We introduce MoRAG, a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation.
The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs), we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore, by utilizing low-level, part-specific motion information, we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module, improving the performance of motion diffusion models.
We prompt large language models (LLMs) to generate part-specific motion descriptions, which are then used to retrieve corresponding motion sequences for each part. These sequences are composed to create a full-body motion sequence that aligns with the input text.
We illustrate the LLM-generated part-specific outputs for text descriptions alongside their corresponding top-1 retrieval results to demonstrate the effectiveness of our prompt strategy. The HumanML3D ID for the retrieved motions is indicated with the # symbol
Specifically prompting for the position of body parts provides their global orientation, leading to improved motion retrieval.
We avoid retrieving separate motions for left and right body parts to prevent asynchronous movements in the composed full-body motion sequence.
The usage of large language models (LLMs) in MoRAG's pipeline enables it to handle text changes
such as rephrasing, substitution, and spell correction, thereby enhancing the model's ability to
retrieve motion sequences with improved accuracy.
The composition workflow involves combining part-specific motions, Rpart, retrieved from their respective part-specific databases, based on part-specific descriptions generated by the large language model(LLM).
We compare the results of text-to-motion generation between ours and the state-of-the-art diffusion based methods on HumanML3D dataset. Our method achieves better semantic relevance, diversity, and multimodality performances. Indicate best results, indicates second best results.
References: MDM[5]; MotionDiffuse[6]; MLD[2]; ReMoDiffuse[7]; FineMoGen[8].
We present qualitative analysis across three key aspects: (1) Generalizability, (2) Zero-shot performance, and (3) Diversity, for both MoRAG (retrieval model) and MoRAG-Diffuse (generative model).
We provide baseline comparisons for generalizability and zero-shot capabilities against TMR++.
We provide baseline comparisons for generalizability and zero-shot capabilities against ReMoDiffuse.
@InProceedings{MoRAG,
author = {Kalakonda, Sai Shashank and Maheshwari, Shubh and Sarvadevabhatla, Ravi Kiran},
title = {MoRAG - Multi-Fusion Retrieval Augmented Generation for Human Motion},
booktitle = {arXiv preprint},
year = {2024},
}