Story-to-Motion: Generate animation based on text story
Story-to-Motion: Generate continuous character animation based on text story content
The project, developed by SenseTime Technology Research Institute, is capable of processing complex text descriptions and converting these descriptions into specific actions and location information.
It can not only generate a single action, but also generate a series of actions continuously to create a coherent animation effect.
A key feature of Story-to-Motion is its ability to generate infinitely long character animations.
This means that, in theory, as long as the provided text story is long enough and the content is continuous, the system can continuously generate corresponding character actions and behaviors based on the text content, thereby creating a continuous animation sequence.
Main principles of Story-to-Motion:
- Text parsing and action scheduling: First, the system uses a large language model to parse the input long text story. This process involves extracting key information from the text, such as a character’s actions, location, and situation. This information is converted into a series of (text, position) pairs for subsequent action generation.
2. Text-driven action retrieval: The system then retrieves appropriate actions based on the extracted information. This step combines action matching technology, action semantic understanding, and trajectory constraints to ensure that the generated actions are not only consistent with the text content, but also spatially reasonable.
3. Action synthesis and transition processing: The system designs a special progressive mask transformer to process the transition between actions. This transformer solves common problems in motion synthesis, such as unnatural postures and foot sliding, ensuring natural and smooth motion.
4. Unlimited animation generation: Since the system can continuously process action descriptions in text, it can generate infinitely long animation sequences. This means that as long as the text story continues, the animation will continue to be generated accordingly.
Projects and demos
Paper
