Researchers at NVIDIA research have introduced a breakthrough in generating videos based on textual prompts by allowing users to control the trajectory and behavior of objects in the synthesized videos. This innovative approach provides a high-level interface through which users can specify the position of an object at various points in the video by providing bounding boxes (bboxes) and corresponding text prompts.
By editing spatial and temporal attention maps during the initial denoising diffusion steps, users can concentrate activation at the desired location of the object. Importantly, this approach does not disrupt the learned text-image association and requires minimal code modifications.
One of the key features of this new method is the ability to keyframe the bounding box, enabling users to control the size and perspective effects of the object. Additionally, keyframing the text prompt allows users to influence the behavior of the subject in the synthesized video.
This intuitive approach offers casual users a seamless video storytelling tool that allows for modifying the trajectory and behavior of the subject over time. By integrating the synthesized subject into a specified environment, users can create natural outcomes, including perspective effects, accurate object motion, and interactions between objects and their surroundings.
The best part is that this method is computationally efficient and does not require model finetuning, training, or online optimization. By utilizing the power of the underlying diffusion model, it produces high-quality outputs with minimal effort from the user.
While this approach brings significant advancements in controlling object trajectories in synthesized videos, some challenges still remain, such as difficulties generating accurate attributes for multiple objects or deformed objects. However, the researchers continue to work on refining the method to overcome these limitations and enhance the user experience.
This breakthrough research opens up new possibilities in video generation, allowing casual users to create dynamic and engaging videos with ease. To learn more about this research, check out the paper and project by the researchers at NVIDIA research.
The source of the article is from the blog bitperfect.pe