VideoDoodles: Easily insert hand-drawn animations into videos and seamlessly integrate them with video content

8 min readAug 28, 2024

Insert hand-drawn animations into your videos! Traditionally this has been a very difficult task, but VideoDoodles makes it possible.

VideoDoodles is an innovative interactive system that allows users to add hand-drawn animations to videos. These animations not only move with the objects in the video, but also automatically adjust the size and angle to look like they are part of the video. This is very useful for creators who want to add personalized effects or educational content to their videos.

The system allows users to seamlessly blend hand-drawn animations with video content by placing flat canvases in a 3D scene. These canvases can be precisely tracked and adjusted based on the perspective and movement of objects in the video, making the hand-drawn animations look like they are part of the video scene.

What is VideoDoodles good for?

Make the video more interesting: You can use it to add creative hand-drawn content to the video, such as drawing a funny little man or a beautiful rainbow to make the video more attractive.
Better explain the content: For example, in an instructional video, you can use hand-drawn animation to highlight the key points and make it easier for the audience to understand what you want to express.
Easy to use: Whether you are a novice or a professional, you can quickly get started and use it to create professional animation effects.

System Advantages

Simplified creation of 3D effects : The system combines 2D drawing with 3D computer vision technology, making it easy for even users without 3D modeling experience to create 3D animations with perspective and occlusion effects.
Friendly to newbies : One of the design goals of the system is to reduce the learning curve, so that users with no animation experience can quickly get started and create complex animation effects.
Balance between user control and automation : While providing powerful automation capabilities, the system also allows users to precisely control the effects of animations through keyframes to achieve their creative intent.

Key Features

3D Flat Canvas

The system allows users to place 3D flat canvases in the video scene and perform hand-drawn animations on these canvases. These canvases are positioned according to the 3D scene of the video. Users can operate them through a simple 2D interface, and the system automatically handles the position and orientation of the canvas in 3D space. In other words, these canvases will automatically adjust according to the perspective and object movement in the video to ensure the correct position and angle of the animation.
Support for dynamic and static canvases:

Automatic tracking and perspective correction

The system has powerful 3D tracking capabilities that allow the canvas to align with static or dynamic objects in the video. Whether the camera moves or the object moves, the system will automatically adjust the perspective of the canvas, making the animation look as natural as if it is embedded in the video scene.
Allows users to anchor the canvas to static or dynamic objects in the video, allowing the animation to follow the movement and rotation of the object, maintaining a natural and realistic effect.

Occlusion processing

VideoDoodles can automatically handle occlusion effects. When objects in the video block the animation, the system will intelligently handle the occlusion to ensure that the animation is visually consistent with the video content.
Multiple 3D motion effects: Through 3D tracking and rotation control of the canvas, the system can achieve a variety of complex 3D animation effects, such as object rotation, dynamic scaling, etc., making the animation effects more expressive.

Fast creation and efficient output

Get started quickly: Users can create complex animation effects in a short time without a complicated learning process, which is suitable for rapid content creation on social media.
Efficient output: Whether it is a static or dynamic canvas, the system can quickly generate and render the final animation effect, meeting the needs of professional users for efficient creation.

Easy-to-use 2D drawing interface

Simplified 2D drawing interface: Users can draw directly on the 2D interface without worrying about perspective distortion. The system will automatically embed the drawing content into the 3D scene in the video and generate the correct perspective effect.
Balance between automation and manual control: The system automatically completes tracking and perspective correction in most cases, and the user only needs to provide a small amount of keyframe input. If necessary, the user can also manually adjust the position and direction of the canvas for more precise animation control. The user can control the movement trajectory and direction of the canvas by setting keyframes. If the automatic tracking does not meet the requirements, the user can manually adjust the keyframes to further refine the animation effect.

Professional function expansion

Although the system is optimized for beginners, it still retains some features suitable for professional users:

Precise keyframe control: The system allows users to precisely control the movement trajectory and direction of the canvas through keyframes, thereby achieving more complex animation effects.
Integration with existing tools: The system can be used with existing video editing tools to provide users with more creative freedom and possibilities.

Technical methods

1. 3D scene reconstruction and preprocessing

The system first constructs a rough 3D model of the scene by extracting depth information and motion data from the video. This reconstruction allows the system to understand the relative positions and directions of movement of objects in the video.
First, the input video is reconstructed in 3D, which includes calculating the camera pose, depth map and optical flow of each frame. This information is obtained through the latest computer vision technology and can provide accurate 3D geometric information for subsequent animation embedding and tracking.
We use Robust Consistent Video Depth EstimationCOLMAP

2. 3D tracking algorithm for flat canvas

Users can place flat canvases in the 3D scene, which are equivalent to the “drawing board” for drawing hand-drawn animations in the video. The system supports placing these canvases on static or dynamic objects and automatically adjusts them according to the depth and perspective effects of the scene.

Keyframe-driven 3D tracking: The user can place a flat canvas in a certain frame of the video and set its position and orientation. The system will automatically deduce the 3D trajectory of the canvas in other frames of the video. To achieve this goal, the system will use the keyframes specified by the user as hard constraints, and use the scene motion and depth information generated in the preprocessing stage to calculate the 3D position of the canvas in each frame through an optimization algorithm.
3D trajectory optimization based on scene flow: The system uses scene flow estimation to preliminarily determine the 3D trajectory of the canvas, and then further stabilizes and refines the trajectory using the Poisson integration method to reduce jitter and drift.

3. Orientation tracking of dynamic canvas

VideoDoodles uses a custom tracking algorithm to associate the canvas with the object in the video. This algorithm can track the movement and rotation of the object, allowing the canvas to update its position and angle in real time to ensure that the animation is consistent with the video content. This tracking includes not only simple translation, but also supports complex rotation and scaling.

The system supports dynamic canvas, that is, the canvas can change as the object in the video moves. To achieve this function, the system will optimize the rotation matrix of the canvas according to the canvas direction keyframe set by the user and the direction of movement in the scene, so that it is consistent with the movement trajectory of the object.
VideoDoodles leverages existing computer vision techniques, such as depth estimation and object tracking, to obtain scene information in videos. This information is used to generate accurate 3D reconstruction and scene perception, thus enabling more accurate animation placement and tracking.
Posture Optimization: The system uses a manifold-based trust region algorithm (Riemannian Trust Region) to optimize the rotation matrix so that the posture of the canvas can smoothly follow the direction of movement in the scene. At the same time, users can fine-tune the posture by setting key frames.

4. Occlusion processing

The system can identify the occlusion relationship between different objects in the video and automatically handle the occlusion effect of hand-drawn animation. This means that the animation on the canvas will be occluded by the foreground object when necessary, enhancing the realism of the animation and video scene.

5. User interaction and 2D drawing interface

VideoDoodles provides an intuitive 2D interface where users can directly manipulate the canvas in the 3D scene. By simply dragging and rotating, users can set the position and angle of the animation. In addition, the system also supports real-time preview, allowing users to see the animation effect immediately.

Although the system performs complex 3D calculations in the background, users only need to draw through a simple 2D interface on the front end. The system aligns the canvas with the video content and displays perspective correction and occlusion effects in real time during the drawing process. Users can directly draw frame-by-frame animations in this interface without having to consider complex 3D transformations.
Automatic perspective correction: When users draw on the 2D interface, the system will automatically correct the content on the canvas to be consistent with the perspective of the video scene.

6. Keyframe-based user control

Users can set the position and orientation keyframes of the canvas at different frames of the video. The system will automatically interpolate smoothly between these keyframes to generate a continuous animation effect. This method greatly reduces the workload of manual adjustment and improves the efficiency of animation production.
The system allows users to control the 3D trajectory and rotation of animations through a small number of keyframe settings, thereby achieving complex animation effects. Users can freely adjust the position, direction and depth of the canvas between keyframes, and the system will automatically calculate the smooth transition between these keyframes.

……

For more info ↓