Google Research has unveiled Lumiere, a cutting-edge text-to-video diffusion model that brings to life extraordinarily realistic videos from prompts of text or image. Although there has been remarkable progress in the generation of still images by tools such as Midjourney and DALL-E, text-to-video models have not quite reached the same level until recently.
Until the introduction of Lumiere, models like those developed by Pika Labs or Stable Video Diffusion provided fairly decent results, but there remained certain constraints in the realism and fluidity of motion. Lumiere has pushed the boundaries with an innovative approach promising video generation that maintains spatial-temporal coherence to ensure visual consistency and smooth movements.
Lumiere provides a number of functionalities such as generating a 5-second video clip from a text prompt, turning an image prompt into a video, stylizing a video to match an image or text prompt, animating certain parts of a stationary image, and video inpainting which can complete or edit a video scene.
Typically, existing text-to-video models would adopt a cascaded design relying on temporal and spatial super-resolution models, which often resulted in videos with temporal inconsistency or disrupted motion. Contrarily, Lumiere uses a Space-Time U-Net architecture that downsamples the signal in both space and time to process all frames simultaneously. This approach ensures globally coherent motion to achieve high-resolution videos.
A user study by Google Research revealed that users greatly preferred Lumiere videos to those generated by other text-to-video models. Although Lumiere is currently limited to 5-second clips and does not handle scene transitions or multi-shot video scenes, it offers an increased level of realism and visual coherency unmatched by current text-to-video solutions, which tend only to create 3-second clips.
Google acknowledges a potential risk of misuse of technology like Lumiere for creating harmful or fake content. Hence, effective preventive measures such as watermarking videos and safeguarding against copyright issues are a prominent focus before Lumiere is made available for extensive use.
