Dr. Tali Dekel (Weizmann Institute)
Dr. Tali Dekel of the laboratory for the study of computer vision in the Computer Science and Applied Mathematics Department at the Weizmann Institute is one of the world’s leading researchers in generative AI. She focuses on the hidden capabilities of existing large-scale deep-learning models and her research with Google led to the development of the recently released video generation service Lumiere.
The rapid rise of generative AI is mind-boggling. In just a few years, these user-friendly applications have become widely accessible, allowing millions to create realistic text, images, and even videos. These AI tools not only mimic human creativity, but also have the potential to conjure entirely new realities.
Once a distant dream, large language models (LLMs) have exploded in capability, surprising even the experts. Decades of research culminated in models that craft complex, believable text. This sudden leap sparked a surge of interest in AI that blends text with visual data, like images and videos. Now, with a simple prompt or reference image, users can conjure realistic videos – from bustling cityscapes to a whimsical squirrel moonwalk. However, despite these awe-inspiring feats, limitations remain. Deep learning networks, particularly in video generation, still have a restricted operational scope. This is the current frontier for researchers, pushing the boundaries of what LLMs can achieve.
Will you offer us a hand? Every gift, regardless of size, fuels our future.
Your critical contribution enables us to maintain our independence from shareholders or wealthy owners, allowing us to keep up reporting without bias. It means we can continue to make Jewish Business News available to everyone.
You can support us for as little as $1 via PayPal at [email protected].
Thank you.
“I define our area of research as ‘re-rendering reality,’ in other words, recreating the visual world using computational tools,” said Dr. Tali Dekel. “We analyze images and videos and focus on their specific aspects, and then we create a new version with different characteristics. My goal is to enhance the way that we see the world, to give us more creativity and even a new kind of interaction with the visual data.”
Dr. Tali Dekel wears two hats: researcher at the Weizmann Institute and Staff Research Scientist at Google. Her Weizmann research tackles limitations in existing AI models, while at Google, she’s on the cutting edge, developing new ones. One such innovation is Lumiere, a groundbreaking text-to-video model recently unveiled.
Lumiere takes a short text description or a reference image and creates impressive videos. It can even edit existing videos. Imagine feeding Lumiere text describing a woman running in a park, and it generates videos of her morphing into figures made of wooden blocks, colorful bricks, or even flowers!
Google’s Lumiere is a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion — a pivotal challenge in video synthesis. To this end, it has a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution — an approach that inherently makes global temporal consistency difficult to achieve.
Luniere says that by deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, its model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales.
