You’ll Soon Be Able to Send Video and Screenshares to Google’s Gemini AI

As part of Mobile World Congress, happening this week in Barcelona, Google has teased that its Gemini AI is finally ready to see the world. Literally.

Originally teased as Project Astra in May of last year, Google’s Gemini Live with Video and Gemini Live with Screenshare features each have a new trailer, showing how the search giant plans to catch up with ChatGPT’s Advanced Voice Mode. Soon, Google One AI Premium subscribers will be able to share their phone screens and real-time video with Gemini, which the chatbot will then be able to use to answer questions.

ChatGPT’s Advanced Voice Mode got these features late last year, but it also requires you to shell out at least $20/month for a ChatGPT Plus subscription. While a Google One AI Premium plan is priced identically, it also throws in 2TB of storage for your Drive and Gmail accounts, so depending on your needs, it could be the better pick for you.

In the trailers, Google shows users accessing Gemini Live via the Gemini phone app to open a real-time conversation with the AI, then tapping either a video or screen sharing button at the bottom of the screen to get started. In the Gemini Live with Video demo, we see someone showing the chatbot some vases they’ve just fired as well as some glaze swatches and asking for advice on which one to choose. In the Screenshare demo, the chatbot instead looks at a store listing for a pair of jeans and gives advice on which type of clothes to pair with them. 

I’d love to meet the type of person who would go through all of the effort of spinning and firing a vase without knowing what color to glaze it, but the point is clear. Gemini will soon be able to use real-time video and screen captures as input when answering prompts.

Unfortunately, Google hasn’t yet said much more than that. When Project Astra was originally teased, it boasted such impressive (and creepy) abilities as “being able to tell where you live just by looking out the window.” These trailers seem significantly scaled back in comparison, but they're also clearly just short demos. I’m curious to see how the feature actually functions once users get their hands on it, which Google says will happen “later this month.”