Google Enhances AI Mode with Multimodal Search Capabilities

Google has unveiled a significant enhancement to its AI Mode in Search, introducing multimodal capabilities that allow users to interact with the search engine using images.

This update enables users to upload or capture photos and pose questions related to them, receiving detailed, context-aware responses enriched with links for further exploration.

The integration of Google Lens with a specialized version of the Gemini AI model empowers AI Mode to comprehend the entirety of an image’s scene. This includes understanding the relationships between objects, their materials, colors, shapes, and arrangements. For instance, users can take a picture of a plant and inquire about its species, care instructions, or potential issues.

AI Mode employs a sophisticated “fan-out technique” to process visual inputs effectively. This method involves generating multiple queries from a single image, allowing the system to access a broad spectrum of information.

Initially available to Google One AI Premium subscribers through Google Labs, AI Mode is now being rolled out to millions more users in the United States.

With these enhancements, Google’s AI Mode positions itself as a formidable competitor to other AI-driven search tools like Perplexity and ChatGPT Search.