Google Leverages Anthropic’s Claude to Enhance Gemini AI Performance

Google is leveraging Anthropic’s AI model Claude for performance benchmarking and for evaluating its Gemini AI model’s outputs against those generated by Claude, TechCrunch is reporting.

Focusing on criteria such as accuracy, truthfulness, and verbosity, this meticulous evaluation process allows up to 30 minutes per prompt, ensuring a thorough comparison between the two AI models.

Internal communications have revealed that Claude’s responses often prioritize safety more rigorously than Gemini’s. For instance, Claude may decline to respond to prompts it deems unsafe, such as those involving role-playing as a different AI assistant. In contrast, there have been instances where Gemini’s responses included inappropriate content.

Anthropic’s terms of service explicitly prohibit the use of Claude for developing or training competing AI models without prior approval. When approached for comment, Google did not confirm whether it had obtained such permission from Anthropic.

Shira McNamara, a spokesperson for Google DeepMind, stated that while comparing model outputs is standard industry practice, any suggestion that Anthropic’s models were used to train Gemini is inaccurate. This also complicates Google’s investment in Anthropic, raising questions about the ethical implications of using a competitor’s AI model for internal evaluations.

Additionally, concerns have been raised about the expertise of contractors evaluating Gemini’s responses. Reports indicate that some contractors are required to assess outputs in areas beyond their professional knowledge, potentially leading to inaccuracies.

As AI technology rapidly evolves, the methods employed by leading tech companies to refine and enhance their models are coming under increased scrutiny.