Apple’s Siri Named 2018’s Most Improved AI Assistant: Loup Ventures

A new analysis of AI assistants pegs Apple’s Siri on HomePod as the year’s most-improved.

Apple analyst Gene Munster of Loup Ventures recently tested the big four AI assistants — Siri, Alexa, Google Assistant, and Cortana — by asking them a series of 800 questions each on the HomePod, Amazon Echo, Google Home Mini, and Harmon Kardon Invoke, respectively.

Munster explains the methodology behind the IQ test, making it as simple and straightforward as possible by asking the same questions across the four services. Here’s how they explain it:

We asked each smart speaker the same 800 questions, and they were graded on two metrics: 1. Did it understand what was said? 2. Did it deliver a correct response? The question set, which is designed to comprehensively test a smart speaker’s ability and utility, is broken into 5 categories:

• Local – Where is the nearest coffee shop?
• Commerce – Can you order me more paper towels?
• Navigation – How do I get to uptown on the bus?
• Information – Who do the Twins play tonight?
• Command – Remind me to call Steve at 2 pm today.

According to the results, Siri correctly answered a respective 74.6 percent of the questions, showing a drastic improvement over last year’s test results of 52.3 percent.

Siri still wasn’t able to top Google Assistant, which managed to correctly answer an impressive 87.9 percent of the questions. Alexa and Cortana brought up the rear of the pack, correctly answering 72.5 percent and 63.4 percent of the questions, respectively.

The ability for the digital personal assistants to understand the query was much closer, even if Assistant once again has a demanding lead with 100 percent of questions understood. Meanwhile, Siri, Alexa, and Cortana understood 99.6 percent, 99.0 percent, and 99.4 percent of the questions, respectively.



Google Assistant managed to correctly answer the most questions in four out of five of the categories mentioned above, but lost to Siri in the “Command” category:

HomePod’s lead in this category may come from the fact that the HomePod will pass on full SiriKit requests like those regarding messaging, lists, and basically anything other than music to the iOS device paired to the speaker. Siri on iPhone has deep integration with email, calendar, messaging, and other areas of focus in our Command category. Our question set also contains a fair amount of music-related queries, which HomePod specializes in.

According to Munster, the HomePod’s drastic improvement can be attributed to “the enabling of more domains in the past year,” as Apple has recently released a number of software updates, improving the speaker’s ability to make and receive phone calls, answer questions, and perform tasks.

Read Loup Ventures‘ complete analysis of today’s AI assistants here.