Photo Source: Loup Ventures
So Siri looks like it’s a little bit smarter this year than in April of 2017. At least that’s what Loup Ventures’ Annual Digital Assistant IQ Test tells us. Does this mean that Siri really is quantifiably better now than a few months ago? Probably? Maybe? Who knows? More to the point- does this test actually matter?
Before we get to that, here is how the numbers actually break down:
Photo Source: Loup Ventures
As you can see here, Siri did score markedly better on this year’s test, up 12.4%. Siri wasn’t alone, either, as Google Assistant and Cortana were also up year over year, as well. Alexa wasn’t put through this particular test last year, because it is performed using the iOS mobile app. That app did not provide full Alexa query capability then, as it does today, so this is its first time in the IQ Test. Ironically, even though Cortana improved year over year, Alexa still beat its score handily.
I think that the information we see here for Siri and Google Assistant is valid, as the questions being asked are the same, and each is being tested using native hardware- Siri on an Apple device running iOS 11.4 (we don’t know which iOS device was used), and Assistant on a Pixel XL (we don’t actually know which version of Android was used here). However, I am not sure we can really trust the numbers for Cortana and Alexa, as they were tested using iOS mobile apps that aren’t native to either digital assistant. I think it would be a much more “apples to apples” comparison to at least test Alexa using a Fire tablet and Cortana on a Surface device.
The way this test is set up and advertised makes it look like a competition, but it really isn’t. There’s no real value in that. This test isn’t controlled or scientific enough to draw those kinds of conclusions. Besides that, the only competition that matters is how each device improves relative to its own previous performance. That can be taken at face value, if we assume that the people doing the testing are maintaining some objectivity.
The real value of this data is finding out if each assistant is moving forward? What is the rate of improvement? As the tests continue on, we will be able to see a trend line form, and see that rate change as the assistant is adapted and improved over time. However, at this point, two points of data isn’t really enough to draw any sweeping conclusions.
So, while it is a positive sign that Siri scored better on Loup Ventures Digital Assistant IQ Test this year, I will not put as much stock in it as I might have in the past. Not yet, at least. Anyone who uses Siri in the real world knows what it can do well, but also knows where it falls down and comes up short. Unfortunately, there are still plenty of the latter, and a 12.4% bump here doesn’t change that. Now, if we see a more of these tests results come back with similar performance increases, then we should also be seeing real-world improvements in Siri, as well. Until then, take this result as a sign that Apple’s AI and machine learning teams are hard at work under new boss John Giannandrea, but also take it with a grain of salt.