Is Gartner's Magic Quadrant really magic?
The disappearing act
Free ESG report : Seamless data management with Avere FXT
Database myths and legends (Part 8) In this series we're looking at the myths and legends of the database world; some are true, some false. The myth under the spotlight today is whether Gartner's Magic Quadrant really is magic.
(True, Gartner's Magic Quadrant (MQ) isn't just applied to databases, but its recent application to the area of BI platforms, for me at least, threw into sharp relief the essence of the MQ.)
Magic?
Many formal definitions of magic refer to "supernatural influences", others, for example Merriam-Webster's, includes references to "the art of producing illusions by sleight of hand". It is unlikely that Gartner means either of these; I assume the word is used to mean "great", "wonderful" and/or "fantastic". So, is the MQ really magic?
The Magic Quadrant
I don't need to tell you that the MQ plots the intersection point of two values; on the X axis "Completeness of Vision" and on the Y "Ability to Execute". Nor do I need to tell you that the graph is then divided into four quadrants - Leaders, Challengers, Visionaries and Niche Players. If you work in IT, you are doubtless familiar with the MQ; you know it of old. As it turns out, very old. So old that even Jenni Lehman, group vice president of research operations at Gartner didn't know exactly. She guessed at 15 years – about 1992.
This presumably explains why it is monochrome, two dimensional and static. It was developed in the days even before Death-by-PowerPoint. We are talking about a time when most people didn't have computers so the MQ appeared on paper and foils (ask a passing Grandparent for details).
This makes it a child of its time - but computing has since grown up. Not only have PCs evolved into superb multi-media platforms, we have also learnt a huge amount about how to present complex information to people. It is still a challenge, but people like Edward Tufte have done ground-breaking work and this has been brought into the computer world by pioneers like Hans Rosling. If you really want to see why I think this is important, check out Hans' talk at TED. This work has revolutionised the way we think about and present data and companies like Microsoft, Spotfire and many others are adopting these ideas into commercial products.
Yet Gartner is still giving us a monochrome, two dimensional, static quadrant. You do have to ask why, so I did.
I talked to Jenni Lehman who said "the value of the Gartner MQ is that it is well-understood throughout the IT world. The MQ provides an easy to understand market overview representing complex data gained through rigorous and independent research. The tool provides an excellent starting point for interactions between clients and analysts as they discuss the market from the context of the client environment".
From Gartner's perspective, I have to agree. The MQ clearly does have value for its clients. I think the MQ provides huge brand recognition for the company. Given all of that, you can see why change is low on the agenda. But, as a modern consumer of MQs, I have a different perspective. Every time one appears in a presentation I find myself groaning (in earlier times inaudibly; as time passes, more and more loudly).
There are two fundamental reasons.
Firstly, I know that we can do much better than this. For example, given the graphical representation in Figure 1, Company E looks good with D and F close behind.
Suppose I tell you that size represents, say, sales strategy and colour, Business model (blue good, red bad), so we get Figure 2:
Does your opinion change at all? Now suppose we add, in Figure 3, lines that show changes in the recent past:
We could even animate the blobs so they move across the screen, the size and colour changing as they moved.
I'm certainly not suggesting that this is an optimal solution (I chose the parameters at random). What I am suggesting is that it is possible to convey a more complete picture in a single graphic without overloading the human brain.
The second major problem I have with the MQ is that, while it does simplify complex data into a digestible form, along the way it erodes the detail in a way I find alarming. To see why, you have to look at how the MQ is produced.
Gartner first defines and describes the market and selects the relevant companies. Then it uses 15 criteria (ranging from product/service to geographical strategy) to evaluate each company.
Incidentally, there is very good evidence that Gartner performs this work very diligently and with great competence. I have absolutely no doubt of the worth of Gartner's work: my concern is with the MQ, not the fundamental research work that the company performs.
In preparing this article, I talked to several companies who have been assessed by Gartner and all reported the same: Gartner does an excellent job of collecting the raw data. For example, when I asked Seán Jackson, marketing manager for Kognitio, he said: "Gartner certainly did due diligence with regards to interviewing our clients. It was very thorough."
So, Gartner collects a huge amount of valuable information; in analytical terms, it collects data over 15 different dimensions. But to produce the MQ it then proceeds to amalgamate seven of the evaluation criteria into one dimension (ability to execute) and the remaining eight into another (completeness of vision). fifteen dimensions of data are collapsed into two.
This is where the problem lies and is best illustrated by an example. Two of the initial criteria Gartner uses are Sales Strategy and Business Model. Suppose that a company develops a world class sales strategy. As it turns out, by error, this strategy fatally compromises its business model. This was clearly a poor choice (but let's face it, one that has been made by several computer companies) and you would probably wish to avoid investing in the company's products.
Let's imagine that Gartner spots the flaw and scores the business model appropriately, but also scores the sales strategy highly, since it is world class. The problem is that when these scores are aggregated with six others into completeness of vision the detail can easily be lost.
I'm not suggesting that such a crude example is likely in practice. But even the process of aggregating data with identical meaning (such as sales figures for a given product) is known to obscure detail. Aggregating non-identical data has a much greater potential to obscure vital information.
Summary
Time has not been kind to the MQ and the magic is long gone.
Does this mean that I think that Gartner is somehow discredited, its analysis flawed? No, that's part of the problem. I have great respect for the value that Gartner's analysis brings to its clients. It is the MQ that has had its day and now is the time for the final magic – the disappearing act.
