Is Gartner's Magic Quadrant really magic?

The disappearing act

Providing a secure and efficient Helpdesk

Database myths and legends (Part 8) In this series we're looking at the myths and legends of the database world; some are true, some false. The myth under the spotlight today is whether Gartner's Magic Quadrant really is magic.

(True, Gartner's Magic Quadrant (MQ) isn't just applied to databases, but its recent application to the area of BI platforms, for me at least, threw into sharp relief the essence of the MQ.)


Many formal definitions of magic refer to "supernatural influences", others, for example Merriam-Webster's, includes references to "the art of producing illusions by sleight of hand". It is unlikely that Gartner means either of these; I assume the word is used to mean "great", "wonderful" and/or "fantastic". So, is the MQ really magic?

The Magic Quadrant

I don't need to tell you that the MQ plots the intersection point of two values; on the X axis "Completeness of Vision" and on the Y "Ability to Execute". Nor do I need to tell you that the graph is then divided into four quadrants - Leaders, Challengers, Visionaries and Niche Players. If you work in IT, you are doubtless familiar with the MQ; you know it of old. As it turns out, very old. So old that even Jenni Lehman, group vice president of research operations at Gartner didn't know exactly. She guessed at 15 years – about 1992.

This presumably explains why it is monochrome, two dimensional and static. It was developed in the days even before Death-by-PowerPoint. We are talking about a time when most people didn't have computers so the MQ appeared on paper and foils (ask a passing Grandparent for details).

This makes it a child of its time - but computing has since grown up. Not only have PCs evolved into superb multi-media platforms, we have also learnt a huge amount about how to present complex information to people. It is still a challenge, but people like Edward Tufte have done ground-breaking work and this has been brought into the computer world by pioneers like Hans Rosling. If you really want to see why I think this is important, check out Hans' talk at TED. This work has revolutionised the way we think about and present data and companies like Microsoft, Spotfire and many others are adopting these ideas into commercial products.

Yet Gartner is still giving us a monochrome, two dimensional, static quadrant. You do have to ask why, so I did.

I talked to Jenni Lehman who said "the value of the Gartner MQ is that it is well-understood throughout the IT world. The MQ provides an easy to understand market overview representing complex data gained through rigorous and independent research. The tool provides an excellent starting point for interactions between clients and analysts as they discuss the market from the context of the client environment".

From Gartner's perspective, I have to agree. The MQ clearly does have value for its clients. I think the MQ provides huge brand recognition for the company. Given all of that, you can see why change is low on the agenda. But, as a modern consumer of MQs, I have a different perspective. Every time one appears in a presentation I find myself groaning (in earlier times inaudibly; as time passes, more and more loudly).

There are two fundamental reasons.

Firstly, I know that we can do much better than this. For example, given the graphical representation in Figure 1, Company E looks good with D and F close behind.

Shows a basic form of Gartner's Magic Quadrant.

Suppose I tell you that size represents, say, sales strategy and colour, Business model (blue good, red bad), so we get Figure 2:

Shows an enhanced form of Gartner's Magic Quadrant.

Does your opinion change at all? Now suppose we add, in Figure 3, lines that show changes in the recent past:

Shows a more enhanced form of Gartner's Magic Quadrant.

We could even animate the blobs so they move across the screen, the size and colour changing as they moved.

I'm certainly not suggesting that this is an optimal solution (I chose the parameters at random). What I am suggesting is that it is possible to convey a more complete picture in a single graphic without overloading the human brain.

The second major problem I have with the MQ is that, while it does simplify complex data into a digestible form, along the way it erodes the detail in a way I find alarming. To see why, you have to look at how the MQ is produced.

Gartner first defines and describes the market and selects the relevant companies. Then it uses 15 criteria (ranging from product/service to geographical strategy) to evaluate each company.

Incidentally, there is very good evidence that Gartner performs this work very diligently and with great competence. I have absolutely no doubt of the worth of Gartner's work: my concern is with the MQ, not the fundamental research work that the company performs.

In preparing this article, I talked to several companies who have been assessed by Gartner and all reported the same: Gartner does an excellent job of collecting the raw data. For example, when I asked Seán Jackson, marketing manager for Kognitio, he said: "Gartner certainly did due diligence with regards to interviewing our clients. It was very thorough."

So, Gartner collects a huge amount of valuable information; in analytical terms, it collects data over 15 different dimensions. But to produce the MQ it then proceeds to amalgamate seven of the evaluation criteria into one dimension (ability to execute) and the remaining eight into another (completeness of vision). fifteen dimensions of data are collapsed into two.

This is where the problem lies and is best illustrated by an example. Two of the initial criteria Gartner uses are Sales Strategy and Business Model. Suppose that a company develops a world class sales strategy. As it turns out, by error, this strategy fatally compromises its business model. This was clearly a poor choice (but let's face it, one that has been made by several computer companies) and you would probably wish to avoid investing in the company's products.

Let's imagine that Gartner spots the flaw and scores the business model appropriately, but also scores the sales strategy highly, since it is world class. The problem is that when these scores are aggregated with six others into completeness of vision the detail can easily be lost.

I'm not suggesting that such a crude example is likely in practice. But even the process of aggregating data with identical meaning (such as sales figures for a given product) is known to obscure detail. Aggregating non-identical data has a much greater potential to obscure vital information.


Time has not been kind to the MQ and the magic is long gone.

Does this mean that I think that Gartner is somehow discredited, its analysis flawed? No, that's part of the problem. I have great respect for the value that Gartner's analysis brings to its clients. It is the MQ that has had its day and now is the time for the final magic – the disappearing act.

Internet Security Threat Report 2014

More from The Register

next story
Microsoft on the Threshold of a new name for Windows next week
Rebranded OS reportedly set to be flung open by Redmond
Business is back, baby! Hasta la VISTA, Win 8... Oh, yeah, Windows 9
Forget touchscreen millennials, Microsoft goes for mouse crowd
Apple: SO sorry for the iOS 8.0.1 UPDATE BUNGLE HORROR
Apple kills 'upgrade'. Hey, Microsoft. You sure you want to be like these guys?
SMASH the Bash bug! Apple and Red Hat scramble for patch batches
'Applying multiple security updates is extremely difficult'
ARM gives Internet of Things a piece of its mind – the Cortex-M7
32-bit core packs some DSP for VIP IoT CPU LOL
Lotus Notes inventor Ozzie invents app to talk to people on your phone
Imagine that. Startup floats with voice collab app for Win iPhone
prev story


Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.