Feeds

Is Gartner's Magic Quadrant really magic?

The disappearing act

Providing a secure and efficient Helpdesk

Database myths and legends (Part 8) In this series we're looking at the myths and legends of the database world; some are true, some false. The myth under the spotlight today is whether Gartner's Magic Quadrant really is magic.

(True, Gartner's Magic Quadrant (MQ) isn't just applied to databases, but its recent application to the area of BI platforms, for me at least, threw into sharp relief the essence of the MQ.)

Magic?

Many formal definitions of magic refer to "supernatural influences", others, for example Merriam-Webster's, includes references to "the art of producing illusions by sleight of hand". It is unlikely that Gartner means either of these; I assume the word is used to mean "great", "wonderful" and/or "fantastic". So, is the MQ really magic?

The Magic Quadrant

I don't need to tell you that the MQ plots the intersection point of two values; on the X axis "Completeness of Vision" and on the Y "Ability to Execute". Nor do I need to tell you that the graph is then divided into four quadrants - Leaders, Challengers, Visionaries and Niche Players. If you work in IT, you are doubtless familiar with the MQ; you know it of old. As it turns out, very old. So old that even Jenni Lehman, group vice president of research operations at Gartner didn't know exactly. She guessed at 15 years – about 1992.

This presumably explains why it is monochrome, two dimensional and static. It was developed in the days even before Death-by-PowerPoint. We are talking about a time when most people didn't have computers so the MQ appeared on paper and foils (ask a passing Grandparent for details).

This makes it a child of its time - but computing has since grown up. Not only have PCs evolved into superb multi-media platforms, we have also learnt a huge amount about how to present complex information to people. It is still a challenge, but people like Edward Tufte have done ground-breaking work and this has been brought into the computer world by pioneers like Hans Rosling. If you really want to see why I think this is important, check out Hans' talk at TED. This work has revolutionised the way we think about and present data and companies like Microsoft, Spotfire and many others are adopting these ideas into commercial products.

Yet Gartner is still giving us a monochrome, two dimensional, static quadrant. You do have to ask why, so I did.

I talked to Jenni Lehman who said "the value of the Gartner MQ is that it is well-understood throughout the IT world. The MQ provides an easy to understand market overview representing complex data gained through rigorous and independent research. The tool provides an excellent starting point for interactions between clients and analysts as they discuss the market from the context of the client environment".

From Gartner's perspective, I have to agree. The MQ clearly does have value for its clients. I think the MQ provides huge brand recognition for the company. Given all of that, you can see why change is low on the agenda. But, as a modern consumer of MQs, I have a different perspective. Every time one appears in a presentation I find myself groaning (in earlier times inaudibly; as time passes, more and more loudly).

There are two fundamental reasons.

Firstly, I know that we can do much better than this. For example, given the graphical representation in Figure 1, Company E looks good with D and F close behind.

Shows a basic form of Gartner's Magic Quadrant.

Suppose I tell you that size represents, say, sales strategy and colour, Business model (blue good, red bad), so we get Figure 2:

Shows an enhanced form of Gartner's Magic Quadrant.

Does your opinion change at all? Now suppose we add, in Figure 3, lines that show changes in the recent past:

Shows a more enhanced form of Gartner's Magic Quadrant.

We could even animate the blobs so they move across the screen, the size and colour changing as they moved.

I'm certainly not suggesting that this is an optimal solution (I chose the parameters at random). What I am suggesting is that it is possible to convey a more complete picture in a single graphic without overloading the human brain.

The second major problem I have with the MQ is that, while it does simplify complex data into a digestible form, along the way it erodes the detail in a way I find alarming. To see why, you have to look at how the MQ is produced.

Gartner first defines and describes the market and selects the relevant companies. Then it uses 15 criteria (ranging from product/service to geographical strategy) to evaluate each company.

Incidentally, there is very good evidence that Gartner performs this work very diligently and with great competence. I have absolutely no doubt of the worth of Gartner's work: my concern is with the MQ, not the fundamental research work that the company performs.

In preparing this article, I talked to several companies who have been assessed by Gartner and all reported the same: Gartner does an excellent job of collecting the raw data. For example, when I asked Seán Jackson, marketing manager for Kognitio, he said: "Gartner certainly did due diligence with regards to interviewing our clients. It was very thorough."

So, Gartner collects a huge amount of valuable information; in analytical terms, it collects data over 15 different dimensions. But to produce the MQ it then proceeds to amalgamate seven of the evaluation criteria into one dimension (ability to execute) and the remaining eight into another (completeness of vision). fifteen dimensions of data are collapsed into two.

This is where the problem lies and is best illustrated by an example. Two of the initial criteria Gartner uses are Sales Strategy and Business Model. Suppose that a company develops a world class sales strategy. As it turns out, by error, this strategy fatally compromises its business model. This was clearly a poor choice (but let's face it, one that has been made by several computer companies) and you would probably wish to avoid investing in the company's products.

Let's imagine that Gartner spots the flaw and scores the business model appropriately, but also scores the sales strategy highly, since it is world class. The problem is that when these scores are aggregated with six others into completeness of vision the detail can easily be lost.

I'm not suggesting that such a crude example is likely in practice. But even the process of aggregating data with identical meaning (such as sales figures for a given product) is known to obscure detail. Aggregating non-identical data has a much greater potential to obscure vital information.

Summary

Time has not been kind to the MQ and the magic is long gone.

Does this mean that I think that Gartner is somehow discredited, its analysis flawed? No, that's part of the problem. I have great respect for the value that Gartner's analysis brings to its clients. It is the MQ that has had its day and now is the time for the final magic – the disappearing act.

Internet Security Threat Report 2014

More from The Register

next story
UNIX greybeards threaten Debian fork over systemd plan
'Veteran Unix Admins' fear desktop emphasis is betraying open source
Netscape Navigator - the browser that started it all - turns 20
It was 20 years ago today, Marc Andreeesen taught the band to play
Redmond top man Satya Nadella: 'Microsoft LOVES Linux'
Open-source 'love' fairly runneth over at cloud event
Chrome 38's new HTML tag support makes fatties FIT and SKINNIER
First browser to protect networks' bandwith using official spec
Google+ goes TITSUP. But WHO knew? How long? Anyone ... Hello ...
Wobbly Gmail, Contacts, Calendar on the other hand ...
Admins! Never mind POODLE, there're NEW OpenSSL bugs to splat
Four new patches for open-source crypto libraries
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.