How not to survey your customers

A great customer turns bad survey questions into valuable feedback.

  • alert
  • submit to reddit

The Power of One eBook: Top reasons to choose HP BladeSystem

Blog [David Norfolk says: I don't like most surveys I see – they're normally from the armed and dangerous wing of the PR industry – but they can be useful, when done properly. So I was interested when Richard Collins (he wrote a piece for Reg Dev on test-driven development here earlier this year) pointed me at this piece from one of his staff – and I thought it was worth sharing with our readers – , Editor]

Bob Cramblitt says:

We’ve all seen them, been frustrated by them, and sometimes been the culprits producing them : the surveys from hell, larded with vague questions, multiple choices that don’t represent any of your choices, off-target assumptions, conclusions that have been reached before the results are in.

Red Gate Software thought its survey was different – more enlightened. And most of it was. But, there were the two questions so poorly designed that people felt compelled to comment, none more elegantly than Mark McGinty.

“It’s amazing that people even took the time to complain, much less honour us with the kind of insight Mark provided,” says Richard Collins, the Red Gate brand manager who compiled the survey. “We not only received a nice tutorial on how a survey question should and shouldn’t be phrased, but we have a great user perspective on issues such as price, performance and ease of use.”

Bob Cramblitt: The offending questions

Let’s first get done with the offending questions, so we can delve into the wisdom of The Great McGinty.

Question 14 followed up on questions about home improvement and the tools that respondents own or wished they owned (there was a method to this madness, but it would take too much time to explain). It asked respondents to rank those qualities they look for in a power tool from 1 (most important) to 9 (least important). The choices were product design, wide range of accessories, power, handling, never breaks down, what the professionals use, multi-functionality, accessible price, and precision.

Question 15 got back to software, asking respondents to rank the qualities they look for in a software tool from 1 (most important) to 9 (least important). Choices included feature rich, support and upgrades, good reputation, speed, attractive UI, good price, ease of use, never crashes, and accuracy.

Both questions forced respondents to place a separate rating value for each characteristic, in a specific order of importance.

Leaving out some complimentary remarks about Red Gate products, [Of course – Ed] here is McGinty’s response, edited for length and consistency.

Mark McGinty's contribution:

The problem with these questions, McGinty says, isn't that they are unclear. It’s that the importance of those characteristics can’t often (maybe can't ever) be placed in an exact order. And even if they could, that order would be shifting constantly, depending on variations in context that are practically infinite.

Performance and price point are similar in that the importance of either/both tends to be relative to what else is available, and at which end of the extreme hey happen to live. The slower something is at completing its job[s], the more important its performance becomes to me, because it causes me to focus on that aspect.

Likewise with price – price is always something of a hinge point in a decision to buy. Does that make it the most important characteristic? Well, maybe – it depends on how much it costs! :-) I once eval'ed this truly stellar protocol analyzer at a client’s request, then just about soiled myself when I found out it cost $25,000 to license! Very difficult for me to picture a positive ROI for that one.

So at a certain level of exorbitance, price can become the most important factor. I think the best case is when something is priced moderately enough so that the specific price of that specific something is rendered fairly unimportant – it’s so much easier that way. In a perfect world price would never be the deciding factor, but this world has constraints, and as such, it’s important.

Performance is special in that it invariably depends on various environmental factors, each of which is subject to ongoing price vs. performance analysis. In the case of SQL Compare, for example, I know I could boost performance by paying for more upstream bandwidth. Case in point, two identical databases: one on my local system took around six seconds to make a snapshot; the other across a WAN link took a minute and a half. My connectivity bill has nothing to do with SQL Compare, but it heavily affects its performance.

Escaping the “lizards”

Ease of use is yet another quagmire – an ambiguous term. I tend to focus on how intuitive a UI is in the context of the problem it’s meant to solve. I also focus on how tedious it becomes to use repeatedly. I prefer complexity that equates to power over mindless “lizards” (my pet name for wizards) that pretend to enable anyone to do anything.

I don’t want a UI that tries to spoon-feed a concept to me every time I use it – like the way the .zip shell extension works in XP: I know a .zip contains files/folders, and I know I'll need a target folder in which to extract them; why must I endure this relentless UI? To do 25 or more of them in a sitting is easy (intellectually), yet at the same time, excruciating!

I like a UI that I can figure out as I use it. If I need help before I can even scratch the surface of some app, I'm usually a little bent by that alone, depending upon how well I understand the concepts involved. Conversely, I have no qualms about using an online reference, provided it’s adequately keyworded and reasonably detailed, to figure out how to do intricate tasks. Sometimes things that aren’t immediately intuitive turn out to be that way after all, given a good explanation.

Ease of use is not a simple characteristic to quantify. Determining whether or not something fits that bill depends heavily on its purpose and intended users. As such, it’s difficult to say how important this is, either before or after purchase – it may turn out to be way more important than anyone anticipated! Bottom line, its place in the order of importance can’t be pinned without some attached context.

Mcginty's final word on surveys

A survey is a set of questions designed to extract my opinions, so my underlying assumptions are: a) there are no wrong answers; and, b) there should be no questions I can’t answer. In practice, it works out that way more often than not.

Question 15 in the Red Gate survey, however, left me no way to respond that I didn't feel substantially misrepresented my opinion in some way.

Surveys must allow multiple characteristics to be assigned equal importance. It must be possible for the underlying inference logically drawn from the answer[s] to a question to be entirely positive [or negative]; exclusivity of choices cannot force the answer to defy real life or actual opinion.

Bob Cramblitt: So much for the garbage in, garbage out cliché.

It just goes to show, Cramblitt says, that if you engage people with good intention, they might not only forgive your survey sins, but reward you for them. It helps, of course, if one of those people is a Mark McGinty.

Bob Cramblitt is a technology writer and Marketing Consultant at Red Gate, and has blithely inflicted some bad surveys on unfortunate readers.

Mark McGinty landed his first real programming position in 1990, working on a Department of Defense mission planning app written for Windows 2.11. Since then he’s worked as a contractor on numerous (C, C++, VB, HTML, VBS, ECMA Script and T-SQL) projects for almost as many companies, with some occasional network consulting projects in between. For the last seven years one client has monopolized his time with work that includes optimizing schema and SQL statements.

HP ProLiant Gen8: Integrated lifecycle automation

More from The Register

next story
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
Do YOU work at Microsoft? Um. Are you SURE about that?
Nokia and marketing types first to get the bullet, says report
Microsoft takes on Chromebook with low-cost Windows laptops
Redmond's chief salesman: We're taking 'hard' decisions
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
EU dons gloves, pokes Google's deals with Android mobe makers
El Reg cops a squint at investigatory letters
Chrome browser has been DRAINING PC batteries for YEARS
Google is only now fixing ancient, energy-sapping bug
Big Blue Apple: IBM to sell iPads, iPhones to enterprises
iOS/2 gear loaded with apps for big biz ... uh oh BlackBerry
prev story


Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.