Feeds

How not to survey your customers

A great customer turns bad survey questions into valuable feedback.

  • alert
  • submit to reddit

High performance access to file storage

Blog [David Norfolk says: I don't like most surveys I see – they're normally from the armed and dangerous wing of the PR industry – but they can be useful, when done properly. So I was interested when Richard Collins (he wrote a piece for Reg Dev on test-driven development here earlier this year) pointed me at this piece from one of his staff – and I thought it was worth sharing with our readers – , Editor]

Bob Cramblitt says:

We’ve all seen them, been frustrated by them, and sometimes been the culprits producing them : the surveys from hell, larded with vague questions, multiple choices that don’t represent any of your choices, off-target assumptions, conclusions that have been reached before the results are in.

Red Gate Software thought its survey was different – more enlightened. And most of it was. But, there were the two questions so poorly designed that people felt compelled to comment, none more elegantly than Mark McGinty.

“It’s amazing that people even took the time to complain, much less honour us with the kind of insight Mark provided,” says Richard Collins, the Red Gate brand manager who compiled the survey. “We not only received a nice tutorial on how a survey question should and shouldn’t be phrased, but we have a great user perspective on issues such as price, performance and ease of use.”

Bob Cramblitt: The offending questions

Let’s first get done with the offending questions, so we can delve into the wisdom of The Great McGinty.

Question 14 followed up on questions about home improvement and the tools that respondents own or wished they owned (there was a method to this madness, but it would take too much time to explain). It asked respondents to rank those qualities they look for in a power tool from 1 (most important) to 9 (least important). The choices were product design, wide range of accessories, power, handling, never breaks down, what the professionals use, multi-functionality, accessible price, and precision.

Question 15 got back to software, asking respondents to rank the qualities they look for in a software tool from 1 (most important) to 9 (least important). Choices included feature rich, support and upgrades, good reputation, speed, attractive UI, good price, ease of use, never crashes, and accuracy.

Both questions forced respondents to place a separate rating value for each characteristic, in a specific order of importance.

Leaving out some complimentary remarks about Red Gate products, [Of course – Ed] here is McGinty’s response, edited for length and consistency.

Mark McGinty's contribution:

The problem with these questions, McGinty says, isn't that they are unclear. It’s that the importance of those characteristics can’t often (maybe can't ever) be placed in an exact order. And even if they could, that order would be shifting constantly, depending on variations in context that are practically infinite.

Performance and price point are similar in that the importance of either/both tends to be relative to what else is available, and at which end of the extreme hey happen to live. The slower something is at completing its job[s], the more important its performance becomes to me, because it causes me to focus on that aspect.

Likewise with price – price is always something of a hinge point in a decision to buy. Does that make it the most important characteristic? Well, maybe – it depends on how much it costs! :-) I once eval'ed this truly stellar protocol analyzer at a client’s request, then just about soiled myself when I found out it cost $25,000 to license! Very difficult for me to picture a positive ROI for that one.

So at a certain level of exorbitance, price can become the most important factor. I think the best case is when something is priced moderately enough so that the specific price of that specific something is rendered fairly unimportant – it’s so much easier that way. In a perfect world price would never be the deciding factor, but this world has constraints, and as such, it’s important.

Performance is special in that it invariably depends on various environmental factors, each of which is subject to ongoing price vs. performance analysis. In the case of SQL Compare, for example, I know I could boost performance by paying for more upstream bandwidth. Case in point, two identical databases: one on my local system took around six seconds to make a snapshot; the other across a WAN link took a minute and a half. My connectivity bill has nothing to do with SQL Compare, but it heavily affects its performance.

Escaping the “lizards”

Ease of use is yet another quagmire – an ambiguous term. I tend to focus on how intuitive a UI is in the context of the problem it’s meant to solve. I also focus on how tedious it becomes to use repeatedly. I prefer complexity that equates to power over mindless “lizards” (my pet name for wizards) that pretend to enable anyone to do anything.

I don’t want a UI that tries to spoon-feed a concept to me every time I use it – like the way the .zip shell extension works in XP: I know a .zip contains files/folders, and I know I'll need a target folder in which to extract them; why must I endure this relentless UI? To do 25 or more of them in a sitting is easy (intellectually), yet at the same time, excruciating!

I like a UI that I can figure out as I use it. If I need help before I can even scratch the surface of some app, I'm usually a little bent by that alone, depending upon how well I understand the concepts involved. Conversely, I have no qualms about using an online reference, provided it’s adequately keyworded and reasonably detailed, to figure out how to do intricate tasks. Sometimes things that aren’t immediately intuitive turn out to be that way after all, given a good explanation.

Ease of use is not a simple characteristic to quantify. Determining whether or not something fits that bill depends heavily on its purpose and intended users. As such, it’s difficult to say how important this is, either before or after purchase – it may turn out to be way more important than anyone anticipated! Bottom line, its place in the order of importance can’t be pinned without some attached context.

Mcginty's final word on surveys

A survey is a set of questions designed to extract my opinions, so my underlying assumptions are: a) there are no wrong answers; and, b) there should be no questions I can’t answer. In practice, it works out that way more often than not.

Question 15 in the Red Gate survey, however, left me no way to respond that I didn't feel substantially misrepresented my opinion in some way.

Surveys must allow multiple characteristics to be assigned equal importance. It must be possible for the underlying inference logically drawn from the answer[s] to a question to be entirely positive [or negative]; exclusivity of choices cannot force the answer to defy real life or actual opinion.

Bob Cramblitt: So much for the garbage in, garbage out cliché.

It just goes to show, Cramblitt says, that if you engage people with good intention, they might not only forgive your survey sins, but reward you for them. It helps, of course, if one of those people is a Mark McGinty.

Bob Cramblitt is a technology writer and Marketing Consultant at Red Gate, and has blithely inflicted some bad surveys on unfortunate readers.

Mark McGinty landed his first real programming position in 1990, working on a Department of Defense mission planning app written for Windows 2.11. Since then he’s worked as a contractor on numerous (C, C++, VB, HTML, VBS, ECMA Script and T-SQL) projects for almost as many companies, with some occasional network consulting projects in between. For the last seven years one client has monopolized his time with work that includes optimizing schema and SQL statements.

High performance access to file storage

More from The Register

next story
Android engineer: We DIDN'T copy Apple OR follow Samsung's orders
Veep testifies for Samsung during Apple patent trial
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Windows XP still has 27 per cent market share on its deathbed
Windows 7 making some gains on XP Death Day
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
US taxman blows Win XP deadline, must now spend millions on custom support
Gov't IT likened to 'a Model T with a lot of things on top of it'
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
HP ArcSight ESM solution helps Finansbank
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.