How not to survey your customers

A great customer turns bad survey questions into valuable feedback.

  • alert
  • submit to reddit

Combat fraud and increase customer satisfaction

Blog [David Norfolk says: I don't like most surveys I see – they're normally from the armed and dangerous wing of the PR industry – but they can be useful, when done properly. So I was interested when Richard Collins (he wrote a piece for Reg Dev on test-driven development here earlier this year) pointed me at this piece from one of his staff – and I thought it was worth sharing with our readers – , Editor]

Bob Cramblitt says:

We’ve all seen them, been frustrated by them, and sometimes been the culprits producing them : the surveys from hell, larded with vague questions, multiple choices that don’t represent any of your choices, off-target assumptions, conclusions that have been reached before the results are in.

Red Gate Software thought its survey was different – more enlightened. And most of it was. But, there were the two questions so poorly designed that people felt compelled to comment, none more elegantly than Mark McGinty.

“It’s amazing that people even took the time to complain, much less honour us with the kind of insight Mark provided,” says Richard Collins, the Red Gate brand manager who compiled the survey. “We not only received a nice tutorial on how a survey question should and shouldn’t be phrased, but we have a great user perspective on issues such as price, performance and ease of use.”

Bob Cramblitt: The offending questions

Let’s first get done with the offending questions, so we can delve into the wisdom of The Great McGinty.

Question 14 followed up on questions about home improvement and the tools that respondents own or wished they owned (there was a method to this madness, but it would take too much time to explain). It asked respondents to rank those qualities they look for in a power tool from 1 (most important) to 9 (least important). The choices were product design, wide range of accessories, power, handling, never breaks down, what the professionals use, multi-functionality, accessible price, and precision.

Question 15 got back to software, asking respondents to rank the qualities they look for in a software tool from 1 (most important) to 9 (least important). Choices included feature rich, support and upgrades, good reputation, speed, attractive UI, good price, ease of use, never crashes, and accuracy.

Both questions forced respondents to place a separate rating value for each characteristic, in a specific order of importance.

Leaving out some complimentary remarks about Red Gate products, [Of course – Ed] here is McGinty’s response, edited for length and consistency.

Mark McGinty's contribution:

The problem with these questions, McGinty says, isn't that they are unclear. It’s that the importance of those characteristics can’t often (maybe can't ever) be placed in an exact order. And even if they could, that order would be shifting constantly, depending on variations in context that are practically infinite.

Performance and price point are similar in that the importance of either/both tends to be relative to what else is available, and at which end of the extreme hey happen to live. The slower something is at completing its job[s], the more important its performance becomes to me, because it causes me to focus on that aspect.

Likewise with price – price is always something of a hinge point in a decision to buy. Does that make it the most important characteristic? Well, maybe – it depends on how much it costs! :-) I once eval'ed this truly stellar protocol analyzer at a client’s request, then just about soiled myself when I found out it cost $25,000 to license! Very difficult for me to picture a positive ROI for that one.

So at a certain level of exorbitance, price can become the most important factor. I think the best case is when something is priced moderately enough so that the specific price of that specific something is rendered fairly unimportant – it’s so much easier that way. In a perfect world price would never be the deciding factor, but this world has constraints, and as such, it’s important.

Performance is special in that it invariably depends on various environmental factors, each of which is subject to ongoing price vs. performance analysis. In the case of SQL Compare, for example, I know I could boost performance by paying for more upstream bandwidth. Case in point, two identical databases: one on my local system took around six seconds to make a snapshot; the other across a WAN link took a minute and a half. My connectivity bill has nothing to do with SQL Compare, but it heavily affects its performance.

Escaping the “lizards”

Ease of use is yet another quagmire – an ambiguous term. I tend to focus on how intuitive a UI is in the context of the problem it’s meant to solve. I also focus on how tedious it becomes to use repeatedly. I prefer complexity that equates to power over mindless “lizards” (my pet name for wizards) that pretend to enable anyone to do anything.

I don’t want a UI that tries to spoon-feed a concept to me every time I use it – like the way the .zip shell extension works in XP: I know a .zip contains files/folders, and I know I'll need a target folder in which to extract them; why must I endure this relentless UI? To do 25 or more of them in a sitting is easy (intellectually), yet at the same time, excruciating!

I like a UI that I can figure out as I use it. If I need help before I can even scratch the surface of some app, I'm usually a little bent by that alone, depending upon how well I understand the concepts involved. Conversely, I have no qualms about using an online reference, provided it’s adequately keyworded and reasonably detailed, to figure out how to do intricate tasks. Sometimes things that aren’t immediately intuitive turn out to be that way after all, given a good explanation.

Ease of use is not a simple characteristic to quantify. Determining whether or not something fits that bill depends heavily on its purpose and intended users. As such, it’s difficult to say how important this is, either before or after purchase – it may turn out to be way more important than anyone anticipated! Bottom line, its place in the order of importance can’t be pinned without some attached context.

Mcginty's final word on surveys

A survey is a set of questions designed to extract my opinions, so my underlying assumptions are: a) there are no wrong answers; and, b) there should be no questions I can’t answer. In practice, it works out that way more often than not.

Question 15 in the Red Gate survey, however, left me no way to respond that I didn't feel substantially misrepresented my opinion in some way.

Surveys must allow multiple characteristics to be assigned equal importance. It must be possible for the underlying inference logically drawn from the answer[s] to a question to be entirely positive [or negative]; exclusivity of choices cannot force the answer to defy real life or actual opinion.

Bob Cramblitt: So much for the garbage in, garbage out cliché.

It just goes to show, Cramblitt says, that if you engage people with good intention, they might not only forgive your survey sins, but reward you for them. It helps, of course, if one of those people is a Mark McGinty.

Bob Cramblitt is a technology writer and Marketing Consultant at Red Gate, and has blithely inflicted some bad surveys on unfortunate readers.

Mark McGinty landed his first real programming position in 1990, working on a Department of Defense mission planning app written for Windows 2.11. Since then he’s worked as a contractor on numerous (C, C++, VB, HTML, VBS, ECMA Script and T-SQL) projects for almost as many companies, with some occasional network consulting projects in between. For the last seven years one client has monopolized his time with work that includes optimizing schema and SQL statements.

SANS - Survey on application security programs

More from The Register

next story
Next Windows obsolescence panic is 450 days from … NOW!
The clock is ticking louder for Windows Server 2003 R2 users
Ubuntu 14.04 LTS: Great changes, but sssh don't mention the...
Why HELLO Amazon! You weren't here last time
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Ditch the sync, paddle in the Streem: Upstart offers syncless sharing
Upload, delete and carry on sharing afterwards?
Microsoft TIER SMEAR changes app prices whether devs ask or not
Some go up, some go down, Redmond goes silent
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Red Hat to ship RHEL 7 release candidate with a taste of container tech
Grab 'near-final' version of next Enterprise Linux next week
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
prev story


Mainstay ROI - Does application security pay?
In this whitepaper learn how you and your enterprise might benefit from better software security.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.