DIY Market Research: Flip or Flop?

More and more DIY tools let you do research cheaper, faster.  But be careful, because one small mistake can have big consequences.

Everyone enjoys a good DIY home improvement show.  HGTV built a network around watching every day homeowners transform a run down shack into the gem of the neighborhood.  We watch and think, “I want that kitchen!  I could do that!”  But, the truth of the matter is, most of us can’t without the help of a professional contractor. Someone to make sure our cabinets are level and the electrical is up to code.

The last few years in market research have brought a trend towards DIY research. Many of the large sample and technology providers have created self-serve tools for in-house marketers to execute research without the aid of a research firm.  These tools range from simple DIY survey platforms to online communities to very advanced analytics capabilities.

While DIY market research isn’t exactly the same as DIY home remodeling (I don’t know many people who dream of well written questionnaires), there are similarities.  They make it look simple.  They make it sound affordable.  They make it seem low risk.  And, often, the end product doesn’t meet expectations.

We’ve done a lot of research on these DIY tools and found many things that raised eyebrows here at The Seidewitz Group. Here are just a few examples.

• Complex Methodologies made simple (and executed poorly) – Conjoint analysis, and sister methodologies like MaxDiff, are powerful tools that are enjoying a lot of popularity right now.  They’re great for helping marketers understand relative importance of features and create various “what if” product/attribute/pricing scenarios.  But they’re also considered advanced by even experienced research consultants.
  
  For example, at The Seidewitz Group, we don’t run conjoint research on our own.  We partner with an analytics expert to help us build and execute these projects. So it is concerning when a DIY tool claims to be able to deliver Conjoint at the push of a button.
  
  So what’s the risk if you try to do conjoint yourself?  The trouble starts with building the survey.  How many attributes can you test?  Should you include competitive product attributes as a baseline?  How do you handle pricing?  These are not simple questions and require significant experience and expertise to answer.  Designing a conjoint study without this experience creates significant risk of “garbage in, garbage out”.
  
  Analyzing results can be even more complicated.  Building the statistical model is nuanced and not always straightforward. And even if the DIY tool creates a decent model, results are not as straightforward as they appear: experienced market researchers often mistake the relative rankings from conjoint for absolute measures. Someone with little research experience using a DIY tool is going to be highly susceptible to misinterpretation of data—possibly leading to invalid conclusions.
• Just because they call it segmentation, doesn’t mean it is – The word segmentation can strike fear in the heart of the even the most seasoned market researcher.  It is a complex methodology that involves significant sampling, advanced statistical modeling and some heavy-duty planning and execution.  Depending on the size of the target audience, segmentation can require samples in the thousands.  So it is a major red flag when a frequently used DIY website features a case study promoting the platform’s ability to segment audiences based on the results of a small, qualitative online discussion board.
  
  The case study makes the argument that quantitative questions thrown into the board, along with qualitative findings, were a solid basis for behavioral segmentation.  This based on a sample of 30, with four different audiences included.
  
  The problem? Obviously, the sample doesn’t come close to the level needed to have confidence in behavioral segments. But more fundamentally, the DIY site promoted developing segments based on a small amount of qualitative data, without discussing the analytic and modeling side of segmentation.  Database tagging?  Not a mention.  Anyone who follows this DIY methodology is going to end up with suboptimal, unreliable segments.
  
  Bad, bad sample – Everyone loves a bargain and acquiring sample can be expensive.  Some sources offer very low cost general population sample (we’re big fans of Toluna Quick Surveys) but not all sample is the same.  For example, a certain search provider (and future ruler of the Galaxy - rhymes with oogle) has a survey development tool that features free or low cost sample options.  They offer a panel they claim can cost effectively reach students and small business owners; two audiences that can be relatively expensive to acquire.  However, the devil is in the detail.  All of the members of the panel are Android users who subscribe to their Opinion Rewards program.  Yes, all Android users.  Given the high market penetration Apple enjoys with young people, excluding them from a sample of students would obviously introduce bias.  This information isn’t readily advertised in the description of the panel. You have to dig a few pages down.

The risk that is inherent in all of these examples is clear: marketers will make business decisions based on bad data without realizing the risk. It’s a classic case of “you don’t know what you don’t know”. 

So what’s a marketing pro who hears the siren song of “do it yourself” to do? Just like changing a light bulb doesn’t require a professional, not all research requires a full-blown agency.  Here are a few principles that will make sure your in-house DIY research will reach your objectives:

• Keep it simple – Simple surveys or qualitative projects can certainly be managed in-house.  What do we mean by simple? A customer service survey with pre-determined measures that goes out after every sale?  Sure.  A short qualitative interview without probing or other advanced interview techniques?  You bet.  But if you start getting into more complicated qualitative research or advanced quantitative techniques, it’s probably not a good idea to do it yourself.
   
• Use the resources available – There are a ton of online resources that can help you optimize your in-house research and many of them are very good. Templates for surveys, prewritten discussion guides or even pre-populated discussion boards.  Take the time to understand the best practices for the methodologies you're using and follow them.
   
• Get Trained – In the methodologies you’re using but also market research overall.  Burke Institute and Riva both offer training in qualitative research for client-side professionals.  Pick someone in your organization to become the “in-house qual expert” and get them trained to fill that role.  More broadly for your whole team, The Seidewitz Group offers several training courses on everything from understanding “when to use qual versus quant” to “writing good concepts for testing”.  Getting everyone up to speed is important and a key part of that training should always be…
   
• Know when to call in the experts – (Hint: it’s before water is spraying all over the kitchen.)  Deciding what can be done in house and what should be taken to a research firm is no a small task. Sometimes you may just want to talk it through. You’d be surprised how many research consultants would take that call.  They can help you refine objectives and think through the pros and cons of doing research yourself vs. outsourcing it to a pro.  Just like an outside consultant can’t be expected to be an expert on your business, you can’t be expected to be a research expert.  Sometimes the smartest thing you can do is call for help.

DIY research can help in-house marketing departments drive the business forward and manage tight budgets and timelines. But, when it comes to important, high-risk decisions and complicated research methodologies, it always makes sense to talk with a consultant.  A good partner will take the time to understand your objectives and work with you to find the best solution, whether that's DIY or bringing in the pros.  It's entertaining when a first time home flipper removes a load-bearing wall on a home improvement show.  It's not as much fun when your DIY market research leads your business down the wrong path.