Synopsis

Market Research is great at compiling the right data, but not so good at making it easy to use. This isn’t about “storytelling”. It’s about the data itself, and clarity on what delivering it actually means. Getting the data out of siloes like tabs and SPSS where it cannot adequately be mined, and out into the big wide world as Tidy Data.

The Problem

Esomar recently reported that Global Market Research growth is at its highest since 2010, driven by new “market research” sectors such as web analytics and social media monitoring. But this joyful headline hides mixed emotions. How many huge traditional quant projects have been axed in recent years, as business gets used to the slick, self-service nature of new sources of data? Were they really not good value for money? Or was it all in the presentation?

Market Research agencies have invested to meet the challenge, with the result that research is now much better at tech, but it still lags behind in its communcation of data. Analytics data is available in engaging dashboard formats at the click of a mouse whereas brand trackers still languish in massive tabulations files or SPSS, one file per month, waiting for an over-worked researcher to scroll through for the one key number, to pop into the MD’s monthly PowerPoint deck. Which he or she never reads.

Tabs remain our default delivery medium. Usually because that’s what the client asks for, preferring to save on the presentation costs and do the analysis in-house. Tabs were born decades ago, from a need to summarise data without coding, in an era where we lacked the desktop computing skills to do this ourselves. With our pathological need to be complete, tab decks tend to be huge. They may be right, they may be complete, but they are not useful to the modern data user.

The Solution

Tidy Data

The minimum standard we should be aiming for is Tidy Data. This concept, championed by data scientist Hadley Wickham, is about data being shaped for analysis, modelling and visualisation. A Tidy Research Data table will have:

  1. one variable per column
  2. one case per row
  3. one table per project (one big rectangle of data)

Compare this to the MR standard. Data often delivered in excel tabs formatted for print; split across sheets and tables, with titles, blank columns/rows and even images separating sections of data.

The implication is that these data tables constitute “delivery”, the agency’s final means of communicating the data without regard for where it goes next.

This is undesirable for a number of reasons. Firstly, for anyone wanting to conduct further analysis on the data, their job just became a lot harder. Say you’re an analyst with the task of summarising a batch of reports of media brands’ website traffic. The data is delivered in multiple reports split by target audience or timescale. With the data in disparate locations, even calculating an average is far from a trivial pursuit. How do you write formulas for data across different files, tabs, and columns? You could, but if you want to get it right you probably shouldn’t.

In the optimal pipeline of analysis, from data source to communication, tidying shouldn’t be the bit taking up time and creating risk of error. Understanding is where we need to invest our efforts:


Source: R For Data Science

With data formatted inefficiently, even tidying will often be beyond the capabilities of the average research analyst, and the final communication becomes the spreadsheet itself or a powerpoint deck that takes weeks to build with lots of error prone copy + pasting. The understanding stage has been bypassed, and the optimal outcome missed.

Visualising your data

It’s widely known that the human brain is much better at interpreting visuals as opposed to numbers and text. Here’s a quick example:

As Maarten rightly points out, this data seems ripe for plotting on an x + y axis. How quickly can you get a solid understanding of what the average oil price has been doing in the last 10 years from the tweet above? Now compare that with a quick glance at a chart of the same data, below:


In the attention poor world, there’s only going to be one winner. It’s therefore vital that we start getting our data out of spreadsheets and into effective data visualisations if we want our research to realise its value. And there’s no better place to do this than the modern day web-browser…

Getting your data in the browser

Browsers are where people expect to find information. And it should be no different when it comes to making sense of your data. Data is what the most cutting edge front-end technologies are built for, and utilising them to build rich web applications is getting easier and easier. The only real prerequisite is tidy data!

At Culture of Insight, we increasingly use R to develop solutions for our clients’ data projects. From reproducible workflows to world reknowned graphing libraries, it literally sets data free where the many popular BI software apps build walled gardens, and charge extortionate fees. And with the shiny package, all of this can be built into bespoke web applications that meet specific project needs and utilise the latest web technologies.

Here’s a simple demo app we built by tidying up some dummy web-traffic data files, previously stored in formatted Excel reports. By munging all the data in one big beautiful tidy rectangle, the user can use filters and selectors in the browser to manipulate the data to their specifications. The absorbing interactivity allows users to quickly strip back to the data they’re interested in and compare brand performance across a variety of metrics. It has has the depth of a thousand PowerPoint slides but the size of one.

comscore app click the image to launch the app

Summary

Armed with your research data in a tidy format, you have the opportunity to stretch the value of your market research to its full potential by furthering analysis opportunities and delivering the results in more engaging ways. Prioritise the communication of the data to the same degree as the collection of the data and visualise it in the browser for maximum accesibility and impact.

This may require a shift from the status quo of market research reporting, but the level of investment or thought required to do this is minimal, the potential benefits transformational.

James Smythe is the MD of Culture of Insight, a media research and analytics app development company working for clients such as Sky, Exterion, Global Radio, Hachette Books, Shortlist Media, News UK and the Royal Shakespeare Company. @datasetfree cultureofinsight.com