How do I go about reporting? Do I need to be a specialist? Isn't this stuff for numbers geeks?
Yes, if by specialist you mean someone who practices a lot.
No, actually this kinda thing is not really ideal for purists. It's too messy.
What are the skills you need to practice?
Lining up behind the "most important goal."
It's good to have lots of things you would like to accomplish. Just make sure they all line up behind a single goal.
Why is it important to know what the goal is? Because without a clear goal, it's very hard to know what to measure. Are you optimizing for short-term gain or long-term pay-off? Quantity or quality? The most new customers or the most repeat customers? To be liked by the hordes and masses? Or to be adored by a select few?
Translating conceptual goals into something you can count.
Goals are often defined around vague concepts like "healthy," "improvement," "satisfaction," and "quality." Your job is to pin down what success looks like quantitatively.
Measuring the right things by constantly questioning assumptions.
Don't fall into the trap of only looking for answers "where the light is good." Also known as, the "easiest thing to count." Instead, treat the data you collect as clues, not definitive answers.
Case Study: A product team wants to measure the impact, hopefully positive, of a new auto-complete function for a search input box. Until now, the "success" of search-related features has been measured by the number of searches the user performs. However, in this case auto-complete is so useful that the user rarely follows through with the actual search, and when they do, result set click-throughs are noticeably decreased. In such a case, how should success be defined?
Sometimes, frequency of use does not necessarily reflect the quality of the relationship you have with your customer.
Working through trial and error and accepting that you will never get it just right.
Intelligent iteration will always be more effective than the most painstaking plan. Figure out over time what you need to collect and how you need to collect it. You will probably end up re-tooling what questions you're trying to ask as well.
Performing the Sniff Test.
Could this possibly be true? Be suspicious of straight lines and clean curves.
Build stories out of scaffolding, so you can tear them down.
Data will rarely tell you something definitive on its own. Instead, human intervention is required to piece together something credible that explains why the data came out the way it did. Then you will hack at that story with all your might until you're satisfied that the story you constructed isn't just another fairy tale.
You'll find yourself saying a lot of things like:
"If that were true, then you would also see..."
"This doesn't make sense because both of these can't be..."
Being paranoid. And making sure your systems fail gracefully.
Figure out all the ways things can go wrong. Connectivity issues...Configuration issues...Multiple users posing as a single user...
When systems fail, make sure there's a way for you to find out something has gone wrong, and have alternate mechanisms in place for getting the data you need. Think army supply lines. Multiple sources of data also allows you to cross-check your data and identify problems early on.
Comparing apples to apples.
Defending your data.
Once you have a credible story, be prepared to defend it if it doesn't align with what others expect or want to hear. Data itself doesn’t lie, but people often misrepresent data for their own ends. So make sure you can show that you're clarifying, not distorting the numbers. This is where your commitment to transparency, attention to detail and sterling communication skills kick in.
Being prepared to "steward" your data.
As your business, organization, service or product moves forward, your data collection system to will need to change and progress along with it. Today, data collection and analysis isn't something you do once a year, like annual reports of times past. It's an ever-evolving system that will require constant stewardship. (That's Gopher, the ship steward from "The Love Boat.")
Our Approach