Tag Archives: KPIs

The Myth of the Single KPI for Testing

Continuous Improvement through testing is a simple idea. That’s no surprise. The simplest, most obvious ideas are often the most powerful. And testing is a powerful idea. An idea that forms and shapes the way digital is done by the companies that do it best. And those same companies have changed the world we live in.

If testing and continuous improvement is a process, analytics is the driver of that process; and as any good driver knows, the more powerful the vehicle, the more careful you have to be as a driver. Testing analytics seems so easy. You run a test, you measure which worked better. You choose the winner.

It’s like reading the scoreboard at a football game. It doesn’t take a lot of brains to figure out who’s ahead.

Except it’s usually not that easy.

Sporting events just are decided by the score. Games have rules and a single goal. Life and business mostly don’t. What makes measuring tests surprising tricky is that you rarely have a single unequivocal measure of success.

Suppose you add a merchandising drive to a section of your store or on the product detail page of your website. You test. And you generate more sales of that product.

Success!

Success?

Let’s start with the obvious caveat. You may have generated more sales, but you gave up margin. Was it worth it? Usually, the majority of buyers with a discount would have bought without one. Still, that kind of cannibalization is fairly easy to baseline and measure.

Here’s a trickier problem. What else changed? Because when you add a merchandising drive to a product, you don’t just shift that product’s buying pattern. The customer who buys might have bought something else. Maybe something with a better margin.

To people who don’t run tests, this may come as a bit of surprise. Shouldn’t tests be designed to limit their impact so that the “winner” is clear? ‘

Part of a good experimental design is, indeed, creating a test that limits external impacts. But this isn’t the lab. Limiting the outside impact of a test isn’t easy and you can  never be sure you’ve actually succeeded in doing that unless you carefully measure.

Worse, the most important tests usually have the most macro-impact. Small creative tests can often be isolated to a single win-loss metric. Sadly, that metric usually doesn’t matter or doesn’t move.

If you need proof of that, check out this meta-study by Will Browne & Mike Jones (those names feel like generic test products, right?) that looked at the impact of different types of test. Their finding? UI changes of the color and call-to-action type had, essentially, zero impact. Sadly, that’s what most folks spend all their time testing. (http://www.qubit.com/sites/default/files/pdf/qubit_meta_analysis.pdf)

If your test actually changes shopper behavior, believe me, there will be macro impacts.

It’s usually straightforward to measure the direct results of a store test. It’s often much harder to determine the macro impact. But it’s something you MUST look at. The macro impact can be as or more important than the direct impact. What’s more, it often – I’ll say usually – runs in the opposite direction.

So if you fail to measure the macro impact of a store test and you focus only on the obvious outcome, you’ll often pick the wrong result or grossly overstate the impact. Either way, you’re not using your analytics to drive appropriately.

Of course, one of the very real challenges you’ll face is that many tools don’t measure the macro impact of tests at all. In the digital world, the vast majority of dedicated testing tools require you to focus on a single KPI and provide absolutely no measurement of macro impacts. They simply assume that the test was completely compartmentalized. That works okay for things like email testing, but it’s flat-out wrong when it comes to testing store or website changes.

If your experiment worked well enough to change a shopper’s behavior and got them to buy something, the chances are quite good that it changed more than just that behavior. You may have given up margin. You likely lost some sales elsewhere. You almost certainly changed what else in the store or the site the shopper engaged with. That stuff matters.

In the store world, most tools don’t measure enough to give you even the immediate win-loss results. To heck with the rest of the story. So it can tempting, when you first have real measurement, to focus on the obvious: which test won. Don’t.

In some of my recent posts, I’ve talked about the ways in which DM1 – our store testing and measurement platform – lets you track the full customer journey, segmentfunnel and compare. Those capabilities are key to doing test measurement right. They give you the ability to see the immediate impact of a test AND the ways in which a change affected macro customer behavior.

You can see an example of how this works (and how important that macro behavior is in store layout) in this DM1 video that focuses on the Comparison capabilities of the tool.

https://www.youtube.com/watch?v=lbpaeSmaE74&t=13s

It’s the right way to use all that power a store testing program can provide.

Productivity is Our Business. And Business isn’t Good

A little while back there was a fascinating article on the lack of productivity growth in the U.S. in the past 4-5 years. I’ll try to summarize the key points below (and then tell you why I think they’re important) – but the full article is very much worth the read.

Productivity Growth

Let’s start with the facts. In the last year, the total number of hours worked in the U.S. rose by 1.9%. GDP growth in the last quarter exactly matched that rate – 1.9%. So we added hours and we got an exact match in output. That might sound okay, but it means that there was zero productivity growth. We didn’t get one whit more efficient in producing stuff. Nor is this just a short term blip. In the last four years, we’ve recorded .4% annual growth in productivity. That’s not very good. Take a look at the chart above (from the New York Times article and originally from the Labor Department) – it looks bad. We’re in late ‘70s and early ‘80s territory. Those weren’t good years.

The Times article advances three theories about why productivity growth has been so tepid. They classify them as the “Depressing” theory, the “Neutral” theory and the “Happy” theory. Here’s a quick description of each.

Depressing Theory

The trend is real and will be sustained. Capex is down. The digital revolution is largely complete. People aren’t getting significantly more productive and the people returning to the work-force post-recession are the least productive segment of our workforce. On this view, we’re not getting richer anytime soon.

Neutral Theory

There’s a lot of imprecision in measuring productivity. With fundamental changes in the economy it may be that the imprecision is increasing – and we’re undercounting true productivity. As measurement professionals, we all know this one needs to be reckoned with.

Happy Theory

We’re in an “investment” period where companies are hiring and investing – resulting in a period of lower-productivity before that investment begins to show returns and productivity accelerates. Interestingly, this story played out in the late ‘90s when productivity slowed and then accelerated sharply in the 2000s.

 

Which theory is right? The Times article doesn’t really draw any firm conclusions – and that’s probably reasonable. When it comes to macro-economic trends, the answers are rarely simple and obvious. From my perspective, though, this lack of productivity is troubling. We live in a profession (analytics) that’s supposed to be the next great driver or productivity. Computers, internet, now analytics. We’re on the hook for the next great advance in productivity. From a macro-economic perspective, no one’s thinking about analytics. But out here in the field, analytics is THE thing companies are investing in to drive productivity.

And the bad news? We’re clearly not delivering.

Now I don’t take it as all bad news. There’s a pretty good chance that the Happy theory is dead-on. Analytics is a difficult transformation and one that many companies struggle with. And while they’re struggling with big data systems and advanced analytics, you have a lot of money getting poured into rather unproductive holes. Word processing was almost certainly more immediately productive than analytics (anybody out there remember Wang?) – but every sea change in how we do things is going to take time, effort and money. Analytics takes more than most.

Here’s the flip side, though. It’s easy to see how all that investment in analytics might turn out to be as unproductive as building nuclear missiles and parking them into the ground. If they were ever used, those missiles would produce a pretty big bang for the buck. In the case of ICBM’s, we’re all happiest when they don’t get used. That’s not what we hope for from analytics.

Of course, I’ve been doing this extended series on the challenges of digital transformation – most of which revolves around why we aren’t more productive with analytics. Those challenges are not, in my opinion, the exception. They’re the rule. The vast majority of enterprises aren’t doing analytics well and aren’t boosting their productivity with it. That doesn’t mean I don’t believe in the power of analytics to drive real productivity. I do. But before those productivity gains start to appear, we have to do better.

Doing better isn’t about one single thing. Heaven knows it’s not just about having the newest technologies. We have those aplenty. It’s about finding highly repeatable methods in analytics so that we can drive improvement without rock-stars. It’s very much about re-thinking the way the organization is setup so that analytics is embedded and operationalized. It’s even more about finding ways to re-tool our thinking so that agile concepts and controlled experimentation are everywhere.

Most companies still need a blueprint for how to turn analytics into increased productivity. That’s what this series on digital transformation is all about.

If you haven’t yet had the opportunity to spin through my 20min presentation on transforming the organization with analytics – check it out.

After all, productivity is our business.

Digital Transformation – How to Get Started, Real KPIs, the Necessary Staff and So Much More!

In the last couple of months, I’ve been writing an extended series on digital transformation that reflects our current practice focus. At the center of this whole series is a simple thesis: if you want to be good at something you have to be able to make good decisions around it. Most enterprises can’t do that in digital. From the top on down, they are setup in ways that make it difficult or impossible for decision-makers to understand how digital systems work and act on that knowledge. It isn’t because people don’t understand what’s necessary to make good decisions. Enterprises have invested in exactly the capabilities that are necessary: analytics, Voice of Customer, customer journey mapping, agile development, and testing. What they haven’t done is changed their processes in ways that take advantage of those capabilities.

I’ve put together what I think is a really compelling presentation of how most organizations make decisions in the digital channel, why it’s ineffective, and what they need to do to get better. I’ve put a lot of time into it (because it’s at the core of our value proposition) and really, it’s one of the best presentations I’ve ever done. If you’re a member of the Digital Analytics Association, you can see a chunk of that presentation in the recent webinar I did on this topic. [Webinars are brutal – by far the hardest kind of speaking I do – because you are just sitting there talking into the phone for 50 minutes – but I think this one, especially the back-half, just went well] Seriously, if you’re a DAA member, I think you’ll find it worthwhile to replay the webinar.

If you’re not, and you really want to see it, drop me a line, I’m told we can get guest registrations setup by request.

At the end of that webinar I got quite a few questions. I didn’t get a chance to answer them all and I promised I would – so that’s what this post is. I think most of the questions have inherent interest and are easily understood without watching the webinar so do read on even if you didn’t catch it (but watch the darn webinar).

Q: Are metrics valuable to stakeholders even if they don’t tie in to revenues/cost savings?

Absolutely. In point of fact, revenue isn’t even the best metric on the positive side of the balance sheet. For many reasons, lifetime value metrics are generally a better choice than revenue. Regardless, not every useful metric has to, can or should tie back to dollars. There are whole classes of metrics that are important but won’t directly tie to dollars: satisfaction metrics, brand awareness metrics and task completion metrics. That being said, the most controversial type of non-revenue metric are proxies for engagement which is, in turn, a kind of proxy for revenue. These, too, can be useful but they are far more dangerous. My advice is to never use a proxy metric unless you’ve done the work to prove it’s a valid proxy. That means no metrics plucked from thin air because they seem reasonable. If you can’t close the loop on performance with behavioral data, use re-survey methods. It’s absolutely critical that the metrics you optimize with be the right ones – and that means spending the extra time to get them right. Finally, I’ve argued for awhile that rather than metrics our focus should be on delivering models embedded in tools – this allows people to run their business not just look at history.

Q: What is your favorite social advertising KPI? I have been using $ / Site Visit and $ / Conversion to measure our campaigns but there is some pushback from the social team that we are not capturing social reach.

A very related question – and it’s interesting because I actually didn’t talk much about KPIs in the webinar! I think the question boils down to this (in addition to everything I just said about metrics) – is reach a valid metric? It can be, but reach shouldn’t be taken as is. As per my answer above, the value of an impression is quite different on every channel. If you’re not doing the work to figure out the value of an impression in a channel then what’s the point of reporting an arbitrary reach number? How can people possibly assess whether any given reach number makes a buy good or bad once they realize that the value of an impression varies dramatically by channel? I also think a strong case can be made that it’s a mistake to try and optimize digital campaigns using reported metrics even direct conversion and dollars. I just saw a tremendous presentation from Drexel’s Elea Feit at the Philadelphia DAA Symposium that echoed (and improved) what I’ve been saying for years. Namely that non-incremental attribution is garbage and that the best way to get true measures of lift is to use control groups. If your social media team thinks reach is important, then it’s worth trying to prove if they are right – whether that’s because those campaigns generate hidden short-term lift or or because they generate brand awareness that track to long term lift.

Q: For companies that are operating in the way you typically see, what is the one thing you would recommend to help get them started?

This is a tough one because it’s still somewhat dependent on the exact shape of the organization. Here are two things I commonly recommend. First, think about a much different kind of VoC program. Constant updating and targeting of surveys, regular socialization with key decision-makers where they drive the research, an enterprise-wide VoC dashboard in something like Tableau that focuses on customer decision-making not NPS. This is a great and relatively inexpensive way to bootstrap a true strategic decision support capability. Second, totally re-think your testing program as a controlled experimentation capability for decision-making. Almost every organization I work with should consider fundamental change in the nature, scope, and process around testing.

Q: How much does this change when there are no clear conversions (i.e., Non-Profit, B2B, etc)?

I don’t think anything changes. But, of course, everything does change. What I mean is that all of the fundamental precepts are identical. VoC, controlled experiments, customer journey mapping, agile analytics, integration of teams – it’s all exactly the same set of lessons regardless of whether or not you have clear conversions on your website. On the other hand, every single measurement is that much harder. I’d argue that the methods I argue for are even more important when you don’t have the relatively straightforward path to optimization that eCommerce provides. In particular, the absolute importance of closing the loop on important measurements simply can’t be understated when you don’t have a clear conversion to optimize to.

Q: What is the minimum size of analytics team to be able to successfully implement this at scale?

Another tricky question to answer but I’ll try not to weasel out of it. Think about it this way, to drive real transformation at enterprise scale, you need at least 1 analyst covering every significant function. That means an analyst for core digital reporting, digital analytics, experimentation, VoC, data science, customer journey, and implementation. For most large enterprises, that’s still an unrealistically small team. You might scrape by with a single analyst in VoC and customer journey, but you’re going to need at least small teams in core digital reporting, analytics, implementation and probably data science as well. If you’re at all successful, the number of analytics, experimentation and data science folks is going to grow larger – possibly much larger.  It’s not like a single person in a startup can’t drive real change, but that’s just not the way things work in the large enterprise. Large enterprise environments are complex in every respect and it takes a significant number of people to drive effective processes.

Q: Sometimes it feels like agile is just a subject line for the weekly meeting. Do you have any examples of organizations using agile well when it comes to digital?

Couldn’t agree more. My rule of thumb is this: if your organization is studying how to be innovative, it never will be. If your organization is meeting about agile, it isn’t. In the IT world, Agile has gone from a truly innovative approach to development to a ludicrous over-engineered process managed, often enough, by teams of consulting PMs. I do see some organizations that I think are actually quite agile when it comes to digital and doing it very well. They are almost all gaming companies, pure-play internet companies or startups. I’ll be honest – a lot of the ideas in my presentation and approach to digital transformation come from observing those types of companies. Whether I’m right that similar approaches can work for a large enterprise is, frankly, unclear.

Q: As a third party measurement company, what is the best way to approach or the best questions to ask customers to really get at and understand their strategic goals around their customer journeys?

This really is too big to answer inside a blog – maybe even too big to reasonably answer as a blog. I’ll say, too, that I’m increasingly skeptical of our ability to do this. As a consultant, I’m honor-bound to claim that as a group we can come in, ask a series of questions of people who have worked in an industry for 10 or 20 years and, in a few days time, understand their strategic goals. Okay…put this way, it’s obviously absurd. And, in fact, that’s really not how consulting companies work. Most of the people leading strategic engagements at top-tier consulting outfits have actually worked in an industry for a long-time and many have worked on the enterprise side and made exactly those strategic decisions. That’s a huge advantage. Most good consultants in a strategic engagement know 90% of what they are going to recommend before they ask a single question.

Having said that, I’m often personally in a situation where I’m asked to do exactly what I’ve just said is absurd and chances are if you’re a third party measurement company you have the same problem. You have to get at something that’s very hard and very complex in a very short amount of time and your expertise (like mine) is in analytics or technology not insurance or plumbing or publishing or automotive.

Here’s a couple of things I’ve found helpful. First, take the journey’s yourself. It’s surprising how many executives have never bought an online policy from their own company, downloaded a whitepaper to generate a lead, or bought advertising on their own site. You may not be able to replicate every journey, but where you can get hands on, do it. Having a customer’s viewpoint on the journey never hurts and it can give you insight your customers should but often don’t have. Second, remember that the internet is your best friend. A little up-front research from analysts is a huge benefit when setting the table for those conversations. And I’m often frantically googling acronyms and keywords when I’m leading those executive conversations. Third, check out the competition. If you do a lead on the client’s website, try it on their top three competitors too. What you’ll see is often a great table-set for understanding where they are in digital and what their strategy needs to be. Finally, get specific on the journey. In my experience, the biggest failing in senior leaders is their tendency to generality. Big generalities are easy and they sound smart but they usually don’t mean much of anything. The very best leaders don’t ever retreat into useless generality, but most of us will fall into it all too easily.

Q: What are some engagement models where an enterprise engages 3rd party consulting? For how long?

The question every consultant loves to hear! There are three main ways we help drive this type of digital transformation. The first is as strategic planners. We do quite a bit of pure digital analytics strategy work, but for this type of work we typically expand the strategic team a bit (beyond our core digital analytics folks) to include subject matter experts in the industry, in customer journey, and in information management. The goal is to create a “deep” analytics strategy that drives toward enterprise transformation. The second model (which can follow the strategic phase) is to supplement enterprise resources with specific expertise to bootstrap capabilities. This can include things like tackling specific highly strategic analytics projects, providing embedded analysts as part of the team to increase capacity and maturity, building out controlled experiment teams, developing VoC systems, etc. We can also provide – and here’s where being part of a big practice really helps – PM and Change Management experts who can help drive a broader transformation strategy. Finally, we can help soup to nuts building the program. Mind you, that doesn’t mean we do everything. I’m a huge believer that a core part of this vision is transformation in the enterprise. Effectively, that means outsourcing to a consultancy is never the right answer. But in a soup-to-nuts model, we keep strategic people on the ground, helping to hire, train, and plan on an ongoing basis.

Obviously, the how-long depends on the model. Strategic planning exercises are typically 10-12 weeks. Specific projects are all over the map, and the soup-to-nuts model is sustained engagement though it usually starts out hot and then gets gradually smaller over time.

Q: Would really like to better understand how you can identify visitor segments in your 2-tier segmentation when we only know they came to the site and left (without any other info on what segment they might represent).  Do you have any examples or other papers that address how/if this can be done?

A couple years back I was on a panel at a Conference in San Diego and one of the panelists started every response with “In my book…”. It didn’t seem to matter much what the question was. The answer (and not just the first three words) were always the same. I told my daughters about it when I got home, and the gentleman is forever immortalized in my household as the “book guy”. Now I’m going to go all book guy on you. The heart of my book, “Measuring the Digital World” is an attempt to answer this exact question. It’s by far the most detailed explication I’ve ever given of the concepts behind 2-tiered segmentation and how to go from behavior to segmentation. That being said, you can only pre-order now. So I’m also going to point out that I have blogged fairly extensively on this topic over the years. Here’s a couple of posts I dredged out that provide a good overview:

http://semphonic.blogs.com/semangel/2012/05/digital-segmentation.html

http://semphonic.blogs.com/semangel/2011/06/building-a-two-tiered-segmentation-semphonics-digital-segmentation-techniques.html

and – even more important – here’s the link to pre-order the book!

That’s it…a pretty darn good list of questions. I hope that’s genuinely reflective of the quality of the webinar. Next week I’m going to break out of this series for a week and write about our recent non-profit analytics hackathon – a very cool event that spurred some new thoughts on the analysis process and the tools we use for it.