Burning Down the House

Nowhere is the challenge of getting people to understand how to use data better illustrated than the methodology wars being fought in the discipline of Psychology. If you haven’t heard of the methodology wars be assured that the battlefields – studies in psychological research – are being fought over like blocks of Stalingrad; and like that famous battle, not much is left standing in the aftermath.

I’m not sure exactly how the methodology wars started. Somehow, somewhere, someone decided to actually re-test a “classic” study in psychology. A study that’s been accepted into the core of the discipline – that established somebody’s reputation, made somebody a career. Only it didn’t replicate. They re-did the experiment as carefully as they could and it didn’t show the same result. Didn’t, usually, show any result at all. Increase the sample size to fix the problem and the signal becomes even clearer. Alas, the signal always seems to be that there is no signal.

Pretty soon people started calling into question nearly every Psych study done over the last fifty years and testing them. And many – and I mean many – have failed.

Slate’s article (and it’s really good – giving a great overview of the issue – so give it a read) recounts the latest block to burn down in psychology’s methodology wars. The research in question centered around the idea that our facial states feedback into our emotions. If we smile (even inadvertently) we will feel happier.

It’s an interesting idea – intuitively plausible – and apparently widely supported by a huge variety of studies in the field. It’s an idea which strikes me as perfectly reasonable and in which I have zero vested interested one way or another.

But when it was submitted to rigorous simultaneous validation in a number of different labs, it failed. Completely.

The original test involved 32 participants and a change in average subjective scoring between the two groups of 4.4 to 5.5. That means each group had sixteen participants. The improved second test had 92 total participants and showed a scoring difference of 4.3 to 51.

That’s a pretty small sample.

Especially for something that became received wisdom. A classic.

So how would people react if it turned out to be wrong?

Well, the Slate article answers that question pretty definitively. Because when the multi-lab tests came back, here’s what happened. Seventeen different labs replicated the experiment with nearly 2000 subjects. In half the participating labs, participants who smiled recorded a slightly higher average on the resulting happiness test (but much lower than in the original experiment). In the other half, it went the other way.

Net, net, there was no correlation at all. Zero.

Okay, so far you have just another sad story of a small sample size failure.

That’s not what really attracted my attention. Nope. What really made me laugh in utter disbelief was the comment of the “scientist” who had done the original research. Here it is, and I quote in full lest you think I’m about to exaggerate:

“Fritz Strack has no regrets about the RRR, but then again, he doesn’t take its findings all that seriously. “I don’t see what we’ve learned,” he said.

Two years ago, while the replication of his work was underway, Strack wrote a takedown of the skeptics’ project with the social psychologist Wolfgang Stroebe. Their piece, called “The Alleged Crisis and the Illusion of Exact Replication,” argued that efforts like the RRR reflect an “epistemological misunderstanding,” since it’s impossible to make a perfect copy of an old experiment. People change, times change, and cultures change, they said. No social psychologist ever steps in the same river twice. Even if a study could be reproduced, they added, a negative result wouldn’t be that interesting, because it wouldn’t explain why the replication didn’t work.

So when Strack looks at the recent data he sees not a total failure but a set of mixed results. Nine labs found the pen-in-mouth effect going in the right direction. Eight labs found the opposite. Instead of averaging these together to get a zero effect, why not try to figure out how the two groups might have differed? Maybe there’s a reason why half the labs could not elicit the effect.

[Bolding is mine]

So here’s a “scientist” who, despite presumably being familiar with the extensive literature on statistics and the methodology wars, somehow believes that because half the labs reported a number slightly above average the key thing to look at is why the other half didn’t. Apparently, the only thing that would satisfy him is if all the labs reported an exactly opposite result. Which, presumably, would result in a new classic paper that frowning makes you happier!

Ring, Ring!

Clue Phone.

It’s random variation calling for you, Dr. Strack!

Cause here’s the thing…you’d expect about half the labs to show a positive result when there is no correlation. If some labs didn’t report a positive result then the correlation would pretty much have to be negative, right?

This isn’t, as he appears to believe, half-corroboration. It’s the way every null result ever found actually looks out here in the real world. I’d advise him to try flipping a coin 100 times, repeatedly, and see how often it comes out 50 heads and 50 tails. He might be surprised to learn that about the half the time this test will yield more heads flips than tail flips. This does not mean that heads is more likely than tails and it does not suggest that researchers should focus on why some trials yielded more heads and other trials yielded more tails.

 

Okay, I get it. You published a study. You made a career out of it. It’s embarrassing that it turns out to be wrong. But it’s hard to know in this case which is the worse response – intellectual dishonesty or sheer stupidity. Frankly, I think the latter. Because I don’t care how dishonest you are, some explanations should be too embarrassing to try on for size. And the idea that the right interpretation of these results would be to look for why some labs had slightly different results than others clearly belongs in that category.

I find the defense based on the difficulties of true replication more respectable. And yet, what are we to make of an experiment so delicate that it can’t be replicated AT ALL even with the most careful controls? How important can any inference we make from such an experiment plausibly be? By definition, it could only fit the most narrow range of cases imaginable. And the idea that replication of an experiment doesn’t matter seems…you know…a tad unscientific.

From my perspective, it isn’t the original study that illustrates the extraordinary problem we have getting people to use data well. Yes, over-reliance on small sample sizes is all too common and all too easy. That’s unfortunate, not shameful. But the deeper problem is that even when data is used well, a lethal combination of self-interest and a near total lack of understanding of basic statistics make it all too possible for people to ignore the data whenever they wish.

As Simon and Garfunkel plaintively observed, “A man hears what he wants to hear, and disregards the rest”.

If it wasn’t so sad, it would be funny.

Dammit. It is funny.

For it’s easy to see that in this version of the psych methodology wars, the defenders have their own unique version of a foxhole – with posterior high in the air and head firmly planted in the sand.

[Getting close to the Digital Analytics Hub. If you love talking analytics, check it out. Would be great to see you there!]

Frictionless Competition and the Surprising Monopolization of the Internet

In the last few months I’ve been spending quite a bit of time thinking about the challenges in physical retail – stores. I’m going to be talking much more about that in the months to come, but thinking about the challenges in physical retail and whether and to what extent digital techniques might help, I’ve also had to think about why digital retail has evolved the way it has.

There’s no doubt that digital has disrupted and hurt traditional retail. But it’s a mistake to attribute that solely to advantages inherent in digital. After all, if it was just a matter of digital being superior to B&M, then Borders should have been fine moving online. That didn’t work out so well.

In fact, one of the most interesting aspects of our digital world is how a perfect leveling of the playing field has produced such a strong tendency to natural monopoly. This isn’t just about retail. In most of the key areas of internet – from retail to video streaming to music to search to ride summoning, we’ve seen an extraordinary tendency toward massive consolidation around a single leader.

It’s not exactly what most of us expected. By eliminating most barriers to entry, creating frictionless geographies, and creating technology environments that scale seamlessly to almost any size, the digital world has removed many of the traditional bastions of monopoly. Old-world monopolies used to spring from cases where scale precluded competition. If, for example, you owned the pipes that carried gas to homes or the wires that carried electricity, it was incredibly hard for anyone else to compete.

In today’s world, that kind of ownership has mostly vanished. You could argue that if you own search you own the pipes to the Web. But the analogy doesn’t hold. It doesn’t hold because anybody can create a competing search system at any time and every single internet user can have instant access to it. It doesn’t hold because there are multiple ways to pipe through the internet besides search. And it doesn’t hold because there really are no physical barriers to building or deploying that alternative search system.

So it wouldn’t be unreasonable to expect the digital world to have morphed into a wild west of tiny artisanal companies with meteoric rises, equally sudden collapses, and constant, ubiquitous competition. Mostly, though, that’s not the way it looks at all. It looks as if monopoly, despite the absence of physical barriers, is actually a more powerful tendency in the digital world than the physical world.

It’s not that hard to understand why things have gone this way. Natural monopolies around things like electricity delivery occurred because of the immense friction involved in setting up the delivery system. Economies of scale were absolutely decisive in such situations. But most traditional markets are resilient to natural monopoly because of fundamental facts of the physical world that worked AGAINST too much scale. In the physical world, it makes perfect sense to have gas stations on the opposite side of a street. And it’s quite likely that two such stations can not only co-exist but thrive despite their close proximity. After all, it’s a pain to cross the street when you want to get gas. I may prefer Whole Foods to Safeway or vice versa. But I often go the grocery store that’s closest to me regardless of brand. And when I lived in San Francisco I bought most of my Diet Coke and impulse snacks at the corner store up my block. No, it wasn’t nice and it wasn’t cheap. But it sure was close. I may like Sol Food in San Rafael better than Los Moles, but so do a lot of other people – and I hate standing in line.

The natural friction that the physical world carries in terms of geographic convenience and capacity help ensure that countless niches for delivery exist. Like my old corner store, in the physical world, you can o be worse at everything except location and still thrive.

That doesn’t happen in the digital world.

It turns out – and I guess this should be no surprise – that in a frictionless world, any small advantage can be decisive. A grocery has to be a LOT better than its competitors to get me to drive an extra 10 minutes. But online, the best grocery is always just a few milliseconds away.

It doesn’t have to be a lot better. In fact, the difference can be incredibly tiny. Absent friction, the size of the advantage is no longer that meaningful. The digital world can make even tiny advantages decisive.

So why doesn’t every aspect of the digital world turn into a monopoly?

The answer lies in segmentation. A very small advantage may be decisive in the digital world. But it’s hard to have an advantage to EVERYONE.

In areas like news and entertainment, for example, it’s impossible to produce content that is better for everyone. Age, education, interest, background, geography and countless other factors create an infinity of micro-fractures. Not only is the content itself differentiated, but it’s creation is almost equally fractured. A.O. Scott could no more produce a version of Real Housewives than Andy Cohen could write a NY Times film review.

Content creation turns out to be friction-full in a way that was somewhat obscured by the old limitations in distribution. In fact, it appears that the market for segmented content and the ability of content to create barriers to consolidation is almost limitless. That’s why there’s almost nothing so important to becoming a good digital company than content creation. It’s the best way there is to guard your marketspace.

All this suggests that there are two paths to success in the digital world. One path involves scale and the other segmentation. They aren’t mutually exclusive and the companies that do both well are formidable indeed.

 

It’s only a little more than a month till the Digital Analytics Hub in Monterey and a chance to talk all things digital – both practical and philosophical. After all, there is no monopoly on great conversation. Looking forward to talking deep analytics, natural monopolies, digital transformation and digital advantage!

A Guided Tour through Digital Analytics (Circa 2016)

I’ve been planning my schedule for the DA Hub in late September and while I find it frustrating (so much interesting stuff!), it’s also enlightening about where digital analytics is right now and where it’s headed. Every conference is a kind of mirror to its industry, of course, but that reflection is often distorted by the needs of the conference – to focus on the cutting-edge, to sell sponsorships, to encourage product adoption, etc.  With DA Hub, the Conference agenda is set by the enterprise practitioners who are leading groups – and it’s what they want to talk about. That makes the conference agenda unusually broad and, it seems to me, uniquely reflective of the state of our industry (at least at the big enterprise level).

So here’s a guided tour of my DA Hub – including what I thought was most interesting, what I choose, and why. At the end I hope that, like Indiana Jones picking the Holy Grail from a murderers row of drinking vessels, I chose wisely.

Session 1 features conversations on Video Tracking, Data Lakes, the Lifecycle of an Analyst, Building Analytics Community, Sexy Dashboards (surely an oxymoron), Innovation, the Agile Enterprise and Personalization. Fortunately, while I’d love to join both Twitch’s June Dershewitz to talk about Data Lakes and Data Swamps or Intuit’s Dylan Lewis for When Harry (Personalization) met Sally (Experimentation), I didn’t have to agonize at all, since I’m scheduled to lead a conversation on Machine Learning in Digital Analtyics. Still, it’s an incredible set of choices and represents just how much breadth there is to digital analytics practice these days.

Session 2 doesn’t make things easier. With topics ranging across Women in Analytics, Personalization, Data Science, IoT, Data Governance, Digital Product Management, Campaign Measurement, Rolling Your Own Technology, and Voice of Customer…Dang. Women in Analytics gets knocked off my list. I’ll eliminate Campaign Measurement even though I’d love to chat with Chip Strieff from Adidas about campaign optimization. I did Tom Bett’s (Financial Times) conversation on rolling your own technology in Europe this year – so I guess I can sacrifice that. Normally I’d cross the data governance session off my list. But not only am I managing some aspects of a data governance process for a client right now, I’ve known Verizon’s Rene Villa for a long time and had some truly fantastic conversations with him. So I’m tempted. On the other hand, retail personalization is of huge interest to me. So talking over personalization with Gautam Madiman from Lowe’s would be a real treat. And did I mention that I’ve become very, very interested in certain forms of IoT tracking? Getting a chance to talk with Vivint’s Brandon Bunker around that would be pretty cool. And, of course, I’ve spent years trying to do more with VoC and hearing Abercrombie & Fitch’s story with Sasha Verbitsky would be sweet. Provisionally, I’m picking IoT. I just don’t get a chance to talk IoT very much and I can’t pass up the opportunity. But personalization might drag me back in.

In the next session I have to choose between Dashboarding (the wretched state of as opposed to the sexiness of), Data Mining Methods, Martech, Next Generation Analytics, Analytics Coaching, Measuring Content Success, Leveraging Tag Management and Using Marketing Couds for Personalization. The choice is a little easier because I did Kyle Keller’s (Vox) conversation on Dashboarding two years ago in Europe. And while that session was probably the most contentious DA Hub group I’ve ever been in (and yes, it was my fault but it was also pretty productive and interesting), I can probably move on. I’m not that involved with tag management these days – a sign that it must be mature – so that’s off my list too. I’m very intrigued by Akhil Anumolu’s (Delta Airlines) session on Can Developers be Marketers? The Emerging Role of MarTech. As a washed-up developer, I still find myself believing that developers are extraordinarily useful people and vastly under-utilized in today’s enterprise. I’m also tempted by my friend David McBride’s session on Next Generation Analytics. Not only because David is one of the most enjoyable people that I’ve ever met to talk with, but because driving analytics forward is, really, my job. But I’m probably going to go with David William’s session on Marketing Clouds. David is brilliant and ASOS is truly cutting edge (they are a giant in the UK and global in reach but not as well known here), and this also happens to be an area where I’m personally involved in steering some client projects. David’s topical focus on single-vendor stacks to deliver personalization is incredibly timely for me.

Next up we have Millennials in the Analytics Workforce, Streaming Video Metrics, Breaking the Analytics Glass Ceiling, Experimentation on Steroids, Data Journalism, Distributed Social Media Platforms, Customer Experience Management, Ethics in Analytics(!), and Customer Segmentation. There are several choices in here that I’d be pretty thrilled with: Dylan’s session on Experimentation, Chip’s session on CEM and, of course, Shari Cleary’s (Viacom) session on Segmentation. After all, segmentation is, like, my favorite thing in the world. But I’m probably going to go with Lynn Lanphier’s (Best Buy) session on Data Journalism. I have more to learn in that space, and it’s an area of analytics I’ve never felt that my practice has delivered on as well as we should.

In the last session, I could choose from more on Customer Experience Management, Driving Analytics to the C-Suite, Optimizing Analytics Career-Oaths, Creating High-Impact Analytics Programs, Building Analytics Teams, Delivering Digital Products, Calculating Analytics Impact, and Moving from Report Monkey to Analytics Advisor. But I don’t get to choose. Because this is where my second session (on driving Enterprise Digital Transformation) resides. I wrote about doing this session in the EU early this summer – it was one of the best conversations around analytics I’ve had the pleasure of being part of. I’m just hoping this session can capture some of that magic. If I didn’t have hosting duties, I think I might gravitate toward Theresa Locklear’s (NFL) conversation on Return on Analytics. When we help our clients create new analytics and digital transformation strategies, we have to help them justify what always amount to significant new expenditures. So much of analytics is exploratory and foundational, however, that we don’t always have great answers about the real return. I’d love to be able to share thoughts on how to think (and talk) about analytics ROI in a more compelling fashion.

All great stuff.

We work in such a fascinating field with so many components to it. We can specialize in data science and analytics method, take care of the fundamental challenges around building data foundations, drive customer communications and personalization, help the enterprise understand and measure it’s performance, optimize relentlessly in and across channels, or try to put all these pieces together and manage the teams and people that come with that. I love that at a Conference like the Hub I get a chance to share knowledge with (very) like-minded folks and participate in conversations where I know I’m truly expert (like segmentation or analytics transformation), areas where I’d like to do better (like Data Journalism), and areas where we’re all pushing the outside of the envelope (IoT and Machine Learning) together. Seems like a wonderful trade-off all the way around.

See you there!
See you there!

https://www.digitalanalyticshub.com/dahub16-us/

 

Seven Pillars of Statistical Wisdom

I don’t review a lot of business books on my blog…mostly because I don’t like a lot of business books. A ridiculous percentage of business books seem to me either to be one-trick ponies (a good idea that could be expressed fully in a magazine article expanded to book length) or thinly veiled self-help books (self help books with ties as described in this spot-on Slate article). I HATE self-help books. Grit, Courage, Indecisiveness. It’s all the same to me.

On the other hand, The Seven Pillars of Statistical Wisdom isn’t really a business book. It’s a short (200 small pages), crisp, philosophical exploration of what makes statistics interesting. Written by a Univ. of Chicago Professor and published by Harvard University Press, it’s the best quasi-business book I’ve read in a long time.

I say quasi-business book because I’m not really sure who the intended audience is. It’s not super technical (thank god you can read it and know very little math), but it sometimes veers into explanations that assume a fairly deep understanding of statistics. Deeper, at least, than I have though I am most certainly not a formally trained statistician.

What Seven Pillars does extraordinarily well is examine a small core set of statistical ideas, explicate their history, and show why they are important, fundamental, and, in some cases, still controversial. In doing this, Seven Pillars provides a profound introduction into how to think statistically – not do statistics. Instead of focusing on how specific methods work, on definitions of statistical methods, or on specific issues in modern statistics (like big data), Seven Pillars tries to define what makes statistics an important way to think.

To give you a sense of this, here are the seven pillars:

Aggregation: Probably the core concept at the heart of all statistical thinking is the idea that you can sometimes GAIN insight while losing data. Stigler delves into basic concepts like the mean, shows how they evolved over the centuries (and it did take centuries) and explains why this fundamental insight is so important. It’s a brilliant discussion.

Information: If we gain information by losing data, how do we know how much information we’ve gained? Or how much data we need? With this pillar, Stigler lays out why more is sometimes less and how the value of observations usually declines sharply. Another terrific discussion around a fundamental insight that comes from statistics but is constantly under siege from folk common-sense.

Likelihood: In this section, Stigler tackles how the concepts around confidence levels and estimation of likelihood evolved over time. This section contains an amusing and historically interesting discussion on arguments for and against the likelihood of miracles!

Intercomparison: Stigler’s fourth pillar is the idea that we can use interior measurements of the data (there’s an excellent discussion of the historical derivation of Standard Deviation for example) to understand it. This section includes a superb discussion of the pitfalls of purely internal comparison and the tendency of humans to find patterns and of data to exhibit patterns that are not meaningful.

Regression: The idea of regression to the mean is fundamental to statistical thinking. It’s an amazingly powerful but consistently non-intuitive concept. Stigler uses a genetics example (and a really cool Quincunx visualization) to help explain the concept. This is one of the best discussions in a very fine book. On the other hand, the last part of this section which covers multivariate and Bayesian developments is less wonderful. If you don’t already understand these concepts, I’m not sure Stigler’s discussion is going to help.

Design: The next pillar is all about experimental design – surely a concept that is fundamental not just to statistics but to our everyday practical application of it. I found the discussion of randomization in this section particularly interesting and potentially noteworthy and thought-provoking.

Residual: Pillar seven is, appropriately enough, about what’s left over. Stigler is concerned here to show how examining the unexplained part of the analysis leads to a great deal of productive thinking in science and elsewhere. The idea of nested models is introduced and this section somehow transitions into a discussion of data visualization with illustrations from Florence Nightingale (apparently a mean hand with a chart). I’m not sure this transition made perfect sense in the context of the chapter, but the discussion is fascinating, enjoyable and pointed enough to generate some real insight.

Stigler concludes with some thoughts around whether and where an eighth pillar might arise. There’s some interesting stuff here that’s highly appropriate to anyone in digital trying to extend analytics into high-dimensional, machine-learning spaces. The discussion is (too) brief but I think intentionally so.

 

Seven Pillars isn’t quite a great book, and I mean that as high-praise. I don’t read many books that I could plausibly describe as almost great. The quality of the explanations is extremely high. But it does a better job explicating the intellectual basis behind simpler statistical concepts than more complicated ones and there are places where I think it’s insufficiently forceful in illuminating the underlying ways of thinking not just the statistical methods. Perhaps that’s inevitable, but greatness isn’t easy!

I do think the book occasionally suffers from a certain ambiguity around its audience. Is it intended as a means to get deep practitioners thinking about more fundamental concepts? I don’t think so – too many of the explanations are historical and basic.

Is it intended for a lay audience? Please.

I think it fits two audiences very well, but perhaps neither perfectly.

First, there are folks like me who use statistics and statistical thinking on an everyday basis but are not formally trained. I’m assuming that’s also a pretty broad swath of my readers. I know I found it both useful and enlightening, with only a few spots where the discussion became obscure and overtly professional.

The second audience is students and potential students of statistics who need something that pulls them away from the trenches (here’s how you do a regression) and gets them to think about what their discipline actually does. For that audience, I think the book is consistently brilliant.

If there’s a better short introduction into the intellectual basis and foundation of statistical thinking, I don’t know it. And for those who confuse statistical thinking with the ability to calculate a standard deviation or run a regression, Seven Pillars is a heady antidote

Organizing the Digital Enterprise

At the Digital Analytics Hub in Europe I facilitated a conversation around enterprise digital transformation. We covered a lot of interesting ground, but organizing digital in the enterprise was the most challenging part of that discussion.

It’s a topic you can easily find yourself going around in circles with as people trot out opinions that sound right but sail past each other. That’s especially true since different organizations start (and want to finish) in very different places.

To get around that, I framed the problem in “state-of-nature” terms. If you were starting a digital organization from scratch in an enterprise, how would you organize and staff it?

But before we could answer that question, we had to consider something even more basic.

Should a “digital” organization be separate?

There’s a pretty strong sense these days that walling off digital from the rest of the organization gets things wrong from the outset. Digital should be embedded right into the DNA of the core organization. In a mature organization, there was a pretty broad consensus that digital isn’t a separate function. On the other hand, what if you’re not mature? Can you embed digital directly and grow it right if it’s inside the huge, complex structures that pervade an existing large enterprise? Even strong proponents of the “digital needs to be organic in the organization” point of view seemed to concede that incubation as a separate organization is often necessary to getting digital done and setup right. Of course, taking the incubation strategy is going to leave you with an organizational debt that at some point will have to be paid. The more successful you are and the larger and faster digital grows, the harder it’s going to be to re-integrate digital back into the organization.

I see both sides of this argument (and I’m sure there are more than two sides to be had). I’m just not a big believer in hard-and-fast right answers when it comes to organizational design.

If you have a strong digitally-experienced leader on your executive team and you have solid relationships between marketing and IT, maybe you try to transform digitally within your existing structures. If you’re not that lucky (and that is pretty lucky), maybe incubation with a strategy for integration is the right answer.

Having gotten to the point where most people conceded that incubation might sometimes be necessary, we returned to the “state-of-nature” question and discussed building out an incubated organization. Most people set product teams at the heart of that organization and agreed that these product teams should be organized very much along the lines that I described in my videos on enterprise transformation: agile-based product teams that include IT, creative and analytics people (behavioral, customer and attitudinal) all working together. In this model, there’s no pass-off from design to implementation to measurement to testing. The same teams that built a project optimize it – and there’s analytics at every step of the process.

I believe this is an incredibly powerful model for getting digital products right – and it’s a model that resonated across a pretty wide swath of different organizations – from giant retailers to very modest start-ups.

But it’s far from a complete answer to creating a digital organization.

Suppose you have these great integrated teams for each digital product, how do you handle all the ancillary functions that the large enterprise has developed? Things like finance and HR, for example. Do they need to be re-created inside a digital organization?

My reaction – and it was common – was that such functions probably don’t need to be re-created inside digital. Including these functions in digital doesn’t seem fundamental to getting digital right.

This point-of-view, however, was immediately challenged when it came to HR. The difficulties of digital hiring are well known – and it isn’t just finding resources. Traditional HR approaches to finding people, vetting candidates, compensation, and promotion bands are all problematic in digital. And if you get the people element wrong, everything else is doomed.

So once again, if you’ve got HR folks willing to work with and adapt to the needs of your digital leader, maybe you can leave existing structures intact and keep HR centralized. But HR is the wrong place to wimp out and leave your digital team without the power to execute the way they need to.

Bottom line? If my digital leader really wanted to own their own HR, I’d say yes.

Other functions? I don’t really know. Is it fundamental to digital execution? Does it need to be done differently in digital? Wherever the answer is yes, then it’s going to be a debate about whether it should live inside an incubated digital organization or be an outside service to it.

There’s another challenge that cuts even closer to the bone and lies at the heart of the challenge to the large enterprise. If you have a single digital product (like a pure-play startup might), you don’t have to worry about the relationship between and across teams and functions. But in a larger enterprise – even when it’s incubated – digital is going to require multiple product teams.

How do management lines work across those teams? Are the IT folks across product teams in the Digital IT organization and are they “managed” by Digital IT? Or are they managed by their Product Owner? In one sense, the answer seems obvious. On a day-to-day basis they are managed by their Product Owner. But who owns their career? What’s a career path like? How do Digital IT folks (or analysts) across product teams communicate? Who makes centralized decisions about key technology infrastructure? Who owns the customer?

Every one of these is a deep, important question with real ramifications for how the organization works and how you take a single product model and scale it into something that preserves the magic of the integrated team but adapts to the reality of the large, multi-function enterprise.

It was here, not surprisingly, that one of the participants in our DA Hub conversation trotted out the “dotted line”. Now it happened to be a consultant from a fellow big-4 and I (too glibly, I’m afraid) responded that “dotted lines are what consultants draw when they don’t have a good answer to a problem”.

I both regret and endorse this answer. I regret it because it was far too glib a response to what is, in one sense, probably the right answer. I endorse it because I think it’s true. God knows I’ve drawn these dotted lines before. When we draw a dotted line we essentially leave it up to the organization to organically figure out how it should work in day-to-day practice. That’s not necessarily a bad thing. It’s probably the right answer in a lot of cases. But we shouldn’t kid ourselves that just because it might be the right answer that makes it a good answer. It’s not. It’s a “we’re not the right people at the right time to answer this question” kind of answer. Knowing enough to know you’re not the right people at the right time is a good thing, but it would be a mistake to confuse that with actually having a good answer to the question.

So here’s my best attempt at a non-dotted line organization that integrates Product Teams into a broader structure. It seems clear to me that you need some centralized capabilities within each function. For Digital IT, as an example, these centralized teams provide shared services including enterprise technology selection, key standards and data governance. In analytics, the centralized team will be responsible for the overall customer journey mapping, analytics technology selection and standardization, a centralized analytics warehouse, and standards around implementation and reporting.

How big and inclusive does the centralized team need to be? Thinking there’s one right answer to this question is a kind of disease akin to thinking there’s some right answer to questions like “how large should government be?” There isn’t. I tend to be in the “as small as practical” school when it comes to centralization – both politically and in the enterprise. The best IT, the best creative, the best analytics is done when it’s closest to the business – that means out there in those Product teams. That also means making sure you don’t incent your best people out of the product teams into centralized roles so that they can “advance” and make more money.

It used to drive me crazy to see good teachers promoted to administrative roles in schools. You can’t blame the teachers. When you’ve got a family to feed, a house to buy, a nice to car to own, you’re not going to stay a teacher when your only path to more money and prestige is becoming an assistant principal. But you don’t see the Cleveland Cavaliers promoting Lebron from player to coach. It’s a terrible mistake to confuse rank with value.

I’m a big believer in WIDE salary bands. One great developer is worth an army of offshore programmers and is likely worth more than the person managing them. Don’t force your best people to Peter Principle themselves into jobs they hate or suck at.

So instead of creating progressions from Product teams to central teams, I’d favor aggressive rotational policies. By rotating people into and out of those central teams, you ensure that central teams stay attuned to the needs of the Product teams where work is actually getting done. You also remove the career-path issues that often drive top talent to gravitate toward centralization.

Communications and collaboration is another tricky problem. Collaboration is one of the most under-invested capabilities in the organization and my Product team structure is going to make it harder to do well. For areas like analytics, though, it’s critical. Analysts need to communicate practices and learnings across – not just within – product teams. So I’d favor having at least one role (and maybe more) per area in the central team whose sole function is driving cross-team communication and sharing. This is one of those band-aids you slap on an organizational structure because it doesn’t do something important well. Every organizational structure will have at least a few of these.

In an ideal world, that collaboration function would probably always have at least two resource slots – and one of those slots would be rotated across different teams.

My final structure features highly integrated product teams that blend resources across every function needed to deliver a great digital experience. Those teams don’t dissolve and they don’t pass off products. They own not just the creation of a product, but its ongoing improvement. Almost needless to say, analytics (customer, behavioral and attitudinal) is embedded in that team right from the get-go and drives continuous improvement.

Those teams are supported by centralized groups organized by function (IT, Design, Analytics) that handle key support, integration and standardization tasks. These centralized teams are kept as small as is practical. Rotational policies are enforced so that people experience both centralized and product roles. Salary bands are kept very wide and the organization tries hard not to incent people out of roles at which they excel. Included in the centralized teams are roles designed to foster collaboration and communication between functional areas embedded in the product teams.

Finally, support functions like HR and Finance are mostly kept external. However, where compelling reasons exist to incubate them with digital, they are embedded in the central structure.

I won’t pretend this is the one right answer to digital organizational structure. Not only does it leave countless questions unanswered, but I’m sure it has many problems that make it fatally flawed in at least some organizations.

There are no final answers when it comes to organizational design. Every decision is a trade-off and every decision needs to be placed in the context of your organization history, culture and your specific people. That’s why you can’t get the right answer out of a book or a blog.

But if you’re building an incubated digital organization, I think there’s more right than wrong here. I’ve tried to keep the cop-outs and dotted lines to a minimum and focused on designing a structure that really will enable digital excellence. I don’t say deliver it, because that’s always up to the people. But if your Product Managers can’t deliver good digital experiences with this organization, at least you know it’s their fault.

Competitive Advantage and Digital Transformation – Optimizing in Travel & Hospitality

In my last post I described a set of analytics projects that drive real competitive advantage in retail and eCommerce. These projects are meant to be the opening of the third and final stage of an analytically driven digital transformation. They are big, complex, important projects that make a real difference to the way the business works.

But I know folks outside retail (and they’re the majority of my client-base) get frustrated because so much of the analytics technology and conversation seems to reflect retail concerns. So in this post, I wanted to describe an alternative set of projects specifically for another industry (I picked Hospitality) and talk a little bit about some of the key analytics flashpoints in different industries. Every business is unique. There is no one right set of projects when you get to this phase of digital transformation, but there are analytics projects that are quite important to almost everyone in a given industry.

Here’s a fairly generic set of projects I’d typically attach to a presentation on digital transformation in hospitality.  You can see that about half the projects are the same as what I recommended for retail.

Digital Transformation Phase III Hospitality

Aggressive personalization is a core part of MOST good digital programs – almost regardless of industry. If you’re in health-care, financial services, retail, travel & hospitality, government or technology, then analytics-driven personalization should be a high priority. It’s actually a lot easier to say where personalization might not be near the top of the list: CPG and maybe manufacturing. In CPG, many web sites are too shallow and lack enough interesting content to make personalization effective. In fact, the Website itself is often pretty unimportant. CPG folks should probably be more worried about their marketing and social media analytics than personalization. Manufacturers might be on the same level, but a lot depends on the type of industry, how many products you have, how many audiences, and how much content. In every case, the more you have (product, audience, content), the more likely it is that personalization should be a strategic priority.

I also included Surprise-based Loyalty. Travel is actually the sector where I first developed these concepts. You can read a somewhat more detailed explanation in an article I recently published in the CIO Outlook for Travel & Hospitality. But there are quite a few reasons why hospitality, in particular, is a great place to build a surprise-based program. First, the hospitality industry has numerous opportunities to deliver surprise-based loyalty at little or no cost. That’s critical. Hospitality also has the requisite data to allow for powerful analytic targeting and has sufficient touches to make the concept powerful and workable. What’s more, most of the rewards programs in hospitality suffer from scale. Sure, a few global giants have the reach to make a traditional loyalty program appealing. But if you’re a boutique or mid-tier chain, your traditional loyalty program will never look particularly attractive. Surprise-based concepts get around all that. With no fixed cost, the ability to target and grow them organically, and real impact on loyalty, they deliver a fundamentally different kind of experience that doesn’t depend on scale and global reach.

My third project is another one that could appear almost everywhere: mobile optimization. For Hospitality, it’s particularly important to create a great mobile on-property experience and build out the mobile experience as the Hub for loyalty. Integration of mobile digital experiences with property systems enables a whole array of real experience difference makers – room selection, automatic upgrading, room bidding, expedited check-in, door control, service requests and, of course, plenty of surprise (and traditional) loyalty opportunities.

Why didn’t this show up in retail? Hey, it could. It might be sixth on my generic list. But many of the retailers I’m working with are struggling to figure out how to make mobile an important part of the experience. With all the beaconing and wifi we’ve seen, most opt-in systems simply don’t get enough adoption to make them worthwhile. I think it’s easier to drive adoption in hospitality. And adoption is critical to driving serious advantage.

When I talk about advanced Revenue Management I’m clearly hovering somewhere on the edge of what might reasonably be considered digital. There are lots of different ways to improve revenue management, but what I have in mind here are two specific types of analysis. The first is using digital view volume to feed demand signals into revenue management. This is a simple but effective technique for taking advantage of your digital data to improve your price planning. I also believe that in the zero-sum game that is room (and flight) planning, there are opportunities to use digital data collection from OTAs to reverse engineer competitor pricing strategies and then optimize your price curve to take advantage of that knowledge.

In retail, I talked about the growing importance of electronic signage and integrated digital experiences and optimizing the measurement of those (largely unmeasured) systems. In Hospitality, I’ve picked something that isn’t quite the same but falls in the same omni-channel category – optimizing the integration of on-property with digital. This cuts in both directions and overlaps with the analytics around mobile (obviously), personalization (obviously), surprise-loyalty (obviously) and revenue management. Revenue Management a little less obviously but most revenue management systems use time-based pricing not customer based pricing – often completely missing differentials in customer value from on-property behavior. Casino’s, of course, are the exception to this.

For resort properties, there are significant opportunities to integrate digital view behavior into on-property drives. But for almost every type of property there are ways to make the on-property experience better. Some of this is ridiculously easy. When I log into my hotel wifi, I almost always get the standard property page. No customization. No personalization. But I’m a heavy consumer of certain types of on-property experiences including some highly-profitable ones like late-night room service dinners. Do I ever get a dinner drive? A special offer? A loyalty treat? Nope. Pretty much never.

I put this digital/on-property integration high on the list mostly because when it comes to hospitality, the on-property experience is THE critical factor. I might love or hate the Website or even the App, but both are just little bumps on the great big behind that is the actual stay. If I can help make the stay experience better with digital, I’ve done something important.

So my top five projects for hospitality are:

  • Personalization
  • Surprise-based Loyalty
  • Digital Additions to Revenue Management
  • Mobile Experience and Loyalty Optimization
  • On-Property digital integration

As with retail, none are easy. Most involve complex integrations AND deep analytics to work well. But they form a powerful and powerfully related nexus of programs that drive real competitive advantage.

Of course, as I’ve tried to make clear, the selection of a top-five is utterly arbitrary. Every business will have its unique strategic priorities, market position, and brand. Those things matter. What’s more, the third phase of an analytics transformation is open-ended. There aren’t just five things. You don’t stop when (if ever) you’ve done these projects.

So it’s natural to ask what are some other commonly important projects that didn’t make the list (and weren’t already captured in the earlier two phases). Here, with some notes about industries, are some more things to chew on:

Digital Acquisition Optimization (Campaign-level): I’ve already covered both a campaign measurement framework and Mix/Attribution in the first two phases. But I haven’t been quite true to myself since I often tell clients to worry about optimizing your individual channels and campaigns first before you worry about attribution. There are more powerful analytic techniques for campaign-specific optimization than attribution – and many, many enterprises would be well-advised focus on those techniques as part of their overall digital transformation. I won’t say that every digital media buy I see sucks. But a lot do. This one isn’t specific to industry; it’s important to anyone dropping significant dime on digital marketing.

Right-Channeling Support:  This analytics project often makes my top-five list in financial services, technology, and health care (but it’s important in a lot of other places too). Not only is the call-center a significant cost for many an enterprise, it’s almost always a significant driver of churn and bad experience. That’s not always because call-centers are bad – it’s hard to do well. And these days, many people (I’m certainly in this bucket) flat out prefer digital servicing in most use-cases. So digital servicing is a big deal and it’s deeply analytic. Bridging digital and call is a huge analytics opportunity and one of the most important projects you can take in a digital transformation.

Digital Sales Support: If a field-sales force is a core part of your business, then digital analytics to support what they do is often in my top-five projects around transformation. Technology, Pharma, and certain areas of Financial Services (like Insurance and Wealth) all need to figure out how their digital assets play with their field sales force. Siloed approaches here are worse than silos in digital marketing attribution. You can NOT do this well unless you tackle it as an integrated effort with consistent measurement across the journey.

Content Attribution: When I was at the Digital Analytics Hub in Europe one of the most interesting parts of the discussion around transformation focused on the need for traditional companies to become, in effect, media companies. There’s nothing terribly original about this idea (not sure who’s it is), but it is terribly important and often it’s a huge stumbling block when it comes to transformation. Companies don’t build nearly enough content to be good at digital and they don’t measure it appropriately. Learning how to measure the content experience and how to take advantage of content are keys to effective digital transformation and anyone focused on building deeper sales cycles should think carefully about making content attribution a prominent part of their initial analytics plan.

Balancing Success:  One of the biggest failure points in digital transformation in my client-base involves situations where a digital property has several very important enterprise functions. Selling and generating leads, advertising and engagement, linear vs. direct consumption, building brands vs. generating revenue. These are all common examples. The problem is that most enterprises are wishy-washy when it comes to balancing these objectives. When I ask senior folks what they really want (or when I look at how people are measured), what I usually hear is both. That’s not helpful. There are analytic approaches to measuring the trade-offs in site real-estate and marketing between driving to multiple types of success. If you haven’t done the analytics work to figure this out and set appropriate incentives and performance measurements, you’re simply not going to be good at all – and perhaps any – of your core functions.

Well, I could go on of course. But I’m almost at four pages now – which I know is excessive. There are a lot of options. That’s why creating a strategic plan for analytics transformation isn’t trivial and it isn’t boilerplate. But as I pointed out in my introduction to the last post, this is the fun stuff.

In my next post, I hope to tackle those organizational issues I’ve been deferring for so long – but I may have one or two more detours up my sleeve!

[BTW – Early bird sign up for the U.S. version of the Digital Analytics Hub is coming up. If you’d like a promo code, just drop me a line!]

Competitive Advantage and Digital Transformation – Optimizing Retail and eCommerce

In my last posts before the DA Hub, I described the first two parts of an analytics driven digital transformation. The first part covered the foundational activities that help an organization understand digital and think and decide about it intelligently. Things like customer journey, 2-tiered segmentation, a comprehensive VoC system and a unified campaign measurement framework form the core of a great digital organization. Done well, they will transform the way your organization thinks about digital. But, of course, thinking isn’t enough. You don’t build culture by talking but by doing. In the beginning was the deed. That’s why my second post dealt with a whole set of techniques for making analytics a constant part of the organization’s processes. Experimentation driven by a comprehensive analytics-driven testing plan, attribution and mix modelling, analytic reporting, re-survey, and a regular cadence of analytics driven briefings make continuous improvement a reality. If you take this seriously and execute fully on these first two phases, you will be good at digital. That’s a promise.

But as powerful, transformative and important as these first two phases are, they still represent only a fraction of what you can achieve with analytics driven-transformation. The third phase of analytics driven transformation targets areas where analytics changes the way a business operates, prices its products, communicates with and supports its customers.

The third phase of digital transformation is unique. In some ways, it’s easier than the first two phases. It involves much less organization and cultural transformation. If you done those first two phases, you’re already there when it comes to having an analytics culture. On the other hand, in this third phase the analytics projects themselves are often MUCH more complex. This is where we tackle big hard problems. Problems that require big data, advanced statistical analysis, and serious imagination. Well, that’s the fun stuff. Seriously, if you’ve gotten through the first two phases of an analytics transformation successfully, doing the projects in Phase Three is like a taking a victory lap.

There isn’t one single blueprint for the third phase of an analytics driven transformation. The work that gets done in the first two phases is surprisingly similar almost regardless of the industry or specific business. I suppose it’s like laying the foundation for a building. No matter what the building looks like, the concrete block at the bottom is going to look pretty much the same. At this third level, however, we’re above the foundation and what you do will depend mightily on your specific business.

I know that it depends on your business is not much of an answer. As a consultant, it’s not unusual to get caught up in conversations like this:

“So how much would it cost?”

“Well, that depends.”

“What kind of things does it depend on?”

“Well, it depends on how deeply you want to go into it, who you want to have do it, and how you want to get it done.”

All of this is true, of course, but none of it is helpful. I usually try to short-circuit these conversations by presenting a couple of real world alternatives.

I think this is more helpful (though it’s also more dangerous). Similarly, when I present the third phase of an analytics driven transformation I try to make it specific to the business in question. And the more I know about the business, the more pointed, interesting, and – I hope – convincing that third phase is going to look. But if I haven’t spent much time a business, I still customize that third phase by industry – picking out high-level analytics projects that are broadly applicable to everyone in the sector.

That’s what I’m going to try to do here, with the added benefit of picking a couple different industries and showing how the differences play out in this third phase. Do keep in mind, though, that the description of this third phase – unlike that of the first two – is meant to be suggestive only. No real-world third phase (certainly no optimal one) is likely to mirror what I lay out here. It might not even be very close. What’s more, unlike the first phase (at least) which is close-ended (when you’ve done the projects I suggest you’re done with that phase), phase three is open-ended. You never stop doing analytics projects at this level. And that’s a good thing.

For the first example, I decided to start with a classic retail e-commerce view of the world. It’s a sector where we all have, at the very least, a consumer’s understanding of how it works. There are many, many possible projects to choose from, but here are five I often present as a typical starting point.

The first is an analytically driven personalization program. With journey-mapping, 2-tiered segmentation and a robust experimentation program, an enterprise should be a in a good position to drive personalization. Most personalization programs bootstrap themselves by starting with fairly straightforward segmentations (already done) and rule-based personalization decisions targeted to “easy” problems like email offers and returning visitors to the Website. That’s fine. The very best way to build a personalization program is organically – build it by doing it with increasing sophistication in more and more channels and at more and more touchpoints.

Merchandising optimization is another very big opportunity. So much of the merchandising optimization I see is focused on product detail pages. That’s fine as far as it goes, but it misses the much larger opportunity to optimize merchandising on search and aisle pages via analytics. Traditional merchandising folks have been slow to understand how critical moving merchandising upstream is to effective digital performance. This turns out to be analytically both very challenging and very rich.

Assortment optimization (and I might be just as likely to pick pricing or demand signals here) has long been a domain of traditional retail analytics. As such, I have to admit I didn’t think much about it until the last few years. But I’ve come to believe that digital analytics can yield powerful preference information that is typically missing in this analysis. To do effective assortment optimization, you need to understand customer’s potential replacement options. In the offline world, this usually involves making simple guesses based on high-level product sales about which products will be substituted. Using online view data, we can do much, much better. This is a case where digital analytics doesn’t so much replace an existing technique as deepen and enrich it with data heretofore undreamed of. Assortment optimization with digital data gives you highly segmented, localized data about product substitution preferences. It’s a lot better.

I’ve become a strong advocated for a fundamental re-think of loyalty programs based on the idea that surprise-based loyalty with no formal earning system is the future of rewards programs. The advantages of surprise-based loyalty are considerable when stacked up against traditional loyalty programs. You can target rewards where you think they will create lift. You can take advantage of inventory problems or opportunities. You don’t incur ANY financial obligations. You create no customer resentment or class issues. You can scale them and localize them to work with a specially trained staff. And, of course, the biggest bonus of all – you actually create far more impact per dollar spent. Surprise-based loyalty is, inherently, analytic. You can’t really do it any other way. Where it’s an option, it’s always one of the biggest changes you can make in the way your business works.

Finally, I’ve picked digital/store integration as my fifth project for analytics-led transformation. There are a number of different ways to take this. The drives between store and site are complex, important and fruitful. Optimizing those drives should be one of the analytics priorities for any omni-channel retail. And that optimization is a combination of testing and analytics. In this case, however, I’ve chosen to focus on measuring and optimizing digital in-store experiences. You’re surely familiar with endless-aisle retail; where digital is integrated into the in-store experience. The vast majority of these physical-digital experiences have been quite ineffective. Almost always, they’ve been executed from a retail perspective. By which I mean that they’ve been built once, dropped into the store, and left to fail. That’s just not doing it right. In-store experiences are getting more digital. Digital signage is growing rapidly. Physical-digital experiences are increasingly common. But if you want actual competitive advantage out of these experiences, you’d better tackle them from a digital test-and-learn/analytics perspective. Anything less is a prescription for failure.

Digital Transformation Phase III Retail

So here’s my first round of Phase Three projects for an analytics driven transformation in retail. Each is big, complex and hard. They are also important. These are the projects that will truly transform your digital business. They are rubber-meets-the-road stuff that drive competitive advantage. It would be a mistake to try and execute on projects like this without first creating a strong analytics foundation in the organization. You’re chances of misfiring on doing or operationalizing the analytics are simply too great without that foundation. But if you don’t move past the first two phases into analytics like this, you’re missing the big stuff. You can churn out lots of incremental improvement in digital without ever touching projects like these. Those incremental improvements aren’t nothing. They may be valuable enough to justify your time and money. But if that’s all you ever do, you’ll likely find yourself wondering if it was all really worth it. Do any of these projects successfully, and you’ll never ask that question again.

Next week I’ll show a different (non-retail) set of projects and break-down what the differences tell us about how to make analytics a strategic asset.

[Just a reminder that if you’re interested in the U.S. version of the Digital Analytics Hub you can register here!]