A lot of stress is placed on being data-driven. Data is nice. It builds confidence in decisions. But it's often taken too far, beyond the point where diminishing returns have burned the marginal value away.

I'd like to share an argument for pulling back the reins a little bit.

Data optimizes funnels. But funnels are influenced by far more than most data captures. Are the users being charmed and feeling butterflies, or are they just grumpy about needing to use a product they're planning to switch off next year? They're engaged, but do they like the product?

Does it matter?

A Case Study

Pro tip: If anyone suggests a design change you don't like, just say "Do you have any data to back that up?" Nobody knows how to answer this question.

—Mark Dalgleish (@markdalgleish)
July 11, 2019

Data-driven decisions are based entirely on proxies, so it can be hard to grasp exactly how much influences them behind the scenes. Let's walk through an example: Assassin's Creed.

The first Assassin's Creed was a risky move for a game developer: a new team building the entire game on new technology in a new genre. It was not perfect, but the risk paid off.

The Assassin's Creed II trilogy resolved some shortcomings and took some more risks. That's what sequels to risky games are meant to do; see also Portal 2.

But by Assassin's Creed III, it was clear Ubisoft had taken a different stance toward the series—the one that milks them dry. They had invented the perfect cow, and now they could sell milk.

The games started looking like yearly sports games. Ubisoft announced proudly that it had spun up a second Assassin's Creed team so they could trade off releases every year. Wow! What a great idea.

This is pure opinion, but starting then, every other Assassin's Creed game was a dud, and the series as a whole was on a downward slope. They were meticulously crafted time sinks, just barely fun enough to keep playing.

This is the fast food industry of gaming. Cold, calculating, efficient—and effective! Assassin's Creed games are wildly successful because of this strategy. But it overlooks one thing.

Loyalty

People catch on. They realize they're being sated, but not satisfied. It might take one game, or two, or ten, but eventually people silently stop bothering to keep buying the games.

The opposite of this strategy is Nintendo. Nintendo doesn't give a shit. They love making amazing games, data be damned.

They take risks, they do weird stuff, and it sometimes flops. But they never milk the cow. The Wii U was a flop. Jumping around your living room to play Wii was a flop. But they weren't mistakes. Each of them was a step towards the Switch.

For every mistake, Nintendo gets something right. The control stick. The 3D platformer. The twin-stick first-person shooter. The home video game system. The handheld console. The open-world adventure game. The metroidvania. Where would the video game industry be if Nintendo milked their cows?

Ubisoft and Nintendo are both successful companies by many measures, so it's hard to point at them and say "I like this one better, so the other one must be flawed". Any company as successful as Ubisoft must have done many things right, and far be it from me to tell them their philosophy is wrong.

But as someone with a couple of successful products, it's important for me to make sure my philosophy is in tune with my ideals.

I want to be like Nintendo. And I don't know for sure to what extent they use data, but anyone who's interacted with their software a lot knows it's at least an order of magnitude less than a traditional tech company.

A Personal Aside

I chose Nintendo as an example here because I like them. I line up about 18 hours before any home Nintendo console launches. It's chill, with people passing cocoa, playing guitar, and being thankful Nintendo console launches don't yield near-death experiences and eBay scalpers like other console launches.

It was this chillfolk vibe for the Gamecube, the Nintendo DS, and the Wii. But for the Wii U, something was different. No one else showed up. I sat alone in the dark all night. An unseen concerned citizen, presumably an employee not knowing why I was there, called the cops on me. By the time the doors opened, only nine people had lined up, all after sunrise.

At those wee hours into the night, staving off sleep with excitement and hot cocoa, I remember thinking Where is everybody? Did I get the date right?.

I did. There was no one there because the Wii U was horribly marketed ("So it's an add-on for Wii?") and had no launch games worth buying one for. Nintendo failed me. I'd done what superfans do: I gave my heart, my time, and my money, and it was all a big let-down.

I don't care. Look at this:

That's my loyalty bar towards Nintendo before the Wii U. Now look at this:

That's my loyalty bar towards Nintendo after the Wii U. Only a handful of brands in my life can take a hit like that. Assassin's Creed certainly isn't one of them.

Nintendo and Risks

Nintendo takes crazy risks based on what sometimes feel like acid trip ideas. Think about the premise of Animal Crossing for a second.

You can't make this up by looking at data, but it worked, and it filled loyalty bars. These bars aren't exposed in data, at least not until a lifetime of it passes you by. But everyone has one.

Bad decisions can reflect well on the data while still being bad decisions. If you make it impossible for users to cancel their subscriptions, will fewer users cancel their subscriptions? Probably! Does that mean you should do it?

Conclusion

Being data-driven is hard. Adding instrumentation and A/B tests, along with extra specs, coordination, and maintenance takes a lot of time that could be spent building.

Try making the 80/20 trade-off. Put an analytics snippet in your footer or whatever the trend is nowadays. But ditch the experiments framework, scaffolding, headache, and personpower in exchange for a 10x velocity on gut-driven features.

Hard-to-measure things like loyalty bars defy the charts, but are at least as important as them. If you one day discovered users just wanted to be charmed and feel butterflies, wouldn't you charm them and build butterflies?

If you're unconvinced, run an A/B test.