How To Use Data

Once upon a time, I was working in an investment bank. This bank had a “black box” valuation model: you just input all the financial statements of an acquisition target and the key assumptions, and it spat out a value that your client should pay. I was appalled. I had worked before business school at an M&A consulting firm where you wrote the discounted cashflow model line by line yourself. This gave you the advantage of really ‘feeling’ the inputs and flow of the analysis within the model. Sure, it took longer, but it was the best grounding possible to understand financial analysis. It made doing my MBA at Wharton feel like a walk in the park.

Anyway, the young analyst on my team put in the data for the industrial company we were looking at. It was pretty small — $20MM in revenue with low profits — but growing at 10 percent per year. It should have been worth about $10–20MM, roughly half to one times revenue.

The analyst came back and announced in a matter-of-fact tone, “That company is worth five hundred million dollars.” 

I started laughing. “It can’t be,” I said. “That wouldn’t make any sense!”

“Yes, it is,” he said defensively. “ValueMac told me so.” That was our internal model.

Before we went to our deal lead, I insisted on reviewing every input. Sure enough, the analyst had typed in growth at 100 percent instead of 10 percent. No wonder it was worth so much.

Moral of the story? Do not believe the data unless it backs up your gut feel or industry standard expectations — or if it is a true surprise, check it upside down and backwards until you are absolutely confident there are no human input errors and that the data itself isn’t flawed.

Imagine you’re standing at the edge of a cliff, and the data says it is just a few inches high. You look down about 300 ft, a long way to the sea. Would you jump? I really hope not.

I ran a multi-billion-dollar business line at a bank, lending customers money. This bank was way ahead of any industry competitor — or indeed any company — in its use of data. The strategically brilliant CEO always told us, “80 percent of the credit risk of the customer can be told by the FICO score; but to understand  closer to 100 percent of the credit risk, we use data that competitors don’t and so we make better lending decisions.” The point: if you choose to lend to a potential customer who appears high risk to a competitor but you know is low risk — and avoid one who appears low risk but is actually high risk — you will outplay the market every single time. However, like anything, this can be played ad absurdum.

In 2010, the concept of big data gained huge traction: getting as much data as possible and analyzing it to death to get that tiny edge over competitors. It requires huge effort, lots of time, and massive server capacity. At this point, the analyst loses any feel for what the answer might be, so companies blindly follow the data.

Big data can be a disaster if thought isn’t put behind its use. In 2012, Target famously used big data to send maternity clothing offers to women whom its analysis suggested were likely pregnant. One included a teenager whose parents didn’t know. Until the mail arrived. Oops.

Big data also causes other problems. In 2017, Equifax, one of the world’s largest credit bureaus, lost data for 147 million people. This was a massive security and governance failure, leading to a $1.4BN fine (ten times net cashflow) and the ouster of their CEO. Leaders, be warned.

Even Google can screw up. In 2008, they created a model to predict flu outbreaks. It overestimated cases by 50 percent, putting everyone in a tizzy. The program was shut down by 2015.

Worst of all, Microsoft created AI model Tay, which ingested Twitter data. Within hours, Tay was generating racist and offensive content due to biased training data. It was immediately shut down — in less than 24 hours.

All these issues could have been avoided by a less cavalier attitude to model outputs. Tight security, pre-testing results, and common sense dramatically improve success chances. That’s not rocket science — even if analyzing big data is common in actual rocket science.

Data bias is real, even when you try to be neutral. Amazon built a recruiting algorithm in 2018 to help hire the best people. They trained it on ten years of data, mostly men. Unsurprisingly, the model was biased against women. Larry Summers, when president of Harvard, suggested that differences in gender representation in math and science (more men in higher levels) might be due to differences in aptitude but this has not proven out in models.

If you train your model only on current customer or employee data, you will have survivorship bias. Test new markets, geographies, demographics, and low-frequency segments.

Even if you’re not the analyst, you need to understand the background of the model. Ask questions. Consider sources carefully. Rubbish in equals rubbish out; bias in equals bias out.

Beware regression to the mean, a concept taught throughout the bank by our afore-mentioned brilliant CEO. This is the statistical tendency for extreme events to be followed by more average ones. A handful of customers buy a ton of your product — they are outliers. Or, your Day One marketing spike collapses by Day Seven — you think it failed. Actually, the lower rate may still be 20 percent above baseline. Well done!

You think you’ve found the perfect customer — your own personal unicorn. You study, model, and lavish them with attention. And then, like clockwork, next year they behave like normal humans. That is regression to the mean: the universe reminding you that perfection is fleeting, and luck is volatile.

The danger isn’t just that unicorns revert to ordinary — it’s that if you chase only “perfect” ones, you end up with a shrinking club of rare creatures, ignoring a whole herd of perfectly good, but less dramatic prospects. In other words, a boutique audience for a one-hit wonder.

How to survive? Broaden the net. Test, sample, retrain. Embrace the ordinary, the messy, the unpredictable. Because while unicorns are fun to chase, it’s the robust, diverse herd that keeps your business alive — and your sanity intact.

Lastly, the biggest risk: if you don’t go outside your data sources for unusual or unexpected signals — for example, to customers you think are uninterested — you will lose. Your model will get better at selecting only the “best” customers until your lending model says, “Shut up, just lend to Bill Gates. He’ll always pay back.” Not helpful. Customer behavior changes. Unattractive customers from the past could be super attractive now. And Bill might not want your money. Like I said, people change, and all your eggs would really be in one basket. Test and learn.

Facebook ads over-targeted likely customers and ended up with few. Retail credit models approved fewer people (only the best), destabilizing portfolios. Netflix trapped users in algorithms, giving only what they had shown they liked. Preferences change. You’ll never discover new options if you select more of the same.

I love Apple News. I discovered I am really interested in geography and ancient civilizations, more than I thought. Apple sends me lots of this content. I thought I liked scientific innovations. I skipped those articles for months. Now I don’t get any. Boo hoo. Apple, improve your algorithm! Go outside the box, test me regularly, please.

So if you’re analyzing data — surely all of us are these days — inject randomness. Use untargeted data. Test something you don’t believe. Test something crazy, but don't bet the farm. When you find that positive surprise and act on it, you will get outsized market gains. Enjoy!

Janet Lewis Matricciani is a two-time CEO who has worked all over the world and is multilingual, now sharing her business lessons publicly. She is a regular contributing writer to this magazine and can be reached at jlmatricciani@gmail.com. Click the link to see more of her articles.