All the models are wrong and all we have are models.

When I was in college I studied Economics. If I could go back in time I’d kick myself but that’s the choice I made.

While studying I observed this tendency in modern economic theory towards looking at what happened and building a mathematical model to explain it. From this model you would try to extract predictive power.

Further, all of these models assume “rational behavior.” I’ve never once in my life seen rational behavior.

None of this ever sat well with me.

If these models had extreme predictive power we would be able to reliably predict a lot of major economic events. Read the news and you’ll see that’s hardly the case. Then there’s the fact that these models are post-rationalizations. Sure they try to cross-check under different conditions but economics isn’t really an experimental science.

There’s a saying I learned while studying that goes “ask 100 economists what they think and you’ll get 101 different answers.” The implication is that economists basically all operate on their own individual viewpoints. And even then they can’t always agree with themselves.

The main thing my study of Economics left me with was a tremendous amount of doubt in statistical models. Not what I was supposed to walk away with but it was probably worth the price of admission.

The weird thing about the world is that there is very little we can actually know for certain. We build mathematical and rational models and use them to predict the world. Some of them are very, very good models. But they’re still just models.

People often think they know things, but that’s different. People definitely want to believe we can know things and reliably predict events. Certainty makes us feel nice and safe. It helps us sleep at night.

This desire can be a little dangerous. Trusting dodgy models creates overconfidence. In fact, the definition of “overconfidence” is exactly this. You have mental models of how the world works that you think are better than they are and you trust them too much.

Then there’s this other tendency that pairs up nicely. People would rather go with a plan that relies on a model than one that goes on any other seemingly irrational, unquantifiable approach. If you can make a statistical model and compose a rational sounding theory, people trust it much more than an idea that is based more on experience and heuristics.

This kind of thinking makes sense on paper of course. But it discounts experience and forgets that models are a type of heuristic.

Let’s try to make this more tangible.

One place that I’ve often seen modeling and reasoning win out over “I have a very bad feeling about this” is outsourcing. I’m a software developer and I’ve been on many projects where people try to outsource to cheaper developers overseas. This makes a lot of sense on paper and you can see all the companies out there doing it and feel safe.

Several projects I’ve been on I was actually brought in because this went less than well.

Here’s the problem with outsourcing that quite often happens. The people you are outsourcing to are paid hourly and not much at that. They are in a wildly different time zone. They don’t actually care about your product. They want to work as many hours as possible to make as much as possible for as long as possible. They are going to do the bare minimum to keep the project going but aren’t going to try to make your product better. They’ll just do what they’re told in a half-assed way and that’s that.

If you’ve worked with a few outsourced projects you likely know what I mean.

The problems here are very hard to model in numbers. I can make a somewhat rational argument for why it’s a problem but I can’t easily assign dollars to it. Some projects seem to do okay with outsourced development. Many stagnate and fail entirely.

Models aren’t inherently bad. They give us insights and help understand the world. But we can't fully understand the world. A fact worth remembering. Models are useful for guiding us towards bets and experiments but practical experience and real-world results are really what matters.

Always question models because reality has a way of doing something else.