You can’t win

I’ve been reading and listening to Jeff Pfeffer on evidence-based management. Basically, the good doctor wants managers to look at the theories, instead of mearely going by beliefs that sound ruthless enough, i.e., in line with with Econ 101, e.g., ‘fire the losers, give stock options to winners, etc’.

Two problems come to mind immediately. Managers may not know how to look at evidence. Those who come from the Arts, may not know any statistics, and may even be scared of numbers. Those who come from the Sciences, while somewhat better equipped, may over-estimate the power of numbers… mistake data for information. Men and women are not screws and nails. Companies are not bridges and towers. Markets are not computer models.

The second problem is that the evidence at hand may be misleading for complex systems. Let’s take, say, stock options. Apparently, there is no evidence that it makes any difference. In some cases, it is counterproductive. So what? The question is not ‘What does it do in general?’ but ‘How do the people who do it right do it?’

(Let me take an extreme example to amplify: There is no evidence that keeping accounts will make your business prosper. All businesses keep accounts of some sort; most fail. Hence, we should conclude that keeping accounts is immaterial to business success.

But we don’t. Instead, we try to figure out the best way of keeping accounts, because commonsense tells us, without the need of any research, that accounts should matter a great deal.)

I have the same problem with this thing as I had with The Bottom Billion by Paul Collier, namely that applying data analysis may be as much as an over-simplification as not applying any. A diverse, representative sample may be as wrong as a sample of a single failure or a single super-success.