Let’s say you want to train a neural network to advise on a healthy lifestyle. You make a randomized questionnaire and hire a bunch of people to answer the questions like “how does drinking
[1 ∨ 2 ∨ 3] cups of water every
[morning ∨ afternoon ∨ evening] makes you feel?”
Now let’s imagine, for the sake of this mental experiment, all your respondents have started to take cocaine. They now feel great all the time no matter how bad their health is getting.
“No sleep for two days?” “Great!” “Jogging for seven hours?” “Super!” “Bourbon for breakfast?” “That would be splendid!”
And that’s what your neural network learns.
And now let’s imagine, for the sake of the experiment, that suddenly the cocaine is gone. It’s just no more. No supplies, no suppliers, it’s all physically gone.
Your respondents are starting to feel the toll of their decisions. They feel bad. They feel awful. They feel like they need a piece of healthy lifestyle advice, so now they are asking your network what to do.
And the network, while not trying to be particularly harmful, can only make things worse. It has the authority of many people contributing to its knowledge, but most of that people were high during teaching the network, and very few of them are ready to admit that.
So bourbon for breakfast remains a “healthy lifestyle choice” for the next generation.
Not surprisingly, people in large quantities behave pretty much like neural networks. Communities, committees, even governments, learn from measurable metrics just like machines.
In software development, Moore’s law was our cocaine for decades, and now it’s gone, and all we have is a bunch of methodologies and practices that suddenly stopped working.
Well, they were never working, to begin with. We only felt great because of Moore’s law. Hardware was getting better faster than the software was getting worse.
Ok, but you might think, “I’ve heard this ‘the end is near’ nonsense before. And nothing changed much so far. Can’t I afford another decade of denial?”
Moore’s law has several limitations.
- First, to make a transistor, you need 3 atoms. That’s about 1 um in size. We’re already at the 7 um process. So even discarding all other problems, we can only make 49 times more transistors fit the same space. That’s enough to feed Moore’s law for less than a decade. And that’s the absolute physical limitation we will never reach anyway.
- Because with extreme UV you can do photolithography as detailed as ~10 um since this is the length of the light wave. But you’ll need exponentially more energy and you will still have more defects when you go smaller. More defects mean more culling and higher prices for the remained product.
- Theoretically, we can fit more transistors in the same space if we discard the planar configuration. Except that we wouldn’t because, without effective heat transfer, the heat will accumulate and melt the bed.
Yes, we still have some growing space, so there will still be some progress in CPU performance in the near future. But it wouldn’t be exponential anymore.
In fact, 2018 was the first year when the performance of top Intel CPUs have not improved but even declined due to Meltdown and Spectre issues mitigation.
Which brings us to the real issue.
What do we do with our “neural network”?
There are practices, methods, and habits developed under Moore’s law. There is a visible consequence of these practices: modern software is crap.
I think, we need to start anew. Just forget everything you learned that comes from the era of self-delusion and start measuring.