This post came about when my wife told me that she was playing a certain game in iOS and nobody she knew could go past a certain level in the game. She tried herself and her nephew also tried and failed. My son who is 4 years old and knew nothing about software or games was able easily go past that level. The reason he was able to do that was because he approach was random, and when he tried a burst of random approaches with no particular logic, he was able to break the level.
There is a software analogy to this as well. When any software gets mature, we tend to frown upon major changes to it, or someone "breaking the rules" and writing code on it. Every architect has been on this side at some point in the career. I have done this myself in cases where I knew for a fact that this would break something. I have also vehemently objected to a requirement which would cause me to break a lot of code while trying to implement it.
In software as in every other field, we tend to frown upon failure of any sort. If we worked on a project and it failed after 6 months, or was unable to attain what we wanted it to attain, this is considered a failure as well.
Is it really?
Today I feel the opposite. Everybody is different. When we tighten the rules and put a straitjacket around a framework or software, we are basically preventing any kind of out of the box activity from happening. We know this is difficult to do, or it breaks something - but how do you know that for sure? - maybe you failed to do something in a certain way - someone else maybe able to do what you were unable to do.
How would you know what was possible, if we prevent someone from trying out a different approach?
Sometimes, even when a different approach "fails", a lot of different things come out of the effort improving the code base in ways we never thought possible - because nobody every looked at the code from that particular perspective before.
Perspective is everything when it comes to large code bases. Because beyond a certain size, it becomes too much for one person to internalize. At that point of time, perspective is the only thing through which you can gain understanding of the system. When it changes, the understanding changes, and even evolves. You find over time that what you thought impossible yesterday is very much possible today.
And none of this is possible, if you never tried out a new approach, or went down the risky path.
A concrete example of this is when I had to add a new mode to an existing software which made it more reliable, but also quite slower than before. I tried out a very different approach to make this as fast as possible - basically indexing files beforehand, instead of doing it when the process was running. This was not enough - so I had to go around and make other parts of the code run 10X faster than before, and utilize multiple cores. The end result was that even if in the new mode, it was as fast as the earlier mode or maybe a bit slower, if you ran in the old mode, it was 10X faster.
If I had never tried the new mode saying it was too difficult to do, I would never have improved the old mode, and found hundreds of small but important things which made the software better in the long run.
How is this related to Nature?
We write this simple, cave-man software and prevent others from trying out new things, or worry about risk. But look at nature which has created plants, birds, mammals and even the human brain. We have reached this stage in evolution because Nature tried every random permutation and combination based on the given parameters and optimized along the path which is just right to reach the current stage today.
It tried all approaches, all risks. Entire species dies, new ones arose, but it still went forward and continues to move forward.
Random approaches and failures are necessary as they provide us with as much information as the successes do. Only through learning both success and failure can we really move forward. Infinite patience is required to find the right solutions for difficult problems. Solutions are not just found through brute force human intelligence or infinite CPU power.
I think a major failure of current software is that we build something and we tighten the rules as we go along, and then new ideas have to create something new from scratch. We never continue to build upon the same thing, evolving it over time. So, we are forever going in a circle, reaching nowhere because of this. All logic is useful, whether it was written in FORTRAN or PASCAL or C++.
Discussing these ideas with others, they mentioned that this might be the reason that in some companies on the west coast, they tend to keep only younger employees and not experienced ones. Because, even though experienced employees are technically competent, they are highly resistant to change and unmalleable to consider new approaches and risky techniques. So, companies keep younger employees who don't know enough not to take risky approaches.
Whatever be the case, I can't generalize that older employees are always resistant to change. They have strong opinions for sure, but I have worked with very constructive senior developers who embrace the fact that we are building something and every time, this has resulted in massive success. On the other hand, I have also worked with very capable senior developers who are very resistant to change and very nonconstructive in terms of showing only negativity when faced with having to build anything new. There have been teams which are highly "aligned" to resist change, even building frameworks which actively straitjacket any new approaches to code. A particular instance was a service architecture where the code would error out if you returned an object with two properties instead of one, or you had to duplicate a class in two assemblies deliberately just to be able to write the service layer code.
