Software Engineering is dual natured: both a particle and a wave

The title of this post maybe pretty obtuse and hard to make sense of. I'll try to explain.This post is to disparage everyone who evaluates software engineers only by checking to see whether they know how to sort an array fast or ask algorithm questions.

The fundamental reason why this is a bad idea is because similar to most natural phenomena and physics in particular, we are making the very wrong assumption that because someone knows everything about the building blocks of something, that person can be a great engineer building applications.

This is completely wrong. And I'll explain why...

In the beginning there were Newton's laws which could explain most natural phenomena at the physical level (laws of motion, etc). This is what I call as the "wave".

Later, it was found that these laws break down at sub atomic levels (basic building blocks). This is what I call the "particle".

Essentially, physicists realized that two completely different sets of rules defined what happens at the sub atomic level and what happens at the larger levels of physical phenomena.

Boiling down the essence of this means that just because you have a deep understanding of how something works internally, you cannot use those equations in your daily life. For that, you can rely on much simpler set of laws like Newton's laws.

Of course, you can do everything at the sub-atomic level all the time, which would mean that you spend all your time on figuring out how those very complex equations map at the physical level.

How easy would it be then to do anything practical fast? - Try to model the path of a ball thrown at 45 degrees using quantum mechanical equations...

This is also why these guys who know the internals very well cannot produce any tangible results to solving big problems. They are applying particle physics in situations where they are supposed to use Newton's laws.

I may have mentioned these very thoughts in some previous posts. Why I am writing this post is because I had a concrete example of this in my recent work.

The particle part of software is the underlying data structures and algorithms internal to a high level framework like .NET and C#.

The wave part of software is when we build complex software to do some workflow using these building blocks. This is more similar to a process in Applied Electronics & Instrumentation more than linked lists and sorting algorithms.

A very distinguished engineer tried to solve the out of memory issue using the particle methodology. Use a more efficient data structure to reduce the memory usage. He tried for months and failed and the software could never handle more than a certain amount of data.

When I saw the problem, I approached it from the wave perspective. Irrespective of the data structure used, or the sorting algorithms used - the problem was because of the way the entire process worked. It was similar to a factory where the pipes feed chemicals into the plant to manufacture something else. The input flow needs to be regulated to the consumption. If this is not done, the process will overload - run out of memory.

I fixed the problem which had remained open for 8 years. I used no sorting algorithms or specialized data structures. Now the problem is completely resolved.

It is true that understanding the building blocks of software can help in rare situations where the higher level code does not work the way you think it may - because of how it is coded internally. However, this is an extremely rare situation. Most software projects fail a lot before it reaches that stage. Just think about how buggy Android is - and it is built by people who are from the top most universities and have the most degrees. The reason for this is firstly their arrogance that they will only hire people from some top colleges to build software which will be used by everyone. This is also because they think too much at the particle level, while they should actually be thinking at the wave level.

The best examples of this kind of follies are: Google Wave, Android and even the most horrible to use Gmail.

More Data

It is always a struggle to provide a more clearer explanation of what I am trying to say here. Let me give you a few more examples which clearly show how a system is not the sum of its parts.

• A factory consists of many parts mechanical, non mechanical and electrical all of which work together for the most part to produce the output. The behavior of the factory cannot be calculated as the sum of its parts because when parts come together they interact in new ways which may not always have been foreseen before. Even if it did, when we add more parts and go on, eventually we reach a point where we have to consider the entire system in a holistic manner rather than try to optimize atomically.
  
• For Software Engineers who love to go back to the building blocks because it allows them to understand "better" how it works - why not go back to the hardware level? No software runs in isolation. It always run on top of some hardware which sometimes changes the behavior of the software. So, if we want to "really", "comprehensively" understand how best to optimize the software or how it works, perhaps, we should first stop at machine language and optimize at that level with compilers, and then go deeper into how the hardware handles the instructions at the chip level and fix that as well.
  
  If this seems nonsensical to you, then the whole talk of data structures should also. Because all structures are static and there is only so much which can be done at that level. Software when running in a system consists of rich interactions between live instances of objects with complex behavior. This is best optimized as a whole process, rather than as the sum of its parts.

Using a Design Pattern does not mean do not design the solution

This is an idea which I have explained a lot to others previously. In that example, I designed a workflow system and compared to an existing design pattern. I found that my design did not have the bug the design pattern had - which was mentioned as a bug in the design pattern. Still, we find scenarios where people think that if you do not know a design pattern that is some indicator that you are not a good developer.

It is not how much information you know - it is about how you use the information you already have which makes a great or so so developer. As Einstein said: "The true sign of intelligence is not knowledge but imagination." - I saw this in my kids kindergarten wall.

Let us go really deep into the implementation of a Producer Consumer design pattern as they seem to call it which was full of holes. This time I can write at length about this because I just finished fixing a problem which nobody else was able to fix so far.

The problem is that we have a set of threads which are spawned from a set of timers. These threads share the load of performing different work items. One such work item comes from a queue which contains two types of items:

1) Which generates an item which can be consumed directly.
2) Which generates an item which generates more items - of both types (producing items and/ or items which can be consumed).

It looks straightforward, but this will not work properly. This is because these items which get into the queue can come in any sequence - which depends on the dataset from which they are generated from. So, we have a very complex system whose behavior is very dynamic and dependent on the million different combinations of the input dataset.

This system would eventually end up either sleeping indefinitely or running out of memory because of the following reasons:

1) There is no control to regulate the fact that one producer can add so many items into the queue that we run out of memory.

2) There is no control to regulate the fact that because the queue can contain items in any sequence, we don't end up in a scenario where there are only producers and no consumers so we again run out of memory.

3) If we do try to sleep the producer temporarily to control the memory usage, we got to do it in such a way that, there are adequate consumers to consume what is in the queue, and some consumers are available always to get the enqueued items below a limit.

The more I encounter real-world applications of software design patterns, the more I see badly designed software which does not work properly because the developer ceded completely to what is written in some book somewhere. 

I see bugs of the worst kind in both design patterns I've seen in the field. This makes me really wary of the next time I see a design pattern in code and someone tells me.. oh.. we are just using the XYZ design pattern here, it takes care of everything you see...

Design is not code behavior and may not even be how the code works in reality

People sometimes use the word software design in a promiscuous manner. Pretty much anything and everything fits into "design". That is not really true. In reality, design is a lot of things and none of them may directly relate to the code - it influences the code, rather than having a direct relationship. I say this because the design may say X and the code may wrongly do Y. Then it is the flaw in the code and not in the design.

What really comprises of software design?

Design is really a set of ideas which are noted down when thinking about the implementation.

- The set of assumptions which you consider when thinking about the implementation. This is the world view of the code. If a new assumption comes along, that can make or break the existing design.

- The goals of the project - "ultimately when all is said and done, this is what the implementation will accomplish".

- A good design may mention stuff which goes tangential to what a developer may think. A good example is in a recent project "by design", the assumption was that the database rows cannot be fully trusted because they could be out of date w.r.t the file system - so they database rows can only be used as an indicator that something may need to be done. The final word needs to come from the file system.

- A design artifact may not always be re-discoverable by reverse engineering the code. In most cases, it may take months of looking at the code behavior to assume that "one of the original intents of the coder may have been to do X and Y". Even then we cannot be fully sure about it. Hence, it is always important to have a design document, because without it, we cannot add or modify to an existing design without risk. The code could be wrong w.r.t the design document, but still it is important to know whether your thoughts are in alignment to the original designer of the code if the major project goals and assumptions remain the same over time.

- A design always considers the high level interactions between the various entities in the system. This need not always be software modules which are written by the developer implementing the project. It could be third party stuff like OS, IIS, file system, database, etc.

- The two minimally required UML artifacts for design are Use Case Diagrams and Activity Diagrams. UML makes it easier to go about designing a complex project where we cannot think of everything "at once". It helps us split the work in a structured fashion into small "bite sized" pieces which are easily understandable and digestable.

Using the concepts of nature in software

This post came about when my wife told me that she was playing a certain game in iOS and nobody she knew could go past a certain level in the game. She tried herself and her nephew also tried and failed. My son who is 4 years old and knew nothing about software or games was able easily go past that level. The reason he was able to do that was because he approach was random, and when he tried a burst of random approaches with no particular logic, he was able to break the level.

There is a software analogy to this as well. When any software gets mature, we tend to frown upon major changes to it, or someone "breaking the rules" and writing code on it. Every architect has been on this side at some point in the career. I have done this myself in cases where I knew for a fact that this would break something. I have also vehemently objected to a requirement which would cause me to break a lot of code while trying to implement it.

In software as in every other field, we tend to frown upon failure of any sort. If we worked on a project and it failed after 6 months, or was unable to attain what we wanted it to attain, this is considered a failure as well.

Is it really?

Today I feel the opposite. Everybody is different. When we tighten the rules and put a straitjacket around a framework or software, we are basically preventing any kind of out of the box activity from happening. We know this is difficult to do, or it breaks something - but how do you know that for sure? - maybe you failed to do something in a certain way - someone else maybe able to do what you were unable to do.

How would you know what was possible, if we prevent someone from trying out a different approach?

Sometimes, even when a different approach "fails", a lot of different things come out of the effort improving the code base in ways we never thought possible - because nobody every looked at the code from that particular perspective before.

Perspective is everything when it comes to large code bases. Because beyond a certain size, it becomes too much for one person to internalize. At that point of time, perspective is the only thing through which you can gain understanding of the system. When it changes, the understanding changes, and even evolves. You find over time that what you thought impossible yesterday is very much possible today.

And none of this is possible, if you never tried out a new approach, or went down the risky path.

A concrete example of this is when I had to add a new mode to an existing software which made it more reliable, but also quite slower than before. I tried out a very different approach to make this as fast as possible - basically indexing files beforehand, instead of doing it when the process was running. This was not enough - so I had to go around and make other parts of the code run 10X faster than before, and utilize multiple cores. The end result was that even if in the new mode, it was as fast as the earlier mode or maybe a bit slower, if you ran in the old mode, it was 10X faster.

If I had never tried the new mode saying it was too difficult to do, I would never have improved the old mode, and found hundreds of small but important things which made the software better in the long run.

How is this related to Nature?

We write this simple, cave-man software and prevent others from trying out new things, or worry about risk. But look at nature which has created plants, birds, mammals and even the human brain. We have reached this stage in evolution because Nature tried every random permutation and combination based on the given parameters and optimized along the path which is just right to reach the current stage today.

It tried all approaches, all risks. Entire species dies, new ones arose, but it still went forward and continues to move forward.

Random approaches and failures are necessary as they provide us with as much information as the successes do. Only through learning both success and failure can we really move forward. Infinite patience is required to find the right solutions for difficult problems. Solutions are not just found through brute force human intelligence or infinite CPU power.

I think a major failure of current software is that we build something and we tighten the rules as we go along, and then new ideas have to create something new from scratch. We never continue to build upon the same thing, evolving it over time. So, we are forever going in a circle, reaching nowhere because of this. All logic is useful, whether it was written in FORTRAN or PASCAL or C++.

Discussing these ideas with others, they mentioned that this might be the reason that in some companies on the west coast, they tend to keep only younger employees and not experienced ones. Because, even though experienced employees are technically competent, they are highly resistant to change and unmalleable to consider new approaches and risky techniques. So, companies keep younger employees who don't know enough not to take risky approaches.

Whatever be the case, I can't generalize that older employees are always resistant to change. They have strong opinions for sure, but I have worked with very constructive senior developers who embrace the fact that we are building something and every time, this has resulted in massive success. On the other hand, I have also worked with very capable senior developers who are very resistant to change and very nonconstructive in terms of showing only negativity when faced with having to build anything new. There have been teams which are highly "aligned" to resist change, even building frameworks which actively straitjacket any new approaches to code. A particular instance was a service architecture where the code would error out if you returned an object with two properties instead of one, or you had to duplicate a class in two assemblies deliberately just to be able to write the service layer code.

The Code Paradox

It is kind of interesting for me to note that when the code is really bad and very buggy, making small changes to it, does not affect its overall quality a lot. In many cases, defects combine to get something working together in a certain way.

What is more interesting is that when the code is quite good, well written, modular, reusable making small changes to it can actually break everything. So, it affects quality a lot more, and small things cause more stuff to break especially because of reuse.

This could be entropy. The natural state is disorder and it stabilizes there.

When we try to bring order, that is an unnatural state. So, I have seen many, many cases where the code is really good, and someone unfamiliar with it makes a slight change and it keeps breaking all over the place.

The real challenge here is - how do we write software which has entropy by design but still works well? - because I believe biological systems which work well with the most diversity and population are like that.

Is it not weird that even in software, which is totally man made, this basic law of entropy remains true? - just shows that in a system where entropy is the law, even for seemingly autonomous entities (like us) who create new things - the things we create however abstract they are also stabilize at entropy.

Embrace the chaos?
Embrace the diversity of thought which makes a single code-base have both good and bad code?

Definitely something to think about.

Lesson 7: Micro level understanding does not translate into macro level excellence

This is an important topic because the number one problem which tech companies face is that they are unable to figure out who to select in an interview. Most of the time, the selected person is unable to fulfill even a percentage of what they thought she/ he could accomplish.

There is a good reason for this anomaly. It is because in the computer industry it is widely assumed that just because someone has understanding of microscopic details regarding computers, some of which maybe arcane, that translates into excellence at the macro level.

From my experience, this has been proven wrong again and again, and I have learnt even from my experience hiring people.

Basically, the theory goes on that say if you understand computer science concepts like data structures, or stuff which is really important at a system level (like operating system concepts, and other details about compilers, linked trees, etc) then wow - you are a find! - if we hire you - the next day you will be productive.

On the other hand, if the guy does not even know data structures, and cannot solve the complex b-tree problem posed in the interview, then he must be a total loser.

This is total nonsense. As Einstein said: "The true sign of intelligence is not knowledge but imagination." - I saw this quote in my kid's kindergarten. Maybe we need to send such people back to kindergarten. What I've learned over the years that it is very unfair to think that just because someone has not done certain work or does not know some concept he is not a good developer. It is now about how much you know. It is about how well you know what you know. Otherwise you end up in the situation where the guy can talk about any subject for hours, but cannot get any specific thing done, or gets confused and ties himself up in a straitjacket when trying to solve a real world problem.

Here is a post about how not knowing a design pattern does not mean you do not know how to code. If the guy is good and designs software himself and does not know design patterns, it probably means he has more successful software projects under his belt.

I was once interviewed at a company like this, where the manager initially rejected me. For some reason, after seeing my experience and talking to me, his manager hired me overriding him.

Then I joined this company where everyone had an MS in computer science, and I was the BS in Applied Electronics and Instrumentation.

I did better work than anyone there... period... without doubt...

Why?

Because most of us are application developers who work at a much higher level in the code. In my case in C#. We work within the bounds of the .NET framework. What matters for excellence here is your experience working with the framework, and how much you know about it, and how much your mental model is for working at this level.

They hired a russian guy who was a colonel in their army who was the best guy who ever passed this "classic" interview. He knew every single thing and more about systems.

He could not understand my code. In fact, he was not even able to maintain it. All I did in my code was extreme code reuse, with generics and not very complex OOPS. I don't claim to be a OOPS guru either.

I need to tell this story to convince the reader categorically that this entire theory is false and very foolish. I am the living proof of that.

To be a great developer, the number one thing you need to do is to write a lot of code. Gain experience. Try out the right and the wrong approaches. When you do this for a long time, you gain an understanding of how the framework works at its level. Then you can do great work with the mental model you have - there are lots of stuff which people who know more concepts, or can do b-tree cannot do.

In 90% of the cases, you will never need to know the b-tree to write excellent code, which surpasses all expectations. In the rest 10% of the cases, by then you will be smart enough to know you need the b-tree and you can handle the condition.

In 99% of the code, programmers do not have any macro level understanding of the code. Hence, they often never reach the level of intricacy where a b-tree knowledge would even help them. They are still struggling to figure out how to show alternate rows in different color on a web page - forget about going any deeper.

I have seen code of the people who know b-trees and other stuff, here is what I found:

1. They don't know how to do code reuse properly.
2. They don't know how to user interfaces properly.
3. They are confused as to when to use loosely coupled code and when to have highly cohesive code.
4. They use the unity to make all their code run via configuration.
5. They tie themselves into a knot, and struggle to free themselves and basically the knot keeps getting tighter and tighter.

Another example of a total asinine belief is that if you cannot write a sorting algorithm from scratch, that means that you cannot be a good developer and "solve the problems we have". Basically, the simple sorting mechanism you wrote cannot scale to 100K rows. This is total bullshit. The reason is because in today's world, you do not need to write a sorting algorithm from scratch anymore. We have inbuilt methods in the frameworks which bypass the need to know an algorithm like that.

If the framework does not support it, in the weird universe you live in, this is still a really asinine idea because you are doing something wrong if you have to write C# code to sort 100K or a million rows. This is why there are databases and they have the sorting support in SQL. For those who say what about in-memory systems, my answer is that your in-memory system will not scale on the cloud, if you have one object with a million rows in it - how do I know? - I've explored the limits of caching in the cloud and came to the conclusion that in-memory is just good for key-value pairs. Sorting like you want to do, does not give good performance at that kind of scale - the paradigm is to avoid doing it. Even if you want to do it, you got to use a totally different algorithm to divide and conquer the work fog big data.

Also, let us look at the biggest reasons why projects run by such people always fail and never get delivered on time - this is called premature optimization. You spend so much time prematurely optimizing your solution that you end up finding that you need to rewrite most of the code because what you spent so much time coding is not useful anymore. The best way to write successful software is to keep it simple and unoptimized  to begin with and optimize over time as you find out that what you wrote meets the need, but does not scale, or is not elegant enough (or whatever). In the real world, you will possibly find a real use for these kind of optimizations way after the product has gone out, and become really mature and there is a real need to say use some of these algorithms. By then, you will know the system so well, that you can apply the optimization in the best possible manner.

This is the reality of the world we live in. Physics is a good example of this. On one hand you have Newtons laws which work at our physical level, and then you have the modern theory of relativity which works at the sub-atomic level.

In computer science, we are making the mistake of assuming that just because someone is a PHD in sub-atomic physics, he is automatically a PHD in Newtons laws when in fact, he may not even pass the middle school level of Newtons laws. Newton's laws are simpler than sub-atomic physics, just like macro level programming knowledge is simpler than knowing systems level knowledge - however, it belies the fact that just because it is simple does not mean that if you have no idea about these laws, you can still solve problems at the level of the physical world.

There is a whole treasure trove of knowledge at the macro software level which is not in any book, or in any design pattern. People who know this from experience and use these laws, are the great programmers today - others look at them and wonder how the person is doing such good work, when he does not know anything about data structures.

Events and Configuration State

I found another interesting application for configuration and state recently.

We were discussing events. The user presses a button, a request goes to a service, and it starts a process. They were saying the start event is when the user presses the button, and the end event is when the service finishes processing.

Wrong!

The user pressing the button is a command - a configuration. Events are related to the service starting something or finishing something - so it is state.

The correct design is that when the service starts processing the command to start doing the work, then its state is "started" - so that is when the event should be fired. Same for when the work gets done.