Introducing the Big P Notation P(N)

An excellent article on Big O Notation:

https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/

I have realized that with lowered cost of computing and with computers in almost all desks in the world, the focus may have to shift to the Big P notation P(N) as I would like to refer to it.

Back in the old days when computers, memory and disk were super expensive, it was necessary to determine the maximum complexity of an algorithm and measure it so that we can optimize the algorithm. Definitely, there are cases where we still need to do this like when scaling upto a million users, etc.

However, today focus might need to shift to the P(N) notation which has become more and more important over the years.

If O(N) refers to time complexity, based loosely on how much CPU time an algorithm will consume, P(N) refers to people time complexity - which is based on how much human time is spent on creating, maintaining and optimizing an algorithm.

If 3 people work on the algorithm it becomes P(3). If you work on an algorithm 3 times as part of the development process, again you get P(5) [Dev - 1, Code Review - 2 people, Unit Tests - 2 people]

People are the most costly resource for computing today. I refer to the hundreds and thousands of well paid programmers. Any algorithm written today needs to consider the cost benefits as well.

If I can write an algorithm which is easy to understand, digest & maintain even by the most cheapest programmer I can hire, that might give you more bang for the buck than to worry about how much CPU time the algorithm will consume.

Sometimes, there are cases where a very computationally intensive O(N) algorithm is actually less preferred because of the high P(N) value.

The best case I can think of is an engine I wrote around 2004 which used reflection (very time consuming on the CPU) to validate DAL code before it hit the database. It took time to run, but it would raise many problems before the code would even hit the Oracle database thereby reducing development time significantly.

I would say it has a very low P(N) value. P(5) - I wrote it once, got it code reviewed, got it unit tested, and it "just worked" although it was relatively slow.

Because of some misguided notion of performance, we removed this algorithm and lost all the validation logic, and each person had to write the validations manually (sometimes missing many validations). Hence the P(N) value became P(N raised to 5). 5 counts of human effort for every person in my team.

So we reduced the O(N) value significantly, but massively increased P(N) value leading to a very unstable, though fast application.

Summary: Programs were written for people and not the other way around. Maybe, now that computing costs have gone down significantly, and people costs rise with inflation, we need to consider P(N) more than O(N) for many cases.

Lesson 8: No solution can be generalized to be always true in all situations. Depends on the system design.

We had a recent situation where we were going through a case where when the code reads from a value stored within a ConcurrentDictionary, this value could be stale.

We had an interesting discussion why this is ok. And more interesting about why we have to make an update to this value to be thread safe.

(1) Why Stale Value is OK in some cases

This is ok because in some cases, the design of the system is like a traffic light - some vehicles may go even when the light is orange, some may flout the "rules" and still the system can function.

If writes are consistent, reads can be stale in some systems. The critical reason why reads could be stale is if the stale value has no permanent effect on the system, and the cost of keeping it not stale is too expensive.

If the stale value caused permanent changes then it is not ok (persistence, database transactions, etc).

If the stale value is a signal which is read often and the effects of it being stale are very temporary in nature, for performance reasons, it is ok to read the stale value even while updating it in the ConcurrentDictionary.

I see this as a case where computers and programmers do not do well in grey area scenarios. Not all answers can be 0 or 1. Sometimes, grey areas exist and provide optimal solutions to real world problems.

(2) Why the update itself should be thread safe

If this ConcurrentDictionary stores a number, and we want to update the number, we want to keep this operation thread safe because if it is not, then depending on the number of threads accessing the write at the same time, the value of the number could careen across wrong values, thereby leading to a permanent bad state.

This is similar to the traffic signal entering a permanent state of malfunction where only some lanes turn green all the time for no particular reason except software error irrespective of the traffic load at each side of the signal.

We want the number to be correct on a permanent basis, but allow the reads to be stale sometimes for performance reasons.

A great way to do this is to use the AddOrUpdate method which I have not verified myself. But it looks like the below may work fine:

concurrentDictionary.AddOrUpdate(key, value, (key, oldValue) => oldValue + 1);

So locking the update in this case is not really being inconsistent when following this philosophy.

Software Engineering is dual natured: both a particle and a wave

The title of this post maybe pretty obtuse and hard to make sense of. I'll try to explain.This post is to disparage everyone who evaluates software engineers only by checking to see whether they know how to sort an array fast or ask algorithm questions.

The fundamental reason why this is a bad idea is because similar to most natural phenomena and physics in particular, we are making the very wrong assumption that because someone knows everything about the building blocks of something, that person can be a great engineer building applications.

This is completely wrong. And I'll explain why...

In the beginning there were Newton's laws which could explain most natural phenomena at the physical level (laws of motion, etc). This is what I call as the "wave".

Later, it was found that these laws break down at sub atomic levels (basic building blocks). This is what I call the "particle".

Essentially, physicists realized that two completely different sets of rules defined what happens at the sub atomic level and what happens at the larger levels of physical phenomena.

Boiling down the essence of this means that just because you have a deep understanding of how something works internally, you cannot use those equations in your daily life. For that, you can rely on much simpler set of laws like Newton's laws.

Of course, you can do everything at the sub-atomic level all the time, which would mean that you spend all your time on figuring out how those very complex equations map at the physical level.

How easy would it be then to do anything practical fast? - Try to model the path of a ball thrown at 45 degrees using quantum mechanical equations...

This is also why these guys who know the internals very well cannot produce any tangible results to solving big problems. They are applying particle physics in situations where they are supposed to use Newton's laws.

I may have mentioned these very thoughts in some previous posts. Why I am writing this post is because I had a concrete example of this in my recent work.

The particle part of software is the underlying data structures and algorithms internal to a high level framework like .NET and C#.

The wave part of software is when we build complex software to do some workflow using these building blocks. This is more similar to a process in Applied Electronics & Instrumentation more than linked lists and sorting algorithms.

A very distinguished engineer tried to solve the out of memory issue using the particle methodology. Use a more efficient data structure to reduce the memory usage. He tried for months and failed and the software could never handle more than a certain amount of data.

When I saw the problem, I approached it from the wave perspective. Irrespective of the data structure used, or the sorting algorithms used - the problem was because of the way the entire process worked. It was similar to a factory where the pipes feed chemicals into the plant to manufacture something else. The input flow needs to be regulated to the consumption. If this is not done, the process will overload - run out of memory.

I fixed the problem which had remained open for 8 years. I used no sorting algorithms or specialized data structures. Now the problem is completely resolved.

It is true that understanding the building blocks of software can help in rare situations where the higher level code does not work the way you think it may - because of how it is coded internally. However, this is an extremely rare situation. Most software projects fail a lot before it reaches that stage. Just think about how buggy Android is - and it is built by people who are from the top most universities and have the most degrees. The reason for this is firstly their arrogance that they will only hire people from some top colleges to build software which will be used by everyone. This is also because they think too much at the particle level, while they should actually be thinking at the wave level.

The best examples of this kind of follies are: Google Wave, Android and even the most horrible to use Gmail.

More Data

It is always a struggle to provide a more clearer explanation of what I am trying to say here. Let me give you a few more examples which clearly show how a system is not the sum of its parts.

• A factory consists of many parts mechanical, non mechanical and electrical all of which work together for the most part to produce the output. The behavior of the factory cannot be calculated as the sum of its parts because when parts come together they interact in new ways which may not always have been foreseen before. Even if it did, when we add more parts and go on, eventually we reach a point where we have to consider the entire system in a holistic manner rather than try to optimize atomically.
  
• For Software Engineers who love to go back to the building blocks because it allows them to understand "better" how it works - why not go back to the hardware level? No software runs in isolation. It always run on top of some hardware which sometimes changes the behavior of the software. So, if we want to "really", "comprehensively" understand how best to optimize the software or how it works, perhaps, we should first stop at machine language and optimize at that level with compilers, and then go deeper into how the hardware handles the instructions at the chip level and fix that as well.
  
  If this seems nonsensical to you, then the whole talk of data structures should also. Because all structures are static and there is only so much which can be done at that level. Software when running in a system consists of rich interactions between live instances of objects with complex behavior. This is best optimized as a whole process, rather than as the sum of its parts.

Using a Design Pattern does not mean do not design the solution

This is an idea which I have explained a lot to others previously. In that example, I designed a workflow system and compared to an existing design pattern. I found that my design did not have the bug the design pattern had - which was mentioned as a bug in the design pattern. Still, we find scenarios where people think that if you do not know a design pattern that is some indicator that you are not a good developer.

It is not how much information you know - it is about how you use the information you already have which makes a great or so so developer. As Einstein said: "The true sign of intelligence is not knowledge but imagination." - I saw this in my kids kindergarten wall.

Let us go really deep into the implementation of a Producer Consumer design pattern as they seem to call it which was full of holes. This time I can write at length about this because I just finished fixing a problem which nobody else was able to fix so far.

The problem is that we have a set of threads which are spawned from a set of timers. These threads share the load of performing different work items. One such work item comes from a queue which contains two types of items:

1) Which generates an item which can be consumed directly.
2) Which generates an item which generates more items - of both types (producing items and/ or items which can be consumed).

It looks straightforward, but this will not work properly. This is because these items which get into the queue can come in any sequence - which depends on the dataset from which they are generated from. So, we have a very complex system whose behavior is very dynamic and dependent on the million different combinations of the input dataset.

This system would eventually end up either sleeping indefinitely or running out of memory because of the following reasons:

1) There is no control to regulate the fact that one producer can add so many items into the queue that we run out of memory.

2) There is no control to regulate the fact that because the queue can contain items in any sequence, we don't end up in a scenario where there are only producers and no consumers so we again run out of memory.

3) If we do try to sleep the producer temporarily to control the memory usage, we got to do it in such a way that, there are adequate consumers to consume what is in the queue, and some consumers are available always to get the enqueued items below a limit.

The more I encounter real-world applications of software design patterns, the more I see badly designed software which does not work properly because the developer ceded completely to what is written in some book somewhere. 

I see bugs of the worst kind in both design patterns I've seen in the field. This makes me really wary of the next time I see a design pattern in code and someone tells me.. oh.. we are just using the XYZ design pattern here, it takes care of everything you see...

Design is not code behavior and may not even be how the code works in reality

People sometimes use the word software design in a promiscuous manner. Pretty much anything and everything fits into "design". That is not really true. In reality, design is a lot of things and none of them may directly relate to the code - it influences the code, rather than having a direct relationship. I say this because the design may say X and the code may wrongly do Y. Then it is the flaw in the code and not in the design.

What really comprises of software design?

Design is really a set of ideas which are noted down when thinking about the implementation.

- The set of assumptions which you consider when thinking about the implementation. This is the world view of the code. If a new assumption comes along, that can make or break the existing design.

- The goals of the project - "ultimately when all is said and done, this is what the implementation will accomplish".

- A good design may mention stuff which goes tangential to what a developer may think. A good example is in a recent project "by design", the assumption was that the database rows cannot be fully trusted because they could be out of date w.r.t the file system - so they database rows can only be used as an indicator that something may need to be done. The final word needs to come from the file system.

- A design artifact may not always be re-discoverable by reverse engineering the code. In most cases, it may take months of looking at the code behavior to assume that "one of the original intents of the coder may have been to do X and Y". Even then we cannot be fully sure about it. Hence, it is always important to have a design document, because without it, we cannot add or modify to an existing design without risk. The code could be wrong w.r.t the design document, but still it is important to know whether your thoughts are in alignment to the original designer of the code if the major project goals and assumptions remain the same over time.

- A design always considers the high level interactions between the various entities in the system. This need not always be software modules which are written by the developer implementing the project. It could be third party stuff like OS, IIS, file system, database, etc.

- The two minimally required UML artifacts for design are Use Case Diagrams and Activity Diagrams. UML makes it easier to go about designing a complex project where we cannot think of everything "at once". It helps us split the work in a structured fashion into small "bite sized" pieces which are easily understandable and digestable.

Using the concepts of nature in software

This post came about when my wife told me that she was playing a certain game in iOS and nobody she knew could go past a certain level in the game. She tried herself and her nephew also tried and failed. My son who is 4 years old and knew nothing about software or games was able easily go past that level. The reason he was able to do that was because he approach was random, and when he tried a burst of random approaches with no particular logic, he was able to break the level.

There is a software analogy to this as well. When any software gets mature, we tend to frown upon major changes to it, or someone "breaking the rules" and writing code on it. Every architect has been on this side at some point in the career. I have done this myself in cases where I knew for a fact that this would break something. I have also vehemently objected to a requirement which would cause me to break a lot of code while trying to implement it.

In software as in every other field, we tend to frown upon failure of any sort. If we worked on a project and it failed after 6 months, or was unable to attain what we wanted it to attain, this is considered a failure as well.

Is it really?

Today I feel the opposite. Everybody is different. When we tighten the rules and put a straitjacket around a framework or software, we are basically preventing any kind of out of the box activity from happening. We know this is difficult to do, or it breaks something - but how do you know that for sure? - maybe you failed to do something in a certain way - someone else maybe able to do what you were unable to do.

How would you know what was possible, if we prevent someone from trying out a different approach?

Sometimes, even when a different approach "fails", a lot of different things come out of the effort improving the code base in ways we never thought possible - because nobody every looked at the code from that particular perspective before.

Perspective is everything when it comes to large code bases. Because beyond a certain size, it becomes too much for one person to internalize. At that point of time, perspective is the only thing through which you can gain understanding of the system. When it changes, the understanding changes, and even evolves. You find over time that what you thought impossible yesterday is very much possible today.

And none of this is possible, if you never tried out a new approach, or went down the risky path.

A concrete example of this is when I had to add a new mode to an existing software which made it more reliable, but also quite slower than before. I tried out a very different approach to make this as fast as possible - basically indexing files beforehand, instead of doing it when the process was running. This was not enough - so I had to go around and make other parts of the code run 10X faster than before, and utilize multiple cores. The end result was that even if in the new mode, it was as fast as the earlier mode or maybe a bit slower, if you ran in the old mode, it was 10X faster.

If I had never tried the new mode saying it was too difficult to do, I would never have improved the old mode, and found hundreds of small but important things which made the software better in the long run.

How is this related to Nature?

We write this simple, cave-man software and prevent others from trying out new things, or worry about risk. But look at nature which has created plants, birds, mammals and even the human brain. We have reached this stage in evolution because Nature tried every random permutation and combination based on the given parameters and optimized along the path which is just right to reach the current stage today.

It tried all approaches, all risks. Entire species dies, new ones arose, but it still went forward and continues to move forward.

Random approaches and failures are necessary as they provide us with as much information as the successes do. Only through learning both success and failure can we really move forward. Infinite patience is required to find the right solutions for difficult problems. Solutions are not just found through brute force human intelligence or infinite CPU power.

I think a major failure of current software is that we build something and we tighten the rules as we go along, and then new ideas have to create something new from scratch. We never continue to build upon the same thing, evolving it over time. So, we are forever going in a circle, reaching nowhere because of this. All logic is useful, whether it was written in FORTRAN or PASCAL or C++.

Discussing these ideas with others, they mentioned that this might be the reason that in some companies on the west coast, they tend to keep only younger employees and not experienced ones. Because, even though experienced employees are technically competent, they are highly resistant to change and unmalleable to consider new approaches and risky techniques. So, companies keep younger employees who don't know enough not to take risky approaches.

Whatever be the case, I can't generalize that older employees are always resistant to change. They have strong opinions for sure, but I have worked with very constructive senior developers who embrace the fact that we are building something and every time, this has resulted in massive success. On the other hand, I have also worked with very capable senior developers who are very resistant to change and very nonconstructive in terms of showing only negativity when faced with having to build anything new. There have been teams which are highly "aligned" to resist change, even building frameworks which actively straitjacket any new approaches to code. A particular instance was a service architecture where the code would error out if you returned an object with two properties instead of one, or you had to duplicate a class in two assemblies deliberately just to be able to write the service layer code.

The Code Paradox

It is kind of interesting for me to note that when the code is really bad and very buggy, making small changes to it, does not affect its overall quality a lot. In many cases, defects combine to get something working together in a certain way.

What is more interesting is that when the code is quite good, well written, modular, reusable making small changes to it can actually break everything. So, it affects quality a lot more, and small things cause more stuff to break especially because of reuse.

This could be entropy. The natural state is disorder and it stabilizes there.

When we try to bring order, that is an unnatural state. So, I have seen many, many cases where the code is really good, and someone unfamiliar with it makes a slight change and it keeps breaking all over the place.

The real challenge here is - how do we write software which has entropy by design but still works well? - because I believe biological systems which work well with the most diversity and population are like that.

Is it not weird that even in software, which is totally man made, this basic law of entropy remains true? - just shows that in a system where entropy is the law, even for seemingly autonomous entities (like us) who create new things - the things we create however abstract they are also stabilize at entropy.

Embrace the chaos?
Embrace the diversity of thought which makes a single code-base have both good and bad code?

Definitely something to think about.

Lesson 7: Micro level understanding does not translate into macro level excellence

This is an important topic because the number one problem which tech companies face is that they are unable to figure out who to select in an interview. Most of the time, the selected person is unable to fulfill even a percentage of what they thought she/ he could accomplish.

There is a good reason for this anomaly. It is because in the computer industry it is widely assumed that just because someone has understanding of microscopic details regarding computers, some of which maybe arcane, that translates into excellence at the macro level.

From my experience, this has been proven wrong again and again, and I have learnt even from my experience hiring people.

Basically, the theory goes on that say if you understand computer science concepts like data structures, or stuff which is really important at a system level (like operating system concepts, and other details about compilers, linked trees, etc) then wow - you are a find! - if we hire you - the next day you will be productive.

On the other hand, if the guy does not even know data structures, and cannot solve the complex b-tree problem posed in the interview, then he must be a total loser.

This is total nonsense. As Einstein said: "The true sign of intelligence is not knowledge but imagination." - I saw this quote in my kid's kindergarten. Maybe we need to send such people back to kindergarten. What I've learned over the years that it is very unfair to think that just because someone has not done certain work or does not know some concept he is not a good developer. It is now about how much you know. It is about how well you know what you know. Otherwise you end up in the situation where the guy can talk about any subject for hours, but cannot get any specific thing done, or gets confused and ties himself up in a straitjacket when trying to solve a real world problem.

Here is a post about how not knowing a design pattern does not mean you do not know how to code. If the guy is good and designs software himself and does not know design patterns, it probably means he has more successful software projects under his belt.

I was once interviewed at a company like this, where the manager initially rejected me. For some reason, after seeing my experience and talking to me, his manager hired me overriding him.

Then I joined this company where everyone had an MS in computer science, and I was the BS in Applied Electronics and Instrumentation.

I did better work than anyone there... period... without doubt...

Why?

Because most of us are application developers who work at a much higher level in the code. In my case in C#. We work within the bounds of the .NET framework. What matters for excellence here is your experience working with the framework, and how much you know about it, and how much your mental model is for working at this level.

They hired a russian guy who was a colonel in their army who was the best guy who ever passed this "classic" interview. He knew every single thing and more about systems.

He could not understand my code. In fact, he was not even able to maintain it. All I did in my code was extreme code reuse, with generics and not very complex OOPS. I don't claim to be a OOPS guru either.

I need to tell this story to convince the reader categorically that this entire theory is false and very foolish. I am the living proof of that.

To be a great developer, the number one thing you need to do is to write a lot of code. Gain experience. Try out the right and the wrong approaches. When you do this for a long time, you gain an understanding of how the framework works at its level. Then you can do great work with the mental model you have - there are lots of stuff which people who know more concepts, or can do b-tree cannot do.

In 90% of the cases, you will never need to know the b-tree to write excellent code, which surpasses all expectations. In the rest 10% of the cases, by then you will be smart enough to know you need the b-tree and you can handle the condition.

In 99% of the code, programmers do not have any macro level understanding of the code. Hence, they often never reach the level of intricacy where a b-tree knowledge would even help them. They are still struggling to figure out how to show alternate rows in different color on a web page - forget about going any deeper.

I have seen code of the people who know b-trees and other stuff, here is what I found:

1. They don't know how to do code reuse properly.
2. They don't know how to user interfaces properly.
3. They are confused as to when to use loosely coupled code and when to have highly cohesive code.
4. They use the unity to make all their code run via configuration.
5. They tie themselves into a knot, and struggle to free themselves and basically the knot keeps getting tighter and tighter.

Another example of a total asinine belief is that if you cannot write a sorting algorithm from scratch, that means that you cannot be a good developer and "solve the problems we have". Basically, the simple sorting mechanism you wrote cannot scale to 100K rows. This is total bullshit. The reason is because in today's world, you do not need to write a sorting algorithm from scratch anymore. We have inbuilt methods in the frameworks which bypass the need to know an algorithm like that.

If the framework does not support it, in the weird universe you live in, this is still a really asinine idea because you are doing something wrong if you have to write C# code to sort 100K or a million rows. This is why there are databases and they have the sorting support in SQL. For those who say what about in-memory systems, my answer is that your in-memory system will not scale on the cloud, if you have one object with a million rows in it - how do I know? - I've explored the limits of caching in the cloud and came to the conclusion that in-memory is just good for key-value pairs. Sorting like you want to do, does not give good performance at that kind of scale - the paradigm is to avoid doing it. Even if you want to do it, you got to use a totally different algorithm to divide and conquer the work fog big data.

Also, let us look at the biggest reasons why projects run by such people always fail and never get delivered on time - this is called premature optimization. You spend so much time prematurely optimizing your solution that you end up finding that you need to rewrite most of the code because what you spent so much time coding is not useful anymore. The best way to write successful software is to keep it simple and unoptimized  to begin with and optimize over time as you find out that what you wrote meets the need, but does not scale, or is not elegant enough (or whatever). In the real world, you will possibly find a real use for these kind of optimizations way after the product has gone out, and become really mature and there is a real need to say use some of these algorithms. By then, you will know the system so well, that you can apply the optimization in the best possible manner.

This is the reality of the world we live in. Physics is a good example of this. On one hand you have Newtons laws which work at our physical level, and then you have the modern theory of relativity which works at the sub-atomic level.

In computer science, we are making the mistake of assuming that just because someone is a PHD in sub-atomic physics, he is automatically a PHD in Newtons laws when in fact, he may not even pass the middle school level of Newtons laws. Newton's laws are simpler than sub-atomic physics, just like macro level programming knowledge is simpler than knowing systems level knowledge - however, it belies the fact that just because it is simple does not mean that if you have no idea about these laws, you can still solve problems at the level of the physical world.

There is a whole treasure trove of knowledge at the macro software level which is not in any book, or in any design pattern. People who know this from experience and use these laws, are the great programmers today - others look at them and wonder how the person is doing such good work, when he does not know anything about data structures.

Events and Configuration State

I found another interesting application for configuration and state recently.

We were discussing events. The user presses a button, a request goes to a service, and it starts a process. They were saying the start event is when the user presses the button, and the end event is when the service finishes processing.

Wrong!

The user pressing the button is a command - a configuration. Events are related to the service starting something or finishing something - so it is state.

The correct design is that when the service starts processing the command to start doing the work, then its state is "started" - so that is when the event should be fired. Same for when the work gets done.

Dealing with Ambiguity: Zero does not mean anything

In software we often come across complex and difficult scenarios which might muddle our mind. One of the key concepts to understand is how to deal with ambiguity.

Basically, this means that if the data we have is not adequate to reach a satisfactory conclusion, then we need to generate the additional information which is necessary for the computer to understand what needs to be done. This automatically means that assuming that something needs to be done in a certain way is a wrong thing to do. In such scenarios, always have extra properties which can go down to the most basic level and which can be parsed to clearly understand the requirements.

A simple example of this is how we deal with nullable types, enums, etc. Enums make it easier to assign a meaning to an esoteric number which is unreadable and not maintainable. Nullable types for values like dates and numbers allow us to know for sure that, there was nothing present to begin with. In a similar manner, we often abuse zero to mean something in applications. Zero is zero, it means nothing, a number can be initialized to zero, to begin with, so, even if we use nullable types, it would still be good practise to say that 0 = default = nothing.

What can be more interesting than the fact that, all these concepts even boil down to simple naming conventions. I have seen reams of code which are completely meaningless start making sense, after I renamed "xyz" to something more meaningful like "index" or "counter" or something else. In this case, the name of the variable was making the usage ambiguous. I gave it a concrete name and everything simply fell together in place.

Another interesting aspect to this which confuses many engineers is that most of the time, when we feel ambiguous about something, it is because we do not have enough inputs. The existing inputs are not adequate for the program to assume something and then exhibit a new behavior, or execute some logic. Most often in such cases, we need to pass these new inputs or parameters right from the user end, through the various layers to the code which needs to decide what to do. And it is difficult for a novice engineer or even someone with experience to realize that ambiguity is tackled by removing it, and one of the ways of doing that is to add more inputs.

I was having a conversation with a friend of mine regarding a very complex problem. I understood only part of the detail, but during the conversation, it became apparent that there was an underlying ambiguity regarding what decision had to be made, and my friend was struggling to come out with an assertion that so and so would mean X and so and so would mean Y. I interjected and said that, the situation is ambiguous and that assertion cannot be made. We added one more column which would say concretely what would result in X and what would result in Y.

Configuration & State in Web Service API's

I was recently commended on the strength of my API design, and there was a remark that elsewhere we did not do it the way I implemented my method. So I thought that I would like to add a concrete "modern" example of configuration and state considerations in web service API's.

Basically, we have an API which lets the customer send us "activate alarm" requests, when the alarm is activated, the device sends back the alarm status. When I designed the API, there was the thought that, why not simply use one database column for both considerations - "what the customer asked us to do" & "what the device told us it's status was".

The reasoning was that, if we are in fact immediately sending the device an alarm activation message, why, it would go into alarm state. Why have extra fields to capture this information?

I put my foot down. Nope, Nada - we won't do it that way. the activate alarm request is user configuration, the state of the alarm activation in the device is a separate entity, both are different and should not share the same storage space.

8 months after the API was deployed, we had a customer issue which we were able to easily debug because we were storing the values separately. That turned out to be a device issue.

Elsewhere, we shared data space, and there, now everyone is in a tizzy trying to fix that entire implementation.

So.. long story short - keep configuration and state different. Always.

Lesson 1: The Difference between Configuration & State

[Please read Introduction first]

The difference between configuration and state is the most basic software concept which nobody has ever heard of. I had the good fortune of working in a company where I was exposed to great engineers. Passionate, experienced people who had decades of knowledge in great software development yet were humble enough to listen to you. This is the most precious jewel which I have learnt in my work life. It comes from a renowned telecom CEO who has contributed to several basic RFC's in the industry.

"Configuration is what the user has entered. It must only be changed by the end user. State is what the machine has reported; it must not be modified in any manner."

This is the basic rule, which I will now explain by example. It all started as a heated discussion which revolved around a reference variable entered by a user into the system. The machine uses this reference value to do something, and there are conditions where the value used by the machine as reference value must increase or decrease. The machine also reports back a value. The VP of Engineering wanted to use just one variable for the first two conditions and another one for the returned value. User enters it, machine changes it as needed and adjusts itself. This violates the rule of configuration and state, and to make a long story short, we ended up understanding why, and implementing it like this, while the VP got demoted.
What is so great about this?. Why not use one variable for this?. The reason is that, there are 3 concepts here:

1. The configuration value which the user entered. If he sees the value changed next day, and he did not do it, he would be confused.
2. The "real" reference value used by the machine. Initially it is the configuration value, but the moment the machine needs to modify it, we need to store it as a separate value, so that, we can track that something has changed and notice the difference between the initially configured value and the current value.
3. The value returned by the machine. This is "state", which is the value picked up by a sensor and returned back to the user. This is the value returned by a sensor. This cannot be modified by us for any reason, except maybe localization or metric changes, because that would be falsifying the truth.

"Good software systems always separate the concepts of configuration and state. This is how really stable, well designed systems become that way. It is also something many telecom engineers know about because their firmware has to be better than desktop software."

I didn't understand this concept initially. But unlike many of the arrogant "know it all" software engineers/ architects today, I looked at my experienced colleagues nodding their heads, and I realized there is something here which I have to learn, because it seems to be important. So I went back to the person and spent time understanding what exactly he was trying to say. When this person finished his contract and left the company, I went to him and said something which I have never said to anybody else and hope that I can say to someone else tomorrow:

"Working with you has been the greatest privilege in my life, because you have made me realize that one should be humble, and there is always something more to learn. We can never underestimate someone, or think we know everything, because when we meet people with real knowledge, we realize we were fools, and we were so ignorant before and there is so much out there to learn, if only we had an open mind."

Once you understand the concept fully, it is amazing to see the number of places where this crops up. Recently I was involved in a design which needed us to bring some data into a database from multiple providers. Our initial database design I felt was "wrong" in some way. And I could not put my finger to the exact reason for this. What we were doing was that, we were merging configuration and state into a single table and shared columns.

If you have a table which is supposed to define the various fields sent to us from remote sensors, this table must contain the definition as we define it. We were having a column which we referred to as "Name", and which would be used as the default display name shown to the user. This was wrong, because the default display name shown to a user is a different concept than the "real" name of the field defined in the row. If we merge the two concepts, we get a skewed database where we cannot understand what the real field name is, because we modified the value to make it user understandable. This was configuration overriding state. What would have happened was that, this would impact the data driven model which we were trying to accomplish, because one of the primary tables driving the data driven approach would corrupt the information coming in.

This popped up in another place, where we had the requirement to store something which the user entered into the system, while we had another field which was populated by data which would come into our system from another system. The DBA wanted to merge the two fields into one column, but I stood my ground saying that these are two different things and cannot be merged into one column. One is what the user entered (configuration), the other is what the other system gave us (state). One must be stored and used for a purpose, the other must be logged. Even if they look like very similar concepts (or the same thing even), we cannot merge them into one variable or store them in the same place, because that is a basic flaw in the design.

There is one more weird condition where in a very complex design decision, what route to take is clarified when we use the concept of separation between configuration & state. Say you have a device in the field, and to configure it, you go through the UI and set the value to what it is. This value is configuration. Now, when the device sends data to the server, you cannot use the stored configuration to determine what device sent the message, because what the device sends is state. Not only should it be stored separately, what it is, should come as part of the packet, and not determined by the configuration done by a person on the UI. The problem with identifying state based on configuration is that, another user may change the configuration, but the device did not change physically on the field, or the configuration got lost from the table, and now we don't know what the device is. It could even be that, by an SQL statement, the configuration changed to a wrong value and now we are doing wrong things based on it. I guess, what this really means is that, state information should be self-sufficient for storage on the server and also "stateless" in some respects. We cannot decide what to do to state based on a value which is user configured. This must be determined from the state packet itself.

You may think you know all about software because you can code well, or know the technology well. But without understanding and using this core concept in your designs, you become just another "good" coder. This is lesson 1, because it is the perfect example of a concept which has nothing to do with technology, resembles philosophy and yet should be one of the binding principles of good software development.

Introducing Software Concept Design & Modelling

I come up with my best ideas in the night. And today 06/13/2010 I am happy to introduce software concept design & modelling or SCDM as a new area in software engineering. This sits somewhere between requirements gathering and software design. Most people don't know that something like this is needed for software work. This is also the key to understanding what is that "unquantifiable" thing which makes some engineers better than others even though they have lesser degrees/ knowledge/ experience. Most people who have it, don't even know that this is their key strength.

I never knew I had it. Then I worked at a company where I saw more senior people who had it too. I knew it was a great help in the software design process, but it was just unquantifiable.

"Software Concept Design & Modelling is all about understanding the requirements and designing the concepts around it, so that what we build as software makes sense as a whole, and provides value to the end user."

Sounds like meaningless jargon?. Not really. And to explain this in more detail, I will br providing concrete real-world examples of where this was done, and it provided immense value to the software development process.

"Understanding Software Concept Design is equivalent to understand the basic rules which make stable software. Such software has less entropy than others. Honing such skills is essential to make software development a repeatable success in the Enterprise."

But right now, I just want to pique your interest. What do these words really mean? - It means that, often, we fudge software design badly because we lose sight of what is really supposed to be built by using big words, or confusing the hell out of ourselves. In such situations, development either comes to a stand still, or worse, something gets built which turns out to be totally useless later.

The field of software concept design is not about technology specifics, or even domain specifics. It is about how we organize the concepts around a solution to ease the technical design. Requirements tell us what the user needs, Technical Design tells us how to implement a solution. SCD comes right in between, and it shows us how to effectively translate the requirements into a technical design.

This can totally change the direction of a software project:

1) Turns out that what we were discussing the past few weeks was of no use because as a complete solution, this makes no sense.

2) Somewhere along the line, we forgot what it is that we were building and why we were building it, because we were too focused on how to implement something.

The first time I encountered this dichtomy was when we were designing an application which would run some "tests" and produce data which would then be shown in various graphs and reports in the system as a reference value. There was a major flaw in this system right through the technical design and implementation.

We were calling it a "test" all the time, and never realized that when we have a test, that can either pass or fail. What we were doing is that, even if it failed, we were using the results of the test as the reference value. What we should have been doing was to realize that behind the thousands of variables, domain experts, and technical terminology, if the test failed - that indicated a problem, and we could not use the value as a "reference" anymore. And the reports and graphs would have no meaning without the same.
This is a good example of how a complete software development process with very intelligent people with lots of experience can completely fail, and the reasons for the failure really have nothing to do with the "technical design" other than that, there was NO conceptual design done here.

Much of this must have been difficult to follow and confusing as the issue seems obvious?. But we often sit quietly and listen to nonsense when everyone else is doing so, or someone very senior is going in a specific direction. Simple things get convoluted, everything seems directionless. All of these are as important as technical stuff.

In the next post, we will discuss the differences between configuration and status, why every engineer should know it, and why it is the most important thing which I never knew earlier. It is an excellent example of something which came down to a person as "tribal knowledge", "from experience", but can be considered as a very significant concept in Software Conceptual Design.

The Powered by Qumana