Thinking in Systems
I have been reading the Thinking in Systems book by Donella H. Meadows. I had it recommended to me by a few folks as a way to "think differently". So far I have really enjoyed it, it is an easy read and a very interesting topic. While there are obviously applications of systems thinking in software engineering, I think some of the other things I now recognize as systems, and systems with hierarchies, are more interesting. For instance one system with hierarchies I recognized was from Ender's Game. The small "squads" Ender and his friends initially start out piloting / operating as are their own systems. Eventually Ender and his group start managing larger subsystems that end up being composed of many of the original smaller squads they were a part of. When I originally read Ender's Game the idea of having squads operate on their own accord and trusting the squad leader just "made sense". The higher level subsystem directing many squads also made sense, but there was not really a "click" in the why, it just seemed obvious. Similar to how an organization operates: teams & managers are driving towards smaller goals that the higher-order subsystem are steering teams towards.
Back to the squads in Ender's Game: another idea Thinking in Systems brings up is information sharing between parts of a system and how much information is shared with the higher subsystem. The example the book uses is organs - your liver cells share more data with each other than with the nervous system, and different kinds of data. Your liver organs also do not share as much data with your heart or kidney subsystems. Similarly to how the data squad A shares internally during a mission versus how much, or what kind, of data they would share with squad B or the commanding officer. Teams in an org pretty much behave the same way, information is just shared differently based on who you are talking to.
I finished Thinking in Systems this week, it is definitely a worthwhile read. A couple sections I think are especially worthwhile discuss traps in systems and living in a world of systems.
Traps can seem simple and obvious, and most people like to believe they can spot traps ahead of time. That is not always true, and even when being "caught" in a systems trap, getting out of it can be difficult. One trap I have seen at a couple of jobs is the paging rotation slowly becoming more and more noisy. It starts as a couple pages through a week before eventually becoming disruptive enough that engineers need to ask for evening rotations to be covered so they can just sleep through the night. But how did the team get to this point? Surely no one likes being woken up in the middle of the night? If the rotation gets that bad then problematic services should be addresses and fixes prioritized? That is definitely true but many other things get in the way, and other stocks / flows that were in place before have fallen into an even worse state of repair than the pager.
A team I work on has two levels of monitors - one goes to Slack and another pages engineers directly. When I first joined this team the Slack monitors were semi-noisy but contained some valuable information about the state of the system we were running. While it still may have been too much information for a single person to process, it was possible to look at the recent messages and get a high level understanding of what was going on. Over time more and more monitors were added to the channel, to the point where many team members have found it unusable. In response the team would go through cycles of removing or tuning monitors going to our channel. Eventually the cycle would come to an end and new monitors for new features or services would be added. It sounds a lot like oscillations in the car inventory system. The team has recently made a decision (that I actually pitched) to create a daytime-only pager service that higher-priority monitors can be routed to. These monitors are supposed to be important enough that someone should look at them when they get paged, but not important enough to need to wake someone up in the middle of the night. While writing this I have started to wonder whether we have simply replaced the Slack channel's effectiveness during business hours with something harder to ignore? We are still in the early phases of implementing and adding monitors to this new pager so time will tell.
I also have a bit of a feeling that we are living with a broken window in the form of the Slack channel right now. Part of the reason more and more monitors get added to it is "because it is a good place to put low-prio monitors". It feels as though the value of the channel is decreasing simply because of the state it is already in. A reinforcing feedback loop?
As far as living in a world of systems I think a few of them are particularly good advice that I am going to try to keep in mind.
Pay Attention to What Is Important, Not Just What Is Quantifiable
Being able to understand that just because something cannot be measured does not mean it is not important is critical. It is not really possible to measure the "culture" of an engineering department. There is not a set a quantifiable metrics that one can measure to say if one culture is better than another. But engineers do have a feeling of what a good culture looks like and what a bad one feels like.
Expose Your Mental Models to the Light of Day
Get feedback on your own models, discuss with others, and be ready to accept feedback. Everything everyone knows is only a model. Get your model challenged and reviewed. Scuttle the model if it is not accurate or if your assumptions are misguided. Do not be afraid to throw out work if the work needs to be restarted.
Honor, Respect, and Distribute Information
Information needs to be accurate before it can be acted on. It also needs to arrive in a timely manner. I cannot tell my boss that we are going to lose a huge client if we do not implement a feature for them, oh and the deadline for the feature is tomorrow.
Make Feedback Policies for Feedback Systems
I do not have a good example of this piece of advice other than the example from the book. Feedback needs feedback but in order to provide good feedback you need to understand what is driving the system. The example used in the book discusses President Carter's plan to try and reduce immigration by investing in the Economy of our southern neighbor. That policy never got passed, but it is an interesting idea about trying to look at the cause of a flow and attempting to create a feedback loop to diminish the flow. Compared to adding more security, which is an action that provides little feedback to the flow it is trying to stop.
Locate Responsibility in the System
See if you can shift the responsibility of a system, or part of the subsystem, somewhere else to strengthen a feedback loop. The example from the book is about a professor (the author in this case) not being able to change the thermostat for their office and needing to call a central maintenance office for temperature changes. She notes that she saw greater oscillations in the room temperature.
Maybe a sub-idea of responsibility for engineering departments would be moving some of the responsibility of an internal migration to the customer teams using the internal product. While this sounds great from a platform engineer's perspective, application engineers definitely do not want to be migrating the internal services they use very frequently. Maybe providing some additional benefits would shift some of the responsibility to them? Although also turning off old versions of tools shifts responsibility to them?
Expand Time Horizons
Think about how the system will work much, much, much later in time. For engineering, I think this would be akin to Amazon's: "Respect what came before" motto but into the future. You need to think much further in advance about how the system you are designing will work and have pity on the engineers who will be maintaining it in the future (it could be you).
I think I have written and rambled a lot here. I do not think it is a bad thing though, I am still just getting started writing.