Complexity and Flexibility

And all these additional efforts are evasive in nature and difficult to foresee and plan, since they can not be derived from first principles nor constructed methodically. And while some significant improvements could be achieved by simplifying and reorganising the process of building and testing software, none of the methodical solutions proposed thus far did deliver on their “breakthrough” promise.

Not only are there no magical silver bullets now in view, the very nature of software makes it unlikely that there will be any — and in order to understand why extended systems of software slip away from methodical control time and again, the difficulties encountered when building and maintaining such systems should be considered.

When establishing a structure of components within the software, in addition to the original purpose of the application, an extended set of principles is added, which define and demarcate the modules. Such a move adds to the complexity, since additional relationships are introduced. Not only is there now some processing, which serves the original purpose of the program (and thus adopts a relationship to something beyond the bounds the system), but now, in addition, a relation between the processing and the guiding principles of the component was added, and also a relation of these principles and concerns of the component to the original purpose of the system, plus further relationships to other components, and, last but not least, to the system as a whole.

Yet, paradoxically, by adding and shaping all those further relationships, a pathway is opened to work with the complexities, transforming them into something different. Since each component is related to a concern, which, as such, can be brought before the mind’s eye, it becomes possible now to consider each of the new relationships in isolation, using the module with its underlying principles and concerns as a common anchor. What was once an excess of demands and concerns of the outer world, has been transmuted into a match or mismatch of the inner character of the system with the situation and task it is meant to be used upon. What was once just “a solution” to do this and that, and was then confronted with an overload of requirements to consider that and care for this, has been grown into a compound of autonomous centres, the components, which collaborate to form a whole to achieve a solution and to meet expectations.

Ideally, the overall function is accomplished by the parts of the system complementing each other. However, collaboration can take problematic forms, because the new inner order of the system, as imposed by the introduction of components, is always artificial to some degree. The underlying ideas now governing the form and arrangement of components were based on past experience with handling similar problems, which involves an element of remoulding and reshaping the problem in subtle ways, so that it fits with a well-known solution. Some discrepancies remain, and will resurface in unexpected ways, so that they must be absorbed by the components working together beyond the initial plan.

Even after decomposing functionality into components, some overarching concerns remain, and to the degree that these are handled in a way that surpasses or crosscuts the initial idea and design of the components, this additional collaboration becomes a liability. While, as such, this is a general tendency, causing internal system structures to “set in” and “overgrow” with time, this trend can be exacerbated by handling necessary adaptations in a naive fashion, treating each new requirement as isolated “story”, which in turn is again conceptualised as a processing, broken up into several steps, that are then distributed all over the system and sewed into the implementation of various components, thereby disregarding their actual purpose. Once admitted, such a bearing rapidly corrupts the flexibility and ability of the system to adapt to further change.

Such an attitude towards adaptations of the software to changed requirements might be considered pragmatic — but it turns out to be unhealthy and corrosive, since it attempts to respond to change by merely an amendment at the level of processing, which is rooted in the naive assumption that processing and data are “the real stuff”, while components and architecture should be considered abstract, and thus “removed from reality”. This seemingly plausible conclusion however misses the context: Indeed, at the beginning the idea might have been to solve some concern by processing of data. Yet it was the actual contact with all the complexities of the world out there, which engulfed and entangled the initial idea of processing, up to a degree that sovereignty was lost.

It is thus not sufficient just to define a structured plan and component layout initially. Rather, the ideas, concepts and mental images underlying the components must be placed centre stage. Instead of allowing one piece of code to grab into another part’s innards, each component should be treated as a partner, and asked for service, playing by rules. And instead of passing data items along, manipulating them to cause some effect downstream, a contract should be exposed and passed rather, allowing the partners to collaborate without tangling. Mutual dependencies and tight coupling can be replaced by queries, instructions and a common interaction protocol.

It takes time to build a software system. And while pretty much any idea of “processing” can be coded up in a single strike, thereby pursuing a coherent line of thought, it takes time to realise its actual consequences. The new idea requires testing, it must be attuned with the other ideas and conceptions, as represented in the other parts and components of the system. It should be coordinated with and adapted to the forms and metaphors used within the user interface, and it needs to be explained and documented, in simple words, intelligible for users and other developers as well. And thereby, as this new idea gradually becomes linked into the system and connected with the periphery, the initial unity of mind is lost. After some time, the original idea has been reduced to an image of mind, entrapped within mental abbreviations, covered under layers and layers of further detailed insight and traces of handling the contingencies of the surrounding world.

All successful software gets changed: once it is found to be useful, people try it in new cases at the edge of, or beyond, the original domain. Users who like the basic function of the software will invent new uses for it. Extended functionality might seem like a logical consequence of the program’s capabilities from the user’s point of view, but can sometimes be difficult to accommodate internally, within the framework of components. New kinds of data must be handled, which might not be congruent with the logical reasoning used to build the implementation up to this point, and this places pressure onto structures not prepared to carry the unexpected load. Some parts will be rebuilt and redesigned, some ideas must be stretched and recast into an extended meaning, contracts and interactions will be reshaped. Over time, a gradual shift of focus and a change of proportions occurs, what once seemed adequate may be deemed problematic now, and what was meant as a workaround first, might become established and accepted practice. In the end, the system as a whole has evolved.

Over time, a software solution either turns out to be not that useful, and will dry up. Or it proves to be prolific, and thus either shifts focus, or even expands its scope, as it continues to be used in ways that were not foreseeable up front, by reasoning alone. So this process of evolution, driven by demand and constrained by complexity, leads to a characteristic form of challenge, which was first described in the context of functional programming as the »expression problem«

On the face of it, this might seem like a problem of coding technique, and indeed several cunning technical “solutions” have been proposed subsequently, which all somehow seem to miss the heart of this problem: what does it mean to add new cases to a system, and how can existing code deal with new cases which were inconceivable at the time of inception? What do we actually want to achieve through “static type safety”? How can a software system be flexible and evolve in several independent ways — without having to re-engineer all mixed cases generated by those independent degrees of freedom?

Taken to its full consequence, this problem poses a tough challenge and it appears time and again, in various shapes, whenever a system becomes flexible in several, unrelated ways. A notorious example are the never ending difficulties related to User Interfaces for plug-ins: Assuming that new processing capabilities can be added, independently from the original construction of the system, through some dynamic extension mechanism — how can the GUI of the system be evolved independently to work on new devices, to adopt new styling and rely on new paradigms of interaction, without breaking and obsoleting the Plug-ins previously released?

In summary, most problematic are the non-local interactions and consequences from independent degrees of freedom. Therefore it is desirable to look for locally coherent, decentralised structures, where change remains possible without unleashing far-reaching consequences. In fact there is a surprising convergence between the structure of software systems, division of labour organisation and questions of governance.

Subsidiarity is a principle of social organisation which has its origins in Catholic social teaching,
[ According to Wikipedia: »Subsidiarity«, the concept has its roots in the natural law philosophy of Thomas Aquinas and was developed further within the Catholic Church, since the 18th century, in response to reformation, socialism and later fascism. Notably it was incorporated into Pope Pius XI’s encyclical Quadragesimo anno, 1931. ]
and became a tenet in various modern frameworks of government, in theories of law, management methods and modern forms of military command. Subsidiarity implies that any issue should be dealt with at the most immediate or local level that is consistent with its resolution; a central authority should have a subsidiary function, performing only those tasks which cannot be provided at a more local level.

When applied to the inner organisation of software systems, in conjunction and interplay with Separation of Concerns, it has the effect to simplify the reasoning about requirements and dependencies, since local solutions for many aspects are preferred, and the structure and layout of facilities is allowed to differ between components, accepting even some degree of redundancy. A solution within local scope can typically be provided in a much simpler and direct way, since only those problems actually relevant within the local neighbourhood need to be considered and the correctness of the code after change is easier to assess. It should be noted though that overall the complexity of the system is increased by this kind of decentralised organisation based on subsidiarity — yet, paradoxically, such a setup is often simpler to handle and can be adapted to changing requirements with lesser effort, since remote consequences to local changes are unlikely, due to reduced tangling and coupling between modules.

But it is not always possible to establish such structures, because the problem domain must provide suitable sub-domains, which are largely self-contained and amenable to subsidiary solutions. Separating and solving the global and overarching concerns without interfering with the decentralised structure does require special attention and dedicated design work at times. And one must resist the temptation to solve some given issue “once and for all”, whenever a slightly boring yet local solution is a possible alternative — a restraint, that runs counter to the engineer’s deeply ingrained ambition towards control and achievement. Complexities of technology occur intertwined with the call and demand for them to be mastered, by the technician, who is trained and capable to accept the challenge, divide et impera, to dissect matters, employ logic and reasoning, and to apply an arsenal of effective tools, routinely. The downside is that it can be difficult to foresee how sustainable a chosen solution avenue will be and how much additional complexity might ensue, once it interacts with further usage of the system. And so it is compelling to forego the reflection of consequences and stick to methods and technology, crisp and clear, apt to deliver one victory after another.

Both Patterns and Antipatterns are related to the phenomenon of mental abbreviations or shortcuts, which, according to contemporary theories of cognition, are a way to cope with the complexities of our world. As such they are ambivalent: creating or using a mental abbreviation can be a powerful tool; instead of working through all the considerations in detail, an evocative and mediating mental image is applied, like a template of prepackaged decisions and actions. Using such a shortcut as leverage can be essential, and enables conscious decision and swift acting, while otherwise working each time through all the painstaking details would surpass the mental capacity of a single person. Without a guiding idea, you wouldn’t see the forest for the trees. But the flip side of the coin is that the use of such abbreviations and abstractions becomes habitual — the mental shortcut must be applied by routine, to be effective. All the details folded away behind the abbreviation will fade away quickly, and tend to be forgotten, once you get used to it. And while the usage tends to become broader over time, gradually more sloppy and less careful and focused, also an inherent tendency towards mystification is involved. A successful Pattern, or Antipattern for that, becomes emotionally loaded with time, habitually ascribed with magical powers, and runs danger to be used as a panacea. If all you know is a hammer, the world consists of nails.

There is a fine line between an effective abstraction, which indeed requires some degree of boldness to be enforced, and an perky abstraction applied in a self-assertive way. Once you get used to the powerful effect of simplification precipitated through abstraction, it is all to easy to walk that route backwards and start with some technique you’d like to implement and apply — and then to cook up a suitable abstracted scheme to justify that decision after the fact. Doing so would be an act of force, as this kind of duplicitous abstraction is not rooted in the problem domain and will bring on some discrepancies, and thus generate accidental complexity later down the road. Yet every abstraction, and in fact any added structure, will incur additional cost somewhere else, in terms of added complexity and mental load to understand the situation, and so it might be compelling to ignore the first subtle signs of mismatch, skip the self reflection and remain in a state of denial regarding the actual effects of this self-indulgent simplification — and taken together this completes the self-reinforcing circle of delusion underlying any Antipattern.

Oversimplification is often amplified by another Antipattern known as »Domain Allergy« (or »Domain Aversion«): by aiming at a strikingly generic solution, the pain of working through all the pesky details of the problem domain can be evaded, at least long enough to offload the “rest” of the problem onto the user. The textbook example for this Antipattern is to build a business application without understanding and modelling the kinds of business transactions and their connections to the entities involved; rather, the data is modelled as key-value pairs to represent “the entities”, “the parameters” and possible relationships dynamically, by actively probing code. Since users tend to describe the requirements by a set of examples, it becomes very easy then to code-up these examples in a case-by-case manner, using dynamic switch-on-selector. The blame for any discrepancies can be deflected conveniently to the user afterwards for providing an “incomplete specification”.

Antipatterns, like Patterns, reinforce each other. They are woven into a fabric of interconnected mental abbreviations, linked together and spread by corresponding forms of meaning and language, which both encase and evoke and regenerate patterns of behaviour.

After having gained some experience, developers often tend to introduce an abundance of flexibilisation mechanisms, so that any future demand they’ll have to face could be resolved by hooking into a chain of decision points spread all over the system. Complemented by the oversimplification, generating an illusion of far reaching and strikingly generic capabilities, implementation can proceed swiftly by distributing yet another special case all over the system. The infamous switch-case statements are a common symptom of such an approach. There are numerous forms of this Antipattern, like pervasive use of SAM-Interfaces,
[ Single Abstract Member, an interface with only a single virtual operation, which is typically a verb, and often something quite generic, like create(). As any of the programming techniques mentioned in this section, such interfaces have valid uses — what makes them questionable is their use without reason, other than to “come in handy”. ]
at places without actual reason for indirection, likewise the dispatch through functors and closures without even an attempt to conceptualise the indirected operation, or excessively shrewd configuration systems, where everything can be overridden and re-patched by global system parameters and environment variables.

A common, almost clichéd trait of many developer cultures is their compulsive hate for “the (l)user” and a deeply rooted dislike for anything related to “peopleware”, which is closely linked to the fact that “those other people” do not rely on the same mental abbreviations and thought forms as we do. A discussion about the fine points of programming language syntax is much rejoiced, yet understanding how the system is hard to use for a non-programmer can be a PITA. So the default response to any kind of tension is to cut the ground under the user’s feet by making everything configurable. And the users, on their behalf, are quick to pick up the rules of the game. Instead of considering and explaining what they actually need, a bunch of isolated features is requested, looking for ways how to tweak and fiddle their way towards that kind of “solution” that seems easily imaginable. So both sides happily agree to introduce GUI options, icons and keyboard shortcuts, themes and text templating macros and whatever helps to sustain the convenient state of denial, while the overall complexity in the system grows and thrives.

When getting engulfed with highly complex matters, a conspicuous Antipattern becomes visible, which I’d like to designate as the »can be solved«-Antipattern: While pursuing a goal, which is challenging in itself, and to which we have acquired some emotional attachment already, we might unfortunately discover a field of further concerns, connected aspects and complexities, which we might not have imagined prior to engaging into the challenge. If taken seriously, together these discoveries would amount to a change of perspective: it would be prudent to step back and reconsider the situation.

But we do not want to let go of our focus, nor leave the comfortable drive we have gained, feeling that our aspirations still might be at arms length. We then silence the nagging doubt and fight off the demand for a reassessment by rather treating each problematic aspect, one by one, with a schematic band-aid, which can be some means of technology at times, or an organisational provision — while notably keeping each of the new aspects isolated, fixed within a remote and abstracted perspective, without unpacking any of the ramifications or looking into the consequences the alleged band-aid solution would have on our beloved goals. We might even be willing to spend considerable effort to pull-in the alleged solution and integrate more technology, as technology is always enticing, as long as it stays at a provisional level, so that we are finally entitled to the claim “this can be solved through XYZ” (which however silently implies a “but not by me, because I could not care less, once I reached my goal”).

This scheme is not just some questionable behaviour, happening on occasion. Rather this is a real Antipattern, because the proclaimed solution sounds plausible and even compelling, as it caused the threat of complexities, slowdowns and roadblocks to disappear, after all. The course of action taken seems justified at that point: A person acting by this pattern will look like a winner, and will trump another person proceeding cautiously, while frequently re-adjusting the plan and possibly even dropping some goals altogether. Especially in the context of a team or group of people, this style will catch-on quickly and be ingrained into the group identity. And this style very effectively muddles the waters: once problems start to creep up, during the integration phase, they can be brushed aside easily, as being related to that stuff which “can” be solved; we even foresaw all those issues, didn’t we? Applying this Antipattern might even be an effective means to silently disable most of the »definition-of-done« criteria, that were established up-front, as we obviously “can not verify that yet”, due to this other issue that “can be solved”…

As with any Antipattern, it is important not to digress into moral judgement. It still remains an uneasy question when to blame a person for doing something obvious, and even more so when numerous other people do the same, to the degree that it seems to be a winning strategy. The fact that we live in a VUCA world is hard to swallow.
[ VUCA stands for volatility, uncertainty, complexity and ambiguity. This revelation can be depressing; in order to act and be successful, a certain degree of confidence is a prerequisite. It is impossible to get moving, and changing the world, for better or worse, without some amount of simplification. ]

Complexity and Flexibility are related. Depending on the perspective, a given situation might appear complex, hard to grasp and impossible to control. Or, by conscious choice, some configuration of structure is established to cope with the current situation, while retaining the flexibility to adapt to changing requirements later.

Flexibility is ambivalent. It can be that decisive move that makes a solution viable. As such, it is essential part of every architecture, because architecture establishes certain confines and guides in order to create an open, positive space, a possibility for things to happen. Good architecture channels the capabilities into a way to resolve conflict before it materialises. Yet when architecture fails, or is absent altogether, then flexibility can be introduced as an pretence to avoid engaging into the hard work to get at the root of the failure. It can be used to evade a conflict and push it towards the future, and to avoid commitment.

In psychology, the term »coping strategy« designates an action, a series of actions, or a thought process used in meeting a stressful or unpleasant situation or in modifying one’s reaction to such a situation. Coping strategies typically involve a conscious and direct approach to problems, in contrast to defence mechanisms.
[ American Psychological Association, Dictionary of Psychology: »Coping Strategy« ]
Coping strategies can be cognitions or behaviours and can be individual or social.
[ Wikipedia: »Coping« ]

Building and maintaining software requires considerable effort, and a commitment to engage into the task. To the degree that people spend part of their life time on that work, use their will and creativity to overcome the obstacles and maybe even identify with the goals set forth, they live through the process of creating software, both as individuals and as a group or team. Working through difficulties thus becomes a form of coping — even when formalised and established methods and procedures are employed.

On a personal level, coping can be emotion-focused or problem-focused, where the latter is directed at the stressor itself: taking steps to remove or to evade it, or to diminish its impact if it cannot be evaded. Persons and groups may develop a distinct style of coping; empirical psychological research shows that some link can be established between personality traits and coping strategies, but the former are much more stable over the lifetime of an individual, while coping strategies are moderated by age, the actual type of the stressor, context, experience and learning.
[ An extended summary of recent research can be found in the article »Personality and Coping«, by Charles S. Carver and Jennifer Connor-Smith, Annual Review of Psychology, Volume 61, 2010.
Quote from the abstract: “Personality psychology addresses views of human nature and individual differences. Biological and goal-based views of human nature provide an especially useful basis for construing coping; the five-factor model of traits adds a useful set of individual differences. Coping—responses to adversity and to the distress that results—is categorized in many ways. Meta-analyses link optimism, extraversion, conscientiousness, and openness to more engagement coping; neuroticism to more disengagement coping; and optimism, conscientiousness, and agreeableness to less disengagement coping. Relations of traits to specific coping responses reveal a more nuanced picture. Several moderators of these associations also emerge: age, stressor severity, and temporal proximity between the coping activity and the coping report. Personality and coping play both independent and interactive roles in influencing physical and mental health.” ]
Whatever we do and undertake is already part of a meaningful context, and similarities can therefore be recognised and put into words. A network of patterns runs through our intentions and goals, our actions and solution strategies — and so a pattern recognised in a successful solution, by means of an evocative name and mental image, can inform a strategy to cope with similar problems in the future, and to do so in a healthy way.