Planning for an exponential future: Software Engineering and Scenario Management

We are living in an era of exponential growth. On the ground, from an everyday perspective, this rapid rate of change isn’t obvious or even plainly visible, but at the cutting edge of software development, it couldn’t be clearer. From a human perspective, software development is likely to shift from coding to the essential fields of software engineering and architecture, where design is at a higher level than detailed coding. These activities will require as much human interaction and expertise as they ever have. Having Humans In the Loop (HIL) will ensure that projects are effective and efficient as well as satisfying contract terms and regulatory and compliance requirements.

Because of the rate of change, it is becoming harder to predict the future. Even in a few weeks (or, increasingly, days), the outlook can change completely. This means that an Agile approach is even more important than in the past because:

• It minimises the risk by only doing what is necessary at the time
• Changes can be rapid and effective
• You know as early as possible if you’re “building the wrong thing”

The rate of change means that our traditional ability to see further into the future has all but evaporated. In the past, it was easy to predict what might happen in three, five and even ten years’ time. Even with something as technologically advanced as a smartphone, it’s fairly easy to predict a better camera, more memory and higher-resolution screens (or not, as we reach the point of “good enough”). You might say that the “angle of expectation” was only, say, 15 degrees. This is not a precise measure, but essentially a visualisation of how wide our margins of predictive thinking need to be. But with the new developments in AI and related fields, that angle has increased to 180 degrees. In other words, we don’t know where to look. Imagine the future is a dot on the horizon. It’s in front of us, but that’s all we know. We can’t tell where on that 180 degree vista the future will appear.

The reason for this is that in an exponential era, as progress speeds up, even a tiny increment on the time axis can lead to a massive jump on the capability axis. So, from our perspective, we don’t know where or when the next giant leap will take place. Almost every day takes us by surprise, and every week we see developments that we would previously have expected to take a decade or more.

Given that we face a wider spectrum of possible outcomes, we should architect our software to be able to work with and manage as wide a range as possible of future scenarios.

Specifically, this would mean

• Building interfaces for AI services
• Analysing processes so that they are ready for the changes likely to be caused by AI
• Monitoring trends - especially exponential ones

To fully embrace the current rate of change we need to avoid letting linear thinking distract us from the reality of exponential change.The only way to effectively plan for the future is to spot exponential trends and leave behind our dependency on reasoning like “this has always happened before, so this will happen in the future” We also need to look for signs of acceleration in all domains: physical, electronic, software and AI. Not everything is accelerating, so part of the task will be to devise reliable but flexible interfaces between accelerating processes and non accelerating processes.

This might sound abstract, but examples are all around us. The biggest indicator of exponential change is when experts in their own field are surprised by new developments. This happens almost daily now, but it is as clear an indicator as it possibly could be that Agile was made for this environment: maintaining productivity in the midst of surprise and minimising risk for the best possible practical outcomes.