One workflow to rule them all?

During the 2024 edition of Agile Lean Europe (but very much aside of the conference), I found myself thinking a lot about processes and workflows: I had a question on my mind: “Is it only me, or we are always reinventing the concept of workflow?”. I discussed it with Vladimirs Ivanovs and Markus Wissekal, and ultimately with Silvana Wasitova

There was some insight, and I like to keep such smart things in my pages 🤓.

Introduction

The business concept of “Workflow” points to different meanings, depending on your context:

• it can be the way one works, how one gets things done
• it can indicate a generalized program
• it can point to a group of technologies, that aim to capture the two above points
• It could suggest techniques that, through analysis, highlight setbacks and bottlenecks.

Here I will explore the merits of some architectural choices, and how they can enable new types of analysis and be in general a treasure trove for observability and tracking.

The core rationale for this discussion lies in the exploration of a ubiquitous pair of “workflow related” requirements:

• The need to execute and implement a specific business process
• The imperative to track the state of execution of a job

Interestingly, these two requirements, upon close examination, seem to reflect the same underlying need. Let’s delve deeper to comprehend how and why.

The business process is a pipeline for humans

Every business functions according to a specific set of tasks and procedures, akin to following a recipe, hence the term “business processes”.

These processes are unique to each company, sculpted through its structure, historical background, business vertical, and the people associated with it. In a lighter vein, you could consider these processes as a company’s equivalent of “Nana’s Old-Fashioned Birthday Cake Recipe.”¹:

Business processes usually evolve through internal human interactions. For example, common business processes include:

• Order Fulfillment Process: This is the sequence of steps from the receipt of an order up to its delivery. Let’s say, for instance, you receive an order of a chocolate cake. Then your business process might have steps like:
  • Checking the availability of required ingredients like chocolate.
  • Determining if the required equipment (oven in this case) is free.
  • Underlining the scheduled time of the cake delivery.

• Employee Time-off Management: How does an employee apply for leave and how are leaves granted?

• Customer Complaint Resolution: How are customer grievances registered and solutions provided?

At each of these steps, companies strive to enforce their own way of executing[^com-adv] (embedded in the enforcing of “our standard” process), track execution and compliance (so that the company knows that something has been done, and nothing has been “lost” ) and, in a later stage, automate to enhance performance and productivity.

Business processes are usually executed at least partially by people, and when that happens, people are the driver of decisions.

The pipeline is an automated process

As an example, let’s consider the “software build pipeline”, essential part of any modern CI/CD.

Generally, this pipeline evolves through code commits and merges from developers within the project. For instance, some common stages in a software build pipeline include:

• Code Compilation: This process converts source code into machine code. For example, if there’s a syntax error in the code, the pipeline provides immediate feedback, enabling developers to correct the mistake before moving on to the next stage.

• Testing: Automated tests are run to ensure that new changes don’t break existing functionality. If a test fails, a feedback loop informs the developers so they can fix the issue.

• Deployment: The verified code is automatically deployed to its respective environment (like staging or production), keeping the application up-to-date and allowing for quick feedback on live performance.

At each of these stages, automation plays a critical role in enhancing efficiency and productivity, with the ultimate goal of transforming these processes into a flawless pipeline.

So a pipeline, effectively behaves like an automated process, with the difference that all the input os at the beginning.

The need for tracking is actually always there

The more a process is automated, the more we need to track it, in order to troubleshoot in case of trouble. Additionally, a business process should be measured for efficiency and effectiveness, so that companies know where their “computational hot spots” are, where there is friction.

This helps a lot when we ( and I mean us programmers and industrial technicians) come into the picture, so that we can help by proposing valuable 20%-80% solutions for our customers.

Process tracking further allows for “retrying/resuming” of operations. When each step of a process maintains a proper state history, it becomes feasible to resume a failed step. This functionality is especially beneficial for systems reliant on other systems, which may encounter temporary downtime, faults, or “bad data” during the initial attempt.

MOAR - the Mother Of All Requirements

Finally, let me explicate what I meant when I said that both requirements are the same: put simply, the requirement is “do what we already do, but better”. Make no mistake: this is an implicit plea for workflows! It requires us to offer

• a single point of contact to execute the task that encompasses multiple systems, what we call “the portal”
• a single point where to evaluate the quality of the work, with metrics and statistics: “the dashboard” as it is usually called
• a way to easily express dependency on other previous tasks or optionality on some input.
• allow for retry and “live-troubleshooting”, which essentially is one of the hidden sub requirements of the “what we already do” part.
• a platform where we can start automating our dependencies on other systems: for example a step that usually would be done manually (send payments) once the system is stable can be automated

Conclusions

When we improve a process for productivity, we implicitly face the challenge of the MOAR. More often than not, we can get away with not doing a lot of the “work” because our tech today is so awesome, and because our users still have a lot of freedom in their processes. I think that getting away easily, in some contexts, is a feature, not a bug: it allows us to be cheaper and more effective ourselves, doing only the relevant part of a process and call it day, without necessarily implementing the whole process support. On the other hand, let’s always keep in mind the real ask, and let’s factor that the bigger the company, the bigger the need for observability.

SAP docet.

Footnotes

1. Some call this competitive advantage, or the soul of a company ↩