Project: Deep Breath

Biological wastewater treatment lives or dies on aeration and dosing, and both were being tuned by hand. Operators reacted to lab results that arrived long after the water had moved on, sensor feeds were noisy and regularly interrupted by calibration and drift, and every plant had its own layout of valves, blowers and instruments. The operator wanted automated control that saved energy and chemicals without ever putting discharge compliance at risk, and that could be rolled out to more than one plant.

We built a control system that sits between the plant's PLCs and its equipment and makes a decision every cycle. An expert engine encodes how experienced operators actually run the process, coordinating valves, blowers and dosing pumps in a layered strategy where the gentlest lever is always tried first. Before any decision is made, a data-quality layer screens out calibration windows, frozen readings and spikes. AI models trained on the plant's own history run alongside the expert engine, but they start with no authority at all: a continuous evaluation loop has to see sustained compliant operation before their influence is raised, one small step at a time. Each plant is described by its own profile (instruments, valves, blowers), so one software image serves every site and models are locked to the plant they were trained on. Around it sit a data health check for onboarding new plants, an offline training lab with closed-loop simulation, a REST API with hot-reloadable configuration and a full audit trail, and a Docker deployment that runs on the plant's own servers.

Machine learning here is an apprentice, not a replacement. Valves, blowers and dosing each get their own model, and each has to qualify on its own. Learning starts before go-live: the system runs a slow, deliberate sweep of the valves across their full range, so the models see how the process responds at every operating point rather than only in the narrow band operators usually hold it in. From then on, every decision the expert engine makes is recorded together with its context and its outcome, building a training set that belongs to that plant alone. Alongside the control models we train a forward model of the process itself, which predicts how water quality will respond to incoming load and a proposed action, so new strategies can be tried offline long before they reach real equipment. Candidates are retrained in the background, away from the control loop, and judged the way they will actually be used: on chronological splits that never let the future leak into training, against an honest "nothing changes" baseline, and in closed-loop replays of real operating days where the model drives a simulated plant end to end and is scored against the expert engine on compliance, chemical use and how smoothly it controls. Only a model that then passes a recurring health check on live operation is given a small share of the decision, and that share grows gradually with each consecutive pass. If anything degrades, its share drops straight back to zero and the expert engine carries on alone.

Work in progress. The control core, the data-quality layer, plant profiles, the API and the full training and evaluation pipeline are built and covered by 252 automated tests, and the system has already been adapted from its first plant layout to a second. The plant runs on the expert engine first; AI models are trained and waiting, and will only start sharing decisions once enough live data from that plant has accumulated and they pass their health checks on it.