An order taken with no signal, delivered to SAP exactly once — never lost, never billed twice

The problem: you cannot tell whether it worked

Every system that sends money-shaped messages over an unreliable network runs into the same wall. A request goes out and no answer comes back. Did the other side process it and lose the reply, or never receive it at all? From the sender's position those two situations are identical, and both possible responses are wrong: retrying can double-charge a customer, and not retrying can silently lose an order.

You do not solve this by guessing more cleverly. You solve it by making a repeat submission harmless — so the client can resend freely, and the server recognises "I have already seen this one" and hands back the original result instead of creating a second one. That property is called idempotency, and building it end to end, across a phone, two backend services and an ERP, is what this project is about.

The setting makes it concrete rather than academic. Traditional-trade beverage distribution means a rep visiting dozens of small shops a day, in and out of coverage, where a duplicated invoice is a phone call from an angry shopkeeper and a lost order is a delivery that never arrives.

The spine: one identity, minted on the phone

The whole design hangs off a single decision — the phone creates the order's identity, not the server. A rep in a dead zone has no server to ask, so the app generates the ID itself, before it knows whether the internet exists. That ID travels as the order's primary key and as the HTTP idempotency key, unchanged, byte for byte, on every attempt forever. If the client ever generated a fresh ID per retry, every safeguard behind it would be protecting a guarantee that had already been lost on the device.

The mobile app treats its own storage as the source of truth. An order is written to local SQLite first and only then sent — never the other way round — so an order survives the phone being killed, the battery dying, or the app being force-quit between "saved" and the network call. That is the same reasoning as the server's outbox, applied one layer earlier.

• UUIDv7 rather than the more common v4: it begins with a timestamp, so IDs sort by capture time as plain text. The offline queue drains in the order the rep actually took the orders, and the database index grows in order instead of at random.
• The queued request body is frozen at capture time and resent unchanged, because the server compares a hash of the body — a retry that re-serialised even slightly differently would be refused as a conflicting resubmission.
• A dead network is recorded as "outcome unknown", never as "failed". Telling a rep their order was lost while it sits confirmed on the server is its own kind of bug.
• The phone's retries never give up. The server can park an order for a human after twelve attempts because someone is watching; nobody is watching a phone in a pocket, so it slows to a five-minute interval and shows the attempt count — turning "silently lost" into "visibly stuck".

Handing the order over exactly once

When an order arrives, the service prices it from its own price list — the phone's prices are only an estimate, and money is never decided by whichever copy of the catalogue is stalest — then saves the order and the instruction to send it to SAP in one database transaction. Both, or neither. This is the transactional outbox pattern, and choosing it meant deliberately not adding a message broker: with the queue living in the same database as the data, there is no window where the order is safely stored but the instruction to deliver it has been lost.

A separate worker drains that outbox. It claims jobs with SELECT … FOR UPDATE SKIP LOCKED, which in plain terms means "give me work nobody else has taken, and do not queue behind the rows they have" — so many workers can run at once without two of them ever posting the same order.

The careful part is what happens when SAP does not answer. Before calling, the worker records that it is about to post. If the reply never arrives, it does not assume anything: the next attempt asks SAP whether a document already exists for this order, and only posts again if SAP definitely says no. That is the difference between a lost reply and a duplicate invoice, and it matters because a real ERP does not promise to de-duplicate for you.

The AI is allowed to suggest, and nothing else

A separate Python service recommends what else a shop is likely to want, and writes the pitch in both Arabic and English — including the Unicode bidirectional isolates needed to stop a Latin product name rendering scrambled inside right-to-left Arabic text. The recommendation itself is a transparent weighted score over three signals: what this shop habitually buys, what tends to be bought alongside today's basket, and what similar shops buy. Confidence is scaled by how much evidence exists, because a product bought on a shop's only recorded visit has a perfect hit rate and is worth nothing.

The important part is the boundary. That service connects to the database as a role with SELECT permission and nothing else, and it audits its own privileges at startup and refuses to boot if it discovers it can write. A large language model is available as an optional re-ranker, off by default, and it can only reorder suggestions the deterministic model already produced — it cannot invent a product, change a price, or add a line to an order. A slow, wrong or hallucinating model can make the app less useful; it cannot make it incorrect.

Running it: Kubernetes, and why not something easier

The whole environment is Terraform — cluster, database, registry, key vault, identities — and teardown was built and tested before anything was built on top of it, because an environment you cannot destroy is an environment you cannot afford to leave running. No password appears in any manifest: pods authenticate with workload identity and secrets are mounted from Azure Key Vault.

Azure Container Apps would have been faster, and an earlier architecture decision record chose it. I overrode that later and wrote a second record superseding the first, because the goal was to understand the orchestration layer well enough to defend it, and Container Apps' main virtue is doing all of that invisibly. The payoff was not theoretical: several real bugs were only findable because Kubernetes exposed the setting that was wrong.

• Two replicas that looked like high availability and were sitting on the same node, because the spread rule said "if convenient" and it was not convenient.
• A resource quota I set myself that deadlocked my own rolling update.
• A shutdown grace period the application never actually used — which turned out to be losing customer orders in production.
• Autoscaling driven by queue depth rather than CPU, because a worker waiting on a slow ERP call uses almost no CPU while its backlog grows.

Proving it — including the bug the proofs missed

Four claims are measured against a live cluster rather than asserted: a reply destroyed after SAP committed, a worker force-killed mid-post, the AI plane's read-only limit checked against the managed database, and a deployment rolled out under load. Thirteen assertions, all passing. Afterwards the database held 566 orders and 566 SAP documents — one to one, nothing lost, nothing duplicated, nothing stuck — with around 300 automated tests underneath.

The more useful story is what auditing the running system found. Two real orders were permanently stuck, and SAP had already created documents for both nineteen hours earlier. SAP believed two shops had ordered; the system believed both had failed. The cause was a shutdown setting: Kubernetes allowed the worker 45 seconds to finish, and the application used about 8, because its framework's graceful shutdown waits for web requests and this service does all its work in a background task. So workers were being killed mid-conversation with SAP every time the fleet scaled in.

Fixing it took three changes — the shutdown now waits, the recovery sweep repairs both affected tables atomically, and the operator tool can rescue orders already stranded. The tempting one-line fix, letting those orders through, would have been far worse: it would allow a second worker to post while the first was merely slow. When the two stranded orders were re-driven, the logs showed the design working as intended — the worker asked SAP, found the documents that already existed, and recorded them rather than creating duplicates.

Two of the original proofs also turned out to pass while proving nothing. One asserted that SAP held exactly one document — which was equally true if the worker posted twice and the mock ERP quietly de-duplicated, meaning the guarantee belonged to the mock rather than to my code. It now asserts that SAP received exactly one POST. Another printed "36 skipped" while the summary still read "12 passed, 0 failed", because a three-second connection timeout had silently removed every assertion in a security test suite. A test that can pass without doing its job is worse than no test, because it manufactures confidence.