System design / HLD + LLDAdvanced · 9 sections · 7 figures · 19 min read

Order Management System

  • Transactions
  • Sagas
  • Outbox
  • Designed · build in progress
The constraint

Money and stock must end up right when payments, webhooks and consumers fail in every order imaginable.

The decision

A modular monolith: one transaction reserves stock and creates the orders; an orchestrated saga does the rest.

0
oversold units, by constraint
35/s
peak checkouts
15 min
max to reconcile a payment

Domain: a pickup-first local marketplace (“QuickMart”). A customer fills one cart from several nearby shops, pays once, and collects each shop’s part when it is ready.

1. Problem and requirements

The multi-shop model

One customer action produces several orders, and they must be modelled separately:

Checkout CHK-1001 (customer pays once: ₹208)
├── ShopOrder SO-1 · Fresh Basket    · 3 items · ₹173 → pickup when READY
└── ShopOrder SO-2 · Corner Pharmacy · 1 item  · ₹35  → pickup when READY
Question Decision
Is payment collected once or per shop? Once per checkout. The marketplace collects and settles to shops later. One payment is simpler for the customer and avoids partial-authorisation states.
Can one shop’s order be cancelled alone? Yes. Cancellation and refunds are per shop order; the others continue.
If one shop rejects its order, what happens to the others? They continue. The rejected shop order is refunded in full automatically.
Can items within one shop order be cancelled? Only by the shop marking them unavailable before preparation; the line is refunded.
How is the refund calculated? Sum of the affected lines plus their share of any checkout-level discount, allocated pro rata by line value and fixed at checkout time.

Functional requirements

# Requirement
F1 Validate a cart: products exist, prices are current, shops are open.
F2 Reserve stock for every line so it cannot be sold twice while the customer pays.
F3 Take one payment for the checkout; confirm all shop orders when it succeeds.
F4 Release reservations if payment fails or does not finish in time.
F5 Notify each shop of its confirmed order; shops accept or reject, then mark preparing and ready.
F6 Customers and shops can cancel within the rules; refunds follow automatically.
F7 Track every order’s status and history; show it to customers and shops.
F8 Publish order events for notifications, analytics and settlement.

Non-functional requirements

# Requirement Target
N1 No overselling Zero, enforced by the database
N2 No charge without an order, no order confirmed without a charge Every mismatch detected and resolved within 15 minutes
N3 No double charge, no double refund Zero, enforced by idempotency keys
N4 Checkout API latency (to payment page) p99 under 800 ms
N5 Order status freshness for shops Under 5 s from payment
N6 Checkout availability 99.95%
N7 Audit Every state change and money movement is recorded and immutable

Out of scope

Delivery logistics (pickup only in this version), shop settlement and payouts (consumes our events), product search and catalogue management, and promotions engine design (we consume a computed price).

2. Capacity estimation

Assumptions: 300,000 checkouts a day; 1.5 shop orders and 4.5 lines per checkout on average; sale events reach 10× the average for an hour; flash deals concentrate demand on a few SKUs; about 8 events per checkout over its life; about 5 KB of rows per checkout including history.

Metric Working Result
Checkouts, average 300,000 / 86,400 ≈ 3.5/s
Checkouts, peak 3.5 × 10 ≈ 35/s
Stock reservation updates, peak 35 × 4.5 ≈ 160 row updates/s
Order status reads (customers polling, shops’ dashboards) Assume 20 reads per checkout ≈ 70 reads/s average, 700 peak
Events 300,000 × 8 / 86,400 ≈ 28/s average, ≈ 280/s peak
Storage 300,000 × 5 KB ≈ 1.5 GB/day, ≈ 550 GB/year
Flash deal 5,000 customers trying to buy 200 units of one SKU in the first minute ≈ 80 attempts/s on one row

What this says: aggregate volume is modest for one PostgreSQL primary. The hard part is contention on hot rows during flash deals and correctness across the payment boundary, not throughput. The design spends its complexity there.

3. Architecture and APIs

High-level architecture

FIG 01ArchitectureHigh-level architecture
Module Owns Never does
Checkout Checkout aggregate, saga state, discount allocation Change stock or payments directly; it calls the owning module
Order Shop orders, lines, status history, cancellation rules Talk to the payment provider
Inventory Stock levels, reservations, expiry Know about payments
Payment Payment intents, attempts, provider events, refunds Change order state directly; it emits events the saga reacts to
Catalog read model Prices and availability as seen at checkout Write anything during checkout

Modules talk through Java interfaces inside one process and own their tables. That boundary is what allows extracting Inventory or Payment later (stage 4).

Which call creates the durable order

POST /checkouts is the only call that creates orders. It creates the checkout, every shop order (state AWAITING_PAYMENT) and every stock reservation in one transaction, and returns a payment session. There is no public POST /orders; that ambiguity is removed on purpose. Payment confirms orders; it never creates them.

API contracts

Method and path Purpose Success Notable errors
POST /checkouts Validate cart, reserve stock, create checkout and shop orders, start payment 201 checkout + payment session 409 OUT_OF_STOCK (with lines), 409 PRICE_CHANGED, 422 SHOP_CLOSED
GET /checkouts/{id} Checkout with its shop orders 200 404
GET /orders/{id} One shop order with history 200 403, 404
POST /orders/{id}/cancel Customer or shop cancels 202 cancellation accepted 409 NOT_CANCELLABLE (already preparing)
POST /orders/{id}/accept, /reject, /ready, /picked-up Shop workflow 200 409 ILLEGAL_TRANSITION
POST /payments/webhooks/{provider} Provider events 200 400 bad signature
POST /inventory/reservations Internal only (module interface, not public)
POST /checkouts
Authorization: Bearer <customer token>
Idempotency-Key: 5c0e…  (generated by the app when the user taps "Pay")

{
  "cartVersion": 14,
  "lines": [
    { "shopId": "S-FB", "sku": "MILK-1L", "qty": 2, "unitPrice": 6400 },
    { "shopId": "S-FB", "sku": "BREAD-W", "qty": 1, "unitPrice": 4500 },
    { "shopId": "S-CP", "sku": "PARA-500", "qty": 1, "unitPrice": 3500 }
  ],
  "couponCode": null,
  "pickupWindow": "TODAY_EVENING"
}
201 Created
{
  "checkoutId": "CHK-1001",
  "status": "PENDING_PAYMENT",
  "total": 20800, "currency": "INR",
  "shopOrders": [
    { "id": "SO-1", "shopId": "S-FB", "status": "AWAITING_PAYMENT", "amount": 17300 },
    { "id": "SO-2", "shopId": "S-CP", "status": "AWAITING_PAYMENT", "amount": 3500 }
  ],
  "payment": { "intentId": "pi_3Px…", "clientSecret": "…", "expiresAt": "2026-10-22T10:15:00Z" },
  "reservationExpiresAt": "2026-10-22T10:15:00Z"
}

Amounts are integers in the smallest currency unit (paise). The client sends the prices it showed; if the server’s price differs, it answers 409 PRICE_CHANGED with the new prices rather than silently charging a different amount.

Deployment

FIG 02ArchitectureDeployment

A synchronous standby is used because this database records money: a failover must not lose committed payments.

4. Internal design and workflows

Class diagram

FIG 03Class modelClass diagram

Shop order state machine

FIG 04State machineShop order state machine

Terminal states are COMPLETED, REFUNDED, EXPIRED and CANCELLED (when nothing was paid). A customer can cancel until the shop starts preparing; after that, only the shop can cancel. Transitions live in one table-driven class, and every transition is a conditional update, so concurrent actions (customer cancels as shop accepts) resolve to exactly one winner:

UPDATE shop_order SET status = :to, version = version + 1, updated_at = now()
WHERE id = :id AND status = ANY(:allowedFrom) AND version = :expectedVersion;

Checkout state machine

FIG 05State machineCheckout state machine

Data model and indexes

FIG 06Data modelData model and indexes
-- Overselling is impossible: stock can never go negative
ALTER TABLE inventory ADD CONSTRAINT stock_non_negative CHECK (available >= 0 AND reserved >= 0);
-- One checkout per customer action
CREATE UNIQUE INDEX checkout_idem ON checkout (customer_id, idempotency_key);
-- Duplicate webhooks are recorded once
-- (PRIMARY KEY (provider, event_id) on payment_event)
-- Expiry sweeper
CREATE INDEX reservation_expiring ON reservation (expires_at) WHERE status = 'HELD';
-- Shop dashboard: live orders for a shop
CREATE INDEX shop_live_orders ON shop_order (shop_id, status) WHERE status IN ('CONFIRMED','ACCEPTED','PREPARING','READY');

Reserving stock without overselling

Each line is one conditional update; all lines of a checkout are in one transaction, so the checkout either reserves everything or nothing:

UPDATE inventory
SET available = available - :qty, reserved = reserved + :qty
WHERE shop_id = :shop AND sku = :sku AND available >= :qty;   -- 0 rows: out of stock
@Transactional
public Checkout start(StartCheckout cmd) {
    var existing = checkouts.findByIdempotencyKey(cmd.customerId(), cmd.key());
    if (existing.isPresent()) return existing.get();                 // retry of the same tap

    var priced = pricing.verify(cmd.lines());                        // 409 PRICE_CHANGED if different
    var lines = priced.sorted(comparing(Line::shopId).thenComparing(Line::sku)); // fixed lock order: no deadlocks
    var shortages = new ArrayList<Line>();
    for (var l : lines) if (!inventory.tryReserve(l.shopId(), l.sku(), l.qty())) shortages.add(l);
    if (!shortages.isEmpty()) throw new OutOfStock(shortages);       // whole transaction rolls back

    var checkout = Checkout.create(cmd, priced, clock.instant().plus(RESERVATION_TTL));
    checkouts.save(checkout);                                         // checkout + shop orders + reservations
    outbox.append(checkout.id(), "CheckoutStarted", checkout.toEvent());
    return checkout;
}
// After commit (outside the transaction): payment.createIntent(checkout, key = checkout.id())

Sorting lines by (shop, sku) gives every transaction the same locking order, which rules out deadlocks between two checkouts that share items. The CHECK constraint is the backstop: even a buggy path cannot drive stock below zero.

Reservations last 15 minutes. A sweeper (every 30 s, FOR UPDATE SKIP LOCKED in batches) releases expired ones: available += qty, reserved -= qty, and expires the checkout. On payment success, reservations are committed: reserved -= qty, sold += qty.

Checkout and payment sequence (the saga)

FIG 07SequenceCheckout and payment sequence (the saga)

Compensation paths

Failure point Compensation
A line is out of stock Nothing to undo: the transaction rolled back; 409 OUT_OF_STOCK lists the lines
Payment intent creation fails after commit Checkout stays PENDING_PAYMENT; the app retries GET /checkouts/{id} which re-creates the intent with the same key; otherwise expiry releases stock
Payment declined PAYMENT_FAILED; customer can retry until the reservation expires; then stock is released
Reservation expires before payment completes Release stock, expire checkout; if payment later succeeds, see the late-payment case below
Shop rejects Refund that shop order in full; other shop orders unaffected
Customer cancels before preparation Refund that shop order; return its stock to available

Transaction boundaries and data ownership

Step Atomic unit Outside the transaction
Start checkout Checkout, shop orders, lines, reservations, stock updates, outbox Creating the payment intent
Payment event payment_event insert, payment status, outbox Provider calls
Confirm Checkout and shop order statuses, reservation commit, outbox Notifications (consumers of the outbox)
Refund Refund row PENDING with key, then provider call, then status

Each module writes only its own tables, even inside a shared transaction. That discipline is what keeps extraction possible.

5. Failure modes and consistency

The four scenarios that prove the design:

1. Payment succeeds, but our update fails or times out

The customer paid; our webhook handler crashed before recording it, or the database was failing over.

  • The provider retries webhooks until it gets a 2xx; our handler is idempotent on (provider, event_id).
  • The reconciliation job runs every 5 minutes: it lists provider payment intents updated in the last hour, compares them with our payment rows, and applies any missing succeeded event through the same code path as the webhook.
  • Result: the order is confirmed late (within minutes), never lost. N2 is met.

2. Payment succeeds after the reservation expired

The customer took 20 minutes in their bank’s app; the sweeper already released the stock.

  • On PaymentSucceeded for an EXPIRED checkout, the saga tries to re-reserve the lines.
  • If stock is still there: confirm normally (the customer never notices).
  • If any line is gone: confirm what can be fulfilled, mark the rest UNAVAILABLE, and refund those lines automatically, with a notification explaining why. A fully unfulfillable checkout is refunded in full.

3. Duplicate webhook

The provider sends payment_intent.succeeded twice (retries, or two event types for one payment).

  • INSERT ... ON CONFLICT (provider, event_id) DO NOTHING records the event once.
  • Even with different event IDs for the same fact, the status transition PENDING → SUCCEEDED is conditional, so the second one changes nothing and emits no second PaymentSucceeded.

4. Stock race on the last unit

Two customers reserve the last pack of PARA-500 within a millisecond.

  • Both run UPDATE ... WHERE available >= 1. PostgreSQL row locking serialises them; the second re-evaluates the condition after the first commits and matches 0 rows.
  • The first gets 201, the second gets 409 OUT_OF_STOCK. No overselling, no lost update.

5. Kafka consumer crashes mid-processing

The notification consumer crashed after sending the shop alert but before committing its offset.

  • On restart, it receives OrderConfirmed again. Consumers keep a processed_event(consumer, event_id) table, or use the notification platform’s idempotency key (order-SO-1-confirmed), so the shop is alerted once.
  • The outbox relay itself can publish an event twice (crash between produce and marking published). Every event carries a unique ID, and every consumer must de-duplicate. This is documented as the system’s contract: events are delivered at least once.

Other scenarios

Scenario Design response
Customer double-taps “Pay” Same idempotency key returns the same checkout; the provider intent is created with key checkoutId, so only one exists
Price changed between cart and checkout 409 PRICE_CHANGED with new prices; never charge a different amount silently
Customer cancels while the shop accepts Conditional update with version: one wins; the loser gets 409 with the current state
Refund call times out Refund row PENDING with an idempotency key sent to the provider; reconciliation resolves it; never issue a second refund
Flash deal: 80 attempts/s on one SKU See stage 5: token pre-allocation in Redis in front of the row
Shop never responds Auto-reject after 10 minutes and refund, so the customer is not left waiting
Database failover Synchronous standby: no committed payment is lost; requests retry with the same keys

Security and abuse

  • Customers see only their checkouts and orders; shops see only their own shop orders (row-level checks in the service, not only in the UI).
  • Webhooks are verified by provider signature and timestamp tolerance; events are re-fetched from the provider API before money-affecting transitions when the signature scheme is weak.
  • Coupons are validated server-side with per-customer usage limits enforced by a unique constraint.
  • Checkout creation is rate-limited per customer and per device, which stops scripted stock hoarding through endless reservations.

6. Scaling evolution

Stage Trigger Design Why
1. MVP Launch Modular Spring Boot monolith, PostgreSQL, synchronous notifications Simplest correct system; one transaction covers checkout
2. Growing traffic CPU or connection limits Stateless replicas, PgBouncer, read replica for order history and dashboards Scale reads and connections without changing the model
3. Slow side effects Notifications and analytics slow checkout Transactional outbox to Kafka; notifications, analytics and settlement become consumers Checkout latency stops depending on downstream systems
4. Independent domains Inventory or Payment need their own release cadence or scale Extract Inventory (and later Payment) into services; the in-process orchestrated saga now calls them over the network with the same commands and compensations The saga was designed for this; only the transport changes
5. Hot SKUs (flash deals) One row over about 50 updates/s, lock waits rising Pre-load deal stock as tokens in Redis (DECR with a floor); only token holders reach the database reservation; tokens are returned on failure Turns thousands of row-lock waits into fast in-memory rejections; the database stays the source of truth
6. High write volume Primary write capacity Partition orders and reservations by shop or region; archive closed orders Shops are natural partitions; cross-shop queries move to the analytics store
7. Multi-region Expansion to new cities with residency or latency needs Region owns its shops, stock and orders (cell architecture); customers are routed to their region; global views come from replicated events No cross-region transactions on the checkout path

7. Operations and validation

Metrics and SLOs

Signal SLO or alert
Checkout latency p99 under 800 ms
Checkout outcome mix Dashboard: success, out of stock, price changed, payment failed, expired
Paid-but-not-confirmed checkouts older than 15 min Page at 1 or more
Confirmed-but-unpaid shop orders Page at 1 or more (should be impossible)
Reservation lock wait time per SKU Alert above 200 ms p99 (hot SKU)
Outbox oldest unpublished event Alert at 30 s
Reconciliation differences found Daily report; alert on any older than 1 hour
Refunds pending more than 1 hour Alert

Test plan (to build)

Test Proves
500 concurrent checkouts for 50 units of one SKU (Testcontainers) Exactly 50 succeed; stock is 0, never negative
Two checkouts with overlapping SKUs in opposite order, repeated 1,000 times No deadlocks
Same webhook delivered 5 times, out of order with a failure event One confirmation, correct final state
Webhook handler killed mid-transaction, reconciliation run Order confirmed, nothing charged twice
Payment succeeds after reservation expiry, with and without remaining stock Re-reserve or partial refund as designed
Customer cancel and shop accept at the same instant Exactly one transition wins
Consumer killed after side effect, before offset commit Side effect happens once

Load test (to measure)

A k6 scenario mixes normal checkouts with a flash deal (one SKU, 200 units, 5,000 virtual users). To be recorded: checkout p95 and p99, success versus out-of-stock counts (must sum correctly, with zero oversell), lock wait times, and outbox lag. Results will be published only after they have been measured.

8. Decision records

DR-1: Orchestrated saga inside a modular monolith

  • Context: checkout spans inventory, orders and payment, and the payment provider can never join our transaction.
  • Decision: one orchestrator (the checkout saga) issues commands and compensations; in stage 1 the participants are in-process modules sharing one database transaction where possible.
  • Rejected: choreography (each service reacting to others’ events): the workflow becomes implicit and hard to trace; compensation logic scatters. Microservices on day one: turns local transactions into distributed ones for no scaling need at 35 checkouts/s.

DR-2: Reserve stock at checkout, not at payment

  • Decision: reserve when the checkout is created, with a 15-minute expiry.
  • Rejected: decrement on payment success (customers would pay for items already sold out, forcing refunds at scale); reserve when added to cart (hoarding, and most carts are abandoned).

DR-3: Transactional outbox, not dual writes

  • Decision: events are written to the outbox table in the same transaction as the state change, then relayed to Kafka.
  • Rejected: writing to the database and then to Kafka (a crash between the two loses or invents events); Kafka transactions (do not include our PostgreSQL write).

DR-4: One payment per checkout

  • Decision: collect once; refund per shop order.
  • Rejected: a payment per shop (several authorisations for one tap; partial failures leave the customer half paid).

9. Limitations, and implemented versus proposed

Known limitations

  • Payment confirmation is eventually consistent: in the worst case an order is confirmed minutes after payment, by reconciliation.
  • A late payment after expiry can produce a partial order; the customer is refunded automatically but may be disappointed.
  • Flash-deal tokens (stage 5) add a second store whose count must be reconciled with the database after the deal.

Status

Item Status
Requirements, multi-shop rules, capacity, HLD, APIs, deployment Designed here
Class, ER, state and sequence diagrams; reservation algorithm; saga and compensations Designed here
Failure scenarios (payment, webhook, stock race, consumer crash), security, scaling, SLOs Designed here
Decision records, trade-offs, limitations Designed here
Java implementation (checkout saga, inventory, payment with a fake provider, outbox) To build
Concurrency, webhook, reconciliation and crash tests To build
Flash-deal load test To measure