Notification Platform

This design builds on production experience moving SMS, email and WhatsApp notification flows onto Kafka. The case study itself is a fresh, general design; nothing here describes a specific employer’s system.

1. Problem and requirements

Internal services (orders, payments, claims, marketing) call one platform to notify people. The platform owns templates, preferences, provider integrations, retries and delivery tracking, so no product team writes its own SMS client again.

Functional requirements

Non-functional requirements

The delivery guarantee, stated precisely

The platform guarantees at-least-once processing of every accepted request, with idempotent sending so that our own retries do not produce duplicates. It cannot guarantee exactly-once delivery to a person. If a worker crashes after a provider has accepted a message but before we record that, we cannot know whether the message went out. For providers that accept an idempotency reference, the retry is de-duplicated by the provider. For those that do not, a rare duplicate is possible and is preferred over a lost OTP. This trade-off is written into the API contract, so callers are not surprised.

Out of scope

In-app inbox, two-way conversational messaging, A/B testing of templates, and the marketing segmentation that decides who receives a campaign (the platform receives already-resolved recipients).

2. Capacity estimation

Assumptions: 50 million notifications a day; a bulk campaign can push intake to 10× the average for an hour; 3 status events per notification (accepted by provider, delivered, opened or failed); a stored notification with its rendered payload is about 1 KB; status events about 150 bytes.

What this says: intake must be faster than delivery, because providers cap how fast we can send. A campaign therefore builds a backlog by design, and the system must keep that backlog durable, drain it at exactly the provider’s rate, and keep critical traffic out of it. Kafka’s throughput is not the constraint; partitioning, priority isolation and provider quotas are.

3. Architecture and APIs

High-level architecture

flowchart LR
  S[Calling services] -->|POST /notifications| API[Intake API]
  API --> PG[(PostgreSQL: notifications, status, outbox)]
  PG --> OB[Outbox relay]
  OB --> T1[[notify.requested]]
  T1 --> RT[Router: preferences, templates, dedup, quotas]
  RT --> C1[[sms.critical]]
  RT --> C2[[sms.standard]]
  RT --> C3[[email.*]]
  RT --> C4[[whatsapp.*]]
  RT --> C5[[push.*]]
  C1 --> W1[SMS workers]
  C2 --> W1
  W1 --> PA1[Provider adapters: SMS A / SMS B]
  C3 --> W2[Email workers] --> PA2[Email adapters]
  C4 --> W3[WhatsApp workers] --> PA3[WhatsApp adapter]
  C5 --> W4[Push workers] --> PA4[FCM / APNs adapters]
  W1 -->|transient failure| RQ[[retry.1m / 10m / 1h]]
  RQ --> W1
  W1 -->|permanent or exhausted| DLQ[[dlq]]
  PA1 -->|webhooks| CB[Callback API]
  CB --> PG
  SCH[Scheduler] --> PG
  RL[(Redis: quotas, dedup keys)] --- RT
  RL --- W1

API contracts

POST /notifications
Authorization: Bearer <service token for team "orders">
Idempotency-Key: order-77812-shipped

{
  "recipient": { "userId": "u_4821" },
  "category": "ORDER_UPDATE",
  "priority": "TRANSACTIONAL",
  "channels": ["PUSH", "SMS"],
  "fallback": "SEQUENTIAL",
  "template": { "id": "order-shipped", "version": 3, "locale": "en-IN",
                "vars": { "orderId": "77812", "eta": "Thursday" } },
  "sendAt": null,
  "expiresAt": "2026-10-23T18:00:00Z"
}

202 Accepted
{ "id": "ntf_01JB8…", "status": "ACCEPTED", "idempotencyKey": "order-77812-shipped" }

• fallback: SEQUENTIAL means: try push; if push is not delivered within its window, send SMS. ALL sends on every listed channel.
• expiresAt stops stale messages: an OTP that is 10 minutes old is dropped, not sent.

Deployment

flowchart LR
  subgraph K8s [Kubernetes]
    I[intake-api x 3, HPA on RPS]
    R[router x 3]
    WS[sms-workers, HPA on lag]
    WE[email-workers, HPA on lag]
    WW[whatsapp-workers]
    WP[push-workers]
    CBK[callback-api x 2]
    RL[outbox-relay x 2, leader-elected]
  end
  subgraph Data
    KF[[Kafka: 3 brokers, RF 3]]
    PG[(PostgreSQL primary + standby)]
    RD[(Redis)]
  end
  I --> PG
  RL --> PG
  RL --> KF
  R --> KF
  WS --> KF
  CBK --> PG

Workers autoscale on consumer lag, capped by the partition count and by the provider quota: adding workers beyond the quota only adds rejected calls.

4. Internal design and workflows

Class diagram

classDiagram
  class NotificationRequest {
    +id
    +tenant
    +recipient
    +category
    +priority Priority
    +channels List~Channel~
    +template TemplateRef
    +sendAt
    +expiresAt
    +idempotencyKey
  }
  class Delivery {
    +notificationId
    +channel
    +attempt int
    +state DeliveryState
    +providerId
    +providerMessageId
    +transition(event) Delivery
  }
  class NotificationTemplate {
    +id
    +version
    +channel
    +locale
    +render(vars) RenderedMessage
  }
  class PreferenceService {
    +allowed(recipient, channel, category, now) Decision
  }
  class NotificationRouter {
    +route(NotificationRequest) List~Delivery~
  }
  class NotificationDispatcher {
    +dispatch(Delivery)
  }
  class ChannelProvider {
    <<interface>>
    +channel() Channel
    +send(RenderedMessage, idempotencyRef) SendResult
    +supportsIdempotency() bool
  }
  class TwilioSmsAdapter
  class SecondarySmsAdapter
  class SesEmailAdapter
  class WhatsAppCloudAdapter
  class FcmPushAdapter
  class ProviderSelector {
    +select(channel, delivery) ChannelProvider
  }
  class RetryPolicy {
    +next(attempt, SendResult) RetryDecision
  }
  class DeliveryTracker {
    +record(delivery, event)
    +applyCallback(providerMessageId, status, at)
  }
  class QuotaGate {
    +acquire(provider) bool
  }
  NotificationRouter --> PreferenceService
  NotificationRouter --> NotificationTemplate
  NotificationDispatcher --> ProviderSelector
  NotificationDispatcher --> QuotaGate
  NotificationDispatcher --> RetryPolicy
  NotificationDispatcher --> DeliveryTracker
  ProviderSelector --> ChannelProvider
  ChannelProvider <|.. TwilioSmsAdapter
  ChannelProvider <|.. SecondarySmsAdapter
  ChannelProvider <|.. SesEmailAdapter
  ChannelProvider <|.. WhatsAppCloudAdapter
  ChannelProvider <|.. FcmPushAdapter
  DeliveryTracker --> Delivery

Provider names are examples of the adapter pattern, not a recommendation of specific vendors.

Delivery state machine

stateDiagram-v2
  [*] --> ACCEPTED
  ACCEPTED --> SUPPRESSED: opted out, quiet hours, storm limit
  ACCEPTED --> SCHEDULED: sendAt in the future
  SCHEDULED --> QUEUED: due
  ACCEPTED --> QUEUED: routed
  QUEUED --> SENDING: worker claims
  SENDING --> SENT: provider accepted
  SENDING --> RETRY_WAIT: transient failure
  RETRY_WAIT --> QUEUED: delay elapsed
  SENDING --> FAILED: permanent failure or retries exhausted
  SENT --> DELIVERED: provider callback
  SENT --> UNDELIVERED: provider callback (bounced, rejected)
  QUEUED --> EXPIRED: expiresAt passed
  RETRY_WAIT --> EXPIRED: expiresAt passed
  ACCEPTED --> CANCELLED: caller cancels
  QUEUED --> CANCELLED: caller cancels
  DELIVERED --> [*]
  UNDELIVERED --> [*]
  FAILED --> [*]
  SUPPRESSED --> [*]
  EXPIRED --> [*]
  CANCELLED --> [*]

Transitions are applied with a conditional update, so only legal moves succeed and a late event cannot drag a delivery backwards:

UPDATE delivery
SET state = :to, updated_at = :at, provider_message_id = coalesce(:pmid, provider_message_id)
WHERE id = :id AND state = ANY(:allowedFrom) AND updated_at <= :at;

The rank of each state (SENT < DELIVERED) plus the event timestamp lets an out-of-order callback (a SENT report arriving after DELIVERED) be recorded in history but ignored for the current state.

Data model and indexes

erDiagram
  NOTIFICATION ||--|{ DELIVERY : "one per channel"
  DELIVERY ||--o{ DELIVERY_EVENT : history
  NOTIFICATION ||--o| OUTBOX : "published via"
  TEMPLATE ||--o{ NOTIFICATION : renders
  RECIPIENT_PREFERENCE }o--|| RECIPIENT : of
  NOTIFICATION {
    uuid id PK
    text tenant
    text idempotency_key
    text recipient_id
    text category
    text priority
    jsonb payload
    timestamptz send_at
    timestamptz expires_at
    timestamptz created_at
  }
  DELIVERY {
    uuid id PK
    uuid notification_id FK
    text channel
    text state
    int attempt
    text provider
    text provider_message_id
    timestamptz updated_at
  }
  DELIVERY_EVENT {
    bigint id PK
    uuid delivery_id FK
    text event
    jsonb detail
    timestamptz at
  }
  OUTBOX {
    bigint id PK
    text topic
    text key
    jsonb payload
    timestamptz created_at
    timestamptz published_at
  }
  TEMPLATE {
    text id PK
    int version PK
    text channel
    text locale
    text body
  }
  RECIPIENT_PREFERENCE {
    text recipient_id PK
    text channel PK
    text category PK
    bool allowed
    text quiet_hours
  }
  RECIPIENT {
    text id PK
    text phone
    text email
    text locale
  }

-- Idempotent intake: one notification per caller and key
CREATE UNIQUE INDEX ntf_idem ON notification (tenant, idempotency_key);
-- Provider callbacks find their delivery
CREATE UNIQUE INDEX dlv_provider_msg ON delivery (provider, provider_message_id);
-- Scheduler: due work only
CREATE INDEX ntf_due ON notification (send_at) WHERE send_at IS NOT NULL;
-- Relay: unpublished outbox rows in order
CREATE INDEX outbox_pending ON outbox (id) WHERE published_at IS NULL;

notification, delivery and delivery_event are range-partitioned by day. Partitions older than 30 days are exported to object storage (Parquet) and dropped, which is far cheaper than DELETE.

Kafka topology

flowchart LR
  REQ[[notify.requested · 48 partitions · key recipientId]] --> RT[router group]
  RT --> SC[[sms.critical · 12p]]
  RT --> SS[[sms.standard · 24p]]
  RT --> SB[[sms.bulk · 24p]]
  SC --> W[sms-workers group]
  SS --> W
  SB --> W
  W -->|transient| R1[[sms.retry.1m]]
  W -->|2nd failure| R2[[sms.retry.10m]]
  W -->|3rd failure| R3[[sms.retry.1h]]
  R1 --> RW[retry consumer: waits until due, then republishes to sms.standard]
  R2 --> RW
  R3 --> RW
  W -->|permanent / exhausted / poison| D[[sms.dlq]]
  D --> OPS[DLQ console: inspect, fix, replay]

• Partition key recipientId: messages to one person stay in order (the “order placed” SMS does not arrive after “order shipped”), and storm suppression for one person happens in one partition.
• Priority by topic, not by message field: workers poll critical first with dedicated threads; a bulk backlog of millions sits in sms.bulk and cannot delay an OTP. Kafka has no priority within a partition, so separate topics are the only reliable way.
• Retry topics with fixed delays: a failed message moves to a delay topic and the main partition keeps flowing. Blocking retries inside the main consumer would stall every message behind the failing one.
• Commit after recording: a worker commits the offset only after the delivery state is written. A crash before that replays the message, and the state machine plus idempotency reference prevent a second send.

Dispatch sequence

sequenceDiagram
  participant K as sms.standard
  participant W as SMS worker
  participant D as PostgreSQL
  participant Q as QuotaGate (Redis)
  participant P as Provider adapter
  participant X as SMS provider
  K->>W: Delivery d (attempt 1)
  W->>D: UPDATE delivery SET state='SENDING' WHERE id=d AND state='QUEUED'
  alt 0 rows (already sending or sent)
    W->>K: commit offset, skip
  else claimed
    W->>Q: acquire(provider A)
    alt no token
      W->>K: pause partition briefly, retry acquire (backpressure)
    end
    W->>P: send(message, ref = d.id + attempt)
    P->>X: HTTP send (timeout 3 s, client reference = ref)
    alt accepted
      X-->>P: messageId
      W->>D: state SENT, provider_message_id
    else transient (timeout, 5xx, 429)
      W->>D: state RETRY_WAIT, attempt+1
      W->>K: produce to sms.retry.1m
    else permanent (invalid number, blocked)
      W->>D: state FAILED, reason
    end
    W->>K: commit offset
  end
  X--)W: later: webhook DELIVERED (via Callback API)

Retry policy

RetryDecision next(int attempt, SendResult r) {
    if (r.permanent()) return RetryDecision.fail(r.reason());              // bad number, unsubscribed
    if (attempt >= 4)  return RetryDecision.deadLetter("retries exhausted");
    Duration base = List.of(Duration.ofMinutes(1), Duration.ofMinutes(10), Duration.ofHours(1)).get(attempt - 1);
    long jitterMs = ThreadLocalRandom.current().nextLong(base.toMillis() / 5);   // up to +20%
    return RetryDecision.after(base.plusMillis(jitterMs), r.retryAfter());    // honour provider Retry-After if larger
}

Critical notifications use a shorter schedule (5 s, 15 s, 30 s) and switch provider on the first failure, because an OTP is worthless after a few minutes.

Storm suppression and deduplication

Two different problems with two different keys:

Transaction boundaries

5. Failure modes and consistency

Consistency summary

• Strong (single database transaction): intake, idempotency, each state transition.
• Eventual: the Kafka pipeline from intake to send; status from provider callbacks.
• Never assumed: that Kafka delivers once, that a provider call that timed out failed, or that callbacks arrive in order.

Security and abuse

• Callers authenticate with service credentials; each caller (tenant) has quotas and an allow-list of templates and categories.
• Templates are reviewed and versioned; callers pass variables, never free-form message bodies, which stops the platform being used for arbitrary spam or phishing.
• Phone numbers and emails are personal data: encrypted at rest, masked in logs and traces, deleted or anonymised after retention.
• Opt-out (STOP replies, unsubscribe links) is processed as a preference update with the highest priority.

6. Scaling evolution

7. Operations and validation

Metrics and SLOs

Traces carry the notification ID from intake through router, worker, adapter and callback, so one ID shows the whole life of a message.

Test plan (to build)

Load test (to measure)

Using fake providers with configurable latency, error rate and quotas: accepted requests per second, delivery throughput per channel, accept-to-sent p95 and p99 per priority, consumer lag over time, and behaviour during a simulated 5-minute provider outage. These will be charted and labelled as measured only after the runs exist.

8. Decision records

DR-1: Kafka, introduced at stage 2, not on day one

• Decision: start with a PostgreSQL-backed job queue; move to Kafka with an outbox when independent consumers, replay or per-channel scaling are needed.
• Rejected for day one: Kafka from the start (operational weight with no benefit below a few hundred messages a second). Rejected for later: RabbitMQ (good for task queues, but replay and multiple independent consumers of the same stream are first-class in Kafka).

DR-2: Priority by topic

• Decision: separate topics per channel and priority; workers drain critical topics first with dedicated capacity.
• Rejected: a priority field on one topic (Kafka partitions are FIFO, so a bulk backlog still blocks critical messages behind it).

DR-3: Non-blocking retries through delay topics

• Decision: failed messages move to retry.1m, retry.10m, retry.1h, then the DLQ.
• Rejected: retrying inside the consumer (one bad provider stalls the whole partition); Thread.sleep in a consumer (triggers rebalances).

9. Limitations, and implemented versus proposed

Known limitations

• Exactly-once delivery to a person is impossible; a rare duplicate is possible with providers that lack idempotency support.
• Per-recipient ordering holds within one channel, not across channels (push and SMS can arrive in either order).
• Delivery and open tracking depend on providers; email opens in particular are unreliable signals.

Status