Skip to main content

Idempotency and Ordering

HealthEx delivers events at least once, in no guaranteed order. Both of those are deliberate: they're what let us retry aggressively enough to survive an outage on either side. The cost is that your handler has to tolerate receiving the same event twice and receiving events out of sequence. Neither is hard to handle, but neither happens by default.

Deduplicate on webhook-id​

The webhook-id header is the identity of an event. Use it as your idempotency key.

webhook-id: evt_01K2YXQ8N3M4P5Q6R7S8T9V0W1

Its guarantees:

  • Stable across retries. All eight delivery attempts for one event carry the same webhook-id.
  • Stable across replays. Manually replaying a delivery from the delivery log reuses the original id, so a correct handler recognizes it as already-processed.
  • Shared across your endpoints. If two of your webhooks both cover a project, one event fans out to both with the same webhook-id. Deduplicate per endpoint, or account for this if your receivers share a store.
  • Unique per event. Two separate withdrawals, even for the same patient a second apart, get different ids.

Record each id you've fully processed, and short-circuit on a repeat. Keep the record for at least 33 hours to cover the full retry window; 30 days matches delivery-log retention and means a manual replay is also caught.

Don't Deduplicate on Payload Contents

Hashing the body, or keying on patientId plus projectId, will silently drop legitimate events — a patient who withdraws, re-grants, and withdraws again produces two genuinely distinct withdrawals with identical-looking data. Key on webhook-id and nothing else.

Why Duplicates Happen​

Worth knowing, because the most common cause is invisible from your side:

  • Your acknowledgement didn't reach us. Your handler ran, succeeded, and returned 2xx — but the connection dropped or exceeded the 10-second timeout before we saw it. From our side that's a failure, so we retry. This is the usual cause, and no amount of care on your end eliminates it.
  • A partial failure. Your handler did its work and then errored before responding.
  • A manual replay. Someone replayed the delivery from the Admin console.
  • Fan-out. Two of your webhooks both matched the event.

Using deliveryAttempt​

The envelope's deliveryAttempt is 1 on the first delivery and increments on each retry.

{
  "type": "consent.withdrawn",
  "deliveryAttempt": 3
}

It's useful for logging and alerting — a spike in deliveries arriving with deliveryAttempt above 1 tells you your endpoint is intermittently failing, which is otherwise easy to miss when the retry eventually succeeds.

Not an Idempotency Key

deliveryAttempt tells you which attempt this is, not which event. It resets to 1 for every new event, so it can't identify a duplicate. Two different events both arrive with deliveryAttempt: 1. Deduplicate on webhook-id.

Ordering Is Not Guaranteed​

Each delivery has its own independent retry chain, so a retried older event can arrive after a newer one that succeeded first. Consider a patient who withdraws consent, re-grants it, and withdraws again over the course of an hour, while your endpoint is briefly unhealthy:

Your endpoint receivesEnvelope timestamp
Second withdrawal, first attempt2026-08-19T15:10:00.000Z
First withdrawal, fourth attempt2026-08-19T14:32:07.421Z

Processing these in arrival order leaves you holding the older state. Order on the envelope's timestamp, never on arrival time, and discard an event that is older than the state you already have for that patient and project.

The practical rule: store the timestamp of the last event you applied per (patientId, projectId), and ignore anything older. For consent.withdrawn you can also lean on the event's semantics — withdrawal is a terminal state for that authorization, so applying it twice is harmless, and a stale withdrawal arriving after a fresh grant is the one case the timestamp check has to catch.

A Safe Handler Shape​

Five steps, in this order:

  1. Read the raw request body, before any JSON parsing.
  2. Verify the signature. Reject with 400 if it fails.
  3. Check webhook-id against your processed-events store. If it's there, return 2xx and stop.
  4. Persist the event and return 2xx immediately.
  5. Process it asynchronously, ordering on timestamp, and mark the id processed when you're done.
app.post("/webhooks/healthex", express.raw({ type: "application/json" }), async (req, res) => {
    let event
    try {
        event = wh.verify(req.body, {
            "webhook-id": req.header("webhook-id"),
            "webhook-timestamp": req.header("webhook-timestamp"),
            "webhook-signature": req.header("webhook-signature"),
        })
    } catch {
        return res.sendStatus(400)
    }

    const eventId = req.header("webhook-id")

    // Atomic insert-if-absent. A duplicate is an acknowledged no-op.
    const isNew = await events.claim(eventId, event)
    if (!isNew) {
        return res.sendStatus(204)
    }

    // Acknowledge before doing the work, so a slow downstream can't trigger a retry.
    res.sendStatus(204)

    try {
        await applyIfNewer(event)
        await events.markProcessed(eventId)
    } catch (error) {
        // Leave it unprocessed for your own retry — we've already been acknowledged.
        logger.error({ eventId, error }, "webhook processing failed")
    }
})

Two details in there that matter:

  • Step 3 and 4 are one atomic operation. claim should be an insert-if-absent — a unique constraint on the event id, or a conditional write. Checking for existence and then inserting as two separate statements leaves a race that two concurrent retries will find.
  • Once you return 2xx, the event is yours. We won't retry it, so a failure after that point needs your own retry path. If you'd rather we retried instead, wait to respond until your work completes, then return success (2xx) or failure (usually 5xx). However, in this case, your work needs to take less than 10 seconds, which can be tricky to guarantee.

Next Steps​