An integration works in a demo when events arrive once and in order. Production needs a more careful contract. A sender may retry after a network interruption, and a later business event may arrive before an earlier one. Your receiving service needs a plan for both.
Separate receipt from business processing
Verify the sender using its documented signature scheme, then durably record the event or enqueue it before acknowledging receipt. Preserve the raw request body when signature verification requires it. Keep slow downstream work out of the acknowledgement path.
Stripe’s webhook documentation explicitly covers duplicate events, retries, and ordering limitations. Check the equivalent guarantees for every provider you integrate with; there is no universal delivery contract that all webhook systems share.
Protect the business operation from repetition
Use a stable event identifier to recognize repeated delivery. Also consider semantic duplicates: two different event records may describe the same business transition. A shipment should not be created twice just because two notifications both indicate that an order is ready.
Store processing state with enough information to resume after a crash. Where possible, commit the state change and its deduplication record atomically. For external side effects, use the destination’s idempotency support or a reconciliation process; a local ‘processed’ flag alone cannot guarantee exactly-once behavior across systems.
Make recovery visible to operators
Provide a view of failed events, the reason for failure, and whether retrying is safe. Test a duplicate, an out-of-order sequence, and a crash after the downstream action succeeds. Add a scheduled reconciliation against the authoritative system so missing notifications can be discovered. A replay tool should preserve traceability and avoid bypassing the same safeguards used during normal delivery.
