UseToolSuite UseToolSuite

Webhooks Explained: Building Reliable Integrations That Don't Break at 3 AM

A practical guide to webhook architecture: HMAC-SHA256 signature verification, avoiding timing attacks, idempotency keys, and asynchronous retry queues.

Necmeddin Cunedioglu Necmeddin Cunedioglu 7 min read
Part of the HTTP Status Codes: The Complete Guide for Developers series

Practice what you learn

Webhook Tester

Try it free →

Webhooks Explained: Building Reliable Integrations That Don’t Break at 3 AM

Here’s a failure mode worth understanding. A Stripe webhook handler assumes every invoice.payment_succeeded event contains a nested discount object. Stripe quietly changes the payload — the field is now null instead of missing — and the handler throws TypeError: Cannot read properties of null. Because the process crashes before returning 200 OK, Stripe assumes delivery failed and starts retrying. Hundreds of payment confirmations get missed before anyone notices.

Webhooks are deceptively simple to get working locally: receive a POST, parse the JSON, write to the database, return 200. Making them reliable, secure, and fault-tolerant in production is the hard part. This guide covers the patterns that prevent those 3 a.m. incidents.

Stop guessing payload structures in production: before connecting your code to a live service, use our local Webhook Tester to build, format, and validate webhook payloads for GitHub, Stripe, Slack, and custom integrations.

1. What Webhooks Actually Are

A webhook is a reverse API. It’s an HTTP POST that an external service (Stripe, GitHub) fires to your server the moment an event happens on their side.

Instead of your server polling an API every few seconds asking “Did anything change?” (wasting CPU and bandwidth), the service pushes the data to you immediately.

Polling (inefficient):
Your Server → "Any new payments?" → Stripe API → "No"
Your Server → "Any new payments?" → Stripe API → "No"
Your Server → "Any new payments?" → Stripe API → "Yes, here is the JSON data."

Webhooks (event-driven):
(A user pays) → Stripe Backend → POST to https://api.your-server.com/webhooks/stripe

The key reality: webhooks are not guaranteed delivery. They’re HTTP requests over the open internet, and HTTP requests fail constantly — your server could be restarting, the connection could drop, your database could time out. Every webhook provider has a retry mechanism, so your handler has to account for duplicate payloads.

2. Receiving Webhooks: A Baseline Express Handler

A minimal endpoint just receives and parses the data.

// A basic Express implementation (no security yet)
app.post('/api/webhooks/stripe', express.raw({ type: 'application/json' }), (req, res) => {
  // Parse the raw body into an object
  const event = JSON.parse(req.body);

  switch (event.type) {
    case 'payment_intent.succeeded':
      handlePaymentSuccess(event.data.object);
      break;
    case 'customer.subscription.deleted':
      handleSubscriptionCanceled(event.data.object);
      break;
    default:
      console.log(`Unhandled event type: ${event.type}`);
  }

  // Respond 200 immediately so the provider stops retrying
  res.status(200).json({ received: true });
});

This works, but deploying it as-is is a security hole. Anyone with the URL could send a fake POST and spoof a payment.

3. Signature Verification (HMAC-SHA256)

Serious webhook providers sign their payloads with an HMAC (Hash-based Message Authentication Code). Skip verification and anyone who finds your endpoint can send fake events — fake payment confirmations, fake account deletions, fake GitHub pushes.

How the signature works

1. The provider computes: HMAC-SHA256(Raw_JSON_Payload_String, Your_Shared_Secret)
2. It sends the payload to your server with the hash in a header (e.g., `Stripe-Signature`).
3. Your server recomputes the same HMAC using the same payload and your stored secret.
4. If your hash matches the header hash, the payload is authentic and came from the provider.

Stripe signature verification (Node.js)

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

app.post('/api/webhooks/stripe',
  // Capture the RAW body. Do NOT use express.json() here.
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signatureHeader = req.headers['stripe-signature'];
    let event;

    try {
      // The Stripe SDK handles the crypto and timing-attack prevention internally
      event = stripe.webhooks.constructEvent(
        req.body, // The raw, unparsed Buffer
        signatureHeader,
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (err) {
      console.error('Signature verification failed:', err.message);
      // Drop the connection with a 400
      return res.status(400).json({ error: 'Invalid signature' });
    }
    
    handleEvent(event);
    res.status(200).json({ received: true });
  }
);

GitHub signature verification (manual)

Without an SDK, do the crypto yourself with Node’s crypto library.

const crypto = require('crypto');

function verifyGitHubSignature(rawPayloadBuffer, signatureHeader, secretKey) {
  // 1. Create the HMAC using the secret
  const hmac = crypto.createHmac('sha256', secretKey);
  
  // 2. Hash the raw payload
  const computedDigest = 'sha256=' + hmac.update(rawPayloadBuffer).digest('hex');
  
  // 3. Use timingSafeEqual to prevent timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(computedDigest),
    Buffer.from(signatureHeader)
  );
}

Timing attacks: always use crypto.timingSafeEqual() for the final comparison, not ===. A normal string comparison fails on the first mismatched character, and an attacker can measure response times to guess the signature one character at a time. timingSafeEqual takes the same time regardless of where the mismatch is.

Serialization: verify the signature against the raw request body buffer, never against re-serialized JSON (JSON.stringify(JSON.parse(req.body))). Serialization changes whitespace and key order, which breaks the byte sequence and makes the HMAC fail every time.

4. Respond Fast, Process Later

Webhook providers enforce timeouts — typically 5–30 seconds. If your handler takes longer than that on a slow database query or PDF generation, the provider drops the connection, assumes a crash, and queues a retry.

The golden pattern: acknowledge immediately (return 200), push the payload onto a queue, and process it asynchronously.

// The route stays fast (< 50ms)
app.post('/api/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const event = verifyAndParse(req);
  if (!event) return res.status(400).send();

  // Push the event onto a queue (Redis/RabbitMQ/Kafka)
  await messageQueue.add('process-stripe-webhook', {
    eventId: event.id,
    type: event.type,
    payload: event,
  });

  // Respond immediately
  res.status(200).json({ received: true });
});

A separate worker (or an SQS-triggered Lambda) pulls events off the queue and does the heavy database writes. If the database goes down, the queue retries the job locally without involving Stripe.

5. Idempotency: Handling Duplicates

Because providers retry on failure, your handler will receive duplicate payloads. If your “Payment Succeeded” handler credits $100 without checking for duplicates, the user eventually gets $200.

The fix is idempotency: design your writes so that running the same operation 100 times has the same result as running it once.

async function processWebhookEvent(event) {
  // 1. Check whether we've already processed this event ID
  const existingRecord = await db.processedEvents.findOne({
    where: { eventId: event.id }
  });

  if (existingRecord) {
    console.log(`Event ${event.id} already processed. Skipping.`);
    return;
  }

  // 2. Run the business logic
  await runBusinessLogic(event);

  // 3. Mark the event as processed to prevent duplicates
  await db.processedEvents.create({
    eventId: event.id,
    processedAt: new Date(),
  });
}

The safer SQL approach: the JavaScript above is vulnerable to a race condition if two duplicate webhooks arrive at the same millisecond. The safest method uses a unique constraint and ON CONFLICT:

-- ❌ Creates duplicate credits if run twice quickly
INSERT INTO user_credits (user_id, amount, reason)
VALUES ('user_123', 100, 'stripe_payment');

-- ✅ Idempotent: can run any number of times safely
INSERT INTO user_credits (user_id, amount, reason, source_event_id)
VALUES ('user_123', 100, 'stripe_payment', 'evt_abc123')
ON CONFLICT (source_event_id) DO NOTHING;

6. Local Development and Tunneling

Your local machine (http://localhost:3000) has no public IP, so Stripe’s servers can’t reach it. You need a tunnel.

Using ngrok

# Start a tunnel to port 3000
ngrok http 3000
# Gives you a public URL like https://abc123xyz.ngrok.io

# Paste https://abc123xyz.ngrok.io/api/webhooks/stripe into the Stripe Dashboard

Testing with cURL

Simulate a webhook locally by firing a test payload at your server with cURL.

curl -X POST http://localhost:3000/api/webhooks/stripe \
  -H "Content-Type: application/json" \
  -d '{
    "id": "evt_test_123",
    "type": "payment_intent.succeeded",
    "data": {
      "object": {
        "id": "pi_test_456",
        "amount": 2000,
        "currency": "usd"
      }
    }
  }'

7. Four Common Mistakes

Mistake 1: hardcoding payload structures

// ❌ Fragile: crashes if `discount` is undefined
const discountAmount = event.data.object.discount.amount;

// ✅ Resilient: optional chaining and a default
const discountAmount = event.data.object?.discount?.amount ?? 0;

Webhook payloads change over time. Providers add fields, make optional fields nullable, and sometimes restructure nested arrays. Use optional chaining and defaults for every deep property access.

Mistake 2: not logging failed payloads

When a handler fails at 3 a.m., you need to know which payload caused it so you can replay it. Log to Datadog or CloudWatch.

try {
  await processStripeEvent(event);
} catch (err) {
  console.error('Webhook processing failed', {
    eventId: event.id,
    error: err.message,
    // Truncate the payload to keep logs reasonable, but keep enough to debug
    payload: JSON.stringify(event).substring(0, 1500), 
  });
  // Still return 200 — we'll retry from our internal queue later
}

Mistake 3: returning 400 for temporary outages

If your database goes down for five minutes and you return 500, Stripe assumes a temporary outage and retries later. If you return 400, Stripe assumes the payload is invalid and stops retrying permanently, which means data loss. Only return 400 for genuinely malformed JSON or failed signatures.

Further Reading


Reliable webhook integrations need careful testing. Use our local Webhook Tester to construct and validate payloads before connecting to live services. The cURL to Code Converter generates the HTTP request logic in your language, and the HMAC Generator lets you test signatures locally.

Necmeddin Cunedioglu
Necmeddin Cunedioglu Author
7 min read
-- views

Software developer and the creator of UseToolSuite. I write about the tools and techniques I use daily as a developer — practical guides based on real experience, not theory.