Webhooks Explained: Building Reliable Integrations That Don't Break at 3 AM

Three months into my last job, our Stripe webhook handler crashed at 2 AM because someone deployed a code change that assumed every invoice.payment_succeeded event had a discount field. Stripe had changed their payload structure slightly, the field was null instead of missing, and our handler threw an unhandled TypeError. We missed 847 payment confirmations before the on-call engineer woke up, saw the alerts, and pushed a fix.

Webhooks are deceptively simple to get working — receive a POST, parse the JSON, do something. Making them reliable is the hard part. This guide covers the patterns that prevent those 3 AM incidents.

What Webhooks Are (and Aren’t)

A webhook is an HTTP POST request that a service sends to your server when something happens. Instead of polling an API every 30 seconds asking “did anything change?”, the service tells you immediately.

Traditional polling:
Your Server → "Any new payments?" → Stripe API → "No"
Your Server → "Any new payments?" → Stripe API → "No"
Your Server → "Any new payments?" → Stripe API → "Yes, here's one"

Webhooks:
(payment happens) → Stripe → POST to your-server.com/webhooks/stripe

Webhooks are not guaranteed delivery. They’re HTTP requests, and HTTP requests fail — your server could be down, the network could drop the connection, your handler could timeout. Every webhook system has a retry mechanism, and your handler needs to account for duplicates.

Receiving Webhooks: The Basics

A minimal webhook endpoint:

// Express.js
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), (req, res) => {
  const event = JSON.parse(req.body);

  switch (event.type) {
    case 'payment_intent.succeeded':
      handlePaymentSuccess(event.data.object);
      break;
    case 'customer.subscription.deleted':
      handleSubscriptionCanceled(event.data.object);
      break;
  }

  // Always respond with 200 quickly
  res.status(200).json({ received: true });
});

# Flask
@app.route('/webhooks/stripe', methods=['POST'])
def stripe_webhook():
    event = request.get_json()

    if event['type'] == 'payment_intent.succeeded':
        handle_payment_success(event['data']['object'])
    elif event['type'] == 'customer.subscription.deleted':
        handle_subscription_canceled(event['data']['object'])

    return jsonify({'received': True}), 200

Build test payloads: Before connecting to a live service, use our Webhook Tester to build and validate webhook payloads for GitHub, Stripe, Slack, and custom integrations. Getting the payload structure right before going live saves debugging time.

Signature Verification: Don’t Skip This

Every serious webhook provider signs their payloads with HMAC. Skipping verification means anyone who discovers your webhook URL can send fake events — fake payment confirmations, fake account deletions, anything.

How Webhook Signatures Work

1. Service computes HMAC-SHA256(payload_body, shared_secret)
2. Service sends the payload with the signature in a header
3. You recompute the HMAC with the same secret
4. If the signatures match, the payload is authentic

Stripe Signature Verification

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

app.post('/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const sig = req.headers['stripe-signature'];

    try {
      const event = stripe.webhooks.constructEvent(
        req.body,            // raw body — NOT parsed JSON
        sig,
        process.env.STRIPE_WEBHOOK_SECRET
      );
      handleEvent(event);
      res.status(200).json({ received: true });
    } catch (err) {
      console.error('Signature verification failed:', err.message);
      res.status(400).json({ error: 'Invalid signature' });
    }
  }
);

GitHub Signature Verification

const crypto = require('crypto');

function verifyGitHubSignature(payload, signature, secret) {
  const hmac = crypto.createHmac('sha256', secret);
  const digest = 'sha256=' + hmac.update(payload).digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(digest),
    Buffer.from(signature)
  );
}

app.post('/webhooks/github', express.raw({ type: 'application/json' }), (req, res) => {
  const signature = req.headers['x-hub-signature-256'];

  if (!verifyGitHubSignature(req.body, signature, process.env.GITHUB_WEBHOOK_SECRET)) {
    return res.status(401).json({ error: 'Invalid signature' });
  }

  const event = JSON.parse(req.body);
  // Process event...
  res.status(200).json({ received: true });
});

Critical: Always use crypto.timingSafeEqual() for signature comparison, not ===. Regular string comparison is vulnerable to timing attacks — an attacker can guess the correct signature one character at a time by measuring response times.

Also critical: Verify the signature against the raw request body, not the parsed-and-re-serialized JSON. JSON serialization doesn’t preserve the exact byte sequence, so the HMAC won’t match.

Respond Fast, Process Later

Webhook providers have timeout limits — typically 5–30 seconds. If your handler takes longer than that, the provider treats it as a failure and retries.

The pattern: acknowledge the webhook immediately, then process it asynchronously.

app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  // Verify signature
  const event = verifyAndParse(req);
  if (!event) return res.status(400).send();

  // Store the event for processing
  await db.webhookEvents.insert({
    provider: 'stripe',
    eventId: event.id,
    type: event.type,
    payload: event,
    status: 'pending',
    receivedAt: new Date(),
  });

  // Respond immediately
  res.status(200).json({ received: true });
});

// Separate worker processes the queue
async function processWebhookQueue() {
  const events = await db.webhookEvents.findAll({
    where: { status: 'pending' },
    order: [['receivedAt', 'ASC']],
  });

  for (const event of events) {
    try {
      await processEvent(event);
      await event.update({ status: 'processed' });
    } catch (err) {
      await event.update({
        status: 'failed',
        error: err.message,
        retryCount: event.retryCount + 1,
      });
    }
  }
}

Idempotency: Handle Duplicates Gracefully

Webhook providers retry on failure, network issues, or even their own bugs. Your handler will receive duplicate events. If your “payment succeeded” handler credits an account without checking for duplicates, users get double credits.

The fix: store event IDs and check before processing.

async function processEvent(event) {
  // Check if we've already processed this event
  const existing = await db.processedEvents.findOne({
    where: { eventId: event.id }
  });

  if (existing) {
    console.log(`Event ${event.id} already processed, skipping`);
    return;
  }

  // Process the event
  await handleEvent(event);

  // Mark as processed
  await db.processedEvents.create({
    eventId: event.id,
    processedAt: new Date(),
  });
}

The key insight: idempotency isn’t just about rejecting duplicates. Every database write should use upserts or conditional updates:

-- WRONG: creates duplicate credits
INSERT INTO credits (user_id, amount, reason)
VALUES ('user_123', 100, 'payment');

-- CORRECT: idempotent — can run multiple times safely
INSERT INTO credits (user_id, amount, reason, event_id)
VALUES ('user_123', 100, 'payment', 'evt_abc123')
ON CONFLICT (event_id) DO NOTHING;

Retry Behavior by Provider

Each provider has different retry strategies. Know yours:

Provider	Retry Attempts	Retry Schedule	Timeout
Stripe	Up to 3 days	Exponential backoff	20 seconds
GitHub	1 redeliver	Manual retry via UI	10 seconds
Slack	3 attempts	30 min intervals	3 seconds
Twilio	Up to 48 hours	Exponential backoff	15 seconds
SendGrid	Up to 72 hours	Exponential backoff	5 seconds

What triggers retries:

Your server returns 5xx — provider retries (temporary failure)
Your server returns 4xx — most providers don’t retry (permanent failure)
Timeout — provider retries (assumed temporary failure)
Connection refused — provider retries (your server is down)

This means: Always return 200 for successfully received (even if processing fails later). Only return 4xx if the payload is genuinely invalid.

Testing Webhooks in Development

Local Development with Tunnels

Your local machine doesn’t have a public URL. Use a tunnel:

# ngrok
ngrok http 3000
# → Gives you https://abc123.ngrok.io

# Then configure the webhook URL in the provider:
# https://abc123.ngrok.io/webhooks/stripe

Manual Testing with cURL

Build a test payload and send it directly:

curl -X POST http://localhost:3000/webhooks/stripe \
  -H "Content-Type: application/json" \
  -d '{
    "id": "evt_test_123",
    "type": "payment_intent.succeeded",
    "data": {
      "object": {
        "id": "pi_test_456",
        "amount": 2000,
        "currency": "usd",
        "status": "succeeded"
      }
    }
  }'

Use the Webhook Tester to build realistic payloads with the correct structure, then send them with the cURL to Code Converter in your preferred language.

Stripe CLI for Local Testing

# Forward Stripe events to your local server
stripe listen --forward-to localhost:3000/webhooks/stripe

# Trigger a specific event
stripe trigger payment_intent.succeeded

Common Mistakes

Mistake 1: Parsing Body Before Verification

// WRONG — body is parsed/re-serialized, signature won't match
app.post('/webhooks', express.json(), (req, res) => {
  verifySignature(JSON.stringify(req.body), sig); // ← different bytes!
});

// CORRECT — use raw body for verification
app.post('/webhooks', express.raw({ type: 'application/json' }), (req, res) => {
  verifySignature(req.body, sig); // ← exact bytes as sent
});

Mistake 2: Doing Heavy Processing Synchronously

// WRONG — might timeout
app.post('/webhooks', async (req, res) => {
  await sendEmail(event);           // 2 seconds
  await updateDatabase(event);       // 1 second
  await notifySlack(event);          // 3 seconds
  await generateInvoicePDF(event);   // 10 seconds ← timeout!
  res.status(200).send();
});

// CORRECT — queue and respond
app.post('/webhooks', async (req, res) => {
  await queue.add('process-webhook', event);
  res.status(200).send(); // Responds in milliseconds
});

Mistake 3: Not Logging Webhook Failures

When a webhook handler fails at 3 AM, you need to know exactly what payload caused the failure. Log everything:

try {
  await processEvent(event);
} catch (err) {
  console.error('Webhook processing failed', {
    eventId: event.id,
    eventType: event.type,
    error: err.message,
    stack: err.stack,
    payload: JSON.stringify(event).substring(0, 1000), // truncate for log size
  });
  // Still return 200 — we'll retry from our own queue
}

Mistake 4: Hardcoding Payload Structure

// FRAGILE — crashes if discount is undefined
const discount = event.data.object.discount.amount;

// RESILIENT — handles missing fields gracefully
const discount = event.data.object?.discount?.amount ?? 0;

Webhook payloads change over time. Providers add fields, make optional fields nullable, and sometimes restructure nested objects. Use optional chaining and default values for every field access.