Webhook Monitoring: Ensure Your Webhooks Never Fail

# Django / DRF
@api_view(['POST'])
def stripe_webhook(request):
    payload = request.body
    
    # Verify signature (Stripe требует это, иначе откроется XSS)
    sig_header = request.META.get('HTTP_STRIPE_SIGNATURE')
    try:
        event = stripe.Webhook.construct_event(
            payload, sig_header, STRIPE_ENDPOINT_SECRET
        )
    except (ValueError, stripe.error.SignatureVerificationError):
        return Response(
            {"error": "Invalid signature"},
            status=400
        )
    
    # Store raw event in DB immediately
    # Before any processing that might fail
    WebhookEvent.objects.create(
        event_id=event['id'],
        event_type=event['type'],
        payload=event,
        status='pending'
    )
    
    # Queue async processing
    process_webhook_event.delay(event['id'])
    
    # Return 202 immediately
    # Tell Stripe: event received, will process async
    return Response({"status": "received"}, status=202)

# Background worker
def process_webhook_event(event_id):
    event = WebhookEvent.objects.get(id=event_id)
    
    # Detect duplicate: webhook_id + type should be unique
    # Stripe sends same webhook multiple times if we reply slowly
    webhook_idempotency_key = f"{event.event_id}-{event.event_type}"
    
    # Check if already processed
    if ProcessedWebhook.objects.filter(
        webhook_id=event.event_id
    ).exists():
        print(f"Webhook {event.event_id} already processed, skipping")
        return
    
    try:
        if event.event_type == 'payment_intent.succeeded':
            handle_payment_success(event.payload)
        elif event.event_type == 'payment_intent.payment_failed':
            handle_payment_failure(event.payload)
        elif event.event_type == 'customer.subscription.updated':
            handle_subscription_update(event.payload)
        
        # Mark as processed ONLY after success
        ProcessedWebhook.objects.create(webhook_id=event.event_id)
        event.status = 'processed'
        event.save()
    except Exception as e:
        # Log error, add to dead letter queue
        # Don't update status to 'processed'
        event.status = 'failed'
        event.error = str(e)
        event.save()
        
        # Alert the team
        send_alert(f"Webhook processing failed: {event.event_id}")
        
        # Add to retry queue with exponential backoff
        process_webhook_event.apply_async(
            args=[event_id],
            countdown=300  # retry after 5 minutes
        )

# DLQ model
class WebhookDeadLetter(models.Model):
    webhook_id = models.CharField(max_length=255, unique=True)
    event_type = models.CharField(max_length=100)
    payload = models.JSONField()
    error_message = models.TextField()
    retry_count = models.IntegerField(default=0)
    last_retry_at = models.DateTimeField(null=True)
    created_at = models.DateTimeField(auto_now_add=True)

# In exception handler:
if event.status_failed_count > 3:  # Failed 3+ times
    WebhookDeadLetter.objects.create(
        webhook_id=event.event_id,
        event_type=event.event_type,
        payload=event.payload,
        error_message=str(e),
        retry_count=3
    )
    # Send alert to ops team
    alert_dlq(f"Webhook {event.event_id} moved to DLQ")

Конфигурация в AtomPing:

Тип: Heartbeat

URL: https://api.yourapp.com/health/last-webhook-heartbeat

Интервал мониторинга: 30 секунд

Expected interval: 120 секунд (webhook heartbeat должен приходить каждые 60s, допуск 2x)

Endpoint должен вернуть timestamp последнего heartbeat'а

GET /health/webhooks

{
  "status": "healthy",
  "last_webhook_received_at": "2026-03-26T10:28:00Z",
  "pending_webhooks": 0,
  "failed_webhooks": 0,
  "dlq_count": 0,
  "worker_alive": true,
  "metrics": {
    "processed_last_hour": 42,
    "errors_last_hour": 1,
    "avg_processing_time_ms": 234
  }
}

Assertions:

✓ $.status equals healthy

✓ $.pending_webhooks equals 0

✓ $.worker_alive equals true

✓ $.dlq_count lt 5 (alert if more than 5 in DLQ)

✓ Проверить полный путь: handler → queue → worker → БД

✓ Измерить latency end-to-end

✓ Убедиться что идемпотентность работает (test webhook приходит дважды)

# Celery Beat task (runs every hour)
@periodic_task(run_every=crontab(minute=0))
def send_test_webhook():
    webhook_event = {
        'type': 'test_webhook',
        'timestamp': now(),
        'data': {'marker': 'internal_test'}
    }
    
    # POST to your own webhook endpoint
    response = requests.post(
        f"{INTERNAL_API_URL}/webhooks/stripe",
        json=webhook_event,
        timeout=10
    )
    
    # Check if received and processed
    processed = ProcessedWebhook.objects.filter(
        webhook_id=webhook_event['id']
    ).exists()
    
    if not processed or response.status_code != 202:
        alert("Test webhook failed", 
              f"Status: {response.status_code}, Processed: {processed}")

Попытка 1: immediately

Попытка 2: 5 минут спустя

Попытка 3: 30 минут спустя

Попытка 4: 2 часа спустя

Попытка 5: 5 часов спустя

Max: 3 дня, потом Stripe даёт up

1. Получили webhook → сохранили в БД → вернули 202

2. Queued обработку на рабочий процесс

3. Рабочий пробует обработать

4. Если fails → exponential backoff в queue (5 min → 30 min → 2 hours)

5. После N failures → Dead Letter Queue

class WebhookAuditLog(models.Model):
    webhook_id = models.CharField(max_length=255)
    received_at = models.DateTimeField(auto_now_add=True)
    processing_started_at = models.DateTimeField(null=True)
    processing_completed_at = models.DateTimeField(null=True)
    status = models.CharField(  # received, processing, success, failed
        max_length=20,
        choices=[...])
    http_status = models.IntegerField()
    payload_hash = models.CharField(max_length=64)  # SHA256
    response_time_ms = models.IntegerField()
    error_message = models.TextField(null=True)
    retry_count = models.IntegerField(default=0)
    
    class Meta:
        indexes = [
            models.Index(fields=['webhook_id']),
            models.Index(fields=['status']),
            models.Index(fields=['received_at']),
        ]

✓ Ваш endpoint доступен (DNS разрешается, firewall открыт)

✓ Signature verification работает

✓ Payload распарсился правильно

✓ Webhook залогирован и виден в вашем логов

ngrok — expose локальный server на интернет. Используйте для local development webhook'ов

Stripe CLI — локальный Stripe webhook emulator. stripe listen --forward-to localhost:8000/webhooks

curl — manual testing. Отправьте сырой webhook'ом себе, проверьте handling

# Load testing with Apache Bench
ab -n 1000 -c 50 -p webhook.json \\
   -H "Content-Type: application/json" \\
   https://api.yourapp.com/webhooks/stripe

# Results:
# Requests per second: 95.2
# Avg response time: 52ms
# 95th percentile: 120ms
# Success: 100%

Архитектура: HTTP 202 immediately, async processing, separate queue

Идемпотентность: webhook_id как unique identifier, detect duplicates

Error handling: log everything, use DLQ for persistent failures, alert team

Signature verification: verify отправителя (Stripe, GitHub, etc.)

Monitoring: heartbeat pattern, health endpoint с metrics, test webhook'ом себя

SSL: monitor certificate expiry, alert за 30 дней

Logging: full audit trail, payload_hash, retry_count, processing_time

Heartbeat Monitoring — паттерн для проверки periodic обработок

API Monitoring Guide — как мониторить REST endpoints

Health Check Endpoint Design — JSON path assertions для /health

SSL Certificate Monitoring — мониторить expiry

Reduce False Alarms — quorum confirmation для webhook'ов

Heartbeat Check — в AtomPing