Webhook System

Event-driven webhooks with HMAC rotation (24h grace), tenant hierarchy subscriptions, circuit breaker, and delivery logs.

Webhook Architecture

The Webhook System enables external integrations by delivering event payloads to subscriber URLs. Each payload is HMAC-signed for verification, and failed deliveries are retried with exponential backoff.

Webhook Delivery & Signature Verification Flow
Domain Event Triggered
Match active WebhookSubscriptions
NFR-02: Size Check & Truncation
SSRF URL Verification
URL resolved & allowed
Compute HMAC-SHA256 Signature
PreviousSecret Exists & Not Expired?
Yes (rotation active)
Dual-Sign (Generate secondary signature)
HTTP POST (30s timeout)
Request dispatched
Receiver Signature Verification
WebhookRetryJob (Backoff queues)

WebhookSubscription Entity

WebhookSubscription.csC#
public class WebhookSubscription : AuditableEntity
{
    public Guid Id { get; set; }
    
    [Required] [MaxLength(500)]
    public string Url { get; set; } = string.Empty;       // Allowed HTTPS destination URL
    
    [Required] [MaxLength(100)]
    public string Secret { get; set; } = string.Empty;    // HMAC signature key
    
    [MaxLength(100)]
    public string? PreviousSecret { get; set; }           // Temporary old key
    public DateTime? PreviousSecretExpiresAt { get; set; } // 24-hour grace period threshold
    
    public bool IsActive { get; set; } = true;            // Toggle state (disabled by circuit breaker)
    public string EventsJson { get; set; } = "[]";        // Subscribed event types JSON array
    public WebhookScope Scope { get; set; }               // Scope filter settings
    
    public int MaxRetries { get; set; } = 4;              // Max retry attempts
    public int MaxConsecutiveFailures { get; set; } = 10; // Circuit breaker threshold
    public int ConsecutiveFailures { get; set; } = 0;     // Consecutive failure counter
    
    public Guid? TenantId { get; set; }                   // Owning tenant ID context
    public virtual Tenant? Tenant { get; set; }
}

HMAC Signing

Every webhook payload is signed with HMAC-SHA256 using the subscription's secret key. The receiver can verify the signature by computing the HMAC of the raw request body and comparing it to the X-Webhook-Signature header.

WebhookSignatureVerification.csC#
// Server-side: Sign payload with timestamp prefix
long timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
string signatureInput = $"{timestamp}.{payloadJson}";
using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(subscription.Secret));
byte[] hashBytes = hmac.ComputeHash(Encoding.UTF8.GetBytes(signatureInput));
string signature = $"sha256={Convert.ToHexString(hashBytes).ToLowerInvariant()}";

// Dispatched HTTP Headers:
// X-Webhook-Signature: sha256={hash}
// X-Webhook-Signature-Old: sha256={oldHash} (only during secret rotation)
// X-Webhook-Timestamp: {timestamp}
// X-Webhook-Event: {eventType}
// X-Webhook-Delivery: {eventDeliveryId} (same for all retries)
// X-Webhook-Attempt: {attemptNumber}

// Receiver: Reconstruct signature input & verify
string receivedSignature = request.Headers["X-Webhook-Signature"];
string receivedTimestamp = request.Headers["X-Webhook-Timestamp"];
string requestBody = await ReadBodyAsStringAsync(request);

string computedInput = $"{receivedTimestamp}.{requestBody}";
using var verifierHmac = new HMACSHA256(Encoding.UTF8.GetBytes(mySecret));
byte[] computedHash = verifierHmac.ComputeHash(Encoding.UTF8.GetBytes(computedInput));
string computedSignature = $"sha256={Convert.ToHexString(computedHash).ToLowerInvariant()}";

bool isValid = CryptographicOperations.FixedTimeEquals(
    Encoding.UTF8.GetBytes(computedSignature),
    Encoding.UTF8.GetBytes(receivedSignature)
);

Secret Rotation (24h Grace Period)

The RotateSecret endpoint generates a new HMAC secret while keeping the old one valid for 24 hours. This prevents missed deliveries during migration the system signs payloads with the new secret and includes the old signature in X-Webhook-Signature-Old header.

Secret Rotation 24-Hour Grace Period Lifecycle
POST /rotate-secret
PreviousSecret = current Secret
Set PreviousSecretExpiresAt = Now + 24h
Secret = new Secret
Active transition
Dual-sign payloads in dispatcher
After 24 hours
WebhookRetryJob removes expired PreviousSecrets

Tenant Hierarchy Subscriptions

When IncludeChildren is enabled on a subscription, the webhook receives events from the subscribing tenant AND all child tenants in the hierarchy. This is perfect for parent companies that need to monitor branch activity.

Webhook Scope TypeTenantId ContextEvent Filtering LogicCommon Use Case
PlatformOnlynullReceives events from the platform level only (TenantId = null)Global analytics / tenant management engines
AllTenantsnull / non-nullReceives all events across all tenants in the systemSuper-admin audits & monitoring dashboard
TenantOnlynon-nullReceives events originating strictly from own tenant contextSingle tenant custom automation hooks
TenantWithChildrennon-nullReceives events from own tenant and all descendant tenantsParent companies monitoring branch subsidiaries

Circuit Breaker (Auto-Disable)

Each subscription has a MaxConsecutiveFailures threshold. When reached, the subscription is automatically disabled (IsActive = false) to avoid hammering failing endpoints. An AuditLog entry is created and the admin is notified.

Webhook Circuit Breaker & Auto-Disable Logic
DeliverAsync execution
Delivery successful (HTTP 2xx)?
Yes
Reset ConsecutiveFailures = 0
ConsecutiveFailures++
ConsecutiveFailures >= Max?
Yes
IsActive = false (Auto-Disabled)
Log security event (WebhookCircuitBroken)

Retry Policy

Failed webhook deliveries are retried with exponential backoff. The system tracks the last attempt time and response status to determine when to retry.

Retry AttemptDelay OffsetCumulative Retries TimelineMechanism Details
Initial AttemptImmediate0 secondsDispatched synchronously in command pipeline
1st Retry10 seconds10 secondsQueued as DeliveryStatus.Retrying, processed by WebhookRetryJob
2nd Retry60 seconds1 minute 10sScheduled with exponential delay offset
3rd Retry5 minutes6 minutes 10sScheduled with exponential delay offset
4th Retry (Final)30 minutes36 minutes 10sLast attempt before dead-lettering
ExhaustedNo further retryDead-letteredStatus = DeadLettered, increments subscription.FailedDeliveries

Delivery Logs

Every delivery attempt is recorded in the WebhookDeliveryLog table with status code, response body, duration, and attempt number. Failed deliveries include error details for debugging.

WebhookDeliveryLog.csC#
public class WebhookDeliveryLog
{
    public Guid Id { get; set; }
    public Guid SubscriptionId { get; set; }
    public Guid EventDeliveryId { get; set; }     // Matches across all retry attempts for de-duplication
    public Guid? EventTenantId { get; set; }       // Tenant context of origin event
    public string EventType { get; set; } = "";
    public string PayloadJson { get; set; } = "";
    
    public string RequestUrl { get; set; } = "";
    public string? RequestHeaders { get; set; }    // JSON string of headers (excluding secrets)
    public int HttpStatusCode { get; set; }        // HTTP response status code (0 for connection issues)
    public string? ResponseBody { get; set; }      // Snapped response snippet (max 2048 chars)
    
    public int AttemptNumber { get; set; }         // Attempt index: 1, 2, 3, etc.
    public double LatencyMs { get; set; }          // Network latency timing
    public bool IsSuccess { get; set; }            // Status flag: HTTP status 2xx
    public string? ErrorMessage { get; set; }      // Network error snippet (if any)
    public DeliveryStatus Status { get; set; }     // State enum: Delivered, Retrying, DeadLettered
    public DateTime CreatedAt { get; set; }
}

Webhook Event Types

Subscribable Event TypePayload AttributesTrigger Point Description
admin.createdAdmin ID, Username, Email, TenantId contextTriggered when a new administrator is registered
admin.updatedModified property changes, old and new valuesTriggered when administrator profile changes are saved
admin.deletedAdmin ID, Soft-delete timestamp, DeletedBy userTriggered when administrator account is soft-deleted
tenant.createdTenant details, plan setting overridesTriggered when a new tenant/workspace is registered
tenant.updatedModified settings or billing preferencesTriggered when workspace configuration is updated
tenant.deletedTenant ID, cascade deletion results statusTriggered when a workspace is soft-deleted
role.createdRole ID, assigned permissions array listTriggered when a new user role is created
role.permissions_changedAdded permissions list, revoked permissions listTriggered when permission scopes are modified on a role
audit.security_eventSecurity event type, host IP, actor detailsTriggered on unauthorized actions or guardian blocks

Subscription Management

MethodEndpointDescriptionAuth
GET/api/v1/webhooksList all subscriptions (tenant-scoped)🔒 Required
GET/api/v1/webhooks/{id}Get subscription details (including stats and state)🔒 Required
POST/api/v1/webhooksCreate subscription (configurable url, events list, scope filters)🔒 Required
PUT/api/v1/webhooks/{id}Update subscription URL, events catalog list, or status🔒 Required
DELETE/api/v1/webhooks/{id}Permanently remove a webhook subscription🔒 Required

Operations & Monitoring

MethodEndpointDescriptionAuth
POST/api/v1/webhooks/{id}/rotate-secretTrigger secret rotation with a 24-hour grace period🔒 Required
PUT/api/v1/webhooks/{id}/toggleManually enable/disable subscription (resets circuit breaker)🔒 Required
POST/api/v1/webhooks/{id}/testDispatch a 'webhook.test' ping payload immediately without retries🔒 Required
GET/api/v1/webhooks/{id}/delivery-logsGet paginated delivery logs (filtering by status, latency)🔒 Required
GET/api/v1/webhooks/available-eventsGet the complete catalog of subscribable event types🔒 Required