Webhook System
Event-driven webhooks with HMAC rotation (24h grace), tenant hierarchy subscriptions, circuit breaker, and delivery logs.
Webhook Architecture
The Webhook System enables external integrations by delivering event payloads to subscriber URLs. Each payload is HMAC-signed for verification, and failed deliveries are retried with exponential backoff.
WebhookSubscription Entity
public class WebhookSubscription : AuditableEntity
{
public Guid Id { get; set; }
[Required] [MaxLength(500)]
public string Url { get; set; } = string.Empty; // Allowed HTTPS destination URL
[Required] [MaxLength(100)]
public string Secret { get; set; } = string.Empty; // HMAC signature key
[MaxLength(100)]
public string? PreviousSecret { get; set; } // Temporary old key
public DateTime? PreviousSecretExpiresAt { get; set; } // 24-hour grace period threshold
public bool IsActive { get; set; } = true; // Toggle state (disabled by circuit breaker)
public string EventsJson { get; set; } = "[]"; // Subscribed event types JSON array
public WebhookScope Scope { get; set; } // Scope filter settings
public int MaxRetries { get; set; } = 4; // Max retry attempts
public int MaxConsecutiveFailures { get; set; } = 10; // Circuit breaker threshold
public int ConsecutiveFailures { get; set; } = 0; // Consecutive failure counter
public Guid? TenantId { get; set; } // Owning tenant ID context
public virtual Tenant? Tenant { get; set; }
}HMAC Signing
Every webhook payload is signed with HMAC-SHA256 using the subscription's secret key. The receiver can verify the signature by computing the HMAC of the raw request body and comparing it to the X-Webhook-Signature header.
// Server-side: Sign payload with timestamp prefix
long timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
string signatureInput = $"{timestamp}.{payloadJson}";
using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(subscription.Secret));
byte[] hashBytes = hmac.ComputeHash(Encoding.UTF8.GetBytes(signatureInput));
string signature = $"sha256={Convert.ToHexString(hashBytes).ToLowerInvariant()}";
// Dispatched HTTP Headers:
// X-Webhook-Signature: sha256={hash}
// X-Webhook-Signature-Old: sha256={oldHash} (only during secret rotation)
// X-Webhook-Timestamp: {timestamp}
// X-Webhook-Event: {eventType}
// X-Webhook-Delivery: {eventDeliveryId} (same for all retries)
// X-Webhook-Attempt: {attemptNumber}
// Receiver: Reconstruct signature input & verify
string receivedSignature = request.Headers["X-Webhook-Signature"];
string receivedTimestamp = request.Headers["X-Webhook-Timestamp"];
string requestBody = await ReadBodyAsStringAsync(request);
string computedInput = $"{receivedTimestamp}.{requestBody}";
using var verifierHmac = new HMACSHA256(Encoding.UTF8.GetBytes(mySecret));
byte[] computedHash = verifierHmac.ComputeHash(Encoding.UTF8.GetBytes(computedInput));
string computedSignature = $"sha256={Convert.ToHexString(computedHash).ToLowerInvariant()}";
bool isValid = CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(computedSignature),
Encoding.UTF8.GetBytes(receivedSignature)
);Secret Rotation (24h Grace Period)
The RotateSecret endpoint generates a new HMAC secret while keeping the old one valid for 24 hours. This prevents missed deliveries during migration the system signs payloads with the new secret and includes the old signature in X-Webhook-Signature-Old header.
Tenant Hierarchy Subscriptions
When IncludeChildren is enabled on a subscription, the webhook receives events from the subscribing tenant AND all child tenants in the hierarchy. This is perfect for parent companies that need to monitor branch activity.
| Webhook Scope Type | TenantId Context | Event Filtering Logic | Common Use Case |
|---|---|---|---|
| PlatformOnly | null | Receives events from the platform level only (TenantId = null) | Global analytics / tenant management engines |
| AllTenants | null / non-null | Receives all events across all tenants in the system | Super-admin audits & monitoring dashboard |
| TenantOnly | non-null | Receives events originating strictly from own tenant context | Single tenant custom automation hooks |
| TenantWithChildren | non-null | Receives events from own tenant and all descendant tenants | Parent companies monitoring branch subsidiaries |
Circuit Breaker (Auto-Disable)
Each subscription has a MaxConsecutiveFailures threshold. When reached, the subscription is automatically disabled (IsActive = false) to avoid hammering failing endpoints. An AuditLog entry is created and the admin is notified.
Retry Policy
Failed webhook deliveries are retried with exponential backoff. The system tracks the last attempt time and response status to determine when to retry.
| Retry Attempt | Delay Offset | Cumulative Retries Timeline | Mechanism Details |
|---|---|---|---|
| Initial Attempt | Immediate | 0 seconds | Dispatched synchronously in command pipeline |
| 1st Retry | 10 seconds | 10 seconds | Queued as DeliveryStatus.Retrying, processed by WebhookRetryJob |
| 2nd Retry | 60 seconds | 1 minute 10s | Scheduled with exponential delay offset |
| 3rd Retry | 5 minutes | 6 minutes 10s | Scheduled with exponential delay offset |
| 4th Retry (Final) | 30 minutes | 36 minutes 10s | Last attempt before dead-lettering |
| Exhausted | No further retry | Dead-lettered | Status = DeadLettered, increments subscription.FailedDeliveries |
Delivery Logs
Every delivery attempt is recorded in the WebhookDeliveryLog table with status code, response body, duration, and attempt number. Failed deliveries include error details for debugging.
public class WebhookDeliveryLog
{
public Guid Id { get; set; }
public Guid SubscriptionId { get; set; }
public Guid EventDeliveryId { get; set; } // Matches across all retry attempts for de-duplication
public Guid? EventTenantId { get; set; } // Tenant context of origin event
public string EventType { get; set; } = "";
public string PayloadJson { get; set; } = "";
public string RequestUrl { get; set; } = "";
public string? RequestHeaders { get; set; } // JSON string of headers (excluding secrets)
public int HttpStatusCode { get; set; } // HTTP response status code (0 for connection issues)
public string? ResponseBody { get; set; } // Snapped response snippet (max 2048 chars)
public int AttemptNumber { get; set; } // Attempt index: 1, 2, 3, etc.
public double LatencyMs { get; set; } // Network latency timing
public bool IsSuccess { get; set; } // Status flag: HTTP status 2xx
public string? ErrorMessage { get; set; } // Network error snippet (if any)
public DeliveryStatus Status { get; set; } // State enum: Delivered, Retrying, DeadLettered
public DateTime CreatedAt { get; set; }
}Webhook Event Types
| Subscribable Event Type | Payload Attributes | Trigger Point Description |
|---|---|---|
| admin.created | Admin ID, Username, Email, TenantId context | Triggered when a new administrator is registered |
| admin.updated | Modified property changes, old and new values | Triggered when administrator profile changes are saved |
| admin.deleted | Admin ID, Soft-delete timestamp, DeletedBy user | Triggered when administrator account is soft-deleted |
| tenant.created | Tenant details, plan setting overrides | Triggered when a new tenant/workspace is registered |
| tenant.updated | Modified settings or billing preferences | Triggered when workspace configuration is updated |
| tenant.deleted | Tenant ID, cascade deletion results status | Triggered when a workspace is soft-deleted |
| role.created | Role ID, assigned permissions array list | Triggered when a new user role is created |
| role.permissions_changed | Added permissions list, revoked permissions list | Triggered when permission scopes are modified on a role |
| audit.security_event | Security event type, host IP, actor details | Triggered on unauthorized actions or guardian blocks |
Subscription Management
| Method | Endpoint | Description | Auth |
|---|---|---|---|
| GET | /api/v1/webhooks | List all subscriptions (tenant-scoped) | 🔒 Required |
| GET | /api/v1/webhooks/{id} | Get subscription details (including stats and state) | 🔒 Required |
| POST | /api/v1/webhooks | Create subscription (configurable url, events list, scope filters) | 🔒 Required |
| PUT | /api/v1/webhooks/{id} | Update subscription URL, events catalog list, or status | 🔒 Required |
| DELETE | /api/v1/webhooks/{id} | Permanently remove a webhook subscription | 🔒 Required |
Operations & Monitoring
| Method | Endpoint | Description | Auth |
|---|---|---|---|
| POST | /api/v1/webhooks/{id}/rotate-secret | Trigger secret rotation with a 24-hour grace period | 🔒 Required |
| PUT | /api/v1/webhooks/{id}/toggle | Manually enable/disable subscription (resets circuit breaker) | 🔒 Required |
| POST | /api/v1/webhooks/{id}/test | Dispatch a 'webhook.test' ping payload immediately without retries | 🔒 Required |
| GET | /api/v1/webhooks/{id}/delivery-logs | Get paginated delivery logs (filtering by status, latency) | 🔒 Required |
| GET | /api/v1/webhooks/available-events | Get the complete catalog of subscribable event types | 🔒 Required |