Handling Service Downtime

Understanding Retry Behavior

When a webhook delivery fails, QueueBear automatically retries with exponential backoff:

With the default 3 retries (4 total attempts), delivery spans ~7 seconds. With maximum 5 retries, ~31 seconds.

The Dead Letter Queue (DLQ)

Messages that fail all retry attempts are automatically moved to the Dead Letter Queue. This is your safety net for prolonged downtime.

Monitoring the DLQ

const { entries } = await qb.dlq.list();

for (const entry of entries) {
  console.log({
    id: entry.id,
    destination: entry.destination,
    failureReason: entry.failureReason,
    totalAttempts: entry.totalAttempts,
    lastAttemptAt: entry.lastAttemptAt,
  });
}

Setting Up DLQ Alerts

// Check DLQ every 5 minutes
await qb.schedules.create({
  destination: "https://app.com/api/admin/dlq-monitor",
  cron: "*/5 * * * *",
  body: JSON.stringify({ action: "check_and_alert" }),
});

Recovering from DLQ

// Retry a single entry
const result = await qb.dlq.retry(dlqId);

// Retry all entries
const results = await qb.dlq.retryAll();

// Selective retry
const { entries } = await qb.dlq.list();
for (const entry of entries) {
  if (entry.destination.includes("payment-service")) {
    await qb.dlq.retry(entry.id);
  }
}

Using Events for External Dependencies

When your workflow depends on a service that may be down, use waitForEvent():

export const POST = serve<OrderInput>(async (ctx) => {
  await ctx.run("create-order", async () => { /* ... */ });

  // Wait for external service to signal readiness
  const externalResult = await ctx.waitForEvent(
    "wait-for-external-service",
    "external.service.ready",
    {
      eventKey: ctx.input.orderId,
      timeoutSeconds: 4 * 60 * 60,  // Wait up to 4 hours
    }
  );

  // Now safe to proceed
  await ctx.run("process-payment", async () => {
    await paymentService.charge(ctx.input.orderId, ctx.input.amount);
  });

  return { success: true };
});

When the service recovers, send the event:

await qb.workflows.sendEvent("external.service.ready", {
  eventKey: orderId,
  payload: { recoveredAt: new Date().toISOString() },
});

Workflow Checkpoint and Resume

Workflows automatically checkpoint after each completed step. On retry, they resume from the last checkpoint.

export const POST = serve<Input>(async (ctx) => {
  // Step 1: Completes, result cached
  const user = await ctx.run("fetch-user", async () => {
    return await userService.get(ctx.input.userId);
  });

  // Step 2: Fails due to service outage
  await ctx.run("update-crm", async () => {
    await crmService.update(user);  // Service is down!
  });

  // Step 3: Never reached
  await ctx.run("send-notification", async () => { ... });
});

When the CRM service recovers:

// Retry the workflow
await qb.workflows.retry(runId);

// Step 1 returns cached result (NOT re-executed)
// Step 2 executes (CRM service now available)
// Step 3 executes

Graceful Degradation Strategies

Skip Non-Critical Steps

export const POST = serve<OrderInput>(async (ctx) => {
  // Critical: Process order
  await ctx.run("process-order", async () => {
    await orderService.process(ctx.input);
  });

  // Non-critical: Update analytics (skip if fails)
  try {
    await ctx.run("update-analytics", async () => {
      await analyticsService.track(ctx.input);
    });
  } catch (error) {
    console.warn("Analytics update failed, continuing:", error);
  }

  return { success: true };
});

Fallback Destinations

await ctx.run("process-with-fallback", async () => {
  try {
    return await primaryService.process(data);
  } catch (error) {
    console.warn("Primary service failed, trying fallback");
    return await fallbackService.process(data);
  }
});

Example: Payment Service Outage Recovery

Scenario: Your payment provider experiences a 4-hour outage.

During Outage

• Orders triggered will complete step 1 (create order)
• Fail at step 2 (payment) after retries
• Workflow status becomes "failed"
• Messages go to DLQ

Recovery

// 1. Check health
const isHealthy = await paymentService.healthCheck();

if (isHealthy) {
  // 2. Retry all DLQ entries
  await qb.dlq.retryAll();

  // 3. Retry failed workflows
  const failedWorkflows = await db.workflowRuns.findMany({
    where: { status: "failed", workflowName: "order-processing" },
  });

  for (const workflow of failedWorkflows) {
    await qb.workflows.retry(workflow.id);
  }
}

Summary