Alerts & Notifications

Get notified immediately when your routes experience failures, high latency, unauthorized access attempts, or rate limit issues.

What Are Alerts?

Alerts monitor your routes in real-time and send notifications when specific conditions are met. Example:

Route: stripe-payments
Condition: 5+ failed requests (5xx errors) within 5 minutes
Action: Send email to ops@example.com + Slack #incidents channel

Use cases:

🚨 Detect API integration failures instantly
⏱️ Monitor response time degradation
🔒 Track unauthorized access attempts
📊 Catch rate limit violations
🛠️ Proactive issue resolution before customers complain

Alert Types

KnoxCall supports four types of alerts:

1. Request Failures

What it monitors: Failed requests (5xx status codes) Common scenarios:

Backend API is down (502, 503, 504)
Internal server errors (500)
Backend timeout issues

Example condition:

{
  "threshold": 5,
  "window_minutes": 5,
  "include_status_codes": [500, 502, 503, 504]
}

Triggers when: 5 or more 5xx errors occur within 5 minutes Use case:

Your Stripe integration suddenly starts returning 503 errors.
Alert triggers after 5 failures → Email sent → Team investigates immediately
→ Discover Stripe API outage → Switch to backup payment processor

2. High Latency

What it monitors: Request response times (P95 percentile) Common scenarios:

Backend database slow queries
Network issues
Service degradation

Example condition:

{
  "threshold_ms": 2000,
  "percentile": 95,
  "window_minutes": 5,
  "min_requests": 10
}

Triggers when: P95 latency exceeds 2 seconds (2000ms) over 5 minutes, with at least 10 requests Use case:

User complaints about slow checkout page.
Alert triggers when P95 latency hits 2.5s → Team investigates → Find database index missing
→ Add index → Latency drops to 300ms

Percentiles explained:

P50 (median): 50% of requests faster than this
P95: 95% of requests faster than this (catches slowest 5%)
P99: 99% of requests faster than this (extreme outliers)

3. Rate Limit Exceeded

What it monitors: Requests hitting your configured rate limits Common scenarios:

Client sending too many requests
Runaway script or bot
DDoS attempt

Example condition:

{
  "threshold_percent": 80,
  "window_minutes": 10
}

Triggers when: Rate limit utilisation exceeds 80% within 10 minutes Use case:

Client accidentally deploys infinite loop calling your API.
Alert triggers → Email sent → You contact client → They fix the bug
→ Prevents unnecessary API costs

4. Unauthorized Client

What it monitors: Requests from non-whitelisted IP addresses Common scenarios:

Security breach attempt
Client using wrong IP after server migration
Misconfigured firewall

Example condition:

{
  "immediate": true,
  "threshold": 1,
  "window_minutes": 5
}

Triggers when: Unauthorized IP tries to access route (immediate alert) Use case:

Random IP address attempts to access your internal API.
Alert triggers immediately → Security team reviews → Block malicious IP
→ Prevent potential breach

5. Vault Destination Monitoring (automatic)

Not a template you pick. The moment your workspace has its first vault, KnoxCall installs a Vault Destination Monitoring alert for you — no Routes required, no configuration, and it appears in your alerts list like any other. Delete it and it stays deleted. Every time a vault releases a value, KnoxCall counts it against the destination it went to. Two rules fire on that counter:

Rule	Fires when
New destination	A vault releases a value to a destination it has not used in the last 30 days.
Volume spike	A window’s count for one vault-and-destination pair is far above the baseline learned for that pair.

The alert names the vault, the destination and the count, and reaches you through the same channels as every other alert. It never contains the token or the value. Live traffic only. Test-mode detokenizes are counted in a separate history — which is what stops a test run teaching the detector that an unfamiliar host is familiar — but they never raise an alert, for the same reason no other alert fires on Test traffic: alerting is a production concern. A destination is bucketed by the pattern it matched on the vault’s allowed_destinations list, so api.stripe.com and files.stripe.com under *.stripe.com are one destination — a second endpoint at a processor you already approved is not an alert; a processor you never approved is.

Notification Channels

Alerts can be sent through multiple channels:

Email Notifications

Configuration:

Email Addresses: ops@example.com, john@example.com, team@example.com

Format:

Subject: [CRITICAL] stripe-payments: Integration Failure
Body:
Alert: Integration Failure
Route: stripe-payments
Severity: Critical
Time: 2025-01-15 10:30:45 UTC

Condition Met: 5+ failures within 5 minutes

Recent Failures:
- 10:30:42 UTC - 503 Service Unavailable
- 10:30:40 UTC - 502 Bad Gateway
- 10:30:38 UTC - 502 Bad Gateway
- 10:30:35 UTC - 503 Service Unavailable
- 10:30:33 UTC - 500 Internal Server Error

View Route: https://knoxcall.com/routes/route_abc123
View Alert: https://knoxcall.com/alerts/alert_def456

Best for: Primary notification channel, detailed information

SMS Notifications

SMS is an Enterprise channel, and every destination must be verified before it can receive alerts: KnoxCall texts a code to the handset and you enter it once. Verify a number under Alerts → Phone numbers, or inline when you add an SMS destination to an alert.

Verification is enforced when the alert is sent, not only when it is saved. A number showing as Unverified on the Phone numbers page is not receiving alerts — any alert pointed at it is failing to deliver, and the reason is recorded on the alert log. Verify it to restore paging.

This applies to alerts and to severity routing defaults alike — a number that has not been verified is refused wherever you try to add it. Configuration:

Phone Numbers: +15551234567, +15559876543   (each verified once, then reusable)

Format:

[KnoxCall CRITICAL] stripe-payments: 5 failures in 5min. Check email for details.

Best for: Critical alerts, on-call engineers, immediate attention

Slack Notifications

Configuration:

Webhook URL: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX

Format:

🚨 CRITICAL ALERT: Integration Failure

Route: stripe-payments
Severity: Critical
Time: 2025-01-15 10:30:45 UTC

5 failures detected within 5 minutes

Recent errors:
• 503 Service Unavailable (3)
• 502 Bad Gateway (2)

[View Route] [View Alert] [Acknowledge]

Best for: Team collaboration, incident response, threaded discussions

Creating Your First Alert

Step 1: Navigate to Alerts

Click Monitoring in sidebar
Select Alerts
Click + Create Alert

Step 2: Choose a Template

KnoxCall provides pre-configured templates: Integration Failure (Recommended)

Alert on any 5xx error
Severity: High
Cooldown: 15 minutes
Great for catching backend issues immediately

Critical Failures

Alert on 5+ failures within 5 minutes
Severity: Critical
Cooldown: 30 minutes
Best for production environments

High Latency

Alert when P95 latency exceeds 2 seconds
Severity: Medium
Cooldown: 20 minutes
Good for performance monitoring

Unauthorized Access Attempt

Immediate alert on unauthorized IP
Severity: High
Cooldown: 60 minutes
Essential for security

Or click Custom Alert to configure from scratch.

Step 3: Configure Alert

Route: Select the route to monitor

stripe-payments

Alert Name:

Stripe Integration Failure

Description (optional):

Alert when Stripe API returns 5xx errors

Alert Type:

Request Failures

Severity:

Low: Informational, review later
Medium: Important, check within hours
High: Urgent, check within 30 minutes
Critical: Emergency, check immediately

Alert Conditions (JSON):

{
  "threshold": 3,
  "window_minutes": 5,
  "include_status_codes": [500, 502, 503, 504]
}

Cooldown (minutes):

Time before alert can trigger again (prevents spam) Aggregation (minutes):

Window for batching multiple events into one alert

Step 4: Configure Notifications

Enable Email

☑ Email Notifications
Email Addresses: ops@example.com, john@example.com

Enable SMS (optional, Enterprise)

☐ SMS Notifications
Phone Numbers: +15551234567   ← must be verified first

Adding an SMS destination offers only numbers you have already verified, plus a Verify another number step that sends a code and confirms it without leaving the form. Numbers are verified once per tenant and can then be used on any alert or routing rule.

Enable Slack (optional)

☐ Slack Notifications
Webhook URL: https://hooks.slack.com/services/...

At least one channel required. You can enable multiple channels to ensure alerts are seen.

Step 5: Custom Message Templates (Optional)

Customize notification content: Email Subject Template:

[{{SEVERITY}}] {{ROUTE_NAME}}: {{ALERT_NAME}}

Email Body Template:

Alert triggered: {{ALERT_NAME}}
Route: {{ROUTE_NAME}}
Severity: {{SEVERITY}}
Time: {{TIMESTAMP}}

Condition: {{CONDITION_DESCRIPTION}}

{{TRIGGER_DETAILS}}

View: https://knoxcall.com/alerts/{{ALERT_ID}}

Available variables:

{{ALERT_NAME}} - Alert name
{{ROUTE_NAME}} - Route name
{{SEVERITY}} - low/medium/high/critical
{{TIMESTAMP}} - When alert triggered
{{CONDITION_DESCRIPTION}} - Human-readable condition
{{TRIGGER_DETAILS}} - Specific trigger info (errors, latency, etc.)
{{ALERT_ID}} - Alert ID
{{ROUTE_ID}} - Route ID

Step 6: Create Alert

Click Create Alert Alert is now active and monitoring your route! 🎉

Alert States

Alerts have three states:

1. OK (Green)

Meaning: Condition is not met, everything normal Example:

Route: stripe-payments
Failures: 0 in last 5 minutes
State: OK ✓

2. Triggered (Red)

Meaning: Condition met, notification sent Example:

Route: stripe-payments
Failures: 6 in last 5 minutes (threshold: 5)
State: TRIGGERED ⚠️
Notification sent: 2025-01-15 10:30:45 UTC

3. Cooldown (Yellow)

Meaning: Recently triggered, waiting for cooldown period Example:

Route: stripe-payments
Last triggered: 5 minutes ago
Cooldown: 15 minutes
State: COOLDOWN (10 minutes remaining)

Purpose of cooldown:

Prevents notification spam
Gives time to fix issue
Won’t trigger again until cooldown expires

Advanced Configuration

Condition Schemas

Different alert types have different configuration options:

Request Failures Schema

{
  "threshold": 5,                        // Number of failures to trigger
  "window_minutes": 5,                   // Time window to count failures
  "include_status_codes": [500, 502, 503, 504],  // Which status codes count as failures
  "exclude_status_codes": []             // Optional: exclude specific codes
}

High Latency Schema

{
  "threshold_ms": 2000,                  // Latency threshold in milliseconds
  "percentile": 95,                      // Percentile to monitor (50, 95, 99)
  "window_minutes": 5,                   // Time window to calculate percentile
  "min_requests": 10                     // Minimum requests needed (prevents false positives)
}

Rate Limit Exceeded Schema

{
  "threshold_percent": 80,               // Percentage of rate limit consumed (1–100) to trigger
  "window_minutes": 10                   // Optional: time window to evaluate utilisation
}

Unauthorized Client Schema

{
  "immediate": true,                     // Trigger on first occurrence
  "threshold": 1,                        // Usually 1 for immediate alerts
  "window_minutes": 5                    // Window for counting attempts
}

Multi-Status Code Filtering

Include only specific errors:

{
  "threshold": 3,
  "window_minutes": 5,
  "include_status_codes": [503, 504]     // Only 503 and 504
}

Exclude specific errors:

{
  "threshold": 3,
  "window_minutes": 5,
  "include_status_codes": [500, 502, 503, 504],
  "exclude_status_codes": [503]          // Ignore 503, alert on others
}

Use case: Ignore expected 503 errors during maintenance.

Latency Percentiles

P50 (Median):

{
  "threshold_ms": 500,
  "percentile": 50,
  "window_minutes": 5,
  "min_requests": 50
}

Alert if typical request exceeds 500ms P95 (Recommended):

{
  "threshold_ms": 2000,
  "percentile": 95,
  "window_minutes": 5,
  "min_requests": 10
}

Alert if slowest 5% exceed 2 seconds P99 (Outliers):

{
  "threshold_ms": 5000,
  "percentile": 99,
  "window_minutes": 5,
  "min_requests": 100
}

Alert on extreme outliers (slowest 1%)

Alert Management

Viewing Alert Status

Navigate to: Monitoring → Alerts List view shows:

Alert name and description
Route name
Current state (OK / Triggered / Cooldown)
Severity
Trigger count (24h)
Last triggered time
Enabled/Disabled status

Filters:

Severity: Low, Medium, High, Critical
State: OK, Triggered, Cooldown
Enabled: All, Enabled, Disabled
Alert Type: Failures, Latency, Rate Limit, Unauthorized
Activity: 0 triggers, 1-5, 6-20, 20+

Viewing Alert Details

Click alert name to see: Overview:

Current state
Trigger history graph
Recent triggers list

Configuration:

Alert type and conditions
Notification channels
Cooldown and aggregation settings

Logs:

When alert triggered
Notification delivery status
Error details that triggered alert

Editing Alerts

Navigate to alert details
Click Edit Alert
Modify settings:
- Change threshold
- Update notification emails/phones
- Adjust cooldown
- Change severity
Click Save Changes

Changes take effect immediately.

Disabling Alerts

Temporarily disable:

Navigate to alert details
Toggle Enabled switch to OFF
Alert stops monitoring (won’t trigger)

Use cases:

Scheduled maintenance
Known issue being fixed
Testing changes without spam

Re-enable:

Toggle Enabled switch to ON
Alert resumes monitoring

Deleting Alerts

Navigate to alert details
Click Delete Alert
Confirm deletion

Deleting an alert is permanent! All trigger history and configuration is lost. Consider disabling instead if you might need it later.

Alert Logs

View alert trigger history: Navigate to: Monitoring → Alert Logs Shows:

When alert triggered
Which route
Severity
Trigger details (error messages, latency values, etc.)
Notification channels used
Delivery status (sent, failed)

Filter by:

Date range
Route
Severity
Alert name

Use cases:

Audit notification history
Troubleshoot missed alerts
Analyze incident patterns

Best Practices

1. Start with Templates

✅ Use built-in templates when creating your first alerts Why:

Pre-configured with sensible defaults
Battle-tested thresholds
Clear descriptions

❌ Don’t create custom alerts immediately

2. Set Appropriate Thresholds

Too sensitive (spam):

{
  "threshold": 1,
  "window_minutes": 1
}

Triggers on single error = too noisy Too lenient (miss issues):

{
  "threshold": 50,
  "window_minutes": 60
}

50 errors in an hour might mean users already frustrated ✅ Just right:

{
  "threshold": 5,
  "window_minutes": 5
}

Catches issues quickly without false positives

3. Use Severity Correctly

Critical:

Production payment processing down
Complete API outage
Security breach attempt

High:

Partial service degradation
Single integration failing
Elevated error rate

Medium:

Performance degradation
Non-critical API slow
Occasional errors

Low:

Informational
Minor issues
For tracking/trending

4. Configure Cooldowns

Problem: Alert triggers every minute → 60 emails in 1 hour Solution: Use cooldown

Cooldown: 30 minutes
Result: 1 alert, wait 30 min, then can alert again if still failing

Recommended cooldowns:

Critical alerts: 30-60 minutes
High alerts: 15-30 minutes
Medium alerts: 60 minutes
Low alerts: 2-4 hours

5. Use Multiple Notification Channels

Redundancy strategy:

Critical Alert:
- Email: ops@example.com
- SMS: +15551234567 (on-call engineer)
- Slack: #incidents channel

Why:

Email might be missed
SMS ensures immediate attention
Slack allows team collaboration

6. Test Your Alerts

Before going live:

Create test alert with low threshold:

{
  "threshold": 1,
  "window_minutes": 1
}

Trigger condition (e.g., send request that returns 500)
Verify notifications received:
- Check email inbox
- Check SMS received
- Check Slack message
Adjust configuration if needed
Set production thresholds

7. Monitor Alert Logs

Weekly review:

Which alerts triggered most?
Any false positives?
Any missed incidents?

Adjust thresholds based on patterns.

8. Document Your Alerts

In alert description, include:

Description: Alerts when Stripe API returns 5xx errors.
Indicates Stripe service outage or network issues between KnoxCall and Stripe.
Runbook: https://wiki.company.com/runbooks/stripe-outage

Benefits:

Team knows what alert means
Clear action steps
Faster incident resolution

Common Alert Scenarios

Scenario 1: Backend API Outage

Problem: Stripe API completely down Alert Configuration:

Alert Type: Request Failures
Condition: {
  "threshold": 3,
  "window_minutes": 5,
  "include_status_codes": [502, 503, 504]
}
Severity: Critical
Cooldown: 30 minutes
Channels: Email + SMS + Slack

Expected behavior:

3 failures → Alert triggers
Notifications sent to all channels
Team investigates
Issue resolved or escalated to Stripe
30 minutes later, if still failing, alert again

Scenario 2: Gradual Performance Degradation

Problem: Database queries getting slower over time Alert Configuration:

Alert Type: High Latency
Condition: {
  "threshold_ms": 1000,
  "percentile": 95,
  "window_minutes": 10,
  "min_requests": 20
}
Severity: Medium
Cooldown: 60 minutes
Channels: Email + Slack

Expected behavior:

P95 latency hits 1.5s → Alert triggers
Team reviews logs
Identifies slow query
Optimizes or adds caching
Latency returns to normal

Scenario 3: Security Incident

Problem: Unknown IP trying to access internal API Alert Configuration:

Alert Type: Unauthorized Client
Condition: {
  "immediate": true,
  "threshold": 1,
  "window_minutes": 5
}
Severity: High
Cooldown: 60 minutes
Channels: Email + SMS + Slack

Expected behavior:

Unauthorized IP makes request → Immediate alert
Security team reviews
IP blocked if malicious
Client contacted if legitimate (e.g., moved servers)

Scenario 4: Rate Limit Abuse

Problem: Client’s script gone rogue, hitting rate limits Alert Configuration:

Alert Type: Rate Limit Exceeded
Condition: {
  "threshold_percent": 80,
  "window_minutes": 10
}
Severity: Medium
Cooldown: 30 minutes
Channels: Email

Expected behavior:

Client reaches 80% of rate limit within 10 min → Alert
Review which client
Contact client
They fix infinite loop
Rate limit stops triggering

Troubleshooting

Issue: “Alert not triggering”

Check:

Alert is enabled (not disabled)
Route is active (not disabled)
Condition threshold is correct (not too high)
Check alert state (might be in cooldown)

Debug:

Route: stripe-payments
Alert: Stripe Integration Failure
State: Cooldown (15 minutes remaining)
Last triggered: 10 minutes ago

Alert won’t trigger again until cooldown expires

Issue: “Too many notifications”

Cause: Threshold too low or cooldown too short Fix:

Increase threshold: 1 → 5
Increase cooldown: 5 minutes → 30 minutes
Increase window: 1 minute → 5 minutes

Issue: “Not receiving email notifications”

Check:

Email addresses correct (no typos)
Check spam folder
Email channel enabled
Alert logs show “sent” status

Debug: Navigate to Alert Logs → Find trigger event → Check delivery status

Issue: “Slack notifications not working”

Check:

Webhook URL correct (starts with https://hooks.slack.com)
Slack channel enabled in alert config
Webhook not revoked in Slack settings

Test webhook:

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H "Content-Type: application/json" \
  -d '{"text": "Test from KnoxCall"}'

Should see message in Slack channel.

API Logs: View detailed request history that triggered alerts
Analytics: Visualize trends and patterns in alert triggers
Audit Logs: Track who created/modified alerts

Next Steps

API Logs

View requests that triggered alerts

Analytics

Analyze alert patterns and trends

Audit Logs

Track alert configuration changes

Routes

Configure routes to monitor

📊 Statistics

Level: beginner to intermediate
Time: 15 minutes

🏷️ Tags

alerts, monitoring, notifications, incidents, email, sms, slack

​Alerts & Notifications

​What Are Alerts?

​Alert Types

​1. Request Failures

​2. High Latency

​3. Rate Limit Exceeded

​4. Unauthorized Client

​5. Vault Destination Monitoring (automatic)

​Notification Channels

​Email Notifications

​SMS Notifications

​Slack Notifications

​Creating Your First Alert

​Step 1: Navigate to Alerts

​Step 2: Choose a Template

​Step 3: Configure Alert

​Step 4: Configure Notifications

​Enable Email

​Enable SMS (optional, Enterprise)

​Enable Slack (optional)

​Step 5: Custom Message Templates (Optional)

​Step 6: Create Alert

​Alert States

​1. OK (Green)

​2. Triggered (Red)

​3. Cooldown (Yellow)

​Advanced Configuration

​Condition Schemas

​Request Failures Schema

​High Latency Schema

​Rate Limit Exceeded Schema

​Unauthorized Client Schema

​Multi-Status Code Filtering

​Latency Percentiles

​Alert Management

​Viewing Alert Status

​Viewing Alert Details

​Editing Alerts

​Disabling Alerts

​Deleting Alerts

​Alert Logs

​Best Practices

​1. Start with Templates

​2. Set Appropriate Thresholds

​3. Use Severity Correctly

​4. Configure Cooldowns

​5. Use Multiple Notification Channels

​6. Test Your Alerts

​7. Monitor Alert Logs

​8. Document Your Alerts

​Common Alert Scenarios

​Scenario 1: Backend API Outage

​Scenario 2: Gradual Performance Degradation

​Scenario 3: Security Incident

​Scenario 4: Rate Limit Abuse

​Troubleshooting

​Issue: “Alert not triggering”

​Issue: “Too many notifications”

​Issue: “Not receiving email notifications”

​Issue: “Slack notifications not working”

​Related Features

​Next Steps

API Logs

Analytics

Audit Logs

Routes

📊 Statistics

🏷️ Tags

Alerts & Notifications

What Are Alerts?

Alert Types

1. Request Failures

2. High Latency

3. Rate Limit Exceeded

4. Unauthorized Client

5. Vault Destination Monitoring (automatic)

Notification Channels

Email Notifications

SMS Notifications

Slack Notifications

Creating Your First Alert

Step 1: Navigate to Alerts

Step 2: Choose a Template

Step 3: Configure Alert

Step 4: Configure Notifications

Enable Email

Enable SMS (optional, Enterprise)

Enable Slack (optional)

Step 5: Custom Message Templates (Optional)

Step 6: Create Alert

Alert States

1. OK (Green)

2. Triggered (Red)

3. Cooldown (Yellow)

Advanced Configuration

Condition Schemas

Request Failures Schema

High Latency Schema

Rate Limit Exceeded Schema

Unauthorized Client Schema

Multi-Status Code Filtering

Latency Percentiles

Alert Management

Viewing Alert Status

Viewing Alert Details

Editing Alerts

Disabling Alerts

Deleting Alerts

Alert Logs

Best Practices

1. Start with Templates

2. Set Appropriate Thresholds

3. Use Severity Correctly

4. Configure Cooldowns

5. Use Multiple Notification Channels

6. Test Your Alerts

7. Monitor Alert Logs

8. Document Your Alerts

Common Alert Scenarios

Scenario 1: Backend API Outage

Scenario 2: Gradual Performance Degradation

Scenario 3: Security Incident

Scenario 4: Rate Limit Abuse

Troubleshooting

Issue: “Alert not triggering”

Issue: “Too many notifications”

Issue: “Not receiving email notifications”

Issue: “Slack notifications not working”

Related Features

Next Steps