Skip to main content

Secret Rotation and Management

Learn how to rotate secrets safely, organize them effectively, and manage per-environment credentials.

Why Rotate Secrets?

Security reasons:
  • Limit blast radius if compromised
  • Meet compliance requirements (PCI DSS, SOC 2)
  • Best practice for production systems
Recommended schedule:
  • Critical secrets (production API keys): Every 30 days
  • Standard secrets: Every 90 days
  • Database passwords: Every 90 days
  • OAuth tokens: When revoked by provider

Safe Rotation Process

Step-by-Step

1. Create new secret with different name:
Old: stripe_api_key
New: stripe_api_key_v2
2. Update routes to use new secret:
{
  "Authorization": "Bearer {{secret:stripe_api_key_v2}}"
}
3. Test thoroughly in staging first 4. Deploy to production 5. Monitor for 24-48 hours 6. Delete old secret when confirmed working

Why This Works

  • Zero downtime
  • Easy rollback (keep old secret temporarily)
  • Clear audit trail

What NOT to Do

Don’t try to “update” a secret
  • You can’t update values (by design)
  • Must create new secret
Don’t delete immediately
  • Keep old secret for 48 hours
  • Allows rollback if issues

Environment-Specific Secrets

KnoxCall supports two approaches for managing secrets across environments: Best for: Single secret with different values per environment Create one secret with environment-specific values:
Secret: stripe_api_key

Environments:
  production:  sk_live_abc123xyz789
  staging:     sk_test_staging_456def
  development: sk_test_dev_789ghi
How to configure:
  1. Create a secret: stripe_api_key
  2. In the secret detail page, select environment from dropdown
  3. Add value for each environment:
    • Switch to “production” → Enter live key
    • Switch to “staging” → Enter test key
    • Switch to “development” → Enter dev key
Using in routes:
{
  "Authorization": "Bearer {{secret:stripe_api_key}}"
}
Behavior:
  • Route in “production” environment → Uses production value
  • Route in “staging” environment → Uses staging value
  • Hard fail if missing: Route returns error if secret lacks the environment value
  • No fallbacks or defaults (prevents accidental production key usage in dev)
Benefits:
  • ✅ One secret name across all environments
  • ✅ Impossible to accidentally use wrong key
  • ✅ Clear environment isolation
  • ✅ Simplified route configuration

Approach 2: Naming Convention (Legacy)

Best for: Backward compatibility or when you prefer separate secrets Create different secrets per environment:
Secrets:
  stripe_production_api_key
  stripe_staging_api_key
  stripe_development_api_key
Production route:
{
  "Authorization": "Bearer {{secret:stripe_production_api_key}}"
}
Staging route:
{
  "Authorization": "Bearer {{secret:stripe_staging_api_key}}"
}
Drawbacks:
  • ❌ Must update route config when changing environments
  • ❌ Possible to reference wrong secret in wrong environment
  • ❌ More secrets to manage

Comparison

FeatureEnvironment ConfigsNaming Convention
Secret count1 per service3+ per service
Route configSame everywhereDifferent per env
SafetyHard fail if missingSilent failure possible
Ease of use⭐⭐⭐⭐⭐⭐⭐⭐
Recommended✅ YesLegacy support
Migration tip: Use environment configs for new secrets, migrate old ones gradually

Organization Strategies

Group by Service

stripe_api_key
stripe_webhook_secret
stripe_publishable_key

sendgrid_api_key
sendgrid_webhook_key

aws_access_key_id
aws_secret_access_key
aws_region

Group by Environment

production_stripe_key
production_sendgrid_key
production_database_url

staging_stripe_key
staging_sendgrid_key
staging_database_url
Choose one strategy and be consistent!

Multiple Secrets in One Route

Inject multiple credentials:
{
  "Authorization": "Bearer {{secret:stripe_api_key}}",
  "X-Partner-Key": "{{secret:partner_api_key}}",
  "X-Database": "{{secret:database_url}}"
}
All injected server-side, none exposed to client.

Monitoring Secret Usage

Track Usage

In the Secrets page, check:
  • Last Used - When secret was last injected
  • Associated Routes - Which routes use this secret

Find Unused Secrets

Look for:
  • Last used > 30 days ago
  • No associated routes
  • Test/temporary secrets
Delete these to reduce attack surface.

Set Up Alerts

Get notified when:
  • Secret not used in 30+ days
  • Multiple failed auth attempts (might indicate compromised key)
  • Secret approaching rotation date

Common Patterns

API Key Rotation

1. Generate new key from provider (Stripe, SendGrid, etc)
2. Create secret: service_key_v2
3. Update routes to use v2
4. Test thoroughly
5. Delete old secret after 48h

Database Password Rotation

1. Create new password in database
2. Create secret: db_password_v2
3. Update routes to use v2
4. Test all database connections
5. Remove old password from database
6. Delete old secret

OAuth Token Refresh

1. Refresh token via OAuth flow
2. Create secret: service_oauth_token_v2
3. Update routes
4. Test authentication
5. Revoke old token with provider
6. Delete old secret

Troubleshooting

Routes failing after rotation:
  • Verify new secret value is correct
  • Check all routes were updated
  • Look for cached old values
  • Roll back to old secret if needed
“Secret not found” errors:
  • Check spelling (case-sensitive)
  • Verify secret exists in Secrets page
  • Confirm route is using correct name
Third-party API returns 401:
  • New secret value might be wrong
  • API key might be expired/revoked
  • Test secret value directly with provider

Rotation Checklist

Use this checklist for each rotation:
  • Document which routes use this secret
  • Generate new credential from provider
  • Create new secret with v2 suffix
  • Update all routes in staging
  • Test thoroughly in staging
  • Update routes in production
  • Monitor for 48 hours
  • Verify no errors
  • Delete old secret
  • Update documentation
  • Schedule next rotation (calendar)

Next Steps

  • Set up environments for per-env secrets
  • Learn about securing your routes
  • Configure monitoring and alerting
Need help? Use the support chat!

📊 Statistics

  • Level: intermediate
  • Time: 10 minutes

🏷️ Tags

secrets, security, rotation, management