Designing for High Availability – Resiliency & Contingency Planning

Enterprise customers – particularly in public sector, finance, healthcare, and critical infrastructure – increasingly face regulatory or governance requirements demanding documented vendor contingency strategies. This is not about distrust of Cloudflare’s resilience; it’s about demonstrating due diligence and risk management to auditors, boards, and compliance bodies.

TL;DR: Cloudflare’s native resilience (anycast, Load Balancing, health checks) handles most availability needs – fallback systems only make sense for compliance-level guarantees. Bypassing Cloudflare via DNS is possible but removes WAF / DDoS protection and exposes origin servers. If contingency is required, use Active-Active (Primary-Primary) DNS with both Cloudflare and an external provider in NS records – traditional read-only Secondary DNS cannot modify records during outages. Additional requirements: publicly trusted TLS certificates, origin firewalls, Infrastructure as Code, and tested runbooks. Platform providers serving many customers need API-first automation; those with heavy edge logic (Workers / KV / DO) may find bypass doesn’t restore functionality. For most organizations, the security risk and operational complexity of bypass outweigh the rare benefit of avoiding brief outages — waiting for restoration is usually the safer choice.

When Contingency Planning Actually Matters

#

Regulatory Drivers

#

• Financial Services: Central Banks or Governments usually mandate operational resilience frameworks with vendor concentration risk assessments.
• Public Sector: often reference sovereignty controls, documented exit capabilities.
• Critical Infrastructure: Network and Information Security Directive 2.0 (NIS2) mentions having robust redundancy measures in place for critical systems.
• Healthcare: US HIPAA business associate agreements and other data residency requirements usually drive contingency planning for protected health information (PHI) systems.

Reality Check

#

Despite these drivers, security / operational cost of parallel infrastructure often exceeds vendor dependency risk. Most organizations are better served by:

• Maximizing Cloudflare’s native resilience features.
• Maintaining strong Cloudflare Enterprise account team or Partner relationships.
• Accepting brief degradation preferable to exposing unprotected origins.
• Investing in origin resilience and disaster recovery.

Cloudflare’s Standard Architecture Is Already Resilient

#

Cloudflare separates its Control Plane (management / API / Dashboard) from its Data Plane (traffic flow / Edge), ensuring traffic continues to flow even if the management dashboard is unavailable. Cloudflare’s global anycast network is architected for resilience – every server in every data center shares the same tech stack and announces IP addresses via anycast, providing inherent redundancy.

Before discussing contingency strategies, recognize that a properly configured Cloudflare deployment already provides exceptional resilience:

• Anycast routing: Traffic automatically routes to the nearest healthy data center.

• N+x redundancy: Multiple servers (Multi-Colo-PoPs) in each location provide failover without manual intervention.

• Control / Data Plane separation: Existing configurations continue operating even during control plane degradation.

• Load Balancing with health checks: Automatic origin failover within Cloudflare.

• Serverless Runtime Workers for custom failover logic: Write programmable traffic routing based on origin health, retries, timeouts, and circuit breakers for dependencies for storage options.

• Geographic distribution and backbone: 13,000+ network interconnects with major ISPs and cloud providers.

Most availability requirements are met through proper Cloudflare configuration – using Load Balancing, health checks, traffic steering, and multiple origins – rather than multi-vendor complexity.

While Cloudflare strives for maximum resilience – constantly improving through learning and innovation – some customers require documented contingency (“break glass”) strategies to meet risk compliance, regulatory, or sovereignty requirements. This document provides a non-exhaustive high-level introduction to architectural patterns for designing failover capabilities while maintaining security posture.

The goal is not to offboard from Cloudflare, but to provide a potential safety net (backup plan) that allows customers to rely on the platform with confidence.

Critical Self-Risk Assessment

#

Before activating any bypass strategy or failover plan, organizations must answer:

At what point does the security risk of exposing infrastructure outweigh the downtime of waiting for service restoration?

Considerations:

• Loss of WAF / DDoS protection: Disabling Cloudflare exposes origin IPs directly to the Internet.
• Integration breakage: Third-party scripts and integrations, Workers, Load Balancing logic, and SaaS integrations may fail – if they are also using Cloudflare.
• Attack surface exposure: Malicious actors monitor DNS changes; bypassing protections during outages creates potential exploitation windows.
• Operational cost: Maintaining parallel infrastructure, training staff on multiple platforms, and designing applications for vendor-agnostic operation requires significant investment, time and additional resources.
• Configuration drift: Multi-vendor setups introduce complexity in maintaining policy parity, certificate management, and configuration coordination.

For most organizations, the answer is: wait for service restoration. The cost of maintaining bypass / a backup infrastructure, the security risk of exposure, and the complexity of multi-vendor operations exceeds the cost of brief service degradation.

Before implementing multi-vendor strategies, exhaust Cloudflare’s native resilience capabilities.

Part 1: Application Services (Reverse Proxy / CDN / WAF)

#

The primary mechanism for circumventing Cloudflare reverse proxy for application traffic relies on DNS architecture and origin security.

Diagram: Emergency Bypass Flow

#

┌────────────────────────────────────────────────────────────────────┐
│                        DNS RESOLUTION LAYER                        │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│    User ──► DNS Query ──► Authoritative DNS                        │
│                               │                                    │
│              ┌────────────────┴────────────────┐                   │
│              ▼                                 ▼                   │
│    ┌─────────────────────┐          ┌─────────────────────┐        │
│    │   PATH A: STANDARD  │          │  PATH B: EMERGENCY  │        │
│    │   (Recommended)     │          │  (Contingency)      │        │
│    └─────────┬───────────┘          └─────────┬───────────┘        │
│              ▼                                 ▼                   │
│    Cloudflare Anycast IPs           Backup Provider IP             │
│              │                      OR Direct Origin IP            │
│              ▼                                 │                   │
│    ┌─────────────────────┐                     │                   │
│    │  Cloudflare Edge    │                     │                   │
│    │  • DDoS Protection  │                     │                   │
│    │  • Bot Management   │                     │                   │
│    │  • WAF              │                     │                   │
│    │  • Rate Limiting    │                     │                   │
│    │  • CDN/Cache        │                     │                   │
│    └─────────┬───────────┘                     │                   │
│              │                                 │                   │
│              └────────────┬────────────────────┘                   │
│                           ▼                                        │
│                    Origin Server(s)                                │
│                    (Must have publicly trusted TLS certs)          │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

DECISION POINT: Switch occurs at the Authoritative DNS layer

Architecture Components & Failover Steps

#

DNS Setup Options

#

Take into account DNS TTL.

Understanding Zone Transfer Directions

#

Critical Distinction: “Secondary DNS” traditionally means read-only zone transfers. For contingency planning, you need write access to modify DNS records during an outage. This requires either:

• Active-Active (Primary-Primary) configuration with bidirectional sync, OR
• Cloudflare as Secondary DNS, OR
• CNAME setup where you control the external authoritative DNS

Multi-Vendor Architecture Options

#

For organizations with resources to maintain parallel infrastructure:

┌──────────────────────────────────────────────────────────────────────┐
│                    MULTI-VENDOR DNS LOAD BALANCING                   │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│                      External DNS Provider                           │
│                      (Route53 / Azure DNS)                           │
│                             │                                        │
│            ┌────────────────┴────────────────┐                       │
│            ▼                                 ▼                       │
│   ┌─────────────────┐               ┌─────────────────┐              │
│   │   Cloudflare    │               │  Backup Vendor  │              │
│   │   (Primary)     │               │  (Fallback)     │              │
│   │                 │               │                 │              │
│   │  Full feature   │               │  Baseline       │              │
│   │  set enabled    │               │  protection     │              │
│   └────────┬────────┘               └────────┬────────┘              │
│            │                                 │                       │
│            └────────────┬────────────────────┘                       │
│                         ▼                                            │
│                   Origin Server(s)                                   │
│                                                                      │
│   Traffic Distribution: Health-check based, performance-based,       │
│                         or weighted round-robin                      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Configuration Management: Maintain parity via Terraform across providers. API-first approach enables automated synchronization.

• Active-Active (Recommended): Both vendors receive traffic continuously. Provides ongoing signal for security tools (i.e. Bot Management, Rate Limiting). Configuration complexity manageable if traffic split is maintained.

• Active-Passive: One vendor receives all traffic normally; backup only used during incidents. Higher cutover risk due to cold configuration and lack of baseline traffic for Machine Learning (ML)-based security features.

Operational Considerations:

• Cache purge and security coordination across vendors required.
• Certificate management complexity (same Custom Certificates on both platforms).
• Log aggregation and normalization to common SIEM format.
• Configuration drift monitoring and automated reconciliation.
• Only really justified for extreme availability requirements or regulatory mandates.

Part 2: SASE (Zero Trust) & Network Services

#

For Zero Trust (WARP Device Client / Secure Web Gateway) and Network services (Magic Transit), contingency planning can be more complex as these services are deeply integrated into employee workflows and network infrastructure.

• “Shadow VPN” Strategy: Pre-deployed but dormant legacy VPN infrastructure (OpenVPN, etc.) that can be activated if Cloudflare Zero Trust becomes unavailable. Requires maintaining separate authentication, DNS, and network routing configurations.

Diagram: SASE Failover Logic

#

┌───────────────────────────────────────────────────────────────────────┐
│                     ZERO TRUST FAILOVER DECISION TREE                 │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│                        ┌─────────────────┐                            │
│                        │  WARP Client    │                            │
│                        │  (User Device)  │                            │
│                        └────────┬────────┘                            │
│                                 │                                     │
│                        ┌────────▼────────┐                            │
│                        │ Service Status? │                            │
│                        └────────┬────────┘                            │
│                    ┌────────────┴────────────┐                        │
│                    ▼                         ▼                        │
│           ┌──────────────┐          ┌──────────────┐                  │
│           │    NORMAL    │          │   FAILURE    │                  │
│           └──────┬───────┘          └──────┬───────┘                  │
│                  ▼                         │                          │
│    ┌─────────────────────┐      ┌──────────┴──────────┐               │
│    │   WARP Tunnel       │      ▼                     ▼               │
│    │        │            │  ┌─────────┐        ┌──────────┐           │
│    │        ▼            │  │FAIL OPEN│        │FAIL CLOSE│           │
│    │ Cloudflare Gateway  │  │(Trigger)│        │(Default) │           │
│    │        │            │  └────┬────┘        └────┬─────┘           │
│    │        ▼            │       ▼                  ▼                 │
│    │ Internet / Company  │  Direct Internet     Block All             │
│    │ Applications        │  (No filtering)      Traffic               │
│    └─────────────────────┘  HIGH AVAILABILITY   HIGH SECURITY         │
│                             LOW SECURITY        ZERO AVAILABILITY     │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────────┐
│                      MAGIC TRANSIT FAILOVER                                │
├────────────────────────────────────────────────────────────────────────────┤
│                                                                            │
│   Normal State:                                                            │
│   Customer IP Prefix ──► Cloudflare BGP Announcement ──► DDoS Scrubbing    │
│                               ──► GRE / IPsec Tunnel ──► Customer Network  │
│                                                                            │
│   Failure State:                                                           │
│   Withdraw BGP from Cloudflare ──► Announce via ISP directly               │
│                                                                            │
│   ⚠ WARNING: Direct ISP announcement removes Cloudflare DDoS protection    │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

Architecture Components & Failover Steps

#

Practical Sample Action Plan for Cloudflare L7 Application Services

#

Context and Scope: This sample action plan assumes the outage affects Cloudflare Layer-7 (HTTP/HTTPS) reverse-proxy / application services. Mitigations differ for other Cloudflare products (Magic Transit, Spectrum, Zero Trust, etc.). The guidance below addresses immediate operational steps, risks, and configuration details for bypassing Cloudflare so traffic reaches origins directly.

Develop and implement a tailored internal action plan and resiliency policies for each vendor you work with.

Immediate Checks (First 60–120 Seconds)

#

1. Confirm Cloudflare outage via Cloudflare Status and public reports
2. Verify Cloudflare API accessibility (API responsiveness required for fast programmatic toggles)
3. Identify which DNS records are proxied (orange cloud / proxied: true) vs DNS-only
4. Verify origin public accessibility, capacity (CPU, connections, bandwidth, autoscaling status) and security (firewall)

Fastest Mitigation (Not Recommended): Disable the Reverse Proxy (DNS Stays on Cloudflare)

#

Action: Set affected DNS records from proxied → DNS only (orange cloud → grey cloud).

Effect: Cloudflare continues serving DNS responses but no longer proxies or terminates TLS/HTTP; clients resolve names to origin IPs directly.

How to Execute:

Via Dashboard:

DNS → Select Record → Toggle Proxy Status → Save

Via API (example for single record):

curl -X PUT "https://api.cloudflare.com/client/v4/zones/{ZONE_ID}/dns_records/{RECORD_ID}" \
  -H "Authorization: Bearer {API_TOKEN}" \
  -H "Content-Type: application/json" \
  --data '{
    "type": "A",
    "name": "www.example.com",
    "content": "198.51.100.4",
    "ttl": 120,
    "comment": "Disabled due to XYZ – write a recognizable comment for auditing",
    "proxied": false
  }'

Post-Change Verification:

• Confirm proxied: false via GET /zones/{zone_id}/dns_records or dig +short to ensure responses are origin IPs
• Test origin responds to HTTP(S) directly; validate TLS handshake and application behavior
• Monitor origin resource utilization (CPU, memory, connections)

Alternative When Cloudflare API is Unavailable: Change DNS to Point Directly to Origin

#

If you operate external DNS (or secondary DNS provider) with CNAME setup:

• Update DNS entries to point to origin host or origin IPs directly
• Replace CNAME target with origin A/AAAA record or CNAME to origin hostname

Considerations:

• If zone is Cloudflare authoritative DNS, switching to another provider requires NS record changes at registrar (not fast during outage – requires pre-planning)
• DNS TTL matters: Long TTLs slow propagation. Use 60–300s TTLs in normal operation for faster failover
• Secondary DNS with zone transfers enables rapid failover if pre-configured

TLS, Certificates and Ports

#

When traffic bypasses Cloudflare:

Certificate Requirements:

• Origin must present valid certificate accepted by clients
• Ensure origin has public CA certificate (not Cloudflare Origin CA only)
• Clients will validate certificate directly; mismatched SANs or expired certs cause failures

Protocol Support:

• Verify origin supports SNI, ALPN, HTTP/2 if clients expect these
• Cloudflare supports non-standard ports; ensure origin listens on same ports clients use

Security and Operational Risks When Bypassing Cloudflare

#

Immediate Loss of Protections:

• WAF, rate limiting, bot management, DDoS mitigation no longer active
• Origin IPs exposed to scanning and direct attack if previously hidden
• Review and update origin firewall rules, fail2ban configurations, and ACLs

Capacity Concerns:

• Predictable surge in traffic and connections
• Monitor resource usage and enable autoscaling if available
• Consider connection limits at origin

Observability:

• Ensure direct-to-origin logs are collected
• Adjust alerting thresholds for increased baseline traffic

Pre-Incident Preparedness Checklist

#

Implement these measures before an outage occurs:

✅ Disaster Runbook: Document exact API calls and operator steps to toggle proxied flags and update DNS records. Keep API tokens secure and accessible.

✅ DNS Resilience:

• Maintain low TTLs (60–300s) for critical records
• Configure tested secondary DNS provider with pre-staged records

✅ Origin TLS:

• Deploy publicly valid TLS certificates with automated renewal
• Keep Cloudflare Origin CA certs only if you also maintain public CA cert for failover

✅ Origin DDoS Protections:

• Implement network ACLs, upstream scrubbing, provider mitigations as fallback
• Use iptables to allowlist only Cloudflare IPs normally, but have rules ready to open during bypass

✅ Health-Check Driven Failover:

• Use DNS provider supporting active/passive failover
• Test failover quarterly / periodically

✅ Multi-CDN Architecture (for extreme requirements):

• Consider active-passive or active-active with traffic steering at DNS or load balancer layer
• Review multi-vendor reference architecture

✅ DNS Architecture (Choose ONE):

Option A - Active-Active (Primary-Primary) (Most robust, higher complexity):

• Configure bidirectional zone synchronization
• Both Cloudflare and external provider in NS records at registrar
• Verify you can modify at either provider and changes sync
• Test emergency DNS modifications at both providers periodically

Option B - Cloudflare as Secondary (Simpler, recommended for most):

• Configure external provider as primary (read-write)
• Set up zone transfers: External → Cloudflare
• Add both providers’ nameservers to NS records at registrar
• Can use Secondary DNS Override to proxy specific records
• Verify you can modify at external primary and changes sync to Cloudflare
• Test emergency DNS modifications at external primary periodically

✅ Registrar Independence:

• Critical: If using Cloudflare Registrar, transfer critical revenue-generating domains to external registrar
• Maintain registrar credentials separately from Cloudflare account
• Store MFA backup codes securely
• Test registrar login and NS change procedures periodically

✅ Edge Logic Assessment:

• Document all application logic implemented in Workers, KV, Durable Objects, R2
• Identify which features will be unavailable during bypass
• For critical features: implement fallback logic at origin OR accept temporary unavailability
• Create feature degradation communication templates for users
• Test application behavior with edge logic disabled

✅ Platform Provider Scale (if serving multiple customers):

• Build API-first automation for bulk DNS/configuration changes
• Pre-provision TLS certificates for critical customers at backup provider
• Implement tiered customer approach (critical vs. standard)
• Create staged rollout procedures
• Test automation on pilot customers periodically

Rollback and Verification After Cloudflare Recovery

#

Re-Enable Protections:

1. Verify Cloudflare services healthy via Status page and test requests
2. Re-enable proxying (proxied: true) for records previously disabled
3. Or re-point authoritative DNS back to Cloudflare if NS records were changed
4. Re-verify origin security rules to allow Cloudflare

Validation:

• Test TLS termination, WAF rules, bot management features
• Review origin access logs and Cloudflare Logs and Analytics to confirm traffic routing normalized
• Verify security features (rate limiting, firewall rules) are active

Post-Incident Review:

• Document actual time to failover vs. RTO targets
• Identify gaps in security, runbook or tooling
• Update incident response procedures with lessons learned

Summary: Technical Tradeoffs

#

Key Insight: Preparation (low TTLs, secondary DNS, origin certs, autoscaling, origin security controls) reduces impact and decreases time to recovery. The fastest mitigations require the most preparation.

Monitoring & Incident Detection

#

Independent monitoring is essential for informed failover decisions.

Summary: Operational Discipline

#

Resilience is not a one-time setup but an ongoing discipline.

Resilience Hierarchy

#

Prioritize resilience strategies in this order:

1. Cloudflare Native Resilience (Primary): Load Balancing, health checks, multiple origins, Workers / Snippets-based failover, Custom Errors.

2. Multi-Vendor DNS (Secondary): External authoritative DNS with Cloudflare as Secondary, or Partial (CNAME) setup.

3. Multi-Vendor Security Proxy (Tertiary): Only for extreme compliance requirements; introduces significant operational overhead.

Key Principles

#

1. Own Your Control Points: Unfettered, secure access to Domain Registrar and MDM platform is non-negotiable, following a principle of least privilege.

2. Infrastructure as Code: Manage all configurations via API / Terraform. Enable rapid, audited, transferable changes. Prevent self-inflicted outages.

3. Secure Origins for Bypass: Ensure origins have publicly trusted SSL certificates and robust security posture (i.e. using iptables) independent of Cloudflare protections.

4. Monitor Externally: Gain unbiased view of service health from user perspective.

5. Test Playbooks: An untested incident response plan is just a paper. Regularly test “break glass” scenarios on non-production / staging subdomains.

6. Accept the Trade-off: For most organizations, the security risk of circumventing Cloudflare exceeds the cost of temporary service degradation. Design for this reality.

7. Configuration Hygiene: Rigorous change management with approval workflows, staging environment testing, and rollback plans prevents self-inflicted incidents.

Recommended Artifacts

#

• Documented runbooks with step-by-step failover procedures for all involved teams.
• Terraform / IaC templates for “Emergency Bypass” configurations, applying the principle of least privilege.
• Pre-staged DNS records (inactive) for rapid failover.
• “Shadow VPN” infrastructure (dormant) for Zero Trust contingency.
• Company-wide communication plans and escalation paths.
• Periodic live failover exercises (not just tabletop): Simulate vendor failures in controlled environments, measure actual traffic rerouting times, and refine response processes under realistic conditions.
• Post-incident review (iteration) process including:
  • What protections were bypassed (WAF, bot management, geo-blocking, etc.) and duration
  • Emergency DNS / routing changes made and approval chains
  • Shadow IT emergence (personal devices, home networks, unsanctioned SaaS)
  • Temporary services stood up “just for now” that became permanent
  • Documented unwinding plan for emergency changes
  • Intentional fallback plan for next incident vs. decentralized improvisation (continuous improvement)

Related Resources

#

• Multi-vendor Application Security and Performance Reference Architecture
• Partial (CNAME) setup
• Cloudflare as Secondary DNS provider
• Cloudflare Status

Disclaimer

#

Educational purposes only.

This blog post is independently created and is not affiliated with, endorsed by, or necessarily representative of the views or opinions of any organizations or services mentioned herein.

The guidelines provided in this post are intended for general educational purposes. They should be customized to fit your specific use cases and situation. You are responsible for configuring settings according to your unique requirements, and it is important to understand their potential impact. Familiarity with Cloudflare concepts such as Phases, Proxy Status, and other relevant features is recommended.

The author of this post is not responsible for any misconfigurations, errors, or unintended consequences that may arise from implementing the guidelines or recommendations discussed herein. You assume full responsibility for any actions taken based on this content and for ensuring that configurations are appropriate for your specific environment.

The images used in this article primarily consist of screenshots from the Cloudflare Dashboard or other publicly available materials, such as Cloudflare webinar slides.