How to use Cloudflare for Service Discovery

Cloudflare runs 3,588 containers, making up 1,264 apps and services that all need to be able to find and discover each other in order to communicate -- a problem solved with service discovery.

You can use Cloudflare for service discovery. By deploying microservices behind Cloudflare, microservices’ origins are masked, secured from DDoS and L7 exploits and authenticated, and service discovery is natively built in. Cloudflare is also cloud platform agnostic, which means that if you have distributed infrastructure deployed across cloud platforms, you still get a holistic view of your services and the ability to manage your security and authentication policies in one place, independent of where services are actually deployed.

Service locations and metadata are stored in a distributed KV store deployed in all 100+ Cloudflare edge locations (the service registry).

Services register themselves to the service registry when they start up and deregister themselves when they spin down via a POST to Cloudflare’s API. Services provide data in the form of a DNS record, either by giving Cloudflare the address of the service in an A (IPv4) or AAAA (IPv6) record, or by providing more metadata like transport protocol and port in an SRV record.

Services are also automatically registered and deregistered by health check monitors so only healthy nodes are sent traffic. Health checks are over HTTP and can be setup with custom configuration so that responses to the health check must return a specific response body and or response code otherwise the nodes are marked as unhealthy.

Traffic is distributed evenly between redundant nodes using a load balancer. Clients of the service discovery query the load balancer directly over DNS. The load balancer receives data from the service registry and returns the corresponding service address. If services are behind Cloudflare, the load balancer returns a Cloudflare IP address to route traffic to the service through Cloudflare’s L7 proxy.

Traffic can also be sent to specific service nodes based on client geography, so the data replication service in North America, for example, can talk to a specific North American version of the billing service, or European data can stay in Europe.

Clients query the service registry over DNS, and service location and metadata is packaged in A, AAAA, CNAME or SRV records. The benefit of this is that no additional client software needs to be installed on service nodes beyond a DNS client. Cloudflare works natively over DNS, meaning that if your services have a DNS client, there’s no extra software to install, manage, upgrade or patch.

While usually, TTL’s in DNS mean that if a service location changes or deregisters, clients may still get stale information, Cloudflare DNS keeps low TTL’s (it’s able to do this and maintain fast performance because of its distributed network) and if you are using Cloudflare as a proxy, the DNS answers always point back to Cloudflare even when the IP’s of services behind Cloudflare change, removing the effect of cache staleness.

If your services communicate over HTTP/S and websockets, you can additionally use Cloudflare as a L7 proxy for added security, authentication and optimization. Cloudflare prevents DDoS attacks from hitting your infrastructure, masks your IP’s behind its network, and routes traffic through an optimized edge PoP to edge PoP route to shave latency off the internet.

Once service <--> service traffic is going through Cloudflare, you can use TLS client certificates to authenticate traffic between your services. Cloudflare can authenticate traffic at the edge by ensuring that the client certificate presented during the TLS handshake is signed by your root CA.

Sign up for Cloudflare account

During the signup process, add all your initial services as DNS records in the DNS editor.

To finish sign up, move DNS to Cloudflare by logging into your registrar and changing your nameservers to the Cloudflare nameservers assigned to you when you signed up for Cloudflare. If you want traffic to those services to be proxied through Cloudflare, click on the cloud next to each DNS record to make it orange.

Run a script on each node so that:

On startup, the node sends a POST to the DNS record API to register itself and PUT to load balancing API to add itself to the origin pool.

On shutdown, the node sends a DELETE to the DNS record API to deregister itself and PUT to load balancing API to remove itself to the origin pool.

These can be accomplished via startup and shutdown scripts on Google Compute Engine or user data scripts or auto scaling lifecycle hooks on AWS.

Registration:

curl -X POST "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"type":"SRV","data":{"service":"_http","proto":"_tcp","name":"name","priority":1,"weight":1,"port":80,"target":"staging.badtortilla.com"},"ttl":1,"zone_name":"badtortilla.com","name":"_http._tcp.name.","content":"SRV 1 1 80 staging.badtortilla.com.","proxied":false,"proxiable":false,"priority":1}'

De-Registration:

curl -X DELETE "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json"

Add or remove an origin from an origin pool (this should be a unique IP per node added to the pool):

curl -X PUT "https://api.cloudflare.com/client/v4/user/load_balancers/pools/17b5962d775c646f3f9725cbc7a53df4" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"description":"Primary data center - Provider XYZ","name":"primary-dc-1","enabled":true,"monitor":"f1aba936b94213e5b8dca0c0dbf1f9cc","origins":[{"name":"app-server-1","address":"1.2.3.4","enabled":true}],"notification_email":"someone@example.com"}'

Create a health check. You can do this in the API or in the Cloudflare dashboard (in the Load Balancer card).

curl -X POST "https://api.cloudflare.com/client/v4/organizations/01a7362d577a6c3019a474fd6f485823/load_balancers/monitors" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"type":"https","description":"Login page monitor","method":"GET","path":"/health","header":{"Host":["example.com"],"X-App-ID":["abc123"]},"timeout":3,"retries":0,"interval":90,"expected_body":"alive","expected_codes":"2xx"}'

Create an initial load balancer, either through the API or in the Cloudflare dashboard.

curl -X POST "https://api.cloudflare.com/client/v4/zones/699d98642c564d2e855e9661899b7252/load_balancers" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"description":"Load Balancer for www.example.com","name":"www.example.com","ttl":30,"fallback_pool":"17b5962d775c646f3f9725cbc7a53df4","default_pools":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196","00920f38ce07c2e2f4df50b1f61d4194"],"region_pools":{"WNAM":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"ENAM":["00920f38ce07c2e2f4df50b1f61d4194"]},"pop_pools":{"LAX":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"LHR":["abd90f38ced07c2e2f4df50b1f61d4194","f9138c5d07c2e2f4df57b1f61d4196"],"SJC":["00920f38ce07c2e2f4df50b1f61d4194"]},"proxied":true}'

(optional) Setup geographic routing rules. You can do this via API or in the Cloudflare dashboard.

curl -X POST "https://api.cloudflare.com/client/v4/zones/699d98642c564d2e855e9661899b7252/load_balancers" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"description":"Load Balancer for www.example.com","name":"www.example.com","ttl":30,"fallback_pool":"17b5962d775c646f3f9725cbc7a53df4","default_pools":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196","00920f38ce07c2e2f4df50b1f61d4194"],"region_pools":{"WNAM":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"ENAM":["00920f38ce07c2e2f4df50b1f61d4194"]},"pop_pools":{"LAX":["de90f38ced07c2e2f4df50b1f61d4194","9290f38c5d07c2e2f4df57b1f61d4196"],"LHR":["abd90f38ced07c2e2f4df50b1f61d4194","f9138c5d07c2e2f4df57b1f61d4196"],"SJC":["00920f38ce07c2e2f4df50b1f61d4194"]},"proxied":true}'

(optional) Setup Argo for faster PoP to PoP transit in the traffic app of the Cloudflare dashboard.

(optional) Setup rate limiting via API or in the dashboard

curl -X POST "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/rate_limits" \
     -H "X-Auth-Email: user@example.com" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json" \
     --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","disabled":false,"description":"Prevent multiple login failures to mitigate brute force attacks","match":{"request":{"methods":["GET","POST"],"schemes":["HTTP","HTTPS"],"url":"*.example.org/path*"},"response":{"status":[401,403],"origin_traffic":true}},"bypass":[{"name":"url","value":"api.example.com/*"}],"threshold":60,"period":900,"action":{"mode":"simulate","timeout":86400,"response":{"content_type":"text/xml","body":"<error>This request has been rate-limited.</error>"}}}'

(optional) Setup TLS client authentication. (Enterprise only) Send your account manager your root CA certificate and which options you would like enabled.