Service Discovery for NGINX Plus Using DNS SRV Records from Consul

Our previous blog post on service discovery withConsul discussed the importance of service discovery in distributed systems, including service-oriented and microservices architectures. It described how to use NGINX Plus’on‑the‑fly reconfiguration API to dynamically add or remove load‑balanced servers that are registered with Consul.

In this blog post, we will go over another method to dynamically reconfigure your upstream servers in NGINX Plus using DNSSRV records obtained fromConsul’s DNS interface. With this method, NGINX Plus periodically re‑resolves the service name using Consul DNS. If the list of IP addresses associated with the service has changed, NGINX Plus immediately starts load balancing across the updated group of servers.

Editor – Support for DNSSRVrecords was introduced in NGINX Plus R9. For an overview of all the new features in that release, seeAnnouncing NGINX Plus R9 on our blog.

Also check out our other blogs about service discovery for NGINX Plus:

• Service Discovery for NGINX Plus Using Consul APIs
• Service Discovery for NGINX Plus with etcd
• Service Discovery for NGINX Plus with ZooKeeper

To make it easier to combine upstream reconfiguration in NGINX Plus with Consul DNS, we’ve created a sample demo,consul-dns-srv-demo, with step‑by‑step instructions for creating the configuration described in this blog post. In this post, we will walk you through the proof of concept. Using tools likeDocker,Docker Compose, andHomebrew, you can spin up a Docker‑based environment in which NGINX Plus load balances HTTP traffic across a group of open source NGINX web servers, with all components running in separate Docker containers.

How the Demo Works

First we spin up a separate Docker container for each of the following apps:

• Consul – Performs service discovery and provides DNS service.
• Registrator – Registers services with Consul. Registrator monitors for containers being started and stopped and updates Consul when a container changes state.
• hello – Simulates a backend server that can be scaled up or down. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page listing the client IP address, the request URI, and the web server’s hostname, IP address, port, and local time. (The diagram shows four instances of this app.)
• NGINX Plus R9 and later – Load balances the backend server apps.

The NGINX Plus container listens on the public port 80, and the built‑in NGINX Plus dashboard on port 8080. The Consul container listens on ports 8300, 8400, 8500, and 53 (the last mapped to port 8600 on the Docker host, which listens for DNS queries over both TCP and UDP). Consul advertises the IP address of the Docker host (configured with the environment variableHOST_IP before starting the containers) for intercontainer communication.

Each time a new Docker container that has exposed ports is launched, Registrator registers the associated service with Consul. When a container quits or is removed, Registrator removes it from the Consul service catalog automatically.

By setting environment variables within the containers, we can be more explicit about how to register the services with Consul. For the hello container, we set the environment variableSERVICE_80_NAME to http to override the defaultservice name of the service object created within Registrator, which later gets proxied to the registry backend (Consul in our case). This tells Consul to return the information (IP address, port, priority, and weight) for all the hello containers identified by service namehttp when NGINX Plus sends a DNSSRV query using theservice=http parameter discussed in the next paragraph.

The NGINX Plus configuration fileapp.conf included in the demo configures DNS‑based load balancing with these directives:

• The resolver directive designates the container named consul, which is the Consul DNS service listening on port 53, as the DNS server. NGINX Plus obtains the IP address of theconsul container from the/etc/hosts file in the NGINX Plus container.
  The valid parameter means that NGINX Plus ignores the TTL in the records provided by Consul DNS and instead re‑resolves them every 10 seconds.
• The service parameter to theserver directive, introduced in NGINX Plus R9, enables NGINX Plus to read DNS SRV records and get not only the IP addresses forservice.consul from the name server, but also port numbers, weights, and priorities. Consul doesn’t currently supportweights and priorities, so in the demo NGINX Plus uses just the port numbers of the backend servers registered with Consul.

resolver consul:53 valid=10s;

upstream backend {

    zone upstream_backend 64k;

    server service.consul service=http resolve;

}

server {

    ...

    location / {

        proxy_pass http://backend;

    }

}

Another feature introduced in NGINX Plus R9 facilitates the use ofSRV records – support for DNS lookups over TCP.SRV records are larger than other types of DNS records, making it more likely that the complete DNS response won’t fit in a single 512 byte UDP datagram. The DNS server flags an incomplete response as “truncated”, and NGINX Plus immediately retries the request over TCP. (This feature also makes it easy for NGINX Plus to handle the very large deployments of servers that are common in microservices applications.) Note that you must also configure Consul to set the truncated flag when appropriate, by setting theenable_truncate flag. In the demo, we do this in the consul_dns_config.json file.

Summary

Using Consul’s DNS interface to dynamically reconfigure your upstream servers in NGINX Plus is an alternative to using the NGINX Plus’ on‑the‑fly reconfiguration API, and is useful for organizations that frequently need to change the configuration of upstream server groups, such as those that use autoscaling. A major benefit is that you can manage the servers in the upstream group without modifying the NGINX Plus configuration directly.

Editor – Support for DNSSRVrecords was introduced in NGINX Plus R9. For an overview of all the new features in that release, seeAnnouncing NGINX Plus R9 on our blog.

Also check out our other blogs about service discovery for NGINX Plus:

• Service Discovery for NGINX Plus Using Consul APIs
• Service Discovery for NGINX Plus with etcd
• Service Discovery for NGINX Plus with ZooKeeper

Try out for yourself automated reconfiguration of NGINX Plus upstream groups using Consul DNS for service discovery:

• Download consul-dns-srv-demo from our GitHub repository
• Start a free 30-day trial of NGINX Plus today or contact us for a live demo