xDS, frame
by frame

An Envoy on its own

Every Envoy starts from a bootstrap file. You can list every listener and cluster in it under static_resources, and for one proxy that works fine.

For a fleet it doesn't. Each change means editing files on many machines and restarting or hot-restarting each process.

Enter the management server

xDS splits the system in two. The proxies form the data plane. A separate process, the management server, holds the configuration you want them to run.

Istio's istiod, Envoy Gateway, Gloo, and anything built on go-control-plane or java-control-plane all play this role.

Configuration arrives on a stream

Each Envoy opens a long-lived, bidirectional gRPC stream to the server. Envoy sends requests up; the server sends configuration down, and keeps sending it whenever something changes.

Nothing restarts. Listeners, routes, and clusters are swapped in while traffic flows.

gRPC is the main transport. REST-JSON polling and watching files on disk use the same resource types.

Every request says who is asking

Requests carry a node with an id and a cluster. The server uses them to pick which configuration that proxy receives.

Two proxies on the same server can see completely different worlds. In go-control-plane this is literally a map from node ID to a snapshot.

Where traffic enters

A Listener binds an address and port and holds filter chains. For HTTP, the important filter is the http_connection_manager.

The connection manager can embed its routes, or name a route table to fetch: rds.route_config_name.

Where a request should go

A RouteConfiguration groups virtual hosts, matched by the Host header. Each virtual host has ordered routes: match a path, headers, or method, then send to a cluster.

Routes change far more often than listeners, which is why they live in a separate resource.

A named group of upstream hosts

A Cluster carries everything about talking to a backend: load balancing policy, timeouts, circuit breakers, health checks, outlier detection, TLS.

With type: EDS, it doesn't list its hosts. It asks for them.

The actual IP addresses

A ClusterLoadAssignment lists endpoints grouped by locality, with weights, priorities, and health status.

Endpoints churn constantly as pods come and go. Keeping them separate means a scale-up never touches cluster settings.

Certificates, delivered on their own

TLS contexts can point to a Secret by name instead of embedding keys. Certificates rotate without rewriting listeners or clusters, and private keys stay out of the main config.

Every arrow is a name

Resources point at each other by string. The highlighted names are the glue: a listener names a route table, a route names a cluster, a cluster names its endpoints and secret.

A name can be referenced before its resource exists. Chapter 8 is about what happens then.

Less common types follow the same pattern: SRDS for scoped routes, VHDS for virtual hosts on demand, RTDS for runtime, ECDS for individual filter configs.

Identity first

node.id and node.cluster are copied into the first request on every stream. If the server keys snapshots by node ID and you get this wrong, the proxy connects fine and receives nothing.

Both can also be set with the --service-node and --service-cluster flags.

The chicken-and-egg cluster

Envoy can't fetch clusters until it knows how to reach the server, and reaching the server needs a cluster. So exactly one cluster stays static: xds_cluster.

gRPC runs over HTTP/2, so that cluster must enable http2_protocol_options. Forgetting it is the most common reason a fresh setup never connects.

One stream for everything

ads_config sets up the Aggregated Discovery Service: a single gRPC stream that carries every resource type. Chapter 7 covers why most deployments choose it.

Only two roots

The bootstrap names only listeners and clusters as dynamic. ads: {} means "fetch them over the ADS stream."

Routes, endpoints, and secrets never appear here. Envoy discovers them because listeners and clusters reference them, which is the graph from chapter 2.

Envoy asks with empty hands

The first DiscoveryRequest has an empty version_info and an empty response_nonce: "I have nothing, and I'm not answering anything."

An empty resource_names on a Cluster stream is a wildcard: send every cluster meant for this node.

The server answers with a version and a nonce

The DiscoveryResponse carries the resources, a version_info naming this state, and a nonce naming this particular message.

The version describes the content. The nonce only identifies the message, so a reply can point at it.

Envoy acknowledges

Envoy validates and applies the clusters, then sends a new request with version_info: "v1" and response_nonce: "n1".

Read the pair as a sentence: "I'm answering message n1, and I now run v1." Since the reported version matches the one sent, this is an ACK.

Config changes, server pushes

Someone adds a payments cluster. The server sends v2 with nonce n2. It sends all three clusters, because in state-of-the-world mode each Cluster response is the complete list.

Every ACK is also the next question

Envoy ACKs v2. That request now sits at the server with nothing to answer it. The server holds it until the config changes, then responds.

There is no separate "subscribe" or "poll" call. The request in flight is the subscription.

The server sends something invalid

Envoy is running v2. The server pushes v3, where the payments cluster has connect_timeout: 0s. The proto requires a timeout greater than zero, so validation fails.

Envoy answers the nonce, keeps the old version

The reply carries response_nonce: "n3", because it is answering message n3. But version_info stays "v2", the last version it accepted, and error_detail explains why.

Traffic keeps flowing on the old configuration. The invalid cluster never takes effect.

Don't retry the same mistake

The NACK is also the next pending request. A server should not answer it by resending v3; it would be rejected again in a loop.

Good control planes surface the error_detail to an operator and wait for the configuration to change.

A fix arrives as a new version

The cluster is corrected. The server sends v4 with nonce n4. Envoy applies it and ACKs: version_info: "v4", response_nonce: "n4", no error.

Rule of thumb: version_info equal to the response's version means ACK. A version_info that is older, plus an error_detail, means NACK.

EDS asks for specific names

Listener and Cluster streams are wildcard: Envoy wants all of them. Route and endpoint streams are named: Envoy lists exactly what it needs in resource_names.

Here Envoy knows one EDS cluster, so it asks for ["users"].

The usual exchange

The server sends the users endpoints at v7 with nonce n5. Envoy ACKs and repeats its subscription. In state-of-the-world mode, every request carries the full list of names Envoy wants.

Both sides talk at once

CDS just delivered a new cluster, orders. Envoy sends a request subscribing to ["users", "orders"], still referencing nonce n5.

At the same moment the server pushes a new users endpoint list as v8, nonce n6. The two messages cross on the wire.

The server ignores the old reply

When Envoy's request arrives, the server has already sent n6. A request that answers n5 is stale, and the spec says the server should not respond to it.

Without this rule, the server could act on Envoy's view from before n6 arrived.

The latest request wins

Envoy ACKs n6, and since every request repeats the whole subscription, that ACK still asks for orders. Nothing was lost. The server answers with v9: both clusters' endpoints.

One stream per type

Each resource type gets its own gRPC service and method: StreamListeners, StreamRoutes, StreamClusters, StreamEndpoints. They can even point at different servers.

The cost: nothing orders updates across streams. A route can arrive before the cluster it names.

Everything on one stream

The Aggregated Discovery Service multiplexes all types over StreamAggregatedResources, distinguished by type_url. One server sees every request and chooses the order of every response.

That ordering control is why Istio and most production control planes use ADS.

SotW: send the whole set

In the original variant, a Listener or Cluster response lists every resource. Change one cluster out of 5,000 and the server sends 5,000.

Deletion is implicit. If a cluster is missing from the list, Envoy removes it.

Delta: send only what changed

Delta xDS (DeltaDiscoveryRequest) carries only changed resources, each with its own version, plus an explicit removed_resources list.

Subscriptions change by difference too: resource_names_subscribe and resource_names_unsubscribe. On reconnect, initial_resource_versions tells the server what Envoy already has.

Pick one cell

The two choices combine into four gRPC methods per type. The ACK and NACK rules from chapters 4 and 5 hold in all of them.

Large meshes increasingly run Delta + ADS: one ordered stream and small payloads. VHDS, for virtual hosts loaded on demand, is only available in delta mode.

A route to nowhere

A new route sends /pay to cluster payments, but the CDS update with that cluster hasn't landed yet.

Envoy doesn't reject the route. It installs it, and requests to /pay fail with 503 and response flag NC: cluster not found.

Make before break

The recommended order for additions is CDS → EDS → LDS → RDS. Each step only references things the previous steps created.

With ADS, the server can enforce this order on the single stream. With separate streams it has to be approximated.

Envoy waits for some dependencies

A new EDS cluster is warming until its first ClusterLoadAssignment arrives. A new listener is warming until its route configuration arrives. Warming resources don't take traffic.

Warming ends after initial_fetch_timeout (15s by default) even if nothing arrives, so a missing resource eventually fails visibly.

Routes don't warm against clusters. That is why scene one breaks.

Break in reverse

Removal goes the other way: first stop referencing a thing, then delete it. Update routes to drop /pay, then remove the payments cluster. Its endpoints go with it.

The same rule covers both directions. A reference must never outlive, or arrive before, what it points to.