What a Service is
Most of us have written a Service manifest. It’s a few lines of YAML and one of the first abstractions you reach for in Kubernetes:
apiVersion: v1
kind: Service
metadata:
name: orders
spec:
selector:
app: orders
ports:
- port: 80
targetPort: 5678It’s also the abstraction whose simplicity hides the most. A Service feels like a single networking primitive: a name that maps to a set of pods, an IP that load-balances across them, a port that forwards. None of that is wrong, but there is so much going on underneath the hood.
The Service object on the API server is three pieces of inert data: a ClusterIP (a virtual IP assigned when the Service is created), a label selector (a query that picks the pods behind the Service), and a port mapping (which Service port maps to which container port).
Those fields are the entire spec. Nothing in them resolves the name, picks a pod, or rewrites the packet. For another pod in the cluster to actually reach a Service by name, three artifacts have to be produced elsewhere:
- a DNS answer turning the Service name into a ClusterIP
- a list of which pods currently back the Service
- iptables rules on each node that rewrite the ClusterIP to a pod IP
These artifacts are produced by four components in Kubernetes, working together.
CoreDNS runs as a pod in kube-system and watches the API for Services; when a pod looks up a specific Service, CoreDNS is what answers with the ClusterIP.
kube-proxy runs on every node, watches the Service and EndpointSlice, and writes iptables rules into the local kernel so any packet destined for the ClusterIP gets rewritten to one of the ready pod IPs.
The EndpointSlice controller watches Services and Pods and writes an EndpointSlice object next to each Service: a list of (pod IP, ready) tuples for the pods matching that Service’s selector.
The kubelet on each node runs the readinessProbe declared on each pod and maintains a Ready condition on the pod; the EndpointSlice controller copies that condition into the slice’s per-entry ready flag.
Four components, all reading from the same API, all producing different runtime artifacts: a DNS answer, an EndpointSlice, and the iptables rules that tie them together. When a Service “works”, all four are working. When something looks like a Service problem, one of them is usually misbehaving while the others keep going. Figuring out which of the four is misbehaving is the whole job.
The rest of this article traces how those pieces interact, hopefully in enough detail such that you can understand the diagram below at the end of the article.
The example
The cluster I’ll use is kind, which runs each Kubernetes node as a Docker container on the host. It’s the easiest way to get a real cluster to poke at; the node-as-container detail will matter later when we need to look at iptables rules on a node. One command brings it up:
$ kind create cluster --name test-clusterThe workload is hashicorp/http-echo returning a fixed string, behind a three-replica Deployment and a Service. Apply with kubectl apply -f:
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
spec:
replicas: 3
selector:
matchLabels:
app: orders
template:
metadata:
labels:
app: orders
spec:
containers:
- name: orders
image: hashicorp/http-echo:1.0
args: ["-text=hello from orders"]
ports:
- containerPort: 5678
name: http
readinessProbe:
httpGet:
path: /
port: 5678
---
apiVersion: v1
kind: Service
metadata:
name: orders
spec:
selector:
app: orders
ports:
- port: 80
targetPort: 5678Once the rollout finishes, three pods are running and the Service has a ClusterIP:
$ kubectl get pods -l app=orders -o wide
NAME READY STATUS IP NODE
orders-6d7c4c4954-dmgkq 1/1 Running 10.244.0.15 test-cluster-control-plane
orders-6d7c4c4954-tjsdr 1/1 Running 10.244.0.16 test-cluster-control-plane
orders-6d7c4c4954-dbnvc 1/1 Running 10.244.0.17 test-cluster-control-plane
$ kubectl get svc orders
NAME TYPE CLUSTER-IP PORT(S)
orders ClusterIP 10.96.165.52 80/TCPI also keep a small debug pod around (kubectl run debug --image=busybox:1.36 --command -- sleep infinity).
The chain a packet takes
A request from one pod to another via a Service name passes through these layers. Each layer transforms what’s in the destination field of the packet, and the next layer reads the new value:
source pod
│
│ curl orders service
▼
┌──────────────────────────────┐
│ 1. DNS │
│ name → ClusterIP │
└──────────────────────────────┘
│
│ packet to ClusterIP
▼
┌──────────────────────────────┐
│ 2. iptables on the node │
│ ClusterIP → pod IP │
│ (DNAT) │
└──────────────────────────────┘
│
│ packet to pod IP
▼
destination podLayer 1 - DNS turns the name into the ClusterIP.
Layer 2 - iptables on the node rewrites the ClusterIP to a real pod IP, and the packet is delivered from there.
Layer 2 is the one with the most moving parts: the rules iptables follows are written by kube-proxy from the EndpointSlice, which is in turn written by the EndpointSlice controller from the Service’s selector and the kubelet’s readiness checks.
Walking the chain
Layer 1 - Name to ClusterIP
CoreDNS is a small Go DNS server that the cluster ships by default, running as a normal Deployment in kube-system fronted by a Service called kube-dns.
Whenever a pod is created on a node, the kubelet writes a /etc/resolv.conf file inside that pod which points at that kube-dns ClusterIP. For example:
$ kubectl exec debug -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5So what does this file do? Well, three things on those lines matter.
nameserver 10.96.0.10 is the ClusterIP of the kube-dns Service. (kubectl get svc -n kube-system kube-dns shows it.) Every DNS query the pod makes is sent to one of the running CoreDNS pods.
search default.svc.cluster.local svc.cluster.local cluster.local is the search list. When something inside the pod looks up a name, a query resolver tries each suffix in this list before giving up. So orders becomes orders.default.svc.cluster.local, then orders.svc.cluster.local, then orders.cluster.local, then the bare orders. The first lookup that resolves wins.
options ndots:5 controls whether the search list runs before or after the bare name. A name with at least ndots dots is treated as absolute and looked up as-is first; a name with fewer dots gets the search list applied first. The default of 5 is high. orders.svc.cluster.local has 3 dots, so even that gets the search list tried first.
When the query for orders.default.svc.cluster.local finally reaches CoreDNS, CoreDNS looks the name up in its in-memory view of the API, finds the orders Service, and answers with that Service’s ClusterIP — 10.96.165.52, the same address kubectl get svc orders showed earlier.
Worth pausing on the two distinct ClusterIPs that have come up so far. The kube-dns ClusterIP 10.96.0.10 is the address the resolver dials to ask the DNS question. It’s wired into every pod’s resolv.conf at startup. The orders ClusterIP 10.96.165.52 is the DNS answer that comes back — the address of the Service the pod was actually trying to reach.
Layer 2 - ClusterIP to pod IP
So we are at the point where the pod’s application initiates a TCP connection to 10.96.165.52 (the orders Service ClusterIP). From the application’s point of view it’s connecting to that address, but no handshake has actually completed against it yet — the kernel just has a packet on its hands whose destination field reads 10.96.165.52. Before following the packet, let’s better understand what that address actually is, because it explains why this next layer has to exist at all.
10.96.165.52 doesn’t actually live on any machine. Normally an IP address is bound to a network interface — the kernel’s name for a port a machine can send and receive on. But if you run ip addr on any node in the cluster and grep for 10.96.165.52, you get nothing back: no interface on any node claims it. It’s purely a number the cluster has agreed to mean “the orders Service”, a target for iptables rules to match on, not an address the kernel routes to in a normal way.
So when the packet leaves the pod and hits the node’s network stack, normal routing has nothing to deliver it to. iptables rules are what make the address mean something.
The EndpointSlice controller watches Services and Pods on the API server. When the orders Service appeared, the controller produced an EndpointSlice for it and keeps the slice in sync as pods come and go. The slice for our Service looks like this:
$ kubectl get endpointslices -l kubernetes.io/service-name=orders
NAME ADDRESSTYPE PORTS ENDPOINTS
orders-8rqq2 IPv4 5678 10.244.0.15,10.244.0.16,10.244.0.17One endpoint here means one (pod IP, ready) tuple, not an HTTP route (which is what “endpoint” usually means in application code).
kube-proxy reads that slice and writes iptables rules on the node. Since the node is a Docker container under kind, I docker exec into it to see them (on a real cluster you’d ssh to the node and run the same iptables-save). In the default mode it produces a small chain per Service, looking like this on our cluster (filtered to just the orders rules, with kube-proxy’s -m comment annotations stripped for readability):
$ docker exec test-cluster-control-plane iptables-save -t nat | grep orders
-A KUBE-SERVICES -d 10.96.165.52/32 -p tcp --dport 80 -j KUBE-SVC-HJT7Q47QJWZKYPK5
-A KUBE-SVC-HJT7Q47QJWZKYPK5 -m statistic --mode random --probability 0.3333 -j KUBE-SEP-VPVAH3C6BYA2P2NV
-A KUBE-SVC-HJT7Q47QJWZKYPK5 -m statistic --mode random --probability 0.5000 -j KUBE-SEP-H52RGD7GLHE2WTXK
-A KUBE-SVC-HJT7Q47QJWZKYPK5 -j KUBE-SEP-JDCLW3233VAUQ7TO
-A KUBE-SEP-VPVAH3C6BYA2P2NV -p tcp -j DNAT --to-destination 10.244.0.15:5678
-A KUBE-SEP-H52RGD7GLHE2WTXK -p tcp -j DNAT --to-destination 10.244.0.16:5678
-A KUBE-SEP-JDCLW3233VAUQ7TO -p tcp -j DNAT --to-destination 10.244.0.17:5678Reading top to bottom, the rules trace three movements:
- Match. A packet destined for
10.96.165.52:80matches the first rule and jumps to the per-Service chainKUBE-SVC-HJT7.... - Pick a backend. That chain picks one of three backends by probability: the first rule fires 33% of the time; if it doesn’t, the second tries 50% of the remaining (which is 33% overall); if neither fires, the third catches the rest. The conditional probabilities net out to the round-robin load-balancing across the three pods that we would expect.
- Rewrite. The chosen pick jumps to a per-endpoint chain that DNATs the packet — rewrites the destination address — to one of the actual pod IPs.
By the time the packet leaves the node, the ClusterIP is gone from the destination field and what gets forwarded is a normal packet bound for a pod IP.
Readiness shapes the DNAT pool
We just saw three KUBE-SEP-* chains — one per pod — plus the weighted jumps in the per-Service KUBE-SVC-... chain that pick between them. Call this set the DNAT pool: the pods a request to the ClusterIP can actually be DNATed to. The pool here has three pods, matching the Deployment’s three replicas. But how does kube-proxy decide which pods belong in the pool — and what happens when a pod isn’t healthy?
Two conditions sit on every pod and they are not the same thing. Running means the container process has started. Ready means the pod’s readinessProbe is currently passing. The probe is a check the kubelet runs against the pod on a fixed interval (HTTP GET, TCP connect, or command exec, depending on how it’s declared). The EndpointSlice controller writes every matching pod into the EndpointSlice. Each entry carries a per-endpoint ready flag that mirrors the pod’s Ready condition, and kube-proxy uses that flag when it writes the Layer 2 iptables rules: a pod with ready: true gets its own KUBE-SEP-* chain and a corresponding jump in the per-Service KUBE-SVC-... chain; a pod with ready: false gets neither. So a pod whose probe is failing stays in the EndpointSlice but doesn’t appear in iptables at all, and because of that traffic routes around it.
You can force this ‘not ready’ traffic bypass with a deliberately bad probe port.
$ kubectl patch deployment orders --type=json
-p='[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":9999}]'
$ kubectl get pods -l app=orders
NAME READY STATUS
orders-5d76c45bf8-zvz9p 0/1 Running
orders-6d7c4c4954-dbnvc 1/1 Running
orders-6d7c4c4954-dmgkq 1/1 Running
orders-6d7c4c4954-tjsdr 1/1 Running
$ kubectl get endpointslices -l kubernetes.io/service-name=orders -o yaml |
grep -E 'addresses|ready:|name:' | head -20
- addresses: [10.244.0.15]
ready: true
name: orders-6d7c4c4954-dmgkq
- addresses: [10.244.0.16]
ready: true
name: orders-6d7c4c4954-tjsdr
- addresses: [10.244.0.17]
ready: true
name: orders-6d7c4c4954-dbnvc
- addresses: [10.244.0.19]
ready: false
name: orders-5d76c45bf8-zvz9pThe fourth pod is in the slice but flagged ready: false, so kube-proxy leaves it out of the DNAT pool.
When DNS works but the connection refuses
With this understanding in hand, one of the more common Kubernetes networking failures is worth looking at: a pod resolves a Service name to an IP, but the connection refuses.
The easiest way to reproduce it is to scale the Deployment to zero. The Service object stays, so its ClusterIP stays, so CoreDNS keeps answering the name lookup with that ClusterIP. But the EndpointSlice empties out, and an empty slice means kube-proxy has no KUBE-SEP-* chains to fill the DNAT pool with:
$ kubectl scale deploy orders --replicas=0
$ kubectl get endpointslices -l kubernetes.io/service-name=orders
NAME ADDRESSTYPE PORTS ENDPOINTS
orders-8rqq2 IPv4 <unset> <unset>
$ kubectl exec debug -- dig +short +search orders A
10.96.165.52
$ kubectl exec debug -- curl -sS --max-time 5 http://orders/
curl: (7) Failed to connect to orders port 80: Could not connect to serverDNS returns the ClusterIP because the Service object hasn’t changed. The connection fails because the per-Service iptables chain has no backends to jump to. The same shape shows up in other ways — a selector that doesn’t match any pods (a label typo on the Deployment), every replica failing its readiness probe, a Deployment that hasn’t rolled out yet. They all collapse to the same diagnostic: an empty EndpointSlice.
The command that surfaces this fastest is kubectl get endpointslices -l kubernetes.io/service-name=<name>. An empty slice points you at selector mismatch, scale-to-zero, or all pods failing readiness; from there the fix is on whichever side broke the link between the selector and the running pods.
Why this is worth knowing
The two-line version of all of this: a Service is a query (the selector) and a virtual address (the ClusterIP). The EndpointSlice controller turns the query into a list of pod IPs in an EndpointSlice; kube-proxy turns the list into iptables rules on every node; CoreDNS turns the Service’s name into the ClusterIP. Most Kubernetes networking failures I’ve debugged have been one of those steps misbehaving.
The mental model you need is short. A Service is not a process. The EndpointSlice is the actual binding between Service and pod, and CoreDNS doesn’t know anything about pod readiness. With that, you can read the symptoms and know where to point your next debugging command.
Further reading
- Kubernetes Services concepts and EndpointSlices
- DNS for Services and Pods — the canonical reference for the cluster DNS schema, ndots, and search behavior
- CoreDNS Corefile syntax and the
kubernetesplugin - kube-proxy modes — iptables, IPVS, nftables, and the trade-offs
- kind — what I used to capture every command and output in this piece