descricao: “Learn how to keep MongoDB Atlas change streams alive during primary elections in a Kubernetes StatefulSet deployment.”
# MongoDB Atlas Change Stream Stops After Replica Set Step‑Down in Kubernetes StatefulSet
I ran into a nasty production‑grade bug last quarter: my microservice, running inside a Kubernetes StatefulSet, would lose its MongoDB Atlas change stream every time the replica set elected a new primary. The stream just hung, no events were delivered, and the pod kept retrying forever. In this post I’ll walk through the exact circumstances that caused the failure, why the usual `resumeAfter` logic didn’t save us, and how I rewired the whole connection handling to survive primary elections reliably.
—
## Why does my change stream stop after a primary election?
When MongoDB picks a new primary, the underlying TCP connections are silently closed. The driver detects the closure, attempts to reconnect, and then re‑issues the `aggregate` command that backs the change stream. In theory this should be seamless – the driver automatically resumes from the last known `resumeToken`.
In practice I discovered three hidden gotchas that break this flow in a Kubernetes StatefulSet:
1. **Pod‑level DNS caching** – the StatefulSet DNS name (`my‑mongo-0.my‑mongo‑svc`) resolves to the pod IP of the primary. When that pod steps down, the DNS entry still points to the old IP until the pod’s `Endpoints` object updates, causing the driver to keep dialing a dead endpoint.
2. **Driver version mismatch** – older MongoDB JavaScript/Node drivers (pre‑4.2) don’t correctly handle `resumeAfter` after a `NotPrimaryError` when the server returns a stale `clusterTime`.
3. **Kubernetes graceful shutdown timeout** – the StatefulSet’s `terminationGracePeriodSeconds` was set to 30 s, but the primary election can take 60 s in a three‑node cluster, so the pod was terminated before the driver could finish its reconnection handshake.
These three issues combine to produce the exact symptom I was seeing: the change stream’s `on(‘change’)` handler never fired again, and the driver logged a flurry of `NetworkError: connection closed` messages.
—
## How can I make change streams survive primary elections in a StatefulSet?
Below is the step‑by‑step recipe that got my service back to 100 % uptime. I’ll split it into three logical layers:
* **Network layer fixes** – ensure the driver always resolves to a healthy member.
* **Driver configuration** – use the right resume semantics and timeout values.
* **Kubernetes pod lifecycle** – give the replica set enough time to settle and avoid premature pod termination.
### 1️⃣ How do I avoid DNS‑caching pitfalls for the MongoDB service?
Kubernetes Service DNS resolves to the **ClusterIP**, not to the individual pod IPs. However, my StatefulSet was using a **headless Service** (`clusterIP: None`) so each pod got its own DNS name. The driver was connecting directly to `my-mongo-0.my-mongo-svc` which is fine when that pod is primary, but when it steps down the DNS entry stays stale for the DNS TTL (typically 30 s).
**Fix:** Use a regular (non‑headless) Service with a stable ClusterIP and enable **SRV** records. The driver then performs a DNS SRV lookup on each connection attempt, automatically picking a current primary.
“`yaml
apiVersion: v1
kind: Service
metadata:
name: mongo-svc
spec:
selector:
app: mongo
ports:
– port: 27017
targetPort: 27017
clusterIP: None # ❌ remove this line for a normal Service
# Add an SRV record via a separate Service if you need it
“`
If you must keep the headless Service (e.g., for direct pod‑to‑pod communication), set a low DNS TTL and enable `ndots` in the pod’s `dnsConfig`:
“`yaml
spec:
dnsConfig:
options:
– name: ndots
value: “1”
– name: attempts
value: “5”
– name: timeout
value: “2”
“`
> **Pro Tip:** Adding `options:
– name: rotate
value: “1”` forces the resolver to round‑robin through the A records, reducing the chance of hitting a stale primary.
### 2️⃣ Which driver settings guarantee proper resume after a `NotPrimaryError`?
I was using the Node.js driver version **3.6.4**. The driver’s default `retryWrites` flag is `true`, but it does **not** automatically retry a change stream after a topology change. Upgrading to **4.4+** introduced the `changeStream` retry logic that respects `resumeAfter` across elections.
Here’s a minimal, production‑ready connection snippet:
“`javascript
const { MongoClient } = require(‘mongodb’);
const uri = ‘mongodb://mongo-svc:27017/?replicaSet=rs0&readPreference=primaryPreferred&retryWrites=true’;
const client = new MongoClient(uri, {
// Important: keep the socket alive long enough for elections
socketTimeoutMS: 0,
// Reduce the heartbeat frequency so we notice elections quicker
heartbeatFrequencyMS: 2000,
// Enable resumable change streams (MongoDB 4.2+)
useUnifiedTopology: true,
});
async function watchOrders() {
await client.connect();
const db = client.db(‘orders’);
const coll = db.collection(‘events’);
const pipeline = [{ $match: { operationType: { $in: [‘insert’, ‘update’] } } }];
const changeStream = coll.watch(pipeline, {
fullDocument: ‘updateLookup’,
// Let the driver handle resume automatically
resumeAfter: null,
// If the stream is idle for more than 5 minutes, force a reconnect
maxAwaitTimeMS: 5 * 60 * 1000,
});
changeStream.on(‘change’, (doc) => {
console.log(‘Change received’, doc);
// Process the event …
});
changeStream.on(‘error’, (err) => {
console.error(‘Change stream error’, err);
// The driver will attempt to resume; you can also trigger a manual restart here
});
}
watchOrders().catch(console.error);
“`
Key points:
* `socketTimeoutMS: 0` disables the driver‑side idle timeout, letting the underlying TCP stay open during the election.
* `heartbeatFrequencyMS` shortens the monitoring interval so the driver discovers a new primary within seconds.
* `maxAwaitTimeMS` caps the server‑side `maxTimeMS` for the aggregate, preventing the server from silently closing the cursor after a long idle period.
If you cannot upgrade the driver, you can implement a manual resume loop:
“`javascript
let resumeToken = null;
async function startStream() {
const cs = coll.watch(pipeline, { resumeAfter: resumeToken });
cs.on(‘change’, (doc) => {
resumeToken = doc._id; // store the token for the next iteration
handle(doc);
});
cs.on(‘error’, async (err) => {
if (err.hasErrorLabel(‘ResumableChangeStreamError’)) {
console.warn(‘Resumable error, restarting stream…’);
await cs.close();
await startStream();
} else {
console.error(‘Unrecoverable error’, err);
}
});
}
“`
> **Pro Tip:** Persist the `resumeToken` to an external store (Redis, ConfigMap) if you ever need to restart the pod entirely – the token survives process restarts.
### 3️⃣ How do I tune the StatefulSet lifecycle to give elections time to finish?
Kubernetes kills a pod after `terminationGracePeriodSeconds` expires. My original StatefulSet looked like this:
“`yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: mongo-svc
replicas: 3
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
– name: mongod
image: mongo:5.0
args: [“–replSet”, “rs0”]
ports:
– containerPort: 27017
readinessProbe:
exec:
command: [“mongo”, “–eval”, “db.adminCommand(‘ping’)”]
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
exec:
command: [“mongo”, “–eval”, “db.adminCommand(‘ping’)”]
initialDelaySeconds: 15
periodSeconds: 20
terminationGracePeriodSeconds: 30
“`
During a rolling update the primary would step down, the election could take up to 60 s, and the pod would be SIGKILLed after 30 s, aborting the reconnection logic.
**Solution:** Raise the grace period to at least **120 seconds** and add a `preStop` hook that blocks until the node is no longer primary.
“`yaml
spec:
terminationGracePeriodSeconds: 120
template:
spec:
containers:
– name: mongod
lifecycle:
preStop:
exec:
command: [“/bin/sh”, “-c”, “mongo –eval \”while (rs.status().myState == 1) { print(‘Waiting to step down’); sleep(2); }\” “]
“`
The `preStop` loop checks `myState == 1` (primary) and sleeps until the node steps down, guaranteeing the driver has a chance to reconnect to the new primary before the container is finally terminated.
### 4️⃣ How do I monitor the health of my change stream in production?
Even with all the fixes, a silent failure can still creep in if the resume token becomes invalid (e.g., after a collection rename). I instrumented two metrics with Prometheus:
* `change_stream_active{service=”order-processor”}` – gauge set to `1` when the stream is open.
* `change_stream_errors_total{service=”order-processor”}` – counter incremented on every non‑resumable error.
In the Node.js code I added:
“`javascript
const client = require(‘prom-client’);
const gauge = new client.Gauge({ name: ‘change_stream_active’, help: ‘Is change stream open’ });
const counter = new client.Counter({ name: ‘change_stream_errors_total’, help: ‘Change stream errors’ });
changeStream.on(‘open’, () => gauge.set(1));
changeStream.on(‘close’, () => gauge.set(0));
changeStream.on(‘error’, (err) => {
counter.inc();
console.error(‘Stream error’, err);
});
“`
Alert on `change_stream_active == 0` for more than 30 seconds and you’ll be notified before customers notice missing events.
—
## FAQ
### How do I configure a MongoDB Atlas cluster to use SRV records with Kubernetes?
Create a standard Service (non‑headless) that points to the Atlas connection string’s host list. Use the `mongodb+srv://` URI in your driver; the driver will resolve the SRV records automatically.
### What driver version is required for resumable change streams after a primary step‑down?
MongoDB drivers **4.2+** (Node, Java, Python) implement the `ResumableChangeStreamError` label and automatically resume after `NotPrimaryError`. Older drivers need manual retry logic.
### Can I persist the resume token across pod restarts?
Yes. Store the token in a durable store (Redis, etcd, ConfigMap) and read it on startup. The driver’s `resumeAfter` option accepts any BSON document.
### Why does a headless Service cause DNS staleness during elections?
Headless Services return pod IPs directly; the DNS TTL is controlled by the kube‑dns ConfigMap (default 30 s). When the primary pod changes, the DNS entry for the old primary remains until the TTL expires, leading the driver to connect to a dead node.
### Is there a way to force the driver to re‑lookup the primary on every operation?
Set `directConnection=false` (default for replica sets) and a short `heartbeatFrequencyMS`. The driver will poll the replica set topology and re‑resolve the primary on each heartbeat.
—
## Conclusion
Ensuring that **MongoDB Atlas change streams survive replica set step‑downs in a Kubernetes StatefulSet** boils down to three concrete actions: use a stable Service with SRV lookups, upgrade and tune the driver’s retry/resume settings, and give the pod enough graceful shutdown time to let elections complete. Once these pieces are in place, change streams become as reliable as any other Kafka‑style event source, even in the face of inevitable primary elections.
From a senior‑engineer perspective, the biggest lesson was to treat the database topology as part of the application’s runtime environment—just like network policies or CPU limits. By wiring observability (Prometheus metrics) into the stream and handling the pod lifecycle deliberately, you avoid the “it worked in dev” surprise when a production election occurs.
> **Pro Tip:** When you first roll out the fixes, enable `debug` logging for the MongoDB driver (`MONGODB_DEBUG=1`). The logs will show the exact sequence of `NotPrimaryError → reconnect → resumeAfter` events, confirming that your stream truly survives elections.
That’s it. Follow SpiritCode for more deep‑dive engineering posts.