Rancher: Waiting for plan to be applied and empty connection info

I have recently hit an interesting Rancher provisioning issue after working on downstream RKE2 clusters managed by Rancher and backed by VMware vSphere.

The visible symptom was not very specific. Rancher UI kept showing the cluster in Reconciling state and reported that it was Waiting for plan to be applied. A new worker VM was created in vSphere, but provisioning did not really move forward. At first glance it looked like a new node provisioning problem, but the actual blocker was an older node in the cluster.

Symptoms

The cluster stayed in reconciliation and Rancher was waiting for a machine plan to be applied. On the affected nodes the rancher-system-agent service was failing immediately:

Fatal error running: unable to parse connection info file: EOF

The connection info file existed, but it was empty:

wc -c /var/lib/rancher/agent/rancher2_connection_info.json
0 /var/lib/rancher/agent/rancher2_connection_info.json

This file is required by rancher-system-agent so that it can connect back to Rancher, watch the machine plan Secret and apply the requested plan. If the file is empty, the agent cannot even start properly.

The failure chain was therefore:

empty rancher2_connection_info.json
-> rancher-system-agent fails with EOF
-> agent cannot watch/apply machine-plan
-> Rancher waits for plan to be applied
-> cluster reconciliation is blocked

One important observation was that a broken older control-plane/etcd node can block cluster-level planning. That can make a newly created worker look broken too, even if the new worker is only waiting behind the older affected node.

Diagnostics

On the affected node, start with the agent logs and the connection info file size:

journalctl -u rancher-system-agent -n 50 --no-pager
wc -c /var/lib/rancher/agent/rancher2_connection_info.json
systemctl is-active rancher-system-agent
systemctl is-active rke2-agent

From the Rancher management cluster, check the Machine object:

kubectl -n fleet-default get machine "$MACHINE" \
  -o jsonpath='phase={.status.phase} node={.status.nodeRef.name}{"\n"}'

The conditions are also useful:

kubectl -n fleet-default get machine "$MACHINE" -o json | jq -r '
  .status.conditions[] |
  [.type,.status,(.reason // "-"),(.message // "-"),(.lastTransitionTime // "-")] |
  @tsv'

And the machine plan Secret can show whether Rancher thinks the plan was already applied:

kubectl -n fleet-default get secret "$MACHINE-machine-plan" -o json | jq '{
  planEqualsAppliedPlan: (.data.plan == .data.appliedPlan),
  successCount: (.data["success-count"] | @base64d),
  failureCount: (.data["failure-count"] | @base64d),
  lastApplyTime: (.data["last-apply-time"] | @base64d)
}'

Healthy state after recovery should look like this:

phase=Running
Ready=True
NodeHealthy=True
PlanApplied=True
planEqualsAppliedPlan=true
failureCount=0

Recovery

There are existing upstream discussions about broken rancher2_connection_info.json files. Some recovery procedures manually reconstruct the JSON file from Rancher Secrets. That works, but I prefer a safer method when the original Machine bootstrap Secret is still available.

The Machine object references its bootstrap Secret in .spec.bootstrap.dataSecretName. That Secret contains the original machine-specific bootstrap script. Running that script again on the affected node can regenerate the correct agent connection info and let rancher-system-agent reconnect.

First find the bootstrap Secret:

kubectl -n fleet-default get machine "$MACHINE" \
  -o jsonpath='{.spec.bootstrap.dataSecretName}{"\n"}'

Extract the bootstrap script:

BOOTSTRAP=$(kubectl -n fleet-default get machine "$MACHINE" \
  -o jsonpath='{.spec.bootstrap.dataSecretName}')

kubectl -n fleet-default get secret "$BOOTSTRAP" \
  -o jsonpath='{.data.value}' | base64 -d > /tmp/${MACHINE}-bootstrap.sh

Check the script before copying it anywhere:

sh -n /tmp/${MACHINE}-bootstrap.sh

Copy it to the affected node. Before running it, back up the current agent data:

cp -a /var/lib/rancher/agent /var/lib/rancher/agent.backup.$(date +%Y%m%d%H%M%S)
sh /tmp/${MACHINE}-bootstrap.sh

Then verify the services and logs:

systemctl is-active rancher-system-agent
systemctl is-active rke2-agent
journalctl -u rancher-system-agent -n 50 --no-pager

The useful log line after a successful recovery is something like:

Command sh [-c run.sh] finished with err: <nil> and exit code: 0
updated plan secret ... with feedback

In our case this recovered multiple affected nodes. Once the blocking control-plane/etcd node was fixed, Rancher planning progressed to the worker nodes.

Notes

The affected Rancher version was v2.13.8. The issue looks related to previously reported problems around broken or invalid rancher2_connection_info.json, but the important point here is that the runtime failure was still reproducible on v2.13.8.

I reported this upstream as:

https://github.com/rancher/rancher/issues/56895

The robust fix should be on the Rancher/system-agent side: do not leave an empty or truncated rancher2_connection_info.json in place. The safer pattern is to write to a temporary file, validate the content, and only then atomically replace the active connection info file.

Until then, restoring the machine-specific bootstrap flow from .spec.bootstrap.dataSecretName is a useful operational recovery method.