Rancher: Migrating server-url to a new domain with split-horizon DNS

I recently migrated a Rancher installation from an old domain name to a new one. The setup was slightly more interesting than a simple DNS rename because Rancher was used from different network locations and the environment used split-horizon DNS.

The simplified topology looked like this:

local downstream clusters  -> internal RR DNS -> Rancher ingress -> internal/self-signed CA
remote downstream clusters -> public DNS      -> F5 VIP          -> public CA certificate
users                      -> public DNS      -> F5 VIP          -> public CA certificate

As an ASCII diagram:

                         +---------------------------+
                         |        Rancher URL         |
                         | rancher.new.example.com    |
                         +-------------+-------------+
                                       |
                  +--------------------+--------------------+
                  |                                         |
                  v                                         v
        +-------------------+                     +-------------------+
        | Internal RR DNS   |                     | Public DNS        |
        | local resolution  |                     | external clients  |
        +---------+---------+                     +---------+---------+
                  |                                         |
                  v                                         v
        +-------------------+                     +-------------------+
        | Rancher Ingress   |                     | F5 VIP            |
        | internal TLS path |                     | public TLS path   |
        +---------+---------+                     +---------+---------+
                  |                                         |
                  +--------------------+--------------------+
                                       |
                                       v
                         +---------------------------+
                         |      Rancher service      |
                         |      cattle-system        |
                         +---------------------------+

Clients using the same hostname may therefore see a different TLS endpoint
depending on where they resolve the name from.

The important part is that the same Rancher service can be reached through different paths depending on where the client is located. That makes the order of changes important. If server-url is changed before DNS, TLS and Ingress are ready, downstream agents can lose connectivity.

Goal

The goal was to move Rancher from the old hostname to a new hostname, for example:

https://rancher.old.example.com
->
https://rancher.new.example.com

The old hostname had to stay working during the migration. This is important because existing agents may still use the old CATTLE_SERVER value until Rancher redeploys the agent manifests and the downstream clusters pick up the new URL.

Preparation

Before touching Rancher itself, I prepared both DNS paths.

For external clients, the new hostname had to resolve to the existing F5 VIP. The VIP also had to present a valid certificate for the new hostname. In our case this meant adding the second certificate/SNI entry on the F5 VIP while keeping the old certificate in place.

For internal clients, I also added a secondary local round-robin DNS record for the new hostname. This made sure local downstream clusters could resolve the new Rancher name internally and continue using the local path to the Rancher ingress.

At this point both names existed:

rancher.old.example.com -> still working
rancher.new.example.com -> resolves internally and externally

Add the new host to Rancher Ingress

The next step was to make the Rancher Ingress accept both hostnames. The old host was kept, and the new host was added as an additional rule and TLS host.

For a standard Rancher installation this is usually the rancher Ingress in the cattle-system namespace:

kubectl -n cattle-system get ingress rancher -o yaml

The target state is a dual-host Ingress similar to this:

spec:
  rules:
  - host: rancher.old.example.com
    http:
      paths:
      - backend:
          service:
            name: rancher
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  - host: rancher.new.example.com
    http:
      paths:
      - backend:
          service:
            name: rancher
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - rancher.old.example.com
    - rancher.new.example.com
    secretName: tls-rancher-ingress

The exact TLS secret name depends on how Rancher was installed. The main point is that the Ingress must be ready for the new hostname before Rancher starts advertising the new server-url to agents.

After the Ingress change, I verified that the Rancher UI loaded through both names.

Change server-url

Only after DNS, F5 TLS and Rancher Ingress were ready did I change the Rancher server-url setting.

The setting can be checked with:

kubectl get setting.management.cattle.io server-url \
  -o jsonpath='{.value}{"\n"}'

And changed with:

kubectl patch setting.management.cattle.io server-url \
  --type=merge \
  -p '{"value":"https://rancher.new.example.com"}'

The same change can also be done in the Rancher UI under global settings.

This setting is more important than the Helm chart hostname value for already running Rancher agents. The Helm hostname primarily controls the generated Ingress. The runtime server-url is what Rancher advertises to agents.

Agent migration

After changing server-url, Rancher automatically redeployed the downstream cluster agent manifests. The cattle-cluster-agent deployments in downstream clusters were updated with the new CATTLE_SERVER value.

I checked the value on downstream clusters with:

kubectl -n cattle-system get deploy cattle-cluster-agent \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="CATTLE_SERVER")].value}{"\n"}'

The expected value was:

https://rancher.new.example.com

In this case the downstream clusters re-registered cleanly against the new Rancher domain. I did not have to manually recreate the agents.

If an agent does not get redeployed, Rancher can be nudged by annotating the management cluster object:

kubectl annotate clusters.management.cattle.io <CLUSTER_ID> \
  io.cattle.agent.force.deploy=true --overwrite

TLS mode note

In split-horizon environments it is worth checking Rancher’s agent TLS mode. If different paths terminate TLS with different trust chains, system-store is often the practical choice:

kubectl get setting.management.cattle.io agent-tls-mode \
  -o jsonpath='{.value}{"\n"}'

To set it:

kubectl patch setting.management.cattle.io agent-tls-mode \
  --type=merge \
  -p '{"value":"system-store"}'

In this mode agents can use the operating system trust store for publicly trusted certificates, while Rancher-managed CA handling can still cover internal/self-signed paths depending on the Rancher version and agent type.

Final cleanup

Once all downstream agents were using the new CATTLE_SERVER, the Helm/Terraform value for the Rancher hostname could be updated to the new hostname as well.

For Terraform-managed Helm releases that usually means updating something like:

set {
  name  = "hostname"
  value = "rancher.new.example.com"
}

I intentionally did this after the runtime migration was already proven. A Terraform plan may not show manual drift in every Kubernetes object created by a Helm release, so I did not use Terraform as the first validation step for the live agent migration.

Summary

The safe order was:

  1. Add external DNS for the new hostname.
  2. Add the new certificate/SNI entry on the F5 VIP.
  3. Add internal RR DNS for the new hostname.
  4. Add the new host and TLS host to the Rancher Ingress while keeping the old host.
  5. Verify Rancher UI works through both hostnames.
  6. Change Rancher server-url to the new URL.
  7. Watch downstream clusters and verify cattle-cluster-agent uses the new CATTLE_SERVER.
  8. Update Helm/Terraform hostname after the migration is confirmed.
  9. Remove the old hostname only after all agents and users have moved away from it.

The key lesson is simple: make the new route valid before Rancher starts advertising it. Once DNS, certificates and Ingress were ready, Rancher handled the downstream agent migration automatically.