Managing multiple k8s config files
So it finally happened! I got myself a blog that’s up and running! And I am inaugurating it with a small blog about how I make my life easier when dealing with multiple Kubernetes clusters and having to connect to them.
What is the problem?
Imagine you are asked to deploy some fancy service in the k8s (Kubernetes) cluster A
and given the kubeconfig for being able to connect to it. Later in the day, you are asked to deploy another fancy service in the k8s cluster B
, for which you are also given its respective kubeconfig. The next day you are now asked to yet again deploy a different fancy service into cluster C
and fix an issue on cluster B
. Ah, from your boss you also heard that cluster A
has been decommissioned so now you need to clean up your configs so the (now useless) information of cluster A
doesn’t stay around unnecessarily.
Hopefully, you see my point now. Over time it becomes tedious to deal with these config files, especially if the way you do it is through a single config file (a.k.a. ~/.kube/config
) as you need some way of adding, removing, and even sharing clusters details.
For the sake of completion, let’s visualize the problem with some dummy kubeconfigs. The file below represents cluster A
, and at this point, we have already saved it to the default location.
1# ~/.kube/config
2apiVersion: v1
3kind: Config
4
5clusters:
6- cluster:
7 server: https://k8s.example.org/k8s/clusters/example-a
8 name: example-a
9
10users:
11- name: example-a
12 user:
13 client-certificate-data: certificate-data-example-a
14 client-key-data: key-data-example-a
15
16contexts:
17- context:
18 name: example-a
The file below represents cluster B
, and it’s saved in $HOME
. Note that in this location kubectl
is not aware of the file.
1# ~/example-b.yaml
2apiVersion: v1
3kind: Config
4
5clusters:
6- cluster:
7 server: https://k8s.example.org/k8s/clusters/example-b
8 name: example-b
9
10users:
11- name: example-b
12 user:
13 client-certificate-data: certificate-data-example-b
14 client-key-data: key-data-example-b
15
16contexts:
17- context:
18 name: example-b
Our options
Now let’s talk about what are our options here. So far, I have seen and used 3 ways of handling this problem, being the last one listed here the most effective for me.
Replace default kubeconfig
The first way I dealt with this was as a caveman. I would move whatever kubeconfig I needed to ~/.kube/config
. So, with our example, I would move ~/.kube/config
into ~/example-a
and then ~/example-b
back into ~/.kube/config
.
As you can imagine, the problem with this approach is that it is way too manual and prone to errors. A wrong move and say goodbye to the one cluster config, or worse, now you have mixed up the clusters configs without being aware of it. It also adds a mental overhead of having to remember where all the files are backed up.
This is a big no.
Merge files
The second option is to merge both files (and any subsequent one) into ~/.kube/config
. That would look like below. This is much nicer than the first option, now we can use something like kubectx for switching clusters with ease. However, I still have two problems with it: 1) having to add new clusters info to the file, and 2) having to remove the info of one cluster when I no longer need it.
For the first problem, I know there’s some bash magic that can merge all new kubeconfig files into one, but I haven’t set that up. And for the second one, I am unaware of how to do the cleanup in “one-click”.
So, while this is a better option, it’s still a no for me.
1# ~/.kube/config
2apiVersion: v1
3kind: Config
4
5clusters:
6- cluster:
7 server: https://k8s.example.org/k8s/clusters/example-a
8 name: example-a
9- cluster:
10 server: https://k8s.example.org/k8s/clusters/example-b
11 name: example-b
12
13users:
14- name: example-a
15 user:
16 client-certificate-data: certificate-data-example-a
17 client-key-data: key-data-example-a
18- name: example-b
19 user:
20 client-certificate-data: certificate-data-example-b
21 client-key-data: key-data-example-b
22
23contexts:
24- context:
25 name: example-a
26- context:
27 name: example-b
Edit the KUBECONFIG env var
This third way so far is what I use up to this day. It consists of modifying the env var KUBECONFIG
and adding all the file paths that we require kubectl
to be aware of. Just this time not as a caveman.
The way how tools like kubectl
/kubectx
/kubens
work is that if no KUBECONFIG
env var is defined they read ~/.kube/config
. If defined, these tools will interpret the string value as a list of paths separated by a colon (:
). What we need now to automate the loading of kubeconfig is making an assumption and a few lines of bash in our ~/.bashrc
/~/.zshrc
.
The assumption we make is that all the config files are going to be saved under the directory ~/.kube/configs/
; all the files in it will be set into KUBECONFIG
. For setting up the variable I have two recipes, one “clear” and another that is a one-liner. Both result in having KUBECONFIG=~/.kube/config:~/.kube/configs/example-a:~/.kube/configs/example-b
, you can choose whatever you prefer.
Bash magic the clear way
1CONFIGS_DIR=$HOME/.kube/configs
2KUBECONFIG="$HOME/.kube/config"
3for entry in "$CONFIGS_DIR"/*
4do
5 if [ -z "$KUBECONFIG" ]
6 then
7 KUBECONFIG="$entry"
8 else
9 KUBECONFIG="$KUBECONFIG:$entry"
10 fi
11done
12export KUBECONFIG
Bash magic the one-liner way
1export KUBECONFIG="$HOME/.kube/config:"$(paste -sd ":" <(find ~/.kube/configs -type f))