How I deleted 10,000 secrets from Kubernetes in a few minutes
Freshservice, the cloud-based ITSM service desk software from the Freshworks suite of products, helps companies manage incidents, assets, and other ITIL features. It is deployed as containers, which are managed by Kubernetes. The secrets management for various microservices in Freshservice was previously done using ejson or rails credentials. For ease of management, we, from the Site Reliability Engineering (SRE) team, decided to start using Kubernetes secrets instead.
Kubernetes has a constraint on naming the secrets. The name of a secret object must be a valid DNS subdomain name. All our secret names contain “_” and this had to be handled. We performed a test migration in our staging environment, replacing “_” with “-” and then came up with the idea of adding an annotation with the original name (with “_”), so that the configuration manager would be able to recognize the secret without any additional transformations. By the time this implementation was ready, we lost sync of the updates to our secrets.
Challenge 1:
We had to re-migrate all the secrets again, which posed a problem because we had to clear all the existing secrets first.
Deleting a secret from a Kubernetes cluster:
kubectl -n <namespace> delete secret <secret_name>
To understand the complexity of the problem, consider this: We have around 100 applications running in our staging environment. Each application has its own set of secrets, with very few common ones since there are no discernable patterns or overlaps. Each application contains around 100-120 secrets. That’s around 10,000 different kubectl commands.
But hey, doesn’t Kubernetes have an option to delete all objects of a particular resource?
Challenge 2:
We already stored some tokens in Kubernetes secrets. Since we couldn’t just delete all secrets, we had to delete only the ones migrated for this specific use case. No other resource could be deleted since the developers, who were actively using the application, could not be disrupted.
One small ray of hope:
All the secrets that I needed to delete had one common word in their name. For example, let’s consider the word “freshservice”. We had a common word but there were no other patterns for recognition.
Does that mean I should have generated around 10,000 unique kubectl delete commands and executed them?
Automation using shell and sublime text shortcuts
I’m a big fan of shell and automation. I have also worked a lot in sublime text and I’m conversant in its few useful shortcuts. I knew I could put all these together and come up with a better solution.
First, I listed the namespaces of my application in a new file in sublime text. You can keep a copy of the following image because it will be used multiple times in the solution.
Press Cmd + A to select all text, and then Cmd + Shift + L to get the cursor on all lines. You can replace Cmd with Ctrl for Windows. It works on sublime text.
Leveraging this workaround, Now, whatever I typed got written on all lines.
As you can see here, navigating each way gets applied to the cursor on all lines. You can select text in parallel with each cursor and copy and paste, and each cursor pastes whatever it originally copied.
So I just had to type once and get the following commands ready:
kubectl -n <namespace> get secrets > <namespace>.txt
Now, I had all the secret names in their respective namespace.txt files. But the files also contained additional content, such as type, date, and age. The files also contained all the secrets of the namespace, from which I still had to extract the ones I needed.
As I already mentioned, all my secrets contained the word “freshservice”. I could have used grep, but that would return the entire line. Instead, I edit the following command a little:
egrep -wo 'freshservice.[A-Za-z0-9-]*' namespace.txt >
namespace_secret_names.txt
- egrep: Grep works with extended regular expression. Include characters, numbers and “-”.
- w: Matches only word(s) instead of the substring.
- o: Displays only matched patterns instead of the whole line.
Using the same way discussed above, you can also generate and run this command for each namespace.
I now got the secret names that I wanted to delete in the files with the name <namespace>_secret_names.txt. Here’s the result of the file from the example above:
Next, I went back to the list of namespaces, selected all lines, put the cursor on all lines, and copied the namespace as well. Then I generated the following command for each namespace. You can paste the namespace copied in place of <namespace>:
awk '{ printf "kubectl -n <namespace> delete secret "; print }'
<namespace>_secret_names.txt > <namespace>_delete_command.sh
- This command prints the secret name next to the content in double-quotes, giving the delete command required. One command for each secret is generated.
- Delete commands specific to the namespace are written as <namespace>_delete_command.sh files.
Following is a list of ‘awk’ commands that were generated:
Here are the resulting delete secret commands in the sh file:
Now that I had generated all the unique and required delete commands in a set of around 100 shell files, one per namespace, I just need to execute the shell files — and my job is done. But I went one more step to make the job easier. So I opened the list of namespaces again and put the cursor on all lines and typed the execute shell file command, and saved it to another master_delete.sh shell file, along with logs to track progress:
From this directory, I ran the following two commands:
chmod +x *.sh
This command is to make all shell files executable.
and
./master_delete.sh
This command is to run the individual files and delete all secrets.
So I got the solution ready to generate around 10,000 different kubectl secret delete commands and run them in a few minutes. I triggered the script, let it run in the background, went for a cup of coffee, and then continued with other work. The entire deletion took about 5 hours to complete.
