K8s on Oracle Cloud [Part 5]: Setting up Sealed Secrets
First service to be deployed on a k8s cluster is Sealed Secrets. This is because Sealed Secrets will be used to protect confidential information for other services installed later on.
To speed the process up, add automation and to make sure the entire installation can be easily replayed we use a set of scripts available on gihub repository: k8s-scripts. While there is some documentation for the scripts and you can look in scripts source code to get more details, this guide expands on the details explaining various options and suggesting optimal settings.
Personal notes: My personal notes on how to setup stuff to make it easier to repeat next time.
Step 1: Prerequisites
- K8s on Oracle Cloud “Part 4” is completed.
Step 2: Configuration
There is not much to configure for Sealed Secrets and you can skip to deployment right away to have working installation. However, there are possible customizations which are optional.
Version adjustment
If you want to use a different version of Sealed Secrets from the default version used by k8s-scripts you can open ~/.tigase-flux/envs/versions.env file and change value of the SS_VER property:
# Sealed secrets
SS_VER="2.1.8"
The simplest way to get the most recent version of the chart is to check the version using helm command:
~/temp/k8s-scripts$ helm search hub sealed-secrets
URL CHART VERSION APP VERSION DESCRIPTION
https://artifacthub.io/packages/helm/bitnami-la... 2.1.8 v0.17.5 Helm chart for the sealed-secrets controller.
https://artifacthub.io/packages/helm/wener/seal... 2.1.8 v0.17.5 Helm chart for the sealed-secrets controller.
https://artifacthub.io/packages/helm/wenerme/se... 2.1.8 v0.17.5 Helm chart for the sealed-secrets controller.
https://artifacthub.io/packages/helm/openinfrad... 1.16.1 v0.16.0 Helm chart for the sealed-secrets controller.
https://artifacthub.io/packages/helm/cloudnativ... 1.0.2 0.7.0 A Helm chart for Sealed Secrets
https://artifacthub.io/packages/helm/redhat-cop... 1.10.2 0.12.1 A Helm chart for Sealed Secrets
https://artifacthub.io/packages/helm/taskmedia/... 1.1.4 latest Deploy IPsec VPN server inside K8s with optiona...
If you look at the ~/.tigase-flux/envs/versions.env file again, you can see that we use helm chart from “bitnami-labs”, and you have to use chart version for this package.
Custom values
The ~/.tigase-flux/envs/ folder contains custom values for any package being installed using k8s-scripts. This includes Sealed Secrets. If you look at the content of the ~/.tigase-flux/envs/ss_values.yaml file, it looks really simple with not much options to change:
ingress:
enabled: false
You can adjust values as needed or add more custom configuration if necessary.
Step 3: Installation
If you look inside the Sealed Secrets installation script, you could find it does 2 things:
- Deploys Sealed Secrets using fluxcd to the cluster.
- Retrieves public key from sealed secrets on the cluster, and adds the key to the git repository.
After each step the script forces fluxcd reconcilation with the changes in git repo and waits for the process to complete. Sometimes it may take a while.
And that’s all.
The public key is then used by subsequent scripts to enrypt sensitive data for other services.
Let’s deploy!
Before running the script check the cluser state to see what changed after deployment:
~/temp/k8s-scripts$ flux get all -A
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system gitrepository/flux-system True Fetched revision: master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system helmrepository/chartmuseum True Fetched revision: d5596b841d07f4ca4ec3e7098ad5d5f5bae528c556cf28aa3584ecf0a5a8d43a d5596b841d07f4ca4ec3e7098ad5d5f5bae528c556cf28aa3584ecf0a5a8d43a False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system kustomization/apps True Applied revision: master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 False
flux-system kustomization/common True Applied revision: master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 False
flux-system kustomization/flux-system True Applied revision: master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 master/ecd4e63d6493f9356d05afe548c1cd7b3541b487 False
So we have a basic cluster with fluxcd controlling the state. We can also “watch” the cluster in real-time to see any changes as they happen:
~/.tigase-flux$ flux get hr -A -w
where hr stands for helmreleases. So far nothing on the list but let’s see what happens agter running the script which deploys Sealed Secrets:
~/temp/k8s-scripts$ ./scripts/cluster-sealed-secrets.sh
Adding sealed-secrets source at https://bitnami-labs.github.io/sealed-secrets
/home/t/.tigase-flux/projects/cluster-name
[master 0d0d266] sealed-secrets deployment
2 files changed, 13 insertions(+), 2 deletions(-)
create mode 100644 infra/common/sources/sealed-secrets.yaml
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 16 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 860 bytes | 860.00 KiB/s, done.
Total 7 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/a/cluster-name
811e054..0d0d266 master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/0d0d266919fdb42dc29763faf8ef714595b8868c
Waiting for the system to be ready
Deploying sealed secrets
+ flux create helmrelease sealed-secrets \
--interval=3h \
--release-name=sealed-secrets-controller \
--source=HelmRepository/sealed-secrets \
--chart-version=2.1.8 \
--chart=sealed-secrets \
--namespace=flux-system \
--target-namespace=flux-system \
--values=/home/t/.tigase-flux/envs/ss_values.yaml \
--crds=CreateReplace \
--export
+ set +x
Update service kustomization
/home/t/.tigase-flux/projects/cluster-name
Update common kustomization
/home/t/.tigase-flux/projects/cluster-name
[master 05590f7] sealed-secrets deployment
3 files changed, 31 insertions(+), 1 deletion(-)
create mode 100644 infra/common/sealed-secrets/kustomization.yaml
create mode 100644 infra/common/sealed-secrets/sealed-secrets.yaml
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 16 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 951 bytes | 951.00 KiB/s, done.
Total 8 (delta 1), reused 3 (delta 1), pack-reused 0
remote: Resolving deltas: 100% (1/1), done.
To https://github.com/a/cluster-name
0d0d266..05590f7 master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/05590f7d2e1d51c07dc1b54f8805531157b38dff
Waiting for the system to be ready
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Handling connection for 8080
100 1724 100 1724 0 0 10579 0 --:--:-- --:--:-- --:--:-- 10641
[master 2dae059] public-key deployment
1 file changed, 28 insertions(+)
create mode 100644 pub-sealed-secrets-cluster-name.pem
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.62 KiB | 1.62 MiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/a/cluster-name
05590f7..2dae059 master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/2dae05931789b84ad20e45fd75b7909b5ccd54ba
At the same time our watch command shows some changes and progress:
~/.tigase-flux$ flux get hr -A -w
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system sealed-secrets False waiting to be reconciled False
flux-system sealed-secrets False waiting to be reconciled False
flux-system sealed-secrets Unknown Reconciliation in progress False
flux-system sealed-secrets False HelmChart 'flux-system/flux-system-sealed-secrets' is not ready False
flux-system sealed-secrets Unknown Reconciliation in progress False
flux-system sealed-secrets True Release reconciliation succeeded 2.1.8 False
And now, if we check the cluster status we should see new service deployed:
~/temp/k8s-scripts$ flux get all -A
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system gitrepository/flux-system True Fetched revision: master/2dae05931789b84ad20e45fd75b7909b5ccd54ba master/2dae05931789b84ad20e45fd75b7909b5ccd54ba False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system helmrepository/chartmuseum True Fetched revision: d5596b841d07f4ca4ec3e7098ad5d5f5bae528c556cf28aa3584ecf0a5a8d43a d5596b841d07f4ca4ec3e7098ad5d5f5bae528c556cf28aa3584ecf0a5a8d43a False
flux-system helmrepository/sealed-secrets True Fetched revision: 0d6c10577d8902f34be668dd1f02d462cc3b3945463625dd2b33c798a6a81fbb 0d6c10577d8902f34be668dd1f02d462cc3b3945463625dd2b33c798a6a81fbb False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system helmchart/flux-system-sealed-secrets True Pulled 'sealed-secrets' chart with version '2.1.8'. 2.1.8 False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system helmrelease/sealed-secrets True Release reconciliation succeeded 2.1.8 False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system kustomization/apps True Applied revision: master/2dae05931789b84ad20e45fd75b7909b5ccd54ba master/2dae05931789b84ad20e45fd75b7909b5ccd54ba False
flux-system kustomization/common True Applied revision: master/2dae05931789b84ad20e45fd75b7909b5ccd54ba master/2dae05931789b84ad20e45fd75b7909b5ccd54ba False
flux-system kustomization/flux-system True Applied revision: master/2dae05931789b84ad20e45fd75b7909b5ccd54ba master/2dae05931789b84ad20e45fd75b7909b5ccd54ba False
Also, kubectl shows new resources deployed to the cluster:
~/temp/k8s-scripts$ kubectl get all -A | grep sealed
flux-system pod/sealed-secrets-controller-6dd8d66b9b-22ndf 1/1 Running 0 7m25s
flux-system service/sealed-secrets-controller ClusterIP 10.96.13.67 <none> 8080/TCP 7m25s
flux-system deployment.apps/sealed-secrets-controller 1/1 1 1 7m25s
flux-system replicaset.apps/sealed-secrets-controller-6dd8d66b9b 1 1 1 7m26s
And finally you can go to your git repository on GitHub and see new content added to the repository. On your local disk you can see the content by searching for files with “sealed” string in their name:
t:~/.tigase-flux/projects/cluster-name$ find . -name "*sealed*"
./pub-sealed-secrets-cluster-name.pem
./infra/common/sources/sealed-secrets.yaml
./infra/common/sealed-secrets
./infra/common/sealed-secrets/sealed-secrets.yaml
Open the files to see what is inside.
Uninstallation
OK, now how to remove Sealed Secrets from our cluster without reinitializing entire cluster?
As explained before, fluxcd controls the cluster and it keeps the cluster state in git repository. Adding/removing/updating git repository is then picked up by fluxcd and synchronized back to the cluster. And this is what all the k8s-scripts do. They change make changes to the git repository and force fluxcd to reconcile these changes.
Therefore, if we want to remove Sealed Secrets we just installed, all we have to do is to remove the service config and manifest files. Go to your cluster git repository and run following commands:
t:~/.tigase-flux/projects/cluster-name$ rm -rf infra/common/sealed-secrets/
Edit file: infra/common/kustomization.yaml and remove line with sealed-secrets and run following commands.
t:~/.tigase-flux/projects/cluster-name$ git commit -am "Removing sealed-secrets"
[master cb097b4] Removing sealed-secrets
2 files changed, 29 deletions(-)
delete mode 100644 infra/common/sealed-secrets/kustomization.yaml
delete mode 100644 infra/common/sealed-secrets/sealed-secrets.yaml
t:~/.tigase-flux/projects/cluster-name$ git push
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 16 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 508 bytes | 508.00 KiB/s, done.
Total 4 (delta 0), reused 2 (delta 0), pack-reused 0
To https://github.com/a/cluster-name
2dae059..cb097b4 master -> master
t:~/.tigase-flux/projects/cluster-name$ flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/cb097b414666f8b9cefbe2f4c3ada19b8438189d
t:~/.tigase-flux/projects/cluster-name$
Now, let’s check the cluster status:
t:~/temp/k8s-scripts$ flux get all -A
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system gitrepository/flux-system True Fetched revision: master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system helmrepository/chartmuseum True Fetched revision: d5596b841d07f4ca4ec3e7098ad5d5f5bae528c556cf28aa3584ecf0a5a8d43a d5596b841d07f4ca4ec3e7098ad5d5f5bae528c556cf28aa3584ecf0a5a8d43a False
flux-system helmrepository/sealed-secrets True Fetched revision: 0d6c10577d8902f34be668dd1f02d462cc3b3945463625dd2b33c798a6a81fbb 0d6c10577d8902f34be668dd1f02d462cc3b3945463625dd2b33c798a6a81fbb False
NAMESPACE NAME READY MESSAGE REVISION SUSPENDED
flux-system kustomization/apps True Applied revision: master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 False
flux-system kustomization/common True Applied revision: master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 False
flux-system kustomization/flux-system True Applied revision: master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 master/d3d7d91a0c062ea1e40661e9ba6ec82dcca61827 False
t:~/temp/k8s-scripts$ kubectl get all -A | grep sealed
t:~/temp/k8s-scripts$
As we can see there is no more Selaed Secrets resources on the cluster.
Please note, if you have any other services deployed which depend on Sealed Secrets, removal of the service may be unsuccessfull as FluxCD may refuse to remove a service which is required by other services. In some case, however, where service using Sealed Secrets but does not signal dependency, FluCD may remove Sealed Secrets and this may lead to a broken installation.