kobit

kobit

Passionate software developer for work and for fun.

K8s on Oracle Cloud [Part 6]: Setting up Ingress Nginx

6 minute read

Ingress Nginx Controller should be installed as one of the first services. It does not really matter if it is the first or second one. This is because it is used by many other services being installed later on.

To speed the process up, add automation and to make sure the entire installation can be easily replayed we use a set of scripts available on gihub repository: k8s-scripts. While there is some documentation for the scripts and you can look in scripts source code to get more details, this guide expands on the details explaining various options and suggesting optimal settings.

Personal notes: My personal notes on how to setup stuff to make it easier to repeat next time.

Step 1: Prerequisites

Step 2: Configuration

Ingress Nginx controller has much more configuration options available. k8s-scripts come with some reasonable defaults. In this guide we will look at some options which are most common to be adjusted. For more details, please refer to the Ingress Nginx project on GitHub.

Version adjustment

k8s-scripts define versions for services which are up to date and tested at the when the project was last updated by it’s developers. These versions may become outdated over time, or perhaps you need/want to use a very specific version of the package.

To adjust the ingress version look at the ~/.tigase-flux/envs/versions.env file and change value of the IN_VER property:

# Ingress Nginx
IN_VER="4.1.0"

To check what is the latest available version for the package run command:

$ helm search hub --max-col-width 70 ingress-nginx
URL                                                                   	CHART VERSION	APP VERSION	DESCRIPTION
https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx      	4.1.1        	1.2.0      	Ingress controller...
https://artifacthub.io/packages/helm/nginx-ingress-controller/ingre...	4.0.13       	1.1.0      	Ingress controller...
https://artifacthub.io/packages/helm/gilangvperdana/ingress-nginx     	4.0.18       	1.1.2      	Ingress controller...
https://artifacthub.io/packages/helm/api/ingress-nginx                	3.29.1       	0.45.0     	Ingress controller...
https://artifacthub.io/packages/helm/drycc/ingress-nginx              	4.0.6        	1.0.4      	Ingress controller...
https://artifacthub.io/packages/helm/mxytest/ingress-nginx            	4.1.1        	1.2.0      	Ingress controller...
https://artifacthub.io/packages/helm/wenerme/ingress-nginx            	4.1.1        	1.2.0      	Ingress controller...
https://artifacthub.io/packages/helm/wener/ingress-nginx              	4.1.1        	1.2.0      	Ingress controller...

k8s-scripts use “ingress-nginx/ingress-nginx” package, so you need to use version for the 1st position on this list.

Custom values

Custom values for the “ingress-nginx” service can be found in ~/.tigase-flux/envs/nginx_values.yaml file. There are a few settings worth checking out, however, let’s focus first on a setting which must be adjusted in order to have a successful ingress service deployment.

Disable metrics

Apparently, the k8s-scripts developers deploy ingress-nginx after monitoring services (prometheus) is installed. Therefore in the configuration file metrics are already enabled. However, on our cluster we do not have any monitoring service yet and with metrics enabled, deployment fails. Therefore, we have to disable metrics before we run installation script.

Following section in the ~/.tigase-flux/envs/nginx_values.yaml must be changed:

  metrics:
    enabled: true

We need to change metrics.enabled to “false”, like this:

  metrics:
    enabled: false

Load Balancer

  service:
    annotations:
      service.beta.kubernetes.io/oci-load-balancer-shape: flexible
      service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: 10
      service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: 10

Above are Oracle Cloud specific settings for load balancer. If the service is being setup on a different provider, these shuold be adjusted accordingly. For Oracle Cloud we use “Flexible Load Balancer” with traffic size 10MB. The size can be adjusted as needed but this may affect installation costs.

Note. After completing ingress-nginx installation, a Flexible Load Balancer will be automatically started for you, which can add to the costs on your Oracle Cloud account.

Number of replicas

  replicaCount: 2

With cluster of 3 nodes, possible other values are 3 - with ingress running on each node for greater redundancy at expense of more resources usage or 1 which further reduces resources consumption but also reduces reliability in case this particular node with ingress service breaking up. K8s would automatically respawn ingress on a different cluster node but there would be possible service interruption.

Resources

  resources:
    limits:
      cpu: 1
      memory: 1024Mi
    requests:
      cpu: 100m
      memory: 128Mi

Default resources comming with k8s-scripts are adjusted for relatively low traffic system and work well in such case. If there is a demand for a higher traffic, I suggest to play with different settings until acceptable performance is reached.

Step 3: Installation

At this point the installtion process is very simple. Actually, for this service, there is not much tweaking in terms of configuration and we could go right to the script from the beginning. The installation script cluster-ingress-nginx.sh is actually very simple. It just adds ingress-nginx manifests and configuration to the repository and forces sync of the fluxcd with repository.

Before running the script I suggest to execute flux get all -A and/or flux get hr -A before and after to see the difference.

To install the ingress-nginx service run the following:

~/temp/k8s-scripts$ ./scripts/cluster-ingress-nginx.sh 
      Adding ingress-nginx source at https://kubernetes.github.io/ingress-nginx
/home/t/.tigase-flux/projects/cluster-name
[master 1665cbf] ingress-nginx deployment
 4 files changed, 91 deletions(-)
 delete mode 100644 infra/common/ingress-nginx/ingress-nginx/ingress-nginx.yaml
 delete mode 100644 infra/common/ingress-nginx/ingress-nginx/kustomization.yaml
 delete mode 100644 infra/common/ingress-nginx/kustomization.yaml
 delete mode 100644 infra/common/ingress-nginx/namespace.yaml
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 16 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 452 bytes | 452.00 KiB/s, done.
Total 4 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/a/cluster-name
   94630d0..1665cbf  master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/1665cbf61f0f0fad93e8efbde61615a24f2886e3
Waiting for the system to be ready
   Deploying ingress-nginx
Creating folder for ingress-nginx namespace...
+ flux create helmrelease ingress-nginx \
    --interval=3h \
    --release-name=ingress-nginx \
    --source=HelmRepository/ingress-nginx \
    --chart-version=4.1.1 \
    --chart=ingress-nginx \
    --namespace=flux-system \
    --target-namespace=ingress-nginx \
    --values=/home/t/.tigase-flux/envs/nginx_values.yaml \
    --create-target-namespace \
    --export
+ set +x
Update service kustomization
/home/t/.tigase-flux/projects/cluster-name
Update namespace kustomization
/home/t/.tigase-flux/projects/cluster-name
Update common kustomization
/home/t/.tigase-flux/projects/cluster-name
[master 491aa23] ingress-nginx deployment
 4 files changed, 91 insertions(+)
 create mode 100644 infra/common/ingress-nginx/ingress-nginx/ingress-nginx.yaml
 create mode 100644 infra/common/ingress-nginx/ingress-nginx/kustomization.yaml
 create mode 100644 infra/common/ingress-nginx/kustomization.yaml
 create mode 100644 infra/common/ingress-nginx/namespace.yaml
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 16 threads
Compressing objects: 100% (9/9), done.
Writing objects: 100% (10/10), 1.68 KiB | 1.68 MiB/s, done.
Total 10 (delta 2), reused 2 (delta 1), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 1 local object.
To https://github.com/a/cluster-name
   1665cbf..491aa23  master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/491aa23bad8a8de334954ef8793710816cd20cf3
Waiting for the system to be ready
2022-05-10 15:56:09 System not ready yet, waiting 20

From the output, looks like deployment was successful. To confirm let’s run flux get hr -A:

~/.tigase-flux/projects/cluster-name$ flux get hr -A
NAMESPACE    	NAME          	READY	MESSAGE                         	REVISION	SUSPENDED 
flux-system  	sealed-secrets	True 	Release reconciliation succeeded	2.1.8   	False    	
ingress-nginx	ingress-nginx 	True 	Release reconciliation succeeded	4.1.1   	False

Running flux get all -A shows more details about cluster state.

To get all the details about ingress service, which is useful in case of deployment problems or errors run the following command:

~/temp/k8s-scripts$ kubectl describe deployment ingress-nginx-controller -n ingress-nginx
Name:                   ingress-nginx-controller
Namespace:              ingress-nginx
CreationTimestamp:      Tue, 10 May 2022 15:55:49 -0700
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=ingress-nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/part-of=ingress-nginx
                        app.kubernetes.io/version=1.2.0
                        helm.sh/chart=ingress-nginx-4.1.1
                        helm.toolkit.fluxcd.io/name=ingress-nginx
                        helm.toolkit.fluxcd.io/namespace=ingress-nginx
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: ingress-nginx
                        meta.helm.sh/release-namespace: ingress-nginx
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/name=ingress-nginx
  Service Account:  ingress-nginx
  Containers:
   controller:
    Image:       k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185
    Ports:       80/TCP, 443/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --update-status-on-shutdown=false
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-nginx-controller-68b6767f7 (2/2 replicas created)
Events:          <none>

Uninstallation

To uninstall ingress-nginx, please follow instructions about Sealed Secrets uninstallation described in K8s on Oracle Cloud “Part 5”.

You may also enjoy

Structure needs cleaning error fix

1 minute read

I have recently experienced an error in my k8s setup “Relabel failed… structure needs cleaning”. As a result one of the k8s container/pod failed to start. ...

VIM tab settings for yaml files

less than 1 minute read

Simple vim configuration adjustments to make it work better with yaml files. Personal notes: My personal notes on how to setup stuff to make it easier to re...

IMAP Accounts Migration

4 minute read

When you move your email account around, either between different hosting providers or to/from self-hosted system, you usually want to take your mailbox with...

Let’s Encrypt and back to RSA keys

2 minute read

A while ago Let’s Encrypt switched over to use more modern, safer and efficient ECDSA private keys algorithm instead of RSA. And made the ECDSA algorithm def...