K8s on Oracle Cloud [Part 9]: Setting up Mailu as email server

25 minute read

Mailu is an email server package with SMTP, IMAP, POP, admin WEB Console, Webmail, virus scanner and more. For more details about Mailu and features set, refer to the Mailu website. This setup aim to leverage all Mailu capabilities.

The k8s installation provides following features set:

  1. Setup on managed k8s cluster
  2. Setup on multiple nodes
  3. Mailu volumes with replication
  4. Automated backups to S3 storage
  5. Automated SSL certificates renewal
  6. K8s cluster managed by FluxCD
  7. LB configuration for Web UI access as well as IMAP/POP/SMTP
  8. External relay service for email delivery
  9. Email service is not an open relay

FluxCD is used to manage and maintain the k8s cluster state. However, while the FluxCD makes things easier and takes care of some cluster management/maintenance hussle, it still leaves many details for manual work. It includes generating passwords, k8s secrets, preparing git repository structure, placing files in a correct location and so on. With manual work, comes a greater chance to make mistake, it slow the whole process down and also makes it difficult to repeat the whole process on a different cluster or by someone else.

To completely remove the manual element and fully automate the process a set of scripts is used. These are really simple bash scripts which create correct folder structure for the git repository, take care of creating and submitting secrets, generate strong passwords if necessary, commit changes, and force fluxcd reconcilation.

These scripts can be found in a public GitHub repository: k8s-scripts. While there is some documentation for the scripts and you can look in scripts source code to get more details, this guide expands on the details explaining various options and suggesting optimal settings.

Personal notes: My personal notes on how to setup stuff to make it easier to repeat next time.

Step 1: Prerequisites

  • K8s on Oracle Cloud “Part 4” Basic Setup is completed. Alternatively setup any k8s cluster.
  • K8s on Oracle Cloud “Part 5” Sealed Secrets is completed.
  • K8s on Oracle Cloud “Part 6” Ingress Nginx is completed.
  • K8s on Oracle Cloud “Part 7” Cert-Manager is completed. At this point it is highly recommended to also configure DNS01 challenge issuer, as otherwise it might be impossible to obtain SSL certificates for both the domain and hostnames. It does not need to be through AWS Route53, though.
  • K8s on Oracle Cloud “Part 8” Longhorn is completed. This not mandatory. However, through Longhorn we get automated replication and automated backups, therefore it is highly recommended.
  • At least one e-mail domain registered and ready. This guide covers DNS setup, so preparing DNS is not necessary before starting this guide.
  • If you use AWS Route53 for DNS Zone hosting, you might also want aws cli installed and configured and also obtain your DNS Zone ID from AWS Console as the Mailu installation script can automatically update DNS records on AWS Route53.
  • External relay service for email delivery. My personal preference is AWS SES. It’s reliable and very cheap compared to other services. I pay only for what you use. With my traffic it is about $0.01/month for all the emails I send.

    This might be confusing. We are setting up an email server, which is supposed to deliver emails. Why do we need an external relay service then. In theory we do not need external relay as Mailu is perfectly capable of delivering emails. However, the problem is that emails sent from your installation will most likely be rejected by most of email services like GMail, Yahoo, MS and basically any other. Even self-hosted corporate email systems for large companies will reject our emails or just put it right in spam folder in the best case. This is because the IP address of our email server is either already flagged as spam (used by spammers in the past) or just unknown and untrusted in the best circumstances.

    Therefore to make sure our emails reach the destination and do not end up in spam folders we need a recognized relay service. There are many options available with different price brackets. I personally prerer and use AWS SES. With my low traffic I stay well within free tier.

    Now questions: Is it safe/secure? Can we trust the relay service? Can we really consider it a self-hosted if we use external relay?

    First and foremost. All the communication between accounts on your server, stays on your server. The relay service is only used for emails going outside our server, that is emails addressed to people who have email accounts on different servers. In such a case, email messages leave our server anyway. And how are they handled outside of our server is out of our control anyway. Usage of external relay service does not change much. Besides…, there really is not much choice. All we can do is to select a relay we trust.

Step 2: Configuration

As with all the services being installed using k8s-scripts there is very minimal requirement to adjust configuration. In most cases, the installation script can be run and we end up with working service on our cluster. The same applies to Mailu setup. There are just a few settings that need to be set and we should be good to go. However, Mailu is relatively complex system with lots of functionality and options available. Therefore I am going to expand on the configuration to get as much as possible out of the service.

Version adjustment

There are 2 contradicting suggestions related to the Mailu version used for the setup. First and foremost, email server setup is very sensitive about the configuration settings. Unfortunately, settings and defaults change from version to version. This guide is prepared for Maily chart version 0.3.2. And everything that works, is guaranteed to work for this version. It ‘should’ work for any subsequent version too.

Therefore, I would suggest to follow the guide to install the same version.

On the other hand, Mailu is constantly being worked and and it is being improved. Therefore, I highly suggest to consider installing the most recent version. However, the guide may not apply to that version.

Maybe, the best approach would be to follow the guide first and install the same version as we use in the guide and once it all works, either upgrade your installation to the most recent version or remove Mailu and install it again from scratch to the most recent version.

In either case, if you are installing a different version and find discrepancies in the guide, please let me know in comments and I will update the guide.

Let’s check the current/most recent version of the Mailu chart using the command:

helm search hub --max-col-width 65 mailu | grep "URL\|mailu/mailu"

Example:

~/temp/k8s-scripts$ helm search hub --max-col-width 65 mailu | grep "URL\|mailu/mailu"
URL                                                 	CHART VERSION	APP VERSION	DESCRIPTION
https://artifacthub.io/packages/helm/mailu/mailu    	0.3.2        	1.9        	Mailu mail system

and, adjust the versions.env file with the latest CHART VERSION.

Custom values and configuration

Mailu chart values ~/.tigase-flux/envs/mailu-values.yaml

If all prerequisities were followed to the point then the settings should work fine without a need to any tweaking. However, here are a few pointers to look at if your k8s cluster as some differencies:

  • annotations section can/should be removed or updated if you do not run this on Oracle Cloud k8s.
  • size for volumens can be adjusted. The main volumen for all the email data is Dovecot, so it should have the largest space allocated. Depending on your needs, size of your mailbox, number of user accounts you may need more or less space. Default is set to 50GB. All others can be left small as they do not keep a lot of data.
  • issuerName for the certmanager is set to letsencrypt-dns. And this works if DNS01 challenge is setup as in the cert-manager guide. However, if your domain is not hosted on AWS Route53 you can setup cert-manager issuer for DNS01 challenge for any other provider of your choice.

    Using DNS01 challenge is necessary to obtain SSL certificates for both the “domain” and “hostname” if the system runs on Oracle Cloud k8s. Commonly used HTTP01 challenge cannot be applied for us because on Oracle Cloud k8s the “domain” and “hostname” are exposed through a separate load balancers, thus a different IP address. And cert-manager can only request HTTP01 challenge over one, main IP/LB.

    Using HTTP01 challenge is possible if you setup your k8s cluster on self-hosted system or on a different provider where both “domain” and “hostname” is exposed on the same IP address (the same load balancer).

Mailu env variables ~/.tigase-flux/envs/mailu.env

If you open the file, all the settings are well explained in comments to each setting. There are a few parameters that must be set:

  • MAILU_DOMAIN_AWS_ZONE_ID, MAILU_HOSTNAME_AWS_ZONE_ID

    are recommended if the domains zones are hosted on AWS Route53. Along with aws cli utility, installed and configured with credentials to change zone records it allows for an automatic adjusting DNS settings for the domain and hostname. It can be later, easily done manually, so this can be skipped.

  • AWS_PROFILE

    is used to select correct AWS profile if you have multiple AWS accounts and need use for DNS updates a non-default one.

  • MAILU_DOMAIN

    primary domain for the mailserver. The server supports multiple email domains and additional domains can be added through admin web ui after the server is installed. At this point, however, we set the first, email domain:

    MAILU_DOMAIN=example.com
    

    Please note. MAILU_DOMAIN is used for the domain part of the user’s email addresses but it is also used as address to the Mailu web interface, for both admin console and web email client. Therefore, MAILU_DOMAIN also requires a valid SSL certificate.

  • MAILU_HOSTNAMES

    hostnames are DNS addresses used to connect to the Mailu email server. It can be the same as what we set for MAILU_DOMAIN or something like: ‘mail.example.com’, ‘imap.example.com’. In any case, settings this property is critical because Mailu will be maintaining SSL certificates for all these hostnames to allow secure connections to the email server.

    MAILU_HOSTNAMES=(mail.${MAILU_DOMAIN} ${MAILU_DOMAIN})
    

    Please note. The order in the above is important for the k8s-scripts to correctly adjust DNS settings for domain and hostnames. The first has to be a hostname used to access email server via IMAP/POP/SMTP and then other domains. If you are going to manully adjust DNS, the order does not matter.

    Please note. You can skip the ${MAILU_DOMAIN} entry (example.com) from the MAILU_HOSTNAMES property if you do not plan on accessing the mail server via Web UI or do not care about having a valid SSL certificate for the web service. Because example.com is here only for this reason - to obtain a valid SSL certificate for WEB UI access.

    Please note. In theory you can setup email server and DNS to use a single domain/hostname example.com to access both, the WEB UI and IMAP/POP/SMTP. However, on Oracle Cloud k8s, you get a separate load balancer (and IP address) for web access and all other protocols. Web access is provided through the cluster main LB over ingress, while IMAP/POP/SMTP is provided over additional LB as “service”.

  • All the settings related to admin account. Yes, we need to somehow access the admin web ui.

    If you leave MAILU_ADMIN_USERNAME or MAILU_ADMIN_PASSWORD empty, the installation script will ask you to provide values. If you press ENTER providing empty value, the script will generate a random 12 characters username and a random 16 character password for you. The password and other mailu access details are then added to ~/.tigase-flux/k8s-secrets file.

  • MAILU_SUBNET is dependent on the k8s cluster settings. You need to retieve the subnet details from the cluster console or some other place. For the k8s cluster on Oracle Cloud, the subnet usually is: ‘10.244.0.0/16’
    MAILU_SUBNET="10.244.0.0/16"
    
  • Settings for the relay host are not mandatory as you can use the Mailu installation to deliver emails. However, as mentioned before, your email delivery might be very inconsistent and unreliable. Using an external relay service with a good reputation is highly recommended.

Domain and hostname(s) configuration

Domains, hostnames and DNS configuration is relatively simple if you understand how it all works. To understand it better, let’s explain what is ‘domains’ and ‘hostnames’ in context of Mailu installation.

  • domain is the DNS part of your email address, like example.com. For example, user1@example.com, user2@example.com, user@mail.example.com. Of course there can be more than 1 domain hosted on Mailu email server like: example-1.com, example-2.com and so on. Then, user@example-1.com and user@example-2.com are 2 different/separate user’s accounts.

    At the Mailu installation time you provide 1 domain only. This is the email server primary and main domain. After installation is completed, you can add as many domains as you like through the admin’s Web UI.

  • hostname is the address of your email server which is used to reach the server over IMAP/POP/SMTP protocol. There can be one or more such addresses (hostnames) and they can be completely separate from the domain address. Typicaly however, hostname is a subdomain of the domain part. For example mail.example.com, imap.example.com and so on.

DNS for hostnames like mail.example.com

You need a dedicated DNS address to access your email server over IMAP/POP/SMTP for the email messages submission. Let’s call it mail.example.com.

This address needs to point to an IP of your k8s cluster load balancer configured for IMAP/POP/SMTP ports. However, for Oracle Cloud k8s you do not know the address before Mailu is installed because this load balancer is created during installation. Therefore you need to adjust DNS settings after Mailu installation is completed. This can be done manually or with additional configuration the installation script can do it automatically. For this to work the mail.example.com has to be the first item on the MAILU_HOSTNAMES list.

For this reason, SSL certificate for the hostname cannot be generated using common HTTP01 challenge, which works only for DNS addresses pointing to the main k8s cluster load balancer (IP) address. Instead, DNS01 challenge has to be used. Therefore, make sure your cert-manager is configured to obtain certificates using DNS01 challenge and exposes this over letsencrypt-dns or any other issuer with name matching issuerName in mailu-values.yaml.

If the hostname is used for email submission, that is to receive emails messages, you have to make sure your email domains have MX records pointing to the hostname. For example, if you have email domains: example-1.com, example-2.com, the MX records should look like this:

example-1.com.	600   IN	MX	10	mail.example.com.
example-2.com.	600   IN	MX	10	mail.example.com.

Regardless what email domain you use for your email accounts, your email clients and apps will always have the same configuration:

Protocol TCP Port Server Name Username Password
IMAP 993 (TLS), 143 (StartTLS) mail.example.com user’s email address user’s password or token
SMTP 465 (TLS), 587 (StartTLS) mail.example.com user’s email address user’s password or token

DNS for domain like example.com or any domain used for email accounts

As explained above, by domain we mean a DNS address used to create domain part of an email account, like **example.com** in the user@example.com email address.

A Mailu server can keep email for any number of domains with any number of accounts in each domain. DNS configuration for each domain is rather simple. In the simplest form, all we need, is an MX record for each domain pointing to the Mailu server hostname. However, to make the email delivery more reliable and user’s life easier we should add a few more entries.

Let’s say, like in the above example our Mailu server hostname is mail.example.com, and we have domains: example.com, domain.com, just-4.linux, mail.company.com, lawfirm.com for which we want to host email accounts on our server. Now, ideally we need to setup following DNS records for each domain like this:

Domain name example.com
DNS MX entry example.com. 600 IN MX 10 mail.example.com.
DNS SPF entries example.com. 600 IN TXT “v=spf1 mx a:mail.example.com ~all”
DNS DKIM entry This can be set after successful Mailu installation
DNS DMARC entry
_dmarc.example.com. 600 IN TXT “v=DMARC1; p=reject; adkim=s; aspf=s”
example.com._report._dmarc.example.com. 600 IN TXT “v=DMARC1”
DNS Client auto-
configuration
RFC6186 entries

_submission._tcp.example.com. 600 IN SRV 1 1 587 mail.example.com.
_imap._tcp.example.com. 600 IN SRV 1 1 143 mail.example.com.
_pop3._tcp.example.com. 600 IN SRV 1 1 110 mail.example.com.
_imaps._tcp.example.com. 600 IN SRV 1 1 993 mail.example.com.
_pop3s._tcp.example.com. 600 IN SRV 1 1 995 mail.example.com.

That is all. It may seem like a lot but once you set these records you will never need to change them, not even during/after Mailu installation, upgrade, restoration, the IP address of the mail.example.com may change but this does not affect any of the email domains and DNS records set above. We do not need to correct them, change or update. One exception is DNS DKIM entry which can only be set after Mailu is installed and DKIM key is generated which is done from a Mailu web admin UI.

DNS for primary domain like example.com

Usually your primary domain is also used for email accounts. But it does not have to. If it is used for email accounts all the comments and setings from previous section apply. If it is not used for email accounts, the DNS settings from previous section can be skipped.

However, the primary domain is primarly used for the email server management and Web access. For this to work correctly the following configuration must be set correctly:

  • IP address for the primary domain must point to the IP of the primary cluster load balancer. The load balancer on which ingress is exposed. This is usually the same load balancer which is created at the time of ingress (nginx) deployment. This can be set manually, even before running Mailu installation script or you can let the script to set the correct IP for domain automatically at the end of the installation process.
  • The primary domain must be set in the MAILU_DOMAIN property in the ~/.tigase-flux/envs/mailu.env file. This is necessary to create correct ingress on the k8s cluster for web access.
  • The primary domain must be added to the end of the MAILU_HOSTNAMES property in the ~/.tigase-flux/envs/mailu.env file. This is necessary to obtain an SSL certificate for the domain which allows secure web access.

All the above settings and IP address are adjusted correctly by default. In practice you only need to set the primary domain in MAILU_DOMAIN and the rest is adjusted automatically in a proper way.

If the domain is hosted on AWS Route53, then the IP address can also be adjusted automatically.

DNS final IP adjustment for primary domain and hostnames

In short, the primary domain (MAILU_DOMAIN) must point to the IP of the primary load balancer of the k8s cluster and each hostname (except primary domain) from the MAILU_HOSTNAMES list must point to the IP of the load balancer created during Mailu deployment.

The Mailu installation script can adjust DNS records for primary domain and the first hostname on the MAILU_HOSTNAMES list automatically. For the automatic DNS adjustment, the following are needed:

  1. The domain zone is hosted on AWS Route53
  2. Both, the primary domain and the hostname are in the same DNS zone
  3. The AWS Zone ID must be provided in the mailu.env configuration file in the property: MAILU_AWS_ZONE_ID
  4. AWS cli aws must be installed and configured with correct permission to change the DNS zone

If the requirements above are met, you will get a message at the end of the Mailu installation like this:

      Updating DNS for example.com and mail.example.com
      Pointing example.com -> 1.2.9.4 and mail.example.com -> 2.7.6.6
CHANGEINFO    /change/XXXXXXXXXXXXX PENDING 2022-06-22T19:45:55.546Z

Otherwise, if the automatic DNS update is not configured and not possible, the MAILU_AWS_ZONE_ID should not be set. The script then does not attempt to update records automatically and only prints a message how to update DNS entries:

      Cannot automatically update DNS for domain and host. Make necessary adjustments:
      Point example.com -> 1.2.9.4 and mail.example.com -> 2.7.6.6

If you have more than one hostname in the MAILU_HOSTNAMES property, you need to adjust DNS settings for them manually and point them to the same IP address as your first hostname entry.

Step 3: Installation

At this point, after we went through all the configuration details, the rest of the process is rather simple. It is just executing a single script which takes care of deploying Mailu to the k8s cluster.

Please note. The whole installation process on the k8s cluster may take long time as there is a lot to do on the cluster to set everything up. I ran the script more times that I can remember and on some runs it took over 15 minutes before Mailu was ready to use. On most runs it took between 2 to 5 minutes to complete the installation. Default timeout is set to 20 minutes. However, if your installation fails with FluxCD error and you find that Mailu is working anyway after waiting a couple of more minutes, you can uninstall Mailu, increase the timeout in mailu.env, property MAILU_FLUXCD_TIMEOUT and run installation again.

The installation script makes the following changes:

  1. Installs Mailu chart source repository in the cluster’s git repository
  2. Creates Helm release manifest with all the configuration options combined in cluster’s git repository
  3. Creates a separate namespace for Mailu on the k8s cluster
  4. Records admin account and password in ~/.tigase-flux/k8s-secrets file
  5. Commits cluster’s changed git repository and pushes the changes
  6. Runs flux reconciliation

Mailu deployment makes following changes on the k8s cluster:

  1. mailu-prod namespace is created
  2. 7 new PVC are created, each for a different Mailu component
  3. Mailu containers are setup and run
  4. SSL certificates are obtained for all hostnames
  5. A new ingress for the primary domain is created
  6. A new service for IMAP/POP/SMTP is created which results in a new load balancer creation

To run the installation process, execute the script:

./scripts/cluster-mailu.sh

Example run:

~/temp/k8s-scripts$ ./scripts/cluster-mailu.sh 
Switched to context "cluster-name".
Provide admin password or ENTER to generate: 
      Adding Mailu helm chart
Switched to context "cluster-name".
/home/k/.tigase-flux/projects/cluster-name
[master f7976db] mailu deployment
 2 files changed, 12 insertions(+)
 create mode 100644 infra/common/sources/mailu.yaml
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 16 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 790 bytes | 790.00 KiB/s, done.
Total 7 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/a/cluster-name
   e1379d6..f7976db  master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/f7976db6da0ab605f022e5c9d970784d813571b7
Waiting for the system to be ready
Update service kustomization for infra/common/sources in .tigase-flux/projects/cn
/home/k/.tigase-flux/projects/cluster-name
      Preparing Mailu deployment
Switched to context "cluster-name".
Update service kustomization
/home/k/.tigase-flux/projects/cluster-name
Update namespace kustomization
/home/k/.tigase-flux/projects/cluster-name
Update common kustomization
/home/k/.tigase-flux/projects/cluster-name
      Deploying changes
[master 3105f40] mailu deployment
 6 files changed, 128 insertions(+), 1 deletion(-)
 create mode 100644 infra/apps/mailu-prod/kustomization.yaml
 create mode 100644 infra/apps/mailu-prod/mailu/kustomization.yaml
 create mode 100644 infra/apps/mailu-prod/mailu/mailu.yaml
 create mode 100644 infra/apps/mailu-prod/namespace.yaml
Enumerating objects: 20, done.
Counting objects: 100% (20/20), done.
Delta compression using up to 16 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 2.05 KiB | 2.05 MiB/s, done.
Total 14 (delta 4), reused 2 (delta 2), pack-reused 0
remote: Resolving deltas: 100% (4/4), completed with 2 local objects.
To https://github.com/a/cluster-name
   f7976db..3105f40  master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/3105f405e4d2d77b08c4d4a5dea737ed806f0f03
Waiting for the system to be ready
2022-06-22 12:43:27 System not ready yet, waiting anoher 20 seconds
2022-06-22 12:43:51 System not ready yet, waiting anoher 20 seconds
2022-06-22 12:44:15 System not ready yet, waiting anoher 20 seconds
2022-06-22 12:44:39 System not ready yet, waiting anoher 20 seconds
2022-06-22 12:45:04 System not ready yet, waiting anoher 20 seconds
2022-06-22 12:45:28 System not ready yet, waiting anoher 20 seconds
      Updating DNS for example.com and mail.example.com 
Setting example.com -> 1.2.9.4 and mail.example.com -> 1.7.5.6
CHANGEINFO		/change/C05669653KLE08O2H7RBO	PENDING	2022-06-22T19:45:55.546Z

During the installation process you may see repeated message:

2022-06-22 12:43:27 System not ready yet, waiting anoher 20 seconds

which can go on for 20 minutes or more, depending on the timeout you set. There is a way to check what is taking so much time. If you run in a separate terminal command like this:

while k get pods -A | grep "0/1" ; do date; sleep 20; done

you could see which component is still not ready. For example:

$ while k get pods -A | grep "0/1" ; do date; sleep 20; done
mailu-prod        mailu-admin-847f9d5f78-trmc7                   0/1     Running             0               34s
mailu-prod        mailu-clamav-7dc6bbf4c4-khlpc                  0/1     Running             0               34s
mailu-prod        mailu-dovecot-696bc8645-pqbg5                  0/1     ContainerCreating   0               34s
mailu-prod        mailu-front-ctskd                              0/1     ContainerCreating   0               34s
mailu-prod        mailu-front-w5cnc                              0/1     ContainerCreating   0               34s
mailu-prod        mailu-front-zfbv4                              0/1     ContainerCreating   0               34s
mailu-prod        mailu-postfix-5845776859-pwfxq                 0/1     Running             0               34s
mailu-prod        mailu-redis-6fb78fd4b9-fsl8l                   0/1     Running             0               34s
mailu-prod        mailu-roundcube-84f7cb8c96-gxbqs               0/1     ContainerCreating   0               34s
mailu-prod        mailu-rspamd-57b5fcd445-5m57s                  0/1     Running             0               34s
Wed Jun 22 12:43:38 PM PDT 2022
mailu-prod        mailu-admin-847f9d5f78-trmc7                   0/1     Running             0               55s
mailu-prod        mailu-clamav-7dc6bbf4c4-khlpc                  0/1     Running             0               55s
mailu-prod        mailu-dovecot-696bc8645-pqbg5                  0/1     Running             0               55s
mailu-prod        mailu-front-ctskd                              0/1     ContainerCreating   0               55s
mailu-prod        mailu-front-w5cnc                              0/1     ContainerCreating   0               55s
mailu-prod        mailu-front-zfbv4                              0/1     ContainerCreating   0               55s
mailu-prod        mailu-roundcube-84f7cb8c96-gxbqs               0/1     Running             0               55s
mailu-prod        mailu-rspamd-57b5fcd445-5m57s                  0/1     Running             0               55s
Wed Jun 22 12:43:59 PM PDT 2022
mailu-prod        mailu-clamav-7dc6bbf4c4-khlpc                  0/1     Running             0               75s
mailu-prod        mailu-front-ctskd                              0/1     ContainerCreating   0               75s
mailu-prod        mailu-front-w5cnc                              0/1     ContainerCreating   0               75s
mailu-prod        mailu-front-zfbv4                              0/1     ContainerCreating   0               75s
mailu-prod        mailu-rspamd-57b5fcd445-5m57s                  0/1     Running             0               75s
Wed Jun 22 12:44:19 PM PDT 2022
mailu-prod        mailu-front-ctskd                              0/1     ContainerCreating   0               95s
mailu-prod        mailu-front-w5cnc                              0/1     ContainerCreating   0               95s
mailu-prod        mailu-front-zfbv4                              0/1     ContainerCreating   0               95s
mailu-prod        mailu-rspamd-57b5fcd445-5m57s                  0/1     Running             0               95s
Wed Jun 22 12:44:40 PM PDT 2022
mailu-prod        mailu-front-ctskd                              0/1     ContainerCreating   0               116s
mailu-prod        mailu-front-w5cnc                              0/1     ContainerCreating   0               116s
mailu-prod        mailu-front-zfbv4                              0/1     ContainerCreating   0               116s
mailu-prod        mailu-rspamd-57b5fcd445-5m57s                  0/1     Running             0               116s
Wed Jun 22 12:45:00 PM PDT 2022
mailu-prod        mailu-front-ctskd                              0/1     Running             0               2m17s
mailu-prod        mailu-front-w5cnc                              0/1     ContainerCreating   0               2m17s
mailu-prod        mailu-front-zfbv4                              0/1     ContainerCreating   0               2m17s
mailu-prod        mailu-rspamd-57b5fcd445-5m57s                  0/1     Running             0               2m17s
Wed Jun 22 12:45:21 PM PDT 2022

In the example above we can see that it took over 2 minutes for mailu-front and mailu-rspamd to be ready. When the command finishes, it means there is no more components waiting to be ready.

The installation script terminates when either Mailu installation is successfully completed or timeout runs out. You can check to see what is the Mailu status reported by FluxCD:

$ flux get hr -A
NAMESPACE      	NAME          	REVISION	SUSPENDED	READY	MESSAGE                          
cert-manager   	cert-manager  	v1.8.1  	False    	True 	Release reconciliation succeeded	
flux-system    	sealed-secrets	2.2.0   	False    	True 	Release reconciliation succeeded	
ingress-nginx  	ingress-nginx 	4.1.4   	False    	True 	Release reconciliation succeeded	
k8s            	dash          	5.7.0   	False    	True 	Release reconciliation succeeded	
longhorn-system	longhorn      	1.3.0   	False    	True 	Release reconciliation succeeded	
mailu-prod     	mailu         	0.3.2   	False    	True 	Release reconciliation succeeded	

Successful installation is when READY=True.

Step 4: Post-installation steps

Assuming everything went well, we have Mailu installed and ready to use as our email server. What should we do next?

Before we can start using it, we have to make sure the correct, new IP address is updated for domain and hostname and already propagated on the internet. Depending on the TTL settings for the domain zone and depending on provider it make take from a few minutes to a few hours. For AWS Route53 it is really fast, almost instant.

We can check this using host command like this:

host example.com

For example:

$ host example.com
example.com has address 1.7.5.6
example.com mail is handled by 10 mail.example.com.

and the same for the hostname:

$ host mail.example.com
mail.example.com has address 1.2.9.4

If the IP is correct in both cases you really can start using your email server.

Check the Web UI

  • Open your domain link in the web browser: https://example.com and login to admin account.

    You can find your admin login credentials in ~/.tigase-flux/k8s-secrets file. Scroll down to the bottom. There will be lines starting with: mailu_admin_user and mailu_admin_pass.

  • You may want to go to Authentication tokens tab and create separate tokens for each email app you plan to use to avoid compromising your main account password.
  • Go to Client setup tab and configure your email client according to instructions. Check if you can connect using IMAP.
  • Check the Mail domains, click Details icon in Actions. Click Regenerate keys and update DNS records for the domain as in the instruction.

Check email server setup health and ensure you do not have an open relay

This step is simple. Go to MXToolBox Diagnostic, enter your hostname, like mail.example.com and click Test Email Server.

As a result you will get a report about the email server health and verification if it is an open relay.

For example:

  Test Result  
W SMTP Reverse DNS Resolution Reverse DNS Resolution - No PTR Record found More info
Ok SMTP TLS OK - Supports TLS.  
Ok SMTP Connection Time 0.226 seconds - Good on Connection time  
Ok SMTP Open Relay OK - Not an open relay.  
Ok SMTP Transaction Time 1.020 seconds - Good on Transaction Time  

If you do not use an external relay service to deliver your email, you may also use MxToolBox to check if your hostname IP addres is on any blacklist. The service also offers some help with removing IP from some black lists. This can improve deliverability of your emails.

Check if you can send and receive emails

Final test is to see if you can actually send and receive emails through your email server. Send an email from one of the public email services (Google, iCloud, etc…) to your admin account. Once you received it, send a reply back.

Uninstallation

To uninstall and completely remove Mailu from your k8s cluster you can run the script with --remove parameter:

./scripts/cluster-mailu.sh --remove

Please note! This will completely remove Mailu installation along with PVC created during installation, hence all data, mailboxes, user accounts will be lost and cannot be recovered unless there is a data backup. Even with backup, restoring Mailu installation might be complex and time consuming.

Example:

~/temp/k8s-scripts$ ./scripts/cluster-mailu.sh --remove
Switched to context "cluster-name".
    Preparing to remove: mailu
    Removing: infra/apps/mailu-prod
Update service kustomization for infra/apps/ in .tigase-flux/projects/cn
/home/k/.tigase-flux/projects/cluster-name
    Removing: infra/common/sources/mailu.yaml
Update service kustomization for infra/common/sources in .tigase-flux/projects/cn
/home/k/.tigase-flux/projects/cluster-name
[master e1379d6] Removing mailu deployment
 7 files changed, 138 deletions(-)
 delete mode 100644 infra/apps/mailu-prod/kustomization.yaml
 delete mode 100644 infra/apps/mailu-prod/mailu/kustomization.yaml
 delete mode 100644 infra/apps/mailu-prod/mailu/mailu.yaml
 delete mode 100644 infra/apps/mailu-prod/namespace.yaml
 delete mode 100644 infra/common/sources/mailu.yaml
Enumerating objects: 14, done.
Counting objects: 100% (14/14), done.
Delta compression using up to 16 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 778 bytes | 778.00 KiB/s, done.
Total 8 (delta 3), reused 2 (delta 1), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/a/cluster-name
   9c86501..e1379d6  master -> master
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/e1379d6af35377115b29dbcd89439ff5efe112eb