Remote Testing

As Jürgen Egeling told you on the German side of this blog the .infrastructure team of punkt.de developed a new email system, to replace the legacy system we had been using for more than a decade. In the course of setting it up, I came up with a simple way of testing which I want to tell you about in this blog post.

Considerations for a new mail system

Like many others we have got two network-zones: internal and external; but we do not use private IPv4-Addresses, like e.g. 192.168.0.0/24. The distinction between internal and external is purely logical and implemented in our firewall. Postfix supports this kind of distinction with the $mynetworks configuration option and takes a list of networks and addresses, that should be considered internal. Additionally, you are able to configure a wide range of restrictions. This is to keep off spam-sending hosts and mail accounts from those you want to receive mail from, often already at the connection level. You do this before you even accept the email, based on for example which network the host is coming from, which protocol is used or if a successful login has occurred. Only listing and explaining all the restrictions and their combinations with the respective outcomes could fill a blog post on its own and is not in the scope of this one. It took us quite some time to figure out the right restriction set for our requirements.

Our legacy system featured two mailin-relays, which were located in the external network and forwarded mails via our firewall into our internal network to a central server. This server provided not only mail access but also file-shares and other services. That means it had to be inaccessible from the internet because some sensitive data was stored on there. This kind of setup was pretty normal for its time - but it also meant you had to be connected to the VPN when you were outside the office in order to use e-mail. Today people are used to being connected everywhere all the time on any device and punkt.de is very open towards working from home or wherever you are. So when we asked colleagues to list their requirements one of the features meantioned the most was to drop the necessity of VPN connections just to use email.

Our main requirements for a new email system were:

Accessibility from the internet over SMTP(S) and IMAP(S)
Support of different rulesets for internal and external sending systems as well as logged-in accounts
High security with transport encryption by default
Early rejection of spam
Free/Libre Open Source Software
Possibility to run on our own proServer product
Possibility to expand the system to consist of several VMs/containers for mailin-relays, mailout-relays and a mailstore

After considering our options and the lessons we learned from running the old system based on good old sendmail, we decided to give Postfix a shot and ended up with the commonly used combination of Postfix and Dovecot. Postfix has gained some weight among the mail transfer agents, especially with its clear and concise configuration file format while preserving the power and flexibility to master complex and challenging requirements. Dovecot has become the de facto standard as IMAP server and can be used as authentication backend for Postfix. Although our environment is clearly not that different from to other companies, every setup has got intricate details. So it becomes necessary to check whether your current configuration is meeting the requirements without compromising security.

Testing

In the .infrastructure team we use Ansible to set up our systems and we developed an Ansible project to automate the setup of our mailsystem. I am a big fan of automated testing because it helps to ensure that your code does exactly what it's supposed to do. So I wanted this safety net while learning to set up a mail server in general combined with learning how to set up Postfix specifically. Ansible, which is written in Python, does not provide the possibility of automatically testing the provisioned system's configuration. In Ansible you define the setup of operating system and application configuration; its job is to ensure that it puts the system into the state you defined. But if your configuration definition is correct and secure does not lie within the scope of Ansible's intentions; if you configure a service to be insecure, then Ansible will make sure the service will have that state. So I have been searching for a possibility to be able to use the technique of Test Driven Development using Python's unittest framework. I wanted to test the configured behaviour of Postfix and Dovecot from different network locations to ensure our configuration was correct concerning our requirements and security. The problem I was facing though was the fact that usually test code is either run locally by the developer or on a CI/CD system. Ansible does deliver an email module with which you can send mail from different hosts, but it is rather restricted in terms of faking email headers like e.g. envelope headers. Normally this is not necessarily a flaw when you really want to send mail automatically, but I wanted to test the behaviour of Postfix depending on certain email headers, too.

In the beginning a bit of research dug up the usual suspects like paramiko or fabric, but those two concentrate on giving you a remote shell, which you then can programmatically pass commands to as strings and return their output as strings as well. I thought about using the smtplib module included in Python but using that in combination with paramiko or fabric¹ would have required me to

eventually send code to the remote host
open a shell on the remote host
call Python to execute the previously sent code
parse the returned output

Another possibility would have been to write a daemon, which would send mail when triggered; but I did not like the prospect of maintaining a code base for a daemon like this and make sure it was running properly on all the remote hosts when needed.

Finally, I stumbled over rpyc. It is a small RPC-library, that can use plumbum underneath to connect via ssh to a remote host and remotely call python modules on that host, with what they call Zero-Deploy, which takes care of transferring the code to the other side and executing it over there. So now I'm able to run pretty simple tests, which try to send email from different hosts to our mail system even with faked headers. After that my method call simply returns or raises an exception and I can let my test pass or fail, depending on which behaviour I configured Postfix to show. It also handles object serialisation, so it is possible to just call the methods of provided proxy objects and get their returned obejcts as if the originals were called.

Example

Because sometimes a few lines of code speak louder than a lot of words I'll show you an example and explain the key parts afterwards:

import unittest
import rpyc
import plumbum as pb
import smtplib
 
from email.message import EmailMessage
 
from rpyc.utils.zerodeploy import DeployedServer
 
 
class ExternalNetworkRestrictions(unittest.TestCase):
 
    def test_logged_out_smtp_expect_client_host_rejected(self):
        message = EmailMessage()
        message["From"] = "sender@example.com"
        message["To"] = "receiver@example.com"
        message["Subject"] = "Test"
 
        machine: pb.SshMachine = pb.SshMachine(host="remote.example.com", user="user",
                                               keyfile="~/.ssh/id_rsa")
        server = DeployedServer(machine)
        connection: rpyc.Connection = server.classic_connect()
 
        with connection.modules.smtplib.SMTP("mail.example.com") as smtp:
            with self.assertRaises(smtplib.SMTPRecipientsRefused) as e:
                smtp.send_message(message,
                                  from_addr="sender@example.com",
                                  to_addrs="receiver@example.com")
 
            self.assertIn("Client host rejected: Access denied", str(e.exception))
 
        connection.close()
        server.close()
        machine.close()
 
 
if __name__ == "__main__":
    unittest.main()

The first lines to discuss are 13-17: Here I'm using Python's included email.message module to create an instance of EmailMessage and set the email headers.
In line 19 I'm configuring the ssh-connection to my remote host, using my system user's ssh-key. This remote host in not to be considered internal by Postfix, so Postfix should not let the host send email to it via unencrypted SMTP at port 25. Contrary to hosts in our internal network, which should be able to, like our mailin-relays.
Line 21 sets up rpyc's Zero-Deploy functionality and line 22 actually connects to the remote host.
The connection object is the key-object for executing code on the remote side. It proxies the whole Python environment of the other side.
In line 24 I use the modules provided by Python on the remote host. This is the same as if I used the following code on the remote host and then executed that code:

import smtplib 

[...]

with smtplib.SMTP(...) as smtp:

    [...]

Line 25 prepares Python's unittest to fail if the following line doesn't raise an SMTPRecipientsRefusedError. I do this because the SMTP connection is made unencrypted on port 25 and I configured Postfix to reject hosts, which open connections via SMTP and no login. The actual rejection happens during the check of Postfix's recipient_restrictions; this is why I expect that kind of exception to be raised.
Now I try to actually send my previously created email message in line 26, adding the envelope-headers to that email.
Finally in line 30 I check whether the error message given by the server after the exception matches "Client host rejected: Access denied", because my sending client should have been rejected with this message.
Afterwards some cleanups and proper closing of resources and that's it.

Conclusion

This blog post shows how the usage of an RPC-Library opened up the possibility of using a unittest framework to remotly test the configuration of an email server. With this it becomes possible to include remote testing in an Ansible project and let a CI/CD run the tests as soon as changes are pushed to git. Now we get instant checking and feedback if we had previously introduced unwanted configuration changes

If you want to see more examples about how to test cases like using transport-encryption with SMTPS or STARTTLS over their respective ports, please have a look at GitHub, where we published a repository including more examples. If you liked this post and want to know more about our new e-mail system, which will be available for customers soon, get in touch.

¹: fabric version 1 does have a method to execute code on a remote host but that is considered deprecated (soon) and fabric version 2 does not yet have the functionality

Remote Testing

Most of our development blog posts are about TYPO3, Neos/Flow and PHP, and clearly this is where our main expertise is at. But today Lars will tell you about a way of testing remotely with Python.

Considerations for a new mail system

Testing

Example

Conclusion

Comments

Other Posts

A toolbar for page properties in TYPO3

Responsive Images in TYPO3 v10