Pentagrid AG (Posts about Encryption)

Nothing new, still broken, insecure by default since then: Python's e-mail libraries and certificate verification

Pentagrid AG — Tue, 14 Nov 2023 09:23:00 GMT

Today, basically every e-mail provider supports TLS for their services and programmatically accessing e-mail services with Python code using TLS-wrapped clients is common. Python offers three libraries shipped with a standard installation for handling e-mail transfer. These modules are smtplib, imaplib, and poplib. While Python programming is usually straightforward, using these Python libraries require passing a magic parameter in the right way to use secure communication. If one has just read the summary on Stackoverflow, read a tutorial that does not mention security settings, or asked ChatGPT not specifically enough, it results in programs that do not defeat active attackers in a machine-in-the-middle (MITM) position. Our journey started, when we wrote an e-mail monitoring plugin in Python and ended for the time being with the notification of various open source projects.

Insecure past

There was a time, when fetching e-mail via clear-text channels was the standard and there was no real alternative to that. SMTP is 40 years old. The encryption program PGP is 31 years old and can be understood as tool that originated from the desire for reasonably useful end-to-end privacy, at a time where the underlying foundation was based on clear-text. In the mid-nineties, the first crypto-protocols for regular users were developed, such as SSH for shell access and SSL mainly for web browsing. For e-mail, STARTTLS was an improvement to upgrade from an insecure communication to a crypto-channel that preserves integrity and confidentiality as long as an active attacker did not prevent STARTTLS. The pre-Snowden era is not that long ago, where websites usually relied on cleartext-HTTP. These old times influenced how things are nowadays. Nevertheless, times are changing, techniques too, and expectations anyway.

The same reasoning applies to libraries with protocol implementations. It is not different with Python. So, everything here is an old hat. It is a "known" problem. It is more or less documented in the corresponding Python documentation. There is a Python Enhancement Proposal, a PEP, which is the Python community's way of coming to decisions. It is PEP 476 – Enabling certificate verification by default for stdlib HTTP clients from 2014, because HTTP libraries had the same issue. They were insecure by default. PEP 476 suggests doing certificate checking for accessing resources via HTTPS and it also mentions that there are other Python libraries with the same issue. These other libraries are smtplib, imaplib, and poplib for mail handling and additional libraries for FTP and NNTP. The PEP explains that the mail libraries may still need insecure defaults for backward compatibility, for example because mail servers may have invalid certificates and therefore with e-mail this certificate verification could be a different thing compared to the PEP's aim at changing certificate verification for HTTP. And there is the very same discussion already on Python's Github tracker.

Making things worse, while imaplib and poplib are mainly used in client to server communication (where you might have a chance to ask the user to do a security decision), SMTP is used for server to server communication as well. While security features are introduced to improve mail security (e.g. MTA-STS), the situation is still challenging. That's why we think projects should configure e-mail as secure as possible, where possible.

Meanwhile it is 2023 and insecure defaults are a problem, especially when our infrastructure depends on many small building blocks. An insecurity here, a shared password there, an outdated component over there. Each vulnerability could be one too much to grant attackers the next level of access. So let us explain the problem.

A vulnerable example

Consider this example of Python script that is intended to connect to an IMAP server:

#!/usr/bin/env python3

import imaplib

imap_host = 'outlook.office365.com'
imap_user = 'nonexisting@outlook.com'
imap_pass = 'secret'

print("IMAP: Try to connect.")
server = imaplib.IMAP4_SSL(host=imap_host, port=993)

print("IMAP: Try IMAP login, which is expected to fail with the data above.")
server.login(imap_user, imap_pass)

At first, the script looks like it should work, which it does. But if this script is used, it will not verify the peer's certificate, which can be tested by just adding an entry to /etc/hosts, for example by associating an IP address from imap.gmail.com with the hostname of outlook.office365.com:

173.194.76.109  outlook.office365.com

If the script now connects to Microsoft's IMAP server, it will instead connect to the one from Google. The certificate will not match the hostname, but running the script now results in an authentication error and not in an ssl.SSLCertVerificationError. As there was no certificate verification error, username and password were sent to the wrong server and the script did not prevent this.

$ ./imaptest.py
IMAP: Try to connect.
IMAP: Try IMAP login, which is expected to fail with the data above.
Traceback (most recent call last):
  File "/tmp/./imaptest.py", line 13, in <module>
    server.login(imap_user, imap_pass)
  File "/usr/lib/python3.10/imaplib.py", line 612, in login
    raise self.error(dat[-1])
imaplib.IMAP4.error: b'[AUTHENTICATIONFAILED] Authentication failed.'

Outch!

Read the manual first

This was just an example with accessing an IMAP server. Technically, it is the same with Python's smtplib and also poplib. This is more or less documented behaviour. The imaplib.IMAP4_SSL has an optional parameter ssl_context, which is described as:

ssl_context is a ssl.SSLContext object which allows bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure. Please read Security considerations for best practices.

The ssl_context parameter was added in Python 3.3 released in 2012. Following the link to the security best practices, the Python documentation further explains:

For client use, if you don’t have any special requirements for your security policy, it is highly recommended that you use the create_default_context() function to create your SSL context. It will load the system’s trusted CA certificates, enable certificate validation and hostname checking, and try to choose reasonably secure protocol and cipher settings. [...]

By contrast, if you create the SSL context by calling the SSLContext constructor yourself, it will not have certificate validation nor hostname checking enabled by default. If you do so, please read the paragraphs below to achieve a good security level. [...]

When calling the SSLContext constructor directly, CERT_NONE is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you’re talking to. Therefore, when in client mode, it is highly recommended to use CERT_REQUIRED. However, it is in itself not sufficient; you also have to check that the server certificate, which can be obtained by calling SSLSocket.getpeercert(), matches the desired service. For many protocols and applications, the service can be identified by the hostname; in this case, the match_hostname() function can be used. This common check is automatically performed when SSLContext.check_hostname is enabled.

Would you say that after reading this, everything is completely clear? Creating a default context to load CA certificates sounds good, but is the impact clear, when one skips this step and just does not pass the right SSL context, when calling the IMAP4_SSL constructor? Does the code call the SSLContext contructor somehow, when you completely ignore the IMAP4_SSL constructor's ssl_context parameter? The Python documentation links the corresponding source code. So one can check, what happens starting from the constructor imaplib.IMAP4_SSL:

def __init__(self, host='', port=IMAP4_SSL_PORT, keyfile=None,
             certfile=None, ssl_context=None, timeout=None):
[...]

    if ssl_context is None:
        ssl_context = ssl._create_stdlib_context(certfile=certfile,
                                                 keyfile=keyfile)
    self.ssl_context = ssl_context
    IMAP4.__init__(self, host, port, timeout)

There, the ssl_context parameter is initialized with None. If no specific value is passed via ssl_context, then _create_stdlib_context is called. The function ssl._create_stdlib_context is defined as:

_create_stdlib_context = _create_unverified_context

Unverified context does not sound good. The corresponding code is shown below. The function _create_unverified_context disables check_hostname and does not load a default trust chain, if not instructed to do this:

def _create_unverified_context(protocol=None, *, cert_reqs=CERT_NONE,
                           check_hostname=False, purpose=Purpose.SERVER_AUTH,
                           certfile=None, keyfile=None,
                           cafile=None, capath=None, cadata=None):
[...]
    context = SSLContext(protocol)
    context.check_hostname = check_hostname
    if cert_reqs is not None:
        context.verify_mode = cert_reqs
    if check_hostname:
        context.check_hostname = True
[...]
    if certfile or keyfile:
        context.load_cert_chain(certfile, keyfile)

    # load CA root certs
    if cafile or capath or cadata:
        context.load_verify_locations(cafile, capath, cadata)
    elif context.verify_mode != CERT_NONE:
        # no explicit cafile, capath or cadata but the verify mode is
        # CERT_OPTIONAL or CERT_REQUIRED. Let's try to load default system
        # root CA certificates for the given purpose. This may fail silently.
        context.load_default_certs(purpose)

Not checking certificates explains why the vulnerable example client in the beginning just continued connecting and sent credentials to another mail server or to the entity that pretends to be one.

How could that be avoided? Just create an SSL context and pass it to the constructor as an additional parameter (examples for each library):

imaplib.IMAP4_SSL(hostname, port, ssl_context=ssl.create_default_context())
smtplib.SMTP_SSL(hostname, port, context=ssl.create_default_context())
poplib.POP3_SSL(hostname, port, context=ssl.create_default_context())

By the way, did you noticed the different parameter names for the SSL context?

Implications

The direct consequence is that a vulnerable Python program that uses the libraries in a vulnerable way may leak credentials for an IMAP or SMTP account (or POP3) to an active attacker in a MITM position. A script may only use SMTP, but often SMTP and IMAP credentials are identical and therefore may result in IMAP being accessed, too. Furthermore, if an e-mail is sent via SMTP, mail contents may leak or could be modified during transit. Due to the flexibility of IMAP, not only mail contents may be retrieved, but also modified or deleted before another, legitimate user can access it. Since a library is affected here, the exact impact depends on the use case of the software that uses the library and it will depend on the type of data being processed via e-mail. Another impact is that with access to e-mail infrastructure, the reachable attack surface increases for the attacker. Access to e-mail infrastructure may be abused for phishing attacks and spamming, for example.

If there is the need to create an SSLContext to enable certificate verification, will everyone have read the documentation? Sadly, there are even tutorials not mentioning the SSL context and ChatGPT won’t tell the full story until one asks... erm... demonstrates prompt engineering skills.

We can use code search engines such as grep.app, search code, and Debian Code Search, looking for constructor calls to imaplib.IMAP4_SSL and smtplib.SMTP_SSL and skim through the source code. Skipping testing code, inactive and some personal code repositories, it is possible to find quite some vulnerable uses. A curated list is given at the end of this article.

We found vulnerabilities in code that is covered by bug bounty programs. For example, we found a vulnerability in an OAuth2 Python script by Google, where a connection to an SMTP/IMAP server is made to test if authentication tokens are valid. We reported this to the Google and Alphabet Vulnerability Reward Program (VRP). They rewarded us with a small bounty. [1] We also filed vulnerability reports at Hackerone for Spotify and Mozilla, but they [2] don't see a problem with not checking server certificates and closed the tickets. The vulnerability in Spotify's Luigi project has been found before, but was likely not reported back then. We were able to report the vulnerability in Mozilla's Bugbot via another channel and it was fixed.

We tried to contact most of the projects, where it was somehow possible to find (security) contact data. Some of them were very happy to receive the report and immediately provided a fix.

Necessary conclusion to draw

It is understandable to prefer not to break existing functionality. We all know ecosystems in software engineering, where things break every now and then. But keeping insecure defaults is not a value that makes sense to get preserved. It is a vulnerability. If a security issue is only documented in the fine print, developers will miss it and will write vulnerable code. RTFM is not an excuse. You don't build unavoidable trip traps and urge caution.

Over a year ago, a ticket was opened in the Cpython project to change the default behaviour of these e-mail libraries, in other words to be secure by default. We are really looking forward to the ticket being implemented and closed.

A few affected tools and frameworks

Infrastructure:

Frameworks:

Frappe: Low code web framework for real world applications, in Python and Javascript,, at several places.
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing,.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows, affected components are smtp.py, email.py and imap.py. – Code is fixed and released with 2.7.0. CVE-2023-39441 was assigned to this.
zvt: modular quant framework.
Python library imap_tools: Work with email by IMAP. – The library itself provides an option to pass a SSL context to MailBoxTls(), but if this parameter is not used, it is also insecure by default. This made Paperless-NGX vulnerable.
Flask-Mail: Flask-Mail adds SMTP mail sending to your Flask applications. – We made a pull request to fix the problem, but it was not accepted yet.
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in..

Security:

LinOTP: The open source solution for two factor authentication,.
King phisher: Phishing Campaign Toolkit,.
SquarePhish is an advanced phishing tool that uses a technique combining the OAuth Device code authentication flow and QR codes..
AIL Project is an open source project to collect and analyse data to produce security intelligence, – It is fixed now.
Impacket: – This could also be a feature to be insecure.

Community:

The source code that powers readthedocs.org, – According to the project, this is dead code. It was removed.
A Mozilla release management tool to send reminders to Firefox developers and improve Bugzilla metadata, – It is fixed now.

End-user tools:

OAuth2 clients:

Google OAuth2 python client,, for example used in – Pentagrid filed a bug report for this in the Google and Alphabet Vulnerability Reward Program (VRP) as described above. It is fixed now.
A contributed script to the Mutt mail client,.
python-oauth2: A fully tested, abstract interface to creating OAuth clients and servers. – Listed here, but it is an unmaintained project.

Teaching Burp a new HTTP Transport Encoding

Pentagrid AG — Wed, 13 Apr 2022 10:42:00 GMT

Today we would like to talk about Burp Suite Professional and extensions again. In this blog post we explain how we can teach Burp Suite to handle a custom Transport Encoding that is spoken between an HTTP client and a server by using Burp extensions.

Introduction

Burp has many nice features, but sometimes it lacks a feature we would like to have or our target application is customized so that it is necessary to teach Burp new things. More specifically:

Burp has no support for the standard HTTP brotli Transport-Encoding, but all modern browsers do. This problem is handled by Burp by removing brotli from the Accept-Encoding request header (see Proxy - Options - Miscellaneous - Remove unsupported encodings from Accept-Encoding headers in incoming requests), which works for modern browsers. However, we encountered situations during tests where a non-browser HTTP client software required brotli and refused to talk to the server when brotli encoding is missing.
Java fat clients sometimes zlib-encode the entire HTTP body when talking to a server and vice-versa. They also often refuse to speak non-zlib and do not signal it at all in an HTTP header, therefore it is necessary to implement zlib-decoding and encoding to be able to use Burp properly.
If a website uses JavaScript to encrypt all HTTP bodies when talking to a server and vice-versa, we also would like to implement a decryption and encryption routing to use the Burp tools. We also encountered the same situation with other HTTP clients that used additional transport encryption, such as mobile banking apps.

The biggest issue for us was to figure out how we approach these situations consistently and we show our approach in this blog post.

In this blog post we often refer to "encoding" or "encrypting", which obviously is not the same, but both operations have to be done at the same place inside Burp. Encryption or encoding is merely an implementation detail in this blog post. Also "cleartext" is used to indicate decoded or decrypted content.

Why are you doing all of this?

If Burp is not seeing decoded messages for one of the reasons above, it won't work properly. Tools such as Burp Scanner, Repeater, Intruder and other extensions will often fail to interpret the encoded HTTP content and will for sure fail with encrypted content. It won't matter if you use the scanner or not, in many cases it won't do anything useful if you don't implement this custom decoder/encoder. On the contrary, very often a tester's task is to make their tools work. This might be rewarded in the end by simply pressing the active scan button in Burp and then finding security issues. To us, this is a huge difference between a good and a bad security analysis.

Usually we would suggest to solve this problem with features or extensions that are already present, for example the Hackvertor extension would be a good candidate. However, Hackvertor won't modify responses, so this isn't an option here.

Burp's API for extensions

Burp provides several APIs an extension can use. The API code didn't get much love in the last few years, but this is hopefully changing when Burp releases the New API and multi-language extensibility. Let's hope for the best, but for now we're stuck with the API we have. There are two main API hooks that matter in the situation of a missing Transport Encoding. We'll use the name of the function name in the rest of this blog post, but the interfaces that provide the functionality are called:

IProxyListener, which means you have to implement the function processProxyMessage.
IHttpListener, which means you have to implement the function processHttpMessage.

While both interfaces sound similar, the proxy listener will only see requests and responses going through the proxy (e.g. from the browser), whereas the HTTP listener will see the requests and responses of all tools (Proxy, Crawler, Scanner, Repeater, Intruder, other extensions, etc.). However, there's one little detail that is important to mention: If you want to change messages going throught the proxy, you have to do that in the processProxyMessage function and you can't do that in the processHttpMessage. It's just a Burp API limitation that you can't modify messages from TOOL_PROXY in the processHttpMessage. That means we have to get around this limitation by choosing the correct API.

While we think it is better to use Kotlin to write large Burp extensions nowadays, rapid prototyping is still quicker with Python. Therefore we use Python (or rather, Jython 2.7 in Burp) in this blog post.

Here's a little Python sample extension that handles the API confusion of processProxyMessage and processHttpMessage, so we don't need to care about it anymore and can just modify messages in the newly created processMessage function:

from burp import IBurpExtender
from burp import IHttpListener
from burp import IProxyListener
from burp import IBurpExtenderCallbacks

NAME = "Pentagrid Extension Template"

class BurpExtender(IBurpExtender, IProxyListener, IHttpListener):

    def registerExtenderCallbacks(self, callbacks):
        # keep a reference to our callbacks object
        self._callbacks = callbacks

        # set our extension name
        callbacks.setExtensionName(NAME)

        # register ourselves as an Proxy listener
        callbacks.registerProxyListener(self)

        # register ourselves as an HTTP listener
        callbacks.registerHttpListener(self)

        print("Loaded "+NAME+" successfully!")

    #
    # implement IHttpListener
    #

    def processHttpMessage(self, toolFlag, messageIsRequest, messageInfo):
        if toolFlag == IBurpExtenderCallbacks.TOOL_PROXY:
            # DONT'T DO ANYTHING HERE
            # Instead, use processProxyMessage below
            # https://github.com/nccgroup/BurpSuiteLoggerPlusPlus/issues/42
            return
        self.processMessage(toolFlag, messageIsRequest, messageInfo)

    #
    # implement IProxyListener
    #
    def processProxyMessage(self, messageIsRequest, message):
        self.processMessage(IBurpExtenderCallbacks.TOOL_PROXY, messageIsRequest, message.getMessageInfo())


    def processMessage(self, toolFlag, messageIsRequest, messageInfo):
        # TODO: Implement whatever you would like to do
        pass

As a bonus, the Burp UI will show the "Edited request" or "Edited response" option in the Proxy tool whenever we change a proxy message. However, for processHttpMessage we are partially blind, the request and responses will simply be changed, but depending on where we measure we might see them in a different state in the UI (discussed in the next section).

Burp's hooking locations

We would like to hook as early as possible and as late as possible. We would like to make sure that we decode or decrypt early, so that all other Burp tools can work with readable non-compressed HTTP requests and responses. We also want to encode or encrypt as late as possible, just before the request leaves Burp for the same reason. The problem here is that processProxyMessage is called before processHttpMessage for requests, but for responses processHttpMessage is called before processProxyMessage. When analysing the exact behavior, we saw it's a non-trivial setup. For example, the built-in Burp Logger is a tab in the Burp UI showing all requests/responses that flow through Burp. But the question is, where does the Burp Logger hook the API itself? Moreover, users who have advanced knowledge of Burp also know that the extension order in the loaded extension list matters. So where should we put our decoder or decrypter and where should we put our encoder or encrypter? This is not easy to explain and it took us a little trial and error, but we figured out that this diagram should be accurate:

Diagram showing how requests and responses are processed inside Burp when we load our Burp extensions. Green is for cleartext (decoded or decrypted) traffic, red for encoded or encrypted traffic.

This diagram shows several interesting details:

We achieve what we want: Target, Intruder, Repeater, Scanner and other extensions can deal with "cleartext" requests. However, we can't make it work with Proxy match/replace rules for requests (they are applied to the encoded content).
Burp's built-in Logger is not last in the chain for responses, it's possible that Burp extensions modify the responses. This means you don't necessarily see what was delivered to the client/server in Logger. You should better use Logger++ and put it last in the extension list to see which requests are sent to the server and which responses are returned after processing in Burp. Of course it depends on what you want to see, but that's what is the most logical setup.
Our extension we would like to write for our purpose has to be first and last (well, second-last before Logger++) to make sure it does its job well. This means we need to write two extensions, an early-decoder that is first in the list for requests (see DECRYPTER top left in the diagram) and responses (see DECRYPTER bottom right) and a late-reencoder that is last in the list for requests (see ENCRYPTER bottom left) and responses (see ENCRYPTER top right). So the order of the extensions has to be like in the following image:

Extension order is important to make sure all extensions see the correct traffic. Early decoder as the first extension, late encoder as the second-last.

Now that we know we need two extensions, how would the Python extensions look like? Here's the early decoder:

from burp import IBurpExtender
from burp import IHttpListener
from burp import IProxyListener
from burp import IBurpExtenderCallbacks

NAME="Pentagrid early decoder"

class BurpExtender(IBurpExtender, IHttpListener, IProxyListener):

    def registerExtenderCallbacks(self, callbacks):

        # keep a reference to our callbacks object
        self._callbacks = callbacks

        # obtain an extension helpers object
        self._helpers = callbacks.getHelpers()

        # set our extension name
        callbacks.setExtensionName(NAME)

        # register ourselves as an HTTP listener
        callbacks.registerHttpListener(self)
        callbacks.registerProxyListener(self)

        print("Loaded "+NAME+" successfully!")

    #
    # implement IHttpListener
    #

    def processHttpMessage(self, toolFlag, messageIsRequest, messageInfo):
        if toolFlag == IBurpExtenderCallbacks.TOOL_PROXY and messageIsRequest:
            # Already processed in processProxyMessage
            return
        self.filter_message(toolFlag, messageIsRequest, messageInfo)

    #
    # implement IProxyListener
    #
    def processProxyMessage(self, messageIsRequest, message):
        # Responses are handled as early as possible in processHttpMessage
        if messageIsRequest:
            self.filter_message(IBurpExtenderCallbacks.TOOL_PROXY, messageIsRequest, message.getMessageInfo())


    def filter_message(self, toolFlag, messageIsRequest, messageInfo):
        # TODO: implement decoding
        pass

And here's the late encoder:

from burp import IBurpExtender
from burp import IHttpListener
from burp import IProxyListener
from burp import IBurpExtenderCallbacks

NAME="Pentagrid late encoder"

class BurpExtender(IBurpExtender, IHttpListener, IProxyListener):

    def registerExtenderCallbacks(self, callbacks):

        # keep a reference to our callbacks object
        self._callbacks = callbacks

        # obtain an extension helpers object
        self._helpers = callbacks.getHelpers()

        # set our extension name
        callbacks.setExtensionName(NAME)

        # register ourselves as an HTTP listener
        callbacks.registerHttpListener(self)
        callbacks.registerProxyListener(self)

        print("Loaded "+NAME+" successfully!")

    #
    # implement IHttpListener
    #

    def processHttpMessage(self, toolFlag, messageIsRequest, messageInfo):
        if toolFlag == IBurpExtenderCallbacks.TOOL_PROXY and not messageIsRequest:
            # Requests are handled as late as possible in processProxyMessage
            return
        self.filter_message(toolFlag, messageIsRequest, messageInfo)

    #
    # implement IProxyListener
    #
    def processProxyMessage(self, messageIsRequest, message):
        # Responses are handled as late as possible in processHttpMessage
        if not messageIsRequest:
            self.filter_message(IBurpExtenderCallbacks.TOOL_PROXY, messageIsRequest, message.getMessageInfo())


    def filter_message(self, toolFlag, messageIsRequest, messageInfo):
        # TODO: implement encoding
        pass

Bonus

To make this a little more convenient for you, we've set up a Github repository with all the examples from above (in the "minimal" subfolder). However, that's not all:

A fully working example for the zlib-body encoding/decoding.
A template that has some more helpful functions if you want to implement your own decoding/encoding.
An additional extension that implements an IMessageEditorTabFactory, so if you encounter an encoded message in Burp you can click the message viewer option of the extension and will see it in decoded form.

More advanced examples

With the above examples it should be no problem for you to also create extensions that do decryption/encryption instead of decoding/encoding in more complicated setups. In one of our security analysis, we were able to implement the decryption/encryption and hardcoded the key at first. However, then we realised we could additionally inject a JavaScript tag into the HTML single-page response of the server to "steal" our own encryption key from our browser. This removed the requirement to hard-code the encryption key. We did this by replacing </body> with:

<script>let lastKey = null; function pentagrid () {let nowKey = window.encryptionKey; if(nowKey && nowKey !== lastKey){lastKey = nowKey; let pentagrid2 = new XMLHttpRequest(); pentagrid2.open("GET", "/pentagridDiscloseKey=" + nowKey, false); pentagrid2.send(null);}}; setInterval(pentagrid, 500);</script></body>

This JavaScript extracted the encryption key from the browser DOM and sent it through Burp, so we could catch it in our decrypter/encrypter extensions and therefore fully intercept cleartext traffic.

Happy hacking!