SMTP Smuggling-Fallout – or: Who should patch now? What and how?
This article is also available in German.
tl;dr 1: All those who run a vulnerable server should patch it really soon.
tl;dr 2: Patching alone usually won’t suffice – in most cases configuration options will have to be changed, too.
This winter season had administrators of of mail servers stirring between christmas cake and new year’s toasts, as the „SMTP Smuggling“ security leak made rounds. It was published on 2023-12-18 by Timo Longin, SEC Consult and presented two weeks later at the (in)famous 37th Chaos Communication Congress (37C3) in Hamburg. The talk has been recorded and is available at media.ccc.de.
The security vulnerability, which allows sending out emails under fake names from valid original servers, affects the standard protocol SMTP which is used worldwide to transfer emails between email servers. Email-Spoofing, i.e. sending out emails under forged sender name, is not exactly a new invention. Nowadays countermeasures like SPF and DMARC (in combination) provide a means to check whether a mail server is allowed to send mails from a certain domain and server – or not. But what, if validated mail servers could be abused to send out illicit emails?
For efficiency reasons the SMTP protocol allows transferring multiple emails in one session. And this is exactly where the „SMTP-Smuggling“ vulnerability kicks in. When sending multiple emails in one go, the emails are separated by the END-OF-DATA marker as defined in RFCs (Request for Comments, the standards documents defining computer data protocols) 5321 and 5322. All email servers worldwide must basically follow these standards (at least to a certain extent) to be able to send and receive emails. Some mail server software allows for some leniency so slightly deviating or broken mail clients still can participate in email exchange. And exactly this where SMTP Smuggling attacks.
The aforementioned END-OF-DATA marker is defined as <CR><LF>.<CR><LF> (<CR> is „carriage return“, as the older ones might remember from typewriters for returning to the first column, also written as \r, whereas <LF> is „line feed“, forwarding the paper to the next line, also written as \n).
So this basically is a sole dot in an otherwise empty line.
While Microsoft Windows systems generally define a the marker for the next line as <CR><LF>, Unix-based operation systems like Linux, BSD, MacOS-X etc only use a solo <LF>. On some older or more obscure operating systems other methods are used (e.g. the old Finder-based MacOS (version 7 and older) use(d) a solo <CR> as next-line-marker).
Knowing this it might not come as surprise that for compatibility reasons quite some mail servers accept „a sole dot in an otherwise empty line“ as END-OF-DATA marker regardless wether with standards conforming next-line markers <CR><LF>.<CR><LF>, <CR>.<CR> , or with Unix-style next-line <LF>.<LF> – or other encodings.
So if now a lenient mail server accepts non-conforming END-OF-DATA markers, an attacker can send one single mail that is interpreted as multiple independent mails which are then delivered as defined by the attacker. As a strictly standards-compliant mail server does not recognize broken END-OF-DATA markers as end-of-data, defense mechanisms against faking sender addresses do not work for the embedded second email, and the one mail including the smuggled second is forwarded to the target mail server.
Security-aware server administrators usually have configured their mail servers so that every email is checked whether the delivering user is actually allowed to send the mail with the given sender address. In Postfix mail server configuration this can be set with the option reject_authenticated_sender_login_mismatch, which matches user names to sender addresses in the $smtpd_sender_login_maps database.
Each email contains at least two addresses: one on denoting the sender (MAIL FROM) and the other one the recipient (RCPT TO). When safeguarding the mail server against forgery it can (depending on software and configuration used) check the emails sender against a list of email addresses the user is allowed to use. If the it is not on the delivering user’s list, delivery of the the mail is rejected. The sole exception is that the original server is allowed to send mails for the (original/first) mmail sender domain. The smuggled second/third emails won’t be checked against maybe existing SPF oder DMARC sender restrictions – as the server only is sending one strictly valid mail after all. This is especially relevant for cloud services with many domains with very many mail addresses sharing the same SPF/DMARC configurations
A single mail dialoge with standards-compliant END-OF-DATA marker <CR><LF>.<CR><LF> looks for example like this (the first two lines defining the envelope, the 4th to 6th line the header) – ending with \r\n.\r\n:
mail FROM: user@sender\r\n
rcpt TO: user@receiver\r\n
data\r\n
From: user@sender\r\n
TO: user@receiver\r\n
Subject: Erste E-Mail\r\n
\r\n
Dies ist die erste E-Mail\r\n
\r\n.\r\n
In contrast to this an attacker’s email might look like this, using a nonstandard marker for mail separation <LF>.<LF> (or \n\n respectively):
mail FROM: \r\n
rcpt TO: \r\n
data\r\n
From: user@sender\r\n
TO: user@receiver\r\n
Subject: Erste E-Mail\r\n
\r\n
Dies ist die erste E-Mail\r\n
\n.\n
mail FROM: admin@sender\r\n
rcpt TO: user@receiver\r\n
data\r\n
From: admin@sender\r\n
TO: user@receiver\r\n
Subject: Geschmuggelte E-Mail\r\n
\r\n
Dies ist die geschmuggelte E-Mail\r\n
\r\n.\r\n
The receiving server accepts the mail and then processes it. If the server is vulnerable to this attack, it leniently accepts <LF>.</LF> (resp. \n.\n) as END-OF-DATA marker and splits the prepared mail into two – separated by the perceived (yet strictly speaking: invalid) marker. As it already accepted the email delivery, it now delivers two mails: one from user@sender and the other by admin@sender – without being able to recognize admin@sender as invalid as all checks already have been passed. Even experts will have a hard time recognising the smuggled mail as such, because originating server, mail headers and SPF and DMARC checks all are valid.
This is especially dangerous for big mail providers. If for example an attacker sends a thusly prepared mail from an account at hypothetically vulnerable Google mail service (e.g. Testuser234@google.com) and smuggles another mail into it from support@gmail.com, the recipient cannot tell, whether that email is legit and that the attachment-to-be-clicked really should be clicked: it originates from the correct server, all headers are like an original one, SPF and DMARC all check as valid. Big cloud and mail providers often use the same SPF/DMARC policy for thousands of domains, so an attacker could send out mail from any of those domains and still pass SPF/DMARC checks. Luckily the big mail providers already filter out malformed END-OF-DATA markers and thus prevent this attack. But as this is the first but probably not last one of such attacks, and because to all mail server admins have patched their systems yet, it is too early to dismiss the warning.
Aggreviating this problem is that not all providers or manufacturers acknowledge this as possible problem. For example Cisco’s mail filter appliance did not filter but „corrected“ malformed END-OF-DATA markers – insisting that of not being a problem (What they nowadays recognize as possible attack vector). As mail recipient you never know how the sending server might be configured. So on your receiving end the mailserver must be patched accordingly and filter out noncompliant mails. A mail service provider will probably notify its customers as soon as its servers prevent sending corrupt END-OF-DATA markers and thus smuggling forged mails. Patching and proper filtering of corrupt markers is the only way to be able to prevent sending out mails with contraband.
Currently security patches are available dor most mail server software – though it might take some time for those to trickle down the supply chain. And sometimes simple patching won’t fix the problem due to compatibility considerations for existing configurations. For example the Open Source project Postfix (included and often used in Ubuntu) „only“ makes filter options available, that have to be explicitly enabled if there already is an individual server configuration. So the proper options must be configured, the mail service restarted and mail flow checked.
We would like to add two notes on two topics where we tripped over ambiguous wording:
The researchers mentioned that two mail servers are necessary to enable smtp smuggling – which can be interpreted in multiple ways. As the vulnerability cannot be triggered locally, a receiving mail server and an SMTP mail delivered to it are needed. Thus a vulnerable mail server that is reachable via SMTP should suffice.
The article mentioned DKIM as one security measure that could be broken by this vulnerability. This usually is not true according to our research for mails that have been signed on the sending mail server – when the original mail is broken up on the receiving system the mail body (originally including the smuggled mail) has changed and this the original DKIM body checksum won’t match any more.
