What SMTP Authentication Is and Why You Can’t Ignore It

Zakhar Yung

5 years ago

How would you feel if anyone could send emails from your mail server? Greenlight for spammers – no way! That’s why Google and other email service providers protect their servers from unauthorized use. Senders need to authenticate and prove that they have a valid account. If they don’t, the server will reject their request. All of this is possible with SMTP authentication, also known as SMTP AUTH.

Why you shouldn’t use SMTP servers without authentication

Let’s say your company supplies an email address for your employees. However, there is no need for authentication to connect to the email server. So, they don’t have to enter a username and password to send an email. What hazards might this involve?

Anonymity is the first thing to consider. Everybody using the mail server remains anonymous since he or she doesn’t have to log in with any credentials. This isn’t that big of a deal. But do you really need this sort of anonymity?

Email spoofing is what you should definitely worry about. No SMTP authentication means that it is possible to spoof the sender. In the best case, someone will use your mail server to send unauthorized sales emails. At worst, the spoofer can request personal information from the recipient and use it for identity theft purposes (phishing).

So, how do you protect the server from spoofing? Earlier, we blogged about email authentication via SPF, DKIM, and DMARC records. Today, let’s talk about authentication mechanisms for the server.

What is SMTP AUTH and how does it work?

SMTP authentication or simply SMTP AUTH is the service extension of the ESMTP. It requires that an email sender (client) must have permission to use the email server. So, only authorized users can send outgoing messages.

Authentication is carried out according to the SASL mechanism. As a rule, servers use the three most common mechanisms: PLAIN, LOGIN, and CRAM-MD5. There are also other options that we’ll talk about later. Now, let’s take a look at authentication in action.

After the SMTP handshake, the client sends the EHLO command to the server. The response is code 250. It is usually attached with additional information, including supported SASL mechanisms. Here is how the SMTP session looks:

S: 220 smtp.mailtrap.io SMTP ready
C: EHLO client.railsware.com
S: 250-smtp.mailtrap.io Hello client.railsware.com
S: 250 AUTH LOGIN PLAIN CRAM-MD5

As an example, we’ll use Mailtrap Email Sandbox as it responds positively to most clients’ commands, and we won’t have to worry about any configuration issues. Additionally, our Email Sandbox prevents test emails from reaching recipients and spamming them.

Let’s try to authenticate via the LOGIN SASL mechanism. For this, the client sends the AUTH LOGIN command. The server responds with a 334 code and requests a username. Once the client provides it, the server follows up with the request for the password. Eventually, the server replies with 235 - Authentication successful.

S: 250 AUTH LOGIN PLAIN CRAM-MD5
C: AUTH LOGIN
S: 334 DYT3jf4sdDR5
C: TXktVXNlcm5hbWU=
S: 334 LIRdf2pekwW3
C: TXktUGFzc3dvcmQ=
S: 235 Authentication successful

DYT3jf4sdDR5 and LIRdf2pekwW3 by the server are the BASE64 encoded texts for “Username” and “Password”, respectively. TXktVXNlcm5hbWU= and TXktUGFzc3dvcmQ= are the client responses that are also BASE64 encoded.

ESMTP replaced POP-before-SMTP

These days, all mail servers use ESMTP authentication via SASL mechanisms. This type of SMTP AUTH replaced the deprecated POP-before-SMTP authentication. The latter, however, still can be found on some old servers. The idea is to authenticate the user at the POP3 service of the same server and then connect them back to the SMTP. If you need to know how POP3 differs from SMTP, check out our dedicated blog post IMAP vs. POP3 vs. SMTP.

This method of authentication is not very secure. Different clients can share the same IP address and authenticate as one user. In addition, the server runs both POP3 and SMTP services on the same system. ESMTP authentication gets an edge since it implements SASL mechanisms.

SASL authentication mechanisms

SMTP authentication mechanisms allow the server to check whether the SMTP client is authorized. They differ in the level of security. As a rule, the server lists which SASL mechanisms it supports as a response to the EHLO command. Most servers support the following options:

PLAIN – the server requests the client to authorize using the username and password. These credentials are transmitted as one string encoded in Base64.

S: 250 AUTH LOGIN PLAIN CRAM-MD5
C: AUTH PLAIN
S: 334
C: vHRjyADROPsdSDIROu=
S: 235 Authentication successful

Alternatively, the client can submit the credentials together with the AUTH PLAIN command in one single line:

S: 250 AUTH LOGIN PLAIN CRAM-MD5
C: AUTH PLAIN vHRjyADROPsdSDIROu=
S: 235 Authentication successful

LOGIN – the server requests the client to authorize using the username and password. The difference from PLAIN is that the credentials encoded in Base64 are transmitted one by one – username and password.

S: 250 AUTH LOGIN PLAIN CRAM-MD5
C: AUTH LOGIN
S: 334 DYT3jf4sdDR5
C: TXktVXNlcm5hbWU=
S: 334 LIRdf2pekwW3
C: TXktUGFzc3dvcmQ=
S: 235 Authentication successful

CRAM-MD5 – the server requests the client to solve a computational challenge using the password. The client response is a string containing a username and a 16-byte digest in hexadecimal notation. The string is BASE64 encoded.

S: 250 AUTH LOGIN PLAIN CRAM-MD5
C: AUTH CRAM-MD5
S: 334 HSU43fnkj47dskmlSH6dsnjn8ndskjnkjnScdDES=
C: adIBUhb7nsddnKsddsfUN8HkjJNJNsdfJIb4LJNKJ=
S: 235 Authentication successful

CRAM-MD5 provides a higher level of security compared to the plaintext authentication mechanisms, PLAIN and LOGIN. The protocol uses a challenge-response principle. The server sends a challenge string. The client decodes the server challenge and replies with an HMAC (Hash-based Message Authentication Code) calculation using the password as a secret key. After that, the hashed challenge is converted to a string of lowercase hex digits. Also, the client prepends the username and a space character to the string. The server gets this concatenation BASE64-encoded.

Spoofers cannot:

decode credentials since they aren’t actually transferred by the client in text or in code
duplicate the hash because they need to know the password
simulate the hash, because the computational challenge is mutable and changes with each login

Now, let’s discover other SASL mechanisms that can be used on SMTP servers:

XOAUTH/XOAUTH2 – is a basic authentication mechanism in mail servers of Gmail, Live.com, and Outlook.com. It is based on OAuth signatures to authenticate users. XOAUTH2 allows the client to send OAuth 2.0 access tokens to the server. The client response is one string encoded in Base64. The AUTH command consists of a single line of text.

S: 250-AUTH LOGIN PLAIN XOAUTH XOAUTH2
C: AUTH XOAUTH2 dXNlcj1zb21ldXNlckBleGFtcGxlLmNvbQFhdXRoPUJlY
XJlciB5YTI5LnZGOWRmdDRxbVNSbGNrQJoZG1semRHRXVZMjl0Q2cBAQ==
S: 235 Accepted

NTLM is a challenge-response authentication mechanism. It is often used in the Integrated Windows Authentication mode and some Microsoft products like MS Exchange. Once the client sends the AUTH command, the server permits NTLM authentication. The client sends a NEGOTIATE_MESSAGE. The server replies with a CHALLENGE_MESSAGE to establish the client’s identity. The client sends an AUTHENTICATE_MESSAGE as a response to the challenge. All the messages are encoded in Base64.

S: 250-AUTH GSSAPI NTLM
C: AUTH NTLM
S: 334 ntlm supported
C: TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAFAs4OAAAADw==
S:334 lRMTVNTUAACAAAAFgAWADgAAAA1goriZt7rI6Uq/ccAAAAAAAAAAGwAbABOAAAABQLODgAAAA9FAFgAQwBIAC0AQwBMAEkALQA2ADYAAgAWAEUAWABDAEgALQBDAEwASQAtADYANgABABYARQBYAEMASAAtAEMATABJAC0ANgA2AAQAFgBlAHgAYwBoAC0AYwBsAGkALQA2ADYAAwAWAGUAeABjAGgALQBjAGwAaQAtADYANgAAAAAA
C:TlRMTVNTUAADAAAAGAAYAHwAAAAYABgAlAAAABYAFgBIAAAACAAIAF4AAAAWABYAZgAAABAAEACsAAAANYKI4gUCzg4AAAAPZQB4AGMAaAAtAGMAbABpAC0ANgA2AHQAZQBzAHQARQBYAEMASAAtAEMATABJAC0ANgA2AAZKkK42dvN2AAAAAAAAAAAAAAAAAAAAABvqCZdJZ0NxuuMaNT5PPn5aZ6imuk9cPZkPUjEYNIRezkCGmTwS5G0=
S: 235 Authentication successful

GSSAPI refers to the Generic Security Service Application Program Interface. By itself, GSSAPI is almost exclusively used with Kerberos, a network authentication protocol. Similar to NTLM, this authentication mechanism is often used in Microsoft’s Windows Servers. The GSSAPI or Kerberos authentication looks as follows:
- The client and server negotiate a shared secret key, cipher, and hash for the session.
- The server provides a host key
- The client authenticates

Here is an example:

S: 250-AUTH GSSAPI PLAIN
C: AUTH GSSAPI
S: 334
C: YIICEQYJKoZIhvcSAQICAQBuggIAMIIB/KADAgEFoQMCAQ6iBwMFACAAAACjggETYYIBDzCCAQugAwIBBaENGwtOQURBLktUSC5TRaIjMCGgAwIBAaEaMBgbBHNtdHAbEHNtdHAubmFkYS5rdGguc2Wjgc8wgcygAwIBEKEDAgEJooG/BIG8msq2xygko4Lv0Agu5pW6SEundUbFK5swuopukvx9kTidWULb/Ab490wQbtnKx3lmM3BFvNFvuUyD3zvh9PHggwz7T7eZYSCDaovIL/QZ0ismF3lZejZBSwBhgLDADQuk4nZHbbeoU9Lk+1jzsMJguNh6Ot3G6o8WLqFZoe8pi3NuxzSdjutjg3O9s/fasuSB9T85bq6oIMWGr5HHRNBNUF4x11tK3ytpsVoMNpKng3d4bY8tLgnxxLCmREakgc8wgcygAwIBEKEDAgEBooG/BIG8SPCDQwKGzJfZGg+MgqQquBiGBXA2uy/08gPE19vuTBP7XyL2H4EaVqtl71MeVxExbat/CNAK
3dMXkNqR6VHxZqb+ky8MYMDo452Z1sN6BfIsKcsy2BcYTwFJMtgdn21vTWVHtMPH3wtXPuPFGn3jigjsXiAyytXi1Y4p4Tni+ox5ndlZuqBJGeThVxyZIpCEI+5rWflxDIYVa/8CAcRUPQqoDpQIs5zkwfoPQtTdfRLdph5VxQ79N9PnvnQ=
S: 334 YGwGCSqGSIb3EgECAgIAb10wW6ADAgEFoQMCAQ+iTzBNoAMCARCiRgRE2FBXYUbT0MVIicgLYE/FKy6CcrvfQxZaoxyt05qqxJBL13kqneza/TKe5i0mjsN0Nc90KW/l4rL0eQ76vWMenaE1Lw8=
S: 334
YD8GCSqGSIb3EgECAgIBBAD/////IGqNk7Rz3+kPdzT9oYPRWnQi/ESL0p3EeQ2yNLWArrmdOzxpBwAgAAQEBAQ=
C: Using system username `jas' as authentication identity.
YD8GCSqGSIb3EgECAgIBBAD/////JhNtx+GhzYe54NY92BltbUHD6i02upmatfXUnIGrBR5vT5yuAQAgAGphcwE=
235 OK Authenticated

The SSL connection is required for plaintext mechanisms. Unencrypted credentials can be sent with no issues. All non-plaintext SASL mechanisms do not require SSL/TLS encryption.

SMTP AUTH listening ports

587 – This is a default SMTP AUTH port. It is also known as the message submission port. 587 is associated with submission servers or mail submission agents (MSAs) and implies the use of authentication.
25 – In some cases, SMTP AUTH can be used on this port as well. 25 is known as the message relay port.

Code 535 – Authentication failed and other SMTP AUTH errors

The SMTP server can reply both positively and negatively to the AUTH command. Positive response codes are:

334 – the requested security mechanism is accepted
235 – authentication is successful

The most frequent negative response is 535 - Authentication failed. It’s likely that you’ll be able to fix this error if you know what has caused it.

Incorrect credentials – Either the username or password (or both) is incorrect. Usually, the username is the same as your email address. And the password is the same as for your email account.

Account disabled – There could be different reasons for this like spamming issues or payment arrears. The best way to fix this is to contact your mail server provider or administrator.

SSL/TLS connection required – Some servers may respond with 535 when an encrypted connection is required.

SMTP authentication is not enabled – Make sure that both your client and server has the SMTP AUTH enabled.

Authentication mechanism is not supported – The client does not support any of the mechanisms the server uses for authentication. In this case, it’s recommended to enable the basic authentication mechanism on the server and use it.

Besides 535, you may face other errors with the AUTH command:

530 – authentication problem that mostly requires the <a href="https://mailtrap.io/blog/starttls-ssl-tls/">STARTTLS</a> command to run
538 – encryption required for a requested authentication mechanism

Code 550 denotes that your email server requires SMTP authentication. It’s mostly a response to the attempt to send a message. For more on SMTP commands and responses, read our dedicated blog post.

How to test the SMTP authentication

Some time ago we blogged about testing SMTP server with a manual Telnet session. Now, let’s use the Telnet client to test SMTP authentication on your mail server.

Step 1: Make sure the Telnet client is installed/activated on your operating system. This mostly refers to Windows users who need to perform some manipulations to get it enabled.

Step 2: Establish a TCP connection (port 25) with the SMTP server using telnet smtp.yourserver.com 25 The server will reply with 220 smtp.yourserver.com This means the SMTP session has begun.

Step 3: Say hello to the server with EHLO smtp.yourserver.com The server will welcome you back and offer a selection of authentication mechanisms. For example: 250-AUTH PLAIN LOGIN CRAM-MD5

Step 4: Let’s authenticate via LOGIN. But before, make sure to encode your credentials in Base64. You can use this tool for encoding. Enter the AUTH LOGIN command. As you remember, the server will reply with 334 followed by the Base64 encoded text. First is the request for username, and then – password. Reply with your Base64 encoded credentials respectively. If authentication is successful, the server replies with 235 OK. The SMTP AUTH test was passed.

SMTP authentication is what you can use to protect your mail server from spoofing and phishing. At the same time, there are many other threats like Malware, DoS attacks, and so on. Read our blog post to learn how to make SMTP secure and protect against all possible vulnerabilities.