Script: How Emails are transported
Summary
- SMTP is used to SEND emails.
- SMTP is used not just to send it from the creator of the email to their outgoing email server, but also between email servers themselves when they forward messages to the next hop.
- Show example SMTP protocol session for a simple plain-text email
- Bonus: Show manual SMTP session, with fake From: and To: addresses
- History of SMTP and why it makes todays Internet email more complicated than it could be if backwards-compatibility weren't such a big issue (7bit encoding, MIME)
- Show example SMTP protocol session for a rich-text email with binary attachment.
Prerequisites
- Recommended: Course about sending email – how the Domain Name Service (DNS) works, what it is, and its role in email delivery.
- Recommended: Very basic knowledge of Internet protocols.
- Course: About Email – A basic understanding of "outgoing email server" and difference between sending and receiving mail.
- Course: About Encodings and Attachments – Email used to be 7bit and in parts still is. About MIME and 8 to 7bit encoding schemes.
Actions
{1} SMTP is used to SEND emails
S5: 0:00.750 [fade in: “SMTP – Simple Mail Transfer Protocol] In this little course I'm going to introduce you to SMTP.
S10: 0:04.850 [show Thunderbird's "Compose Message" window; animation: click send, followed by a graphic (with arrows) how the message is delivered to the/an outgoing server] The simple mail transfer protocol is used by your email software to deliver an email to your outgoing email server after you pressed the SEND button.
{2} SMTP is used to FORWARD emails between MTAs
S15: 0:14.000[same graphic as above – extend it to show how that server delivers the email to the next hop] SMTP is also used by the email servers themselves to forward emails from one to the other, until the message reaches the final destination with the mailbox of the recipient on it. 0:25.000 The protocol in both cases is exactly the same, meaning the language of the conversation between your email software and an email server is exactly the same as the one between email servers. The two differences are these: While your
S30: 0:40.300 [same graphic as above: highlight step #1] email software already knows where to send emails to, because you configured an outgoing email server, that server does
S35: 0:46.500 [same graphic as above: step #2, animation: show Internet cloud "millions of servers", question "Which server should I forward the email to for domain domain to?" (the one used in our example)] not know where to send the message next, but has to find this out somehow using the Domain Name System, a process which I won't describe here.
{3} Show example SMTP session
S40: 0:58.000 [show part of an SMTP protocol session, fade-in/out] Instead of talking about it I'm just going to show you SMTP. That is possible because like most protocols used on the Internet it is text-based and human-readable!
S45: 1:09.700 [show Thunderbird's email composition window with (ANIMATION) a short email being written] I'm going to write a very simple plain-text email and send it. Here it is. In order to be able to see what is going on under the hood I'm using
S50: 1:19.075 [START CAMTASIA CAPTURE; show network sniffer] a special software that allows me to see the traffic on a network interface. I won't dwell on this subject and simply show you the result of the capture session. I started capturing network traffic just before pressing SEND in the email software, and ended it when my email had been delivered. 1:35.250 [show full capture result] I then told the capture software to show only the traffic 1:38.050 [show only selected traffic] for the conversation between my email software and the outgoing email server, and to concentrate on the 1:43.700 [show TCP stream; END CAMTASIA CAPTURE] CONTENTS of the TCP stream inside the IP packets. If you don't know what that means don't worry, you don't really need to! Your email software also does not know a thing about these things, it too only knows how to "speak SMTP".
S55: 1:58.450 [show resulting SMTP conversation] So here we have the result – this is the entire conversation between our email software and the outgoing mail server. This conversation consists of several steps.
S60: 2:08.150 [highlight SMTP greeting] First the machines say hello to each other. There is no technical reason for this. It was implemented by the original creators of SMTP simply because they felt like doing it – when the Internet only consisted of a few hundred machines at Universities people wanted to be nice to each other, and they wanted their computers to mirror this attitude. Today a small piece of information is indeed embedded in this greeting:
220 smtp.perfora.net (mrus1) Welcome to Nemesis ESMTP server EHLO [192.168.2.102] 250-mrus1.perfora.net pleased to meet you
S65: 2:22.100 [highlight EHLO and write comment "used to be HELO - all keywords have 4 letters so one 'L' is missing"] Originally this was HELO. I'm going to talk about the background of why we now have EHLO later, but let us first look at the rest of the SMTP conversation.
S70: 2:34.650 [highlight section] Right after the greeting the receiving machine tells us a few things about itself. This is part of the extended SMTP, the original SMTP did not have this. Here the mailserver tells us which extensions to the SMTP protocol it supports. This server supports S71: 2:51.650 [highlight STARTTLS] the extension for encrypting the communication, S72: 2:54.350 [highlight AUTH lines] it requires authentication before it will accept any further communication, S73: 2:58.100 [highlight PIPELINING] it supports pipelining, i.e. we could send more than one email in this session, S74: 3:02.500 [highlight SIZE] it accepts messages that are up to 52428800 Bytes large, and S75: 3:08.350 [highlight HELP] it supports the HELP keyword. That last keyword, by the way, is useful for humans but not really for machines, so you can see that SMTP is meant to be used by humans too and not just by software. This is a feature common to most Internet protocols and helps the technical people who maintain services and/or machines on the Internet to find and fix problems more easily by being able to read and even speak the language of the software.
250-STARTTLS 250-AUTH PLAIN LOGIN 250-AUTH=PLAIN LOGIN 250-PIPELINING 250-SIZE 52428800 250 HELP
S80: 3:34.650 [highlight AUTH; show text bubble pointing to the base64'd passphrase: "This is a plain-text password! If you decode it using a BASE64 decoder, available on the Internet and part of most programming language libraries, you get the login data I used: <NUL>fake@hasenstein.com<NUL>hahaha (<NUL> is a Byte with the value "0", which is not represented by any character"] This is the authentication section. It didn't exist in the original SMTP protocol but is part of the extended SMTP. Originally everyone could use any email server to send emails, just like you can drop a postal letter into any mailbox on any street. The reason why today we have authentication is SPAM. 3:54.000 Spammers like to send anonymously using fake addresses. To make their live harder mailservers require that you authenticate yourself. This way if you send spam the owner of the mailserver can identify you and remove your right to send messages through this particular server.
AUTH PLAIN AGZha2VAaGFzZW5zdGVpbi5jb20AaGFoYWhh 235 authentication finished successfully
S85: 4:10.500 [highlight MAIL FROM] Here the sender tells the recieving machine whom the email is from and how large it is. Your email software, when talking to your outgoing mailserver, uses your configured email address here. This is the address error messages will be delivered to. For example, if the delivery of the email fails the sender most likely would like to know about it, so the email server sends an error report to this email address given here.
MAIL FROM:<mh@hasenstein.com> SIZE=414 250 mail from: ok
S90: 4:33.100 [highlight RCPT TO] Next the sender tells the receiving machine which addresses to send the email to. In this case there is just one recipient. Your email software, when talking to your outgoing mailserver, lists all addresses you entered into the To, Cc, or the Bcc fields here.
RCPT TO:<hasenstein@yahoo.com> 250 ok
S95: 4:48.050 [text: "Envelope", highlight MAIL FROM and RCPT TO] These two fields form the so-called envelope of an email. This term is a very important one, because the recipient of your email is never going to see the envelope! The final receiving mailserver, the one that sorts the email into the recipients mailbox instead of delivering it to another mailserver, throws away this information. It acts like a good secretary that takes all postal mail out of the envelope before giving only the contents to you!
S100: 5:13.900 [highlight DATA section, show Yahoo mail view] This command starts the real email. Everything that comes next is what the recipient is going to see when they look at your email.
DATA 354 Enter mail, end with "." on a line by itself Message-ID: <472E0104.1090708@hasenstein.com> Date: Sun, 04 Nov 2007 18:27:32 +0100 From: Michael Hasenstein <mh@hasenstein.com> Organization: Letexa User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Michael Hasenstein <hasenstein@yahoo.com> Subject: Demo Email Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit This is a VERY simple Email.
S105: 5.21.550 [highlight email headers, point out standard- vs. normally invisible headers] As you can see this section contains the From:, To:, Subject: and other headers that the recipient gets to see. It is just like a postal business letter on which this kind of data is repeated too, because among businesses it is expected that the envelope is thrown away and that therefore things like who sent it, whom is it addressed to, when was it sent, are repeated on page one of the actual letter.
S110: 5:44.050 [highlight "."] The end of the message transmission is indicated to the receiving mailserver by a dot preceded and followed by a newline (RETURN). Next the sender says QUIT to end the connection.
. 250 Message 0MKpCa-1IojG304zV-0000CK accepted by mrus1.perfora.net QUIT 221 smtp.perfora.net Bye
S115: 5:54.350 [show full headers as seen in receiving Yahoo mail account and highlight the additional headers] There is one strange thing though: When we look at the full header of this email once it has been received by the recipient, which includes the header lines that your email client normally hides from you because they only contain technical information inserted by the various pieces of software and not by the user who wrote the message, we can see a lot more header lines! Someone or something added more headers on the way. There are some more, but only the interesting ones are highlighted. The Received headers are inserted one at a time by the individual mail servers that transported this email.
From Michael Hasenstein Sun Nov 4 09:27:32 2007 Return-Path: <mh@hasenstein.com> Authentication-Results: mta320.mail.mud.yahoo.com from=; domainkeys=neutral (no sig) Received: from 74.208.4.196 (EHLO mout.perfora.net) (74.208.4.196) by mta320.mail.mud.yahoo.com with SMTP; Mon, 05 Nov 2007 02:23:26 -0800 Received: from [192.168.2.102] (pD9E524C1.dip.t-dialin.net [217.229.36.193]) by mrelay.perfora.net (node=mrus0) with ESMTP (Nemesis) id 0MKp8S-1Ioz751LC2-0005ut; Mon, 05 Nov 2007 05:23:26 -0500 Message-ID: <472E0104.1090708@hasenstein.com> Date: Sun, 04 Nov 2007 18:27:32 +0100 From: Michael Hasenstein <mh@hasenstein.com> Organization: Letexa User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Michael Hasenstein <hasenstein@yahoo.com> Subject: Demo Email Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit
S120: 6:21.500 [show graphic depicting the flow from sender to recipients inbox, hop to hop; ANIMATE: start at sender, build the graph 1/second and highlight appropriate header section we get this info from] From those headers we can see the path that our email took, and at what time it was transmitted to the next mailserver.
{4} Bonus: Manual SMTP session
S400: 6.29.050 [graphic: email software talking to mailserver, 2nd row YOU (human) talking!] So. Now you have seen that computers use a human-readable text protocol for relaying emails from one to another. Let me show you that you don't need an email application in order to send an email to someone, S405: 6:41.000 instead you can talk to a mailserver yourself!
S410: 6:44.800 [show MS Windows command line "cmd"] You can do this from all operating systems that have a telnet command line program. Telnet is a piece of software created a loooong time ago when text-based terminals ruled the world. 6:54.800 I won't explain the details here, let me just tell you we do that...
S420: 6:57.500 [start animation: manual email] ...by telling telnet not just the name of the mailserver, but also the port on that machine. If we didn't specify a port it would use the port on which it would normally connect to the terminal service. We use port 25, which is the standard port that any mailserver software listens on for incoming SMTP connections. 7:16.700 Now I am connected, and all I have to do is what you have previously seen – speak "SMTP".
S430: 7:23.400 [show text: EHLO vs. HELO; then type EHLO] I start by saying hello. Whether or not I use EHLO or the old HELO does not matter. If I say HELO the server is going to assume I don't know about the extensions to SMTP and won't confuse me by offering them. If I say EHLO it is going to assume I know these extensions and is going to tell me which of those it supports, but I don't have to use them. So I'll say EHLO.
S440: 7:54.900 [enter authorization incl. base64 code; show text bubble explaining how I arrived at the code] Next the server tells me about its extensions, as expected. Since this is a publically accessible mailserver on the Internet I'm going to have to authenticate myself, because it does not allow strangers to send emails through it, which is because of the spam-problem that I mentioned.
S450: 8:11.000 [enter MAIL FROM] I've been accepted, so now I can tell the mailserver whom the email is from.
S460: 8:15.500 [enter RCPT TO] I'm going to send the email to myself, so that I can show you how it is going to look like.
S470: 8:20.850 [enter all DATA] After the DATA keyword I enter the real email. As I already said when we analyzed the captured SMTP session, what the final recipient is going to see is only the text entered here. The previous commands only form the envelope, which is destroyed when the mail reaches the final destination.
C:\> telnet smtp.1and1.com 25 220 smtp.perfora.net (mrus1) Welcome to Nemesis ESMTP server EHLO mha 250-mrus1.perfora.net pleased to meet you 250-STARTTLS 250-AUTH PLAIN LOGIN 250-AUTH=PLAIN LOGIN 250-PIPELINING 250-SIZE 52428800 250 HELP AUTH PLAIN AGZha2VAaGFzZW5zdGVpbi5jb20AaGFoYWhh 235 authentication finished successfully MAIL FROM: <president@whitehouse.gov> 250 mail from: <president@whitehouse.gov> ok RCPT TO: <mh@hasenstein.com> 250 <mh@hasenstein.com> ok DATA 354 Enter mail, end with "." on a line by itself From: George Bush <president@whitehouse.gov> To: My Friend Michael <mh@hasenstein.com> Subject: Let's have dinner next week? Hello Mike, How about dinner next week in the whitehouse? Everybody hates me, even my own party! I need somebody to talk to. George . 250 Message 0MKpCa-1Ip9nk2Cg7-0000Bq accepted by mrus1.perfora.net QUIT 221 smtp.perfora.net Bye
S480: 8:38.550 [enter "." and QUIT] I finish the email by entering a dot, and then tell the server that I would like to end this SMTP session. The message has been accepted and has already been delivered, so let us now look S485: 8:49.150 [show picture] at this email and how it looks like in my email client Thunderbird.
S490: 8:53.650 [show the learning points] You should learn a few things from this example. One, how SMTP can be used manually. Other Internet protocols like the most common one, HTTP, are also text based and the same principles apply. Second, Emails can be faked. Everything in an email can be faked. So if you receive an email from your boss who asks you to send the latest design of the new stealth plane to address@china.cn, or the customer database to address@competitor.com, don't do it... of course, in the real world things are more subtle than in this example.
{5} 7bit History
S500: 9:30.850 [] I already mentioned that the Simple Mail Transfer Protocol has been extended, and that when in the SMTP greeting the client says EHLO instead of the original HELO the server sees this as a sign that the client knows about these extensions and may want to use them. 9:46.200 I'm not going to tell you all the very long history of mail transfer protocols, which starts in the 1960s. Let me just very briefly mention that in the 90s, when an exponential growth of networked computers on the Internet started, the original SMTP had to be extended. 10:01.500 As almost always in computing this had to be done so that both the old and the new could coexist. Even today a client or a mailserver that only speaks the old SMTP would work.
S510: 10:12.500 [show the items] The SMTP extensions support
- Auhorization, which ensures that only authorized users can use a server to send email,
- Transport layer encryption, which encrypts the entire SMTP session,
- 8bit transmission, so that one doesn't need to encode email contents as 7bit values
- Pipelining to send more than one message in one session
- and quite a few more features.
10:34.300 You can see the different SMTP greetings of the client that results in the server offering or not offering the extended SMTP.
SMTP (1982)
220 linux-x13b.site ESMTP Postfix HELO machine 250 mail.domain.com QUIT 221 2.0.0 ByeExtended (E)SMTP (1995) with extensions
220 linux-x13b.site ESMTP Postfix EHLO machine 250-mail.domain.com 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN QUIT 221 2.0.0 Bye
{6} Show rich email example session
S600: 10:44.000 [show email composition window] To finish your introduction to SMTP let us repeat the previous exercise of capturing an SMTP session. The first time we sent a very simple email. This time we are going to send this rich-text HTML email. It also includes an image attachment.
S605: 10:59.400 [show email] This is how the email looks like on the recipient side, i.e. in the Yahoo webmail view. [Pause 4s]
S610: 11:10.250 [show capture, scrollable] There's no need to watch the capture process once more, so I'll simply show you the SMTP session I captured. You can look at it and scroll around yourself while I'm talking!
11:20.800 The session starts as usual. Until we get to the SMTP keyword DATA there is no difference to the previous simple example. That is because anything new in this session does not really have anything to do with SMTP. This means this SMTP course was over before this example started, but I'll show you anyway as preparation for another course about Email encodings. Besides, all parts of the Email infrastructure somehow do work together.
11:49.150 What is new is that compared to the simple email we cannot easily read this one, because the few bits of text are embedded in lots of HTML code, the same language used to write webpages. At the bottom part of the capture you can see the encoded image file that is part of this email. I deleted most of it, because all those lines look like the ones I left in and you cannot learn anything new by seeing them all. Email is a 7bit medium, but binary files like Images, Microsoft Word or PDF attachments are 8bit data. That is why the email client encodes that data. This process drastically increases the size of emails with encoded attachments by a third! So if you send a 9MB attachment to someone whos email server only accepts messages up to 10MB in size it won't work, because after encoding your email is going to exceed that limit.
12:50.750 That's all in this course! I hope you learned something about how emails are transported on the Internet.
220 smtp.perfora.net (mrus1) Welcome to Nemesis ESMTP server EHLO [192.168.2.102] 250-mrus1.perfora.net pleased to meet you 250-STARTTLS 250-AUTH PLAIN LOGIN 250-AUTH=PLAIN LOGIN 250-PIPELINING 250-SIZE 52428800 250 HELP AUTH PLAIN AGZha2VAaGFzZW5zdGVpbi5jb20AaGFoYWh 235 authentication finished successfully MAIL FROM:<mh@hasenstein.com> SIZE=5597 250 mail from: <mh@hasenstein.com> ok RCPT TO:<hasenstein@yahoo.com> 250 <hasenstein@yahoo.com> ok DATA 354 Enter mail, end with "." on a line by itself Message-ID: <472E050C.5050706@hasenstein.com> Date: Sun, 04 Nov 2007 18:44:44 +0100 From: Michael Hasenstein <mh@hasenstein.com> Organization: Letexa User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Michael Hasenstein <hasenstein@yahoo.com> Subject: Demo Email (rich) Content-Type: multipart/related; boundary="------------080405040301090904010302" This is a multi-part message in MIME format. --------------080405040301090904010302 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> </head> <body bgcolor=3D"#ffffff" text=3D"#000000"> Hi,<br> <br> This is a rich text <b>email </b>with an attachment (picture)!<br> <br> <br> <div class=3D"moz-signature"> <meta http-equiv=3D"Content-Type" content=3D"text/html; "> <title>Signature</title> -- <div style=3D"margin: 1.2em 1em 10em 0pt; float: left;"><a href=3D"http://hasenstein.com/"><img src=3D"cid:par t1.07000905.02010606@hasenstein.com" alt=3D"M=C3=B6rre" id=3D"Image1" border=3D"0" height=3D"60" width=3D"80"></a></div> <h1 style=3D"font-variant: small-caps; letter-spacing: 0.1em; color: rgb(0, = 0, 0);"><span style=3D"font-size: larger;">L</span>etexa</h1> <h2>Learning and Teaching Exchange</h2> <p style=3D"font-family: Arial,Helvetica,sans-serif;">This <span style=3D"color: rgb(0, 153, 51);">signature</span> demonstrates how to use a rich-text HTML signature with <a href=3D"http://mozilla.com/">Mozil= la Thunderbird</a>.</p> </div> </body> </html> --------------080405040301090904010302 Content-Type: image/gif; name="Letexa-Logo_80x60-BW.gif" Content-Transfer-Encoding: base64 Content-ID: <part1.07000905.02010606@hasenstein.com> Content-Disposition: inline; filename="Letexa-Logo_80x60-BW.gif" R0lGODlhUAA8AOeuAAAAAAEBAQICAgMDAwQEBAUFBQYGBgcHBwgICAkJCQoKCgsLCwwMDA0N DQ4ODg8PDxAQEBERERISEhMTExQUFBUVFRYWFhcXFxgYGBkZGRoaGhsbGxwcHB0dHR4eHh8f ...lots more such lines (hundreds or even thousands for large attachments)... 6KYEEMDndjTUgAY4wAE+ioADHjCBCnhINORlAQ6g0AhmfEMc4iAHcjtcq5F9rEMbbkkGKpIx DWCwoQcpyIAGVqAkMZTBDGeAQx/ogIY05EEQTkoxdGerj1hdlCKC2sc8OpUPcmADs5ZIhmzD IY5xAAgdUxrHOdJhJUerKnMCUdRqW2vqIqO6Ihp+taxnTeta2/rWuM61rhUSEAA7 --------------080405040301090904010302-- . 250 Message 0MKpCa-1IojWg2Z9f-00005o accepted by mrus1.perfora.net QUIT 221 smtp.perfora.net Bye
END: 12:56.500