1 <!doctype HTML public "-//W3O//DTD W3 HTML 3.2//EN">
4 <TITLE>Design notes on fetchmail</TITLE>
5 <link rev=made href=mailto:esr@snark.thyrsus.com>
6 <meta name="description" content="Design notes on fetchmail.">
7 <meta name="keywords" content="fetchmail, POP, POP2, POP3, IMAP, remote mail">
8 <base href="http://info.internet.isi.edu:80/in-notes/rfc/files/">
11 <table width="100%" cellpadding=0><tr>
12 <td width="30%">Back to <a href="//localhost/~esr/index.html">Fetchmail Home Page</a>
13 <td width="30%" align=center>To <a href="//localhost/~esr/sitemap.html">Site Map</a>
14 <td width="30%" align=right>$Date: 1998/11/07 21:16:30 $
17 <H1 ALIGN=CENTER>Design Notes On Fetchmail</H1>
19 These notes are for the benefit of future hackers and maintainers.
20 The following sections are both functional and narrative, read from
25 A direct ancestor of the fetchmail program was originally authored
26 (under the name popclient) by Carl Harris <ceharris@mal.com>. I took
27 over development in June 1996 and subsequently renamed the program
28 `fetchmail' to reflect the addition of IMAP support. In early
29 November 1996 Carl officially ended support for the last popclient
32 Before accepting responsibility for the popclient sources from Carl, I
33 had investigated and used and tinkered with every other UNIX
34 remote-mail forwarder I could find, including fetchpop1.9,
35 PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc,
36 popmail-1.6 and upop. My major goal was to get a header-rewrite
37 feature like fetchmail's working so I wouldn't have reply problems
40 Despite having done a good bit of work on fetchpop1.9, when I found
41 popclient I quickly concluded that it offered the solidest base for
42 future development. I was convinced of this primarily by the presence
43 of multiple-protocol support. The competition didn't do
44 POP2/RPOP/APOP, and I was already having vague thoughts of maybe
45 adding IMAP. (This would advance two other goals: learn IMAP and get
46 comfortable writing TCP/IP client software.)<P>
48 Until popclient 3.05 I was simply following out the implications of
49 Carl's basic design. He already had daemon.c in the distribution,
50 and I wanted daemon mode almost as badly as I wanted the header
51 rewrite feature. The other things I added were bug fixes or
54 After 3.1, when I put in SMTP-forwarding support (more about this
55 below) the nature of the project changed -- it became a
56 carefully-thought-out attempt to render obsolete every other program
57 in its class. The name change quickly followed.<P>
59 <H1>The rewrite option</H1>
61 RFC 1123 stipulates that MTAs ought to canonicalize the addresses of
62 outgoing mail so that From:, To:, Cc:, Bcc: and other address headers
63 contain only fully qualified domain names. Failure to do so can break
64 the reply function on many mailers.<P>
66 This problem only becomes obvious when a reply is generated on a
67 machine different from where the message was delivered. The
68 two machines will have different local username spaces, potentially
69 leading to misrouted mail.<P>
71 Most MTAs (and sendmail in particular) do not canonicalize address headers
72 in this way (violating RFC 1123). Fetchmail therefore has to do it. This
73 is the first feature I added to the ancestral popclient.<P>
75 <H1>Reorganization</H1>
77 The second thing I did reorganize and simplify popclient a lot. Carl
78 Harris's implementation was very sound, but exhibited a kind of
79 unnecessary complexity common to many C programmers. He treated the
80 code as central and the data structures as support for the code. As a
81 result, the code was beautiful but the data structure design ad-hoc
82 and rather ugly (at least to this old LISP hacker).<P>
84 I was able to improve matters significantly by reorganizing most of the
85 program around the `query' data structure and eliminating a bunch of
86 global context. This especially simplified the main sequence in
87 fetchmail.c and was critical in enabling the daemon mode changes.<P>
89 <H1>IMAP support and the method table</H1>
91 The next step was IMAP support. I initially wrote the IMAP code
92 as a generic query driver and a method table. The idea was to have
93 all the protocol-independent setup logic and flow of control in the
94 driver, and the protocol-specific stuff in the method table.<P>
96 Once this worked, I rewrote the POP3 code to use the same organization.
97 The POP2 code kept its own driver for a couple more releases, until
98 I found sources of a POP2 server to test against (the breed seems
99 to be nearly extinct).<P>
101 The purpose of this reorganization, of course, is to trivialize
102 the development of support for future protocols as much as possible.
103 All mail-retrieval protocols have to have pretty similar logical
104 design by the nature of the task. By abstracting out that common
105 logic and its interface to the rest of the program, both the common
106 and protocol-specific parts become easier to understand.<P>
108 Furthermore, many kinds of new features can instantly be supported
109 across all protocols by modifying the one driver module.<P>
111 <H1>Implications of smtp forwarding</H1>
113 The direction of the project changed radically when Harry Hochheiser
114 sent me his scratch code for forwarding fetched mail to the SMTP port.
115 I realized almost immediately that a reliable implementation of this
116 feature would make all the other delivery modes obsolete.<P>
118 Why mess with all the complexity of configuring an MDA or setting up
119 lock-and-append on a mailbox when port 25 is guaranteed to be there on
120 any platform with TCP/IP support in the first place? Especially when
121 this means retrieved mail is guaranteed to look like normal sender-
122 initiated SMTP mail, which is really what we want anyway.<P>
124 Clearly, the right thing to do was (1) hack SMTP forwarding support
125 into the generic driver, (2) make it the default mode, and (3) eventually
126 throw out all the other delivery modes. <P>
128 I hesitated over step 3 for some time, fearing to upset long-time
129 popclient users dependent on the alternate delivery mechanisms. In
130 theory, they could immediately switch to .forward files or their
131 non-sendmail equivalents to get the same effects. In practice the
132 transition might have been messy.<P>
134 But when I did it (see the NEWS note on the great options massacre)
135 the benefits proved huge. The cruftiest parts of the driver code
136 vanished. Configuration got radically simpler -- no more grovelling
137 around for the system MDA and user's mailbox, no more worries about
138 whether the underlying OS supports file locking.<P>
140 Also, the only way to lose mail vanished. If you specified localfolder
141 and the disk got full, your mail got lost. This can't happen with
142 SMTP forwarding because your SMTP listener won't return OK unless
143 the message can be spooled or processed.<P>
145 Also, performance improved (though not so you'd notice it in a single
146 run). Another not insignificant benefit of this change was that the
147 manual page got a lot simpler.<P>
149 Later, I had to bring --mda back in order to allow handling of some
150 obscure situations involving dynamic SLIP. But I found a much simpler
153 The moral? Don't hesitate to throw away superannuated features when
154 you can do it without loss of effectiveness. I tanked a couple I'd
155 added myself and have no regrets at all. As Saint-Exupery said,
156 "Perfection [in design] is achieved not when there is nothing more to
157 add, but rather when there is nothing more to take away." This
158 program isn't perfect, but it's trying.<P>
160 <H1>The most-requested features that I will never add, and why not:</H1>
162 <H2>Password encryption in .fetchmailrc</H2>
164 The reason there's no facility to store passwords encrypted in the
165 .fetchmailrc file is because this doesn't actually add protection.<P>
167 Anyone who's acquired the 0600 permissions needed to read your
168 .fetchmailrc file will be able to run fetchmail as you anyway -- and
169 if it's your password they're after, they'd be able to rip the
170 necessary decoder out of the fetchmail code itself to get it.<P>
172 All .fetchmailrc encryption would do is give a false sense of
173 security to people who don't think very hard.<P>
175 <H2>Truly concurrent queries to multiple hosts</H2>
177 Occasionally I get a request for this on "efficiency" grounds. These
178 people aren't thinking either. True concurrency would do nothing to lessen
179 fetchmail's total IP volume. The best it could possibly do is change the
180 usage profile to shorten the duration of the active part of a poll cycle
181 at the cost of increasing its demand on IP volume per unit time.<P>
183 If one could thread the protocol code so that fetchmail didn't block
184 on waiting for a protocol response, but rather switched to trying to
185 process another host query, one might get an efficiency gain (close to
186 constant loading at the single-host level).<P>
188 Fortunately, I've only seldom seen a server that incurred significant
189 wait time on an individual response. I judge the gain from this not
190 worth the hideous complexity increase it would require in the code.<P>
192 <H2>Multiple concurrent instances of fetchmail</H1>
194 Fetchmail locking is on a per-invoking-user because finer-grained
195 locks would be really hard to implement in a portable way. The
196 problem is that you don't want two fetchmails querying the same site
197 for the same remote user at the same time.<P>
199 To handle this optimally, multiple fetchmails would have to associate
200 a system-wide semaphore with each active pair of a remote user and
201 host canonical address. A fetchmail would have to block until getting
202 this semaphore at the start of a query, and release it at the end of a
205 This would be way too complicated to do just for an "it might be nice"
206 feature. Instead, you can run a single root fetchmail polling for
207 multiple users in either single-drop or multidrop mode.<P>
209 The fundamental problem here is how an instance of fetchmail polling
210 host foo can assert that it's doing so in a way visible to all other
211 fetchmails. System V semaphores would be ideal for this purpose, but
212 they're not portable.<P>
214 I've thought about this a lot and roughed up several designs. All are
215 complicated and fragile, with a bunch of the standard problems (what
216 happens if a fetchmail aborts before clearing its semaphore, and how
217 do we recover reliably?).<P>
219 I'm just not satisfied that there's enough functional gain here to pay
220 for the large increase in complexity that adding these semaphores
223 <H1>Multidrop and alias handling</H1>
225 I decided to add the multidrop support partly because some users were
226 clamoring for it, but mostly because I thought it would shake bugs out
227 of the single-drop code by forcing me to deal with addressing in full
228 generality. And so it proved.<P>
230 There are two important aspects of the features for handling
231 multiple-drop aliases and mailing lists which future hackers should be
232 careful to preserve.<P>
236 The logic path for single-recipient mailboxes doesn't involve header
237 parsing or DNS lookups at all. This is important -- it means the code
238 for the most common case can be much simpler and more robust.<P>
241 The multidrop handing does <EM>not</EM> rely on doing the equivalent of passing
242 the message to sendmail -oem -t. Instead, it explicitly mines members
243 of a specified set of local usernames out of the header.<P>
246 We do <EM>not</EM> attempt delivery to multidrop mailboxes in the presence of DNS
247 errors. Before each multidrop poll we probe DNS to see if we have a
248 nameserver handy. If not, the poll is skipped. If DNS crashes during a
249 poll, the error return from the next nameserver lookup aborts message
250 delivery and ends the poll. The daemon mode will then quietly spin until
251 DNS comes up again, at which point it will resume delivering mail.<P>
254 When I designed this support, I was terrified of doing anything that could
255 conceivably cause a mail loop (you should be too). That's why the code
256 as written can only append <EM>local</EM> names (never @-addresses) to the
259 The code in mxget.c is nasty, no two ways about it. But it's utterly
260 necessary, there are a lot of MX pointers out there. It really ought
261 to be a (documented!) entry point in the bind library.<P>
263 <H1>DNS error handling</H1>
265 Fetchmail's behavior on DNS errors is to suppress forwarding and
266 deletion of the individual message that each occurs in, leaving it
267 queued on the server for retrieval on a subsequent poll. The
268 assumption is that DNS errors are transient, due to temporary server
271 Unfortunately this means that if a DNS error is permanent a message
272 can be perpetually stuck in the server mailbox. We've had a couple
273 bug reports of this kind due to subtle RFC822 parsing errors in the fetchmail
274 code that resulted in impossible things getting passed to the DNS lookup
277 Alternative ways to handle the problem: ignore DNS errors (treating
278 them as a non-match on the mailserver domain), or forward messages
279 with errors to fetchmail's invoking user in addition to any other
280 recipients. These would fit an assumption that DNS lookup errors are
281 likely to be permanent problems associated with an address.<P>
283 <H1>IPv6 and IPSEC</H1>
285 The IPv6 support patches are really more protocol-family independence
286 patches. Because of this, in most places, "ports" (numbers) have been
287 replaced with "services" (strings, that may be digits). This allows us
288 to run with certain protocols that use strings as "service names"
289 where we in the IP world think of port numbers. Someday we'll plumb
290 strings all over and then, if inet6 is not enabled, do a
291 getservbyname() down in SocketOpen. The IPv6 support patches use
292 getaddrinfo(), which is a POSIX p1003.1g mandated function. So, in the
293 not too distant future, we'll zap the ifdefs and just let autoconf
294 check for getaddrinfo. IPv6 support comes pretty much automatically
295 once you have protocol family independence.<P>
297 Craig Metz used his inner_connect() function to handle most of the
298 connect work. This is a nonstandard function not likely to ever exist
299 in a system's libc, but we can just include that source file if the
300 day comes when we want to support IPv6 without the inet6-apps
301 library. It just makes life easier.<P>
303 <H1>Checklist for Adding Options</H1>
305 Adding a control option is not complicated in principle, but there are
306 a lot of fiddly details in the process. You'll need to do the
307 following minimum steps.
310 <LI>Add a field to represent the control in <code>struct query</code> or
311 <code>struct hostdata</code>.
313 <LI>Pick a token to declare the option in the .fetchmailrc file. Add
314 the token to <code>rcfile_l</code>.
316 <LI>Go to <code>rcfile_y.y</code>. Add the token to the grammar. Don't
317 forget the <code>%token</code> declaration.
319 <LI>Pick a long-form option name, and a one-letter short option if any
320 are left. Go to <code>options.c</code>. Pick a new <code>LA_</code>
321 value. Hack the <code>longoptions</code> table to set up the
322 association. Hack the big switch statement to set the option.
323 Hack the `?' message to describe it.
325 <LI>If the default is nonzero, set it in <code>def_opts</code> near the top of
326 <code>load_params</code> in <code>fetchmail.c</code>.
328 <LI>Add code to dump the option value in <code>fetchmail.c:dump_params</code>.
330 <LI>Add proper <code>FLAG_MERGE</code> actions in fetchmail.c's
333 <LI>Document the option in fetchmail.man. This will require at least
334 two changes; one to the collected table of options, and one full
335 text description of the option.
337 <LI>Add the new token and a brief description to the header comment of
340 <LI>Hack fetchmailconf to configure it. Bump the fetchmailconf version.
342 <LI>Hack conf.c to dump the option so we won't have a version-skew problem.
344 <LI>Add an entry to NEWS.
347 There may be other things you have to do in the way of logic, of course.<P>
349 <H1>Lessons learned</H1>
351 <H3>1. Server-side state is essential</H3>
353 The person(s) responsible for removing LAST from POP3 deserve to suffer.
354 Without it, a client has no way to know which messages in a box have been
355 read by other means, such as an MUA running on the server.<P>
357 The POP3 UID feature described in RFC1725 to replace LAST is
358 insufficient. The only problem it solves is tracking which messages
359 have been read <EM>by this client</EM> -- and even that requires
360 tricky, fragile implementation.<P>
362 The underlying lesson is that maintaining accessible server-side
363 `seen' state bits associated with Status headers is indispensible in a
364 Unix/RFC822 mail server protocol. IMAP gets this right.<P>
366 <H3>2. Readable text protocol transactions are a Good Thing</H3>
368 A nice thing about the general class of text-based protocols that SMTP,
369 POP2, POP3, and IMAP belongs to is that client/server transactions are
370 easy to watch and transaction code correspondingly easy to debug. Given
371 a decent layer of socket utility functions (which Carl provided) it's
372 easy to write protocol engines and not hard to show that they're working
375 This is an advantage not to be despised! Because of it, this project has
376 been interesting and fun -- no serious or persistent bugs, no long
377 hours spent looking for subtle pathologies.<P>
379 <H3>3. IMAP is a Good Thing.</H3>
381 If there were a standard IMAP equivalent of the POP3 APOP validation,
382 POP3 would be completely obsolete.<P>
384 <H3>4. SMTP is the Right Thing</H3>
386 In retrospect it seems clear that this program (and others like it)
387 should have been designed to forward via SMTP from the beginning.
388 This lesson may be applicable to other Unix programs that now call the
389 local MDA/MTA as a program.<P>
391 <H3>5. Syntactic noise can be your friend</H3>
393 The optional `noise' keywords in the rc file syntax started out as
394 a late-night experiment. The English-like syntax they allow is
395 considerably more readable than the traditional terse keyword-value
396 pairs you get when you strip them all out. I think there may be a
397 wider lesson here.<P>
399 <H1>Motivation and validation</H1>
401 It is truly written: the best hacks start out as personal solutions to
402 the author's everyday problems, and spread because the problem turns
403 out to be typical for a large class of users. So it was with Carl Harris
404 and the ancestral popclient, and so with me and fetchmail.<P>
406 It's gratifying that fetchmail has become so popular. Until just before
407 1.9 I was designing strictly to my own taste. The multi-drop mailbox
408 support and the new --limit option were the first features to go in that
409 I didn't need myself.<P>
411 By 1.9, four months after I started hacking on popclient and a month
412 after the first fetchmail release, there were literally a hundred
413 people on the fetchmail-friends contact list. That's pretty powerful
414 motivation. And they were a good crowd, too, sending fixes and
415 intelligent bug reports in volume. A user population like that is
416 a gift from the gods, and this is my expression of gratitude.<P>
418 The beta testers didn't know it at the time, but they were also the
419 subjects of a sociological experiment. The results are described in
420 my paper, <A HREF="//localhost/~esr/writings/cathedral-bazaar/">The Cathedral
425 Special thanks go to Carl Harris, who built a good solid code base
426 and then tolerated me hacking it out of recognition. And to Harry
427 Hochheiser, who gave me the idea of the SMTP-forwarding delivery mode.<P>
429 Other significant contributors to the code have included Dave Bodenstab
430 (error.c code and --syslog), George Sipe (--monitor and --interface),
431 Gordon Matzigkeit (netrc.c), Al Longyear (UIDL support), Chris
432 Hanson (Kerberos V4 support), and Craig Metz (OPIE, IPv6, IPSEC).<P>
436 At this point, the fetchmail code appears to be pretty stable.
437 It will probably undergo substantial change only if and when support
438 for a new retrieval protocol or authentication method is added.<P>
440 <H1>Relevant RFCS</H1>
442 Not all of these describe standards explicitly used in fetchmail, but they
443 all shaped the design in one way or another.<P>
446 <DT><A HREF="rfc821.txt">RFC821</A>
448 <DT><A HREF="rfc822.txt">RFC822</A>
449 <DD> Mail header format
450 <DT><A HREF="rfc937.txt">RFC937</A>
451 <DD> Post Office Protocol - Version 2
452 <DT><A HREF="rfc974.txt">RFC974</A>
454 <DT><A HREF="rfc976.txt">RFC976</A>
455 <DD> UUCP mail format
456 <DT><A HREF="rfc1081.txt">RFC1081</A>
457 <DD> Post Office Protocol - Version 3
458 <DT><A HREF="rfc1123.txt">RFC1123</A>
459 <DD> Host requirements (modifies 821, 822, and 974)
460 <DT><A HREF="rfc1176.txt">RFC1176</A>
461 <DD> Interactive Mail Access Protocol - Version 2
462 <DT><A HREF="rfc1203.txt">RFC1203</A>
463 <DD> Interactive Mail Access Protocol - Version 3
464 <DT><A HREF="rfc1225.txt">RFC1225</A>
465 <DD> Post Office Protocol - Version 3
466 <DT><A HREF="rfc1344.txt">RFC1344</A>
467 <DD> Implications of MIME for Internet Mail Gateways
468 <DT><A HREF="rfc1413.txt">RFC1413</A>
469 <DD> Identification server
470 <DT><A HREF="rfc1428.txt">RFC1428</A>
471 <DD> Transition of Internet Mail from Just-Send-8 to 8-bit SMTP/MIME
472 <DT><A HREF="rfc1460.txt">RFC1460</A>
473 <DD> Post Office Protocol - Version 3
474 <DT><A HREF="rfc1521.txt">RFC1521</A>
475 <DD> MIME: Multipurpose Internet Mail Extensions
476 <DT><A HREF="rfc1869.txt">RFC1869</A>
477 <DD> SMTP Service Extensions (ESMTP spec)
478 <DT><A HREF="rfc1652.txt">RFC1652</A>
479 <DD> SMTP Service Extension for 8bit-MIMEtransport
480 <DT><A HREF="rfc1725.txt">RFC1725</A>
481 <DD> Post Office Protocol - Version 3
482 <DT><A HREF="rfc1730.txt">RFC1730</A>
483 <DD> Interactive Mail Access Protocol - Version 4
484 <DT><A HREF="rfc1731.txt">RFC1731</A>
485 <DD> IMAP4 Authentication Mechanisms
486 <DT><A HREF="rfc1732.txt">RFC1732</A>
487 <DD> IMAP4 Compatibility With IMAP2 And IMAP2bis
488 <DT><A HREF="rfc1734.txt">RFC1734</A>
489 <DD> POP3 AUTHentication command
490 <DT><A HREF="rfc1870.txt">RFC1870</A>
491 <DD> SMTP Service Extension for Message Size Declaration
492 <DT><A HREF="rfc1891.txt">RFC1891</A>
493 <DD> SMTP Service Extension for Delivery Status Notifications
494 <DT><A HREF="rfc1892.txt">RFC1892</A>
495 <DD> The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages
496 <DT><A HREF="rfc1893.txt">RFC1893</A>
497 <DD> Enhanced Mail System Status Codes
498 <DT><A HREF="rfc1894.txt">RFC1894</A>
499 <DD> An Extensible Message Format for Delivery Status Notifications
500 <DT><A HREF="rfc1938.txt">RFC1938</A>
501 <DD> A One-Time Password System
502 <DT><A HREF="rfc1939.txt">RFC1939</A>
503 <DD> Post Office Protocol - Version 3
504 <DT><A HREF="rfc1985.txt">RFC1985</A>
505 <DD> SMTP Service Extension for Remote Message Queue Starting
506 <DT><A HREF="rfc2033.txt">RFC2033</A>
507 <DD> Local Mail Transfer Protocol
508 <DT><A HREF="rfc2060.txt">RFC2060</A>
509 <DD> Internet Message Access Protocol - Version 4rev1
510 <DT><A HREF="rfc2061.txt">RFC2061</A>
511 <DD> IMAP4 Compatibility With IMAP2bis
512 <DT><A HREF="rfc2062.txt">RFC2062</A>
513 <DD> Internet Message Access Protocol - Obsolete Syntax
517 <table width="100%" cellpadding=0><tr>
518 <td width="30%">Back to <a href="index.html">Fetchmail Home Page</a>
519 <td width="30%" align=center>To <a href="/~esr/sitemap.html">Site Map</a>
520 <td width="30%" align=right>$Date: 1998/11/07 21:16:30 $
523 <P><ADDRESS>Eric S. Raymond <A HREF="mailto:esr@thyrsus.com"><esr@snark.thyrsus.com></A></ADDRESS>
projects / ~andy / fetchmail / blob