-<!doctype HTML public "-//W3O//DTD W3 HTML 2.0//EN">
+<!doctype HTML public "-//W3O//DTD W3 HTML 3.2//EN">
<HTML>
<HEAD>
<TITLE>Design notes on fetchmail</TITLE>
-<link rev=made href=mailto:esr@snark.thyrsus.com>
+<link rev=made href="mailto:esr@snark.thyrsus.com">
<meta name="description" content="Design notes on fetchmail.">
<meta name="keywords" content="fetchmail, POP, POP2, POP3, IMAP, remote mail">
</HEAD>
<BODY>
-<H1><center>Design Notes On Fetchmail</center></H1>
-
-Back to <A HREF="index.html">Fetchmail Home Page</A>.
-<hr>
+<table width="100%" cellpadding=0><tr>
+<td width="30%">Back to <a href="/~esr/index.html">Fetchmail Home Page</a>
+<td width="30%" align=center>To <a href="/~esr/sitemap.html">Site Map</a>
+<td width="30%" align=right>$Date: 2000/01/03 22:59:48 $
+</table>
+<HR>
+<H1 ALIGN=CENTER>Design Notes On Fetchmail</H1>
-This notes are for the benefit of future hackers and maintainers.
+These notes are for the benefit of future hackers and maintainers.
The following sections are both functional and narrative, read from
beginning to end.<P>
<H1>History</H1>
A direct ancestor of the fetchmail program was originally authored
-(under the name popclient) by Carl Harris <ceharris@mal.com>. I took
+(under the name popclient) by Carl Harris <ceharris@mal.com>. I took
over development in June 1996 and subsequently renamed the program
`fetchmail' to reflect the addition of IMAP support. In early
November 1996 Carl officially ended support for the last popclient
<H1>The most-requested features that I will never add, and why not:</H1>
-<H2>1. Password encryption in .fetchmailrc</H2>
+<H2>Password encryption in .fetchmailrc</H2>
The reason there's no facility to store passwords encrypted in the
.fetchmailrc file is because this doesn't actually add protection.<P>
All .fetchmailrc encryption would do is give a false sense of
security to people who don't think very hard.<P>
-<H2>2. Truly concurrent queries to multiple hosts</H2>
+<H2>Truly concurrent queries to multiple hosts</H2>
Occasionally I get a request for this on "efficiency" grounds. These
people aren't thinking either. True concurrency would do nothing to lessen
wait time on an individual response. I judge the gain from this not
worth the hideous complexity increase it would require in the code.<P>
+<H2>Multiple concurrent instances of fetchmail</H2>
+
+Fetchmail locking is on a per-invoking-user because finer-grained
+locks would be really hard to implement in a portable way. The
+problem is that you don't want two fetchmails querying the same site
+for the same remote user at the same time.<P>
+
+To handle this optimally, multiple fetchmails would have to associate
+a system-wide semaphore with each active pair of a remote user and
+host canonical address. A fetchmail would have to block until getting
+this semaphore at the start of a query, and release it at the end of a
+query.<P>
+
+This would be way too complicated to do just for an "it might be nice"
+feature. Instead, you can run a single root fetchmail polling for
+multiple users in either single-drop or multidrop mode.<P>
+
+The fundamental problem here is how an instance of fetchmail polling
+host foo can assert that it's doing so in a way visible to all other
+fetchmails. System V semaphores would be ideal for this purpose, but
+they're not portable.<P>
+
+I've thought about this a lot and roughed up several designs. All are
+complicated and fragile, with a bunch of the standard problems (what
+happens if a fetchmail aborts before clearing its semaphore, and how
+do we recover reliably?).<P>
+
+I'm just not satisfied that there's enough functional gain here to pay
+for the large increase in complexity that adding these semaphores
+would entail.<P>
+
<H1>Multidrop and alias handling</H1>
I decided to add the multidrop support partly because some users were
for the most common case can be much simpler and more robust.<P>
<LI>
- The multidrop handing does <EM>not</EM> rely on doing the equivalent of passing
- the message to sendmail -oem -t. Instead, it explicitly mines members
- of a specified set of local usernames out of the header.<P>
+ The multidrop handing does <EM>not</EM> rely on doing the equivalent of
+ passing the message to sendmail -oem -t. Instead, it explicitly mines
+
members of a specified set of local usernames out of the header.<P>
<LI>
- We do <EM>not</EM> attempt delivery to multidrop mailboxes in the presence of DNS
- errors. Before each multidrop poll we probe DNS to see if we have a
+ We do <EM>not</EM> attempt delivery to multidrop mailboxes in the presence
+ of DNS errors. Before each multidrop poll we probe DNS to see if we have a
nameserver handy. If not, the poll is skipped. If DNS crashes during a
poll, the error return from the next nameserver lookup aborts message
delivery and ends the poll. The daemon mode will then quietly spin until
- DNS comes up again, at which point it will resume delivering mail.<P>
+ DNS comes up again, at which point it will resume delivering mail.
</OL>
When I designed this support, I was terrified of doing anything that could
recipients. These would fit an assumption that DNS lookup errors are
likely to be permanent problems associated with an address.<P>
+<H1>IPv6 and IPSEC</H1>
+
+The IPv6 support patches are really more protocol-family independence
+patches. Because of this, in most places, "ports" (numbers) have been
+replaced with "services" (strings, that may be digits). This allows us
+to run with certain protocols that use strings as "service names"
+where we in the IP world think of port numbers. Someday we'll plumb
+strings all over and then, if inet6 is not enabled, do a
+getservbyname() down in SocketOpen. The IPv6 support patches use
+getaddrinfo(), which is a POSIX p1003.1g mandated function. So, in the
+not too distant future, we'll zap the ifdefs and just let autoconf
+check for getaddrinfo. IPv6 support comes pretty much automatically
+once you have protocol family independence.<P>
+
+<H1>Internationalization</H1>
+
+Internationalization is handled using GNU gettext (see the file
+ABOUT_NLS in the source distribution). This places some
+minor constraints on the code.<P>
+
+Strings that must be subject to translation should be wrapped with _()
+or N_() -- the former in functuib arguments, the latter in static
+initializers and other non-function-argument contexts.<p>
+
+To test the support
+
+<H1>Checklist for Adding Options</H1>
+
+Adding a control option is not complicated in principle, but there are
+a lot of fiddly details in the process. You'll need to do the
+following minimum steps.
+
+<UL>
+<LI>Add a field to represent the control in <code>struct run</code>,
+ <code>struct query</code>, or <code>struct hostdata</code>.
+
+<LI>Go to <code>rcfile_y.y</code>. Add the token to the grammar. Don't
+ forget the <code>%token</code> declaration.
+
+<LI>Pick an actual string to declare the option in the .fetchmailrc file. Add
+ the token to <code>rcfile_l</code>.
+
+<LI>Pick a long-form option name, and a one-letter short option if any
+ are left. Go to <code>options.c</code>. Pick a new <code>LA_</code>
+ value. Hack the <code>longoptions</code> table to set up the
+ association. Hack the big switch statement to set the option.
+ Hack the `?' message to describe it.
+
+<LI>If the default is nonzero, set it in <code>def_opts</code> near the top of
+ <code>load_params</code> in <code>fetchmail.c</code>.
+
+<LI>Add code to dump the option value in <code>fetchmail.c:dump_params</code>.
+
+<LI>Add proper <code>FLAG_MERGE</code> actions in fetchmail.c's
+ optmerge() function. (If it's a global option, add an override
+ at the end of load_params.)
+
+<LI>Document the option in fetchmail.man. This will require at least
+ two changes; one to the collected table of options, and one full
+ text description of the option.
+
+<LI>Add the new token and a brief description to the header comment of
+ sample.rcfile.
+
+<LI>Hack fetchmailconf to configure it. Bump the fetchmailconf version.
+
+<LI>Hack conf.c to dump the option so we won't have a version-skew problem.
+
+<LI>Add an entry to NEWS.
+</UL>
+
+There may be other things you have to do in the way of logic, of course.<P>
+
+Before you implement an option, though, think hard. Is there any way
+to make fetchmail automatically detect the circumstances under which
+it should change its behavior? If so, don't write an option. Just do
+the check!<p>
+
<H1>Lessons learned</H1>
<H3>1. Server-side state is essential</H3>
The beta testers didn't know it at the time, but they were also the
subjects of a sociological experiment. The results are described in
-my paper, <cite>The Cathedral And The Bazaar</cite>, available on the
-<a href="http://www.ccil.org/~esr/fetchmail">Fetchmail home page</a>.
+my paper, <A
+HREF="//www.tuxedo.org/~esr/writings/cathedral-bazaar/">The Cathedral
+And The Bazaar</A>.<P>
<H1>Credits</H1>
Other significant contributors to the code have included Dave Bodenstab
(error.c code and --syslog), George Sipe (--monitor and --interface),
-Gordon Matzigkeit (netrc.c), Al Longyear (UIDL support), and Nalin
-
Dahyabhai (Kerberos V4 support).<P>
+Gordon Matzigkeit (netrc.c), Al Longyear (UIDL support), Chris
+
Hanson (Kerberos V4 support), and Craig Metz (OPIE, IPv6, IPSEC).<P>
<H1>Conclusion</H1>
all shaped the design in one way or another.<P>
<DL>
-<DT>RFC821<DD> SMTP protocol
-<DT>RFC822<DD> Mail header format
-<DT>RFC937<DD> Post Office Protocol - Version 2
-<DT>RFC974<DD> MX routing
-<DT>RFC976<DD> UUCP mail format
-<DT>RFC1081<DD> Post Office Protocol - Version 3
-<DT>RFC1123<DD> Host requirements (modifies 821, 822, and 974)
-<DT>RFC1176<DD> Interactive Mail Access Protocol - Version 2
-<DT>RFC1203<DD> Interactive Mail Access Protocol - Version 3
-<DT>RFC1225<DD> Post Office Protocol - Version 3
-<DT>RFC1344<DD> Implications of MIME for Internet Mail Gateways
-<DT>RFC1413<DD> Identification server
-<DT>RFC1428<DD> Transition of Internet Mail from Just-Send-8 to 8-bit SMTP/MIME
-<DT>RFC1460<DD> Post Office Protocol - Version 3
-<DT>RFC1521<DD> MIME: Multipurpose Internet Mail Extensions
-<DT>RFC1869<DD> SMTP Service Extensions (ESMTP spec)
-<DT>RFC1652<DD> SMTP Service Extension for 8bit-MIMEtransport
-<DT>RFC1725<DD> Post Office Protocol - Version 3
-<DT>RFC1730<DD> Interactive Mail Access Protocol - Version 4
-<DT>RFC1731<DD> IMAP4 Authentication Mechanisms
-<DT>RFC1732<DD> IMAP4 Compatibility With IMAP2 And IMAP2bis
-<DT>RFC1734<DD> POP3 AUTHentication command
-<DT>RFC1870<DD> SMTP Service Extension for Message Size Declaration
-<DT>RFC1891<DD> SMTP Service Extension for Delivery Status Notifications
-<DT>RFC1893<DD> Enhanced Mail System Status Codes
-<DT>RFC1894<DD> An Extensible Message Format for Delivery Status Notifications
-<DT>RFC1938<DD> A One-Time Password System
-<DT>RFC1939<DD> Post Office Protocol - Version 3
-<DT>RFC1985<DD> SMTP Service Extension for Remote Message Queue Starting
-<DT>RFC2060<DD> Internet Message Access Protocol - Version 4rev1
-<DT>RFC2061<DD> IMAP4 Compatibility With IMAP2bis
-<DT>RFC2062<DD> Internet Message Access Protocol - Obsolete Syntax
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc821.txt">RFC821</A>
+<DD> SMTP protocol
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc822.txt">RFC822</A>
+<DD> Mail header format
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc937.txt">RFC937</A>
+<DD> Post Office Protocol - Version 2
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc974.txt">RFC974</A>
+<DD> MX routing
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc976.txt">RFC976</A>
+<DD> UUCP mail format
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1081.txt">RFC1081</A>
+<DD> Post Office Protocol - Version 3
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1123.txt">RFC1123</A>
+<DD> Host requirements (modifies 821, 822, and 974)
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1176.txt">RFC1176</A>
+<DD> Interactive Mail Access Protocol - Version 2
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1203.txt">RFC1203</A>
+<DD> Interactive Mail Access Protocol - Version 3
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1225.txt">RFC1225</A>
+<DD> Post Office Protocol - Version 3
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1344.txt">RFC1344</A>
+<DD> Implications of MIME for Internet Mail Gateways
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1413.txt">RFC1413</A>
+<DD> Identification server
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1428.txt">RFC1428</A>
+<DD> Transition of Internet Mail from Just-Send-8 to 8-bit SMTP/MIME
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1460.txt">RFC1460</A>
+<DD> Post Office Protocol - Version 3
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1521.txt">RFC1521</A>
+<DD> MIME: Multipurpose Internet Mail Extensions
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1869.txt">RFC1869</A>
+<DD> SMTP Service Extensions (ESMTP spec)
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1652.txt">RFC1652</A>
+<DD> SMTP Service Extension for 8bit-MIMEtransport
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1725.txt">RFC1725</A>
+<DD> Post Office Protocol - Version 3
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1730.txt">RFC1730</A>
+<DD> Interactive Mail Access Protocol - Version 4
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1731.txt">RFC1731</A>
+<DD> IMAP4 Authentication Mechanisms
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1732.txt">RFC1732</A>
+<DD> IMAP4 Compatibility With IMAP2 And IMAP2bis
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1734.txt">RFC1734</A>
+<DD> POP3 AUTHentication command
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1870.txt">RFC1870</A>
+<DD> SMTP Service Extension for Message Size Declaration
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1891.txt">RFC1891</A>
+<DD> SMTP Service Extension for Delivery Status Notifications
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1892.txt">RFC1892</A>
+<DD> The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1894.txt">RFC1894</A>
+<DD>An Extensible Message Format for Delivery Status Notifications
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1893.txt">RFC1893</A>
+<DD> Enhanced Mail System Status Codes
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1894.txt">RFC1894</A>
+<DD> An Extensible Message Format for Delivery Status Notifications
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1938.txt">RFC1938</A>
+<DD> A One-Time Password System
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1939.txt">RFC1939</A>
+<DD> Post Office Protocol - Version 3
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc1985.txt">RFC1985</A>
+<DD> SMTP Service Extension for Remote Message Queue Starting
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc2033.txt">RFC2033</A>
+<DD> Local Mail Transfer Protocol
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc2060.txt">RFC2060</A>
+<DD> Internet Message Access Protocol - Version 4rev1
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc2061.txt">RFC2061</A>
+<DD> IMAP4 Compatibility With IMAP2bis
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc2062.txt">RFC2062</A>
+<DD> Internet Message Access Protocol - Obsolete Syntax
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc2195.txt">RFC2449</A>
+<DD> IMAP/POP AUTHorize Extension for Simple Challenge/Response
+<DT><A HREF="ftp://ftp.isi.edu/in-notes/rfc2449.txt">RFC2449</A>
+<DD> POP3 Extension Mechanism
</DL>
<HR>
-Back to <A HREF="index.html">Fetchmail Home Page</A>.<P>
-<ADDRESS>Eric S. Raymond <A HREF="mailto:esr@thyrsus.com"><esr@snark.thyrsus.com></A></ADDRESS>
+<table width="100%" cellpadding=0><tr>
+<td width="30%">Back to <a href="index.html">Fetchmail Home Page</a>
+<td width="30%" align=center>To <a href="/~esr/sitemap.html">Site Map</a>
+<td width="30%" align=right>$Date: 2000/01/03 22:59:48 $
+</table>
+
+<P><ADDRESS>Eric S. Raymond <A HREF="mailto:esr@thyrsus.com"><esr@snark.thyrsus.com></A></ADDRESS>
</BODY>
</HTML>
projects / ~andy / fetchmail / blobdiff