X-Git-Url: http://pileus.org/git/?a=blobdiff_plain;f=design-notes.html;h=d7af2650cafcfcfc372dee783a173f66a06ba53f;hb=9c5aa38035005272a381fb280781efd1a2199c90;hp=134f00669f35870ffac596e34672ef3adce42c47;hpb=5b6b38cbd82468ac720d25b1889f63f4251bf4db;p=~andy%2Ffetchmail diff --git a/design-notes.html b/design-notes.html index 134f0066..d7af2650 100644 --- a/design-notes.html +++ b/design-notes.html @@ -1,383 +1,130 @@ - - -
-- -
- -Before accepting responsibility for the popclient sources from Carl, I -had investigated and used and tinkered with every other UNIX -remote-mail forwarder I could find, including fetchpop1.9, -PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc, -popmail-1.6 and upop. My major goal was to get a header-rewrite -feature like fetchmail's working so I wouldn't have reply problems -anymore.
- -Despite having done a good bit of work on fetchpop1.9, when I found -popclient I quickly concluded that it offered the solidest base for -future development. I was convinced of this primarily by the presence -of multiple-protocol support. The competition didn't do -POP2/RPOP/APOP, and I was already having vague thoughts of maybe -adding IMAP. (This would advance two other goals: learn IMAP and get -comfortable writing TCP/IP client software.)
- -Until popclient 3.05 I was simply following out the implications of -Carl's basic design. He already had daemon.c in the distribution, -and I wanted daemon mode almost as badly as I wanted the header -rewrite feature. The other things I added were bug fixes or -minor extensions.
- -After 3.1, when I put in SMTP-forwarding support (more about this -below) the nature of the project changed -- it became a -carefully-thought-out attempt to render obsolete every other program -in its class. The name change quickly followed.
- -
- -This problem only becomes obvious when a reply is generated on a -machine different from where the message was delivered. The -two machines will have different local username spaces, potentially -leading to misrouted mail.
- -Most MTAs (and sendmail in particular) do not canonicalize address headers -in this way (violating RFC 1123). Fetchmail therefore has to do it. This -is the first feature I added to the ancestral popclient.
- -
- -I was able to improve matters significantly by reorganizing most of the -program around the `query' data structure and eliminating a bunch of -global context. This especially simplified the main sequence in -fetchmail.c and was critical in enabling the daemon mode changes.
- -
- -Once this worked, I rewrote the POP3 code to use the same organization. -The POP2 code kept its own driver for a couple more releases, until -I found sources of a POP2 server to test against (the breed seems -to be nearly extinct).
- -The purpose of this reorganization, of course, is to trivialize -the development of support for future protocols as much as possible. -All mail-retrieval protocols have to have pretty similar logical -design by the nature of the task. By abstracting out that common -logic and its interface to the rest of the program, both the common -and protocol-specific parts become easier to understand.
- -Furthermore, many kinds of new features can instantly be supported -across all protocols by modifying the one driver module.
- -
- -Why mess with all the complexity of configuring an MDA or setting up -lock-and-append on a mailbox when port 25 is guaranteed to be there on -any platform with TCP/IP support in the first place? Especially when -this means retrieved mail is guaranteed to look like normal sender- -initiated SMTP mail, which is really what we want anyway.
- -Clearly, the right thing to do was (1) hack SMTP forwarding support -into the generic driver, (2) make it the default mode, and (3) eventually -throw out all the other delivery modes.
- -I hesitated over step 3 for some time, fearing to upset long-time -popclient users dependent on the alternate delivery mechanisms. In -theory, they could immediately switch to .forward files or their -non-sendmail equivalents to get the same effects. In practice the -transition might have been messy.
- -But when I did it (see the NEWS note on the great options massacre) -the benefits proved huge. The cruftiest parts of the driver code -vanished. Configuration got radically simpler -- no more grovelling -around for the system MDA and user's mailbox, no more worries about -whether the underlying OS supports file locking.
- -Also, the only way to lose mail vanished. If you specified localfolder -and the disk got full, your mail got lost. This can't happen with -SMTP forwarding because your SMTP listener won't return OK unless -the message can be spooled or processed.
- -Also, performance improved (though not so you'd notice it in a single -run). Another not insignificant benefit of this change was that the -manual page got a lot simpler.
- -Later, I had to bring --mda back in order to allow handling of some -obscure situations involving dynamic SLIP. But I found a much simpler -way to do it.
- -The moral? Don't hesitate to throw away superannuated features when -you can do it without loss of effectiveness. I tanked a couple I'd -added myself and have no regrets at all. As Saint-Exupery said, -"Perfection [in design] is achieved not when there is nothing more to -add, but rather when there is nothing more to take away." This -program isn't perfect, but it's trying.
- -
- -Anyone who's acquired the 0600 permissions needed to read your -.fetchmailrc file will be able to run fetchmail as you anyway -- and -if it's your password they're after, they'd be able to rip the -necessary decoder out of the fetchmail code itself to get it.
- -All .fetchmailrc encryption would do is give a false sense of -security to people who don't think very hard.
- -
- -If one could thread the protocol code so that fetchmail didn't block -on waiting for a protocol response, but rather switched to trying to -process another host query, one might get an efficiency gain (close to -constant loading at the single-host level).
- -Fortunately, I've only seldom seen a server that incurred significant -wait time on an individual response. I judge the gain from this not -worth the hideous complexity increase it would require in the code.
- -
- -There are two important aspects of the features for handling -multiple-drop aliases and mailing lists which future hackers should be -careful to preserve.
- -
- -
- -
-
- -The code in mxget.c is nasty, no two ways about it. But it's utterly -necessary, there are a lot of MX pointers out there. It really ought -to be a (documented!) entry point in the bind library.
- -
- -Unfortunately this means that if a DNS error is permanent a message -can be perpetually stuck in the server mailbox. We've had a couple -bug reports of this kind due to subtle RFC822 parsing errors in the fetchmail -code that resulted in impossible things getting passed to the DNS lookup -routines.
- -Alternative ways to handle the problem: ignore DNS errors (treating -them as a non-match on the mailserver domain), or forward messages -with errors to fetchmail's invoking user in addition to any other -recipients. These would fit an assumption that DNS lookup errors are -likely to be permanent problems associated with an address.
- -
- -The POP3 UID feature described in RFC1725 to replace LAST is -insufficient. The only problem it solves is tracking which messages -have been read by this client -- and even that requires -tricky, fragile implementation.
- -The underlying lesson is that maintaining accessible server-side -`seen' state bits associated with Status headers is indispensible in a -Unix/RFC822 mail server protocol. IMAP gets this right.
- -
- -This is an advantage not to be despised! Because of it, this project has -been interesting and fun -- no serious or persistent bugs, no long -hours spent looking for subtle pathologies.
- -
- -
- -
- -
- -It's gratifying that fetchmail has become so popular. Until just before -1.9 I was designing strictly to my own taste. The multi-drop mailbox -support and the new --limit option were the first features to go in that -I didn't need myself.
- -By 1.9, four months after I started hacking on popclient and a month -after the first fetchmail release, there were literally a hundred -people on the fetchmail-friends contact list. That's pretty powerful -motivation. And they were a good crowd, too, sending fixes and -intelligent bug reports in volume. A user population like that is -a gift from the gods, and this is my expression of gratitude.
- -The beta testers didn't know it at the time, but they were also the -subjects of a sociological experiment. The results are described in -my paper, The Cathedral And The Bazaar, available on the -Fetchmail home page. - -
- -Other significant contributors to the code have included Dave Bodenstab -(error.c code and --syslog), George Sipe (--monitor and --interface), -Gordon Matzigkeit (netrc.c), Al Longyear (UIDL support), and Nalin -Dahyabhai (Kerberos V4 support).
- -
- -
- -
-
Eric S. Raymond <esr@snark.thyrsus.com> - - + + + + +Back to Fetchmail Home Page | +$Date$ | +
This document is supposed to complement Eric S. Raymond's (ESR's) + design notes. The new maintainers don't agree with some of the decisions +ESR made previously, and the differences and new directions will be laid +out in this document. It is therefore a sort of a TODO document, until +the necessary code revisions have been made.
+ +Fetchmail was handed over in a pretty poor shape, security-wise. It will +happily talk to the network with root privileges, use sscanf() to read +remotely received data into fixed-length stack-based buffers without +length limitation and so on. A full audit is required and security +concepts will have to be applied. Random bits are:
+ +Fetchmail's multidrop and rewrite options will process addresses +received from remote sites. Special care must be taken so these +features cannot be abused to relay mail to foreign sites.
+ +ESR's attempt to make fetchmail use SMTP exclusively failed, +fetchmail got LMTP and --mda options – the latter has a lot of +flaws unfortunately, is inconsistent with the SMTP forwarder and needs +to be reviewed and probably bugfixed. --mda doesn't properly work with +multiple recipients, it cannot properly communicate errors and is best +avoided for now.
+ +ESR asserted that server-side state were essential and those persons +responsible for removing the LAST command from POP3 deserved to +suffer. ESR is right in stating that the POP3 UID tracks which messages +have been read by this client – and that is exactly what +we need to do.
+ +If fetchmail is supposed to retrieve all mail from a mailbox +reliably, without being disturbed by someone occasionally using another +client on another host, or a webmailer, or similar, then +client-side tracking of the state is indispensable. This is +also needed to match behavior to ETRN and ODMR or to support read-only +mailboxes in --keep mode.
+ +Fetchmail supports client-side state in POP3 if the UIDL option is +used (which is strongly recommended). Similar effort needs to be made to +track IMAP state by means of UIDVALIDITY and UID.
+ +This will also mean that the UID handling code be revised an perhaps +use one file per account or per folder.
+ +ESR refused to make fetchmail query multiple hosts or accounts +concurrently, on the grounds that finer-grained locks would be hard to +implement portably.
+ +The idea of using one file per folder or account to track UIDs on the +client-side will make solving this locking problem easy – the lock can +be placed on the UID file instead.
+ +Fetchmail tries to guess recipients from headers that are not routing +relevant, for instance, To:, Cc:, or Resent-headers (which are rare +anyways). It is important that fetchmail insists on the real envelope +operation for multidrop. This is detailed in my + article "Requisites for working multidrop + mailboxes".
+ +As Terry Lambert pointed out in the FreeBSD-arch mailing list on +2001-02-17 under the subject "UUCP must stay; fetchmail sucks", +fetchmail performs DNS MX lookups to determine domains for which +multidrop is valid, on the assumption that the receiving SMTP host +upstream were the same as the IMAP or POP3 server.
+ +Back to Fetchmail Home Page | +$Date$ | +