X-Git-Url: http://pileus.org/git/?a=blobdiff_plain;f=design-notes.html;h=ef2cc07af7068c35ed121b882f87b7032e665acd;hb=cb8d898683ffccdf8de42a5b1236a6cf3cdbb6ce;hp=b6ddec2de3c91d27841019649ba11cd196fb0574;hpb=0aa7e4307aae2252986fb0e440461642c833a677;p=~andy%2Ffetchmail diff --git a/design-notes.html b/design-notes.html index b6ddec2d..ef2cc07a 100644 --- a/design-notes.html +++ b/design-notes.html @@ -1,471 +1,130 @@ - - -
-Back to Fetchmail Home Page - | To Site Map - | $Date: 1998/03/06 04:17:44 $
+
+
+
+
+
- Design Notes On Fetchmail-These notes are for the benefit of future hackers and maintainers. -The following sections are both functional and narrative, read from -beginning to end.- - History- -A direct ancestor of the fetchmail program was originally authored -(under the name popclient) by Carl Harris- -Before accepting responsibility for the popclient sources from Carl, I -had investigated and used and tinkered with every other UNIX -remote-mail forwarder I could find, including fetchpop1.9, -PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc, -popmail-1.6 and upop. My major goal was to get a header-rewrite -feature like fetchmail's working so I wouldn't have reply problems -anymore. - -Despite having done a good bit of work on fetchpop1.9, when I found -popclient I quickly concluded that it offered the solidest base for -future development. I was convinced of this primarily by the presence -of multiple-protocol support. The competition didn't do -POP2/RPOP/APOP, and I was already having vague thoughts of maybe -adding IMAP. (This would advance two other goals: learn IMAP and get -comfortable writing TCP/IP client software.) - -Until popclient 3.05 I was simply following out the implications of -Carl's basic design. He already had daemon.c in the distribution, -and I wanted daemon mode almost as badly as I wanted the header -rewrite feature. The other things I added were bug fixes or -minor extensions. - -After 3.1, when I put in SMTP-forwarding support (more about this -below) the nature of the project changed -- it became a -carefully-thought-out attempt to render obsolete every other program -in its class. The name change quickly followed. - - The rewrite option- -RFC 1123 stipulates that MTAs ought to canonicalize the addresses of -outgoing mail so that From:, To:, Cc:, Bcc: and other address headers -contain only fully qualified domain names. Failure to do so can break -the reply function on many mailers.- -This problem only becomes obvious when a reply is generated on a -machine different from where the message was delivered. The -two machines will have different local username spaces, potentially -leading to misrouted mail. - -Most MTAs (and sendmail in particular) do not canonicalize address headers -in this way (violating RFC 1123). Fetchmail therefore has to do it. This -is the first feature I added to the ancestral popclient. - - Reorganization- -The second thing I did reorganize and simplify popclient a lot. Carl -Harris's implementation was very sound, but exhibited a kind of -unnecessary complexity common to many C programmers. He treated the -code as central and the data structures as support for the code. As a -result, the code was beautiful but the data structure design ad-hoc -and rather ugly (at least to this old LISP hacker).- -I was able to improve matters significantly by reorganizing most of the -program around the `query' data structure and eliminating a bunch of -global context. This especially simplified the main sequence in -fetchmail.c and was critical in enabling the daemon mode changes. - - IMAP support and the method table- -The next step was IMAP support. I initially wrote the IMAP code -as a generic query driver and a method table. The idea was to have -all the protocol-independent setup logic and flow of control in the -driver, and the protocol-specific stuff in the method table.- -Once this worked, I rewrote the POP3 code to use the same organization. -The POP2 code kept its own driver for a couple more releases, until -I found sources of a POP2 server to test against (the breed seems -to be nearly extinct). - -The purpose of this reorganization, of course, is to trivialize -the development of support for future protocols as much as possible. -All mail-retrieval protocols have to have pretty similar logical -design by the nature of the task. By abstracting out that common -logic and its interface to the rest of the program, both the common -and protocol-specific parts become easier to understand. - -Furthermore, many kinds of new features can instantly be supported -across all protocols by modifying the one driver module. - - Implications of smtp forwarding- -The direction of the project changed radically when Harry Hochheiser -sent me his scratch code for forwarding fetched mail to the SMTP port. -I realized almost immediately that a reliable implementation of this -feature would make all the other delivery modes obsolete.- -Why mess with all the complexity of configuring an MDA or setting up -lock-and-append on a mailbox when port 25 is guaranteed to be there on -any platform with TCP/IP support in the first place? Especially when -this means retrieved mail is guaranteed to look like normal sender- -initiated SMTP mail, which is really what we want anyway. - -Clearly, the right thing to do was (1) hack SMTP forwarding support -into the generic driver, (2) make it the default mode, and (3) eventually -throw out all the other delivery modes. - -I hesitated over step 3 for some time, fearing to upset long-time -popclient users dependent on the alternate delivery mechanisms. In -theory, they could immediately switch to .forward files or their -non-sendmail equivalents to get the same effects. In practice the -transition might have been messy. - -But when I did it (see the NEWS note on the great options massacre) -the benefits proved huge. The cruftiest parts of the driver code -vanished. Configuration got radically simpler -- no more grovelling -around for the system MDA and user's mailbox, no more worries about -whether the underlying OS supports file locking. - -Also, the only way to lose mail vanished. If you specified localfolder -and the disk got full, your mail got lost. This can't happen with -SMTP forwarding because your SMTP listener won't return OK unless -the message can be spooled or processed. - -Also, performance improved (though not so you'd notice it in a single -run). Another not insignificant benefit of this change was that the -manual page got a lot simpler. - -Later, I had to bring --mda back in order to allow handling of some -obscure situations involving dynamic SLIP. But I found a much simpler -way to do it. - -The moral? Don't hesitate to throw away superannuated features when -you can do it without loss of effectiveness. I tanked a couple I'd -added myself and have no regrets at all. As Saint-Exupery said, -"Perfection [in design] is achieved not when there is nothing more to -add, but rather when there is nothing more to take away." This -program isn't perfect, but it's trying. - - The most-requested features that I will never add, and why not:- -1. Password encryption in .fetchmailrc- -The reason there's no facility to store passwords encrypted in the -.fetchmailrc file is because this doesn't actually add protection.- -Anyone who's acquired the 0600 permissions needed to read your -.fetchmailrc file will be able to run fetchmail as you anyway -- and -if it's your password they're after, they'd be able to rip the -necessary decoder out of the fetchmail code itself to get it. - -All .fetchmailrc encryption would do is give a false sense of -security to people who don't think very hard. - - Truly concurrent queries to multiple hosts- -Occasionally I get a request for this on "efficiency" grounds. These -people aren't thinking either. True concurrency would do nothing to lessen -fetchmail's total IP volume. The best it could possibly do is change the -usage profile to shorten the duration of the active part of a poll cycle -at the cost of increasing its demand on IP volume per unit time.- -If one could thread the protocol code so that fetchmail didn't block -on waiting for a protocol response, but rather switched to trying to -process another host query, one might get an efficiency gain (close to -constant loading at the single-host level). - -Fortunately, I've only seldom seen a server that incurred significant -wait time on an individual response. I judge the gain from this not -worth the hideous complexity increase it would require in the code. - - Multiple concurrent instances of fetchmail - -What would be required for this is a per-host semaphore asserted -during each poll.- -The fundamental problem here is how an instance of fetchmail polling -host foo can assert that it's doing so in a way visible to all other -fetchmails. System V semaphores would be ideal for this purpose, but -they're not portable. - -I've thought about this a lot and roughed up several designs. All are -complicated and fragile, with a bunch of the standard problems (what -happens if a fetchmail aborts before clearing its semaphore, and how -do we recover reliably?) . - -I'm not satisfied that there's enough functional gain here to pay -for the large increase in complexity that adding these semaphores -would entail. - - Multidrop and alias handling- -I decided to add the multidrop support partly because some users were -clamoring for it, but mostly because I thought it would shake bugs out -of the single-drop code by forcing me to deal with addressing in full -generality. And so it proved.- -There are two important aspects of the features for handling -multiple-drop aliases and mailing lists which future hackers should be -careful to preserve. - -
- -The code in mxget.c is nasty, no two ways about it. But it's utterly -necessary, there are a lot of MX pointers out there. It really ought -to be a (documented!) entry point in the bind library. - - DNS error handling- -Fetchmail's behavior on DNS errors is to suppress forwarding and -deletion of the individual message that each occurs in, leaving it -queued on the server for retrieval on a subsequent poll. The -assumption is that DNS errors are transient, due to temporary server -outages.- -Unfortunately this means that if a DNS error is permanent a message -can be perpetually stuck in the server mailbox. We've had a couple -bug reports of this kind due to subtle RFC822 parsing errors in the fetchmail -code that resulted in impossible things getting passed to the DNS lookup -routines. - -Alternative ways to handle the problem: ignore DNS errors (treating -them as a non-match on the mailserver domain), or forward messages -with errors to fetchmail's invoking user in addition to any other -recipients. These would fit an assumption that DNS lookup errors are -likely to be permanent problems associated with an address. - - IPv6 and IPSEC- -The IPv6 support patches are really more protocol-family independence -patches. Because of this, in most places, "ports" (numbers) have been -replaced with "services" (strings, that may be digits). This allows us -to run with certain protocols that use strings as "service names" -where we in the IP world think of port numbers. Someday we'll plumb -strings all over and then, if inet6 is not enabled, do a -getservbyname() down in SocketOpen. The IPv6 support patches use -getaddrinfo(), which is a POSIX p1003.1g mandated function. So, in the -not too distant future, we'll zap the ifdefs and just let autoconf -check for getaddrinfo. IPv6 support comes pretty much automatically -once you have protocol family independence.- -Craig Metz used his inner_connect() function to handle most of the -connect work. This is a nonstandard function not likely to ever exist -in a system's libc, but we can just include that source file if the -day comes when we want to support IPv6 without the inet6-apps -library. It just makes life easier. - - Checklist for Adding Options
-
-Adding a control option is not complicated in principle, but there are
-a lot of fiddly details in the process. You'll need to do the
-following minimum steps.
-
-struct query or
- struct hostdata .
-
-rcfile_l .
-
-rcfile_y.y . Add the token to the grammar. Don't
- forget the %token declaration. Add proper
- FLAG_FORCE and FLAG_MERGE actions.
-
-options.c . Pick a new LA_ |
Back to Fetchmail Home Page - | To Site Map - | $Date: 1998/03/06 04:17:44 $
+ + Design Notes On Fetchmail+ +Introduction+ +This document is supposed to complement Eric S. Raymond's (ESR's) + design notes. The new maintainers don't agree with some of the decisions +ESR made previously, and the differences and new directions will be laid +out in this document. It is therefore a sort of a TODO document, until +the necessary code revisions have been made. + +Security+ +Fetchmail was handed over in a pretty poor shape, security-wise. It will +happily talk to the network with root privileges, use sscanf() to read +remotely received data into fixed-length stack-based buffers without +length limitation and so on. A full audit is required and security +concepts will have to be applied. Random bits are: + +
SMTP forwarding+ +Fetchmail's multidrop and rewrite options will process addresses +received from remote sites. Special care must be taken so these +features cannot be abused to relay mail to foreign sites. + +ESR's attempt to make fetchmail use SMTP exclusively failed, +fetchmail got LMTP and --mda options – the latter has a lot of +flaws unfortunately, is inconsistent with the SMTP forwarder and needs +to be reviewed and probably bugfixed. --mda doesn't properly work with +multiple recipients, it cannot properly communicate errors and is best +avoided for now. + +Server-side vs. client-side state.+ +Why we need client-side tracking+ +ESR asserted that server-side state were essential and those persons +repsonsible for removing the LAST command from POP3 deserved to +suffer. ESR is right in stating that the POP3 UID tracks which messages +have been read by this client – and that is exactly what +we need to do. + +If fetchmail is supposed to retrieve all mail from a mailbox +reliably, without being disturbed by someone occasionally using another +client on another host, or a webmailer, or similar, then +client-side tracking of the state is indispensable. This is +also needed to match behavior to ETRN and ODMR or to support read-only +mailboxes in --keep mode. + +Present and future+ +Fetchmail supports client-side state in POP3 if the UIDL option is +used (which is strongly recommended). Similar effort needs to be made to +track IMAP state by means of UIDVALIDITY and UID. + +This will also mean that the UID handling code be revised an perhaps +use one file per account or per folder. + +Concurrent queries/concurrent fetchmail instances+ +ESR refused to make fetchmail query multiple hosts or accounts +concurrently, on the grounds that finer-grained locks would be hard to +implement portably. + +The idea of using one file per folder or account to track UIDs on the +client-side will make solving this locking problem easy – the lock can +be placed on the UID file instead. + +Multidrop issues+ +Fetchmail tries to guess recipients from headers that are not routing +relevant, for instance, To:, Cc:, or Resent-headers (which are rare +anyways). It is important that fetchmail insists on the real envelope +operation for multidrop. This is detailed in my + article "Requisites for working multidrop + mailboxes". + +As Terry Lambert pointed out in the FreeBSD-arch mailing list on +2001-02-17 under the subject "UUCP must stay; fetchmail sucks", +fetchmail performs DNS MX lookups to determine domains for which +multidrop is valid, on the assumption that the receiving SMTP host +upstream were the same as the IMAP or POP3 server. + ++
+Matthias Andree <matthias.andree@gmx.de> + + |