-<p>These notes are for the benefit of future hackers and
-maintainers. The following sections are both functional and
-narrative, read from beginning to end.</p>
-
-<h1>History</h1>
-
-<p>A direct ancestor of the fetchmail program was originally
-authored (under the name popclient) by Carl Harris
-<ceharris@mal.com>. I took over development in June 1996 and
-subsequently renamed the program `fetchmail' to reflect the
-addition of IMAP support and SMTP delivery. In early November 1996
-Carl officially ended support for the last popclient versions.</p>
-
-<p>Before accepting responsibility for the popclient sources from
-Carl, I had investigated and used and tinkered with every other
-UNIX remote-mail forwarder I could find, including fetchpop1.9,
-PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc,
-popmail-1.6 and upop. My major goal was to get a header-rewrite
-feature like fetchmail's working so I wouldn't have reply problems
-anymore.</p>
-
-<p>Despite having done a good bit of work on fetchpop1.9, when I
-found popclient I quickly concluded that it offered the solidest
-base for future development. I was convinced of this primarily by
-the presence of multiple-protocol support. The competition didn't
-do POP2/RPOP/APOP, and I was already having vague thoughts of maybe
-adding IMAP. (This would advance two other goals: learn IMAP and
-get comfortable writing TCP/IP client software.)</p>
-
-<p>Until popclient 3.05 I was simply following out the implications
-of Carl's basic design. He already had daemon.c in the
-distribution, and I wanted daemon mode almost as badly as I wanted
-the header rewrite feature. The other things I added were bug fixes
-or minor extensions.</p>
-
-<p>After 3.1, when I put in SMTP-forwarding support (more about
-this below) the nature of the project changed -- it became a
-carefully-thought-out attempt to render obsolete every other
-program in its class. The name change quickly followed.</p>
-
-<h1>The rewrite option</h1>
-
-<p>MTAs ought to canonicalize the addresses of outgoing non-local
-mail so that From:, To:, Cc:, Bcc: and other address headers
-contain only fully qualified domain names. Failure to do so can
-break the reply function on many mailers. (Sendmail has an option
-to do this.)</p>
-
-<p>This problem only becomes obvious when a reply is generated on a
-machine different from where the message was delivered. The two
-machines will have different local username spaces, potentially
-leading to misrouted mail.</p>
-
-<p>Most MTAs (and sendmail in particular) do not canonicalize
-address headers in this way (violating RFC 1123). Fetchmail
-therefore has to do it. This is the first feature I added to the
-ancestral popclient.</p>
-
-<h1>Reorganization</h1>
-
-<p>The second thing I did reorganize and simplify popclient a lot.
-Carl Harris's implementation was very sound, but exhibited a kind
-of unnecessary complexity common to many C programmers. He treated
-the code as central and the data structures as support for the
-code. As a result, the code was beautiful but the data structure
-design ad-hoc and rather ugly (at least to this old LISP
-hacker).</p>
-
-<p>I was able to improve matters significantly by reorganizing most
-of the program around the `query' data structure and eliminating a
-bunch of global context. This especially simplified the main
-sequence in fetchmail.c and was critical in enabling the daemon
-mode changes.</p>
-
-<h1>IMAP support and the method table</h1>
-
-<p>The next step was IMAP support. I initially wrote the IMAP code
-as a generic query driver and a method table. The idea was to have
-all the protocol-independent setup logic and flow of control in the
-driver, and the protocol-specific stuff in the method table.</p>
-
-<p>Once this worked, I rewrote the POP3 code to use the same
-organization. The POP2 code kept its own driver for a couple more
-releases, until I found sources of a POP2 server to test against
-(the breed seems to be nearly extinct).</p>
-
-<p>The purpose of this reorganization, of course, is to trivialize
-the development of support for future protocols as much as
-possible. All mail-retrieval protocols have to have pretty similar
-logical design by the nature of the task. By abstracting out that
-common logic and its interface to the rest of the program, both the
-common and protocol-specific parts become easier to understand.</p>
-
-<p>Furthermore, many kinds of new features can instantly be
-supported across all protocols by modifying the one driver
-module.</p>
-
-<h1>Implications of smtp forwarding</h1>
-
-<p>The direction of the project changed radically when Harry
-Hochheiser sent me his scratch code for forwarding fetched mail to
-the SMTP port. I realized almost immediately that a reliable
-implementation of this feature would make all the other delivery
-modes obsolete.</p>
-
-<p>Why mess with all the complexity of configuring an MDA or
-setting up lock-and-append on a mailbox when port 25 is guaranteed
-to be there on any platform with TCP/IP support in the first place?
-Especially when this means retrieved mail is guaranteed to look
-like normal sender- initiated SMTP mail, which is really what we
-want anyway.</p>
-
-<p>Clearly, the right thing to do was (1) hack SMTP forwarding
-support into the generic driver, (2) make it the default mode, and
-(3) eventually throw out all the other delivery modes.</p>
-
-<p>I hesitated over step 3 for some time, fearing to upset
-long-time popclient users dependent on the alternate delivery
-mechanisms. In theory, they could immediately switch to .forward
-files or their non-sendmail equivalents to get the same effects. In
-practice the transition might have been messy.</p>
-
-<p>But when I did it (see the NEWS note on the great options
-massacre) the benefits proved huge. The cruftiest parts of the
-driver code vanished. Configuration got radically simpler -- no
-more grovelling around for the system MDA and user's mailbox, no
-more worries about whether the underlying OS supports file
-locking.</p>
-
-<p>Also, the only way to lose mail vanished. If you specified
-localfolder and the disk got full, your mail got lost. This can't
-happen with SMTP forwarding because your SMTP listener won't return
-OK unless the message can be spooled or processed.</p>
-
-<p>Also, performance improved (though not so you'd notice it in a
-single run). Another not insignificant benefit of this change was
-that the manual page got a lot simpler.</p>
-
-<p>Later, I had to bring --mda back in order to allow handling of
-some obscure situations involving dynamic SLIP. But I found a much
-simpler way to do it.</p>
-
-<p>The moral? Don't hesitate to throw away superannuated features
-when you can do it without loss of effectiveness. I tanked a couple
-I'd added myself and have no regrets at all. As Saint-Exupery said,
-"Perfection [in design] is achieved not when there is nothing more
-to add, but rather when there is nothing more to take away." This
-program isn't perfect, but it's trying.</p>
-
-<h1>The most-requested features that I will never add, and why
-not:</h1>
-
-<h2>Password encryption in .fetchmailrc</h2>
-
-<p>The reason there's no facility to store passwords encrypted in
-the .fetchmailrc file is because this doesn't actually add
-protection.</p>
-
-<p>Anyone who's acquired the 0600 permissions needed to read your
-.fetchmailrc file will be able to run fetchmail as you anyway --
-and if it's your password they're after, they'd be able to rip the
-necessary decoder out of the fetchmail code itself to get it.</p>
-
-<p>All .fetchmailrc encryption would do is give a false sense of
-security to people who don't think very hard.</p>
-
-<h2>Truly concurrent queries to multiple hosts</h2>
-
-<p>Occasionally I get a request for this on "efficiency" grounds.
-These people aren't thinking either. True concurrency would do
-nothing to lessen fetchmail's total IP volume. The best it could
-possibly do is change the usage profile to shorten the duration of
-the active part of a poll cycle at the cost of increasing its
-demand on IP volume per unit time.</p>
-
-<p>If one could thread the protocol code so that fetchmail didn't
-block on waiting for a protocol response, but rather switched to
-trying to process another host query, one might get an efficiency
-gain (close to constant loading at the single-host level).</p>
-
-<p>Fortunately, I've only seldom seen a server that incurred
-significant wait time on an individual response. I judge the gain
-from this not worth the hideous complexity increase it would
-require in the code.</p>
-
-<h2>Multiple concurrent instances of fetchmail</h2>
-
-<p>Fetchmail locking is on a per-invoking-user because
-finer-grained locks would be really hard to implement in a portable
-way. The problem is that you don't want two fetchmails querying the
-same site for the same remote user at the same time.</p>
-
-<p>To handle this optimally, multiple fetchmails would have to
-associate a system-wide semaphore with each active pair of a remote
-user and host canonical address. A fetchmail would have to block
-until getting this semaphore at the start of a query, and release
-it at the end of a query.</p>
-
-<p>This would be way too complicated to do just for an "it might be
-nice" feature. Instead, you can run a single root fetchmail polling
-for multiple users in either single-drop or multidrop mode.</p>
-
-<p>The fundamental problem here is how an instance of fetchmail
-polling host foo can assert that it's doing so in a way visible to
-all other fetchmails. System V semaphores would be ideal for this
-purpose, but they're not portable.</p>
-
-<p>I've thought about this a lot and roughed up several designs.
-All are complicated and fragile, with a bunch of the standard
-problems (what happens if a fetchmail aborts before clearing its
-semaphore, and how do we recover reliably?).</p>
-
-<p>I'm just not satisfied that there's enough functional gain here
-to pay for the large increase in complexity that adding these
-semaphores would entail.</p>