CVS commit: pkgsrc/mail/spamdyke

[Amitai Schlair <schmonz%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   schmonz
Date:           Mon Feb 25 07:38:00 UTC 2008

Modified Files:
        pkgsrc/mail/spamdyke: Makefile PLIST distinfo options.mk
        pkgsrc/mail/spamdyke/patches: patch-aa patch-ab patch-ac

Log Message:
Update to 3.1.6. Note that some command-line options and the format
of some log entries have changed since 2.6.3; see UPGRADING.txt.
Also, pkgsrc no longer installs the random extra utilities that are
explicitly marked as unnecessary for spamdyke operation. From the
changelog:

VERSION 3.1.6 -- 2/11/2008
  Fixed a serious bug in middleman() -- when the remote server sent its message
    data and QUIT command in a burst and disconnected before spamdyke read() all
    of the data, the last data returned from read() was printed twice.  This
    could cause message corruption, especially in the case of attachments.
  Fixed a serious bug in middleman() -- when the remote server sent its data
    in bursts of 4096 bytes AND there were two lines of text in the data
    AND the 4096th character was not a newline AND there was a delay between the
    data bursts, memmove()ing the buffered data was causing corruption because
    the moved data was not being properly re-terminated.  While processing the
    remaining buffered data (and waiting for another burst from the remote
    server), strchr() would seek past the end of the data to an old newline
    character and middleman() would erroneously conclude the next line of data
    was complete, ready for processing.  Many thanks to Andreas Galatis and
    Dragomir Denev for reporting and helping me reproduce this one.
  Added a -W flag to sendrecv to introduce a delay between message data bursts.
  Added a -o flag to smtpdummy to save the message data to a file.

VERSION 3.1.5 -- 1/22/2008
  Fixed sendrecv to correctly process corrupted TLS negotiations instead of
    covering up bugs in spamdyke.
  Fixed spamdyke to not add garbage output at the beginning of TLS passthrough
    negotiations.  This was causing SSL handshakes to fail.  Thanks to Ronnie
    Tartar for reporting this one.

VERSION 3.1.4 -- 1/21/2008
  Fixed all of the Makefiles to remove a symbols directory Leopard's gcc seems
    to create when compiling in debug mode.
  Fixed middleman() to log the timeout message only once.
  Fixed middleman() to not expect input from the child process when the child
    process' input is being ignored or after the child process has exited.
  Fixed middleman() to correctly handle a rare situation -- when the child
    process was too slow responding that spamdyke's idle timeout was passed
    AND spamdyke was processing TLS data AND there was still data in the SSL
    buffer, spamdyke would loop infinitely, consuming 100% CPU.  This was a
    very tricky bug to find and fix.  Thanks to Pablo Gonzalez and Paolo for
    reporting this one and helping me debug it.
  Fixed middleman() to send message data to the child process line-by-line,
    even when the buffer is full.
  Added a new test program: smtpdummy.  This one simulates an SMTP server and
    can add delays after specific commands.
  Changed sendrecv to use a 64K buffer for input and output data.
  Changed sendrecv to kill the its child process after its timeout expires.
  Changed sendrecv to optionally continue sending data in bursts after the end
    for the message data.  Some mail servers do this.
  Changed sendrecv to deliberately send corrupt data while TLS is active.
  Changed test regression_009 to build its message payload at runtime instead
    of including a 0.75M file.  This file was unnecesarily increasing the size
    of the spamdyke tarball.
  Fixed compiling on Solaris.  Again.  Thanks to Davide Bozzelli for reporting
    this.  Again.  Sigh.

VERSION 3.1.3 -- 1/3/2008
  Fixed the format string LOG_INFO_DNS_TXT to assign the parameters correctly
    and prevent bus errors when the DNS response text is long.  Thanks to
    Stephan Rosenke for reporting this one.

VERSION 3.1.2 -- 12/11/2007
  Fixed smtp_filter() to set a flag after some SMTP commands to force
    middleman() to wait for input from the child process before proceeding.
    Some (nonspammer) mail servers send their data in bursts without waiting for
    responses.  This was causing spamdyke to skip logging (but not filtering)
    if the DATA command was sent in a burst with RCPT TO.  Thanks to Sebastien
    Guilbaud and Bucky Carr for reporting this one.
  Added a "-b" flag to sendrecv to simulate servers that send their message data
    (but not their SMTP commands) in bursts.

VERSION 3.1.1 -- 11/12/2007
  Added excessive logging to search_domain_directory() to log the directory
    search pattern.
  Changed all calls to spamdyke_log() to use the macros SPAMDYKE_LOG_NONE(),
    SPAMDYKE_LOG_ERROR(), SPAMDYKE_LOG_INFO(), SPAMDYKE_LOG_DEBUG() and
    SPAMDYKE_LOG_EXCESSIVE() instead.  The macro tests the current log level
    without forcing a function call and also paves the way toward eliminating
    some logging code at compile-time.
  Fixed process_access() to correctly search for the RELAYCLIENT variable in
    spamdyke's environment.  Thanks to Steve Cole for reporting this one.

VERSION 3.1.0 -- 11/5/2007
  Changed the "graylist-dir" and "no-graylist-dir" options to take multiple
    directories for servers that are hosting so many domains that they can't
    create enough domain folders in one place (wow).
  Added minimum and maximum values to all integer options and changed
    set_config_value() to generate error messages when values are out of range.
  Change usage() to print minimum and maximum integer values.
  Alphabetized the option list by long option name and changed
    process_config_file() to use a binary search algorithm when identifying
    directives, a theoretical improvement from O(n/2) to O(log n).
  Changed prepare_settings() to create an array of options indexed by the short
    option code.  This introduces some constant-time work (O(1)) and greater
    memory usage.
  Changed process_command_line() to use the indexed array of options,
    theoretically reducing command line parsing work from O(n/2) to O(1).
    This is a win if the command line has many parameters or if it has
    parameters that are near the end of the unindexed option array.
    Testing confirms a small performance gain.
  Added command line options "config-test-smtpauth-username" and
    "config-test-smtpauth-password".
  Changed config_test_smtpauth() to run the authentication command(s) if a
    username and password are provided.  This incorporates the functionality of
    checkpassword into spamdyke.
  Added the command line option "config-test-user" to change user and group IDs
    before running the configuration tests.  This makes it easier to simulate
    running as the mail server.
  Changed process_config_file() and process_command_line() to print errors and
    stop when they encounter an option that is not legal in that location.  At
    the moment, "help", "version", "config-test",
    "config-test-smtpauth-username", "config-test-smtpauth-password" and
    "config-test-user" are not valid in files; all options are valid on the
    command line.
  Changed config_test_dir_read() and config_test_graylist() to never examine the
    "." or ".." folders, even if readdir() and/or stat() report they are not
    folders.  Thanks to Paulo Henrique for reporting this one.
  Changed set_config_value() to remove trailing slashes from directory paths.
  Added test_spamdyke_binary() to check if the spamdyke binary is setuid root
    (it should not be).
  Renamed test_settings() to config_test().
  Moved all of the configuration test functions to config_test.[ch] -- they were
    cluttering up configuration.c.
  Made a few small updates to the help message text.
  Added additional vchkpw exit codes to exec_checkpassword() to explain why
    vchkpw exited, since it doesn't follow DJB's published checkpassword API.
  Moved md5.[ch] from the "utils" folder to the "spamdyke" folder and updated
    Makefile to compile them into spamdyke.
  Removed passwordcheck from the "utils" folder since spamdyke now contains its
    functionality.
  Added a README file to the "utils" folder to answer the biggest FAQ about
    those utilities.
  Fixed exec_command() to connect the output pipe to the child process's stdin
    instead of file descriptor 3.  The bug was due to copying
    exec_checkpassword() and forgetting to change the value.
  Renamed exec_checkpassword() to exec_checkpassword_argv() and changed its
    arguments to expect a filename and an argument array.
  Added exec_checkpassword() to parse a command string into an argument array
    and call exec_checkpassword_argv().
  Renamed exec_command() to exec_command_argv() and changed its
    arguments to expect a filename and an argument array.
  Added exec_command() to parse a command string into an argument array
    and call exec_command_argv().
  Fixed numerous bugs in exec_command_argv() that were preventing it from
    actually gathering any input from the child process.
  Changed exec_command_argv() and exec_checkpassword_argv() to always log their
    child process errors to syslog, regardless of the user's preferences.
    Otherwise, the errors will be lost.
  Added the function find_path() to search the PATH for the given command
    without executing it.
  Changed exec_command_argv() and exec_checkpassword_argv() to use find_path()
    to locate the executable before fork()ing to catch typos.  The child
    processes then use execve() to execute the command instead of exec_path().
    Otherwise, the parent has a hard time determining that the child process
    quit because the command path was invalid.
  Changed exec_command_argv() and exec_checkpassword_argv() not to wait
    indefinitely for the child to exit after the timeout expires.
  Changed dns_txt(), dns_ptr_lookup() and dns_mx() to limit the total number of
    queries they will recursively perform.  This is to prevent a DoS situation
    where some domain has an unreasonable number of chained (non-circular) CNAME
    records.  The limit is (arbitrarily) set at 16.
  Added the function config_test_child_capabilities() to test the qmail binary
    for SMTP AUTH and TLS patches.  Depending on what is found, recommendations
    for spamdyke flags are made.
  Changed check_rdns_keywords() to allow top-level domains (like .com) to be
    used as keywords.  This allows a way to reject connections from remote
    servers with rDNS names that contain the IP address and a two-letter country
    code.  Unlike check_country_code(), specific country codes can now be
    chosen.
  Fixed do_spamdyke() not to wait indefinitely for all child processes to exit.
    This behavior was causing problems with DJB's recordio because recordio
    fork()s and uses its parent process to exec() spamdyke.  This is very
    unusual.  Changing wait(NULL) to waitpid() fixes the problem.  Thanks to
    Bob Hutchinson for reporting this one.
  Added dns_initialize() and dns_get() to perform DNS queries by sending UDP
    packets instead of using the resolver library to do it.  The resolver
    functions are just too slow and they try to do too much unnecessary work.
    dns_get() performs multiple requests for records (one for each kind of
    desired record) and, if no responses are received, sends requests to the
    secondary nameservers as well.  Timeouts and retransmission times can now
    be controlled.  This has resulted in a significant speedup in DNS
    resolutions; testing shows as much as a 10x performance increase in some
    situations.
  Changed dns_txt(), dns_ptr_lookup() and dns_mx() to search all of the answers
    for the desired answer type before recursively querying CNAME answers.  Some
    nameservers always put the CNAME answers first, even if other answer types
    are also given.  This should allow spamdyke to find answers faster when
    domain admins have used a lot of CNAMEs.
  Added dns_a() to perform A record queries and changed all uses of
    gethostbyname() to use dns_a() instead.
  Changed dnsa, dnsmx, dnsns, dnsptr, dnssoa and dnstxt in the "utils" folder to
    only perform their specific queries, not ask for CNAME records as well.
  Changed dnsa, dnsmx, dnsns, dnsptr, dnssoa and dnstxt in the "utils" folder to
    send their own UDP packets instead of using the resolver library.
  Added dnscname to the "utils" folder to perform CNAME queries.
  Added dnsany to the "utils" folder to perform ANY queries and perform
    recursive CNAME lookups.
  Added "log-target" option to allow logging to stderr instead of syslog.  Some
    people apparently like using the qmail-style "multilog" instead of syslog.
    I can't understand why but I'm here to serve.  Thanks to John Hallam for
    suggesting this one.
  Changed all of the error messages about unexpected file types to specify what
    file type was found -- "non-regular file" was too vague to be useful.
  Changed the header in the files created by full logging to include the
    spamdyke version.
  Changed tls_end_inner() to use SSL_get_shutdown() to see if a shutdown signal
    has already been received.  If SSL_shutdown() is used on a closed file
    descriptor, spamdyke will crash with SIGPIPE.
  Changed all instances of read(), write(), SSL_read() and SSL_write() to read
    or write as many bytes as possible in each call.  This should provide a
    significant performance increase.  The single-byte read()s and write()s
    were only used because I had badly misunderstood the relationship between
    select() and read()/write() -- blocking only occurs when select() indicates
    a file descriptor is not ready.  If it is ready, read() and write() will
    handle as many bytes as they can without blocking.  Thanks to Trog for
    setting me straight on this one.
  Rewrote most of sendrecv in the "tests" folder to use a multi-byte read().
    Also took the opportunity to make sendrecv much faster and more polite, so
    it doesn't consume 100% CPU while waiting for qmail output.
  Fixed compiling errors on 64 bit Linux systems (Debian Etch x86_64 and Gentoo
    AMD64).  Thanks to Juha-Pekka Jarvenpaa and FireBall for reporting this.
  Added config_test_file_type() to use stat() to find a file's type if readdir()
    either doesn't report it (Solaris) or reports "unknown" for all files (XFS).
    Thanks to Paulo Henrique for reporting this one.
  Fixed compiling errors on Solaris.  Thanks to Limperis Antonis for reporting
    this.
  Changed the logging severity of the "unable to write X bytes to file
    descriptor" to debug instead of error.  99% of the time, the error occurs
    because the remote client disconnected unexpectedly and there's nothing
    the administrator can do about it anyway.
  Changed do_spamdyke() to ignore SIGPIPE signals.
  Changed do_spamdyke(), exec_command_argv() and exec_command_checkpassword()
    to change the SIGPIPE signal handler back to default for child processes
    after fork()ing but before exec()ing.
  Added a new logging level: excessive (4).  It's to be used for printing very
    detailed debugging statements.
  Changed process_access() to permit access when no matching lines are found in
    the access file.  Although DJB's tcprules documentation doesn't explicitly
    say so, no matching lines should allow access.  Thanks to Steve Cole for
    reporting this one.

VERSION 3.0.1 -- 9/12/2007
  Fixed "configure" to remove the "_beta1" tag from the version number.  That
    should never have been published.
  Changed usage() to show that optional values to long commands must be
    separated by an equals sign.  getopt_long() is really becoming a hassle.
    Thanks to Richard Kreider for reporting this one.
  Fixed find_address() to accept addresses that aren't correctly delimited with
    <> characters and/or have multiple (illegal) spaces after the colon.  Thanks
    to Davide Bozzelli for reporting this one.
  Fixed prepare_settings() to set the idle timeout seconds to the correct
    variable instead of setting the connection timeout variable.  Thanks to
    Carlo Blohm for reporting this one.
  Fixed smtp_filter() to print the rejection message to HELO and EHLO, even if
    those commands appear in an improper place in the protocol.
  Fixed smtp_filter() to print the rejection message with an error code in
    response to STARTTLS if the command is given in an improper place in the
    protocol.
  Added some regression tests to find these bugs in the future.
  Fixed the usage statement in sendrecv to show the -w flag.

VERSION 3.0.0 -- 9/11/2007
  Added command line options never-graylist-rdns-dir, always-graylist-rdns-dir
    and rdns-whitelist-dir to search domain directory structures just like
    rdns-blacklist-dir.
  Added the command line option rdns-blacklist-file to search a file just like
    rdns-whitelist-file.
  Moved the command line option labels into configuration.c so they can be
    shared with the config file parser.
  Changed process_command_line() to build the list of short options from the
    list of long options instead of hardcoding them.  Less maintenance this way.
  Modified check_rdns_keywords(), search_file() and search_tcprules_file() to
    correctly track line numbers and return the matching line number instead of
    just 1.
  Changed logging to allow the amount of information to be turned up or down.
    This should make spamdyke less chatty in the syslog for small errors.
  Modified smtp_filter() and run_tests() to report the matching filename and
    line number from check_rdns_keywords(), search_file() and
    search_tcprules_file() in syslog if the logging level is high enough.
  Fixed find_address() to locate the real email address and ignore BATV tags,
    relay paths and bang paths.  Thanks to Walter Russo for reporting this one
    (again).
  Changed middleman() to obey minimums and maximums for the amount of time to
    select() for traffic.  If spamdyke waits too long, the qmail process might
    not get wait()ed for a while, leaving a lot of defunct/zombie processes
    around.  On a busy server, this could be a problem.  Thanks to Jason M for
    reporting this one.
  Added process_config_file() to process configuration files instead of
    requiring all configuration to be done on the command line.  At the moment,
    the file just uses the same (long option) directives as the command line.
  Added test_settings() to run tests on every configuration option and
    (hopefully) identify misconfigurations before someone makes them on a live
    server.
  Added the command line option "config-test" to run test_settings().
  Renamed log_writeln() and log_write_rejection() to output_writeln() and
    output_write_rejection(), respectively, to make it clearer what they're
    doing.
  Changed smtp_filter() to allow multiple authentication attempts.  Some
    clients retry authentication several times, presumably to deal with servers
    that can't use the authentication method they prefer.
  Changed middleman() to collect (and send) whole lines of input instead of
    single characters.  Single character write()s were causing problems with
    Nagios and Windows clients.
  Changed output_write_rejection() to create a single output line and send it
    to output_writeln() all at once instead of sending a piece at a time.  This
    keeps packets together for stupid Windows clients that just can't handle
    reassembling TCP packets correctly.
  Changed main() to always run spamdyke (as opposed to starting qmail without
    spamdyke listening) even if a whitelist is matched.  This way, spamdyke
    can report all traffic to syslog, not just traffic that _may_ be filtered.
  Changed smtp_filter() and middleman() to catch the return codes from qmail
    when the remote client gives the recipient address.  Now, if spamdyke
    doesn't block the recipient command but qmail does (e.g. for relaying),
    spamdyke will log the correct message.
  Incorporated GNU autoconf to create a "configure" script for spamdyke and the
    "utils" folder.  The days of "make no_tls" and "make bsd" are thankfully
    over.
  Renamed all of the test folders to group them by function so it's easier to
    see what tests exist.  Sequential numbers just weren't working.
  Changed dns_mx() to lookup the MX record before returning success.  This means
    the sender MX filter now requires a mail exchanger record _and_ at least one
    mail exchanger must have an IP address.  Before, the MX record was enough,
    even if there was no corresponding A record.
  Changed usage() to read the options and help text from get_spamdyke_options()
    in configuration.c so the help message won't ever be out of sync with the
    available options again.
  Added the command line option "tls-privatekey-password-file" to allow the SSL
    private key password to be read from a file instead of the command line.
    This way, the password isn't visible to everyone who can view a process
    list.
  Changed search_file(), search_tcprules_file() and check_rdns_keywords() so
    they no longer build their fscanf() patterns into a stack variable but
    instead use a literal search pattern assembled at compile time with
    STRINGIFY().
  Added the command line options "hostname-file" and "hostname-command" to
    support reading the local hostname from a file or from a command (e.g.
    "hostname -f") instead of forcing it to be specified on the command line.
  Changed middleman() and smtp_filter() to always monitor and trust
    authentication carried out by qmail, even if "smtp-auth-command" was not
    given.  This means spamdyke will always disable its filters for
    authenticated users even if it can't check the authentication itself.
    I'm not sure why I didn't design spamdyke this way in the first place.
  Added command line options recipient-whitelist-file and sender-whitelist-file
    so specific sender and recipient addresses can bypass the filters.  Sender
    addresses are very easy to fake and recipient addresses are, of course,
    known to spammers, so both of these options are ill-advised.  I've only
    added them due to popular demand.
  Added command line option check-rhsbl to check righthand-side blacklists.
    Both the server's rDNS domain name and the sender's email domain name are
    checked.
  Added command line options check-dns-whitelist and check-rhs-whitelist to
    allow DNS RBLs and RHSBLs to act as whitelists instead of blacklists.
    Anyone using DNS-based blacklists _and_ whitelists had better have some
    seriously fast DNS servers.
  Changed dns_txt(), dns_mx() and dns_ptr_lookup() to pass a stack of previous
    queries whenever they recursively lookup CNAME records, to prevent a cylical
    CNAME structure from leading to infinite recursion.
  NOT BACKWARDS COMPATIBLE: Changed the syslog entry format: renamed "origin" to
    "origin_ip", added "origin_rdns:" before the rDNS name, added "auth:"
    before the authenticated username and added "reason:" before the rejection
    reason when a timeout occurs.
  Changed process_command_line() to assume the remote IP address is 0.0.0.0 if
    the environment variable TCPREMOTEIP is not set.
  Added a ton more test scripts for all of the new options and for testing
    config files.
  Added dnsa, dnsns and dnssoa to the "utils" folder for performing DNS queries
    of A, NS and SOA records, respectively.  Wouldn't it be AMAZING if the
    libc maintainers added standard functions to do these queries?!
  NOT BACKWARDS COMPATIBLE: Changed the "flag" options to take optional
    arguments instead of simply assuming "true" when the option was given.
    Unfortunately, getopt_long() is too stupid to handle them properly, which
    means clustered options (e.g. -rRc) can no longer be used.  They must be
    separated (e.g. -r -R -c).  Also, arguments given with the short version
    must not be separated by a space (e.g. -l3).
  NOT BACKWARDS COMPATIBLE: Renamed the long command line option "use-syslog"
    to "log-level".
  Fixed middleman() to completely bypass all processing when TLS passthrough is
    active.  The additional processing was buffering TLS traffic until the data
    contained a newline character (purely by coincidence).  This buffering was
    preventing the passthrough from functioning properly.  Thanks to Dominik
    Dausch for reporting this one.


To generate a diff of this commit:
cvs rdiff -r1.9 -r1.10 pkgsrc/mail/spamdyke/Makefile \
    pkgsrc/mail/spamdyke/distinfo
cvs rdiff -r1.3 -r1.4 pkgsrc/mail/spamdyke/PLIST
cvs rdiff -r1.1 -r1.2 pkgsrc/mail/spamdyke/options.mk
cvs rdiff -r1.6 -r1.7 pkgsrc/mail/spamdyke/patches/patch-aa
cvs rdiff -r1.5 -r1.6 pkgsrc/mail/spamdyke/patches/patch-ab \
    pkgsrc/mail/spamdyke/patches/patch-ac

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.