Gentoo mailfiltering gateway guide

Content:
1. Introduction

2. Installing the programs needed

3. Every good rule has good exceptions as well

4. Adding more rules

5. Testing and finishing up

6. Autolearning and sidelining emails

7. Greylisting

8. SPF (Sender Policy Framework)

9. Configuring amavisd-new to use MySQL

10. Configuring Spamassassin to use MySQL

11. Troubleshooting

12. Resources

1. Introduction

This guide describe step by step how to install a spam and virus filtering mail gateway. It is quite simple to adopt this to a single server solution.

The big picture

This document describe how to setup a spam filtering mail gateway with multiple domains. This server is meant to run in front of the mail servers actually keeping the mail accounts i.e. Microsoft Exchange or Lotus Notes.

In this setup applications with good security records and readable configuration files have been chosen. The email MTA is postfix which has a good security record and is fairly easy to setup right. Postfix will listen normally on port 25 for incoming mail. Upon reception it will forward it to Amavisd-new on port 10024. Amavisd-new will then filter the mail through different filters before passing the mail back to Postfix on port 10025 which in turn will forward the mail to the next mail server.

Amavisd-new is a content filtering framework utilizing helper applications for virus filtering and spam filtering. In this setup we will be using two helper applications one ClamAV for filtering virus mails and Spamassassin for filtering spam. Spamassassin itself can function as yet another layer of content filtering framework and utilize the helper applications Vipul's Razor2 and DCC.

Unlike many other spam fighting technologies like RBLs and others Spamassassin does not simply accept or reject a given email based on one single test. It uses a lot of internal tests and external helper applications to calculate a spam score for every mail passed through. This score is based on the following tests:

Bayesian filtering

Static rules based on regular expressions

Distributed and collaborative networks:

RBLs

Razor2

Pyzor

DCC

The first part (chapters 1 to 4) of the guide will describe the basic setup of a mailfiltering gateway. The next chapters can be implemented individually with no dependence between each chapter. These chapters describe how to:

setup special IMAP folders for learning of the Bayesian filter and for delivery of false positives

setup greylisting with Postfix

setup Amavisd-new to use a MySQL backend for user preferences

setup Spamassassin to use a MySQL backend for AWL and Bayes data

Note: The IMAP folders will be using the maildir format. Having each mail in a separate file makes handling much simpler. If you're using mbox I propose to give maildir a try. If you're not already using maildir emerge the necessary tools with emerge courier-imap.

A planned fifth part will contain various tips regarding performance and things you may want to know (running chrooted, postfix restrictions, etc.).

Note: Delegating responsibility to third parties is not without risks. You have to know and trust these third parties. In this setup only the decision to quarantine virus mails are based on a single third party. Using Spamassassin's scoring system the decision to stop spam mails are not made by a single authority except perhaps Spamassassins own static rules.

Warning: When rejecting spam mails at the MTA level you have to be very careful when selecting the RBL's you want to use, i.e. SpamCop is a bad RBL to use at the MTA level because you will experience false positives because sometimes their listing is just too aggressive. Further info at Realtime Blackhole Lists Are Bad and The Spam Problem: Moving Beyond RBLs

Preparations

Before you start make sure that you have a working Postfix installation where you can send and receive mails also you need a backend mailserver. If you're not experienced with setting up Postfix it might quickly become too complicated if all should be set up at once. If you need help you can find it in the excellent Virtual Mailhosting System with Postfix Guide in the Gentoo Documentation.

2. Installing the programs needed

We start out by installing the most important programs: Amavisd-new, Spamassassin and ClamAV.

Code Listing 2.1: Installing Amavis, Spamassassin and Clamav

# emerge amavisd-new spamassassin clamav

Note: As previously mentioned you should already have a working postfix instance running on the box. Basically this shouldn't be much more than emerge postfix and have a basic understanding of how Postfix is working.

Setting up DNS

Note: If you're not setting up a gateway server but have the mailboxes on the same server you only have to create the MX-Record.

While the programs are emerging fire up another shell and create the needed DNS records.

Start out by creating a MX record for the mail gateway and an A record for the next destination.

Code Listing 2.2: Setting up DNS

(Create a MX record for the gateway server)
MX 10 mailgateway.mydomain.tld.
(Create an A record for the gateway server)
mailgateway A mgw.ip.add.here
(Create an A record for the next hop mail server)
mail A ms.ip.add.here

Note: Some ADSL providers might block port 25 and force you to relay mail through one of their servers. Typically you have to create a secondary MX-Record like MX 20 backup-mx.some-isp.tld

Opening the firewall

In addition to allowing normal mail traffic you have to allow a few services through your firewall to allow the network checks to communicate with the servers.

Application | Protocol | Port
DCC | UDP | 6277
Razor(outgoing ping) | TCP | 7
Razor | TCP | 2703

Razor uses pings to discover what servers are closest to it.

Configuring Postfix

First we have to tell postfix to listen on port 10025 and we remove most of the restrictions as they have already been applied by the postfix instance listening on port 25. Also we ensure that it will only listen for local connections on port 10025. To accomplish this we have to add the following at the end of /etc/postfix/master.cf

Code Listing 2.3: Changing the master.cf file

smtp-amavis unix - - n - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
#Equivalently when using lmtp:
#lmtp-amavis unix - - n - 2 lmtp
# -o lmtp_data_done_timeout=1200
# -o lmtp_send_xforward_command=yes

127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000

#If you want to use proxy filtering instead
#smtp inet n - n - 8 smtpd
# -o smtpd_proxy_filter=127.0.0.1:10024
# -o smtpd_client_connection_count_limit=4
#If you don't want to scan outgoing mail use this
#10.0.0.2:smtp inet n - n - - smtpd
#-o content_filter=

Note: The smtp-amavis line specifies that a maximum of two of these processes may run at any time. If you need a greater degree of concurrency tune this number to fit your needs. Remember that to match the number with $max_servers in amavisd.conf. Keep in mind that amavisd-new is quite memory-intensive and raising the amount of amavisd-new processes too high can easily lead to memory starvation and heavy swapping, which leads to drastically reduced performance.

Note: If you want to reject spam early on in the process you can use the Before-Queue (proxy) method instead of the filter method. If you uncomment the three lines you will have to set content_filter= in main.cf. This is not recommended for high traffic servers as the number of concurrent connections are limited to the number of amavisd instances.

Warning: The Before-Queue(proxy) method is still not properly tested.

Note: If you, for any reason whatsoever, want to send mail from this box and don't want it scanned, add another postfix instance by uncommenting the last two lines and substitute with a proper IP.

The file master.cf tells the postfix master program how to run each individual postfix process. More info with man 8 master.

Next we need the main postfix instance listening on port 25 to filter the mail through amavisd-new listening on port 10024.

We also need to set the next hop destination for mail. Tell Postfix to filter all mail through an external content filter and enable explicit routing to let Postfix know where to forward the mail to.

Code Listing 2.4: Modifying /etc/postfix/main.cf

biff = no
empty_address_recipient = MAILER-DAEMON
queue_minfree = 120000000

content_filter = smtp-amavis:[127.0.0.1]:10024
#Equivalently when using lmtp:
#content_filter = lmtp-amavis:[127.0.0.1]:10024

# TRANSPORT MAP
#
# Insert text from sample-transport.cf if you need explicit routing.
transport_maps = hash:/etc/postfix/transport

relay_domains = $transport_maps

Postfix has a lot of options set in main.cf. For further explanation of the file please consult man 5 postconf or the same online Postfix Configuration Parameters.

The format of the transport file is the normal Postfix hash file. Mail to the domain on the left hand side is forwarded to the destination on the right hand side.

Code Listing 2.5: /etc/postfix/transport

mydomain.tld smtp:mail.mydomain.tld

After we have edited the file we need to run the postmap command. Postfix does not actually read this file so we have to convert it to the proper format with postmap /etc/postfix/transport. This creates the file /etc/postfix/transport.db. There is no need to reload Postfix as it will automatically pick up the changes.

Note: If the next hop mail server is not listening on the standard SMTP port 25 you can tell postfix to use a given port number, like smtp:mail.mydomain.tld:25000.

If your first attempts to send mail result in messages bouncing, you've likely made a configuration error somewhere. Try temporarily enabling soft_bounce while you work out your configuration issues. This prevents postfix from bouncing mails on delivery errors by treating them as temporary errors. It keeps mails in the mail queue until soft_bounce is disabled or removed.

Code Listing 2.6: Enabling soft_bounce

# postconf -e "soft_bounce = yes"
# /etc/init.d/postfix reload

Once you've finished creating a working configuration, be sure to disable or remove soft_bounce and reload postfix.

Configuring Amavisd-new

Amavisd-new is used to handle all the filtering and allows you to easily glue together severel different technologies. Upon reception of a mail message it will extract the mail, filter it through some custom filters, handle white and black listing, filter the mail through various virus scanners and finally it will filter the mail using SpamAssassin.

Amavisd-new itself has a number of extra features:

it identifies dangerous file attachments and has policies to handle them

per-user, per-domain and system-wide policies for:

whitelists

blacklists

spam score thresholds

virus and spam policies

Apart from postfix and freshclam we will run all applications as the user amavis.

Edit the following lines in /etc/amavisd.conf

Code Listing 2.7: Editing /etc/amavisd.conf

(Insert the domains to be scanned)
$mydomain = 'example.com';
(Bind only to loopback interface)
$inet_socket_bind = '127.0.0.1';
(Forward to Postfix on port 10025)
$forward_method = 'smtp:127.0.0.1:10025';
$notify_method = $forward_method;
(Define the account to send virus alert emails)
$virus_admin = "virusalert\@$mydomain";
(Always add spam headers)
$sa_tag_level_deflt = -100;
(Add spam detected header aka X-Spam-Status: Yes)
$sa_tag2_level_deflt = 5;
(Trigger evasive action at this spam level)
$sa_kill_level_deflt = $sa_tag2_level_deflt;
(Do not send delivery status notification to sender. It does not affect
delivery of spam to recipient. To do that, use the kill_level)
$sa_dsn_cutoff_level = 10;
Don't bounce messages left and right, quarantine
instead
$final_virus_destiny = D_DISCARD; # (defaults to D_DISCARD)
$final_banned_destiny = D_DISCARD; # (defaults to D_BOUNCE)
$final_spam_destiny = D_DISCARD; # (defaults to D_BOUNCE)

Note: With this line $sa_tag2_level_deflt = 5; you set the Spamassassin spam score to 5. This might be a bit low. As you might have noticed the Amavisd-new default is 6.3. If you don't want to see a single spam mail in your mail folder choose 5, but if you don't want to deal with false positives choose 6.3.

Create a quarantine directory for the virus mails as we don't want these delivered to our users.

Code Listing 2.8: Create a quarantine directory for the virus mails

# mkdir /var/amavis/virusmails
# chown amavis:amavis /var/amavis/virusmails
# chmod 750 /var/amavis/virusmails

Note: Amavisd-new offers finer policy tuning by using policy banks.

Configuring ClamAV

As virus scanner we use ClamAV as it has a fine detection rate comparable with commercial offerings, it is very fast and it is Open Source Software. We love log files, so make clamd log using syslog and turn on verbose logging. Also do not run clamd as root. Now edit /etc/clamd.conf

Code Listing 2.9: Edit /etc/clamd.conf

(Verbose logging with syslog)
LogSyslog
LogVerbose
LogFacility LOG_MAIL
(Change pid file location)
PidFile /var/run/amavis/clamd.pid
(Set the clamav socket)
LocalSocket /var/amavis/clamd
(Close the connection when this limit is exceeded)
StreamMaxLength 10M
(Don't run clamd as root)
User amavis
(Newer versions require you to uncomment this)
ScanMail
ScanArchive

Note: Also remember to remove the Example directive to make ClamAV work

ClamAV comes with the freshclam deamon dedicated to periodical checks of virus signature updates. Instead of updating virus signatures twice a day we will make freshclam update virus signatures every two hours.

Code Listing 2.10: Edit /etc/freshclam.conf

(Syslog logging)
LogSyslog
(Verbose logging)
LogVerbose
(Explicitly drop root privileges)
DatabaseOwner clamav
(Check for updates every two hours. That is the official recommendation)
Checks 12
(Use the mirror closest to you. Replace XY with your country code
DatabaseMirror db.XY.clamav.net

Start clamd with freshclam using the init scripts by modifying /etc/conf.d/clamd.

Code Listing 2.11: Modifying /etc/conf.d/clamd

START_CLAMD=yes
FRESHCLAM_OPTS="-d"

At last modify amavisd.conf with the new location of the socket.

Code Listing 2.12: Modifying /etc/amavisd.conf

(Uncomment the clamav scanner and modify socket location)
['ClamAV-clamd',
\&ask_daemon, ["CONTSCAN {}\n", "/var/amavis/clamd"],
qr/\bOK$/, qr/\bFOUND$/,
qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],

Warning: Do NOT modify the $unix_socketname unless you know what you're doing.

Configuring Vipul's Razor

Razor2 is a collaborative and distributed spam checksum network. Install it with emerge razor and create the needed configuration files. Do this as user amavis by running su - amavis followed razor-admin -create.

Code Listing 2.13: Creating the required configuration files

# emerge razor

(Temporarily set amavis' shell to bash)
# usermod -s /bin/bash amavis
# su - amavis
$ razor-admin -create
$ exit

(Reset the shell to /bin/false)
# usermod -s /bin/false amavis

Configuring Distributed Checksum Clearinghouse (dcc)

Like Razor2, dcc is a collaborative and distributed spam checksum network. Its philosopy is to count the number of recipients of a given mail identifying each mail with a fuzzy checksum.

Code Listing 2.14: Installing DCC

# emerge dcc

Configuring Spamassassin

Amavis is using the Spamassassin Perl libraries directly so there is no need to start the service. Also this creates some confusion about the configuration as some Spamassassin settings are configured in /etc/mail/spamassassin/local.cf and overridden by options in /etc/amavisd.conf.

Code Listing 2.15: Create /etc/mail/spamassassin/local.cf

# Enable the Bayes system
use_bayes 1

# Enable all network checks
skip_rbl_checks 0

# Mail using languages used in these country codes will not be marked
# as being possibly spam in a foreign language.
# - danish english norwegian swedish
ok_languages da en no sv

# Mail using locales used in these country codes will not be marked
# as being possibly spam in a foreign language.
ok_locales en

# Use a sensible bayes path
bayes_path /var/amavis/.spamassassin/bayes

Note: With Spamassassin version 3.1 you have to enable DCC, Razor2 by uncommenting the corresponding lines in v310.pre.

Note: You can find inspiration for your local.cf file by trying the SpamAssassin Configuration Generator.

Note: You might also want to switch the ok_languages and ok_locales.

3. Every good rule has good exceptions as well

Once mail really starts passing through this mail gateway you will probably discover that the above setup is not perfect. Maybe some of your customers like to receive mails that others wouldn't. You can whitelist/blacklist envelope senders quite easily. Uncomment the following line in amavisd.conf.

Code Listing 3.1: Modifying amavisd.conf to do sitewide scoring

read_hash("/var/amavis/sender_scores_sitewide"),

In the sender_scores_sitewide file you put complete email addresses or just the domian parts and then note a positive/negative score to add to the spam score.

Code Listing 3.2: whitelist_sender example

(Whitelist all emails from the specific email address)
[email-masked] -3.0
(Whitelist all emails from the example.net excluding subdomains)
.example.net 1.0

Note: See /etc/amavisd.conf for more examples.

Note: Placing these addresses outside amavisd.conf is a cleaner and safer solution.

Note: Alternatively it can be done in Spamassassin's configuration file /etc/mail/spamassassin/local.cf but I think it is cleaner to do it in /etc/amavisd.conf.

Note: In a later chapter I will show how to implement per-user policies using MySQL.

While waiting for a better method you can add the following to amavisd.conf to bypass spam checks for postmaster and abuse mailboxes.

Code Listing 3.3: By pass spam filters for all postmaster and abuse mails

map { $bypass_spam_checks{lc($_)}=1 } (qw(
postmaster@
abuse@
));

Important: While we are at it we should never automatically discard mails to the postmaster or the abuse accounts. See RFC 2142 MAILBOX NAMES FOR COMMON SERVICES, ROLES AND FUNCTIONS. Otherwise your domains might end up listed in some of the evil lists over at rfc-ignorant.org.

4. Adding more rules

If you want to use more rules provided by the SARE Ninjas over at the SpamAssassin Rules Emporium you can easily add and update them using the sa-update mechanism included in Spamassassin.

A brief guide to using SARE rulesets with sa-update can be found here.

5. Testing and finishing up

Testing the setup

Now before you start freshclam you can manually verify that it works.

Code Listing 5.1: Testing freshclam

# freshclam
ClamAV update process started at Sun May 2 09:13:41 2004
Reading CVD header (main.cvd): OK
Downloading main.cvd [*]
main.cvd updated (version: 22, sigs: 20229, f-level: 1, builder: tkojm)
Reading CVD header (daily.cvd): OK
Downloading daily.cvd [*]
daily.cvd updated (version: 298, sigs: 1141, f-level: 2, builder: diego)
Database updated (21370 signatures) from database.clamav.net (193.1.219.100).

Now you have updated virus definitions and you know that freshclam.conf is working properly.

Test freshclam and amavisd from the cli and amavisd testmails. Start clamd and amavis with the following commands:

Code Listing 5.2: Start amavisd and clamd and reload postfix configuration

# /etc/init.d/clamd start
# /etc/init.d/amavisd start
# /etc/init.d/postfix reload

If everything went well postfix should now be listening for mails on port 25 and for reinjected mails on port 10024. To verify this check your log file.

Code Listing 5.3: Checking log files

# tail -f /var/log/mail.log

Note: Depending on your log settings the correct path might be /var/log/messages.

Now if no strange messages appear in the log file it is time for a new test.

Use netcat to manually connect to amavisd on port 10024 and postfix on port 10025.

Note: Netcat can be used as an advanced replacement for telnet. Install it with emerge netcat.

Note: For some unknown reason you can not complete a manual mail injection to amavisd with netcat. Use telnet instead.

Code Listing 5.4: Manually checking that amavisd and postfix are listning to the new ports

# nc localhost 10024
(Amavis working)
220 [127.0.0.1] ESMTP amavisd-new service ready
nc localhost 10025
(Postfix reinject working)
220 example.com ESMTP Postfix

Note: If you want to see the complete output from amavisd-new start amavisd debug-sa as the amavis user and send a mail. For this to work you might have to change the default shell in /etc/passwd.

Add amavisd and clamd to the default runlevel.

Code Listing 5.5: Add amavisd and clamd to the default runlevel

# rc-update add clamd default
# rc-update add amavisd default

Note: We do not add spamd to the default runlevel as amavisd uses the Spamassassin Perl libraries directly.

Note: You might notice Net::Server: Couldn't POSIX::setuid to ... [] lines in your log. According to amavis chroot README, if the process UID remains 0 (root), the program will terminate, otherwise you can consider the message just as informative. This is because POSIX::setuid() returns a string 0 but true.

Important: If you enabled login for amavis remember to set back the login shell in /etc/passwd to /bin/false.

6. Autolearning and sidelining emails

Creating the spamtrap user

Create the spamtrap account and directories.

Code Listing 6.1: Create spamtrap account

# useradd -m spamtrap
# maildirmake /home/spamtrap/.maildir
# chown -R spamtrap:mailusers /home/spamtrap/.maildir
(Give the spamtrap user a sensible password)
# passwd spamtrap

If you manually want to check some of the mails to ensure that you have no false positives you can use the following procmail recipe to sideline spam found into different mail folders.

Creating .procmailrc

Code Listing 6.2: Creating /home/spamtrap/.procmailrc

#Set some default variables
MAILDIR=$HOME/.maildir

SPAM_FOLDER=$MAILDIR/.spam-found/

LIKELY_SPAM_FOLDER=$MAILDIR/.likely-spam-found/

#Sort mails with a spamscore of 7+ to the spamfolder
:0:
* ^X-Spam-Status: Yes
* ^X-Spam-Level: \*\*\*\*\*\*\*
$SPAM_FOLDER

#Sort mail with a spamscore between 5-7 to the likely spam folder
:0:
* ^X-Spam-Status: Yes
$LIKELY_SPAM_FOLDER

#Sort all other mails to the inbox
:0
*
./

Warning: If your mail server is going to receive a lot of mail you should NOT use the likely-spam recipe. Instead set $sa_tag2_level_deflt high enough to avoid false positives and filter it directly to $SPAM_FOLDER.

Note: If you haven't already installed procmail do it with emerge procmail.

Now make sure that Postfix uses procmail to deliver mail.

Code Listing 6.3: Modifying /etc/postfix/main.cf

mailbox_command = /usr/bin/procmail -a "DOMAIN"

Create mailfolders

Now we will create shared folders for ham and spam.

Code Listing 6.4: Create the necessary mailfolders

# maildirmake /var/amavis/.maildir
# maildirmake -S /var/amavis/.maildir/Bayes
# maildirmake -s write -f spam /var/amavis/.maildir/Bayes
# maildirmake -s write -f ham /var/amavis/.maildir/Bayes
# maildirmake -s write -f redeliver /var/amavis/.maildir/Bayes

Amavisd-new needs to be able to read these files as well as all mailusers. Therefore we add all the relevant users to the mailuser group along with amavis.

Code Listing 6.5: Setting the proper permissions

# groupadd mailusers
# usermod -G mailusers spamtrap
# chown -R amavis:mailusers /var/amavis/.maildir/
# chown amavis:mailusers /var/amavis/
# chmod -R 1733 /var/amavis/.maildir/Bayes/
# chmod g+rx /var/amavis/.maildir/
# chmod g+rx /var/amavis/.maildir/Bayes/

Warning: This grants members of the mailusers groups access to amavis mail.

This makes the spam and ham folders writable but not readable. This way users can safely submit their ham without anyone else being able to read it.

Then run the following command as the spamtrap user:

Code Listing 6.6: Adding the shared folders to the users mailfolder

$ maildirmake --add Bayes=/var/amavis/.maildir/Bayes $HOME/.maildir

Note: We have to give the group read permissions on the Bayes folder in order for the mail client to be able to see the subdirectories used by IMAP.

Adding cron jobs

Now run crontab -u amavis -e to edit the amavis crontab to enable automatic learning of the Bayes filter every hour.

Code Listing 6.7: amavis crontab

#Auto learn
0 * * * * /usr/bin/sa-learn --spam /var/amavis/.maildir/Bayes/.spam/{cur,new} \
> /dev/null 2>&1
0 * * * * /usr/bin/sa-learn --ham /var/amavis/.maildir/Bayes/.ham/{cur,new} > \
/dev/null 2>&1

Note: amavis has to be a member of the cron group to run crons.

Note: It seems like the shared maildir folders will make sa-learn examine all messages twice. This should not be a problem. The output will also show that the maximum of messages learned from is half or less than the messages examined.

Modifying amavisd.conf

Now modify amavis to redirect spam emails to the spamtrap account and keep spamheaders.

Code Listing 6.8: Modifying /etc/amavisd.conf

(Define the account to send virus spam emails)
$spam_quarantine_to = "spamtrap\@$myhostname";

Redelivering false positives

If you set the spam score very low like we do you will probably have some false positives. These are filtered into the folder likely-spam. These are manually reviewed and any false positive is moved to the redeliver mailfolder. From there it is first fed through sa-learn --ham and then redelivered with all headers intact using a patched version of smtpclient by Ralf S. Engelschall.

The original version 1.0.0 of smtpclient is already in Portage however to keep all headers intact we use a modified version. Here is how you install the revised version:

Code Listing 6.9: Installing revised smtpclient

(Switch to your portage OVERLAY)
# wget http://home.coming.dk/files/smtpclient.tar.gz
# tar xzf smtpclient.tar.gz
# echo "mail-client/smtpclient ~x86" >> /etc/portage/package.keywords
# echo "mail-client/smtpclient fullheaders" >> /etc/portage/package.use
# emerge smtpclient

Check for mails in the redeliver folder every minute using cron.

Code Listing 6.10: amavis crontab

#Redeliver false positives
* * * * * find /var/amavis/.maildir/Bayes/.redeliver/cur/ -type f -name \
"[0-9]*" -exec cp {} /var/amavis/.maildir/Bayes/.ham/cur/ \; \
&& find /var/amavis/.maildir/Bayes/.redeliver/cur/ -type f \
-name "[0-9]*" -exec /usr/local/bin/redeliver.pl {} \;

Now we only have to copy the redeliver.pl file to /usr/local/bin/. Download it or use the version below.

Code Listing 6.11: redeliver.pl

#!/usr/bin/perl -w

# Redelivers mail using a modified version of smtpclient
# By: Jens Hilligsoe <[email-masked]>

use strict;

if(!($#ARGV == 0)) {
die "Usage:\n$0 maildir_mail\n";
}

my $mail = $ARGV[0];
my $to = "";
my $from = "";

sub prunefile ( $ );

# Retrieve To and From envelope adresses
open (MAIL, $mail) or die "Could not open $mail: $?\n";
while(<MAIL>) {
if(($to eq "") || ($from eq "")) {
chop;
(my $key, my $value) = split (/:/);
if($key eq "X-Envelope-To") {
$to = $value;
$to =~ s/[\<\>,]//g; # Remove "<", ">" and ","
$to =~ s/^\s+|\s+$//g; #Remove whitespace before and after
}
if($key eq "X-Envelope-From") {
$from = $value;
$from =~ s/[\<\>,]//g;
$from =~ s/^\s+|\s+$//g;
if($from eq "") {
$from = "postmaster";
}
}
}
}

if($to eq "") {
prunefile($ARGV[0]); # Just nuke it if to is empty
} else {
my $redelivercmd = "cat $ARGV[0] | smtpclient -F -S 127.0.0.1 -P 10025 -f $from $to";
unless (system($redelivercmd) == 0 ) {
die "Unable to redeliver: $?";
}
prunefile($ARGV[0]); # Clean up
}

sub prunefile ( $ ) {
my ($file) = @_;
unless (unlink $file) {
die "Unable to remove mail: $?";
}
}

Cleaning up

We don't want to keep mail forever so we use tmpwatch to clean up regularily. Emerge it with emerge tmpwatch. Only root is able to run tmpwatch so we have to edit the root crontab.

Code Listing 6.12: Modifying root crontab

# Clean up
# Keep virusmails for a week (24*7 hours)
15 0 * * * /usr/sbin/tmpwatch -c -f -d --quiet 168 /var/amavis/virusmails/
# Delete spam and ham learned after a week
15 0 * * * /usr/sbin/tmpwatch -c -f -d --quiet 168 /var/amavis/.maildir/Bayes/

7. Greylisting

Introduction

Greylisting is one of the newer weapons in the spam fighting arsenal. As the name implies it is much like whitelisting and blacklisting. Each time an unknown mailserver tries to deliver mail the mail is rejected with a try again later message. This means that mail gets delayed but also that stupid spam bots that do not implement the RFC protocol will drop the attempt to deliver the spam and never retry. With time spam bots will probably adjust, however it will give other technologies more time to identify the spam.

Note: If your ISP blocks incoming traffic on port 25 and relays all mail to you through their own mail server greylisting will not work.

Postfix 2.1 come with a simple Perl greylisting policy server that implements such a scheme. However it suffers from unpredictable results when the partition holding the greylisting database run out of space. There exists an improved version that do not suffer this problem. First I will show how to install the builtin greylisting support that come with Postfix and then I will show how to configure the more robust replacement.

Note: There are other greylisting policy servers for Postfix around (such as Gld, which is in Portage, and SQLgrey). Some of them support database backends, auto whitelisting and other neat features.

Simple greylisting

Note: If you prefer to use the improved greylisting with postgrey you can safely skip this section.

We need the file greylist.pl but unfortunately the ebuild does not install it as default.

Code Listing 7.1: Getting greylist.pl

# cp /usr/portage/distfiles/postfix-your-version-here.tar.gz /root/
# tar xzf postfix-your-version-here.tar.gz
# cp postfix-2.1.0/examples/smtpd-policy/greylist.pl /usr/bin/

Now we have the file in place we need to create the directory to hold the greylisting database:

Code Listing 7.2: Creating directory for the greylisting database

# mkdir /var/mta
# chown nobody /var/mta

Warning: Do not create the greylisting database directory on a partition that might run out of space. While postfix can recover from no-space-left situations for the mail queue and mail box situations, this is not the case with the greylisting database. If the file becomes corrupted you may not be able to receive mail at all until you delete the file by hand.

Configuring greylisting

Now that we have all this ready all that is left is to add it to the postfix configuration. First we add the necessary information to the master.cf:

Code Listing 7.3: Modifying master.cf to use greylisting

policy-greylist unix - n n - - spawn
user=nobody argv=/usr/bin/perl /usr/bin/greylist.pl

The postfix spawn daemon normally kills its child processes after 1000 seconds but this is too short for the greylisting process so we have to increase the timelimit in main.cf:

Code Listing 7.4: Modifying main.cf to use greylisting

policy-greylist_time_limit = 3600
(Under smtpd_recipient_restrictions add:)
check_sender_access hash:/etc/postfix/sender_access
(Later on add:)
restriction_classes = greylist
greylist = check_policy_service unix:private/policy-greylist

Warning: Be sure to specify check_sender_access AFTER reject_unauth_destination or else your system could become an open mail relay.

Note: The greylist database gets polluted quickly with bogus addresses. It helps if you protect greylist lookups with other restrictions that reject unknown senders and/or recipients.

We don't want to use greylisting for all domains but only for those frequently abused by spammers. After all it will delay mail delivery. A list of frequently forged MAIL FROM domains can be found online. Add the domains you receive a lot of spam from to /etc/postfix/sender_access:

Code Listing 7.5: Format of sender_access

aol.com greylist
hotmail.com greylist
bigfoot.com greylist

If you want a more extensive list:

Code Listing 7.6: Adding all domains to sender_access

# wget http://www.monkeys.com/anti-spam/filtering/sender-domain-validate.in
# cat sender-domain-validate.in | sort | awk {'print $1 "\t\t greylist"'} > /etc/postfix/sender_access

Now we only have to initialize the sender_access database:

Code Listing 7.7: Initialize sender_access

# postmap /etc/postfix/sender_access

Now the setup of simple greylisting is complete.

Warning: I tried this on one box handling thousands of mails daily and the results were almost a complete disaster. After four days the box was bogged down with hundreds of old greylist.pl processes.

Configuring improved greylisting with postgrey

You can install the enhanced greylisting policy server with a simple emerge:

Code Listing 7.8: Installing postgrey

# emerge postgrey

After installing postgrey we have to edit main.cf. Changes are almost exactly like the built in greylisting.

Code Listing 7.9: Modifying main.cf to use greylisting

(Under smtpd_recipient_restrictions add:)
check_sender_access hash:/etc/postfix/sender_access
(Later on add:)
smtpd_restriction_classes = greylist
greylist = check_policy_service inet:127.0.0.1:10030

Note: The Postfix SMTPD_POLICY_README only uses restriction_classes but that does not appear to work.

Note: If you want to greylist everything instead add check_policy_service inet:127.0.0.1:10030.

Finally, start the server and add it to the proper runlevel.

Code Listing 7.10: Starting postgrey

# /etc/init.d/postgrey start
# rc-update add postgrey default

Note: Some people like to get their mail fast and thus greylisting is worthless. However if you employ a backup mail server you can safely setup greylisting on that server. My limited experiences tell me that it can stop up to a third of the spam received.

8. SPF (Sender Policy Framework)

Introduction

SPF allows domain owners to state in their DNS records which IP addressess should be allowed to send mails from their domain. This will prevent spammers from spoofing the Return-Path.

Note: If your ISP blocks incoming traffic on port 25 and relays all mail to you through their own mail server SPF will not work.

First domain owners have to create a special TXT DNS record. Then an SPF-enabled MTA can read this and if the mail originates from a server that is not described in the SPF record the mail can be rejected. An example entry could look like this:

Code Listing 8.1: Example SPF record

example.com. IN TXT "v=spf1 a mx ptr -all"

The -all means to reject all mail by default but allow mail from the A(a), MX(mx) and PTR(ptr) DNS records. For more info consult further resources below.

Note: If you relay outgoing mail through your ISP you will have to add: include:yourisp.com.

Spamassassin 3.0 has support for SPF, however it is not enabled by default and the new policy daemon in Postfix supports SPF so let's install SPF support for Postfix.

Note: If you want to use SPF with Spamassassin instead simply emerge dev-perl/Mail-SPF-Query and restart Amavisd-new.

Preparations

First you have to install Postfix 2.1 as described above. When you have fetched the source grab the spf.pl with:

Code Listing 8.2: Installing spf.pl

# cp postfix-<version>/examples/smtpd-policy/spf.pl /usr/local/bin/

Note: The spf.pl coming with Postfix is slightly buggy so find and uncomment the following line: push @HANDLERS, "sender_permitted_from"; use Mail::SPF::Query;. Furthermore in about line 199 substitute comemnt with comment. Alternatively you can download a development version.

This Perl script also needs some Perl libraries that are not in portage but it is still quite simple to install them:

Code Listing 8.3: Installing the needed Perl libraries

# emerge Mail-SPF-Query Net-CIDR-Lite Sys-Hostname-Long

Now that we have everything in place all we need is to configure Postfix to use this new policy.

Code Listing 8.4: Modifying master.cf to use SPF

policy-spf unix - n n - - spawn
user=nobody argv=/usr/bin/perl /usr/local/bin/spf.pl

Now add the SPF check in main.cf. Properly configured SPF should do no harm so we could check SPF for all domains:

Code Listing 8.5: Modifying main.cf to use SPF

(Under smtpd_recipient_restrictions add:)
check_policy_service unix:private/policy-spf

Note: If you're experiencing problems with SPF, e.g. when using fetchmail, you might want to enable SPF for certain domains only.

9. Configuring amavisd-new to use MySQL

Configuring MySQL

Note: This has not been tested on versions higher than 2.2. Feedback is welcome :)

For large domains the default values you can set in amavisd.conf might not suit all users. If you configure amavisd-new with MySQL support you can have individual settings for users or groups of users.

Code Listing 9.1: Creating the MySQL database and user

# mysql -u root -p mysql
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 78 to server version: 4.0.18-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> create database maildb;
mysql> GRANT INSERT,UPDATE,DELETE,SELECT ON maildb.* TO 'mail'@'localhost' IDENTIFIED BY 'very_secret_password';
mysql> use maildb;

Now that the database is created we'll need to create the necessary tables. You can cut and paste the following into the mysql prompt:

Code Listing 9.2: MySQL table layout

CREATE TABLE users (
id int unsigned NOT NULL auto_increment,
priority int NOT NULL DEFAULT '7', -- 0 is low priority
policy_id int unsigned NOT NULL DEFAULT '1',
email varchar(255) NOT NULL,
fullname varchar(255) DEFAULT NULL, -- not used by amavisd-new
local char(1), -- Y/N (optional field, see note further down)
PRIMARY KEY (id),
KEY email (email)
);
CREATE UNIQUE INDEX users_idx_email ON users(email);

(any e-mail address, external or local, used as senders in wblist)
CREATE TABLE mailaddr (
id int unsigned NOT NULL auto_increment,
priority int NOT NULL DEFAULT '7', -- 0 is low priority
email varchar(255) NOT NULL,
PRIMARY KEY (id),
KEY email (email)
);
CREATE UNIQUE INDEX mailaddr_idx_email ON mailaddr(email);

(-- per-recipient whitelist and/or blacklist,
-- puts sender and recipient in relation wb)
(white or blacklisted sender)
CREATE TABLE wblist (
rid int unsigned NOT NULL, -- recipient: users.id
sid int unsigned NOT NULL, -- sender: mailaddr.id
wb char(1) NOT NULL, -- W or Y / B or N / space=neutral
PRIMARY KEY (rid,sid)
);

CREATE TABLE policy (
id int unsigned NOT NULL auto_increment,
policy_name varchar(32), -- not used by amavisd-new
virus_lover char(1), -- Y/N
spam_lover char(1), -- Y/N (optional field)
banned_files_lover char(1), -- Y/N (optional field)
bad_header_lover char(1), -- Y/N (optional field)
bypass_virus_checks char(1), -- Y/N
bypass_spam_checks char(1), -- Y/N
bypass_banned_checks char(1), -- Y/N (optional field)
bypass_header_checks char(1), -- Y/N (optional field)
spam_modifies_subj char(1), -- Y/N (optional field)
spam_quarantine_to varchar(64) DEFAULT NULL, -- (optional field)
spam_tag_level float, -- higher score inserts spam info headers
spam_tag2_level float DEFAULT NULL, -- higher score inserts
-- 'declared spam' info header fields
spam_kill_level float, -- higher score activates evasive actions, e.g.
-- reject/drop, quarantine, ...
-- (subject to final_spam_destiny setting)
PRIMARY KEY (id)
);

Note: If you have problems using copy/paste you might have to copy this somewhere else and clean out the unneeded characters.

Note: Lookups trying to match email are done with raw (rfc2821-unquoted and unbracketed) addresses as a key, i.e.: John "Funny" [email-masked]

Note: Lookups are performed in the following order: SQL, LDAP, hash, ACL, regexp, constant. The first that returns a definitive answer (not undef/NULL) stops the search.

If you wish to use whitelisting and blacklisting you must add the sender and receiver to mailadr after which you create the relation between the two e-mail addresses in wblist and state if it is whitelisting (W) or blacklisting (B).

Now that we have created the tables let's insert a test user and a test policy:

Code Listing 9.3: Create test user and test policy

INSERT INTO users
SET
id =1,
priority =9,
policy_id =1,
email ="[email-masked]",
fullname ="John Doe",
local ="Y";

INSERT INTO policy
SET
id =1,
policy_name ="Test policy 1",
virus_lover ="N",
spam_lover ="N",
banned_files_lover ="N",
bad_header_lover ="N",
bypass_virus_checks ="N",
bypass_spam_checks ="N",
bypass_banned_checks ="N",
bypass_header_checks ="N",
spam_modifies_subj ="N",
spam_quarantine_to =NULL,
spam_tag_level =-50.0,
spam_tag2_level =7.0,
spam_kill_level =10.0;

Note: Copy this to somewhere else and adjust to suit your own environment.

Note: local should be set to Y otherwise the mail will not be scanned for spam.

This inserts a test user and a Test policy. Adjust these examples to fit your needs. Further explanation of the configuration names can be found in amavisd.conf.

Configuring amavisd to use MySQL

Now that MySQL is ready we need to tell amavis to use it:

Code Listing 9.4: Modifying amavisd.conf

@lookup_sql_dsn =
( ['DBI:mysql:maildb:host1', 'mail', 'very_secret_password'] );

(For clarity uncomment the default)
$sql_select_policy = 'SELECT *,users.id FROM users,policy'.
' WHERE (users.policy_id=policy.id) AND (users.email IN (%k))'.
' ORDER BY users.priority DESC';

(If you want sender white/blacklisting)
$sql_select_white_black_list = 'SELECT wb FROM wblist,mailaddr'.
' WHERE (wblist.rid=?) AND (wblist.sid=mailaddr.id)'.
' AND (mailaddr.email IN (%k))'.
' ORDER BY mailaddr.priority DESC';

10. Configuring Spamassassin to use MySQL

As of Spamassassin 3.0 it is possible to store the Bayes and AWL data in a MySQL database. We will use MySQL as the backend as it can generally outperform other databases. Also, using MySQL for both sets of data makes system management much easier. Here I will show how to easily accomplish this.

First start out by creating the new MySQL user and then create the needed tables.

Code Listing 10.1: Creating the new MySQL database and user

# mysql -u root -p mysql
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 78 to server version: 4.0.18-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> create database dbname;
mysql> GRANT INSERT,UPDATE,DELETE,SELECT ON dbname.* TO 'dbuser'@'localhost' IDENTIFIED BY 'another_very_secret_password';
mysql> use dbname;

Now that the database is created we'll create the necessary tables. You can cut and paste the following into the mysql prompt:

Code Listing 10.2: MySQL table layout

CREATE TABLE bayes_expire (
id int(11) NOT NULL default '0',
runtime int(11) NOT NULL default '0',
KEY bayes_expire_idx1 (id)
) TYPE=MyISAM;

CREATE TABLE bayes_global_vars (
variable varchar(30) NOT NULL default '',
value varchar(200) NOT NULL default '',
PRIMARY KEY (variable)
) TYPE=MyISAM;

INSERT INTO bayes_global_vars VALUES ('VERSION','3');

CREATE TABLE bayes_seen (
id int(11) NOT NULL default '0',
msgid varchar(200) binary NOT NULL default '',
flag char(1) NOT NULL default '',
PRIMARY KEY (id,msgid)
) TYPE=MyISAM;

CREATE TABLE bayes_token (
id int(11) NOT NULL default '0',
token char(5) NOT NULL default '',
spam_count int(11) NOT NULL default '0',
ham_count int(11) NOT NULL default '0',
atime int(11) NOT NULL default '0',
PRIMARY KEY (id, token),
INDEX (id, atime)
) TYPE=MyISAM;

CREATE TABLE bayes_vars (
id int(11) NOT NULL AUTO_INCREMENT,
username varchar(200) NOT NULL default '',
spam_count int(11) NOT NULL default '0',
ham_count int(11) NOT NULL default '0',
token_count int(11) NOT NULL default '0',
last_expire int(11) NOT NULL default '0',
last_atime_delta int(11) NOT NULL default '0',
last_expire_reduce int(11) NOT NULL default '0',
oldest_token_age int(11) NOT NULL default '2147483647',
newest_token_age int(11) NOT NULL default '0',
PRIMARY KEY (id),
UNIQUE bayes_vars_idx1 (username)
) TYPE=MyISAM;

CREATE TABLE awl (
username varchar(100) NOT NULL default '',
email varchar(200) NOT NULL default '',
ip varchar(10) NOT NULL default '',
count int(11) default '0',
totscore float default '0',
PRIMARY KEY (username,email,ip)
) TYPE=MyISAM;

Important: The INSERT line is needed otherwise Spamassassin will not work.

Note: This is also available in the source tarball in the files awl_mysql.sql and bayes_mysql.sql.

Configuring Spamassassin to use the MySQL backend

If you have an old Bayes database in the DBM database and want to keep it follow these instructions:

Code Listing 10.3: Converting Bayes data from a DBM Database

su - amavis
sa-learn --sync
sa-learn --backup > backup.txt
sa-learn --restore backup.txt

Note: Note that the last step should only be performed after the MySQL database and secrets.cf have been updated.

Now give Spamassassin the required info:

Code Listing 10.4: Modifying /etc/mail/spamassassin/secrets.cf

(Tell Spamassassin to use MySQL for bayes data
bayes_store_module Mail::SpamAssassin::BayesStore::SQL
bayes_sql_dsn DBI:mysql:sa_bayes:localhost:3306
bayes_sql_username db_name
bayes_sql_password another_very_secret_password

(Tell Spamassassin to use MySQL for AWL data
auto_whitelist_factory Mail::SpamAssassin::SQLBasedAddrList
user_awl_dsn DBI:mysql:sa_bayes:localhost:3306
user_awl_sql_username db_name
user_awl_sql_password another_very_secret_password

Next, change its permissions for proper security:

Code Listing 10.5: Changing permissions

# chmod 400 /etc/mail/spamassassin/secrets.cf

Note: To create a very secret password use emerge app-admin/makepasswd and makepasswd -chars=8

Now all you have to do is /etc/init.d/amavisd restart.

11. Troubleshooting

Amavisd-new

To troubleshoot Amavisd-new start out by stopping it with /etc/init.d/amavisd stop and then start it manually in the foreground with amavisd debug and watch it for anomalies in the output.

Spamassassin

To troubleshoot Spamassassin you can filter an email through it with spamassassin -D < mail. To ensure that the headers are intact you can move it from another machine with IMAP.

Note: If you need to troubleshoot you have to enable login for the user amavis by changing the login shell in /etc/passwd to /bin/bash.

If you want you can make get the same information and more with Amavisd-new using amavisd debug-sa.

Getting help

If you need help a good place to go is the amavis-user mailing list. Before postting a question try searching the Amavis User mailing list archives. If you find no answer here you can subscribe to the Amavis User mailing list

If your question is specific to SpamAssassin, DCC, Razor, or Postfix, please refer to their respective home pages listed below.

12. Resources

For further information

Amavisd-new INSTALL

Amavisd-new Postfix README

Amavisd-new Policy bank documentation

Spamassassin SQL README

Greylisting

Postfix SMTPD_POLICY_README

Blocking spammers with Postfix HELO controls

SPF Overview

Jim Seymour's Postfix Anti-UCE Cheat Sheet

General resources

Spamassassin

Amavisd-new

Amavisd-new documentation bits and pieces

Vipuls's Razor

Pyzor

Distributed Checksum Clearinghouse

Maia Mailguard

Other howtos

Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC

The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.

Print

Updated August 2, 2007

Summary: This guide is step-by-step guide for installing spam fighting technologies for Postfix. Among them Amavisd-new using Spamassassin and ClamAV, greylisting and SPF.

Sune Kloppenborg Jeppesen
Author

Jens Hilligsøe
Contributor

Joshua Saddler
Editor

Donate to support our development efforts.

VR Hosted

Tek Alchemy

SevenL.net

Global Netoptex Inc.

Copyright 2001-2007 Gentoo Foundation, Inc. Questions, Comments? Contact us.