1. Managing Spam Filtering
Module 11
Spam consumes precious network bandwidth and lowers the productivity of employees. We will explore how to combat spammers in this module.
After completing this modules, students will be able to:
List the tests SpamAssassin utilizes
Implement whitelists and blacklists
Modify the responses sent in response to spam
Write rules for SpamAssassin
Explain SpamAssassin scoring
Examine quarantined viruses and spam
Block messages based on header and body checks
Block specific attachment types
Managing Spam Filtering

2. SpamAssassin Spam Cyrus IMAP OpenLDAP ClamAV Apache ProFTP Assassin
AMaViS
Postfix
SCO OpenServer

3. SpamAssassin SpamAssassin uses numerous tests
SpamAssassin is configured in:
/opt/insight/etc/mail/local.cf
/opt/insight/share/spamassassin/*.cf
Do not modify files in share/spamassassin
After modifying configuration files, run:
spamassassin --lint
/opt/insight/etc/rc/amavisd restart
SpamAssassin analyzes messages searching for spam. To achieve this goal, SpamAssassin uses numerous tests (header analysis, text analysis, blacklists, Vipul’s Razor, etc.). For a comprehensive list of SpamAssassin’s tests, see Supplemental rule sets designed to handle a variety of different types of Spam are available at
SpamAssasssin’s configuration files reside in /opt/insight/etc/mail/*.cf and /opt/insight/share/spamassassin/*.cf. Do not modify configuration files in the share/spamassassin directory. Files in that directory are overwritten when SpamAssassin is updated. Instead, make changes to /opt/insight/etc/mail/local.cf. Entries in this file override entries in the share/spamassassin directory. Whenever changes are made to SpamAssassin’s configuration, it is important to run spamassassin --lint to verify that the configuration files contain no syntactical errors.
The combination of configuration settings in these two locations define the site-wide configuration. In some implementations of SpamAssassin, users can customize their own configuration. SCOoffice Server does not provide this capability.
Because the SpamAssassin distribution in SCOoffice Server is built into AMaVis, you must restart amavisd for changes to the configuration files to be recognized:
# /opt/insight/etc/rc/amavisd restart
??? amavisd restart fails to stop amavisd successfully sometimes (leaving amavisd running on port 10024). Killing the master amavisd with a SIGTERM does the trick. ???

4. SpamAssassin Every SpamAssassin administrator should know:
required_hits
report_contact
report_safe
Whitelisting
Blacklisting
There are a few settings every SpamAssassin administrator should know.
The required_hits parameter specifies how many hits (a.k.a. the score) required for SpamAssassin to consider a message spam. The default is 5.0. While the scores in SpamAssassin’s rule base have been extensively tuned, it is not possible to anticipate every mail environment. If your mail environment is getting too much spam, lower this value. Conversely, if your mail environment is getting too many false positives (i.e. non-spam flagged as possible spam), increase this value.
WARNING: Setting required_hits too low can result in false positives.
The report_contact setting is set to postmaster by default. This is the address address senders see when their bounces as spam.
SCOoffice Server sets report_safe to 1. This instructs SpamAssassin to encapsulate the original message as a “message/rfc822” MIME part. This approach preserves the original message. If this behavior is not desired, setting report_safe to 0 instructs SpamAssassin not to MIME encapsulate the original message and to simply add X-Spam headers instead. Alternatively, setting report_safe to 2 instructs SpamAssassin to MIME encapsulate the original message as “message/rfc822”.
If messages from specific sources are being incorrectly marked spam, these sources can be added to a whitelist:
whitelist_from
whitelist_from
The whitelist_from entries support limited file-glob style patterns (i.e. * and ? wildcards are allowed). The first whitelist example above allows messages from The second allows messages from anyone whose address ends in The whitelist_from_rcvd setting works like whitelist_from with the added capability of matching against the Received headers.
The 60_whitelist.cf configuration file included in the SpamAssassin distribution shows numerous whitelisting examples. Note that whitelist entries in the distribution use def_whitelist_from_rcvd setting.
Blacklisting does the opposite of whitelisting. Blacklists can be established using blacklist_from and blacklist_to.
See Mail::SpamAssassin::Conf(3) for more information.

5. SpamAssassin Customizing headers SpamAssassin headers begin “X-Spam”
X-Spam-Checker-Version is mandatory
Modify headers with:
remove_header
clear_headers
add_header
SpamAssassin adds headers to messages. Administrators can customize these messages using remove_header, clear_headers, and add_header.
All headers can be removed except the X-Spam-Checker-Version header which is mandatory.
See Mail::SpamAssassin::Conf(3) for more information.

6. SpamAssassin Report message:
Spam detection software, running on the system "_HOSTNAME_", has
identified this incoming as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or block
similar future . If you have any questions, see
_CONTACTADDRESS_ for details.
Content preview: _PREVIEW_
Content analysis details: (_HITS_ points, _REQD_ required)
" pts rule name description"
_SUMMARY_
When SpamAssassin notifies a recipient that a message contained spam, it does so using the report shown in the slide above. This message is configurable. Within this report, tags (e.g. _HOSTNAME_, _CONTACTADDRESS_, _PREVIEW_, _HITS_, _REQD, and _SUMMARY_) are used to fill in relevant information about the message. The definition of all the tags can be found in Mail::SpamAssassin::Conf(3).
See 10_misc.cf for an example.
Each report entry in the configuration files adds to the report template. When creating a customized report template in local.cf, use clear_report_template to clear the report template set in /opt/insight/share/spamassassin/10_misc.cf (or elsewhere in /opt/insight/share/spamassassin).

7. SpamAssassin Spamtrap message:
Subject: this address is no longer available [this message has been automatically generated] Please note that this address is no longer in use, and nowadays receives nothing but unsolicited commercial mail. Accordingly, any mail sent to it is added to several spam-tracking databases, then automatically deleted. If you genuinely want to contact the owner of the address, please re-check your contact lists, or search the web, to find their current address. The mail you sent is reproduced in full below, for resending to the correct address. Sorry for the inconvenience! [-- Signed: the SpamAssassin mail filter]
When SpamAssassin notifies a sender that a message has been spam trapped, it sends the explanatory text set by the spamtrap entries. If the initial lines in the spamtrap text are headers, these headers will be used in the sent to the sender.
See 10_misc.cf for an example.
Tags cannot be used in the spamtrap report.
See trapping and for more information.
Each spamtrap entry in the configuration files adds to the spamtrap template. When creating a customized spamtrap template in local.cf, use clear_spamtrap_template to clear the spamtrap template set in /opt/insight/share/spamassassin/10_misc.cf (or elsewhere in /opt/insight/share/spamassassin).

8. SpamAssassin Unsafe_report message:
The original message was not completely plain text, and may be unsafe to
open with some clients; in particular, it may contain a virus,
or confirm that your address can receive spam. If you wish to view
it, it may be safer to save it to a file and open it with an editor.
When SpamAssassin notifies a recipient of an unsafe message, it does so using the template shown in the slide above. This message is configurable. Within this report, tags can be used. The definition of all the tags can be found in Mail::SpamAssassin::Conf(3).
See 10_misc.cf for an example.
Each unsafe_report entry in the configuration files adds to the unsafe_report template. When creating a customized unsafe_report template in local.cf, use clear_unsafe_report_template to clear the unsafe_report template set in /opt/insight/share/spamassassin/10_misc.cf (or elsewhere in /opt/insight/share/spamassassin).

9. SpamAssassin Areas tested: header body rawbody full uri
The tests specified in SpamAssassin’s .cf files can target different portions of the messages analyzed:
header
Defines a test which analyzes one or more of a message’s headers (e.g. Subject, To, Cc, etc.)
body
Defines a test which analyzes the textual portions of a message’s body. Non-text MIME parts are stripped as are HTML tags and line breaks. Encoded portions of the body (e.g. Base64, Quoted-Printable) are decoded.
rawbody
Defines a test which works like a body test, but HTML tags and line breaks are not removed.
full
Defines a test which analyzes the message in its undecoded form (including attachments, images, etc.).
uri
Defines a test which analyzes the URIs in the message.
See Mail::SpamAssassin::Conf(3) for more information.

10. Perl regular expression
SpamAssassin
Header test example:
Perl regex operator
Name of rule
header NO_REAL_NAME From =~
SpamAssassin’s 20_head_tests.cf configuration file specifies the following header test:
header NO_REAL_NAME From =~
Name of rule
By convention, rules are upper case and limited to 22 characters.
Header to match
Can be Subject, To, From, Cc, etc. ALL means match against all of the message’s headers. Cc means match agains both the To and Cc headers
Perl regex operator
Can be =~ (contains regular expression) or !~ (does not contain regular expression).
Perl regular expression
Uses Perl regular expression syntax to specify a pattern to match against in the specified message header.
Header to match
Perl regular expression

11. SCOoffice uses this score
SpamAssassin
Header test definitions only define the test
Header test definitions don’t define:
The test’s description
The test’s score
20_head_tests.cf specifies:
50_scores.cf specifies:
header NO_REAL_NAME From =~
describe NO_REAL_NAME From: does not include a real name
Notice that nowhere in the example header test on the previous slide did the test:
Specify what message (if any) should be displayed if the test succeeds
Specify the score of the test
Each test defines its description and its score using the describe and score settings.
By convention, descriptions are limited to 50 characters.
A score setting will specify either 1 number or 4 numbers. When a score setting specifies a single number, that number is the test’s score. When a score setting specifies 4 numbers, the second number the number used by SCOoffice Server’s SpamAssassin configuration. If no score is specified for a test, the default score is 1.0. For information about the other three numbers, see Mail::SpamAssassin::Conf(3).
SCOoffice uses this score
score NO_REAL_NAME

12. SpamAssassin Meta-match (boolean expression)
body CLICK_BELOW_CAPS /CLICK\s.{0,30}(?:HERE|BELOW)/s
describe CLICK_BELOW_CAPS Asks you to click below (in capital letters)
body __CLICK_BELOW /click\s.{0,30}(?:here|below)/is
meta CLICK_BELOW (__CLICK_BELOW && !CLICK_BELOW_CAPS)
describe CLICK_BELOW Asks you to click below
Meta-match rules enable administrators to create more complex rules in SpamAssassin. There are two types of meta-match rules: those that use boolean expressions, and those that use boolean arithmetic expressions
The following creates a meta rule called CLICK_BELOW:
meta CLICK_BELOW (__CLICK_BELOW && !CLICK_BELOW_CAPS)
This meta rule is based on two other rules: CLICK_BELOW_CAPS and __CLICK_BELOW.
Meta-rules allow the use of the boolean &&, ||, and ! operators for AND, OR, and NOT, respectively.
Rules beginning with __ (double underscore) are meta-match sub-rules. These rules are intended for inclusion in meta rule definitions. Meta-match sub-rules are not scored by SpamAssassin.

13. SpamAssassin Meta-match (boolean arithmetic expression)
body __NIGERIAN_CODE_CONDUCT /\bcode of conduct\b/i
body __NIGERIAN_CIV_SERVICE /\bcivil service\b/i
body __NIGERIAN_TOP_SECRET /\btop secret\b/I
body __NIGERIAN_HONESTY /\btransparent honesty\b/i
meta NIGERIAN_BODY_GOVT ((__NIGERIAN_CODE_CONDUCT +
__NIGERIAN_CIV_SERVICE +
__NIGERIAN_TOP_SECRET +
__NIGERIAN_HONESTY) >= 2)
describe NIGERIAN_BODY_GOVT Message body has many indications of nigerian scam
score NIGERIAN_BODY_GOVT
Meta-match rules which use boolean arithmetic expressions can also be used to create more complex rules in SpamAssassin.
The NIGERIAN_BODY_GOVT meta-match rule defined above only succeeds if two or more of its meta-match sub-rules succeed.
NOTE: The sample rules above have been modified for clarity. In fact, the multi-line format used above is syntactically invalid in SpamAssassin. See 20_phrases.cf for the actual definitions of these rules.

14. Quaranting Viruses and Spam
By default, SCOoffice Server:
Quarantines messages containing viruses
Does not quarantine messages containing spam
By default, SCOoffice Server quarantines messages containing viruses but does not quarantine messages containing spam.
Messages that contain viruses are sent to the user. However, the message sent to the user does not contain the virus. Rather, the message containing the virus is quarantined in /opt/insight/var/virusmail. The message sent to the user contains the exact name of the quarantine file.

15. Quaranting Viruses and Spam
Messages containing viruses are quarantined by AMaViS.
By default, AMaViS quarantines messages containing viruses. The intended recipient of the message is sent a message from “virusalert” which identifies:
The name of the virus
Who sent the virus
Where the message containing the virus is quarantined
Many aspects of how messages containing viruses are handled are configurable via the amavisd.conf file. Administrators might want to consider modifying $final_virus_destiny which can take four possible values:
D_PASS -- Message is passed to users regardless whether it contains a virus
D_DISCARD -- Message is not delivered to recipient and sender is not notified
D_BOUNCE -- Message is not delivered to recipient, but the sender is sent a non-delivery notification
D_REJECT -- Message is not delivered to recipient and sender is notified via an SMTP permanent reject response
D_DISCARD is the default.

16. Quaranting Viruses and Spam
Headers added to messages containing spam:
X-Virus-Scanned
X-Spam-Status
X-Spam-Level
X-Spam-Flag
Subject
By default, SpamAssassin adds headers to messages which contain spam. This enables users to easily configure their clients to sort these messages accordingly.
The X-Virus-Scanned header indicates the name of the system that performed the amavisd scan. The X-Spam-Status header reports the score assigned to the message by SpamAssassin and provides information about which SpamAssassin tests were triggered by the message. The X-Spam-Level header provides a visual indication of the score assigned to the message by SpamAssassin. The X-Spam-Flag indicates whether the SpamAssassin considers the message to be spam. The Subject: header is modified to include the string “***SPAM***” if SpamAssassin considers the message to be spam.
In the example above, the score (a.k.a. “hits”) is exceptionally high simply because GTUBE (the Generic Test for Unsolicited Bulk ) was triggered.
Many aspects of how messages containing spam are handled are configurable via the amavisd.conf file. Administrators might want to consider modifying $final_spam_destiny which can take four possible values:
D_PASS -- Message is passed to users regardless whether it contains spam
D_DISCARD -- Message is not delivered to recipient and sender is not notified
D_BOUNCE -- Message is not delivered to recipient, but the sender is sent a non-delivery notification
D_REJECT -- Message is not delivered to recipient and sender is notified via an SMTP permanent reject response
D_DISCARD is the default.

17. Quaranting Viruses and Spam
AMaViS can be configured to quarantine spam
Configured in amavisd.conf
$final_spam_destiny
$QUARANTINEDIR
$spam_quarantine_to
AMaViS can be configured to quarantine spam. This is accomplished by setting parameters in amavisd.conf.
$final_spam_destiny work like $final_virus_destiny. Both accept the same values: D_PASS, D_BOUNCE, D_DISCARD, and D_REJECT.
$QUARANTINEDIR specifies the directory (or file) in which spam is quarantined if spam quarantining is enabled.
If $spam_quarantine_to is undefined, then spam is not quarantined.
If $spam_quarantine_to is set to a string which does not contain an then spam is quarantined in the directory specified by $QUARANTINEDIR.
If $spam_quarantine_to is set to an address, spam is ed to the user specified by that address.

18. Quaranting Viruses and Spam
To quarantine spam to a directory, configure amavisd.conf:
$final_spam_destiny = D_PASS
$QUARANTINEDIR = /opt/insight/var/virusmails
$spam_quarantine_to = ‘spam-quarantine’
AMaViS can be configured to quarantine spam. This is accomplished by setting parameters in amavisd.conf.
$final_spam_destiny work like $final_virus_destiny. Both accept the same values: D_PASS, D_BOUNCE, D_DISCARD, and D_REJECT.
$QUARANTINEDIR specifies the directory (or file) in which spam is quarantined if spam quarantining is enabled.
If $spam_quarantine_to is undefined, then spam is not quarantined.
If $spam_quarantine_to is set to a string which does not contain an then spam is quarantined in the directory specified by $QUARANTINEDIR.
If $spam_quarantine_to is set to an address, spam is ed to the user specified by that address.
By quarantining spam to a specific directory, administrators can centralize the spam messages for analysis.
To configure the system to quarantine spam in a directory, configure amavisd.conf:
If you want users to see the spam messages in addition to the spam messages being quarantined, set $final_spam_destiny = D_PASS. If you want spam messages quarantined but you don’t want users to see the spam messages, set $final_spam_destiny = D_DISCARD.
$QUARANTINEDIR (the directory in which spam messages (and viruses) are quarantined) is set to /opt/insight/var/virusmails.
Set $spam_quarantine_to = ‘spam-quarantine’

19. Header Checks To block emails based on headers:
In /opt/insight/etc/postfix/main.cf:
header_checks = pcre:/opt/insight/etc/postfix/header_checks
In /opt/insight/etc/postfix/header_checks:
/^subject: known_message_subject/ REJECT
While SpamAssassin and ClamAV do a fine job of blocking spam and viruses, both require CPU resources and Postfix must queue up each message. If an administrator is aware of a specific SPAM or virus alert which could potentially flood the mail server with messages, the administrator can use header checks (see also body checks) to reject the messages before they are even queued by Postfix.
In /opt/insight/etc/postfix/main.cf:
header_checks = pcre:/opt/insight/etc/postfix/header_checks
Do not forget to issue postfix reload.
Each line in the header_checks file consists of two fields. The first field is a Perl compatible regular expression used to match against header lines in messages. The remainder of the line specifies what Postfix should do with the message if the regular expression matches. The REJECT can be followed by an optional message.
Since the header_checks file is pcre, it is not necessary to run postmap(1).
One of the limitations of header (and body) checks is that the message is not decoded (e.g. Base64 decoding). If necessary, you can craft your regular expression(s) to match against the encoded message. Alternatively, you can speed up header (and body) checks significantly using the suggestion at
# First skip over base 64 encoded text to save CPU cycles.
~^[[:alnum:]+/]{60,}$~ OK
Other limitations of header (and body) checks can be found at

20. Blocking Attachments by Extension
To block s containing .exe, .bat, etc. attachments:
In /opt/insight/etc/postfix/main.cf:
header_checks = pcre:/opt/insight/etc/postfix/header_checks
In /opt/insight/etc/postfix/header_checks:
/^content-type:.*name[[:space:]]*=.*\.(exe|bat)/ REJECT Rejected file extension: $1
Administrators who do not want to allow messages with attachments containing .exe, .bat, .vbs, etc. extensions can reject such messages even before Postfix queues them. Implementing this involves setting the header_checks parameter in main.cf.
In /opt/insight/etc/postfix/main.cf:
header_checks = pcre:/opt/insight/etc/postfix/header_checks
Do not forget to issue postfix reload.
Each line in the header_checks file consists of two fields. The first field is a Perl compatible regular expression specifying the content types (i.e. file extensions) being rejected. The remainder of the line specifies to reject the message if the content type is disallowed. Notice that REJECT can be followed by an optional message.
Since the header_checks file is pcre, it is not necessary to run postmap(1).