• 
Table of Contents

• 
Index

sendmail Milters A Guide for Fighting Spam

By Bryan Costales, Marcia Flynt

Publisher
: Addison Wesley Professional

Pub Date
: January 24, 2005

ISBN
: 0-321-21333-5

Pages
: 352

"Bryan and Marcia have not only provided tips and tricks for detecting and blocking spam and email fraud but have also written the first of its kind Milter reference guide. This book will help you start writing your own special-purpose mail filters quickly and easily."

--Gregory Neil Shapiro, coauthor of the Milter interface

As a Mail Administrator You Need to Understand How to Successfully Monitor and Fight Spam.

Milters are among the most powerful antispamming tools available. Until now, there has been no clear and helpful resource for you to learn how to set up and use Milters. sendmail Milters: A Guide for Fighting Spam is the first in-depth guide to writing powerful Milters to block even the most clever spammers.

Inside this definitive new reference, you will find

An exhaustive description of the Milter interface

Insightful details on what spam is, its harmful effects, and the diverse techniques used by spammers

A step-by-step guide to luring spammers using a honeypot network

Ways to decode the common encoding methods used in spam email

A reference on the Milter library and its use

Techniques for expanding software to deal with future spamming methods

This book is an indispensable aid to combating spam, now and in the future. If you administer a sendmail server, you need to own a copy.

All of the program code described in the book is available for download at http://spambook.bcx.org.

• 
Table of Contents

• 
Index

sendmail Milters A Guide for Fighting Spam

By Bryan Costales, Marcia Flynt

Publisher
: Addison Wesley Professional

Pub Date
: January 24, 2005

ISBN
: 0-321-21333-5

Pages
: 352

|  | Copyright
|  | Preface
|  |  | Audience
|  |  | Whence Spam?
|  |  | Are Milters the Only Answer?
|  |  | The Organization of This Book
|  |  | Acknowledgments
|  |  | About the Authors
|  | Part I.  The Nefarious Spam Problem
|  |  |     Chapter 1.  Gorilla Versus Guerrilla
|  |  | Section 1.1.  When a Gorilla Sneezes
|  |  | Section 1.2.  When a Guerrilla Masquerades as You
|  |  | Section 1.3.  The Major Proposals for Standards
|  |  | Section 1.4.  Email Fraud
|  |  | Section 1.5.  The Cost of Spam Suppression
|  |  | Section 1.6.  Vikings
|  |  |     Chapter 2.  The Characteristics of Spam Email
|  |  | Section 2.1.  Connection Behavior
|  |  | Section 2.2.  Relaying through MX Servers
|  |  | Section 2.3.  Falsifying the Envelope Sender Address
|  |  | Section 2.4.  Disguising the Subject: Header
|  |  | Section 2.5.  Camouflaging the HTML Body
|  |  | Section 2.6.  Attempting to Fool Signature Detectors
|  |  | Section 2.7.  Unnecessary Encoding
|  |  | Section 2.8.  Grokking the Site
|  |  | Section 2.9.  Loose Ends
|  |  | Section 2.10.  Think Like a Spammer
|  | Part II.  Creating a Test Environment
|  |  |     Chapter 3.  Setting Up a Bait Machine
|  |  | Section 3.1.  Choose Your Platform
|  |  | Section 3.2.  Set Up DNS Records
|  |  | Section 3.3.  Configure sendmail
|  |  | Section 3.4.  Set Up Logging
|  |  | Section 3.5.  Excluding Non-email Ports
|  |  | Section 3.6.  Make Sure the Machine Reboots
|  |  |     Chapter 4.  Baiting the Hook
|  |  | Section 4.1.  Create Fake Recipients
|  |  | Section 4.2.  Protect Good Email
|  |  | Section 4.3.  Run a Web Server
|  |  | Section 4.4.  Post to a Usenet Group
|  |  |     Chapter 5.  Preventive Measures
|  |  | Section 5.1.  Tell Users about Plus Addressing
|  |  | Section 5.2.  Turn Off EXPN and VRFY
|  |  | Section 5.3.  Mask Web Addresses
|  |  | Section 5.4.  Watch Out for finger
|  | Part III.  The Parts of a sendmail Milter
|  |  |     Chapter 6.  The Roles and Flow of a Milter
|  |  | Section 6.1.  A Milter's Role in the Middle
|  |  | Section 6.2.  A Milter from the Point of View of sendmail
|  |  | Section 6.3.  The Milter Flow
|  |  |     Chapter 7.  The Milter-Library
|  |  | Section 7.1.  Overview
|  |  | Section 7.2.  main()
|  |  | Section 7.3.  The smfi Data Access Routines
|  |  | Section 7.4.  The smfi Modifier Routines
|  |  |     Chapter 8.  The xxfi Handler Milter Functions
|  |  | Section 8.1.  Overview
|  |  | Section 8.2.  The xxfi Orientations
|  |  | Section 8.3.  Abort Logic
|  |  | Section 8.4.  xxfi_connect() Reviews the Connection
|  |  | Section 8.5.  xxfi_helo() Reviews SMTP HELO/EHLO
|  |  | Section 8.6.  xxfi_envfrom() Reviews SMTP MAIL FROM
|  |  | Section 8.7.  xxfi_envrcpt() Reviews SMTP RCPT TO
|  |  | Section 8.8.  xxfi_header() Reviews Headers
|  |  | Section 8.9.  xxfi_eoh Reviews at End of Headers
|  |  | Section 8.10.  xxfi_body Reviews Each Body Chunk
|  |  | Section 8.11.  xxfi_eom Reviews at End of Envelope
|  |  | Section 8.12.  xxfi_abort Handles Envelope Abort
|  |  | Section 8.13.  xxfi_close Handles Connection Cleanup
|  | Part IV.  Nuts and Bolts
|  |  |     Chapter 9.  Milters and the Environment
|  |  | Section 9.1.  Where to Run Your Milter
|  |  | Section 9.2.  Your Milter's User ID
|  |  | Section 9.3.  How to Start and Stop Your Milter
|  |  | Section 9.4.  Put Your Milter into the Background
|  |  | Section 9.5.  Handle Signals
|  |  | Section 9.6.  Anticipate MX Servers
|  |  | Section 9.7.  Status and Logging
|  |  | Section 9.8.  Consider Portability Early
|  |  | Section 9.9.  Avoid Memory Leaks
|  |  | Section 9.10.  Final Words
|  |  |     Chapter 10.  User and Temporal Feedback
|  |  | Section 10.1.  Consider Architecture
|  |  | Section 10.2.  Model the End User
|  |  | Section 10.3.  Maintain a History
|  |  | Section 10.4.  Possible Feedback Mechanisms
|  |  | Section 10.5.  Whitelisting
|  |  | Section 10.6.  Graylisting
|  |  | Section 10.7.  Archive, Reject, or Pass Through Spam
|  |  | Section 10.8.  Dynamic Configurations
|  |  | Section 10.9.  In Summary
|  |  |     Chapter 11.  Handy Routines for the Message Body
|  |  | Section 11.1.  Parsing MIME-Encoded Boundaries
|  |  | Section 11.2.  Decoding Base64 Encoding
|  |  | Section 11.3.  Decoding Quoted-Printable Encoding
|  |  | Section 11.4.  Decoding Character-Entity Encoding
|  |  | Section 11.5.  Decoding Url-Encoding
|  |  | Section 11.6.  Stripping Html Comments
|  |  | Section 11.7.  How to Use /etc/magic
|  |  | Section 11.8.  How to Use /usr/share/dict/words
|  |  | Section 11.9.  More
|  |  |     Appendix A.  Useful Source Code
|  |  | Section A.1.  Examples of Milters
|  |  | Section A.2.  Chapter Examples
|  | Bibliography
|  |  | Books
|  |  | Journals and Magazines
|  |  | Requests for Comments
|  |  | Websites
|  | Index

Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:

U. S. Corporate and Government Sales
(800) 382-3419
corpsales@pearsontechgroup.com

For sales outside the U. S., please contact:

International Sales
international@pearsoned.com

Visit us on the Web: www.awprofessional.com

Library of Congress Cataloging-in-Publication Data

Costales, Bryan.
 Sendmail milters : a guide for fighting spam / Bryan Costales and
Marcia Flynt.
       p. cm.
 Includes bibliographical references and index.
 ISBN 0-321-21333-5 (pbk. : alk. paper)
1. Electronic mail systems -- Security measures. I. Flynt, Marcia.
II.  Title.
  TK5105.73 .C67 2005
  005.8 ' 22dc22                                    2004023747

Copyright © 2005 Bryan Costales and Marcia Flynt

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to:

Pearson Education, Inc.
Rights and Contracts Department
One Lake Street
Upper Saddle River, NJ 07458

Text printed in the United States on recycled paper at Courier in Stoughton, Massachusetts.

First printing, January 2005

Preface

Some have likened the fight against spam email to a war in which each side strives to outgun the other. Others use a cops-and-robbers analogy in which the robbers use their ill-gotten gains to buy better technology, while the cops must wait for voters to approve new budget items.

Still others have found the spam email problem so intractable and cancerlike that they have given up in despair. Many government offices, for example, have eliminated email altogether and adopted web forms in its place.

Unfortunately, spam email is a moving target. Books (once published) become etched in time and are no longer able to adapt to a world that continues to changeand spam email continues to change. Spammers will even use this book to learn the techniques being used against them and, like rats, adapt.

But someone must write a book like this. The spam email problem is too serious to let it run unopposed. Spam can be successfully fought only after it is fully understood and when knowledge of its techniques can be collected and collated for use in countermeasures. It is our hope that this book will represent a significant advance in the fight against spam.

Audience

This book targets users of the UNIX, Linux, and BSD family of operating systems. It presumes that sendmail is running as the mail transfer agent (MTA), but running the sendmail program is not strictly required (Parts I and II, for example, provide general lessons).

The bulk of this book (Parts III and IV) targets the sendmail administrator and the C language programmer. Milters are written in C and are specifically interfaced to sendmail. Although Milters can also be written in Perl, this book focuses exclusively on C as the language of choice.

No book can be all things to all people, so we recommend that you read this book with other books by your side:

sendmail, 3rd edition, by Bryan Costales and Eric Allman, O'Reilly & Associates, 2002.

sendmail Cookbook, by Craig Hunt, O'Reilly & Associates, 2003.

sendmail Performance Tuning, by Nick Christenson, Addison-Wesley, 2002.

UNIX System Administration Handbook, 3rd edition, by Evi Nemeth, Garth Snyder, Scott Seebass, and Trent R. Hein, Prentice Hall, 2000.

This book targets many potential groups that may prefer to write their own software rather than buy commercial software. Here are examples:

A business that sells antispam software and can benefit from creating products that fit with existing MTAs.

Small sites that can benefit from the low cost of self-written and self-maintained software.

Large sites that can benefit from the ease of upgrading an open source solution versus the high cost of commercial updates.

Financial institutions that require open source so that filtering behavior can be audited.

Whence Spam?

SPAM (all uppercase) is a trademark of Hormel Foods.[1] SPAM is most often followed by "Luncheon Meat."

[1] See http://www.spam.com for recipes and product information.

In contrast, the term "spam" (lowercase or lexically capitalized as at the start of a sentence) refers to unsolicited email. This latter term, "spam," is attributed to a Monty Python skit in which a group of Vikings sang "Spam, spam, spam," increasing in power and volume until it eventually overpowered all other conversation. The slang term "spam" was first used to describe commercial postings to Usenet (a global bulletin board system). Over time, the word "spam" became associated with unsolicited email.

In this book, when we use the terms "spam," "spamming," and "spammers," we refer to unsolicited email and not to the product described by the trademark SPAM.

Are Milters the Only Answer?

This book is primarily about how sendmail Milters can be used in the fight against spam. We have tended to gloss over other techniques in favor of a Milter approach. Although this can be viewed as a deficiency in a book about spam, a book about sendmail Milters should rightly focus on sendmail Milters.

In all fairness, however, there are many other techniques in use today, and many of them do a remarkably fine job of filtering out unwanted email. In the broadest terms, these other techniques can be grouped into five categories:

Domain Name Services (DNS)-based services that identify offending spam sites using IP numbers

DNS-based services that validate the identity of the sender

Products that employ Bayesian filtering to recognize and eliminate spam

Services that hand-screen URLs to eliminate spam

Companies that use postage as a means to fight spam

DNS-Based Services

When a site connects to sendmail for the purpose of sending email, sendmail can easily discover the IP number of the sending site. There are several Blackhole List (BL) sites on the Internet that allow sendmail to look up the connecting site's IP number to discover whether the IP number is that of a spam-sending site. Some BL sites use domain names, instead of IP numbers, to perform the same kind of screening. Both approaches can be very effective, but the complaint is often heard that they are too effective, often blocking legitimate email.

In addition to BL sites, there are sites that list the addresses of open relays and the addresses of hijacked PCs. Some also list all the IP numbers that are used by machines without fixed IP numbers. These sites store lists of thousands of entries and are used by mail-receiving servers all over the world.

The trouble with all these DNS-based approaches is that it takes only one complaint to cause a site or address to become listed as bad, but it takes a great deal of effort for a legitimate sender to get removed from a list. Thus, the chief flaw with BL screening is that it presumes a sender is guilty until proven innocent.

See http://www.mail-abuse.com/ and the other sites listed in the Bibliography for additional information.

DNS Sender Identification

Recently there have been a number of proposals to identify the machine that sends the email from a given domain as the official machine that is allowed to send email for that domain. Referred to as sender identification, these proposals are discussed in section 1.3 of this book.

Note that none of these proposals has yet seen wide acceptance, so it is too early to determine how effective they might be.

Bayesian Filters

A Bayesian filter divides email messages into small, well-defined units and then computes a signature and a spam rating for each unit. Some use individual words as the units, whereas others have discovered more efficient ways to categorize text to achieve a better recognition rate. But whatever the method, each unit is rated based on the likelihood that it is spam email.

Thereafter, all future pieces of email can be divided into units and their appropriate units compared to those previously rated. If the new email scores a significant spam rating, it can be branded as spam.

Bayesian filtering requires constant upkeep so that new patterns can be added to its history, but as spam email evolves and matures, classification becomes more difficult. When it is well maintained, Bayesian filtering can be very effective. But without constant upkeep, its efficiency quickly dwindles, consequently placing Bayesian filtering into the hands of a few specialty businesses that can perform the upkeep on behalf of customers.

See http://www.paulgraham.com/spam.html for a description of this technique, and http://www.mozilla.org/mailnews/spam.html for an example of its application to spam filtering.

Services That Screen URLs

In section 2.8 we describe the many ways spammers try to obscure URLs embedded in spam email.

We know of only one company that uses this technique to recognize and suppress spam, largely because URL detection, like Bayesian filtering, requires constant human monitoring and updates. Still, when frequently updated, URL detection can be extremely accurate.

See http://www.miavia.com/ for additional information about this approach.

Email Postage to Fight Spam

Some claim that the chief cause of spam is that sending email costs too little. Essentially, anyone with a low-cost DSL connection and a cheap PC can easily spam tens of millions of email addresses, and all the cost is borne by the receiving sites. The receiving site must pay not only for greater capacity (typically nine times the size that might be needed in a nonspam world) but also for spam-fighting software.

Email postage involves charging a small amount to send each piece of email. If it cost US$ 0.001 to send a message, for example, a spammer that sends ten million messages (small by today's standard) would find the total cost to be US$ 10,000.00not a trivial amount. Yet for the home user who sends only perhaps 10 messages per day, the daily cost would be only a penny.

See http://www.goodmailsystems.com/ for more details about this approach.

The Organization of This Book

This book is organized into four parts. Part I provides an overview of spam email, its impact, and the techniques used to fight it. Part II shows how to set up a bait site to gather spam email for analysis. Part III is a reference that describes the milter-library and explains how to use it. Part IV illustrates the code used to screen email for spam. Appendix A describes the source code featured in this book, and the Bibliography lists a number of useful references.

Acknowledgments

Special thanks must go to Greg Shapiro, who reviewed the Milter reference chapters to ensure accuracy. Thanks also to George Jansen, the best editor ever, who nit-picked every chapter for clarity of writing.

Scott Palmer used Part II to set up a bait machine to collect spam examples and provided helpful feedback. Chris Fedde reviewed Part II from the point of view of a system administrator and gave that part his stamp of approval.

About the Authors

Bryan Costales has been involved with sendmail for more than fifteen years and is author of the O'Reilly sendmail book (the "bat" book). He has also written books about C language programming and UNIX communications. He currently lives in the San Francisco Bay area with his dog, a fine German shepherd named Java.

Marcia Flynt has been involved in software engineering for more than twenty-five years and has managed email development for opt-in custom email businesses. She lives in Boulder, Colorado, with her husband and two children.

Part I: The Nefarious Spam Problem

Before we show you how to filter out spam, we will first explain the nature, history, direction, and costs of spam email. That is, if you will forgive an old adage, you must know your enemy before you can defeat your enemy.

This first part covers the following:

In Chapter 1, Gorilla Versus Guerrilla, we explain the origins of spam email and describe how it has come to represent an evil to society.

Chapter 2, The Characteristics of Spam Email, reveals (in gory detail) the tricks and techniques used by spammers to bypass spam suppression tools.

Chapter 1. Gorilla Versus Guerrilla

Over the past few years, spam email has evolved from nuisance to scourge, and now headlines speak of an "antispam arms race" and "millions of dollars lost in the battle with spam."

The fight against spam has become a full-blown war. The war is being fought between the huge Internet Service Providers (ISPs) on the one hand (the gorillas) and duck-and-run spammers on the other (the guerrillas), leaving most ordinary citizens in the role of a downtrodden populace. Our allies are the gorillas. Our enemies are the guerrillas. But beware that, in war, your friends can morph into enemies when you least expect it.

Section 1.1 shows how innocent businesses can suffer when a large ISP unilaterally decides to take drastic action to curb spam.

Section 1.2 describes how innocent mail recipients are targeted to receive bounced spam email, thereby causing the innocent to be blamed for sending the spam.

Section 1.3 shows the current favorite proposed standards for validating the sending host.

Section 1.4 shows that all spam is not just advertising. Some spam is an attempt to get personal information for identity theft.

Section 1.5 shows how to determine the cost of spam and explains how your business can benefit from spam suppression software.

1.1. When a Gorilla Sneezes

Large ISPs receive more spam than anyone else. The load of spam flowing into an ISP is so large, in fact, that some have resorted to extreme measures to handle the load. One large ISP, for example, found that it could easily and simply eliminate much of its spam email by rejecting connections from any machine that used nonfixed IP addresses.[1] Unfortunately, most PCs with digital subscriber line (DSL), cable, or dial-up accounts have nonfixed IP addresses.

[1] This form of IP blacklisting is growing in popularity, so the lesson here can be extended to ISPs of all sizes.

Still, this action might not have been so serious if the ISP had simply limited it to nonfixed IP addresses. But because of the difficulty of determining which addresses are fixed and which are not, the ISP expanded its rejection to all DSL, cable, and dial-up accounts, even if some of those PCs had fixed IP addresses. The effect on thousands of small businesses was disastrous.

Consider, for example, the case of a musician who sold his CDs on the Internet. He hosted his own website on a small PC running Linux in his garage. On an average day, he would sell five or so CDs from his site. He allowed purchases directly from his site with a local bank providing Web-based credit card services. Each time someone bought a CD, the musician's machine would automatically send an email confirmation.

One day, without warning, more than half of his email purchase confirmations began to bounce with a message that looked like this:

554- The IP address you are using to connect to us is a dynamic
554- IP address. We will not accept future e-mail transactions
554- from this IP address until your ISP removes it from its
554- list of dynamic IP addresses. For additional information,
554- please visit http://whynot.example.com.

This puzzled our innocent small-business person because he had a fixed IP address on a DSL connection with the local phone company. He visited the site mentioned in the message, but it simply restated the same information in different words. It instructed him to contact his ISP and ask it to issue him a fixed IP addressbut he already had a fixed IP address.

He spent a week on the phone with his ISP (the phone company), but it could not help him. He looked up his problem on the Web but found no solutions there either. Finally he sought help from a sendmail expert, who recommended he use sendmail's SMART_HOST option to relay mail through his ISP's mail server.

After a week of long distance calls and lost sales, he was finally able to send email again to the large ISP that had blocked him. He contacted it to see whether he could be reimbursed for lost income but was referred to its legal department, which never answered.

This is one reason to beware of large ISPs. Their policies are often set without regard to the harm they cause others. Unless they find a better way to block spam, they will continue to act as if they own the Internet, small businesses be damned.

Unfortunately, just as all media are slowly being gathered into the hands of a few large corporations, email is slowly collecting in the hands of a few large ISPs. To witness some threats on the horizon, just look at the proposals we document in section 1.3. When it comes to your rights to send email, those rights are being progressively eroded.

1.2. When a Guerrilla Masquerades as You

Spammers hide behind hijacked PCs and fire at email recipients and ISPs. In this war, no one is safe from spam email, but spam affects the public most when spammers use an innocent person's email address as the bounce address.[2]

[2] A bounce is a mail message that is returned to the sender because delivery failed.

The bounce address is the address specified in the MAIL FROM part of Simple Mail Transfer Protocol (SMTP) email transmission. That address is called the envelope sender address and can be different from the header sender (the From:) address. Normally, the envelope sender address is never seen by the recipient. It is intended merely as a way of specifying a bounce address that is different from the reply address.

If a spammer were to send 10 million spam messages and if 10 percent of those were to bounce, the spammer would have to be prepared to handle 1 million bounce messages. But spammers do not care whether an address is bad. They never even remove bad addresses from their mailing lists. So, rather than handle bounces themselves, they specify fictional addresses as envelope sender addresses. This would be harmless enough, but they sometimes use real addresses instead of fictional addresses. And that is the core of the problem.

If your address is used, your mailbox suddenly fills with thousands of bounce notification messages. You are overwhelmed, and you don't even have someone to complain to.

So spam becomes increasingly harmful. The big ISPs refuse email from small sites. Everyone receives dozens if not hundreds of spam messages a day, and now everyone runs the risk of receiving thousands if not millions of bounce messages out of the blue and with no recourse.

And to add insult to injury, many sites consider the envelope sender to be the source of the spam email. Not only do you risk receiving many bounce messages, but also you may find your site blocked because you will be considered the source.

1.3. The Major Proposals for Standards

Along with the growth of spam comes a growth in proposals for eliminating it. The four most popular share a common characteristic in that they require the sending site to set up a TXT record under Domain Name Dervice (DNS):

Atkinson Caller-Id, sponsored by Microsoft, requires the sending site to identify its outbound Mail Transfer Agent (MTA) by creating an extensible markup language (XML)-encoded TXT record for the host _ep in that domain.[3]

[3] As of this writing http://www.ietf.org/internet-drafts/draft-atkinson-callerid-00.txt.

Domain Keys requires the sending site to add a special header in its outbound email (which contains a digitally signed message signature) and to create a TXT record for the special domain _domainkey that provides the public key that corresponds to the private key used to create the special header.[4]

[4] Domain Keys is a trademark of Yahoo!, and documented, as of this writing, at http://www.ietf.org/internet-drafts/draft-delany-domainkeys-base-00.txt.

Sender Policy Framework (SPF) requires the sending site to identify its outbound MTA by creating a TXT record for a special PTR record in the _srv._smtp._perm domain.[5]

[5] An executive overview is available at http://www.microsoft.com/mscorp/twc/privacy/spam_senderid.mspx.

MTAMARK (Marking Mail Transfer Agents) is a simple authentication scheme that uses marking of hosts in reverse DNS (in-addr.arpa zone) to allow the receiving mail transfer agents to decide whether or not the connecting (sending) host is a designated mail transfer agent.[6]

[6] As of his writing, see http://www.space.net/~maex/Drafts/dns-metamark/ for more information.

The problem with all proposals that promote a TXT record type as a solution is that they favor the big ISPs over the small-business person. Unless standards are developed to force ISPs to allow any DNS customer to create TXT records, these proposals will benefit only operations large enough to run their own local name servers.[7]

[7] As of this writing many suppliers of IP numbers don't let the customer associate a TXT record with a PTR record. That policy is not about to change anytime soon.

1.4. Email Fraud

Spam email is now more than a major annoyance. It has evolved into schemes intended to bilk the recipient out of money or out of information needed to enable identify theft. Three major forms of email fraud are making the rounds.

1.4.1. The Nigerian Fraud

In this fraud, mail is sent claiming that the sender has millions of dollars in an account that must be shipped out of his country. The recipient is asked to cover the expenses (a mere few thousands of dollars), and in return the millions will be transferred into the recipient's bank account. If the recipient is foolish enough to reply, the evil game begins. The recipient will be asked to cover increasing expenses before the millions are transferred and eventually may even be conned into flying to that country, whereupon even more terrible things will happen.

Although this scam has become known as the Nigerian fraud, it can be staged from almost any country. Organized crime is reputed to be behind this scam, so care must be exercised when dealing with it.

Warn your users that such frauds are dangerous and that they should never reply to them.

Warn your users that such frauds can appear in many other guises, such as a scam where a valuable item is alleged to be held in a bonded warehouse, and the user is invited to spend money to get the item released.

This kind of scam is officially known to law enforcement agents as Advance Fee fraud or 419 fraud, after the section of the Nigerian penal code that specifically prohibits it. In response to scams, a movement has formed called 419 baiting, which seeks to exhaust scammers' resources and to gather evidence about them by replying in a carefully crafted manner.

1.4.2. Theft of Bank or Credit Card Information

Another email scam of growing concern involves email made to appear as if it is from a bank or credit card company. It generally consists of a warning that the recipient's account will be closed or blocked unless personal information is validated. A link is provided for the email recipient visit what appears to be a legitimate website.

But the email and the website are only masquerading as legitimate; they are really maintained by someone wanting to steal personal information. Anyone who is tricked into responding will be asked for a social security number, date of birth, bank account numbers, and so on. Clearly such a scam is intended either to steal money from the recipient's checking account or to perform identity theft.

No bank or credit card company would ever send email like that sent by such scam artists, and such email should never be replied to; nor should the links in it ever be clicked on.

Educate your users to report such scams to the fraud department at the bank or credit card company being impersonated.

Educate your users to never click on such suspect links. They risk unknowingly downloading spyware onto their machines, which can capture keystrokes, including user names and passwords.

1.4.3. Theft of Passwords

Another form of spam sends you information about a new "free" membership in some organization, club, or service. The email often includes a default user name and password to use. If the recipient clicks on the link provided, a profile page appears requesting personal information, a new user name, and a new password.

Although this may seem innocent, spammers know that most people use only a single user name and password for all their web accounts. By gathering the recipient's preferred user name and password, they now have the information needed to connect to various bank and credit card sites to see whether that user name and password work there, too.

Educate your users to never give a user name or password to anyone.

Educate your users to always use a different user name and password at each site, and show them a secure way to keep track of their user names and passwords.

1.4.4. Viruses and Worms

Microsoft Windows PCs are the target of a growing class of viruses and worms that perform keystroke logging. Here, a program is sent via email using various flaws in various Hypertext Markup Language (HTML)-enabled email readers to execute a program without the user being aware. That program records all the keystrokes made by the user and sends a record of those keystrokes to the scam artist.

Because many users of Microsoft Windows PCs connect to bank sites to perform banking and other financial transactions, the record of keystrokes contains user names and passwords used to access those sites. It is a simple matter for the scam artist to extract that information and use it to steal money or to perform identity theft.

A Windows PC should never be connected directly to the Internet. Instead it should be behind a firewall or router that restricts inbound and outbound traffic.

All Windows PCs should be screened for viruses and worms daily using software that is available for this purpose.

Note that viruses and worms can also turn a PC into a zombie mail machine: a machine that sleeps until the spammer wakes it and attempts to relay spam email through it.

1.5. The Cost of Spam Suppression

The ongoing increase in the levels of spam email have spawned a wide range of companies that attempt to aid businesses in their battle against spam. The products are generally expensive and affordable only for medium to large businesses. However, many open source products are also available for use by small businesses and individuals.

To determine how much spam suppression you can afford, you first need to understand the cost of spam. At a small site, it may be sufficient to merely interview each user to get an idea of how much productivity is lost. At a large organizationsuch as an ISP, a university, or a large business, you may need to perform actual complex cost analyses to determine whether the benefit of spam suppression offsets its cost.[8]

[8] Search for spam calculators on the web.

Calculate your monthly equipment cost (its purchase price amortized over its life).

Calculate your monthly cost of support on that equipment (including software upgrades, training, book support contracts, and so on).

Calculate the monthly cost of interest on the loans obtained to purchase equipment or to run the company.

Calculate the monthly cost of Internet access.

Calculate the cost of power to run your computing infrastructure (include lighting and environmental controls).

Calculate or estimate the number of spam messages received per hour and the average amount of time spent handling each spam message. (It takes only a half second to press Delete, but how much time is spent deciding whether something is really a spam message?)

These figures allow you to calculate the cost of unfiltered spam and the loss of revenue it creates. If the cost of spam suppression software is less than the money saved by the resulting reduction in spam, the spam software may be cost-effective for your company and may produce a positive return on investment.

You should note, too, that the cost of spam can vary depending on the nature of your business, and the benefits of suppression are not always easy to recognize. An ISP, for example, can lose customers when the volume of spam becomes oppressive, but it may find effective spam software too expensive. But spam software that offers per-user configurations might justify its cost because the ISP can sell access to custom configurations to its customers (thereby offsetting the cost of the spam software and making it affordable).

Performing a cost-benefit calculation for your site may seem daunting, but it is necessary. Otherwise, you might fall into a two-sided trap, where spam costs your company money but weak spam suppression costs it even more. Remember that spam suppression must have a financial benefit.

Note that the growth of spam is exponential over time. That is, if your site receives 1,000 spam messages today, next year it will receive 10,000, and the year after that, 100,000. This means that today's software solution may not be cost-effective next year. In the war against spam, be prepared to bring out bigger guns each year.

Finally, note that false positives (meaning that nonspam email is wrongly classed as spam) can adversly affect the viability of your business. Because inexpensive spam supression may have a much higher false positive rate than more expensive supression, you should consider the impact of a large number of false positives before selecting a spam supression technique, software, or service.

1.6. Vikings

Who could have imagined on December 15, 1970, that the "Spam Song," written and composed by Terry Jones and Eric Idle (with help from Fred Tomlinson) on Monty Python's Flying Circus, was anything more than a passing joke? On that show, a band of "Viking warriors" chanted "spam, spam, spam" over and over until all normal communication became impossible.

Now, more than thirty years later, the "spam, spam, spam" of that song has become emblematic of a worldwide problem. The volume of spam email in Asia, for example, recently exceeded the volume of spam in the United States. Clearly, the problem has grown beyond the boundaries of any one country.

A true global solution to the spam problem will need to be implemented at the treaty level if it is to have any significant impact. There are currently few treaty negotiations under way to solve the spam problem, but worldwide interest is growing.

Chapter 2. The Characteristics of Spam Email

It is easy for a person to look at a piece of email and say, "This isn't something I asked for. It looks like an advertisement, and I don't want it, so it must be spam." But although it is easy for humans to recognize spam, it is much harder for software to recognize it. And, after all, the point of spam-blocking software is to eliminate the need for humans to recognize spam.

In this chapter we discuss spam as it appears to software and not as it appears to a human. For example, Figure 2.1 shows how a piece of spam email might appear to you when viewed in your email reader.

Figure 2.1. How spam email might appear to a human.

The message in Figure 2.1, however, might look like the following to software:

<a href="dvds.example.com"><img src="dvds.example.com/images/em1"></a>

This is a form of HTML code called a clickable link, and although software lacks human eyes, it can easily detect that this is a clickable link.

In this chapter we reveal, at the code level, how spam email is often internally structured. But bear in mind that spam email is constantly changing and evolving, and the lessons you learn in this chapteralthough forward-lookingare not the total picture. Thus, we encourage you to collect spam to examine for yourself. By doing so, you will note trends not covered here.

One way to do this is to collect spam in your normal mailbox, but because this can get a bit messy, we instead advise you to set up a bait machine (see Chapter 3).[1]

[1] Or you may prefer to simply collect spam in your own mailbox.

2.1. Connection Behavior

Spam email is sent like all other email. The sending site connects to the local sendmail daemon and waits for the local sendmail to say it is ready.

Sender connects and waits for:
220 local.hostname...

Here, the line beginning with 220 is from the local sendmail and tells the sending client that it can begin to send messages.

A well-behaved client will wait for the 220 before sending anything, but some spamming software will not. To speed up the spamming process, some spamming software will send the entire message without waiting for the 220.

Sender connects and immediately sends the whole message.
Then it disconnects and moves on to the next host in line.
220 local.hostname...

This works for spammers because email runs on top of a protocol called Transmission Control Protocol/Internet Protocol (TCP/IP). With TCP/IP the operating system sets up a buffer to hold the incoming message, and because sendmail looks only at the buffer, rather than at the actual connection, it does not see that the inbound sender has already sent everything. Thus, sendmail will obligingly accept the message and probably deliver it.

Fortunately, version 8.13 of sendmail adds a new feature to catch just this sort of ill-mannered behavior. Called the greetpause FEATURE, it causes sendmail to sleep briefly before sending the 220. After the sleep and before sending the 220, sendmail checks to see whether input from the sender is already present in the buffer. If it is, sendmail thereafter rejects all SMTP commands from that sender, for that connection, issuing a 554 error.

Naturally it would seem better to just drop the connection, but sendmail dares not do that because doing so could leave unread information in the TCP buffer, eventually filling that buffer.

In the future, spamming sites will probably adapt to this form of detection, modify their behavior, and start sending only after receiving the 220 greeting, after the EHLO, after the MAIL FROM, or after a RCPT TO. To adapt, spamming software will need to move its asynchronous attack further into the synchronous protocol. As this competition continues, spamming software will slowly be forced to evolve into normal, well-behaved email sending software.

2.2. Relaying through MX Servers

It is important to realize that mail is not sent directly to the host whose name follows the @ in the email address. Instead, that host name is first looked up using DNS to find the names of its MX (mail exchange) servers. To illustrate, consider the email address bob@example.com. Before mail is sent to example.com, that host's MX records are looked up:

example.com.   MX 10 mail.example.com.
example.com.   MX 20 bob.fallback.host.

Here, the host example.com has two MX records, each beginning with the literal MX. Each MX is then followed by a number. The lower the number, the greater the preference given to deliver to that particular record. Following each number is the name of the host to which mail should be sent. Here, mail should be sent to mail.example.com first, and if that fails, it should be sent to bob.fallback.host instead.

MX records exist as a safety net so that mail will always be delivered somewhere. The idea in our example is that mail will be held by bob.fallback.host in the event that mail.example.com is down. Later, when mail.example.com returns to service, bob.fallback.host will forward the delayed mail to it.[2]

[2] Actually, it is perfectly legal to have only one MX server (but it is discouraged to have none). However, we use two and more here for the sake of example.

Because normal email is delivered using the order specified by such MX records, mail from fallback hosts should be rare. But many spam sites take advantage of this behavior to avoid being rejected based on their connecting IP numbers. Their assumption is that spam email screening normally done on a main mail machine will not necessarily be done on a fallback machine, andbecause most fallback services are provided by ISPs and so do not share the same policy or software of the main mail machinethis assumption is frequently correct.

Thus, many spam sites look up MX records and purposely send their spam email to the host with the highest number (lowest preference). When they do this, the main mail machine sees only the IP number of the fallback host, and the IP number of the spam sender remains hidden.[3]

[3] It is actually available in Received: headers, but those headers are difficult to parse and cannot always be trusted.

If you intend to filter email based on connecting site IP numbers, you will need to arrange your MX records to specifically trap this sort of IP number subterfuge. Consider again our example of the site example.com, which has MX records that look like this:

example.com.   MX 10 mail.example.com.
example.com.   MX 20 bob.fallback.host.

If example.com is set up like that, much spam email will be sent to bob.fallback.host as spammers attempt to avoid IP-number suppression. Instead, example.com might be set up to detect spam sending IP numbers, like this:

example.com.   MX 10 mail.example.com.
example.com.   MX 20 bob.fallback.host.
example.com.   MX 30 mail.example.com.
example.com.   MX 40 mail.example.com.

Now, normal mail will be delivered just as before, and if mail.example.com is down or busy, mail will instead be delivered to bob.fallback.host, also just as before. If both are down or busy, mail will be deferred at sending sites. But now, spam sites that send to the highest MX record will instead send directly to mail.example.com, where IP detection and suppression is running. This minimizes the number of spam sites that will connect to bob.fallback.host.

Note that, to accomplish this, we use two high number records, both pointing to mail.example.com. Clearly, a savvy spammer would recognize this trick and adapt to it. A better way to organize MX records requires you to control your own domain records. Consider another way to set up the preceding records:

example.com.   MX 10 mail.example.com.
example.com.   MX 20 mx1.example.com.
example.com.   MX 30 mx2.example.com.
example.com.   MX 40 mx3.example.com.

Here, all MX hosts are under your control, so the spam sender must look up IP numbers for these records in order to figure out where each is actually pointing. In this case, the IP number for mx1.example.com would be the same as that for bob.fallback.com, and the IP numbers for mx2.example.com and mx3.example.com would both be the same as the IP number for mail.example.com.

Modern operating systems allow each network interface to be assigned multiple IP numbers. If yours is one such operating system, consider assigning a different IP number to each of mx2.example.com and mx3.example.com, where those two differ from the IP number for mail.example.com.

The more work you impose on spam-sending sites, the less effective they become and the more costly their operation will be. Because spam email is a low-margin business, even an incremental increase in their costs may drive some out of business.

2.3. Falsifying the Envelope Sender Address

All Internet email is sent using a protocol called SMTP (Simple Mail Transfer Protocol). In SMTP, mail is sent in units called envelopes. First the address of the envelope sender is sent, followed by one or more envelope recipient addresses. Finally the actual message is sent, headers and body together. Note that the final recipient sees only the headers and body. Envelope information is generally visible only to sendmail and its Milters.

When a user replies to an email message, the reply is generally sent to the header senderthe address listed in the From: header. But when a message is bounced, the bounce reply is generally sent to the envelope sender address.

Spam sites have learned about this property of email and can use it to their advantage. Spammers know that a large percentage of the email they send will bounce, but they do not care about bounces. Even if an address in a list continues to bounce, spammers will not remove it, because the cost of cleaning their lists is too high.

Some spam detection software looks at the envelope sender on the theory that the envelope sender will point to the spam-sending site, because that is where bounced email goes. But to avoid this type of detection (and also the potentially high volume of bounce return messages), spam-sending sites often use a false envelope sender. In other words, they lie about their identity. And not only do they lie, they also forge. That is, because many MTAs (sendmail included) reject the envelope sender if the host part of the address does not exist, spammers often use real envelope sender addresses (but not their own).

The result is that some poor user somewhere in the world will receive all of a spam site's bounces. This is certainly evil, but it is not illegal. There is nothing in the SMTP standard that can cause a sending site to tell the truth in the envelope sender specification.

Because spamming sites never clean their address lists and because spamming sites often lie about the envelope sender, you should probably never bounce (reject) spam email. If you do, you may be unwittingly adding to the problem by bouncing to an innocent user. In general, it is better to accept and discard spam email or to accept and archive it.[4]

[4] We fudge for simplicity here. Actually a bounce occurs only when failure notification is mailed back to the envelope sender. A rejection does not necessarily generate a bounce.

An alternative (if you have some way to keep count) is to bounce spam the first time it is received from a site. In that way, if good email is bounced, the sender (a real person) can contact someone at your site to have the problem corrected.[5]

[5] In your bounce (reject) message, always try to include helpful information that can aid in correcting a wrongful rejection of good email.

2.4. Disguising the Subject: Header

It is tempting to examine the Subject: header to identify spam email. Unfortunately, the Subject: header is entirely under the spammer's control and thus can never be trusted.

But not all spam senders are created equal. Some will do everything possible to make the Subject: header appear benign, but others are inept and will provide clues in that header.

Clearly, it can benefit you to try to pick off these inept spammers. But first, understand that, by doing so, you run the risk of identifying good email as spam. To illustrate, consider the following examples of actual spam Subject: headers:

Subject: Learn how thousands are making a fortune with eBay...
Subject: Earn While You Learn  sjdv
Subject: Visit PlayboyPlus for New Playmate Pics
Subject: Discreet delivery
Subject: naked and cute. watch my movie.
Subject: ,^refina'nce now and ;save-    r
Subject: Extend your auto warranty, free quote
Subject:
Subject: Spring Special
Subject: <a href="http://www.example.com/mp/axis/">or not
Subject: get the edge
Subject: Re:  jock
Subject: V1AGR*A final1y fOund a t0ugh cOmpet1t0r -- CIAL1-S
Subject: Email Verification!Please take a look.  wyl
Subject: Celebrity Secret to looking young!
Subject: You've seen ads for Levitra on TV, Does it Work?i s k
Subject: Refinance your Home and skip a payment
Subject: Important notify about your e-mail account.

Although all of these appear to be spam Subject: headers, in actuality the two shown in bold were real email messages to users who wanted to receive them.

In general, it is unwise to employ a spam-screening strategy that examines only Subject: headers. Such screening is prone to errors, and as spammers mature, the use of such headers will likely decline. We expect spam email Subject: headers of the future to look more like these:

Subject: Yesterday was fun
Subject: Re: updating my address
Subject: Thanks again!
Subject: Email Statement
Subject: next appointment
Subject: I hope so

Another trick used by spam senders is to base64-encode the Subject: header. This has the advantage of making it hard for detection software to see the header, while still allowing the end user to see a readable subject line. Such base64 code might look like this:

Subject: =?iso-8859-1?b?OTAgZGF5cyB0byB5b3V0aA==?=

The end user will see this:

Subject: 90 days to youth

Unfortunately, however, base64 encoding of a Subject: does not always indicate a spam message, because it is also the only method that allows some foreign languages to insert legal headers into email. For example, in the United States a user might consider =?.GB2312? (which indicates simple Chinese) to be a solid indicator of spam email. But in China, that same encoding might indicate good email, whereas =?US-ASCII? might indicate spam. Thus, again, we recommend that you screen Subject: headers sparingly, if at all.

2.5. Camouflaging the HTML Body

HTML (Hypertext Markup Language) is a language that describes how text and objects should be displayed. HTML also controls the interaction between the user and the message. Most, but not all, email is formatted using HTML. This is a boon to users and businesses, which benefit from the ability to display attractive fonts and images as part of an email message. But it is also a boon to spam email senders because HTML-capable mail programs are liberal in what they accept, and that allows spam email to be easily disguised.

In this section we first describe the use of HTML comments as a means to disguise content. Then we describe two forms of encoding often used to disguise content: character-entity encoding and URL encoding.

2.5.1. HTML Comments

Consider this HTML comment:

<!-- this is a comment -->

An HTML comment begins with the four-character sequence <!-- and ends with the three-character sequence -->. Comments may span multiple lines and may enclose and hide arbitrary text.

Early in spam history, spammers discovered that words could be broken up with HTML comments, making those words appear normally when displayed. Here's an example:

V<!--THYME-->ia<!--ONION-->gr<!--ALMOND-->a

Here, the word "Viagra" is hidden among the names of various foods. But because the comments are ignored by HTML-capable mail programs, this text is displayed like this:

Viagra

More recently, spammers have discovered that any unknown HTML keyword in angle brackets acts just like a comment.[6] Thus the comment technique has been modified by spammers and now looks like this:

[6] We warned you that HTML-capable mail programs were far too forgiving.

V<xyxxx>ia<ftmmwe>gr</gvwquu>a

Clearly, to eliminate comments from HTML you must now possess a full understanding of valid HTML keywords. For example:

<font>       Yes, a keyword
<fonts>      No, not a keyword, but instead a comment

Similarly, spam-screening software needs to know that <p>, <b>, and <i>, for example, are HTML keywords, but <q>, <t>, and <c> are not.

To make things more complicated, any words that begin with an ! or a ? and any non-HTML keywords that begin with a / character are also treated as comments:

<!xxx>        Also a comment
<?xxx>        Also a comment
</xxx>        Also a comment
</font>       A keyword, not a comment

Be aware, too, that intervening newlines may also be included in comments. Consider, for example, the following:

V<<OICE
GIMM>>I
A<<LIVE
ABLO>>G
R<<IVER
ITUN>>A

Here, we show the effect of both unbalanced angle brackets and a newline between the matching angle brackets of the comment. When viewed by HTML-capable mail programs, the preceding lines will look like this:

V<>I A<>G R<>A

In short, spammers can play many tricks to disguise HTML. Newlines may be included in comments. Extra angle brackets may become part of revealed messages.

As an aside, one trick that may be useful when you attempt to match angle brackets is to scan from the left to the right and look only ahead, never back. To illustrate, examine the following HTML fragment, in which we have indicated matching angle brackets using bold font:

<xxx<yyy<zzz<>www>
<aaa<bbb>ccc>ddd>eee>

Reading from left to right, note that the first left angle bracket starts the xxx comment. The following right angle bracket (even if there are intervening left angle brackets) ends the comment, revealing www as clear text. The next left angle bracket (that is not part of the comment) begins the aaa comment and ends just before ddd. The result, when viewed on HTML-capable mail programs, will look like this:

zzz> ccc>ddd>eee>

2.5.2. Character-Entity Encoding

Character-entity encoding provides the means to include special characters in HTML documents. It can be invoked in either of two forms. Both begin with an ampersand character and end with a semicolon character. In between are either a keyword, such as lt, or a literal # followed by a three-digit decimal value. For example:

&lt;           A Keyword that produces a < character.
&#060;         # and three digits produces a < character.

Without character-entity encoding, the following special characters (to list only two) would not be possible:

&copy;      Produces the © symbol.
&#159;      Produces the Ù symbol.

The HTML standard allows all characters, and not just special characters, to be encoded in this way and provides a handy way for spammers to hide otherwise obvious information. Consider this example:

<a
href="mailto:&#098;&#111;&#098;&#064;&#101;&#120;&#097;&#109;
&#112;&#108;&#101;&#046;&#099;&#111;&#109;">

After decoding, the obscured email address is revealed to be an easily recognized address:

<a href="mailto:bob@example.com">

Clearly, it is necessary to decode character entities before attempting to parse HTML for clues about spam. In Section 11.4 we show you the C language code you can use to decode character-entity-encoded text.

2.5.3. URL Encoding

A URL (Uniform Resource Locator) is a web reference. Every time you click on a link with your web browser, you are invoking a URL. The URL standard assigns special meanings to special characters. For example, a ? character is used to separate a web reference from its arguments:

www.example.com/cgi-bin/search?name=bob

To prevent confusion, such special characters, when not in their special role, must be URL-encoded. To URL-encode a character, first prefix it with a % character, and then convert the character itself into its two-digit hexadecimal ASCII value.[7] To illustrate, consider a search program whose name includes a ? character:

[7] On UNIX machines, conversion from ASCII to hexadecimal is usually documented in the file /usr/pub/ascii. But it might alternatively be found in /usr/share/lib/pub/ascii or /usr/share/misc/ascii. Under Red Hat and Fedora Linux, it is found in the online manual ascii(7).

big?search.pl

To use this program name as part of a URL, the ? character must be converted into its hexadecimal value, 3F, which must be preceded by a % character:

www.example.com/cgi-bin/big%3Fsearch.pl?name=bob

But as we have shown, almost anything will be misused by spammers to hide message content. Consider, for example, the following obscured web reference:

href="%77%77%77%2E%65%78%61%6D%70%6C%65%2E%63%6F%6D"

Clearly, before you can screen a message for spam content, you may need to URL-decode all such URL-encoded expressions.[8] The example, when decoded, will look like the following:

[8] When International Domain Names (IDNs) come into full use, decoding may no longer remain optional.

href="www.example.com"

In section 11.5, we show you the C language code you can use to decode URL-encoded text.

2.5.4. The Order of Encoding

Character-entity decoding is performed by a web browser before it displays the text on the screen, but URL decoding is performed later, as a result of an HTML action, such as clicking on a web reference. Thus, web references may be first URL-encoded and then character-entity-encoded, and the result will be understood by HTML-capable mail programs. When decoding email to examine it for spam content, always character-entity-decode first and then URL-decode.

2.6. Attempting to Fool Signature Detectors

Signature detectors work by computing a value for a block (or chunk) of text in a message. For example, suppose a message contained this text:

We sell herbs at the lowest price you will
ever find on the net.

A signature-generating program might create an expression that represents this text, such as the following:

244372015810742154622705

This "signature" is saved in a file or database. Later, when another message arrives, its chunks are also given signatures. Then each signature is looked up, and, if it is found, that serves as an indication that the message may have been seen before. Clearly, several signatures will have to match so that one message may be considered significantly like another.

Similar spam detection software recognizes phonemes (distinct parts of words) instead of chunks of words. Other software performs permutations of the divided text to increase the number of signatures used, and still others perform statistical analysis on individual words and then store the probabilities.

But all these forms of spam detection share the common method of examining the message's text. Spammers, recognizing that text analysis is being used, have responded by adding large chunks of random text to each message.

Random text can be actual words and names in random order:

dissonant deanna heron aphasia restaurateur circulate
controllable corporeal cranston giuliano helmholtz bertha
albany shank eye asphyxiate commentary gaston aide filler
chipboard prostheses perturb cryptographer atlantic bernice

Random text can also be random combinations of characters:

eyhydxre yaceyaxv gesmveu vmlpv wmgrxa drgcah mqbjneq
wbfqzkmwr fdbkqogtgzwv lsunhut wuwnp- hivrkef dhdpfhcu
ndowgkx cjxrofun yepjhxp rhbxag ncgvmv

Random text can also be a solid stream of random characters:

hyfaqjimgdalmrymmolaktivajvctikdhpfzaplgumufsvtjgu
tccqenngjwtodktenkrvefpmkiherqymsccysqfbmapkkvxuo
tauimesuijmivglyefqlgclxvyjsxfgsfadrhvnrhzacfncmssx
awlzrjilipsbuuenbbdtievlmkpycivegidatnlccffyajnbmqw

Finally, random text can also be an actual abstract from real text, where a different abstract is used in each message:[9]

[9] As if spammers would care, random insertion of actual text abstracts runs the risk of copyright infringement.

Mother called me home that night with a shout
that told me there was trouble. "Mom," I yelled,
pounding up the back steps. "What's wrong, Mom?"

When you parse a message to detect spam, your goal is to find a way to skip such random text and to run signatures only on the portions of the message that do not change. Portions of a message that should be checked include the following:

Common images

Web references

Email addresses

Phone numbers

2.7. Unnecessary Encoding

MIME (Multipurpose Internet Mail Extensions), a protocol that defines how various objects may be embedded in email, is the standard used to create attachments. MIME also provides the standards for encoding attachments for mail transport.

When an attachment is binary (such as a photograph or a word processor document), it must be base64-encoded before email transport. When lines are overly long (as is much HTML code), that text must be quoted-printable-encoded before email transmission. Unfortunately, spammers have adopted these two valuable forms of encoding to help disguise spam email content.

The kind of MIME encoding used in an email message (if any) is specified by the Content-Transfer-Encoding: header:

Content-Transfer-Encoding: base64
Content-Transfer-Encoding: quoted-printable

We will show how to decode base64 in section 11.2, and how to decode quoted-printable in section 11.3. Here we illustrate how they are used to disguise spam email. Consider this example:

Content-Type: text/html
Content-Transfer-Encoding: base64

PEZPTlQgZmFjZT0iVmVyZGFuYSIgc2l6ZT0zLjU+DQpDaW5kZXJlbGxhISAgUGV0ZXIgUGFuISAg
dWxsIG9mIGl0OyBlYXQgaXQgZXZlcnkgZGF5IGZvciBicmVha2Zhc3Q=

Here, the message content is of type HTML (the text/html), but rather than allow the HTML to be transmitted as is, the spammer has base64-encoded it to make it difficult to recognize. Clearly, you must first decode this base64-encoded text before you can screen it for spam.

In addition to its primary use (encoding long lines for transport), quoted-printable can be used to obscure text so that it is difficult to parse:

Content-Type: text/html
Content-Transfer-Encoding: quoted-printable

<br>
<a href=3D"http://bob.example.com/py/">
<img src=3D"http://amy.example.com/2/"></a>

Here, the internal equal signs have been turned into =3D expressions using quoted-printable encoding. You need to decode them back into equal signs before screening the message for spam content.

2.8. Grokking the Site

Spam messages generally contain a method for the recipient to contact the spammer (or the business on whose behalf the spam is being sent). Some offer an email address and others a phone number, but most try to get the recipient to go to a website. To do this, spammers include a clickable web reference (URL) in the body of the spam email. But because spammers seek to keep their identities a secret, they generally try to disguise all web references. In this section we show some of the tricks they use.

First note that a web reference is usually specified as part of an HTML a command:

<a href="http://www.example.com"> visible text </a>

The <a (followed by a space) begins the command. The <a is followed by one or more keywords particular to that command, all terminated by the > character. The HRef= keyword indicates a web reference (URL). Following the > is the actual text that will appear on the screen and that must be clicked to invoke the web reference. The </a> ends the command.

In the following, we examine the pieces of this web reference one item at a time.

Section 2.8.1 examines the leading <a part.

Section 2.8.2 explains case insensitivity.

Section 2.8.3 examines the http: part.

Section 2.8.4 shows how email addresses can mask the www.example.com part.

Section 2.8.5 shows that IP numbers and hexadecimal representations of IP numbers are used by spammers to disguise host names.

Section 2.8.6 shows how the www.example.com part can be hidden with redirects.

Section 2.8.7 shows how the www.example.com part can be ridiculously stretched out to say aa.bb.cc.dd.ee.ff.gg.hh.ii.jj.kk.example.com.

Section 2.8.8 shows how the www.example.com part can be hidden behind CNAME records.

Section 2.8.9 shows that URLs can also be used as comments.

One common method of using URLs to fight spam is to record the host names from those URLs in a database. Each time a new piece of email shows up, the URL is found and the new host name is looked up, and if the new name is found in the database, it is interpreted as spam. In addition to host names, a well-designed antispam database includes IP numbers.

To illustrate, consider a database that contains the host name spam.example.com and that host's IP number (192.168.33.44). When new mail arrives, the host name in the arriving mail is looked up in the database. If that new host also has the address 192.168.22.44, it too is rejected even though the new and old host names may be different.

2.8.1. The HTML Keyword

The <a command always contains a web reference, but other HTML commands can also contain web references. The <a command indicates a web reference by using an href= expression. For example:

<a href="http://www.example.com">

But other HTML commands use different expressions to indicate the web reference. Table 2.1 lists the HTML commands that allow URLs and the expression used by each.

Table 2.1. HTML Commands That Reference URLs

Command

Expression

Description

<a

href=

Create a hyperlink (HRef=) or identifier (name=) in a document.

<applet

codebase=

Define an executable applet with a document.

<area

href=

Define a mouse-clickable area within a map.

<base

HRef=

The base for all URLs in the document.

<body

background=

Set background to a URL.

<del

cite=

Citation reference for deleted information.

<embed

src=

Embed an object in a document.

<form

action=

URL to use on submission of a form.

<frame

src=

Define a frame within a frameset.

<iframe

src=

Embed a frame inside a document.

<img

dynsrc=

Specify a video clip to play.

<img

lowsrc=

Specify a low-resolution image to preload.

<img

src=

Specify an image to load.

<img

usemap=

Coordinates list for a map.

<ins

cite=

Citation for inserted commentary.

<input

src=

Image to select for an input choice.

<isindex

action=

Create a searchable document.

<link

HRef=

Create an interdocument link.

<link

src=

Specify an external style sheet to use.

<meta

url=

Reference for an HTTP refresh.

<object

classid=

Identify the class of an object.

<object

codebase=

Source of the code base for the object.

<object

data=

Source of data for the object.

<object

name=

Source for the name of the object.

<object

usemap=

Specify the image map to use with the object.

<q

cite=

Citation for the enclosed quotation.

<script

src=

Source for external language code to run.

<table

background=

Source of background image to load.

<td

background=

Source of background image to load.

<th

background=

Source of background image to load.

<tr

background=

Source of background image to load.

2.8.2. Just in Case

Before digging deeply into URLs, we first need to comment on one of the (sometimes overlooked) characteristics of HTML in general.

First, note that all HTML commands and the URLs they reference are case insensitive. That is, all the following are the same:

<a href="www.example.com">
<A HREF="WWW.EXAMPLE.COM">
<a HrEf="wWw.ExAmPlE.CoM">

Note, too, that host and domain names are also case insensitive.

2.8.3. The Protocol Specification

A protocol can be thought of as a language used by programs to communicate over a network connection.[10] Usually a request is sent from the client software to the server software, and an answer (or reply or data) is returned. The most common protocols are http (Hypertext Transport Protocol), https (HTTP with Secure Sockets Layer, or SSL), ftp (File Transfer Protocol), and file (for viewing local files).

[10] Some, such as file, interact with the local file system.

In a URL, the expression that identifies the protocol prefixes the host or domain:

<a href="http://www.example.com" ... >

That prefix expression (here, the http://) is followed by a host or domain specification (or an IP number) and whatever additional information is needed:

<a href="http://www.example.com/cgi-bin/search?who=bob">
<a href="http://192.168.44.55">

Note that the protocol can be excluded, in which case it is automatically set to the document's default. The protocol, host or domain, and other information are normally enclosed in quotation marks to protect HTML-capable mail programs from stumbling over illegal characters. The quotation marks may be double or single, but whichever is used, they must pair up (a double quotation mark may not be mixed with a single quotation mark):

href="http://www.example.com"
href="http://www.example.com"
href="http://www.example.com"     Won't work.

Note, however, that quotation marks can often be safely omitted, so you should not count on their presence when parsing spam email.

If a spam HTML message is in a language other than English, the quotes may be present but specified using the other language's encoding:

href=EF2Dhttp://www.example.comEF2D

Here, the EF2D is hexadecimal that represents two binary byte values (not four characters) that specify quotation marks appropriate to the language. So again it is better not to depend on quotation marks when parsing URLs.

Although the protocol, when present, is always specified with a trailing ://, in actuality all that is really needed is the colon.[11] Thus, all three of the following produce the same URL result:

[11] We fudge for simplicity. Technically, two forward slashes are supposed to indicate a network path, and one forward slash is supposed to indicate an absolute (local) path (ref. RFC2396, section 3).

http://www.example.com
http:www.example.com
http:////////////www.example.com

Notice that the number of forward slashes is unimportant. The single required character is the colon.

Also note that there can be no space between the protocol and its colon, but the colon can be followed by arbitrary white space characters.

http :www.example.com    Space before colon won't work.
http: www.example.com    Space after the colon is OK.
http:                    A new line is OK.
www.example.com

Note also that the protocol does not need to actually be present with each URL. A <base command (if present) sets a prefix that will precede all URLs that do not specify a protocol. The prefix is always terminated by a forward slash (/) even if one is omitted from the <base command.

<base href="http://www.example.com">
<img src="images/bob.jpg">

These two commands are the equivalent of the following (single) command:

<img src="http://www.example.com/images/bob.jpg">

If a <base is omitted and if the URL omits a protocol, the default is generally the http:// protocol:

<a href="www.example.com">     The protocol defaults to http://.

2.8.4. Email Addresses Mask URLs

To protect from inappropriate input, some HTML-compliant mail readers interpret an email address that is part of an http:// reference to be the same as a host or domain specification.[12] For example, the email address in the first line is interpreted (in the second line) as if the user part and the @ were omitted:

[12] Technically, a mail reader discovers a username:password to the left of the @ with the :password missing. Note that this form of address is actually useful for ftp:// references (see RFC2396, section G.2).

<a href="http://bob@www.example.com">
<a href="http://www.example.com">

The lesson is that whenever you are parsing a host or domain specification, you will need to start parsing over again when you encounter an @ character.

2.8.5. IP Numbers Too

Spammers are aware that the domain part of the URL does not need to be expressed in host.domain form (as www.example.com), and they use that fact to help disguise the host's name. For example, the following replaces the host.domain form with the IP number of the host www.example.com:

<a href="http://192.168.22.33">

Here 192.168.22.33 is the IP number for www.example.com. So be aware, when parsing URLs, that the host.domain part can be expressed as an IP number, too.

Also note that IP numbers can be expressed in decimal or in hexadecimal when prefixed with a literal 0x, thus making them even harder to detect:

<a href="http://0xC0A81621">      IP number in hexadecimal
<a href="http://3232241185">      IP number in decimal

All this effort to disguise a host's name with a cryptic-looking hexadecimal address allows spam email to double as a means to accomplish fraud. Consider, for example, the following web reference and surrounding text:

Your ATM card PIN number has expired. For security
reasons, connect to our
<a href="https://www.ABCDE-Bank.com:Secure@0xC0A8162">
secure server</a> and select
a new PIN as soon as possible.

Users who click on this URL are taken to the spammer's fraud site at @0xC0A8162, and not to the bank's site as they would expect. Even more dangerous, users see the following literal link address in the browser's link window, thereby being further fooled into thinking that the link is legitimate:

This example shows why it is crucial to start over when you encounter an @ when parsing a URL and why you need to allow for host.domain, IP address, and hexadecimal forms of addresses when parsing the URL.

2.8.6. Dealing with Redirects

A redirecting site is a host that takes a web reference that points to itself, strips away the self-referencing part, and then issues an HTTP redirect command back to the user's browser with the remaining part of the reference. The effect for the user is to view the redirected-to site and not the redirecting host's site.

http://redirect.example.com/*http://www.real.host
http://redirect.example.com/*http://www.real.host
http://www/real.host

Here, redirect.example.com is a redirecting site. When a browser visits it with the full URL shown on the first line, it recognizes the self-referencing part (bold on the second line), strips that self-reference, and returns an HTTP redirect to the actual host (shown on the third line).

The presence of redirect sites increases the complexity of parsing an actual site from a URL in spam email. Currently it is sufficient to simply start parsing over again when you encounter an http: or an https: while parsing a URL. As spammers gain skill and experience, however, such a simple solution will become less effective.

There are many redirect sites on the Internet. One that spammers know well is rd.yahoo.com.[13] They all share the same characteristics. The URL for the redirecting host is listed first, followed by a forward slash or backslash, then a special character, and then the URL for the actual host.

[13] Another is g.msn.com.

<a href="http://rd.yahoo.com/xxxx/*http://real.site">

For the rd.yahoo.com family of servers, the special character is an asterisk. For others the special character varies. The one common characteristic among all redirect servers is that the special character is not one that would normally appear as part of a URL.[14]

[14] Don't be surprised if a spammer organization forms to host a redirect site that uses a special character that can appear in a URL.

The redirect site can be followed by an arbitrary amount of URL information. For example:

href="http://rd.yahoo.com/bypass/winkie/food/thumb*big/dairy/
noisy/gyroscope/middle/fred/*http://real.site"

Recall that the special character must follow a forward slash. So here the * in thumb*big is not special. That is, it does not begin a URL for the real site.

Note that backslashes are the equivalent of forward slashes in the redirect portion of the URL because they are essentially ignored by the redirect server:

[View full width]

href="http://rd.yahoo.com\bypass/winkie/food\thumb*big/dairy/noisy/gyroscope/\middle/fred
/http://www.chunky.example.com/acidrain/moose/jane/wheel\*http://real.site"

Here, the actual site (always last) is real.site. But beware: real.site is the spammer's site and subject to the spammer's rules. It is not unreasonable to expect spammer sites to employ new methods that will make the real.site appear perhaps second from the last or even third from the last:

[View full width]

href="http://rd.yahoo.com\bypass/winkie/food\thumb*big/dairy/noisy/gyroscope/\middle/fred
/http://www.chunky.example.com/acidrain/moose/jane/wheel\*http://real.site?search=http:/
/some.bogus.site&referenece=http://another.bogus.site"

Just as it is important to begin parsing over again when you find an http: or https:, it is also necessary to stop parsing when a URL argument (the ?) appears. Clearly, spammers will go to great lengths to disguise the actual URL, and the examples we have shown here are only a hint of the techniques you will see in the future.

2.8.7. Wildcard DNS Records

The host name part of a URL can be subject to the same random word masquerading as the body, making it difficult to detect. To illustrate, consider these two URLs:

href="http://bob.biff.bonny.bill.betty.boop.example.com"
href="http://andy.able.alex.annie.alice.boop.example.com"

If you look at only one of these, there is no way to know for certain where the randomizing ends. But looking closely at two, one might surmise that the host name is boop.example.com. But one might be wrong. The real host might actually be example.com. To find out which is right, we need to delve briefly into DNS records.

When an HTML-capable mail program needs to connect to a URL's site, it first must look up the address of that site. For both of the sample randomized host names, it would (for example) find the address 192.168.111.44. But look at what happens when we look up the two names we suspect to be the real host names for these URLs:

boop.example.com       192.168.1.23
example.com            192.168.111.44

Clearly, our presumptionthat the boop.example.com host was the real hostwas wrong. Instead, the real host is example.com because it has the address 192.168.111.44.

But a savvy spammer might anticipate this logic and use a host name that appears random but is actually the real host name.

boop.example.com    some innocent's address
example.com    some innocent's address
able.alex.andy.boop.example.com 192.168.111.44

Here, if you decide to use the address that you suppose is the real host, you might cause some innocent site's address to be interpreted as that of a spamming site. The correct way to record the spamming address for later use is to look up and record the full domain name in the reference, even if it appears to be random.

2.8.8. CNAME Records and URLs

When an HTML-capable mail program attempts to look up a URL's host or domain, it may receive another host name in return rather than the expected address. That new host name is called a CNAME record. Here's how it works:

You find the URL http://example.com.

You look up the host name example.com.

You expect an address, such as 198.162.33.44, but instead you get the host name www.example.net.

When you look up a URL's host name and expect an address but instead get another host name in return, you need to do an additional lookup to find the actual address.[15]

[15] We fudge for simplicity. Sometimes the new host and its address are returned at the same time.

    example.com     www.example.net
www.example.net     192.168.33.44

But CNAMEs can lead to other CNAMEs, thus creating a long thread of potential lookups, and CNAMEs can even form infinite loops:

    example.com www.example.net
www.example.net www.example.com
www.example.com example.com      An infinite loop!

When you combine the risk of CNAMEs with the need to decipher long host names, the thread of lookups might get ugly indeed:

able.alex.andy.boop.example.com www.example.net
                www.example.net boop.example.com
               boop.example.com 192.168.111.44
     alex.andy.boop.example.com 192.168.22.4
          andy.boop.example.com 10.9.4.2
               boop.example.com 192.168.111.44
                    example.com example.net
                    example.net example.com
                    example.com example.net
                    example.net example.com
                         etc. in an infinite loop

When you write code to decipher a long host name, be sure to account for the possibility of infinite loops.

2.8.9. URLs Used as Comments

Just as non-HTML words can be used to create comments (see section 2.5), so can URLs. For example, consider the following:

[View full width]

V<a href="bob.example.com"></a>i<a href="jane.example.com></a>a<a href="alice.example.com"><
/a>g<a href="dan.example.com"></a>ra

When viewed on an HTML-aware mail reader, the preceding would appear like this:

Viagra

The use of URLs as comments is intended to make it difficult to find the actual URL in the message. When you look for the URL, it is not enough to simply look for the </a> abutting the reference, because arbitrary nonprint HTML can also appear between the two:

[View full width]

V<a href="bob.example.com"><font size="+5"></font></a>i<a href="jane.example.com"><font
 color="white"></font></a>

Finding the URL when this technique is used requires your spam scanner to act almost as an actual HTML parser.

2.8.10. JavaScript.Encode URLs

Last herebut certainly not the last word in hiding the URLis the technique of encoding a URL using JavaScript.Encode. The spammer's idea in this strategy is to wrap the URL in JavaScript so that it will be decoded by the browser. For example, consider the following obscured URL (wrapped to fit the page):

[View full width]

<script language="JScript.Encode">#@~^hQAAAA=3D=3D~@#@&[Km!:+ YcADbYn'E@> !
(o"bHA~?"Z'r4OYa)Jz+!+ O, FF+R8*f&^kxV 4YhVr~qq9:C{!P_2&!C:'TPwI)\AAr"92"> '!,j/I}SdqHMxE
 WE@*@!&qwI)\A@*BbI@#@&AyIAAA==^#~@</script>

Here, the HTML tag, <script, tells the interpreter to decode what follows using the JavaScript.Encode protocol. That protocol will then decode everything between the leading <script and the ending </script>. When decoded, the URL becomes the following:

[View full width]

document.write('<IFRAME SRC="http://192.168.23.45/link.html" WIDTH=0 HEIGHT=0
 FRAMEBORDER=0 SCROLLING=0>')

After decoding, it is clear that the encoded URL contains a web reference that the spammer wants to keep secret. But we can use it to record the URL of the spamming site. Here, that site is represented by the IP address 192.168.23.45, but in other JavaScript. Encoded URLs the reference may be a host name or may be further obscured by other means.

See http://www.virtualconspiracy.com/ for C language source examples of ways to decode JavaScript.Encode.

2.9. Loose Ends

Most spam email is trying to sell something to the recipient and thus needs to entice the recipient into contacting the spammer. As you have seen, finding a website buried inside spam email can be tricky. But the effort is worthwhile because most spam email sells by tricking or enticing the user into clicking on a website.

Some spam omits a web reference and instead provides a phone number. Because there is no universal standard for the appearance of phone numbers, detection of phone numbers is more an art than a science. Consider, for example, the following ways to hide phone numbers:

Call now! 1 8 0 0 (yes the call is free!) 555 12 12
Dial one eight-hundred 555 1234
Ring us at 800 LLL-LUCK

Fortunately, the use of phone numbers is rare (probably because spammers don't want to give out their phone numbers). But if a URL is missing and if a phone number is easy to find (that is, if it is not an image), look for a phone number instead.

Even more rare, instead of a URL you will find only an email address. On those rare instances when only an email address is available, record it.

Note that some spam doesn't want you to connect at all. It merely recommends a stock or a product and hopes the recommendation will cause you to buy. Obviously this type of spam is very difficult to detect the first time detection is attempted. But note that its pattern can later be recognized as spam once a human has categorized it, as with a Bayesian filter (see section 11.8).

Finally, of course, a tiny amount of spam is not motivated by sales at all. Some is religiously or politically motivated. During one month, for example, the authors were spammed with dozens of requests to read and live by a psalm in the Bible, where the spam recommended a different psalm in each mailing. Such altruistic spam is the hardest of all to detect.

2.10. Think Like a Spammer

Spammers are not creatures apart from the rest of humanity. They are human, just like you and us, and are motivated just the same, to make a reasonable living.

Imagine yourself out of work, desperate and needing to make a buck. All you have is your computer, lots of computer savvy, and a money-making scam in mind. What would you do to send spam email? What would you do that is different from what spammers are doing now? How would you try to foil current spam detection software?

By trying to outspam the spammers, you will be better prepared to fight them when they later mature to your level of thinking.

Part II: Creating a Test Environment

This part of the book explains why you must test spam filtering before deployment. Without this step, you may lose valuable or mission-critical email.

Chapter 3, Setting Up a Bait Machine, shows how to set up a machine that will gather a huge amount of spam so that your filters can be exercised to the maximum.

Chapter 4, Baiting the Hook, shows how to set up many fake addresses on the test machine and how to subscribe them to lists that will generate the most spam.

Chapter 5, Preventive Measures, is the mirror image of Chapter 4. Whereas Chapter 4 shows how to maximize exposure to spam, this chapter shows how to minimize your users' exposure to spam.

Chapter 3. Setting Up a Bait Machine

In this chapter, we illustrate one way to set up a test environment to develop sendmail Milters, a type of email filter.

A sendmail Milter can be your first line of defense against spam email because it is uniquely aware of every connection and reviews every piece of email. In more technical terms, a sendmail Milter is a threaded daemon that runs in parallel to the sendmail program. It provides direct access to raw envelope information about every inbound and outbound piece of email, which includes all headers and the message body.

Before you begin to filter inbound email, you must make sure that real, valuable, or mission-critical email will never be wrongly rejected by your filters. The correct way to test spam filters is with a machine that is set aside for this purposea test machine that never handles real email unless (as we will demonstrate) it handles only expendable copies of email.

Chapter 4 shows you how to manage addresses and how to gather spam. In this chapter we restrict our discussion to the initial setting up of a test environment.

The following topics are covered in this chapter:

Choosing a platform (section 3.1)

Setting up DNS records (section 3.2)

Configuring sendmail (section 3.3)

Setting up logging (section 3.4)

Excluding non-email ports (section 3.5)

Rebooting the test machine (section 3.6)

3.1. Choose Your Platform

A test machine needs to support Posix threads as well as run sendmail. For example, the authors used an old Sun Ultra 1, loaded Solaris 9 on it, and installed gcc(1) version 3.2. Your hardware will doubtless differ, but the lessons learned will be the same. In this section we discuss the following:

Posix threads (section 3.1.1)

The version of sendmail you should use (section 3.1.2)

Choice of compiler (section 3.1.3)

Installing a Milter library (section 3.1.4)

Network connections (section 3.1.5)

Database support (section 3.1.6)

3.1.1. Posix Threads

To compile a Milter, your system must support posix threads (also called Pthreads).[1] Because threads are implemented in the kernel, you need an operating system that supports Posix threads. Most modern operating systems do, but older ones do not. For example, Solaris 5.4 does not, but Solaris 8 does.

[1] Pthreads are defined by the IEEE standard 1003.1c.

One way to tell whether your operating system supports Pthreads is to enter the following command:

$ man pthreads

If a manual page is displayed, you indeed have Pthreads support. If the manual is not found, enter the following command to see whether Pthreads support is available, even though the manual is not:

$ ls /usr/include/pthread.h

Again, if you don't get an error, then you do have Pthreads support.

But note that Pthread support by itself is not all that is necessary. You must also have a thread-safe C language library. If you are running the absolutely latest release of your operating system, you probably do. Otherwise, contact your vendor to find out how thread-safe your libraries are.

3.1.2. Version of sendmail

Although all versions of sendmail, beginning with V8.10, support Milters, only V8.12 and later support them officially. For a variety of reasons, we recommend that you always run the latest version of sendmail. For the examples in this book, we are running V8.13.1 sendmail, because that is the latest version as of this writing and because that version added several useful new routines to the Milter library.

The latest version of sendmail is available from http://www.sendmail.org. Just point your browser at that site and follow the directions you find there. We don't describe how to build and install sendmail in this book, because that subject is amply covered in the third edition of the book sendmail, O'Reilly & Associates, 2002.

To add support for Milters to your sendmail binary, you must rebuild sendmail with MILTER defined. Just add a line like the following to your m4 Build file, and rebuild sendmail as you normally would:

APPENDDEF(`confENVDEF´, `-DMILTER=1´)

Note that you can safely run a sendmail binary with Milter support without a Milter. Thus, you can now run the installed sendmail to test its ability to send and receive email. Later (see section 3.3.1), we will show you how to modify your sendmail configuration file to tell sendmail you are using a Milter.

Note that if you already have sendmail installed with Milter support, you can skip these installation steps. For example, on Linux, you may only need to install the libmilter Redhat Package Manager (RPM).

3.1.3. Choice of Compiler

As a rule, you should keep your C language compiler up-to-date with the latest version and latest patches. It doesn't matter if your compiler is from your vendor or you use the GNU compiler. Either one will work to build a Milter. For the examples in this book, we ran the following:

[View full width]

% gcc -v
Reading specs from /usr/local/lib/gcc-lib/sparc-sun-solaris2.9/3.2/specs
Configured with: ../configure --with-as=/usr/ccs/bin/as -- with-ld=/usr/ccs/bin/ld
 --disable-nls
Thread model: posix
gcc version 3.2

Note that the thread model supported (the Thread model line) is posix, which supports Pthreads, exactly what we want.

3.1.4. Install a Milter Library

In addition to building and installing sendmail, you will need to build and install the Milter library. To do that, perform the following steps (note that the last step will probably need to be run by root):

$ cd /usr/local/src/sendmail-8.13.1/libmilter
$ ./Build -c
# ./Build install

In the first line you need to change into the libmilter directory under the source distribution. The exact path to specify will depend on which version of sendmail you are running and where you installed the source.

Before you build and install, be sure to read the file named README in the libmilter directory. It always contains the latest information about how to build and install the Milter library.

3.1.5. Network Connections

The test machine you set up probably should not be connected directly to your internal or production network. After all, this is a test machine that will be handling a lot of spam email. Thus, if you can do so, try to connect the machine only to the outside world. This arrangement will reduce risk to your normal network.

To manage the test machine, you can either ssh(1) into it or log in on the console interface. As you will see in section 3.5, we reduce services on the test machine to the bare bones to minimize the risk of intrusion into it.

3.1.6. Database Support

We recommend that you download and install the latest version of Sleepycat DB for database support. As of this writing, the latest version is 4.1.52, the version we use for the examples in this book. Sleepycat DB is available from http://www.sleepycat.com/.

In this book we presume that you have installed Sleepycat DB in its standard location:

/usr/local/BerkeleyDB.4.1

If you have installed it in a different location, you will need to adjust our examples accordingly. If it is preinstalled, as on many FreeBSD and Linux systems, we recommend that you use the preinstalled version instead of installing your own. Otherwise, you will have to replace a version integrated into the operating system with one that is not.

3.2. Set Up DNS Records

To ensure that test email never reaches your production network, it is vital that DNS records for the test machine be set up properly. In this section we discuss the following:

Domains and subdomains (see section 3.2.1)

Adding MX records (see section 3.2.2)

Using other host aliases (see section 3.2.3)

3.2.1. Domain Versus Subdomain

A domain name is a name of your choice, followed by a dot and the name of one of the top-level domains, such as .com, .net, .biz, and so on. In this book we use the domain example.gov in our examples that require a domain name. Note that there can be many hosts under a domain, such as www.example.gov and mail.example.gov (the host part being the leftmost part of the name, the www and mail).

A subdomain is another domain that can exist under your main domain. For example, research.example.gov might be the research division of example.gov. A subdomain, like a domain, can have many hosts under it, such as www.research.example.gov.

Before you set up a host machine to use for gathering spam email, you should decide whether you want to use a host name inside your existing domain, a host under a separate subdomain, or a host under a new domain. There are advantages and disadvantages to each approach.

The most cost-effective way to set up a test machine is to simply add a new host to your existing domainperhaps something with a realistic-sounding name like inmail.example.gov. Or you can use an existing machine to receive the spam email, such as bob.example.gov. The risk here is that existing MX records (see section 3.2.2) will prevent mail from going to such an existing host.

If you do not already use subdomains, you probably should not try to set up a new subdomain for testing antispam measures. Subdomains require you to run internal name servers and to set up your external name servers to reference your subdomain name servers. If you are already set up for subdomains, you will already know how to do this. Otherwise, we recommend that you use an existing or new domain instead because we don't cover setting up subdomains in this book.

If you have the money, you might prefer to register a new domain name for testingperhaps something like fubilfabby22222.gov.[2] What you choose, of course, is entirely up to you.

[2] As of this writing, the domain fubilfabby22222.gov does not exist.

The important point here is that you should choose a machine name that is easy for you to set up and administer.

3.2.2. Add MX Records

Mail is seldom routed to a host based on that host's name. Instead, the host name is looked up to see whether it has any MX (mail exchange) records. If it does (and it should) mail is sent to the hosts specified by those records instead of to the original destination host (although both may be the same). For this example we will use the host name inmail.example.gov.

You can use the dig program to look up the MX records for your new host:

$ dig mx inmail.example.gov
; <<>> DiG 8.3 <<>> mx inmail.example.gov
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 2, ADDITIONAL: 3
;; QUERY SECTION:
;;      inmail.example.gov, type = MX, class = IN

;; ANSWER SECTION:
inmail.example.gov. 1H IN MX 10 mx.example.gov.
;; Total query time: 1356 msec
;; FROM: u.i.sl3d.com to SERVER: default -- 0.0.0.0
;; WHEN: Thu Jul 17 14:03:58 2003
;; MSG SIZE sent: 25 rcvd: 228

This sample output shows that whoever set up the records for this new host, set the default MX record to be the same as that used by the entire domain (meaning that all mail will be sent to mx.example.gov, even if the mail was addressed to inmail.example.gov).[3] But this won't work because mail addressed to inmail.example.gov should be delivered directly to inmail.example.gov, so the MX record needs to be corrected.

[3] Additional lines of output have been eliminated to simplify this example.

If your new host has no MX records or has the wrong MX records, you will need to set them up correctly for your test machine. The correct MX record for the inmail.example.gov machine, the one that will cause all mail to be sent only to it, might look like this:

inmail.example.gov. 1H IN MX 0 inmail.example.gov.

Make sure that you set up the MX records for your test machine so that mail goes only to that machine. If you mistakenly add MX records that point to your normal mail-handling machines, you may surprise actual users with a great deal of spam email.[4]

[4] Use care when your ISP manages your DNS records. Many ISP administrators presume they know better than you how to set up your domain. If you ask for the unusual, they will silently give you the wrong setting because they assume you didn't know what to ask for.

3.2.3. Using Other Host Aliases

The last step in setting up the DNS records for your test machine is to add common aliases for it. If, for example, the name of your testing host is fabboulldy222.gov, you might also want to reference it as www.fabboulldy222.gov. You can add such aliases for a host by adding CNAME records to its DNS listing. Aliases for this host might look like the following:

www.fabboulldy222.gov.  1H IN CNAME fabboulldy222.gov.
ftp.fabboulldy222.gov.  1H IN CNAME fabboulldy222.gov.
mail.fabboulldy222.gov. 1H IN CNAME fabboulldy222.gov.

After these records are in place, email to x@www.fabboulldy222.gov, for example, will be delivered as if it were addressed to x@fabboulldy222.gov.[5]

[5] CNAME records may not have any other records associated with them. Don't be tempted to wrongly give your aliases MX records too.

Note that additional information about how to configure DNS records can be found in the book DNS and BIND by Paul Albitz and Cricket Liu (O'Reilly & Associates, 2001).

3.3. Configure sendmail

By default, the sendmail configuration file does not support Milters. In this section we show you how to add Milter support to sendmail's configuration file and how to accept email for your test and real domains:

Add Milter support (section 3.3.1)

Set up local-host-names (section 3.3.2)

A minimal aliases file (section 3.3.3)

3.3.1. Add Milter Support

The vanilla sendmail configuration file lacks Milter support. This makes sense when you consider that sendmail cannot know ahead of time what you will call your Milter.

You add Milter support to your configuration file by editing your mc configuration file and then rebuilding the sendmail configuration file from that mc input. To illustrate this process, we assume you maintain your mc configuration under the cf/cf directory in the sendmail source directory and that you call your mc file sendmail.mc.

$ pwd
/usr/local/src/sendmail/sendmail-8.13.1/cf/cf
$ ls sendmail.mc
sendmail.mc

The first few lines of our sendmail.mc file look like this:

OSTYPE(solaris8)
MASQUERADE_AS(`inmail.example.gov´)
INPUT_MAIL_FILTER(`ourmilter´, `S=unix:/var/run/ourmilter.sock´)
define(`confINPUT_MAIL_FILTERS´, `ourmilter´)

The first line says that we are running Solaris 9.[6] The second line says that all outbound email will appear to be from the test machine, here inmail.example.gov. The last two lines define our Milter. Note that there will likely be many more lines in your mc configuration file, but the last two lines are the two you will add to your mc configuration file to add Milter support.

[6] Solaris 8 and Solaris 9 both share the solaris8 definitions.

The INPUT_MAIL_FILTER mc command that you add takes two arguments. The first tells sendmail that our Milter will be named ourmilter. The second argument tells sendmail that our Milter will communicate using a UNIX domain socket (the unix:) and that the name of that socket is /var/run/ourmilter.sock.

The confINPUT_MAIL_FILTERS mc macro that you add is defined to have the value ourmilter. If we had defined more than one Milter (using INPUT_MAIL_FILTER), the confINPUT_MAIL_FILTERS mc macro would list the order in which each Milter would be called. Without confINPUT_MAIL_FILTERS, Milters are called in the order defined by your INPUT_MAIL_FILTER lines.[7]

[7] Note that this ordering cannot be changed or set by sendmail rule sets.

Note, too, that enabling the delay_checks feature in your mc file may lead to unexpected effects when you run your Milter. This feature prevents rule set check_mail from being called just after the sending site sends the MAIL FROM: command, and it prevents the rule set check_relay from being called just after a connection is accepted. Instead, these rule sets are called as part of the check_rcpt rule set, which is called after each envelope recipient is specified. The net effect is that they are called in the order check_rcpt, check_mail, and then check_relay, which is the reverse of what you might normally expect.

Be aware that if you plan to initialize special macros inside check_mail or check_relay for use in your Milter, those macros may not be initialized when you expect them to be.

After you have added these two new Milter lines to your mc configuration file, you can build a new sendmail configuration file as you normally would. For example, because we named our mc file sendmail.mc and kept it in cf/cf, we can build using the Makefile in that directory:

$ make sendmail.cf
rm -f sendmail.cf
m4 ../m4/cf.m4 sendmail.mc > sendmail.cf || ( rm -f sendmail.cf && exit 1 )
chmod 444 sendmail.cf
# make install-cf
../../devtools/bin/install.sh -c -o root -g bin -m 0444 sendmail.cf /etc/mail/sendmail.cf
../../devtools/bin/install.sh -c -o root -g bin -m 0444 submit.cf /etc/mail/submit.cf

The first command creates a sendmail.cf file from the sendmail.mc input file. If you had made any mistakes (a misspelling perhaps), an error would show up here.

The second command (which likely will need to be run by root) installs the newly created sendmail.cf file in the /etc/mail directory so that sendmail can use it. It also installs submit.cf so that outbound (user-submitted) email will work.

With your new configuration file in place, you will need to kill and restart sendmail using a command that looks like this:

# kill -HUP `head -1 /var/run/sendmail.pid`

If any mail arrives at this machine before the actual Milter is written and installed, the following warnings will be logged for each piece of email received:

Milter (ourmilter): local socket name /var/run/ourmilter.sock unsafe
Milter (ourmilter): to error state

These warnings are harmless and can be ignored for now.

3.3.2. Set Up local-host-names

Recall [see section 3.2.3] that we added other names for the current host. They included ftp, www, and mail. Unless you tell it to, sendmail will not accept mail addressed to those other names. To get sendmail to accept mail for those names, you need to add another line to your mc configuration and list those additional names in a special file.

First, add the following line to your mc configuration file (if it is not already there):

FEATURE('use_cw_file')

This FEATURE tells sendmail to look in the file /etc/mail/local-host-names for other host names it should recognize as local.

Second, add your list of additional names to that file. Our example file (based on the CNAME records in section 3.2.3) looks like this:

www.fabboulldy222.gov
ftp.fabboulldy222.gov
mail.fabboulldy222.gov

Your entries will, of course, differ from those we show here.

Finally, build and install a new configuration file, and restart sendmail.

3.3.3. A Minimal Aliases File

A minimal /etc/mail/aliases (or /etc/aliases) file on the test machine should include at least the following aliases. Later, we will add more:

root:           /dev/null
Postmaster:     root
MAILER-DAEMON:  root
abuse:          root
audit:          root
bin:            root
daemon:         root
debug:          root
decode:         root
dumper:         root
ftp:            root
http:           root
ingres:         root
manager:        root
news:           root
nobody:         root
operator:       root
security:       root
sync:           root
sys:            root
sysdiag:        root
system:         root
tcpd:           root
toor:           root
uucp:           root
webmaster:      root

Note that the first line causes all mail to the user root to be delivered instead to the file /dev/null, which causes all mail to be discarded. The rest of the lines cause all mail to each to be delivered to root, which in turn delivers to /dev/null, so they too are discarded.

If you wish, instead, to receive all mail (rather than discard it), change the first line to the following, where you is your login name:

root:           you

You may want more or fewer names on the left side of your /etc/mail/aliases file. We leave that up to you. Just make sure that you alias each to be delivered to root.

After you have saved your aliases file to disk, you next need to rebuild the aliases database. You do that by using the newaliases command:

# newaliases
/etc/mail/aliases: 26 aliases, longest 13 bytes, 551 bytes total

Remember, every time you change the /etc/mail/aliases file, you also need to run newaliases in order for your changes to go into effect.

3.4. Set Up Logging

A Milter can log what it is doing, and sometimes it may prove desirable to have your Milter log to a separate file. In this section, we describe syslog(8) and explain how to set up the syslog.conf file for use with your Milters.

A brief overview of logging (section 3.4.1)

Set up local# logging (section 3.4.2)

Arrange to rotate logs (section 3.4.3)

3.4.1. A Brief Overview of Logging

Logging is controlled by the /etc/syslog.conf file. Take a moment to study your /etc/syslog.conf file. One of the lines in it might look like the following:

mail.debug    /var/log/syslog

This line is the one used by sendmail (on a Solaris system). The facility mail is the facility used by sendmail. The dot and the debug tell syslog that all logging messages at a level of debug and above will be logged. The /var/log/syslog specifies that all the logging messages for the facility mail, at a level of debug and above, will be appended to the file /var/log/syslog.

Note that your mail entry may differ. On FreeBSD and Linux systems, it might look like the following:

mail.info     /var/log/maillog

But no matter what yours looks like, the effect is similar. Here, logging at a level of info and above will be appended to the file /var/log/maillog.

3.4.2. Set Up local# Logging

Nine logging facilities are available for nonsystem programs:

local0
local1
local2
local3
local4
local5
local6
local7
local8

You can choose any of these that are not currently used by other programs on your system. We chose local3 for the examples in this book.

Because local3 does not already exist in our /etc/syslog.conf file, we need to add it. Simply edit the /etc/syslog.conf file and add a line like the following:

local3.debug      /var/log/milter.log

Note that the dotted expression to the left must be separated from the path specification on the right by tab characters, not spaces!

After you add a new entry to the /etc/syslog.conf file but before you restart the syslogd daemon, you need to create the target file (the one whose path is specified on the right). The following command-line expression, run by root, does just that:

# cp /dev/null /var/log/milter.log

This creates a zero-length file to which logging messages will be appended.

After these changes have been made, you can restart the syslogd daemon. Methods of doing so vary from one version of UNIX to another. On most modern versions of UNIX, for example, you execute the following command:[8]

[8] Sometimes this file is called /var/run/syslogd.pid.

# kill -HUP `cat /var/run/syslog.pid`

Examine your online manuals and various boot-time scripts to determine how to restart syslogd on your system.

3.4.3. Arrange to Rotate Logs

One weakness with the syslog system is that unattended log files will continue to grow forever unless you take steps to limit them. Most UNIX systems have programs for handling this need. On Solaris 9, for example, the logadm program can be used to rotate your new log file and to discard old logs. On FreeBSD, the /root/bin/roll shell script does much the same thing, and on Linux systems you can use /usr/sbin/logrotate.

Using the mechanism provided on your system, arrange for the /var/log/milter.log file to be rotated daily, and for copies more than a week old to be discarded.

3.5. Excluding Non-email Ports

To minimize outside interactions with the test machine, consider minimizing the number of entries in the /etc/inetd.conf file and turning off unneeded daemons. In this section we cover the following:

Scanning your test machine (section 3.5.1)

Minimizing inetd.conf (section 3.5.2)

Turning off unwanted daemons (section 3.5.3)

3.5.1. Scan Your Test Machine

A good program for scanning your test machine is the nmap program available from http://www.nmap.org/.

To use nmap you run it on a different machine from your test machine and then point the program at your test machine. For example:

$ nmap inmail.example.gov
Interesting ports on inmail.example.gov (12.34.56.78):
Port    State       Protocol  Service
21      open        tcp       ftp
22      open        tcp       ssh
25      open        tcp       smtp
80      open        tcp       http
111     open        tcp       sunrpc
512     open        tcp       exec
514     open        tcp       shell
7100    open        tcp       font-service
32771   open        tcp       sometimes-rpc5
32774   open        tcp       sometimes-rpc11

Nmap run completed -- 1 IP address (1 host up) scanned in 46 seconds

Note that your list of port numbers (the numbers to the left) will probably be different from those on this list, but the lesson is the same: There are some services listening on ports that are not necessary for the test machine.

Edit various files to eliminate unwanted listeners. Which files you edit will vary depending on your flavor of UNIX. One common file across all versions of UNIX is the /etc/inetd.conf file. Others will exist in /etc/rc* files, in the /etc/init.d directory, or elsewhere. Read your online manuals to determine what you need to edit on your version of UNIX.

3.5.2. Minimize inetd.conf

The /etc/inetd.conf file describes one listening program per line.[9] Lines that begin with a # character are commented out. Examine your inetd.conf file now. Notice that (if you have not already tuned it) it contains many uncommented lines. Some security experts claim that you can comment out all the lines in inetd.conf. Others recommend running the ftp daemon, for one, from that file.

[9] On Linux, it is /etc/xinetd.conf, and the file uses multiple lines per item and includes multiple files, one per item.

As you study your inetd.conf file, strive to learn why each line is commented or uncommented. Comment out as many items as you can based on what you learn.

After you have tuned inetd.conf you need to restart the inetd daemon. On most modern versions of UNIX, you restart inetd this way:

# kill -HUP `cat /var/run/inetd.pid`

But the way you restart inetd on your system may be different. Consult your online manuals for guidance.

3.5.3. Turn Off Unwanted Daemons

Not all listening is done by inetd. Some daemons are started that listen on their own. One example is the apache HTTP server. On some systems that server is started by default. Recall the earlier nmap output and the line that indicated an HTTP listener:

80      open        tcp       http

On a Solaris machine, for example, the HTTP listener is sometimes started with the file /etc/init.d/apache. To disable the service, edit that file and insert the following lines immediately after the first line:

#!/sbin/sh
# Disabled 12/12/2004 you    Add this line
exit 0                       Add this line
...

Methods of turning off listening daemons vary from one version of UNIX to another. Consult your online manuals to learn the proper way to turn off listening daemons on your version of UNIX.

3.6. Make Sure the Machine Reboots

In this chapter you have made many system-level changes to the test machine, but these changes are useless if you made mistakes that prevent the machine from booting.

Before proceeding to the next chapter, reboot the test machine. Make certain that it comes completely up and that it does not print any errors to the console screen. Try logging in to it and performing your normal tasks (such as looking at disk contents and running a text editor).

If all has gone well, you are ready to make this machine the target of a great deal of spam.

Chapter 4. Baiting the Hook

This chapter describes how to create test addresses and spread them out to the Internet so that your bait machine will receive an ever-increasing amount of spam. We also describe techniques for copying nonspam email onto the bait machine so that false positives can be detected.

Creating fake recipients (section 4.1)

Protecting wanted email (section 4.2)

Running a web server (section 4.3)

Posting to a usenet group (section 4.4)

Note that we omit the technique of attracting spam email by subscribing to various company mailing lists. Most companies do not share or sell your email address. The few that do, however, create a huge spam problem, and to avoid libel issues, we omit naming such offending companies.

4.1. Create Fake Recipients

Fortunately, under UNIX and with sendmail, it is not necessary for a person to have an actual login account to receive email. Fictional persons can be created that receive email just as if they are actual persons, because there is no way to tell the difference from outside the bait machine.

The method used to create this bit of subterfuge is the aliases file. In Chapter 3 you saw how to set up a minimal aliases file. In this section we show you how to create realistic aliases suitable for use in gathering spam.

Before we begin, however, we need to cover a few ground rules:

If possible, avoid using names that are the same as login names or the email names of your real users. This may not be as difficult as you imagine, even if you are a large ISP or university.[1] If, for example, your user base employs the standard of first initial followed by last name (e.g., jdoe), you can set up your spam addresses to use the standard of last name followed by first initial (e.g., doej). Or if, for example, your policy allows only letters and digits in names (e.g., bob23), you can set up your spam addresses to include an underscore character (e.g., bob_23).

[1] Large ISPs likely have all user names in a central database. In such a situation, it is sufficient to set up fake user names for spam collection.

Avoid using names that correspond to real services you wish kept private. For example, don't use private internal administrative names such as ford-sales, boardmeet, devusers, internalwebhelp, and so on.

Avoid using common UNIX administrative names such as ftp, bin, lib, and so on.

Avoid the temptation to write a quick script or program to automatically generate addresses. Although this technique can save you a bit of time, it prevents you from easily screening the result for accidental inclusion of objectionable addresses.

You add addresses for testing antispam techniques by editing the /etc/aliases file and then rebuilding the aliases database.

Recall from Chapter 3 that you created an initial /etc/aliases file that looked something like this:

root:           /dev/null
Postmaster:     root
MAILER-DAEMON:  root
abuse:          root
audit:          root
bin:            root
daemon:         root
debug:          root
decode:         root
dumper:         root
ftp:            root
http:           root
ingres:         root
manager:        root
news:           root
nobody:         root
operator:       root
security:       root
sync:           root
sys:            root
sysdiag:        root
system:         root
tcpd:           root
toor:           root
uucp:           root
webmaster:      root

This list (of names on the left) constitutes aliases that should not be used by the Milter we design. Instead, we will add names to this list.

In the following example, we will append names to this list just to show you how it is done. We presume, of course, that you will provide names of your own choice here. For the example, we chose to use the names of California towns:

# Mail to these addresses shall always be considered spam
yes_spam:        /dev/null
eureka:          yes_spam
arcata:          yes_spam
oakland:         yes_spam
davis:           yes_spam

Here, the first line is a comment. The second line ensures that all mail to any of the aliases that follow it will be deliverable (discarded by delivery to /dev/null).

Note that (beginning with the third line) each address on the left is aliased to yes_spam. Remember this aliases transformation. You can, of course, choose an alias of your own, but if you change it here, you will need to change it later when we write the Milter.

The list of new addresses begins with a comment line (a line that begins with a # character). We recommend you make the comment informative, such as by stating who added the aliases and why:

# Mail to these addresses shall always be considered spam
# First batch added 11/13/2003 bob

You don't need to add very many spam aliases at first. Before you do, read the rest of this chapter to see how complicated it will be to generate spam for each. Then return to this step and expand the list as much as you think necessary based on actual experience.

4.2. Protect Good Email

One problem with rejecting spam email is the need to avoid wrongly rejecting nonspam email. To your Milter program, the two may appear very similar, and generating false positives (wrongly rejecting good email) may be worse than accidentally letting some spam through.

Of course, it is impossible to tell whether your filter is creating false positives unless you can get it to filter known good email, too. In this section, we describe one way to forward a copy of good email to the testing machine. Other methods are also suggested.

The easiest way to forward copies of good email to the bait machine is at the personal level, with your own .forward file in your own home directory. But that may not be a wise strategy because, even before setting up a bait machine, you probably already receive spam email. Instead, and for the sake of example, we show how to forward good mail for a user who receives no spam (perhaps because that user is a member of a mailing list that receives only internal mail).

Most people don't have .forward files, and others may have empty ones. The .forward file on your home machine tells sendmail where and how it should deliver your email. One such .forward file might look like this:

|"/usr/local/nmh/lib/slocal -user you || exit 75"
# Uncomment below for vacation
# |"/usr/ucb/vacation you || exit 75"

Here, the first line causes sendmail to deliver your email by piping (the "|" character) it through the /usr/local/nmh/lib/slocal program. The single argument to slocal is the -user you, which tells slocal that the email is for you. The || exit 75 tells sendmail to queue the message (defer delivery) if slocal fails.

The last two lines in this .forward file are comments (each line begins with a # character) and are ignored by sendmail.

Add the following line to this file. It will cause a copy of your email to be sent to the spam-testing machine:

|"/usr/local/nmh/lib/slocal -user you || exit 75"
# Uncomment below for vacation
# |"/usr/ucb/vacation you || exit 75"
okaymail@inmail.example.gov

This line causes sendmail to forward a copy of that email to the user okaymail at your spam-testing machine, in addition to delivering a copy to you via the slocal program.

You don't have to use the name okaymail, of course. You can choose any name that makes sense to you. Just substitute your name for the name okaymail in the examples to follow.

But before the bait machine can accept email to the user okaymail, its aliases file must be edited again. So far, the last few lines of the /etc/mail/aliases file on the bait machine look like this:

arcata:          yes_spam
oakland:         yes_spam
davis:           yes_spam

Now add a few more aliases, such as the following, to that list:

#The following addresses can get nonspam email
not_spam:        /dev/null
okaymail:        not_spam

The first line is a comment that lets others know why the last two lines are present. The first of those new aliases defines not_spam as an address that will be discarded by delivering it to the /dev/null file. The last line defines the user okaymail as not_spam. This is the same okaymail user we forwarded by editing your .forward file.

After you finish editing the /etc/mail/aliases file, you need only run the newaliases program to make the new entries active.

4.3. Run a Web Server

If you have not already set up a web server, we won't tell you how. Instead we leave that as an exercise for you. For example, the Apache web server is available from, and is documented at, http://www.apache.org/.

In this section, we merely show you how to set up addresses to be gathered for spamming.

Most email addresses that are gathered from web pages are gathered by automatic programs. Such programs connect to your web server and recursively search all pages for the mailto: expression:

<a href="mailto:jenny@your.domain">sales</a>

This line in an HTML document says that email to sales should go to the address jenny@your.domain. Much has been written about how to disguise email addresses on web pages so that they cannot be gathered, but for your spam-testing machine (or on your real web server) you should insert addresses that are easy to gather so that they will generate spam email for your bait machine.

Examples (based on the aliases listed in the preceding section) might look like the following:

<a href="mailto:oakland@inmail.example.gov"></a>
<a href="mailto:arcata@inmail.example.gov"></a>
<a href="mailto:davis@inmail.example.gov"></a>

Note two things here: First, whether these entries are on the spam-testing machine's web server or on another web server, the mail will still be sent to the spam-testing machine (here, inmail.example.gov). Second, note that there is nothing between the <a> phrase and the </a> phrase of each HTML line, meaning that nothing will appear on the actual page, although the address is still available for gathering.

To illustrate, consider the following web page code:

<html><body>
This site is down.
<a href="mailto:oakland@inmail.example.gov"></a>
<a href="mailto:arcata@inmail.example.gov"></a>
<a href="mailto:davis@inmail.example.gov"></a>
</body></html>

If this were the main web page, a gathering program could issue the following HTTP command to download this information:[2]

[2] You may need to follow this line with a Host: specification if your site uses a proxy server.

GET / HTTP/1.0

To try this yourself, telnet to a known web server on port 80 and issue the preceding command followed by two newlines. The main web page will be displayed for your detection of email addresses:

$ telnet www.some.site 80
Trying 192.167.45.67...
Connected to www.some.site.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.1 200 OK
Date: Wed, 30 Jul 2003 18:25:07 GMT
Server: Apache/1.3.22 (Unix) mod_perl/1.25
Last-Modified: Sun, 27 Jul 2003 20:25:12 GMT
ETag: "1a807-86-3f243528"
Accept-Ranges: bytes
Content-Length: 134
Connection: close
Content-Type: text/html

<html><head>

<title>Title For Page Here</title></head>
<body>
Web page contents...
For sales contact <a href="mailto:sales@some.site">
our friendly sales folk</a>.
</body></html>
Connection to localhost closed by foreign host.

As you can see, it is trivial to write a program to detect email addresses, so we will leave that to you.

4.4. Post to a Usenet Group

By far the easiest way to get an email address spammed is by posting an article to a Usenet group using that address. The easiest way to do that is to use a program that allows you to set the email address rather than a program that insists on using your own address.

We illustrate one such program called Fudgenews.[3] Note that the numbers to the left in the listing merely explain the code and are not a part of the code itself. To compile this program, run a command something like this:

[3] See Appendix A for guidance in finding this software source.

gcc -o fudgenews fudgenews.c -lresolv -lsocket

On a Solaris system you will have to add -lnsl to the end of the command line. This program should build as is on most systems, but you may have to tweak it to work on other systems. We leave porting this program to your version of UNIX as an exercise for you.

The program begins (lines 1 through 10) by including the standard system files necessary for compilation.

 1 #include <stdio.h>
 2 #include <sys/types.h>
 3 #include <netinet/in.h>
 4 #include <arpa/nameser.h>
 5 #include <resolv.h>
 6 #include <netdb.h>
 7 #include <netdb.h>
 8 #include <stdlib.h>
 9 #include <errno.h>
10 #include <time.h>

Next (line 12), we define the name of the program for use in error messages.

11
12 # define PROG "fudgenews"
13

In lines 28 through 40 the program parses its command line. Note that it is run with three mandatory switches. The -h switch specifies the host name to use for posting to Usenet news, which may be something like news.example.gov. The -a switch specifies the email address to use, and the -n switch specifies the newsgroup to which to post.

14 int
15 main(int argc, char **argv)
16 {
17     int ch;
18     char *newshost = NULL;
19     char *address = NULL;
20       char *newsgroup = NULL;
21       struct hostent *hp;
22     union {
23             struct sockaddr    sa;
24             struct sockaddr_in sin;
25         } addr;
26      int sock, addrlen;
27
28      while ((ch = getopt(argc, argv, "h:a:n:")) != EOF)
29        {
30           switch(ch)
31          {
32                case 'h':   newshost = optarg;
33                      break;
34                case 'a':   address = optarg;
35                      break;
36                case 'n':   newsgroup = optarg;
37                                 break;
38                default:    break;
39          }
40        }

If any of the command-line switches are missing, the program prints a usage message and exits.

41         if (newshost == NULL ||
42          address == NULL ||
43          newsgroup == NULL)
44         {
45               printf("Usage: %s -h host -a address -n newsgroup\n",
46           PROG);
47          exit(EINVAL);
48

After all the command-line arguments are present, the program next opens a connection (lines 49 through 80) to the news posting host. Note that line 66 checks to see that this is an IPv4 address. If it is not (line 70), an error is printed and the program exits.

49         if ((sock = socket(AF_INET, SOCK_STREAM, 0)) < 0)
50         {
51               (void) fprintf(stderr,"%s(%d): %s: %s\n",
52                      PROG, __LINE__, newshost, strerror(errno));
53               exit(errno);
54         }
55         hp = gethostbyname(newshost);
56         if (hp == NULL)
57         {
58         (void) fprintf(stderr,"%s: %s: %s\n",
59                    PROG, newshost, hstrerror(h_errno));
60              exit(h_errno);
61        }
62        addr.sa.sa_family =  hp->h_addrtype;
63        addr.sin.sin_port = 119;
64        switch (hp->h_addrtype)
65        {
66              case AF_INET:
67                     memmove(&addr.sin.sin_addr, hp->h_addr, INADDRSZ);
68               addrlen = sizeof (struct sockaddr_in);
69                   break;
70              default:
71                    fprintf(stderr,"%s: %s: %d unknown protocol\n",
72                          PROG, newshost, hp->h_addrtype);
73                exit(EINVAL);
74        }
75        if (connect(sock, (struct sockaddr *) &addr, addrlen) < 0)
76        {
77              (void) fprintf(stderr,"%s: %s: %s\n",
78             PROG, newshost, strerror(errno));
79              exit(errno);
80      }

After the connection has been successfully made to the news server host, the program calls its post() subroutine (line 81) to actually post the new article. Any error in posting will cause the error to be printed.

81      if (post(sock, address, newsgroup) != 0)
82      {
83               (void) fprintf(stderr,"%s: %s: %s\n",
84                     PROG, newshost, strerror(errno));
85          exit(errno);
86      }

After the article has been posted, the program cleans up and exits.

87      (void) close(sock);
88        return 0;
89  }

The post() subroutine sends the article to the newsgroup host.

90  int
91  post(int sock, char *address, char *newsgroup)
92  {
93      FILE *infp, *outfp;
94      int nsock;
95        char buf[BUFSIZ];
96        time_t t;
97        struct tm *tt;
98        char hostname[MAXHOSTNAMELEN];
99
100        if ((infp = fdopen(sock, "r")) == NULL)
101         return errno;
102      if ((nsock = dup(sock)) < 0)
103               return errno;
104        if ((outfp = fdopen(nsock, "w")) == NULL)
105              return errno;
106        if (fgets(buf, BUFSIZ, infp) == NULL)
107         return errno;

The first thing that the program does after the article is sent is to wait for the news server host to send its greeting. That greeting (if successful) will begin with a 2yz return code. For example:

200 news.example.gov News server 1.0 (posting ok).

If the news server does not allow a connection from your host, it will reject that connection with a 5yz code. For example:

502 You have no permission to talk.  Goodbye.

Line 108 performs a test for a good greeting (which must start with a 2yz code).

108       if (strlen(buf) > 1 && *buf != '2')
109       {
110             (void)fprintf(stderr, "%s\n", buf);
111             return EPERM;
112       }

After the greeting, we issue the GROUP command (line 113) followed by the name of the newsgroup you specified to ensure that it exists and is served by the news server. If the newsgroup does and is, the server will return a 2yz code (checked for in line 117).

113       (void)fprintf(outfp, "GROUP %s\r\n", newsgroup);
114       (void)fflush(outfp);
115       if (fgets(buf, BUFSIZ, infp) == NULL)
116        return errno;
117       if (strlen(buf) > 1 && *buf != '2')
118       {
119             (void)fprintf(stderr, "%s\n", buf);
120            return EPERM;
121       }

Next, the POST command (line 122) is issued to let the server know that a new news posting is being made. If the news server allows posting, it replies with a 3yz code (checked for in line 126).

122        (void)fprintf(outfp, "POST\r\n", buf);
123        (void)fflush(outfp);
124        if (fgets(buf, BUFSIZ, infp) == NULL)
125              return errno;
126     if (strlen(buf) > 1 && *buf != '3')
127        {
128              (void)fprintf(stderr, "%s\n", buf);
129              return EPERM;
130        }

If posting is allowed, the program sends the actual posting. First, the mandatory headers are sent:

131        (void)fprintf(outfp, "From: %s\r\n", address);
132        (void)fprintf(outfp, "User-Agent: %s\r\n", PROG);
133        (void)fprintf(outfp, "Newsgroups: %s\r\n", newsgroup);
134        (void)fprintf(outfp, "Subject: test\r\n");
135        (void)fprintf(outfp, "Lines: 2\r\n");

The Date: header must be specially constructed to match the news standard:

136        (void)time(&t);
137        tt = localtime(&t);
138        (void)snprintf(buf, BUFSIZ, "Date: ");
139        (void)strftime(buf+strlen(buf), BUFSIZ-strlen(buf),
140         "%a, %b %e %Y %T %Z", tt);
141        (void)fprintf(outfp, "%s\r\n", buf);

This code will produce a date that looks like this:

Date: Thu, Jul 31 2003 12:04:31 PDT

Next, the Message-Id: header needs to be sent. It, too, is specially constructed:

142     if (gethostname(hostname, MAXHOSTNAMELEN) != 0)
143         (void)sprintf(hostname, "localhost");
144       (void)snprintf(buf, BUFSIZ, "Message-ID: <%x.%s@%s>",
145        t, newsgroup, hostname);
146       (void)fprintf(outfp, "%s\r\n", buf);

Finally the message itself is sent following a blank line. The message we use is "Just a test." But you can, of course, use a different message if you so desire. Just be certain to adjust the Lines: header (line 135) to show the correct number of lines in the body. The message body terminates with a line composed of a single dot. After the whole message has been posted, the output stream is flushed.

147     (void)fprintf(outfp, "\r\n\r\nJust a test.\r\n.\r\n");
148       (void)fflush(outfp);

The program then waits for the news server to acknowledge the posting. If the news server accepted the posting, it will reply with a 2yz code. Note that in line 151 the reply is printed so that you will see the result of the posting:

149     if (fgets(buf, BUFSIZ, infp) == NULL)
150         return errno;
151       (void)fprintf(stderr, "%s\n", buf);
152     if (strlen(buf) > 1 && *buf != '2')
153         return EPERM;

Finally, the QUIT command is sent to conclude the session, and the program is finished. There is no need to wait for a reply to the QUIT command.

154     (void)fprintf(outfp, "QUIT\r\n", buf);
155       (void)fflush(outfp);
156     return 0;
157 }

As a final note, be aware that you should not post to groups that will frown on the testing message. Suggested groups to use are alt.test, ba.test, and so on.

Chapter 5. Preventive Measures

Chapter 4 showed you how to encourage spam to be sent to fake addresses. This chapter, in contrast, explains some of the many methods spammers use to gather (harvest) addresses, including certain less obvious means such as use of finger and identd. We show some ways to harden websites against harvesting and explain some methods of administration that allow users to use custom addresses when signing up for services so that the source of spam can be traced. By its nature, this approach is preventive, disallowing addresses from being discovered in the first place.

Telling users about plus addressing (section 5.1)

Turning off EXPN and VRFY (section 5.2)

Masking web addresses (section 5.3)

Watching out for finger (section 5.4)

5.1. Tell Users about Plus Addressing

Plus addressing, under sendmail, is the ability to receive email that is sent to you with a literal "+" character inserted in the user part of the address. For example, if your address is you@example.gov email addressed as you+irs@example.gov will be sent to you in the normal fashion. Unless an alias exists to tell sendmail to do otherwise, you followed by a plus and then anything, at your domain, will be delivered to you as if the plus and the anything were not there.

you@example.gov            Delivered normally
you+anything@example.gov   Also delivered normally

Plus addressing can be used to track the source of spam and determine who may have sold your email address. To illustrate, consider signing up with your bank to receive bank statements via email. Although some websites do not accept a plus in an address, most do, and we will assume that your bank (www.bank.gov) does.

When you sign up for email services with this bank, you would normally sign up with your regular email address:

you@your.domain

But to track spam and discover whether this bank sells your address, you can sign up instead with a plus address, in which the +anything part (as shown earlier) will be +bank.

you+bank@your.domain

Now any mail from the bank will still be delivered to you, but the address in the To: header line of your received message will be

To: you+bank@your.domain (Your Name)

Now you are in a position to deal with two potential problems:

The problem of businesses that presume that your having an account with them automatically grants them permission to email you with unrelated product promotions

The problem of businesses that sell or share your email address with third parties, resulting in unexpected spam email

Both problems can usually be detected using plus addressing.[1] Any email that shows up with a To: header that contains a +bank, for example, should only contain the expected informationfor example, a bank statement. Any other content would be suspect, especially if it is received from a source other than the bank.

[1] Some spam software automatically strips the +anything from addresses.

5.1.1. Plus Addressing with Usenet News

In section 4.4 we described how email addresses are gathered from Usenet news postings and explained how to use that knowledge to seed postings with bait addresses. Usenet news is a great way for spammers to gather addresses. Similarly, Usenet news is a highly risky place for users who post messages.

As an administrator, you should educate your user base about the spam risks of Usenet news. At a minimum, show them how to use plus addressing to help track address gathering.

We don't demonstrate techniques to implement plus addressing for specific news-posting software in this book, because those techniques are as varied as the software used. Instead, familiarize yourself with the news-posting software on your machine, and learn how to customize the email address using each. When a person needs only to post (and not to receive email feedback), encourage that person to use a totally bogus address (one that will always bounce). If someone does need email feedback from a posting, consider obfuscating the email address and putting it somewhere in the body of the posting.

5.2. Turn Off EXPN and VRFY

Email is sent using a protocol called SMTP. Under SMTP, the sender machine issues commands, and the receiver machine performs the requested tasks. Two of these tasks that can be helpful to spammers are the EXPN and VRFY commands.[2] The EXPN command tells the receiving machine to expand (show the ultimate recipient or recipients of) an address. The VRFY command tells the receiving machine to verify that the address is a good one (can have email delivered to it).

[2] They can also use RCPT TO:, but that cannot be prevented.

Spam address-gathering software will sometimes use EXPN or VRFY to detect which addresses at your site are good. For example, one such search in an SMTP conversation might look like this:

VRFY <aa@your.domain>
550 5.1.1 <aa@your.domain>... User unknown
VRFY <you@your.domain>
250 2.1.5 Your Name <you@your.domain>

Here, the sending (spam address-gathering) machine issues the VRFY command to detect whether the address aa@your.domain is a good one. Because it is not good, your machine replies with a 550 error.

Next, the sending machine uses VRFY to determine whether the address you@your.domain is good. Because it is good, your machine replies with a 220 success code, and the spam address-gathering machine has easily found at least one good address at your site.

By issuing many VRFY commands, that machine can find many good addresses. Clearly, it is best to turn off VRFY and EXPN services at your site. The easiest way to do that is by adding the following line to your mc configuration file:

define(`confPRIVACY_FLAGS´, ``novrfy, noexpn´´)

Rebuild your configuration file and install it as described in section 3.3. Thereafter, sendmail will reject all VRFY and EXPN commands such as the following, no matter what address is given:

EXPN <you@your.domain>
502 5.7.0 Sorry, we do not allow this operation
VRFY <you@your.domain>
252 2.5.2 Cannot VRFY user; try RCPT to attempt delivery (or try finger)

Note, in the last line, that sendmail prints hints about other ways to find out whether an address is good. We discuss use of the finger(1) command in section 5.4.

5.3. Mask Web Addresses

Another method used to gather email addresses for spam lists is to take them from web page sources. Inside HTML sources, spammers examine the mailto: command, which lists an email address. For example:

Contact <a href="mailto:sales@example.gov">sales</a>
for further information.

When displayed on a web browser, this HTML source will look like this:

Contact sales for further information.

This, of course, allows the viewer to simply click on the sales link to send email to the user called sales. This is made possible with the HTML mailto: command. Clearly, it is possible for spammers to search for the expression mailto: and gather email addresses for their spam lists.

One technique used to fight this sort of spam gathering is to encode mailto: addresses so that gathering software cannot use them. You can do this by changing the special character @ into its HTML-encoded form:

mailto:sales&#064;example.gov

The idea here is to replace the @ character in the email address with its encoded counterpart so that address-gathering software will find an address that cannot be emailed to. Unfortunately, spam-gathering software has matured to the point where it is no longer fooled by this kind of expression.

Another technique used to fight spam gathering of email addresses is to represent addresses as plain text outside the mailto: command. This eliminates the ease of just clicking to mail but reduces the ability of spammers to gather such addresses by software:

Contact <b>sales</b> at example.gov for further information.

Note that there is no mailto: command in this HTML source. Instead, it relies on the reader to recognize the address and to cut and paste that address into a mail-sending program:

Contact sales at example.gov for further information.

Unfortunately, modern software can also recognize this sort of email addresses.

A better way to embed addresses in HTML source is to express it as abstractly as possible for the truly human reader. Consider, for example, the following HTML source:

<p>For further help and information, please
communicate with one of the following addresses
at this site:
<li>"sales" for retail customers</li>
<li>"wholesale" for business customers</li>

There is nothing in this text to indicate that two email addresses are suggested by it. Automatic software will have trouble finding those addresses, but customers will easily understand what to do:

For further help and information, please communicate with one of the following addresses at this site:

"sales" for retail customers

"wholesale" for business customers

Further obfuscation can be achieved using JavaScript and other tools.[3] For example, the following JavaScript can be used to obscure an email address:

[3] See http://innerpeace.org/escrambler.shtml.

function mailto(here)
{
        here = here.replace(/\s+at\s+/, '@');
        here = here.replace(/\s+dot\s+/g, '.');
        location.href = "mailto:" + here;
}

function mailme()
{
        mailto('bob at example dot com');
}

With this code in place, the email address bob@example.com can be hidden this way:[4]

[4] If you wish to download this source, see Appendix A for more information.

<html>
<head>
<script language="JavaScript1.1" src="mailto.js"></script>
</head>
<body>
<a href="javascript:mailme()">Your Name</a>
</body>
</html>

5.4. Watch Out for finger

The finger(1) program was originally intended to be a way to discover whether or not a user is logged in. The assumption was that, if the user was logged in, that user would be able to respond to queries using the write(1) program.

This was convenient in the days before the current epidemic of spam, but the finger(1) program, when improperly configured, can be used to discover email addresses on your machine:

% finger you@example.gov
Your Name (you) is not presently logged in.
Last seen at example.gov on Thu Jul 31 20:11:07 2003 from www.example.gov
% finger nosuchuser@example.com
Nobody matching nosuchuser ever logged in.

Here, someone at a spam address-gathering machine ran finger to discover good email addresses at your site, one of which turned out to be your address.

Unless there is some compelling reason to do so, you should never allow finger to run at your site. Generally, finger is run by inetd. To disable finger, edit inetd's configuration file (/etc/inetd.conf) and comment out the finger line. When done, signal inetd with a SIGHUP to restart it.

If you do need to run finger internally, you can instead block outside access to port 79 at your router or firewall.

Part III: The Parts of a sendmail Milter

A Milter (mail filter) is a Posix threaded library of routines supplied with the open-source version of sendmail. This part of the book shows how to use the components needed to write a Milter.

Chapter 6, The Roles and Flow of a Milter, describes the overall steps that the Milter uses to parse email and shows what the Milter can and cannot do.

Chapter 7, The Milter Library, describes the library routines supplied for you to use in writing your Milter.

Chapter 8, The xxfi Handler Milter Functions, describes the routines you write to interface with the Milter library.

Chapter 6. The Roles and Flow of a Milter

A Milter is a multithreaded program that is called by (connected to) sendmail whenever mail is received using SMTP. Because sendmail forks a copy of itself for each delivery, there is no part of the sendmail system that is aware of the big picture. A well-designed Milter fulfills a central role in such an arrangement because it can view the big picture. Such a Milter can be a very powerful companion to sendmail.

The sendmail program can support multiple Milter programs and can call each in a predefined order. Also, beginning with sendmail V8.13, Milters can be called in a per-daemon order. For example, the daemon bound to the external network interface might prefer one list of Milters, whereas the daemon bound to the internal network interface might prefer another list.

The role of a Milter depends on two points of view.

From the point of view of a single sendmail process, there are (possibly) many Milters that it can call in sequence.

From the point of view of a single Milter, it receives individual envelopes (messages) in a well-defined order from possibly many simultaneous connections from multiple sendmail processes.

6.1. A Milter's Role in the Middle

A Milter is called by sendmail during each step of the SMTP connection. For example, a Milter is called once just after receipt of the SMTP MAIL FROM: command, and then again just after each SMTP RCPT TO: command.

When the sending site sends a MAIL FROM: command, the address and sendmail macros set by that command are passed to the Milter for review. The Milter can accept, reject, discard, or quarantine the message based on that information.

Figure 6.1 shows a Milter's role in the flow of an email message being sent via SMTP. Notice that it shows two inbound SMTP streams (two inbound email messages) being accepted in parallel by two different invocations of sendmail. The Milter is a multithreaded application that sits between the two invocations and accepts information from both in parallel and asynchronously. That is, the sendmail on the left might be receiving a huge message with many recipients (many RCPT TO: lines), while the sendmail on the right might be receiving a small message with only a single recipient. The two sendmail programs can send information to the Milter at different rates and at different phases of the SMTP conversation.

Figure 6.1. A Milter's role in the flow of SMTP.

Neither invocation of sendmail knows about the existence of the other, whereas (depending on its design) the Milter in the middle can be aware that it is communicating with both.

6.2. A Milter from the Point of View of sendmail

From the point of view of a single sendmail process, the flow of commands when email is received looks like the following, where each Process step is composed of a call to a Milter routine. Note the logic of how sendmail interweaves the calls to (possibly) many Milters:

For each connection, do
|  For each Milter, Process connection
|  For each Milter, Process ELHO/HELO
|  For each envelope, do
|  |  For each Milter, Process the sender
|  |  For each recipient, do
|  |  |  |  For each Milter, Process a recipient
|  |  For each Milter, do
|  |  |  For each header, Process a header
|  |  |  Process end of headers
|  |  |  For each body chunk, Process a body chunk
|  |  |  Process the end-of-data (end-of-message)
|  For each Milter, Process close of envelope (clean up)

For example, when there are multiple Milters, each will review the same connection information before any is given the envelope to review. But each individual Milter is unaware that any other Milter exists.

6.3. The Milter Flow

Individually, each Milter program is called by sendmail in a well-defined order. Each such contact (phase) is one of the steps that constitute the flow of email reception by sendmail. Table 6.1 shows phases of this flow. We will talk about them (in logical order) in the sections to follow.

Table 6.1. The Milter Process Phases

Phase

Section

Description

Connect

6.3.2

The sending machine has just connected to the listening server.

HELO/EHLO

6.3.2

The sending machine greets the listening server.

MAIL FROM:

6.3.3

The sending machine states the envelope sender.

RCPT TO:

6.3.4

The sending machine states one of possibly many envelope recipients.

Headers

6.3.5.1

Each of possibly many header lines inside the message is provided one at a time.

End of headers

6.3.5.1

Final review after all headers have been processed.

Body

6.3.5.2

The body of the message is presented in one or more chunks.

End of message

6.3.6

The entire message has arrived and the envelope is finished.

Abort

6.3.7

Some part of sendmail has stopped accepting the message. The Milter should abort the message.

Cleanup

6.3.8

The connection is finished.

In Chapter 7, when we discuss the Milter library, you will see that these phases have corresponding subroutines in the milter library.

As part of a Milter's startup phase (and before it is sent anything by sendmail), it registers with its library, stating what information it wants to process and what information it does not want to process. For example, a Milter might be interested only in SMTP MAIL FROM: information. Milter startups are covered in section 7.2.

Here, we discuss only the SMTP phases during which a Milter can be contacted and the replies that the Milter can give to sendmail as a result of processing each phase.

Note that, in addition to SMTP commands, Milters have access to selected sendmail macros (see section 7.3.1).

6.3.1. Return Values from Multiple Milters

In the next several sections, we show specifically what happens when a Milter accepts, continues, rejects, and tempfails an aspect of an envelope.[1] Here we briefly describe how those decisions interact when there are multiple Milters.

[1] Tempfails means temporarily fails, generally resulting in the sending site trying to deliver again at a later time.

For example, suppose that four Milters are reviewing a connection:

MilterA accepts connection
MilterB continues
MilterC rejects
MilterD is never asked to review

Here, MilterA accepted the connection. This tells the milter-library that this Milter will not need to review any more of the envelope. Its work is done.

MilterB elects to continue. This tells the milter-library that it does not object to the connection but would still like to review more of the envelope.

MilterC rejects the connection. This completes all Milter work concerning this connection. There is no need to continue to MilterD because there is no way to "unreject" a connection.

MilterB would like to review the rest of the envelope but will not be given the opportunity because a rejection of the connection causes all envelopes sent over that connection to be rejected.

The effects of decisions, as you can see, vary depending on how many Milters there are and what decision each makes in the chain of Milters. In general, only the reject and tempfail decisions affect downstream Milters. Other decisions, including the accept decision, do not.

6.3.2. Milter Replies for Connect and EHLO/HELO

We group the connection information and the EHLO/HELO information here because both are passed to each Milter in sequence for review before any Milter is given the envelope information. As a consequence, if one Milter rejects a connection, later Milters will not be given the opportunity to review the same information.

When sendmail receives a connection request from a sending host, it can call the first (of possibly more than one) Milter and supply it information about that request before deciding to accept or reject the connection. This is called the Connection phase. The information sent is the connecting host's name, its IP address, and related sendmail macros. If the current Milter chooses not to see this information or if it sees the information and replies with Accept or Continue (see Table 6.2), sendmail may accept the connection, although a later Milter can still cause sendmail to reject the connection.[2]

[2] Rule sets (or rule set lookups in the access database) can still reject the connection even if the Milter has accepted it with an Accept or Continue reply.

Table 6.2. Actions for Connect and HELO/EHLO

Milter Reply

Effect on sendmail

Accept

Accept the current connection; no further review by this Milter is needed. The Milter will not be contacted again until the connection closes. A subsequent Milter can still reject the connection.

Reject

If rejecting the Connection phase, the connecting site is given a 554 reply code.[a] Otherwise, for the EHLO/HELO phase, the connecting site is given a 220 reply, and each subsequent command from that connecting site is given a 550 reply except for QUIT (given a 221 reply) and NOOP (given a 250 reply). The Milter will not be connected to again for this connection. Subsequent Milters will not get a chance to review this connection.

Tempfail

If processing the Connect phase, the connection is rejected with a 421 SMTP reply code if the Milter has been configured to use that code. For EHLO/HELO rejection, subsequent SMTP commands from the sending site for this session will be rejected with a 4yz SMTP reply code. The Milter will not be contacted again for this connection. Subsequent Milters will not get a chance to review this connection.

Discard

All messages sent during this SMTP session (e.g., all messages sent in the current connection) will be silently discarded (and possibly logged). The filter will not be contacted for the rest of this SMTP session (for this connection). Subsequent Milters will not get a chance to review this connection.

Continue

Acknowledge the information received. The filter will be contacted with the other requested information when and if that information becomes available.

[a] Reply codes are documented in RFC2821.

If no Milter rejects the connection, sendmail sends its initial 220 greeting and waits for the other side to issue an EHLO or HELO SMTP command. When the EHLO or HELO arrives, sendmail supplies the information following that greeting, and related sendmail macros, to the first Milter. Again, if that Milter chooses not to use this information or if it sees this information and replies with Continue, then sendmail will tentatively accept the HELO or EHLO for that Milter but will still contact additional Milters, if any, which may reject it. If no Milter has rejected the HELO or EHLO, sendmail will accept the HELO or EHLO and continue with the SMTP conversation.

In summary, each Milter gets to review the connection information and then each Milter gets to review the EHLO or HELO informationall this occurring before any Milter gets the envelope information.

But note that a potential problem can arise because sendmail can be configured to allow the sending site to skip the HELO and proceed to the envelope. This means that a Milter should not necessarily expect to get EHLO or HELO information.

Also note that another problem can arise because EHLO or HELO can arrive at unexpected times (it is treated as a RSET by sendmail). This means that this EHLO or HELO phase should never presume that it will be called immediately following the Connect phase.

In fact, if an envelope is being processed when EHLO or HELO arrives, the Milter will be asked to abort processing the current envelope altogether. And even if the envelope is not aborted, EHLO or HELO may fail to be called, even when you planned and expected it to be.

6.3.3. Milter Replies to MAIL FROM:

After the initial connection is accepted and EHLO or HELO sent and accepted, the sending host next sends the SMTP MAIL FROM: command to specify the envelope sender for the message. This begins the envelope (or MAIL FROM:) phase of review (see Table 6.3). Recall that each Milter is given a chance to review the envelope sender before any Milter is given the list of recipients.

Table 6.3. Milter Replies to the SMTP MAIL FROM: Command

Milter Reply

Effect on sendmail

Accept

Accept the current message (envelope); no further review by this Milter is needed. The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope sender.

Reject

The MAIL FROM: command is rejected with a 5yz SMTP reply code.[a] The Milter will not be connected to again for this envelope but can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope sender or envelope.

Tempfail

The MAIL FROM: command is rejected with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope sender or envelope.

Discard

The current envelope is discarded. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope sender or envelope.

Continue

Acknowledge the information received. The filter will be contacted with the other requested information when and if that information becomes available.

[a] Reply codes are documented in RFC2821.

The envelope sender address (as specified in the MAIL FROM: command) is given to the Milter as is, as well as any Extended SMTP (ESMTP) arguments to the MAIL FROM: command. Selected sendmail macros (see section 7.3.1) are also passed to the Milter.

Note that there can be only one MAIL FROM: command for each envelope, and it must be accepted (not rejected) for the envelope as a whole to be accepted.

6.3.4. Milter Replies to RCPT TO:

After the MAIL FROM: envelope sender, the sending site will submit one or more recipients to the receiving sendmail program using the SMTP RCPT TO: command. In this transaction each recipient specified is submitted to the Milter, one at a time, for review. When there are multiple Milters, all of them review each recipient before any is allowed to review the next recipient.

If the Milter replies with Continue or if the Milter does not want envelope recipient information, that envelope recipient is accepted.[3]

[3] It may later be rejected by a rule set (or by rule set lookups in the access database).

Table 6.4 shows the possible replies that the Milter can make for each envelope recipient reviewed. Additional information about each recipient is available in sendmail macros (see section 7.3.1). Remember that each Milter is called once for each recipient and that there may be many recipients.

Table 6.4. Milter Replies to the SMTP RCPT TO: Command

Milter Reply

Effect on sendmail

Accept

Accept the current message (envelope); no further review by this Milter is needed. The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject this recipient.

Reject

Reject the current recipient with a 5yz SMTP reply code.[a] This recipient part of the Milter can be contacted again if there are multiple recipients, or sendmail might go directly to the SMTP DATA phase after all recipients have been processed (if there is at least one good recipient). Subsequent Milters will not get a chance to review this recipient.

Tempfail

Temporarily reject the current recipient with a 4yz SMTP reply code. This recipient part of the Milter may be contacted again if there are multiple recipients, or sendmail might go directly to the SMTP DATA phase after all recipients have been processed (if there was at least one good recipient). Subsequent Milters will not get a chance to review this recipient.

Discard

The current envelope is discarded. The Milter will not be contacted again for this envelope, even if there are more recipients. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this recipient.

Continue

Acknowledge the information received. The filter will be contacted with other requested information when and if that information becomes available.

[a] Reply codes are documented in RFC2821.

6.3.5. Milter Replies with SMTP DATA

After the envelope sender and all envelope recipients have been processed, and if there are any good recipients remaining, sendmail proceeds with the SMTP DATA part of message reception. The data part is composed of the entire message (headers and body together), and sendmail first provides the headers to the Milter, one header at a time, followed by the message body one chunk at time.

This is the last part of the envelope review phase. Recall that each Milter is given the entire data part to review before the next Milter is given a chance to review the data part. That is, MilterA gets to review (and possibly change) headers and the message body before MilterB gets a turn to do the same.

We cover headers first and then the message body, and we conclude with the end-of-data (end-of-message) processing.

6.3.5.1 Milter Handles Headers

The sendmail program sends every header to the Milter, one header at a time, until all the headers have been reviewed or until the message is rejected because of a header. There is no provision for headers to be provided in any particular order other than the order provided to sendmail by the sending site, so the Milter must perform any ordering itself (if special ordering is important to it).

For each header reviewed, the Milter can reply with any of the replies listed in Table 6.5.

Table 6.5. Milter Replies for Each Header Reviewed

Milter Reply

Effect on sendmail

Accept

The current header marks the message as acceptable to the Milter, and the message is accepted. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope.

Reject

Reject the current SMTP DATA phase with a 5yz SMTP reply code.[a] The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope,if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Tempfail

Temporarily reject the SMTP DATA phase with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Discard

The current envelope is discarded. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Continue

Acknowledge the information received. The filter will be contacted with other requested information when and if that information becomes available.

[a] Reply codes are documented in RFC2821.

Even though the Milter receives headers one at a time, it can, if it prefers, gather them up until all have been received and then review them in a preferred order or by using a more complex logic. This means that the Milter is called once per header and then called again after all headers have been received but before the message body has been received.

The separate call after all headers have been received allows the Milter to decide what to do if, for example, a header was missing or what to do if the count of a particular header is wrong, or simply to review time-ordered headers such as the Received: header.

Note that the Milter can remove headers, add new headers, and change existing headers only during the End-of-message phase (see section 6.3.6).

6.3.5.2 Milter Replies per Body Chunk

The sendmail program sends the message body to the Milter, one chunk at a time, until the entire body has been reviewed or until the message is rejected because of an objectionable chunk. There is no guarantee that body chunks will be line oriented or that all of them will be the same size. Chunks may be split anywhere, even in the middle of a word for which you may be searching.[4]

[4] If possible, it's a good idea to buffer chunks in memory or in a temporary disk file so that the body can be fully assembled before you parse it for spam.

When there are multiple Milters, each reviews the entire body (through the end-of-data) before the next Milter is allowed to review the headers and body.

For each chunk of body the Milter can reply with any of the replies listed in Table 6.6.

Table 6.6. Milter Replies for Each Body Chunk Reviewed

Milter Reply

Effect on sendmail

Accept

The current body chunk under review marks the message as acceptable to the Milter, and the message is accepted. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope for body chunk reasons.

Reject

Reject the current SMTP DATA phase with a 5yz SMTP reply code.[a] The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Tempfail

Temporarily reject the SMTP DATA phase with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Discard

The current envelope is discarded. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Continue

Acknowledge the information received. The filter will be contacted with other requested information when and if that information becomes available.

[a] Reply codes are documented in RFC2821.

6.3.6. Milter Replies at End-of-Message

After the Milter has received all the body chunks (after the SMTP DATA phase is complete), the Milter is called again to perform many tasks before the acknowledgment of the SMTP dot is sent by sendmail. The following are among the tasks that the Milter can perform at this time:

Remove one or more headers from the message.

Add one or more new headers to a message.

Replace the value part of any header with a new value.

Replace the message body with a new or rewritten body.

Cause the message to be quarantined (as of sendmail V8.13 ).

Remove a recipient from the envelope recipient list.

Add a recipient to the envelope recipient list.

Deallocate allocated envelope-specific resources.

Here we only note these extra capabilities, but we will discuss them in detail in the chapters to follow. In the following we focus on the replies the Milter can give to sendmail at the end of message (see Table 6.7). Remember that these replies follow the SMTP DATA phase but are ahead of the acknowledgment to the dot.

Table 6.7. Milter Replies at End-of-Message

Milter Reply

Effect on sendmail

Accept

Accept the message. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope, if there is one on this same connection. A subsequent Milter can still reject the envelope.

Reject

Reject the message. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Tempfail

Temporarily reject the message. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Discard

Discard the message. The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Quarantine

Beginning with sendmail V8.13, the envelope can be quarantined (held in the queue pending review). The Milter will not be contacted again for this envelope. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

Continue

The same as Accept.

6.3.7. Milter Handles Abort

Milters are not the only things that can reject, temporarily fail, discard, or quarantine a message. Rule sets in the configuration file can also dispose of a message, as can rule set lookups in the access database. In the event that sendmail, rather than the Milter, rejects the message, sendmail will call the Milter's abort-handling routine to let it know.

Ordinarily, the end-of-message routine is called last for an envelope. If the message is aborted, this Abort phase will be called in place of that end-of-message routine. For any given envelope, both will never be called.

A Milter must use this as an opportunity to clean up after itself. The Milter merely cleans up at this phase. It should not, and sendmail doesn't expect it to, return any reply information.

Note that abort can arrive only after the envelope sender review and the end-of-data review.

6.3.8. Milter Post-Connection Cleanup

A Milter, like any other program, will likely have allocated certain resources to the task of reviewing a connection or envelopes. Such dedicated resources might be ones such as open file descriptors or allocated memory. The Milter should clean up after itself all at once after the connection is finished (after zero or more envelopes have been received over that connection).

Remember that a Milter is a multithreaded program. Nonthreaded programs, such as sendmail, fork to handle multiple I/O streams, and the fork, and corresponding exit, automatically deallocates resources. But a multithreaded program does not have that luxury. Instead, it must be very careful to always deallocate its own resources, lest it accidentally use them up and become unable to continue.

Note that the Milter merely cleans up during this phase. It should not, and sendmail doesn't expect it to, return any reply information.

Chapter 7. The Milter-Library

Milters are constructed from routines found in the milter-library (libmilter.a or libmilter.so) that are used by your Milter to communicate decisions back to sendmail. These routines are called smfi routines because the name of each begins with the smfi_ prefix.

One routine, smfi_register(), tells sendmail (via the milter-library) which of your Milter's functions handles which phase of analysis. For example, if your Milter is interested only in envelope recipients, it might use smfi_register() to tell sendmail (via the milter-library) that sendmail should contact your Milter only during the envelope recipient phase and call a handler function inside your Milter, called xxfi_rcpt(), to handle that phase.[1] (We describe handler functions in Chapter 8.)

[1] We use the prefix xxfi_ to keep this book in line with the libmilter/docs documentation. You are actually free to use almost any name you wish.

In this chapter, we first show how the main() routine inside your Milter source code should interact with the milter-library. Then we explain all the smfi routines and how to use each one.

7.1. Overview

The milter-library provides a set of routines that your Milter can call to perform message review actions. Each entry point into the library begins with the smfi_ prefix. The complete list is shown in Table 7.1.

Table 7.1. smfi Milter-Library Entry Points

smfiRoutine

Section

EOM

Description

smfi_register()

7.2.1

Declare which handler functions to call for which phase and set flags.

smfi_setconn()

7.2.3

Specify the interface socket to use.

smfi_opensocket()

7.2.3

Create the interface socket (sendmail V8.13 and later).

smfi_settimeout()

7.2.4

Set sendmail/Milter timeout.

smfi_main()

7.2.5

Begin multithreaded operations.

smfi_getsymval()

7.3.1

Fetch a sendmail macro's value.

smfi_getpriv()

7.3.3

Fetch private context data pointer.

smfi_setpriv()

7.3.2

Initialize private context data pointer.

smfi_setreply()

7.3.4

Set SMTP reply code.

smfi_setmlreply()

7.3.5

Set a multiline SMTP reply code (sendmail V8.13 and later).

smfi_setdbg()

7.3.6

Set the debugging level (sendmail V8.13 and later).

smfi_setbacklog()

7.3.7

Set the listen(2) queue size (sendmail V8.13 and later).

smfi_addheader()

7.4.2

*

Add a header.

smfi_insheader()

7.4.3

*

Insert a header.

smfi_chgheader()

7.4.4

*

Change or delete a header.

smfi_addrcpt()

7.4.5

*

Add envelope recipient.

smfi_delrcpt()

7.4.6

*

Delete envelope recipient.

smfi_replacebody()

7.4.7

*

Replace message body.

smfi_progress()

7.4.8

*

Report operation in progress (sendmail V8.13 and later).

smfi_quarantine()

7.4.9

*

Quarantine a message (sendmail V8.13 and later).

smfi_stop()

7.4.10

Cause a controlled shutdown (sendmail V8.13 and later).

Note that those marked with an asterisk under EOM in Table 7.1 can be called only from within the end-of-message handler function.

7.1.1. HTML Documentation

In addition to the documentation that is in this book, the open source sendmail provides documentation in the directory libmilter/docs in HTML format. You can read it using a set of commands like this:

$ cd libmilter/docs
$ netscape file://'pwd'/index.html   If running X-11
$ lynx index.html                    If  not running X-11

You will find the information we discuss in this chapter under the link API Documentation. You may find it helpful to read that documentation in parallel with this chapter, but we have attempted to write this chapter so that it stands alone and is complete on its own.

7.1.2. xxfi VERSUS smfi

When you write your Milter, you will be calling milter-library routines to perform certain tasks. All milter-library routines begin with the smfi_ prefix. They are all listed in Table 7.1.

When you write your Milter, you will also compose routines inside your Milter to handle specific phases of the Milter's interaction with sendmail (described in the Chapter 6). Each phase corresponds to an entry point inside your Milter. For convenience, and for correspondence to the HTML Milter documentation, the routines you will write are given names that are prefixed with xxfi_. We call these your handler functions.

We discuss the smfi milter-library entry points in this chapter and your handler functions in Chapter 8.

7.2. main()

All C language programs begin execution in a function called main(). Your Milter is no exception. In general, main() is where you perform initialization tasks and begin program execution. For a Milter, a few specific bits of initialization are required:

Before you can use any Milter-library routine, you must include the required Milter header file:

#include <libmilter/mfapi.h>

Before you can use the milter-library, you must declare which Milter phases you will accept and which you will ignore. You can make this declaration inside main().

The Milter communicates with sendmail over a socket. Before you call smfi_main() you must define and initialize the socket to use.

Replies to sendmail are sent over the socket. By default, those replies can time out if any takes more than 7210 seconds (approximately 2 hours). Inside main() you can change that default timeout.

Each thread of a multithreaded Milter needs private variables dedicated to the particular email message being screened. Inside main(), or later, you declare what information will be made available to your handler functions.

After all initialization, main() (or a function called from main()) calls smfi_main() to begin the Milter's execution.

7.2.1. Structure smfiDesc and smfi_register()

Inside your Milter, you will write functions to handle the various phases of an email message's screening. You declare the phases you choose to accept and those you choose to ignore (generally inside your main() routine) by filling out a special structure called smfiDesc. Table 7.2 shows the items that structure must contain and the order in which they must be declared. After the structure has been declared and filled out, you must then register it by calling smfi_register().

Table 7.2. smfiDesc Structure Items

Item

Section

Description

name

7.2.1

The name of this Milter

version

n/a

The constant SMFI_VERSION

flags

7.2.2

Tune behavior

connect

8.4

Your xxfi_connect() routine or NULL if ignored

helo

8.5

Your xxfi_helo() routine or NULL if ignored

envfrom

8.6

Your xxfi_envfrom() routine or NULL if ignored

envrcpt

8.7

Your xxfi_envfrcpt() routine or NULL if ignored

header

8.8

Your xxfi_header() routine or NULL if ignored

eoh

8.9

Your xxfi_eoh() routine or NULL if ignored

body

8.10

Your xxfi_body() routine or NULL if ignored

Eom

8.11

Your xxfi_eom() routine or NULL if ignored

abort

8.12

Your xxfi_abort() routine or NULL if ignored

close

8.13

Your xxfi_close() routine or NULL if ignored

The structure smfiDesc can be global (outside your main() routine) or local (inside your main() routine). The location is up to you. In the following code we show a way to declare it outside of main(), but you can declare it in a position and style that makes you most comfortable. Note that the numbers to the left of the code are only for reference and are not part of the code.

158    struct smfiDesc smfilter =
159    {
160      "MyMilter",       /* filter name */
161      SMFI_VERSION,     /* a constant */
162      SMFIF_ADDHDRS,    /* flags */
163      xxfi_connect,     /* handle connection */
164      xxfi_helo,        /* handle SMTP HELO/EHLO */
165      xxfi              /* handle envelope sender */
166      xxfi_envrcpt,     /* handle envelope recipient */
167      xxfi_header,      /* handle header */
168      xxfi_eoh,         /* handle end-of-headers */
169      xxfi_body,        /* handle body chunk */
170      xxfi_eom,         /* handle end-of-message */
171      xxfi_abort,       /* handle message aborted */
172      xxfi_close,       /* handle connection cleanup */
173    };

When you initialize a structure, you do so (line 158) by declaring it with the struct keyword followed by its type (the smfiDesc) and then giving it a name (here smfilter) that will later be passed to smfi_register().

The first item initialized inside this structure (line 160) is the name you gave your Milter, which should be the same name you gave to sendmail when you declared this Milter inside sendmail's mc configuration file.

"MyMilter", /* filter name */     InsmfiDesc 
INPUT_MAIL_FILTER(`MyMilter', ...            Inmc file 

Although the names are not required to be the same, it is good form and will help prevent later confusion, especially if you have installed many Milters. But note that this is the name that will be used when the Milter produces syslog records.

Note that if you set the name to an empty string (nothing between the quotation marks), the name will be accepted as an empty string. If you specify the name using a NULL pointer, the name will be silently converted to "Unknown" by the smfi_register() routine.

The second item (line 161) is the version of the milter-library. This is a constant that must always be the literal expression SMFI_VERSION. If you use any other value here, the following message may be logged, and smfi_register() will fail:

name: smfi_register: version mismatch application: bad_num != milter: SMFI_VERSION

Here, name is the name you declared, bad_num is the bad value you specified for the version (line 161), and SMFI_VERSION is the value of that constant as specified in libmilter.h.

The third item (line 162) is the flags, which we discuss soon.

The flags are followed by the names of 10 functions. These are functions you will write yourself. Each corresponds to a Milter phase. If you wish to exclude any (ignore it and cause sendmail to instead use its default behavior), you can specify NULL here instead of a function name. For example, if you intend to ignore end-of-message, you might change (line 170) to look like this:

169    xxfi_body,        /* handle body block */
170    NULL,             /* handle end-of-message */
171    xxfi_abort,       /* handle message aborted */

We suggest that these handler function names begin with the xxfi_ prefix. We do this so that this book and the sendmail open source HTML documentation will share the same conventions. The function names you choose do not, of course, need to begin with that prefix.

Each of these 10 xxfi_ prefixed function declarations corresponds to the Process steps outlined in Chapter 6:

For each connection, do
|  For each Milter, call xxfi_connect()
|  For each Milter, call xxfi_helo()
|  For each envelope, do
|  |  For each Milter, call xxfi_envfrom()
|  |  For each recipient, do
|  |  |  |  For each Milter, call xxfi_envrcpt()
|  |  For each Milter, do
|  |  |  For each header, call xxfi_header()
|  |  |  call xxfi_eoh()
|  |  |  For each body chunk, call xxfi_body()
|  |  |  call xxfi_eom()
|  For each Milter, call xxfi_close()

7.2.1.1 Register the smfiDesc Structure with smfi_register()

After you have filled out the struct smfiDesc with the required information, you next call the smfi_register() routine:

if (smfi_register(smfilter) == MI_FAILURE)
{
   fprintf(stderr, "smfi_register failed\n");
   exit(EX_UNAVAILABLE);
}

Here, smfi_register() is given, as its single argument, the variable name you used when declaring the smfiDesc structure. Recall (line 158) that the name you gave the smfiDesc structure was smfilter, so that is the name supplied as the single argument to smfi_register().

Always check for the success of this smfi_register() call. If that call returns MI_FAILURE you must exit your Milter immediately. Possible reasons for failure include the following:

The milter-library could not allocate needed memory.

The version did not match the current version (this error is logged).

7.2.2. The smfiDesc Structure's Flags

The flags line in smfiDesc (line 162) consists of zero or more constant expressions that are bitwise OR'd together to specify the capabilities your Milter will possess. The complete list of those constant expressions is shown in Table 7.3, along with the capabilities each bestows.

Table 7.3. smfiDesc Flags

flag

Ability

SMFIF_ADDHDRS

This Milter can add new headers to those already present in the message. The added headers will be seen by subsequent Milters.

SMFIF_CHGHDRS

This Milter can remove existing headers and can change the value of any header into a new value. The changed headers will be seen by subsequent Milters.

SMFIF_CHGBODY

This Milter can replace all of the body of the message with a new body (partial replacements are not allowed). Note that subsequent Milters will see the new body and not the original body. Changing the body can impact performance.

SMFIF_ADDRCPT

This Milter can add recipients to the existing recipient list. If the original message had no legal recipients, this opportunity will not present itself.

SMFIF_DELRCPT

This Milter can remove recipients from the existing recipient list.

SMFIF_QUARANTINE

This Milter can quarantine this message (sendmail V8.13 and later).

SMFIF_NONE

This Milter will perform none of the above.

Most Milters will want to perform one or more of these activities. So for each desired activity, bitwise OR that activity with the others (the bitwise OR symbol is a single vertical bar):

SMFIF_ADDRCPT | SMFIF_DELRCPT | SMFIF_CHGBODY, /* flags */

There is no constant that causes all activities to be allowed, so if that is your intention, you will need to list (bitwise OR) all of them together. If you wish to disallow all activities you can specify SMFIF_NONE.

SMFIF_NONE, /* no special flags */

7.2.3. The smfi_setconn() and smfi_opensocket() Routines

Milters can communicate with sendmail over a named pipe or over a TCP network socket.[2] Specify which of these your Milter will use by calling smfi_setconn() with an appropriate argument. Those arguments (shown in Table 7.4) are strings that begin with one of four possible special prefixes. Note that the colon is a literal part of each prefix.

[2] A TCP socket is useful if one Milter will service many sendmail machines.

Table 7.4. smfi_setconn() Prefix Meanings

Prefix

Meaning

unix:

A UNIX named socket

local:

Same as unix:

inet:

An IPv4 TCP network socket

inet6:

An IPv6 TCP network socket

7.2.3.1 The unix: or local: Prefix Defines a Named Pipe

For a named socket, the smfi_setconn() prefix is followed by the full path to the named socket, including the file name of the socket. For example:

smfi_setconn("unix:/var/run/MyMilter.sock");

When the path is hard coded, as with the quoted string here, the Milter can be run by root, as long as all components of the path are writable only by root. If, on the other hand, you read the path, or full socket declaration, from the command line, you should never run the Milter as root.[3]

[3] You are generally advised not to run any Milter as root unless absolutely necessaryif the Milter needs to bind to a privileged port, for example. But even then, it is advisable to bind as root and then immediately change to a non-root user for the rest of the session.

case 'p':
  smfi_setconn(optarg);            /* Don't do this if run as root */

Also note that the socket itself will be created after the program has started up in multithreaded mode (after the call to the smfi_main() routine).[4] As a consequence, the socket will be created using the umask of your process. Just to be safe, set that umask(2) to 0700 (or 0770 if you defined the RunAsUser option in your sendmail mc configuration file, because sendmail and your Milter must both be able to read and write with that socket).

[4] Unless you call smfi_opensocket() (see section 7.2.3.4) yourself before calling smfi_main().

(void) umask(0700);
smfi_setconn("unix:/var/run/MyMilt.sock");

Remember that the prefix and path you specify with smfi_setconn() must match the prefix and path you specify when you declare this Milter in your sendmail mc configuration file:[5]

[5] The prefix may be unix: in one and local: in the other without harm.

INPUT_MAIL_FILTER(`MyMilter', `S=unix:/var/run/MyMilt.sock')

If they differ, sendmail will not be able to communicate with your Milter. Note that sendmail does not create the named socket; the Milter does. This means that Milters should generally be started before sendmail.

7.2.3.2 The inet: and inet6: Prefixes Define TCP Sockets

The inet: and inet6: prefixes cause the Milter to listen for connections from sendmail using a TCP network socket. The inet: prefix specifies an IPv4 socket, whereas the inet6: specifies an IPv6 socket. Either is followed by information in two possible forms:

port@interface_hostname
port@interface_address

Here, the port is the port number on which the Milter will listen. The @ is literal. The interface_hostname is the host name or domain name associated with the network interface to which the Milter will bind to listen. The interface_address is the IP address to which the Milter will bind to listen.

The entire expression (the prefix, port, @, and name or address) is passed to the smfi_setconn() routine as a single string. For example:

smfi_setconn("inet:9999@localhost");

Remember that the socket you specify with smfi_setconn() must match the socket you specify when you declare this Milter in your sendmail mc configuration file:

INPUT_MAIL_FILTER(`MyMilter', `S=inet:9999@localhost')

If they differ, sendmail will not be able to communicate with your Milter.

Note that the TCP socket form of communication (as opposed to the UNIX domain socket described earlier) allows your Milter to run as root.[6] But beware of accepting the socket name from the command line, because the socket name may include a specification that should never be run as root.[7]

[6] It allows it, that is, provided you do not create, overwrite, or remove files. If you do, you would do better to design your Milter to run as a user other than root.

[7] In one configuration, the UNIX domain socket can be removed by root on startup.

7.2.3.3 smfi_setconn() Errors

The smfi_setconn() routine fails only if it cannot allocate memory. Errors that occur when the connection is set up will not be discovered until later (when the Milter begins to run in multithreaded mode) because smfi_setconn() does not set up the actual connection (see also section 7.2.3.4).

The smfi_setconn() routine, including error checking, is used this way:

(void) umask(0700);
if (smfi_setconn("unix:/var/run/MyMilter.sock") == MI_FAILURE)
{
   fprintf(stderr, "smfi_setconn() %s\n",
      strerror(errno));
   exit(EX_SOFTWARE);
}

Thus it may be a good idea to set up the connection ahead of time (to check for errors before entering multithreaded mode) by calling the smfi_opensocket() routine.

7.2.3.4 The smfi_opensocket() Routine Opens a Socket (V8.13 and Later)

The smfi_opensocket() routine causes your socket to actually be set up. You can use it to make certain that the socket specification was proper and that all permissions worked. If you don't want to precheck, you can skip this routine and it will be called on your behalf automatically when the program later enters multithreaded operation.

You use smfi_opensocket() this way:

if (smfi_opensocket(flag) == MI_FAILURE)
{
   fprintf(stderr, "smfi_opensocket() %s\n",
      strerror(errno));
   exit(EX_SOFTWARE);
}

The smfi_opensocket() routine takes a single argument, flag, which indicates whether or not to remove any existing UNIX domain socket before trying to create a new one. If flag is false (has a zero value), no removal will be attempted; if it is true (has a nonzero value) and if a UNIX domain socket is specified, then smfi_opensocket() will try to remove the socket.

The smfi_opensocket() routine causes the argument you supplied earlier to smfi_setconn() to actually set up a socket. Because smfi_opensocket() can potentially undertake many complex steps, it can fail for a wide variety of reasons:

A directory in the path to a named socket did not exist.

The Milter lacked write permission to a directory or to an existing named socket.

A disk I/O error prevented the requested action.

Memory could not be allocated.

A socket was already in use by another process.

A port number was in the reserved range, and the Milter lacked permission.

When an error occurs, the global variable errno will be set with a value that indicates the reason for the failure. Hence, the code uses the following to report the error:

fprintf(stderr, "smfi_opensocket(): %s\n",
   strerror(errno));

Here, the strerror() routine causes the error in errno to print in a human-understandable form. For example:

smfi_opensocket(): No such file or directory

Note that the smfi_opensocket() routine is more valuable when the connection information is collected from the command line and less valuable when the connection information is hard coded.

7.2.4. The smfi_settimeout() Routine Initializes the Connection Timeout

Recall that a Milter is a server, listening for connections from sendmail. The Milter protocol begins with sendmail contacting the Milter and asking it to send it information for review. The Milter will wait (listen) a specified amount of time for that connection to be established before considering the wait too long. If the wait time limit is exceeded, the Milter simply times out and listens again with the same timeout criteria.

The default wait is 7210 seconds (approximately 2 hours). This default is sufficient for most needs. You will need to change it only in rare circumstances.

The smfi_settimeout() routine is used this way:

(void) smfi_settimeout(seconds);

Here, seconds is an integer that specifies the amount of time to wait in seconds. No check is made for absurdly large or negative values. Because no useful information is returned, the call is prefixed with a (void) declaration.

7.2.5. The smfi_main() Routine Launches Multithreaded Mode

The last item in your main() function, the call to smfi_main(), performs two tasks on your behalf:

It establishes your listening connection with smfi_opensocket() (see section 7.2.3.4) if you have not already done so. Note that if smfi_opensocket() fails, smfi_main() will also fail and will return MI_FAILURE.

It puts your program into multithreaded operation, waiting for connections from sendmail and handling each one by calling an appropriate handler function when appropriate.

The smfi_main() routine takes no arguments and generally appears near the end of your main() function:

if (smfi_main() == MI_FAILURE)
{
   /* Handle connection failure here */
}
/* handle normal end of program cleanup here */

Note that smfi_main() does not put your program into the background to run as a daemon. You must take care of that yourself. You can do so either by writing your own code or by launching your Milter from a script that puts it in the background upon startup.

7.2.6. In Summary, a Minimal main() Function

You have seen all the pieces that go into developing a main() function. Here, we gather those lessons into one place and present a minimal main() function. Note again that, in the following, the numbers at the left are for reference only and are not a part of the code.

 1  #include <stdio.h>
 2  #include <errno.h>
 3  #include <libmilter/mfapi.h>
 4  #include <sysexits.h>
 5  #include <string.h>
 6
 7  int
 8  main(argc, argv)
 9    int argc;
10    char **argv;
11  {
12    struct smfiDesc smfilter = {
13      "MyMilter",     /* milter name */
14      SMFI_VERSION,   /* a constant */
15      SMFIF_NONE,     /* no flags */
16      NULL,    /* no handler functions yet */
17      NULL,
18      NULL,
19      NULL,
20      NULL,
21      NULL,
22      NULL,
23      NULL,
24      NULL,
25      NULL
26    };
27
28    if (smfi_register(smfilter) == MI_FAILURE)
29    {
30       fprintf(stderr, "smfi_register failed\n");
31       exit(EX_UNAVAILABLE);
32    }
33
34    if (smfi_setconn("inet:9999@localhost") == MI_FAILURE)
35    {
36       fprintf(stderr, "smfi_setconn() %s\n",
37         strerror(errno));
38       exit(EX_SOFTWARE);
39    }
40
41    if (smfi_opensocket() == MI_FAILURE)
42    {
43       fprintf(stderr, "smfi_opensocket() %s\n",
44         strerror(errno));
45       exit(EX_SOFTWARE);
46    }
47
48    (void) smfi_settimeout(600);
49
50    /* close any unused file descriptors here */
51
52    return smfi_main();
53  }

The program begins by including five #include files:

The stdio.h included file (line 1) defines needed macros, such as NULL.

The errno.h included file (line 2) defines the errno variable (and possibly the strerror() library function).

The libmilter/mfapi.h included file (line 3) defines Milter macros, such as SMFI_VERSION, and the smfiDesc structure type itself.

The sysexits.h included file (line 4) defines EX_UNAVAILABLE and EX_SOFTWARE.

The string.h included file (line 5) defines the prototype for strerror(3).

The main() function itself begins in the standard way (line 8), where we declare the arguments it will accept. The main() function doesn't use these arguments, but we declare them because it will use them eventually, as the code develops.

Inside main(), we first declare the smfiDesc structure (line 12) that will determine how this Milter will operate. Note that it doesn't yet set any flags (line 15), nor does it declare any handler functions (line 16). We cover the handler functions in Chapter 8 and add flags only when those routines need them.

Next, main() calls smfi_register() (line 28) to register its smfiDesc structure with the milter-library. This step is needed so that the milter-library will know how and when to call this Milter. If registration fails, the program prints an error (line 30) and exits (line 31).

In this example, we hard-coded the connection (line 34) as a TCP socket (the inet: prefix) listening on port 9999 of the localhost loopback interface. The program calls smfi_setconn() to register this connection. If the registration fails, the program prints an error message (line 36) and exits (line 38).

But rather than wait for smfi_main() to set up the socket, we prefer to perform that task ourselves inside main(). The socket is set up (line 41) with a call to smfi_opensocket(). If the socket cannot be established, the program prints an error (line 43) and exits (line 45).

Although it is not strictly necessary, we set the interval (line 48) for the Milter to wait for a connection from sendmail to 10 minutes (600 seconds) by having the program call smfi_settimeout().

As the last step in main(), we call smfi_main() to begin multithreaded operation (line 52) and arrange for any error returned from smfi_main() to be returned (using the return keyword) directly to the calling environment.

This minimal program (we call it main.c) will compile and run.[8] It does nothing yet, so just run it and then kill it.

[8] This example of compiling is specific to Solaris.

$ gcc -D_REENTRANT main.c -lmilter -lresolv -lsocket -lnsl -lpthread
$ ./a.out
^C

As you can see, the command line needed to compile this small Milter is quite long. The -l items are libraries that are needed. Most are needed because of the milter-library. The -lnsl is required for the Solaris operating system but is not needed for other operating systems. The -D_REENTRANT may or may not be required, depending on your compiler and operating system, but you should include it for safety. It won't hurt even if it is not needed. Most of these complications will be hidden later inside a makefile, but, for now, just type in the whole line.

After you run (the ./a.out), then kill the Milter by pressing Ctrl + C. It will take a few moments for your prompt to return. This delay is normal and is present because the milter-library needs time to shut down all its threads.

In Chapter 8 we will show you how to write the handler functions defined by the smfiDesc structure.

7.3. The smfi Data Access Routines

In this section we describe seven smfi milter-library routines that are available for use by all (or most of) your handler functions. These are categorized as data access routines because they allow each of your handler functions to access (put, get, or set) needed data:

The smfi_getsymval() routine fetches sendmail macros and their values.

The smfi_setpriv() routine defines a private data type that is sharable among all your handler functions.

The smfi_getpriv() routine fetches the private data type defined by smfi_setpriv().

The smfi_setreply() routine defines SMTP codes, DSN codes, and error notification text.

The smfi_setmlreply() routine defines SMTP codes, DSN codes, and multiline error notification text (sendmail V8.13 and later).

The smfi_setdbg() routine sets the level of debugging printing or logging desired (sendmail V8.13 and later).

The smfi_setbacklog() routine sets the size of the listen(2)'s queue (sendmail V8.13 and later).

As you will see in Chapter 8, every handler function is given at least one argument. A ctx pointer that contains the private information for each connection. For example:

xxfi_eom(SMFICTX *ctx)

Remember that a Milter is multithreaded. Any given handler function can belong to any thread or to multiple threads. The only way any given handler function knows which connection it belongs to is by use of this ctx pointer.

Each of these smfi routines also receives as its first argument the same ctx pointer that its enclosing handler function received.[9] For example:

[9] Note that this pointer is a context data pointer and that there is one per connection.

context = smfi_getpriv(ctx);

7.3.1. The smfi_getsymval() Routine Fetches Macro Values

The smfi_getsymval() routine is used to fetch the value of sendmail macros. It can be called from almost any of the handler functions, but it makes the most sense to call it only from the one that has initial access to the desired macro's value.

The xxfi_connect() function (see section 8.4) generally has access to the ${daemon_name}, ${if_name}, ${if_addr}, $j, and $_ macros.

The xxfi_helo() function (see section 8.5) generally has access to the ${tls_version}, ${cipher}, ${cipher_bits}, ${cert_subject}, and ${cert_issuer} macros.

The xxfi_envfrom() function (see section 8.6) generally has access to the $i, ${auth_type}, ${auth_authen}, ${auth_ssf}, ${auth_author}, ${mail_mailer}, ${mail_host}, and ${mail_addr} macros.

The xxfi_envrcpt() function (see section 8.7) generally has access to the ${rcpt_mailer}, ${rcpt_host}, and ${rcpt_addr} macros.

Note that we say, "generally has access to" because macros, after transferred from sendmail to the Milter, tend to persist throughout a given envelope's context. That is, macros sent with xxfi_connect() tend to persist from the time the connection is initially made until the connection is closed, whereas macros sent with xxfi_envrcpt() persist only for that single recipient. Thus, it is advisable for you to fetch values only for macros that are initially available to the appropriate handler function.

You call smfi_getsymval() this way:

char *value;

value = smfi_getsymval(ctx, macroname);

The ctx is the context pointer that was originally passed to the enclosing handler function. If it is NULL, then smfi_getsymval() returns NULL.

The macroname is a string that specifies the name of the macro whose value is to be looked up. The leading $ must be omitted from the name, but the enclosing curly braces must be included. If the macro name is a single character, the curly braces are optional:

"${_}"                     Bad, has leading $ character
"{_}"                      Good
"_"                        Also good
"{mail_host} {mail_addr}"  Bad, multiple macro names
""                         Bad, an empty string

If the macroname is empty or NULL, then smfi_getsymval() returns NULL. Otherwise, the list of known macros is searched to find the one specified. If one is not found, then smfi_getsymval() returns NULL. Otherwise, smfi_getsymval() returns the address of the value for that macro.

The address for the value is a pointer to a string. Memory is not allocated, so if you need to modify the value, you will need to copy it and then modify the copy. The string is held in the Milter's context and will be rewritten by the Milter for the next envelope or connection.

In general, an envelope-specific macro is valid only for the current envelope, and a connection-specific macro is valid only for the current connection.

If you need sendmail to pass a macro and value pair that is not normally passed, you can do so by adding one or more of the sendmail mc macros shown in Table 7.5 to your sendmail mc file and then creating and using the new configuration file.

Table 7.5. mc Macros for Passing sendmail Macros to a Milter

mcMacro

Description

confMILTER_MACROS_CONNECT

Defines the set of macros passed to the xxfi_connect() handler function.

confMILTER_MACROS_HELO

Defines the set of macros passed to the xxfi_helo() handler function.

confMILTER_MACROS_ENVFROM

Defines the set of macros passed to the xxfi_envfrom() handler function.

confMILTER_MACROS_ENVRCPT

Defines the set of macros passed to the xxfi_envrcpt() handler function.

confMILTER_MACROS_EOM

Defines the set of macros passed to the xxfi_eom() handler function (sendmail V8.13 and later)

Note that you should avoid adding a sendmail macro to one of these mc macros if the added sendmail macro is not given a value until later.[10] If you do, the sendmail macro that is sent will always have an empty value, even though that macro eventually gets a value as rule sets are processed. Timing is everything.

[10] Clearly, it makes no sense to pass a recipient-defined macro during the Connection phase.

Finally, note that sendmail macros seldom hold all the information you need in order to make an informed decision. Consider examining the message headers and the message body as well.

7.3.2. The smfi_setpriv() Routine Registers Private Data

Your various handler functions will likely need to possess and share data among them. Recall that your handler functions are part of a multithreaded environment, meaning that they have no prior knowledge about which connection is being handled at any given time. That information is passed as part of the ctx connection information pointer to each handler function when it is called. One piece of this ctx pointer is a bit of private data space.

Naturally, you are not allowed to access that private data space directly. Instead, you are expected to declare it (allocate it) separately and then to tie it into your ctx pointer using this smfi_setpriv() milter-library routine.

To set up and tie in your private data, follow these steps:

1.

Declare and allocate your private data in the first handler function called or the first one in which you need private data.

2.

Set up your private data for use, and write any support routines you may need to manage it.

3.

Register it with the milter-library by calling the smfi_setpriv() milter-library routine.

To illustrate, we show a simple example of a way to count the number of recipients (as usual, the numbers on the left are not a part of the code):

 1  /*
 2  ** Process the RCPT TO: recipient address.
 3  ** This may be called multiple times, once
 4  ** for each RCPT TO: command.
 5  */
 6  sfsistat
 7  xxfi_envrcpt(ctx, argv)
 8          SMFICTX *ctx;
 9          char **argv;
10  {
11         int *datap;
12
13    datap = (int *)smfi_getpriv(ctx);
14    if (datap == NULL)
15    {
16       datap = (int *)malloc(sizeof(int));
17       if (datap == NULL)
18          return SMFIS_TEMPFAIL;
19       *datap = 1;
20       if (smfi_setpriv(ctx, datap) == MI_FAILURE)
21          return SMFIS_TEMPFAIL;
22       return SMFIS_CONTINUE;
23    }
24    ++(*datap);
25    return SMFIS_CONTINUE
26  }

Inside your Milter code, the xxfi_envrcpt() function (see section 8.7) is called to process each envelope recipient specified by the sending sites' SMTP RCPT TO: commands for the current message. Here, we show a simple xxfi_envrcpt() function (line 7), which counts the number of recipients.

First, the private data is fetched (line 13) with the smfi_getpriv() routine (see section 7.3.3). If this routine returns NULL (line 14), it means that this is the first time through this function during the current connection and message and that the private data has not already been fetched. Therefore, we need to use this smfi_setpriv() routine to set up the private data for keeping count of recipients.

Inside the if clause (line 15), we allocate memory to the private data pointer (line 16). The size of the allocated memory is the sizeof(3) integer so that we can use that allocated memory to count recipients. We preset the count (the value inside the allocated memory) to 1 (line 19) and then use this smfi_setpriv() routine (line 20) to save the location of the private data.

The first argument to smfi_setpriv() is the ctx pointer that was passed to the enclosing xxfi_envrcpt() function. The second argument that smfi_setpriv() expects is a void * pointer, meaning that there is no need to cast datap during the call.

For the second or any subsequent recipients, we simply increment the existing counter (line 24).

The smfi_setpriv() routine can be used to store much more complex data than just a simple integer. For example, you could instead allocate space for a complex structure and then save its address using this smfi_setpriv() routine.

The smfi_setpriv() routine stores its data pointer in a ctx pointer that is unique for each connection. Because multiple messages (envelopes) may be processed during a single connection, envelope-oriented private data should be freed at the end of each envelope, and smfi_setpriv() must be called to store NULL.

The xxfi_envfrom() Milter function (see section 8.6) is called at the beginning of each message (envelope), the xxfi_eom() Milter function (see section 8.11) is called at the end of each message, and the xxfi_abort() Milter function (see section 8.12) is called in the event that something outside the control of this Milter rejected the message.

It is important that any memory you have allocated as part of setting up your private data be deallocated (freed) at the end of the current connection. The xxfi_close() Milter function (see section 8.13) is called at the end of the current connection (the end of the current ctx pointer's life). But your Milter will continue to run after the current connection has ended, and failure to deallocate temporary resources will cause your Milter to gradually waste more memory and grow without bounds.

7.3.3. The smfi_getpriv() Routine Fetches Private Data

As you have seen (in the section 7.3.2 example), the smfi_getpriv() milter-library routine is used to fetch the current private data pointer's value:

datap = (int *)smfi_getpriv(ctx);

The smfi_getpriv() routine takes a single argument, which is the ctx pointer passed to the enclosing handler function. The smfi_getpriv() milter-library routine returns the type void *, so its return value needs to be cast to the type you expect. Note that if smfi_setpriv() has not yet been called to set up the private data, smfi_getpriv() will return NULL.

To illustrate, consider the following extract from an xxfi_connect() handler function, which creates and initializes a complex set of private data.

 1  struct Pdata {
 2     char *sender;
 3     char *send_host;
 4     int   num_recipients;
 5     int   flags;
 6     char **headers;
 7  };
 8
 9  sfsistat
10  xxfi_connect(cts, hostname, hostaddr)
11    SMFICTX *ctx;
12    char *hostname;
13    _SOCK_ADDR *hostaddr;
14  {
15    struct Pdata *pdatap;
16
17    pdatap =(struct Pdata *)malloc(sizeof (struct Pdata));
18    if (pdatap == NULL)
19       return SMFIS_TEMPFAIL;
20    memset(pdatap, '\0', sizeof(struct Pdata));
21    (void) smfi_setpriv(ctx, pdatap);
22

First, outside all functions, we declare a structure type named Pdata that can hold a wide variety of values, including a list of all headers (line 6).

The xxfi_connect() Milter function (see section 8.4) is declared next (line 10). We don't explain the arguments passed to xxfi_connect() here, but instead focus on the use of smfi_setpriv().

Remember that the smfi_setpriv() routine can be used only to store a single pointer. That is, it cannot be used to store the private data itself, only a pointer to that data. To create an appropriate pointer, we declare one (line 15) and then use it to store the address of malloc'd space. The space we malloc is the size of the structure Pdata (line 17).

We zero (line 20) that allocated Pdata structure (the structure that pdatap points to) and conclude by calling smfi_setpriv() (line 21) to store pdatap (the pointer to the private data).

Be careful to avoid the mistake of storing a private data pointer when one has already been stored. The smfi_setpriv() milter-library routine does no checking and will silently overwrite the old value. When in doubt, it is wise to call smfi_getpriv() first to see whether a private data pointer has already been set. Note that if it has not been set, then smfi_getpriv() will return NULL.

It is an error, and your program will fail to compile, if you attempt to store the actual data by using smfi_setpriv() instead of a pointer.

The smfi_setpriv() milter-library routine will return MI_FAILURE if the ctx pointer is not a valid one. Otherwise, it returns MI_SUCCESS. Therefore, the complete call to smfi_setpriv() might look like this:

if (smfi_setpriv(ctx, pdatap) == MI_FAILURE)
  return SMFIS_TEMPFAIL;

Here, we return a temporary failure. Because smfi_setpriv() will fail only if ctx is bad, we might want to exit the Milter here and log a software error.

7.3.4. The smfi_setreply() Routine Changes the SMTP Reply

Normally, when a Milter rejects an SMTP command, it causes the sending site to receive a 5yz reply code. When a Milter tempfails an SMTP command, it causes the sending site to receive a 4yz reply code. The exact default codes vary depending on the SMTP command, so we show the defaults for each SMTP phase and command in Table 7.6.

Table 7.6. Default SMTP Replies Caused by Milter Rejects and Tempfails

SMTP Command

Rejection

Default Reply

Initial connection

Reject

554 your.domain ESMTP not accepting messages

Then offer only ENHANCEDSTATUSCODES in reply to EHLO,and reject all other SMTP commands except QUIT and NOOP.

Tempfail

220, then 250 in reply to HELO/EHLO, then in reply to the MAIL FROM: command tempfail with

451 4.7.1 Please try again later

But with SASL and TLS, it will reply with a 454 prefix.

HELO/EHLO

Reject

250 and offer only ENHANCEDSTATUSCODES in reply to the EHLO command, then reply to all other SMTP commands with

550 5.0.0 Command rejected

And accept only QUIT and NOOP.

Tempfail

250, then, in reply to the MAIL FROM: command, issue

451 4.7.1 Please try again later

But with SASL and TLS, it will reply with a 454 prefix.

MAIL FROM:

Reject

550 5.7.1 <user@domain>... Command rejected

Tempfail

451 4.7.1 Please try again later

RCPT TO:

Reject

550 5.7.1 <user@domain>... Command rejected

Tempfail

451 4.7.1 Please try again later

DATA

Reject

In reply to final dot:

554 5.7.1 Command rejected

Tempfail

451 4.7.1 Please try again later

If these defaults are suitable for you, you may want to leave them unchanged. Sometimes, however, you may wish to provide a different or more informative reply than the default. For such situations, the smfi_setreply() milter-library routine can prove useful. It allows you to

Change the SMTP reply code (say, from 550 to 554)

Change the DSN reply code (say, from 5.7.1 to 5.7.2)

Change the message (say, from "Command Rejected" to "Retired User")

Disconnect a connection by rejecting with tempfail (sendmail V8.13)

You use smfi_setreply() this way:

ret = smfi_setreply(ctx, smtp, dsn, msg);

The smfi_setreply() milter-library routine returns MI_SUCCESS on success, and otherwise MI_FAILURE. Because it can fail for syntax errors as well as memory-allocation errors, there is no guarantee that any value in errno will be useful. Therefore, a complete call to smfi_setreply(), including error checking, might look like this:

errno = 0;
if (smfi_setreply(ctx, smtp, dsn, msg) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_setreply: %s",
     errno == 0 ? "syntax error" : strerror(errno));
}

Here, we use strerror() to print the error if errno is nonzero (a memory-allocation error); otherwise, we print "syntax error."

There are four arguments to smfi_setreply(). The first is ctx, which you will recall is the context pointer for the current connection that was passed to the enclosing handler function.

The second argument is smtp, which is a three-digit SMTP reply code in the form of a string. For example:

"421"

If the argument consists of anything other than three digits, if it begins with a digit other than a 4 or 5, or if it is NULL, then the smfi_setreply() routine will return MI_FAILURE, indicating a syntax error. This argument must be present, syntactically correct, and non-NULL.

The third argument is dsn, which must be either NULL or three integers with two dots separating them in the form of a string. For example:

"5.7.0"

If the first integer is not a 4 or 5, if the three integers are not composed of all digits, or if the character positions that should be occupied by dots are not occupied by dots, then the smfi_setreply() routine will return MI_FAILURE, indicating a syntax error. If this argument is NULL it is ignored, and a default DSN return value will be generated by sendmail.

Note that each of the integers in this third argument may contain multiple digits (see RFC1893). For example, "5.17.212" is also legal.

The fourth argument is msg, which can be either a string or NULL. For example:

"We hate spammers, go away"

If the string is longer than 980 characters, or if it contains a carriage-return (\r) or linefeed character (\n), then the smfi_setreply() routine will return MI_FAILURE, indicating a syntax error. If this argument is NULL it is ignored, and no message will be issued as part of the reply.

All the components of your customized SMTP reply are gathered into a single string by smfi_setreply(), which allocates memory to store the new string. If a prior reply was already allocated, then smfi_setreply() frees the old reply before allocating and storing the new reply. If smfi_setreply() is unable to allocate memory, it returns MI_FAILURE and sets errno to an appropriate value.

When rejecting a connection, envelope, or recipient, you must take care to set the SMTP code correctly for the type of rejection. If you reject and specify a 4yz SMTP code with smfi_setreply(), your customized reply will be ignored because of the mismatch (you should have used a 5yz code). Similarly, if you tempfail and specify a 5yz SMTP code with smfi_setreply(), your customized reply will also be ignored because of the mismatch (you should have used a 4yz code). Your customized reply is silently ignored on mismatch, and nothing is logged.

As a special case (beginning with sendmail V8.13), if you set the SMTP reply code to 421 and tempfail a connection, the connection is immediately terminated with a 421 error code. Contrast this to the default rejection, which accepts the HELO or EHLO and rejects all subsequent SMTP commands except QUIT and NOOP. Note that with this new 421 rejection the dsn and msg arguments to smfi_setreply() are always ignored, even if non-NULL.

7.3.5. The smfi_setmlreply() Routine Sets Multiline Replies (V8.13 and Later)

The smfi_setmlreply() milter-library routine is used in exactly the same manner as the smfi_setreply() milter-library routine described in section 7.3.4. The only difference is that this smfi_setmlreply() routine takes multiple last arguments instead of only one:

ret = smfi_setreply(ctx, smtp, dsn, msg);
ret = smfi_setmlreply(ctx, smtp, dsn, msg1, msg2, ..., NULL);

Here, each additional argument to smfi_setmlreply() is just like the msg argument to smfi_setreply(). That is, each is a string or NULL. For example:

"We hate spammers, go away"

If any string is longer than 980 characters, if it contains a carriage-return (\r) or linefeed (\n) character, or if there are more than 32 string arguments given, then the smfi_setmlreply() routine will return MI_FAILURE, indicating a syntax error. The last argument, the NULL, is used to terminate the list of messages. Any additional messages following the NULL will be ignored.

To illustrate one use of this smfi_setmlreply() routine, we first consider the following use of smfi_setreply():

smfi_setreply(ctx, "554", "5.5.0", "Don't spam us!");

This, when set in one of your handler functions that screens the message body, might bounce the message like this:

554 5.5.0 Don't spam us!

There are times, however, when you may want to express far more information as part of a bounce. This smfi_setmlreply() milter-library routine provides one way to do just that. For example:

smfi_setmlreply(ctx, "554", "5.5.0",
  "This message body contained \"src=cid:\"",
  "Remove inline images and resend, or",
  "email bounce-help@our.domain for help,",
  "or telephone 555-123-1234.", NULL);

Here, we use smfi_setmlreply() to send four lines of bounce information instead of only a single line. The sending site might see a bounce that looks like this:

554-5.5.0 This message body contained "src=cid:"
554-5.5.0 Remove inline images and resend, or
554-5.5.0 email bounce-help@our.domain for help,
554 5.5.0 or telephone 555-123-1234.

But be aware that some MTA software will save only the first or last line of such multiline reports for presentation in bounce messages. We recommend that you do not use multiline replies unless you have an overwhelming reason to do so. As an alternative, you can embed a web reference into your single-line reply that can be used to provide the verbose information needed. That is, use smfi_setreply() instead of smfi_setmlreply():

smfi_setreply(ctx, "554", "5.5.0",
  "Don't spam. See http://www.our.site/nospam/");

Here, we use smfi_setreply() to bounce with a single-line error that contains a web reference, which, when visited, informs others about your policies.

7.3.6. The smfi_setdbg() Routine Sets the Milter Debugging Level (V8.13 and Later)

Normally, the milter-library is silent about its errors. Instead of printing or logging them, it prefers to return error codes to your handler functions for them to handle. There may be times, however perhaps as part of your own Milter development when you will want these errors reported directly by the library routines.

The smfi_setdbg() milter-library routine sets the milter-library's internal debugging level to a new level so that code details can be traced. The smfi_setdbg() routine is used this way:

ret = smfi_setdbg(level);

The smfi_setdbg() milter-library routine always returns (to ret, as shown here) the return value MI_SUCCESS, which can be safely ignored.

(void) smfi_setdbg(level);

The debugging level is changed from its current value to the new value that is defined by level. The default level when your Milter first starts to run is zero (no debugging). Any level above zero turns on debugging, where higher values cause more debugging information to be printed or logged.

If you wish to detect the debugging level currently in use to employ, for example, in your own handler functions you can do so by checking the value of ctx->ctx_dbg

Be aware, however, that this technique uses a milter-library internal that is not guaranteed to exist for all releases of the milter-library.

Table 7.7 shows the currently defined levels and what is logged for each. This table is current as of sendmail V8.13.

Table 7.7. smfi_setdbg() Debugging Levels

Level

At and Above Which the Following Is Logged

1

The Milter is about to open the listening socket.

2

The Milter engine received an unknown command from sendmail, logged or printed as

[id] cmd 'letter' unknown

Where id is the thread-context identifier and letter is the unknown command.

2

The Milter engine received a known but not yet implemented command from sendmail, logged or printed as

[id] cmd 'letter' not impl

Where id is the thread-context identifier and letter is the unknown command.

2

The Milter engine is state driven and has just detected an attempt to transition to a new state in the wrong order (as from the HELO state to a RCPT state with no intervening MAIL state), logged or printed as

[id] abort: cur now new bad next mask

Where id is the thread-context identifier, now and bad are the prior state and the next but bad state (printed in decimal and parenthetically in hexadecimal), and mask is a bitmap of the state table in hexadecimal. This error causes the processing of the current envelope to abort (other envelopes that may arrive as part of the current connection may still be processed).

4

The Milter was semaphored to stop by a signal or by a handler function, printed or logged as

[id] milter abort

Where id is the thread-context identifier.

5

Show the command just received by the Milter engine from sendmail, logged or printed as

[id] got cmd 'letter' len len

Where id is the thread-context identifier, letter is the command, and len is the length of the command buffer.

6

The Milter engine detected an error while reading a command from sendmail, shown as

[id] mi_engine: mi_rd_cmd error (err)

Where id is the thread-context identifier, and err is the error code displayed in hexadecimal.

6

The Milter engine is state driven and has just switched from the old state to the new state, logged or printed as

[id] cur old new next nextmask mask

Where id is the thread-context identifier, old and next are the prior and next states printed in hexadecimal, and mask is the set of remaining states as a bitmap in hexadecimal.

6

An internal function returned abort as its status, logged or printed as

[id] function returned abort

Where id is the thread-context identifier.

Note that when you set a debugging level with smfi_setdbg() before calling smfi_main() (see section 7.2.5), it affects the debugging level for all subsequent handler functions. If, however, you set a debugging level from inside your handler functions, it will affect only threads launched after that point in time and will not affect the current thread or any that are currently running.

7.3.7. The smfi_setbacklog() Routine Sets the listen(2) Queue Size (V8.13 and Later)

When a Milter listens for incoming connections from sendmail, it calls listen(3) with two arguments:

listen(socket, backlog)

The second argument, backlog, defines the maximum number of pending connections in the listen(3) queue. It is generally a small integer, such as 20.[11]

[11] On Solaris the system default is 5, whereas on FreeBSD and Linux the system defaults are both 128.

This smfi_setbacklog() milter-library routine allows you to reset that second argument to a different value. It is used this way:

smfi_setbacklog(newbacklog);

This routine should be called prior to smfi_main(). It replaces the old backlog with the new and will return MI_SUCCESS unless newbacklog is less than or equal to zero. In that case it returns MI_FAILURE.

If you don't set a new backlog value with smfi_setbacklog() and if the system does not set a default that is greater than 20, then a default of 20 is used. Otherwise, the system default is used. In general, the default provided by your system or the milter-library will be sufficient for most needs. But if yours is a very busy system, with many machines using a central Milter on a filtering server, that filtering server could reasonably double or even triple the size of backlog.

Finally, note that some systems provide a listen(3) routine that does not allow you to reset its backlog from the system default. Others, such as Linux after version 2.2, have reinterpreted this parameter.

7.4. The smfi Modifier Routines

A Milter can do more than just review SMTP commands, headers, and the message body. It can also rewrite them using the xxfi_eom() handler function (see section 8.11). The ways a message can be modified include the following:

Add a new header or replace an existing header (see section 7.4.2).

Insert a new header into the list of existing headers (see section 7.4.3).

Change the value of an existing header or delete the header (see section 7.4.4).

Add a recipient to those already present (see section 7.4.5).

Remove a recipient from the list of existing recipients (see section 7.4.6).

Replace the message body with a new message body (see section 7.4.7).

Notify sendmail that you need more time (V8.13 and later) (see section 7.4.8).

Cause the message to be quarantined (V8.13 and later) (see section 7.4.9).

Cause the Milter to shut down in an orderly manner (see section 7.4.10).

Recall that a Milter can perform these additional actions only if it set the appropriate flags in smfiDesc. Review section 7.2.2 to see the meaning of the various flags and how to set them. Note that smfi_progress() (see section 7.4.8) requires no flags for its use. We will show the appropriate flags in the sections to follow.

7.4.1. Logging

Actions taken by a Milter can be logged by that Milter or by sendmail. We cover logging by the Milter in later chapters. Here we discuss logging by sendmail, which results from using one of the smfi milter-library routines.

Inside the sendmail configuration file is the Milter.LogLevel option, which determines what will be logged by sendmail as a result of its interactions with Milters. By default, that option is set to the same value as the LogLevel option (which itself defaults to 9). If you wish the Milter.LogLevel option to use a different value, you can do so with a declaration such as the following in your sendmail mc configuration file:

define(`confMILTER_LOG_LEVEL', `level')

Here, level is an integer value that sets the Milter.LogLevel option's value. The effects of various level settings are listed in Table 7.8.[12] Note that higher levels include information logged by lower levels.

[12] See the sendmail source for details of the actual log messages.

Table 7.8. The Effect of Various Milter.LogLevel Settings

Milter.LogLevel

Effect

0

No logging.

1

Only serious errors are logged.

9

Header and recipient adds, changes, removals.

11

Connections, aborts, rejections, tempfails, discards, etc.

13

Continues and connection errors.

18

Each piece of data sent to a Milter (headers and body chunks).

22

The time it took for a Milter to complete a command.

Note that log messages caused by the Milter.LogLevel setting are logged along with all the usual sendmail log messages and that these messages are logged by sendmail. This is different from logs produced by the Milter itself and should not be confused with them. In the rest of this chapter we discuss only Milter.LogLevel logging.

7.4.2. The smfi_addheader() Routine Adds a Header to the Message

Sometimes a Milter may need or desire to add a header to the existing headers in a message. The smfi_addheader() milter-library routine (which must be called only from the xxfi_eom() handler function (see section 8.11)) does just that.

But before you can add headers to a message, you must declare your intent to do so by adding the SMFIF_ADDHDRS flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name */
   SMFI_VERSION,    /* a constant */
   SMFIF_ADDHDRS,   /* flags */
   ...

If you fail to set this flag, smfi_addheader() will return MI_FAILURE every time you try to use it.

After you have used the SMFIF_ADDHDRS flag to declare your intention to add headers, you can add those headers by calling the smfi_addheader() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_addheader(ctx, name, value);

The smfi_addheader() routine returns (to ret) either MI_SUCCESS (if the header was added) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_addheader() might look like this:

errno = 0;
if (smfi_addheader(ctx, name, value) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_addheader: %s",
     errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_addheader() can fail for syntactic reasons (for example, a bad ctx pointer) or for system-error reasons (for example, a network error). Note that syntax errors do not set errno to a useful value, whereas system errors do.

The first argument to smfi_addheader() is the ctx connection-context pointer. It is the same ctx pointer that was passed to the enclosing xxfi_eom() handler function. It must not be NULL but must be a valid pointer.

The second argument to smfi_addheader() is the name of the header in the form of a string. Recall that a header name is the text to the left of the colon:

Name: value

Header names must conform to RFC standards. The milter-library performs no standards checking to ensure validity. It is your responsibility to ensure that no standards are violated.

If the header name is NULL or if it is an empty string, then smfi_addheader() will return MI_FAILURE.

Note that any capitalization in the name is up to you. Whatever capitalization you choose will be preserved.

The third and last argument to smfi_addheader() is the value of the header. This value can be NULL or an empty string, in which case the header will be inserted with no value.

Recall that a header value is the text to the right of the colon:

Name: value

The string containing the value should be fewer than 998 characters. If the value is too long, sendmail may silently truncate it. If you need to extend the value over multiple lines, you can do so by inserting newline characters, each followed by a space or tab.[13] For example:

[13] Do not use carriage-return/linefeed pairs here. When needed they will be added by sendmail. Using such pairs will yield a syntax error.

"Spam check report\n\tAttachments=0\n\tSpam=NO"

When later viewed by the message recipient, this value will look like this:

Name: Spam check report
  Attachments=0
  Spam=NO

Note that a special logic inside sendmail is used when a header is added by a Milter. The existing header names are scanned to see whether the new name already exists. If the header name exists, if the header is not a trace header (such as Received:), and if the header is not one added by another Milter, then sendmail will silently replace the existing named header's value with the new value, rather than add the new header.[14]

[14] Only trace headers, X- headers, and Milter-added headers can exist in multiple occurrences.

If the sendmail configuration file's Milter.LogLevel option has a value of 8 or less, nothing is logged about added or changed headers. Otherwise, if an existing header had its value changed, the following will be logged:

Milter change: default header existing value with newvalue

Or if a new header was added, the following message will be logged:

Milter add: header: name: value

This means that you cannot add, for example, a Date: header if one already exists. You can, however, add a Received: header if one already exists, and you can always add a header that is prefixed with a literal X-, such as an X-milter: header.

Finally, recall that multiple Milters can review any message. When your Milter changes a header's value, that change will not be seen by prior Milters but will be seen by subsequent Milters. An exception to this rule occurs if, later in your xfi_eom() handler function, you reject or discard that message. In that case, subsequent Milters will not see it.

Note that the current Milter will not get the opportunity to add a header if a prior Milter has rejected the message. Therefore, you should never use a custom-added header with the expectation that it can convey information to subsequent Milters, because there is no guarantee that a subsequent Milter will see it.

Also note that the order in which Milters are called is defined by your sendmail configuration file. Changes in that configuration file can change the order of Milters, and there is no way for any given Milter to know that its position in the sequence of Milters has been changed. Even if you set the order in the configuration file, you cannot assume that it will remain in the correct order forever, nor can you build that assumption into your Milter code. Someone may change it later without telling you.

7.4.3. The smfi_insheader() Routine Inserts a Header into the Message

Sometimes a Milter needs to insert a header into the list of existing headers in a message, even if that new header already exists in the message. The smfi_insheader() milter-library routine (new with V8.13) does just that.

Note that it must be called only from the xxfi_eom() handler function (see section 8.11) and that before you can insert new headers into a message, you must declare your intention to do so by adding the SMFIF_ADDHDRS flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name */
   SMFI_VERSION,    /* a constant */
   SMFIF_ADDHDRS,   /* flags */
   ...

If you fail to set this flag, then smfi_insheader() will return MI_FAILURE every time you try to use it.

After you have declared the SMFIF_ADDHDRS flag, you can insert those headers by calling the smfi_insheader() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_insheader(ctx, index, name, value);

The smfi_insheader() routine returns (to ret) either MI_SUCCESS (if the header was inserted) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_insheader() might look like this:

errno = 0;
if (smfi_insheader(ctx, index, name, value) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_addheader: %s",
     errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_insheader() can fail for syntactic reasons (for example, a bad ctx pointer) or for system-error reasons (for example, a network error). Note that syntax errors do not set errno to a useful value, whereas system errors do.

The first argument to smfi_insheader() is the ctx connection-context pointer. It is the same ctx pointer that was passed to the xxfi_eom() handler function that encloses it. It must not be NULL and must be a valid pointer.

The second argument to smfi_insheader() is an index into the list of existing headers. This list is in the same order that the headers will appear in the message upon local delivery. This is also the order in which they are transmitted over an SMTP session as part of the DATA phase. The index is a count of headers. The new header will be added after the header indicated by the index number:

1  From: bob
2  To: fred
3  Subject: Test

Here, for example, if index is 2, the new header will be inserted after the To: header. If index is zero, the new header will be inserted ahead of all existing headers, that is, in front of the From: header in this example. If index is greater than the number of existing headers, the new header will be inserted following the last header. If index is less than zero, smfi_insheader() will return MI_FAILURE.

The third argument to smfi_insheader() is the name of the header in the form of a string. Recall that a header name is the text to the left of the colon:

Name: value

Header names must conform to RFC standards. The milter-library performs no standards checking to ensure validity. It is your responsibility to ensure that no standards are violated.

If the header name is NULL or if it is an empty string, then smfi_insheader() will return MI_FAILURE.

Note that any capitalization in the name is up to you. Whatever capitalization you choose will be preserved. This is important when you add headers in which special meaning is applied to capitalization.

The fourth and last argument to smfi_insheader() is the value of the header. If the value is NULL, smfi_insheader() will return MI_FAILURE. If the value is an empty string, smfi_insheader() will insert a header with no value.

Recall that a header value is the text to the right of the colon:

Name: value

The string containing the value must be fewer than 2048 characters long and should be less than 998 characters. If the value is too long, sendmail may silently truncate it. If you need to extend the value over multiple lines, you can do so by inserting newline characters, each followed by a space or tab.[15] For example:

[15] Do not use carriage-return/linefeed pairs here. When needed they will be added by sendmail. Using such pairs will yield a syntax error.

"Spam check report\n\tAttachments=0\n\tSpam=NO"

When later viewed by the message recipient, this value will look like this:

Name: Spam check report
   Attachments=0
   Spam=NO

If the sendmail configuration file's Milter.LogLevel option has a value of 8 or less, nothing is logged regarding added or changed headers. Otherwise, if a new header was added, the following message will be logged:

Milter add: header: name: value

Finally, recall that multiple Milters can review any message. When your Milter changes a header's value, that change will not be seen by prior Milters but will be seen by subsequent Milters. An exception to this rule occurs if, later in your xxfi_eom() handler function, you accept, reject, or discard that message. In that case, subsequent Milters will not see it.

Note that the current Milter may not get the opportunity to add a header if a prior Milter has rejected the message. Therefore, you should never use a custom-inserted header to convey information to subsequent Milters, because there is no guarantee that a subsequent Milter will see it.

Also note that the order in which Milters are called is defined by your sendmail configuration file. Changes in that configuration file can change the order of Milters, and there is no way for any given Milter to know that its position in the sequence of Milters has been changed.

7.4.4. The smfi_chgheader() Routine Changes or Removes a Header

Sometimes a Milter may need to change the value of an existing header in a message. The smfi_chgheader() milter-library routine (which must be called only from the xxfi_eom() handler function (see section 8.11)) replaces an existing header's value with a new value.

But before you can modify header values, you must first declare your intent to do so by adding the SMFIF_CHGHDRS flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
  "MyMilter",      /* filter name */
  SMFI_VERSION,    /* a constant */
  SMFIF_CHGHDRS,   /* flags */
  ...

If you fail to declare the SMFIF_CHGHDRS flag, every call to smfi_chgheader() will return MI_FAILURE.

After you have declared your intention to modify header values (by adding the SMFIF_CHGHDRS flag to smfiDesc), you can modify or delete headers by calling the smfi_chgheader() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_chgheader(ctx, name, index, value);

The smfi_chgheader() routine returns (to ret) either MI_SUCCESS (if the header whose name is name has its value successfully changed to value) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_chgheader() might look like this:

errno = 0;
if (smfi_chgheader(ctx, name, index, value) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_chgheader: %s",
     errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_chgheader() can fail for syntactic reasons (for example, a bad ctx pointer) or for system-error reasons (for example, a network error). Syntax errors do not set errno to a useful value, whereas system errors do.

The first argument to smfi_chgheader() is the ctx connection-context pointer. It is the same ctx pointer that was passed to the enclosing xxfi_eom() handler function. It must not be NULL but must be a valid pointer.

The second argument to smfi_chgheader() is the name of a header that exists in the message, specified in the form of a string. Recall that a header name is the text to the left of the colon:

Name: value

If the header name does not already exist, a new header with that name will be created. If the value is empty, the header with that name will be deleted. If the header name is NULL or if it is an empty string, then smfi_chgheader() will return MI_FAILURE.

The third argument, index, is an offset and must be greater than zero. Normally, index should be set to 1, meaning that you are changing the value of the first occurrence of a header with the name Name. But some header names can appear multiple times (the Bcc: header is one), and when they do, you can set index to a count that changes the value of a particular occurrence of that header.

But note that if the offset is greater than the number of headers with that name, a new header will be created.

When you change the value of existing headers, it is vital to keep track of the offset of the header you intend to change when they are first reviewed by the xxfi_header() handler function (see section 8.8). In that way, you can avoid accidentally creating new headers when you intended only to modify the value of existing headers.

The fourth and last argument to smfi_chgheader() is the new value to give to an existing header. If value is NULL or is an empty string, the header will be removed.

Recall that a header value is the text to the right of the colon:

Name: value

When you change a header's value, the string containing the new value must be fewer than 2048 characters long and should be fewer than 998 characters. If the value is too long, sendmail may silently truncate it. If you need to extend the value over multiple lines, you can do so by inserting newline characters, each followed by a space or tab.[16] For example:

[16] Do not use carriage-return/linefeed pairs here. When needed they will be added by sendmail. Using such pairs will yield a syntax error.

"Spam check report\n\tAttachments=0\n\tSpam=NO"

When later viewed by the message recipient, this value will look like this:

Name: Spam check report
  Attachments=0
  Spam=NO

If the sendmail configuration file's Milter.LogLevel option has a value of 9 or greater, the following will be logged when an existing header's value was changed:

Milter change: header name type: from old value to new value

Here, type is either the name of a trace-type header (such as Received:) or a user-added header (such as X-milter:), or it is the literal phrase (default header), meaning one defined by RFCs (such as Date:). If there was no old value, that field will print as a literal <NULL>.

If you deleted a header and if the sendmail configuration file's Milter.LogLevel option has a value of 9 or greater, then the following will be logged:

Milter delete: header name type: old value

Here, type is as described earlier. If there was no old value, that field will print as a literal <NULL>.

Recall that multiple Milters can review any message. When your Milter changes a header's value, that change will not be seen by prior Milters but will be seen by subsequent Milters. An exception to this rule occurs if, later in your xxfi_eom() handler function, you accept, reject, or discard that message. In that case, subsequent Milters will not see it.

Note that the current Milter may not get the opportunity to change a header value if a prior Milter has rejected the message. Therefore, you should never modify a header value to convey information to subsequent Milters, because there is no guarantee that a subsequent Milter will see that header.

Also note that the order in which Milters are called is defined by your sendmail configuration file. Changes in the configuration file can alter the order of Milters, so there is no way for any given Milter to know that its position in the sequence of Milters has been changed.

Finally, note that just because the current Milter has removed a header, there is no guarantee that a subsequent Milter will not add it back in.

7.4.5. The smfi_addrcpt() Routine Adds a Recipient

Sometimes a Milter may need to add a recipient to the existing list of envelope recipients. The smfi_addrcpt() milter-library routine (which must be called only from the xxfi_eom() handler function (see section 8.11)) adds a new recipient to the envelope.

But before you can add any recipients, you must first declare your intent to do so by adding the SMFIF_ADDRCPT flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
   "MyMilter",    /*  filter name */
   SMFI_VERSION,  /*  a constant */
   SMFIF_ADDRCPT,  /* flags */
   ...

If you fail to declare the SMFIF_ADDRCPT flag but call smfi_addrcpt() anyway, then that call will return MI_FAILURE.

After you have declared your intention to add recipients (by adding the SMFIF_ADDRCPT flag to smfiDesc), you can add recipients one at a time by calling the smfi_addrcpt() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_addrcpt(ctx, address);

The smfi_addrcpt() routine returns (to ret) either MI_SUCCESS (if the recipient whose email address is address is successfully added to the envelope list of recipients) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_addrcpt() might look like this:

errno = 0;
if (smfi_addrcpt(ctx, address) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_addrcpt(%s): %s",
     address,
     errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_addrcpt() can fail for syntactic reasons (for example, a bad ctx pointer, a NULL recipient address, or an empty recipient address) or for system-error reasons (for example, a network error). Syntax errors do not set errno to a useful value, whereas system errors do.

The first argument to smfi_addrcpt() is the ctx connection-context pointer. It is the same ctx pointer passed to the enclosing xxfi_eom() function. It must not be NULL but must be a valid pointer.

The other argument to smfi_addrcpt() is the address to be added, in the form of a string. Recall that an address is composed of a user part and a host part separated by an @ character:

user@host.domain

Local addresses can omit the @ and the host.domain part:

user

The new address is added by sendmail. If there is a problem with the address, the problem will be handled by sendmail without notification to the Milter. You can optionally enclose the address in angle brackets with no change in effect.

If the sendmail configuration file's Milter.LogLevel option has a value of 9 or greater, the following will be logged when the new address is submitted by the Milter to sendmail but before sendmail actually adds the new address internally:

Milter add: rcpt: address

Recall that multiple Milters can review any message and that all recipients are given to each Milter before any xxfi_eom() handler functions are called for any Milter. Therefore, when your Milter adds a new recipient, that new recipient cannot be detected by subsequent Milters.

Note that the current Milter may not get the opportunity to add a recipient if a prior Milter has rejected the message.

Also note that the order in which Milters are called is defined by your sendmail configuration file. Changes in that configuration file can change that order, so there is no way for any given Milter to know that its position in the sequence of Milters has been changed.

Finally, note that the current Milter can add a recipient, and a subsequent Milter can remove it.

7.4.6. The smfi_delrcpt() Routine Removes a Recipient

Sometimes a Milter may need to remove a recipient from the existing list of envelope recipients. The smfi_delrcpt() milter-library routine (which must be called only from the xxfi_eom() handler function (see section 8.11)) deletes an existing recipient from the envelope.

But before you can remove any recipients, you must first declare your intent to do so by adding the SMFIF_DELRCPT flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
  "MyMilter",     /* filter name */
  SMFI_VERSION,   /* a constant */
  SMFIF_DELRCPT,  /* flags */
  ...

If you fail to declare the SMFIF_DELRCPT flag but call smfi_delrcpt() anyway, then that call will return MI_FAILURE.

After you have declared your intention to remove recipients (by adding the SMFIF_DELRCPT flag to smfiDesc), you can delete recipients one at a time by calling the smfi_delrcpt() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_delrcpt(ctx, address);

The smfi_delrcpt() routine returns (here to ret) either MI_SUCCESS (if the recipient whose email address is address is successfully removed from the envelope list of recipients) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_delrcpt() might look like this:

errno = 0;
if (smfi_delrcpt(ctx, address) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_delrcpt(%s): %s",
     address,
     errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_delrcpt() can fail for syntactic reasons (for example, a bad ctx pointer, a NULL recipient address, or an empty recipient address) or for system-error reasons (for example, a network error). Syntax errors do not set errno to a useful value, whereas system errors do.

Also note that a successful return does not mean that the recipient actually existed in the list of envelope recipients. The smfi_delrcpt() routine, in effect, only suggests to sendmail that it remove a recipient. The request will always succeed (provided there are no system or syntax errors) even if the recipient does not exist in the recipient list.

Be careful when removing recipients because if a message is left with no recipients, it will be discarded even if it is otherwise acceptable.

The first argument to smfi_delrcpt() is the ctx connection-context pointer. It is the same ctx pointer passed to the enclosing xxfi_eom() function. It must not be NULL but must be a valid pointer.

The other argument to smfi_delrcpt() is the address to be deleted, in the form of a string. Recall that an envelope recipient address (as given to the SMTP RCPT TO: command) is normally composed of a user part and a host part separated by an @ character, all surrounded by angle brackets:

<user@host.domain>

If an existing address exactly matches the address specified, it is removed by sendmail from the envelope recipient list.[17] Note that the address must exactly match an existing address in the envelope recipient list. One way to ensure that it does is to save the address that was given to the handler xxfi_envrcpt() function (see section 8.7) and then to give back exactly the same address to this smfi_delrcpt() milter-library routine.

[17] Note that old MTAs can allow envelope sender addresses to go out without enclosing angle brackets, or with nested brackets, such as <name <address>>.

But note that, although rule sets and aliasing can modify a recipient list between the time the xxfi_envrcpt() handler function first sees that list and the time you later wish to delete an address from that list, sendmail ensures that your xxfi_eom() handler function will have access to the original addresses that were detected during your Milter's xxfi_envrcpt() processing. Consequently, your Milter can safely ignore (otherwise undetectable) rule set and aliasing changes.

If the sendmail configuration file's Milter.LogLevel option has a value of 9 or greater, the following will be logged when the address is submitted by the Milter to sendmail but before sendmail actually removes the address internally:

Milter delete: rcpt: address

Recall that multiple Milters can review any message and that all recipients are given to each Milter before any xxfi_eom() handler functions are called for any Milter. Therefore, when your Milter adds a new recipient, that new recipient cannot be detected by subsequent Milters.

Note that the current Milter may not get the opportunity to add a recipient if a prior Milter has rejected the message.

Also note that the order that Milters are called is defined by your sendmail configuration file. Changes in that configuration file can alter the order of Milters, so there is no way for any given Milter to know that its position in the sequence of Milters has been changed.

Finally, note that just because the current Milter has removed a recipient, there is no guarantee that a subsequent Milter will not add it back in.

7.4.7. The smfi_replacebody() Routine Replaces the Message Body

The SMTP DATA portion of an envelope contains two parts: Headers are first, then an empty line, and then the body part. A Milter receives a copy of the body in its xxfi_body() handler function (see section 8.10). If it intends to modify or replace the body, it must first either modify the copy it made or compose a new body.

But before you can replace the body, you must first declare your intent to do so by adding the SMFIF_CHGBODY flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
  "MyMilter",    /* filter name */
  SMFI_VERSION,  /* a constant */
  SMFIF_CHGBODY, /* flags */
  ...

If you fail to declare the SMFIF_CHGBODY flag but call smfi_replacebody() anyway, that call will return MI_FAILURE.

The SMFIF_CHGBODY flag must be declared with great care. Message bodies can be very large, and each Milter that processes the message body can consume a great deal of time. Remember that, while the Milter performs its work, the sending site has already sent its dot and is now waiting for acceptance. If a Milter takes too much time or if too many Milters need to review the body, the wait by the sending site may time out. When you run multiple Milters, we recommend that you allow only one to review and possibly rewrite the body.

After you have declared your intention to replace the body (by adding the SMFIF_CHGBODY flag to smfiDesc), you can perform that replacement by calling the smfi_replacebody() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_replacebody(ctx, buffer, length);

The smfi_replacebody() routine returns (here to ret) either MI_SUCCESS (if the body is successfully replaced) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_replacebody() might look like this:

errno = 0;
if (smfi_replacebody(ctx, buffer, length) == MI_FAILURE)
{
  syslog(LOG_INFO, "smfi_replacebody(): %s",
    errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_replacebody() can fail for syntactic reasons (for example, a bad ctx pointer or a NULL buffer address) or for system-error reasons (for example, a network error). Syntax errors do not set errno to a useful value, whereas system errors do.

Also note that smfi_replacebody() may be called multiple times, each call appending a next body part to the replacement body created by earlier calls. This capability can be useful if you are replacing a huge body (perhaps buffered to disk) and cannot read the entire body into memory at once. As a side effect, also note that the first time you call smfi_replacebody() for an envelope, the body is truncated to zero length and the new body chunk replaces the old. Subsequent calls to smfi_replacebody() for that envelope append to the new body without first truncating the old.

The first argument to smfi_replacebody() is the ctx connection-context pointer. It must be the same ctx pointer passed to the enclosing xxfi_eom() function. It must be a valid pointer.

The second argument to smfi_replacebody() is the address of the buffer containing the replacement body, which is a buffer of type unsigned char. For example:

unsigned char *buffer;

The third argument to smfi_replacebody() is the length of the buffer and is of type int. If the length is less than zero, smfi_replacebody() returns MI_FAILURE. If the buffer is NULL and if the length is greater than zero, then smfi_replacebody() returns MI_FAILURE. Otherwise, smfi_replacebody() sends the replacement body to sendmail and returns MI_SUCCESS if the replacement worked, or MI_FAILURE if it did not.

This smfi_replacebody() routine causes sendmail to truncate the df queue file and then to rewrite it. Therefore, if buffer is NULL and if length is zero, it simply truncates the df file to a zero size.

Note that, because this routine writes directly to the df file, you must be certain that every line in the new body buffer ends in a carriage-return/newline pair. It is not an error to overlook this rule, but you are highly encouraged to obey it.

If the sendmail configuration file's Milter.LogLevel option has a value of 9 or greater, the following will be logged when the body is replaced:

Milter message: body replaced

Recall that multiple Milters can review any message. When your Milter replaces a body, that replacement cannot be detected by prior Milters but can be detected by subsequent Milters (which perhaps expect the body to be base64-decoded).

Note that the current Milter may not get the opportunity to replace the body if a prior Milter has rejected the envelope. Therefore, you should never modify a message body to convey information to subsequent Milters because there is no guarantee that a subsequent Milter will be able to notice the change.

Also note that the order in which Milters are called is defined by your sendmail configuration file. The configuration file can change the order of Milters, and there is no way for any given Milter to know that its position in the sequence of Milters has changed.

7.4.8. The smfi_progress() Routine Buys More Time (V8.13 and Later)

The sendmail program imposes a timeout on the amount of time that a Milter can spend in its xxfi_eom() handler function without contacting sendmail. That timeout defaults to 300 seconds (5 minutes).[18] If your xxfi_eom() function determines that it needs more time to perform a task, it can reset the timeout by using the smfi_progress() routine.

[18] This timeout can be changed by adding an E: timeout to the sendmail configuration file's X command.

Before we show you how to use smfi_progress(), remember that the sending site is waiting for a reply to its dot while your xxfi_eom() function is busy. Even if you let sendmail know that you need more time, there is no guarantee that the sending site will not time out on its own.

The smfi_progress() routine is used this way:

ret = smfi_progress(ctx);

The smfi_progress() routine returns (here to ret) MI_FAILURE if ctx is invalid or if there is a network error; otherwise, it returns MI_SUCCESS. Thus, a complete call to smfi_progress() might look like this:

errno = 0;
if (smfi_progress(ctx) == MI_FAILURE)
{
  syslog(LOG_INFO, "smfi_progress(): %s",
   strerror(errno));
}

The only argument to smfi_progress() is the ctx connection-context pointer. It is the same ctx pointer that was passed to the enclosing xxfi_eom() handler function. It must not be NULL but must be a valid pointer.

The smfi_progress() routine is available only with sendmail V8.13 and later.

7.4.9. The smfi_quarantine() Routine Quarantines Messages (V8.13 and Later)

Beginning with sendmail V8.13, a Milter can now quarantine messages. Quarantining a message means that the queued file will not be seen by queue-processing invocations of sendmail unless sendmail is run manually to process them. A quarantined message is left unprocessed in the queue so that the administrator can review it before allowing or not allowing it to be delivered.

Milters can quarantine messages, as can rule sets, the command line, and the access database (via rule sets).

But before you can quarantine a message, you must first declare your intent to do so by adding the SMFIF_QUARANTINE flag to the flags portion of the smfiDesc structure (see section 7.2.2). For example:

struct smfiDesc smfilter =
{
  "MyMilter",        /* filter name */
  SMFI_VERSION,      /* a constant */
  SMFIF_QUARANTINE,  /* flags */
  ...

If you fail to declare the SMFIF_QUARANTINE flag but call smfi_quarantine() anyway, then that call will return MI_FAILURE.

After you have declared your intention to quarantine messages (by adding the SMFIF_QUARANTINE flag to smfiDesc), you can quarantine a message by calling the smfi_quarantine() milter-library routine from inside your xxfi_eom() handler function. For example:

ret = smfi_quarantine(ctx, message);

The smfi_quarantine() routine returns (here to ret) either MI_SUCCESS (if the message was successfully quarantined) or MI_FAILURE (if anything went wrong). Thus, a complete call to smfi_quarantine() might look like this:

errno = 0;
if (smfi_quarantine(ctx, message) == MI_FAILURE)
{
   syslog(LOG_INFO, "smfi_quarantine(): %s",
     errno == 0 ? "syntax error" : strerror(errno));
}

Note that smfi_quarantine() can fail for syntactic reasons (for example, a bad ctx pointer or a NULL message pointer) or for system-error reasons (for example, a network error). Syntax errors do not set errno to a useful value, whereas system errors do.

The first argument to smfi_quarantine() is the ctx connection-context pointer. It is the same ctx pointer that was passed to the enclosing xxfi_eom() handler function. It must not be NULL and must be a valid pointer.

The other argument to smfi_quarantine(), a string, is the address of the buffer containing the message to use for quarantining:

smfi_quarantine(ctx, "Detected a virus");

If the second argument is NULL or is an empty string, then smfi_quarantine() will return MI_FAILURE. This makes sense when you consider that sendmail, and the humans using it, can determine a quarantine reason only by examining this string.

If the sendmail configuration file's Milter.LogLevel option has a value of 4 or greater, the following will be logged when the message is quarantined:

Milter=name, quarantined=message

Here, the name is the name you gave this Milter in the sendmail configuration file, and message is the message you passed with the smfi_quarantine() call.

Note that multiple Milters can review any message. When your Milter quarantines a message, it merely marks the message as being quarantined, but the message in quarantined form is not stored until all Milters have reviewed the message.

Also note that a Milter can only quarantine a message and not de-quarantine it. This makes sense when you consider that sendmail can de-quarantine only when run by hand using special command-line arguments.

7.4.10. The smfi_stop() Routine Quits the Milter (V8.13 and Later)

Normal practice, when you write a Milter, is to call exit(2) when a fatal (nonrecoverable) error occurs. For example, many system errors can leave a Milter with no alternative except to give up and exit:

if (ferror(fp))
{
   syslog(LOG_INFO, "%s: %s", fname, strerror(errno));
   exit(errno);
}

Unlike exit(2), which causes all threads to exit immediately without concluding the connection processing, the smfi_stop() milter-library routine allows your Milter to shut down in stages. It does this by setting an internal flag that advises all threads to conclude their business and return. That flag also tells the milter's thread engine that it should launch no new threads and that it should return from smfi_main() (see section 7.2.5) when it is finished.

The smfi_stop() routine is used this way:

ret = smfi_stop();

The smfi_stop() routine returns (to ret, in this instance) the return value MI_SUCCESS no matter what, so its return value can be safely ignored. When you handle an error inside one of your handler functions, a complete call to smfi_stop() might look like this:

if (some error here)
{
   (void) smfi_stop();
   return SMFIS_TEMPFAIL;
}

The smfi_stop() routine takes no argument. Note that after you call smfi_stop(), the shutdown of your Milter becomes a foregone conclusion. There is no way to abort a call to smfi_stop() after it has started.

The smfi_stop() routine was available before V8.13, but was undocumented and therefore was unsupported. It is now documented and supported as of sendmail V8.13.

Chapter 8. The xxfi Handler Milter Functions

Milters are written using routines found in the Milter-library (libmilter.a or libmilter.so). Inside that library are routines used by your Milter to communicate decisions back to sendmail. Those routines are called smfi routines (sendmail filter routines) because the name of each begins with the prefix "smfi_."

One of those routines, smfi_register(), tells sendmail (via the milter-library) which of your Milter's functions handles which phase of analysis. For example, if your Milter is interested only in envelope senders, it might use smfi_register() to tell sendmail that it should call your Milter only during the envelope recipient phase and that it should call xxfi_envfrom(), an internal routine inside your Milter, to handle that phase.

The functions you write to handle each of the Milter's phases are called xxfi functions because each begins with the prefix xxfi_, in the libmilter/docs documentation. Although we maintain that convention in this chapter, be aware that the xxfi_ prefix is only one of many possible ways to refer to these handler functions. In practice, you can name them almost anything. For example, the envelope sender handling function (xxfi_envfrom) could alternatively be called any of the following:

handle_mail_from()
do_mail_from()
envelope_sender()

We will refer to these functions with the xxfi_ prefix, but for improved clarity, we will also call them handler functionsfor example, "the xxfi_envrcpt() handler function."

In addition to the documentation provided in this book, documentation regarding handler functions is provided in HTML format in the directory libmilter/docs as part of the sendmail source distribution. To view these HTML documents, change to that directory and run a command such as the following:

$ netscape file://`pwd´/index.html
$ lynx index.html

After your browser is up and displaying the initial documentation page, click on the link API Documentation and scroll down that next page to the section labeled "Callbacks."

In Table 8.1, we list all the xxfi handler functions (the ones you write) that handle the Milter phases. We describe them in the sections that follow.

Table 8.1. xxfi Handler Functions

Item

Section

Description

xxfi_connect()

8.4

Called once, upon initial connection by the sending site to the listening sendmail daemon.

xxfi_helo()

8.5

Normally called once, after the sending site sends its HELO or EHLO. But it can be called any time thereafter or may never be called.

xxfi_envfrom()

8.6

Called once per envelope, just after the sending site sends its MAIL FROM: envelope sender.

xxfi_envrcpt()

8.7

Called multiple times, once just after the sending site sends each of its RCPT TO: envelope recipients.

xxfi_header()

8.8

Called multiple times, once for each header received.

xxfi_eoh()

8.9

Called once per envelope, after all headers have been received.

xxfi_body()

8.10

Called multiple times, once for each chunk of the message body.

xxfi_eom()

8.11

Called once per envelope, after the entire body has been received (either xxfi_eom or xxfi_abort will be called but not both).

xxfi_abort()

§8.12

Called once, if the message was rejected outside the current Milter (either xxfi_eom or xxfi_abort will be called but not both).

xxfi_close()

§8.13

Called once, when the connection is normally closed.

8.1. Overview

Each xxfi handler function is declared in a similar manner:

sfistat
xxfi_name (SMFICTX *ctx, ...)

There are two characteristics common to all xxfi_ functions.

All return the same type (the sfsistat).

All receive a ctx pointer as the first argument (the ctx).

But there are also differences in that many take additional, and phase-specific, arguments (as indicated in the "..." in the code example).

8.1.1. Possible Return Value Macros

All the xxfi handler functions return the same type, sfsistat. Those return values must be one of five that are defined for you as macros, as shown in Table 8.2.

Table 8.2. xxfi Handler Function Return Value Macros

Macro

Description

SMFIS_TEMPFAIL

Reject with a temporary error (a 4xy SMTP reply code).

SMFIS_REJECT

Reject with a permanent error (a 5xy SMTP reply code).

SMFIS_DISCARD

Accept and discard.

SMFIS_ACCEPT

Accept; no further review is needed.

SMFIS_CONTINUE

No decision has yet been made, so just continue.

Returning any other value is an undefined error and should be avoided.

return SMFIS_ACCEPT;     Good, returns a well-defined macro
return errno;            Bad

8.1.2. The Common First ctx Argument

All the xxfi handler functions take as their first argument a variable that defines the connection context under which that function is running.

Consider, for example, two sendmail programs that are connected to a single Milter. Inside your Milter the xxfi_envfrom() function is called to process the envelope sender information. Your Milter doesn't know which sendmail (which inbound connection) it is dealing with. All it knows is that some inbound connection has just received an SMTP MAIL FROM command.

When an inbound site first connects to sendmail but before sendmail accepts that connection, your Milter is called, and a unique context identifier is assigned to that connection. The information that constitutes that identifier is encapsulated in the connection-context type. That connection-context type is defined in include/milter.h and is used this way:

xxfi_envfrom(SMFICTX * ctx, ...)

Every one of the xxfi_ handler functions begins with a first argument (ctx) that tells a Milter which connection it is dealing with.

8.2. The xxfi Orientations

There are four types of xxfi handler functions:

Recipient-oriented handler functions affect the review of individual recipients. Such routines can decide only whether a recipient should be accepted. They generally cannot affect the status of the entire message (envelope) nor the status of the connection (although V8.13 allows SMTP 421 to be returned using smfi_setreply(), which, in fact, drops the connection). The xxfi_envrcpt() handler function is the only recipient-oriented one.

Message-oriented (envelope-oriented) handler functions affect individual messages. Such routines can decide only whether the entire message (envelope) should be accepted. They affect all recipients contained in a message (by enclosing them) but cannot affect the status of the connection (although V8.13 allows SMTP 421 to be returned using smfi_setreply(), which drops the connection). The xxfi_envfrom(), xxfi_header(), xxfi_eoh(), xxfi_body(), and xxfi_eom() routines are all message oriented.

Connection-oriented handler functions affect an entire connection, during which multiple messages (envelopes) may be delivered to multiple sets of recipients. The xxfi_connect(), xxfi_helo(), and xxfi_close() routines are connection oriented.

Advisory-oriented handler functions accept status information only and make no decisions. The xxfi_abort() routine is advisory oriented.

Before any xxfi_ function can be used, it must be listed in the smfiDesc structure. Recall (see section 7.2.1) that an smfiDesc structure is declared this way:

struct smfiDesc smfilter =
{
   "MyMilter",       /* filter name (see section 7.2.1) */
   SMFI_VERSION,     /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,    /* flags (see section 7.2.2) */
   xxfi_connect,     /* the connection (see section 8.4) */
   xxfi_helo,        /* HELO/EHLO (see section 8.5) */
   xxfi_envfrom,     /* MAIL FROM (see section 8.6) */
   xxfi_envrcpt,     /* RVPT TO (see section 8.7) */
   xxfi_header,      /* headers (see section 8.8) */
   xxfi_eoh,         /* end of headers (see section 8.9) */
   xxfi_body,        /* a body chunk (see section 8.10) */
   xxfi_eom,         /* end of message (see section 8.11) */
   xxfi_abort,       /* message aborted (see section 8.12) */
   xxfi_close,       /* the cleanup (see section 8.13) */
};

In this section we describe all the xxfi_ handler functions in the order that they appear in an smfiDesc structure declaration, which, for the most part, parallels the order in which they will be called during a typical SMTP transaction.

8.3. Abort Logic

In general, you can think of a ctx pointer's context as extending from (a) the initial connection by the sending site to sendmail to (b) the close of that connection. Note that after xxfi_connect() is called for the initial connection, it is guaranteed that xxfi_close() will be called at the close of that connection. This logic allows you to plan to deallocate connection-oriented resources in xxfi_close().

But the logic is less straightforward for each envelope inside a connection. Each envelope comes into being with sendmail's receipt of a MAIL FROM command and exists until receipt of the concluding dot of the SMTP DATA phase.

You would expect that resources allocated before receipt of the concluding dot would be deallocated when that dot is processed (with the xxfi_eom() function) or upon envelope failure (with the xxfi_abort() function). But such is not necessarily the case.

If an envelope is rejected, tempfailed, quarantined, or deferred by sendmail or by any Milter other than the current Milter, then the current Milter's xxfi_abort() routine will be called in place of xxfi_eom(). But if the envelope is accepted, rejected, tempfailed, or deferred by the current Milter, then neither xxfi_eom() nor xxfi_abort() will be called.

This means that you must take special care inside each of your envelope-handling functions to deallocate resources if you accept, defer, tempfail, or reject the envelope from inside that function.

The xxfi_envfrom() handler function (see section 8.6) can accept, discard, and reject the envelope sender and thus the envelope, and if it does, neither this Milter's xxfi_eom() function nor its xxfi_abort() function will be called.

The xxfi_envrcpt() handler function (see section 8.7) can reject envelope recipients. If all envelope recipients are rejected, then sendmail will reject the envelope, and the current Milter's xxfi_abort() function will be called.

The xxfi_header() handler function (see section 8.8) can accept, reject, discard, or defer the current envelope, and, if it does, neither this Milter's xxfi_eom() function nor its xxfi_abort() function will be called.

The xxfi_eoh() handler function (see section 8.9) can accept, reject, discard, or defer the current envelope, and, if it does, neither this Milter's xxfi_eom() function nor its xxfi_abort() function will be called.

The xxfi_body() handler function (see section 8.10) can accept, reject, discard, or defer the current envelope, and, if it does, neither this Milter's xxfi_eom() function nor its xxfi_abort() function will be called.

Note that if the xxfi_eom() handler function (see section 8.8) is called, the xxfi_abort() function will not be called. Also note that if the xxfi_abort() handler function is called, the xxfi_eom() function will not be called. That is, if either is called, the other will not be called.

8.4. xxfi_connect() Reviews the Connection

The xxfi_connect() handler function is called to review a connection just after sendmail is connected to by another site but before sendmail sends its 220 SMTP code accepting that connection. If another site connects multiple times, xxfi_connect() will be called multiple timesonce for each connection. But note that each such call will be supplied with a separate connection context.

The xxfi_connect() handler function is called this way:

sfsistat
xxfi_connect(SMFICTX *ctx, char *hname, _SOCK_ADDR *haddr)

The return type is sfsistat, meaning that xxfi_connect() can return any of the values shown earlier in Table 8.2. We will discuss the effect of each of these possible return values shortly.

The xxfi_connect() handler function takes three arguments. The first argument is the usual ctx private-context pointer (see section 8.1.2), which defines which connection context is being handled.

The hname argument, a string containing the name of the connecting host, comes second.[1] That host name is derived by a reverse lookup of the IP number supplied in the next argument and will never be NULL, but it might be an empty string if sendmail could not look it up.

[1] Recall that a Milter can listen on a TCP/IP socket, so it can be connected to by sendmail processes running on many machines. The xxfi_connect() function is given the host name and IP number of the host connecting to sendmail, and not the sendmail-host connecting to the Milter.

The haddr argument, which comes third, is the address of a structure that contains a binary representation of the IP number of the connecting host, along with the type of the connection.

Internally, sendmail is connected to by a host. First, sendmail uses getpeername(3) to look up the address of the host that connected to it. If the connection was not over a socket (perhaps an SMTP connection using -bs with sendmail over standard input) or if the connection was on a socket type on which getpeername(3) does not work, then the third argument (haddr) is set to NULL. Otherwise, the third argument (haddr) is assigned the IP address found in binary form.

Next, sendmail performs a reverse lookup of the IP address it found and assigns the resulting host name to the second argument (the string hname). If the reverse lookup fails, the second argument (hname) is given the original IP number in text form, surrounded by square brackets (for example, [192.168.12.34]). But if the IP number could not be found, the second argument (hname) will be an empty string.

8.4.1. How to Use xxfi_connect()

Before you can use an xxfi_connect() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_connect() handler function's position in that structure includes the name of a function, the function named will be called each time a connection is made to sendmail:[2]

[2] Recall that we only use the name xxfi_connect() to parallel the names used in the sendmail documentation. You can, of course, use any name that makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",     /* filter name (see section 7.2.1) */
   SMFI_VERSION,   /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,  /* flags (see section 7.2.2) */
   xxfi_connect,   /* this function will be called upon a connection */

If the position of the xxfi_connect() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter at connection time:

struct smfiDesc smfilter =
{
   "MyMilter",     /* filter name (see section 7.2.1) */
   SMFI_VERSION,   /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,  /* flags (see section 7.2.2) */
   NULL,           /* don't call this milter upon a connection */

Remember that the value returned by the xxfi_connect() handler function is of type sfsistat. Table 8.3 shows the effect of each of the possible return values.

Table 8.3. Effect of Return Values for xxfi_connect()

Return Value

Effect on sendmail

SMFIS_ACCEPT

Accept the current connection; no further review by this Milter is needed. The Milter will not be contacted again until the connection closes, at which time the xxfi_close() handler function (see section 8.13) will be called. A subsequent Milter can still reject the connection.

SMFIS_REJECT

The connecting host is given a 554 reply code. The Milter will not be connected to again for this connection, except that the xxfi_close() handler function (see section 8.13) will be called. Subsequent Milters will not get a chance to review this connection.

SMFIS_TEMPFAIL

The connection is rejected with a 451 SMTP reply code, and all following client commands are tempfailed except QUIT. The Milter will not be contacted again for this connection, except that the xxfi_close() handler function (see section 8.13) will be called. Subsequent Milters will not get a chance to review this connection.

SMFIS_DISCARD

The same as SMFIS_ACCEPT.[a]

SMFIS_CONTINUE

Acknowledge the information received. This Milter will be contacted with the other requested information when and if that information becomes available. If the smfiDesc structure's position for the connect function is NULL, this becomes the default return value.

[a] . The milter-library only allows envelopes to be discarded. Because no envelope has been instantiated as of the connection, SMFIS_DISCARD must default to SMFIS_ACCEPT at this point in the SMTP conversation.

8.4.2. An Example of xxfi_connect()

To illustrate one use for an xxfi_connect() handler function, consider the possibility of skipping checks if the connection is not a network connection. Note that the numbers to the left are for reference only and are not a part of the code.

 4  sfsistat
 5  xxfi_connect(SMFICTX *ctx, char *hname, _SOCK_ADDR *haddr)
 6  {
 7    if (haddr == NULL ||
 8      (haddr->sa_family != AF_INET &&
 9        haddr->sa_family != AF_INET6))
10      return SMFIS_ACCEPT;
11
12    /* proceed with checking */

Here, we start the xxfi_connect() handler function (line 7) by checking its third argument, haddr. If haddr is NULL (as it would be if the connection was on the standard input) or if its family is not AF_INET (not an IPv4 network connection) and not AF_INET6 (an IPv6 network connection), then we return SMFIS_ACCEPT so that this Milter will not be called again (except at connection close). If the connection is a network connection, we continue to process it normally.

8.4.3. xxfi_connect and sendmail Macros

In addition to its own arguments, the xxfi_connect() handler function has access to selected sendmail macros whose values can be fetched by calling the smfi_getsymval() milter-library routine (see section 7.3.1). The following are the default macros made available to the xxfi_connect() handler function:

The ${daemon_name} macro holds the listening daemon's name, as it is configured on the sendmail host calling the Milter. For example, if there are two sendmail DaemonPortOptions settingsone listening to the inside LAN and another listening to the Internetthe Milter can use this macro to differentiate between the two and perhaps apply different rules to each.

The ${if_name} macro holds the name associated with the receiving (listening) interface on the sendmail host calling the Milter. Some operating systems allow a single interface to have multiple names and IP numbers. In such cases it can benefit a Milter to be able to differentiate among the various names by using this macro.

The ${if_addr} macro holds the IP address associated with the receiving (listening) interface on the sendmail host calling the Milter. Some operating systems allow a single interface to have multiple names and IP numbers. In such cases it can benefit a Milter to be able to differentiate among the various IP numbers by using this macro.

The ${j} macro holds the official canonical name of the sendmail host calling the Milter. When a Milter listens for connecting sendmail clients on the Internet, it needs some way to know which sendmail is connecting to it so that it can reject queries from unknown hosts. Although the value in this macro can be faked, it nonetheless provides one simple means for this kind of authentication.

The ${_} macro holds the RFC1413 validation (if any), also known as ident lookup, performed by the sendmail host calling the Milter to evaluate the connecting IP number. One possible use for this macro is to check for attempts at IP source routing.

In addition to these defaults, other macros can be useful during a Milter's review of the connection. To add a macro to the default list, you can include a line such as the following in your sendmail mc configuration file.

define(`confMILTER_MACROS_CONNECT´,
  confMILTER_MACROS_CONNECT``,{client_resolve}´´)

Note the use of single-quote pairs, which is required because of the inclusion of a comma. Here is one macro that might aid in a connection review:

The ${client_resolve} macro holds the result of a lookup of the connecting host's name as a constant that is either OK, FAILED, FORGED, or TEMP. A value of FORGED means that the forward lookup doesn't match the reverse lookup and suggests a basis for your Milter to reject the connection.

8.4.4. Other Uses for xxfi_connect()

Following are other uses for the xxfi_connect() handler function.

Keeping track of connections on a single machine to ensure that the simultaneous connections from any arbitrary IP number are not excessive. (Be sure to execute a mutex on any global variables to protect them from simultaneous access by multiple threads.)

Looking up the host name and IP number in a database to see whether it requires special processing (such as whitelisting or blacklisting).

Detecting whether the connection is on the loopback interface (and thus is locally submitted mail).

Logging (recording) every connection, whether or not it results in actual delivery of mail.

8.5. xxfi_helo() Reviews SMTP HELO/EHLO

The xxfi_helo() handler function is called just after sendmail receives a HELO or EHLO command from the connecting site but before sendmail sends its reply to that command. The xxfi_helo() handler function is generally called after the connection has been accepted and before any envelope information is processed. Note, however, that the connecting site can send HELO or EHLO at unexpected times and multiple times, and, if it does, xxfi_helo() will also be called. Thus, this function's design should not presume any particular timing.

Also note that unless you configure sendmail to require the HELO or EHLO command before the MAIL FROM: command, a sending site may choose to skip sending the HELO or EHLO command, meaning that this xxfi_helo() will not be called. To configure sendmail to require HELO or EHLO, you can include one of the following in your mc configuration file:

define(`confPRIVACY_FLAGS´,`goaway´)
define(`confPRIVACY_FLAGS´,`needmailhelo´)

8.5.1. How to Use xxfi_helo()

The xxfi_helo() handler function is called this way:

sfsistat
xxfi_helo(SMFICTX *ctx, char *host)

The return type is sfsistat, and thus xxfi_helo() can return any of the values shown earlier in Table 8.2. We discuss the effect of each of these possible return values shortly.

The xxfi_helo() handler function takes two arguments. The first argument is the usual ctx private-context pointer (see section 8.1.2), which defines which connection is being handled.

The host argument comes second. It is a string containing the host name that the connecting site gave as part of its HELO or EHLO command. Note that this is literally what the other host gave and is not cleaned up by sendmail in any manner.

HELO client.example.gov
HELO bob
HELO none.of.your.business.who.I.am

All of the preceding will be accepted by sendmail. Internally, sendmail cares only that the host name is present and that it appears to be a host name. It is up to your xxfi_helo() handler function to determine the validity of the given host name and whether or not to accept it.

Before you can use an xxfi_helo() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine. If the xxfi_helo() handler function's position in that structure includes the name of a function, the function named will be called each time a client issues a HELO or EHLO command:[3]

[3] . Recall that we only use xxfi_helo() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",     /* filter name (see section 7.2.1) */
   SMFI_VERSION,   /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,   /* the connection ((see section 8.4)) */
   xxfi_helo,      /* this function reviews HELO/EHLO */

If the position of the xxfi_helo() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter just after receiving a HELO or EHLO command:

struct smfiDesc smfilter =
{
   "MyMilter",    /* filter name (see section 7.2.1) */
   SMFI_VERSION,  /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS, /* flags (see section 7.2.2) */
   xxfi_connect,  /* the connection (see section 8.4) */
   NULL,          /* don't call this milter to review HELO/EHLO */

Remember that the value returned by the xxfi_helo() handler function is of type sfsistat. Table 8.4 shows the effect of each of the possible return values.

Table 8.4. Effect of Return Values for xxfi_helo()

Return Value

Effect on sendmail

SMFIS_ACCEPT

Accept the current connection; no further review by this Milter is needed. The Milter will not be contacted again until the connection closes, whereupon xxfi_close() will be called. A subsequent Milter can still reject the connection.

SMFIS_REJECT

The connecting host is given a 250 reply, and each subsequent command from that connecting site is given a 550 reply except for QUIT (given a 221 reply) and NOOP (given a 250 reply). The Milter will not be connected to again for this connection, except that xxfi_close() will be called upon the connection's close. Subsequent Milters will not get a chance to review this connection.

SMFIS_TEMPFAIL

The HELO or EHLO command is tempfailed with a 451 reply, and all subsequent commands are also tempfailed except for QUIT (given a 221 reply) and NOOP (given a 250 reply). The Milter will not be contacted again for this connection, except that xxfi_close() will be called upon the connection's close. Subsequent Milters will not get a chance to review this connection.

SMFIS_DISCARD

The same as SMFIS_ACCEPT.[a]

SMFIS_CONTINUE

Acknowledge the information received. This Milter will be contacted with the other requested information when and if that information becomes available. If the smfiDesc structure's position for the HELO function is NULL, this becomes the default return value.

[a] . The milter-library allows only envelopes to be discarded. Because no envelope has been instantiated as of the connection, SMFIS_DISCARD must default to SMFIS_ACCEPT at this point in the SMTP conversation.

8.5.2. An Example of xxfi_helo()

To illustrate one use for an xxfi_helo() handler function, consider the need to defer a connection if the specified host name cannot be looked up.

 1  sfsistat
 2  xxfi_helo(SMFICTX *ctx, char *host)
 3  {
 4    struct hostent *hp;
 5
 6    if ((hp = getipnodebyname(host,
 7      AF_INET, AI_DEFAULT, NULL)) == NULL)
 8    {
 9      return SMFIS_TEMPFAIL;
10    }
11    (void) freehostent(hp);
12    /* proceed with more checking */

Here, we start the xxfi_helo() handler function (line 6) by looking up the given host name using the getipnodebyname() C library routine. If the lookup returns NULL (as it would if the host name could not be found), then we return SMFIS_TEMPFAIL (line 9) so that the connection will be deferred and so that this Milter will not be called again until the connection closes. If we succeed with the lookup, we free the value returned by getipnodebyname() (line 11) and then continue to process the HELO/EHLO information.

8.5.3. xxfi_helo() and sendmail Macros

In addition to its own arguments, the xxfi_helo() handler function has access to selected sendmail macros. Any of the following macros can be fetched by calling the smfi_getsymval() Milter-library routine (see section 7.3.1). The following default macros are available to your Milter:

The ${tls_version} macro holds the Transaction Layer/Secure Socket Layer (TLS/SSL) version of the connection as established on the sendmail host contacting the Milter.

The ${cipher} macro holds the cipher suite used for the connection as established on the sendmail host calling the Milter.

The ${cipher_bits} macro holds the TLS encryption key length as established on the sendmail host calling the Milter.

The ${cert_subject} macro holds the common name of the certificate sent to the sendmail host calling the Milter.

The ${cert_issuer} macro holds the certificate issuer's common name for the certificate sent to the sendmail host calling the Milter.

8.5.4. Other Uses for xxfi_helo()

Following are other uses for the xxfi_helo() handler function.

Verifying that the correct cipher suite was used for the connection

Detecting patterns in the HELO/EHLO string that are known to indicate spamming software.

8.6. xxfi_envfrom() Reviews SMTP MAIL FROM

The xxfi_envfrom() handler function is called after sendmail receives a MAIL FROM command from the connecting site but before sendmail sends its reply to that command. The MAIL FROM command supplies the envelope sender address and (possibly) additional information about how to handle that sender's address.

The xxfi_envfrom() handler function is generally called after the HELO or EHLO has been accepted and before any envelope recipients are processed. Note, however, that although the connecting site can send MAIL FROM at unexpected times, sendmail will connect to the xxfi_envfrom() only once per envelope. Defensive programming, however, encourages you not to presume any particular timing.

8.6.1. How to Use xxfi_envfrom()

The xxfi_envfrom() handler function is called this way:

sfsistat
xxfi_envfrom(SMFICTX *ctx, char **argv)

The return type is sfsistat, and thus xxfi_envfrom() can return any of the values shown earlier in Table 8.2. We discuss the effect of each of these possible return values shortly.

The xxfi_envfrom() handler function takes two arguments. The first argument is the usual ctx private-context pointer (see section 8.1.2), which defines which connection context is being handled.

The argv argument is second and is an array of pointers to strings. That array contains the address given to the MAIL FROM address as its zeroth element (argv[0]) and has NULL as its last element.

Before the Milter receives this array, sendmail has already reviewed it for values that are out of bounds or are illegal. If sendmail found anything to object to, it will reject this envelope sender and will not call this routine with the same connection context, unless another envelope sender is sent for the same current connection.

The MAIL FROM command looks like this:

MAIL FROM:<address>

Here, address is the address of the envelope sender. This is the address to which error notification will be sent if the message later fails to be delivered. If the sending site sent EHLO as its greeting instead of HELO and if sendmail accepted the EHLO, the sender is allowed to use extended SMTP commands. Among those extensions are additional arguments that can be appended to the MAIL FROM command following the address. For example:

MAIL FROM:<address> SIZE=bytes
MAIL FROM:<address> AUTH=address
MAIL FROM:<address> ENVID=string

The items that follow the MAIL FROM address vary depending on the desires of the sending site. There may be no additional items, there may be one as just shown, or there may be many (following we show two):

MAIL FROM:<address> SIZE=bytes ENVID=string

If this MAIL FROM command were passed to your xxfi_envfrom() handler function, the argv array would contain the following elements:

argv[0] = "<address>"
argv[1] = "SIZE=bytes"
argv[2] = "ENVID=string"
argv[3] = NULL

If the message is a bounce message, the envelope sender may appear as an empty set of angle brackets:

argv[0] = "<>"

Your program should be prepared to handle such empty addresses, as well as the odd, but legal, addresses that old MTAs may give to sendmail. For example, consider the following:

argv[0] = "bob@example.com"                Missing angle brackets
argv[0] = "<Jeff <jeff@example.com>>"      Nested angle brackets

Before you can use an xxfi_envfrom() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_envfrom() handler function's position in that structure includes the name of a function, the function named will be called each time a MAIL FROM command is made to sendmail:[4]

[4] . Recall that we only use xxfi_envfrom() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",     /* filter name (see section 7.2.1) */
   SMFI_VERSION,   /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,  /* flags (see section 7.2.2) */
   xxfi_connect,   /* the connection (see section 8.4) */
   xxfi_helo,      /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,   /* this function reviews MAIL FROM */

If the position of the xxfi_envfrom() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter after receiving a MAIL FROM command:

struct smfiDesc smfilter =
{
   "MyMilter",     /* filter name (see section 7.2.1) */
   SMFI_VERSION,   /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,  /* flags (see section 7.2.2) */
   xxfi_connect,   /* the connection (see section 8.4) */
   xxfi_helo,      /* the HELO/EHLO (see section 8.5) */
   NULL,           /* don't call this milter to review MAIL FROM */

Remember that the value returned by the xxfi_envfrom() handler function is of type sfsistat. Table 8.5 shows the effect of each of the possible return values.

Table 8.5. Effect of Return Values for xxfi_envfrom()

Return Value

Effect on sendmail

SMFIS_ACCEPT

Accept the current message (envelope). No further review by this Milter is needed, and the Milter will not be contacted again for this envelope (but will be contacted again on connection close). However, the Milter can be contacted for a next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope sender.

SMFIS_REJECT

The MAIL FROM: command is rejected with a 5yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted again on connection close). However, the Milter can be contacted for a next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope sender or envelope.

SMFIS_TEMPFAIL

The MAIL FROM: command is rejected with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted again on connection close). However, the Milter can be contacted for a next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope sender or envelope.

SMFIS_DISCARD

The current envelope is discarded. The Milter will not be contacted again for this envelope (but will be contacted again on connection close). However, the Milter can be contacted for a next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope sender or envelope.

SMFIS_CONTINUE

Acknowledge that the information was received. The Milter can be contacted again when and if that additional information becomes available. If the envelope sender position in the smfiDesc structure is NULL, this value is the default.

If your Milter accepts, rejects, defers, or discards the envelope sender, you must remember to deallocate envelope-specific resources at the same time. Remember that any return other than SMFIS_CONTINUE will prevent both your xxfi_eom() function (see section 8.11) and your xxfi_abort() function (see section 8.12) from being called for this envelope. Also remember that the xxfi_close() function (see section 8.13) will still be called when the connection is closed.

8.6.2. An Example of xxfi_envfrom()

To illustrate one use for an xxfi_envfrom() handler function, consider the need to save the envelope sender's address for later comparison to the header sender address. Remember that at this point in the SMTP conversation we have not yet seen any headers.

 1  sfsistat
 2  xxfi_envfrom(SMFICTX *ctx, char **argv)
 3  {
 4    struct priv_struct *priv;
 5
 6    priv = smfi_getpriv(ctx);
 7    if (priv == NULL)
 8      return SMFIS_TEMPFAIL;
 9
10    if (argv[0] != NULL)
11    {
12      priv->mailfrom = strdup(argv[0]);
13      if (priv->mailfrom == NULL)
14        return SMFIS_TEMPFAIL;
15    }
16    return SMFIS_CONTINUE;
17  }

Here, we start the xxfi_envfrom() handler function by declaring a pointer (line 4) that will point to our private data storage. We declare that pointer as type struct priv_struct and pretend, for the sake of example, that you declared the form for that structure elsewhere.

Next, smfi_getpriv() (see section 7.3.3) is called to fetch the private data storage and to assign the address of that stored data to the priv pointer (line 6). If the smfi_getpriv() milter-library routine returns NULL (line 7), then this xxfi_envfrom() function returns SMFIS_TEMPFAIL so that delivery will be deferred.

Notice (line 12) that the sender's address (from argv[0]) is copied only if it is not NULL.[5]

[5] Always program defensively, and never presume that anything is impossible.

If the copy step fails (line 13), this xxfi_envfrom() function again returns SMFIS_TEMPFAIL so that delivery will be deferred. But if all goes well, it returns SMFIS_CONTINUE (line 16).

8.6.3. xxfi_envfrom() and sendmail Macros

In addition to its own arguments, the xxfi_envfrom() handler function has access to selected sendmail macros. Any of the following macros can be fetched, and their values used, by calling the smfi_getsymval() milter-library routine (see section 7.3.1). The following are the default macros sent to xxfi_envfrom(), but you can redefine this list or replace it in your mc file:

The ${i} macro holds the sendmail identifier used for the queued message. (Remember that all messages are normally queued for safety.)

The ${auth_type} macro holds the mechanism selected for authentication, if any.

The ${auth_authen} macro holds the approved authentication credentials, if any.

The ${auth_ssf} macro holds an integer representation of the number of bits used in the key length associated with the encryption used.

The ${auth_author} macro holds the address of a user other than the envelope sender, if there was one, who is allowed to authenticate on behalf of the envelope sender.

The ${mail_mailer} macro holds the $# portion (the delivery agent name) of the delivery agent triple that was selected for the envelope sender.

The ${mail_host} macro holds the $@ portion (the relay host part) for the delivery agent triple that was selected for the envelope sender.

The ${mail_addr} macro holds the $: portion (the sender address) for the delivery agent triple that was selected for the envelope sender.

8.6.4. Other Uses for xxfi_envfrom()

Following are other uses for the xxfi_envfrom() handler function.

Rejecting connections that use an encryption key length that is too small

Looking up the envelope sender's host and comparing its IP address to a list of rejected addresses

Looking up the envelope sender's address in a whitelist database to see whether this sender is allowed to send mail to this domain

8.7. xxfi_envrcpt() Reviews SMTP RCPT TO

The xxfi_envrcpt() handler function is called each time sendmail receives a RCPT TO command from the connecting site but before sendmail sends its reply to that command. Each RCPT TO command supplies an envelope recipient address and (possibly) additional information about how to handle the recipient address.

The xxfi_envrcpt() handler function is generally called after the MAIL FROM envelope sender has been accepted and before the message body is reviewed. Note, however, that although the connecting site can send RCPT TO at unexpected times, sendmail will call your Milter only if that command is received after MAIL FROM: and before DATA. Defensive programming, however, suggests that you design your xxfi_envrcpt() to avoid presuming any particular timing.

An envelope may contain an arbitrary number of recipients (subject to limitations imposed by sendmail), so this xxfi_envrcpt() function may be called multiple times for each envelope. It is called once per recipient and provides the information about a single recipient each time it is called.

8.7.1. How to Use xxfi_envrcpt()

The xxfi_envrcpt() handler function is called this way:

sfsistat
xxfi_envrcpt(SMFICTX *ctx, char **argv)

The return type is sfsistat, and thus xxfi_envrcpt() can return any of the values shown earlier in Table 8.2. We will discuss the effect of each of these possible return values shortly.

The xxfi_envrcpt() handler function takes two arguments. The first argument is the usual ctx private-context pointer (see section 8.1.2), which defines the connection context being handled.

The argv argument is second and is an array of pointers to strings. That array contains the address given to the RCPT TO command as its zeroth element (argv[0]), and the entire array has NULL as its last element.

Before the Milter receives this array, sendmail has already reviewed its contents for any values that are out of bounds or illegal. If sendmail found anything to object to, it will reject this envelope recipient, and the Milter will not see it.

The RCPT TO command looks like this:

RCPT TO:<address>

Here, address is the address of a single envelope recipient. There may be multiple RCPT TO commands (one per recipient) if there are multiple recipients. But note that your Milter will be called only for good recipients (that is, sendmail will not let your Milter review recipients it has rejected itself).[6]

[6] If the delay_checks feature is used when sendmail is configured, sendmail will pass recipients that it has not already screened itself to xxfi_envrcpt().

If the sending site sent EHLO as its greeting (instead of HELO) and if sendmail accepted the EHLO, then the sending site is allowed to use extended SMTP commands. Among those extensions are additional arguments that can be appended to the RCPT TO command following the address. For example:

RCPT TO:<address> ORCPT=rfc822;gw@example.gov

The items that follow the RCPT TO address vary depending on the desires of the sending site. There may be no additional items, there may be one (shown), or there may be many.

When the address and optional arguments are passed from a RCPT TO command to your xxfi_envrcpt() handler function, the argv array might contain, for example, the following elements:

argv[0] = "<address>"
argv[1] = "ORCPT=rfc822;gw@example.gov"
argv[2] = NULL

Note that your program should be prepared to handle the odd, but legal, addresses that old MTAs may give to sendmail. For example, consider the following:

argv[0] = "sally@example.com"             Missing angle brackets
argv[0] = "<Betty <betty@example.com>>"   Nested angle brackets

Before you can use an xxfi_envrcpt() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_envrcpt() handler function's position in that structure includes the name of a function, the function named will be called each time a RCPT TO command is made to sendmail:[7]

[7] Recall that we only use xxfi_envrcpt() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* this function reviews RCPT TO */

If the position of the xxfi_envrcpt() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter after receiving a RCPT TO command:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   NULL,            /* don't call this milter to review RCPT TO */

Remember that the value returned by the xxfi_envrcpt() handler function is of type sfsistat. Table 8.6 shows the effect of each of the possible return values.

Table 8.6. Effect of Return Values for xxfi_envrcpt()

Return Value

Effect on sendmail

SMFIS_ACCEPT

Accept the current envelope (not just the current recipient). No further review by this Milter is needed, so it will not be contacted again (until the connection is closed). However, the Milter can be contacted for a next envelope if there is one on this same connection. A subsequent Milter can still reject this recipient.

SMFIS_REJECT

Reject the current recipient with a 5yz SMTP reply code. This recipient part of the Milter can be contacted again if there are multiple recipients, or sendmail might go directly to the DATA phase after all recipients have been processed (if there is at least one good recipient). Subsequent Milters will not get a chance to review this recipient. If rejection results in zero recipients, then sendmail will reject the envelope, and the current Milter's xxfi_abort() function will be called.

SMFIS_TEMPFAIL

Temporarily reject the current recipient with a 4yz SMTP reply code. This recipient part of the Milter can be contacted again if there are multiple recipients, or sendmail might go directly to the DATA phase after all recipients have been processed (if there was at least one good recipient). Subsequent Milters will not get a chance to review this recipient. If tempfail results in zero recipients, then sendmail will cease processing the envelope, and the current Milter's xxfi_abort() function will be called.

SMFIS_DISCARD

The current envelope is discarded. The Milter will not be contacted again for this envelope, even if there are more recipients. The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this recipient.

SMFIS_CONTINUE

Acknowledge the information received. The Milter will be contacted with other requested information when and if that information becomes available. If the envelope recipient position in the smfiDesc structure is NULL, the return value becomes the default.

If your Milter rejects or defers so many recipients that there are none left, sendmail will reject the message. Remember that this rejection is by sendmail, so your xxfi_abort() function (see section 8.12) will be called.

8.7.2. An Example of xxfi_envrcpt()

To illustrate one use for an xxfi_envrcpt() handler function, consider the need to log and reject recipients addressed to your printer.

 1  sfsistat
 2  xxfi_envrcpt(SMFICTX *ctx, char **argv)
 3  {
 4    char *ph = "lpr.your.domain";
 5    char *cp;
 6
 7    if (argv[0] == NULL)
 8      return SMFIS_CONTINUE;
 9
10    if ((cp = strchr(argv[0], '@')) != NULL)
11      ++cp;
12    else
13      return SMFIS_CONTINUE;
14    if (strncasecmp(ph, cp, strlen(ph)) == 0)
15    {
16      syslog(LOGINFO, "%s print host rejected", ph);
17      return SMFIS_REJECT;
18    }
19    return SMFIS_CONTINUE;
20  }

Here, the xxfi_envrcpt() handler function (line 4) begins by declaring the name of the printer host. In a more complex Milter, this might be a list of hosts or (perhaps) the result of reading a list of hosts from a file or database.

Next, the recipient's address is searched for an @ character (line 10). If it is not found, SMFIS_CONTINUE is returned (line 13) because the address does not contain a host part, so it might be a local address. If the recipient address contains an @ character, the cp pointer (line 11) is set to point just after that character (that is, at the first character of the host part). Notice (line 7) that the recipient's address (from argv[0]) causes this check to be skipped if it is NULL.[8]

[8] Always program defensively, and never presume that anything is impossible.

The printer's host name is compared to the recipient host name (line 14), and if they are the same, that recipient is rejected (line 17). The fact that the printer address was rejected is also recorded (line 16) using syslog(3).

If the host part of the recipient's address is not the address of the printer (line 19), then SMFIS_CONTINUE is returned.

8.7.3. xxfi_envrcpt() and sendmail Macros

In addition to its own arguments, the xxfi_envrcpt() handler function has access to selected sendmail macros. Any of the following macros can be fetched, and their values used, by calling the smfi_getsymval() milter-library routine (see section 7.3.1). The following are the default macros sent to xxfi_envrcpt(), but you can redefine this list or replace it in your mc file:

The ${rcpt_mailer} macro holds the $# portion (the delivery agent name) for the delivery agent triple that was selected for the envelope recipient.

The ${rcpt_host} macro holds the $@ portion (the relay host) for the delivery agent triple that was selected for the envelope recipient.

The ${rcpt_addr} macro holds the $: portion (the recipient address) for the delivery agent triple that was selected for the envelope recipient.

8.7.4. Other Uses for xxfi_envrcpt()

Following are other uses for the xxfi_envrcpt() handler function.

Counting the number of good recipients, and later, in the xxfi_eom() routine (see section 8.11), comparing that count to the number of bad recipients and rejecting the message if the ratio is too high

Comparing each recipient to the envelope sender to validate whitelisting pairs, perhaps using a database lookup

Setting up a list of honey-pot (bait) recipients and comparing each to every envelope recipient so that possible address harvesting can be detected

Flagging selected recipients so that expected missing recipients can later be found (as for mandatory archiving)

8.8. xxfi_header() Reviews Headers

After all recipients have been reviewed, the data portion of the message is reviewed. Recall that the data portion is composed of two parts: the message headers and the message body. These two parts are separated by a single blank line within the data portion. The headers are always first, and the message body is always second. Each Milter reviews the data portion (header and body) before the next Milter is allowed to review that data portion.

The xxfi_header() handler function is called each time sendmail gathers a header. Recall that headers are composed of two parts: the name to the left of the colon and the value to the right.

Name: value

The xxfi_header() handler function is called once for each header that sendmail has gathered from the data portion of the message. Some headers (such as Message-Id:) should appear only once in a message, whereas others (such as Received:) can appear multiple times. The sendmail program passes headers to a Milter in the order that they appear in a message. Each Milter will see all headers (and the body) before the next Milter sees them.

Remember that the Milter will be called again after all headers have been reviewed so that missing headers can be detected. Also remember that other Milters, sendmail, and sendmail rule sets are also reviewing headers, and their decisions may result in a message being rejected before your Milter gets a chance to review the headers in that message.

Also remember that headers can span multiple lines in a message. For example, a Received: header might look like this:

Received: from example.gov (mx.example.gov [1.2.3.4])
  by smtp.gov with ESMTP id i08KjvWt014695
  for <bob@example.gov>; Thu, 8 Jan 2004 13:46:10 -0700

This Received: header occupies three lines. The first line begins with the header name. The last two lines each begin with an indentation character, here a tab (but a space or spaces will have the same effect). When a multiline header value is reviewed by a Milter, the multiline value will be assembled into a single line, with the indentation characters and carriage-return/linefeed character pairs (ending each line) preserved. The following is a single such line, wrapped to fit the page:

[View full width]

Received: from example.gov (mx.example.gov [1.2.3.4])\r\n\tby smtp.gov with ESMTP id
 i08KjvWt014695\r\n\tfor <bob@example.gov>; Thu, 8 Jan 2004 13:46:10 -0700

Here, the indentation character (a tab) is represented as \t, and a carriage-return/linefeed pair is represented as \r\n.

The data portion of a message can contain an arbitrary number of headers, so this xxfi_header() function can be called multiple times for each data portion.[9] It is called once per header and provides the information about a single header each time it is called.

[9] This is subject to any limitations imposed by sendmail.

Note that MIME allows headers to appear in the message body. This xxfi_header() function is called only for message headers and is not called for MIME headers that may appear in the message body.

8.8.1. How to Use xxfi_header()

The xxfi_header() handler function is called this way:

sfsistat
xxfi_header(SMFICTX *ctx, char *name, char *value)

The return type is sfsistat, and thus xxfi_header() can return any of the values shown earlier in Table 8.2. We discuss the effect of each of the possible return values shortly.

The xxfi_header() handler function takes three arguments. The first argument is the usual ctx private-context pointer (see section 8.1.2), which defines the connection context that is being handled.

The name argument comes second. It is the name portion of the headerthat is, the portion to the left of the colon (but omitting the colon). Thus, for the following, the name portion is the "To" as a string. The name portion is not case sensitive (that is, to, To, and TO are all the same).

To: bob

The value argument is third and is the value portion of the header, the portion to the right of the colon. So for the following, the value portion is the "bob" as a string.

To: bob

Before the Milter receives a header, sendmail has already reviewed that header for any values that are out of bounds or are illegal. If sendmail finds, for example, an overly long header, it will truncate that header's value before passing it to the Milter.[10]

[10] Although RFC2822 imposes no limit on the size of a header's value, sendmail carefully screens for absurdly long header values.

The sendmail program does not strip the high bit from header-value characters that have the high bit set, so a Milter may be useful in screening for such illegal header values.

Before you can use an xxfi_header() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_header() handler function's position in that structure includes the name of a function, the function named will be called each time a header line needs to be reviewed:[11]

[11] Recall that we only use xxfi_header() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* this function reviews each header */

If the position of the xxfi_header() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter to process each header:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   NULL,            /* don't call this milter to review headers */

Remember that the value returned by the xxfi_header() handler function is of type sfsistat. Table 8.7 shows the effect of each of the possible return values.

Table 8.7. Effect of Return Values for xxfi_header()

Return Value

Effect on sendmail

SMFIS_ACCEPT

The current header marks the entire envelope (not just the header) as acceptable to the Milter, so the envelope is accepted. The Milter will not be contacted again for this envelope until the connection is closed. The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope.

SMFIS_REJECT

Reject the current DATA phase with a 5yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_TEMPFAIL

Temporarily reject the DATA phase with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_DISCARD

The current envelope is discarded. The Milter will not be contacted again for this envelope (but will be contacted when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_CONTINUE

Acknowledge the information received. The Milter will be contacted with other requested information when and if that information becomes available. If the header portion of the smfiDesc structure is NULL, this becomes the default returned value.

If your Milter accepts, rejects, defers, or discards the envelope because of a header, you must remember to deallocate envelope-specific resources at the same time. Remember that this envelope rejection will prevent both your xxfi_eom() function (see section 8.11) and your xxfi_abort() function (see section 8.12) from being called for this envelope. But note that xxfi_close() will still be called when the connection is closed.

8.8.2. An Example of xxfi_header()

To illustrate one use for an xxfi_header() handler function, consider the need to reject any message whose Subject: header is in Chinese.

 1  sfsistat
 2  xxfi_header(SMFICTX *ctx, char *name, char *value)
 3  {
 4    char **c;
 5    static char *cf[] = {
 6      "=?EUC", "=?GB2312", "=?HZ", "=?Mac", "=?Big5",
 7      "=?CNS", "=?Eten", NULL};
 8
 9    if (name == NULL || value == NULL)
10      return SMFIS_CONTINUE;
11    if (strcasecmp(name, ¨subject¨) == 0)
12    {
13      for(c = cf; *c != NULL; ++c)
14      {
15        if (strncasecmp(*c, value,
16             strlen(*c) == 0)
17        }
18          /* ... deallocate envelope data here */
19          return SMFIS_REJECT;
20        }
21      }
22      return SMFIS_CONTINUE;
23    }
24    /* ...other headers reviewed from here down */
25    return SMFIS_CONTINUE;
26  }

Here, we start the xxfi_header() handler function (line 5) by declaring a variable (cf) that holds, as its value, a list of expressions that can begin a Subject: header's value part. These expressions all indicate that the Subject: header's value is in one or another form of Chinese encoding. For example:

Subject: =?Big5?B?pHCsR6jGpGq5RLJ6LS2xeqRdpmKkq7jMttw=?=

The list in cf is terminated by a NULL (line 7).

The first portion of code (line 9) skips any header that lacks a name or a value.[12]

[12] Always program defensively, and never presume that anything is impossible.

The name of each valid header is reviewed to see whether it is a Subject: header. The comparison (line 11) is case insensitive because header names are case insensitive (that is, Subject:, subject:, and SuBjEcT: are all the same).

If this function has received a Subject: header to review, that header's value is compared to each of the beginning expressions (line 13) in cf to see whether the value begins with one of those (line 15).[13] If the header's value begins with one of the objectionable prefixes, the message is rejected (line 19). Note (line 18) that rejecting a header rejects the envelope, so neither xxfi_eom() nor xxfi_abort() will be called. Envelope cleanup must happen here unless it can be deferred to connection cleanup in xxfi_close().

[13] In practice this MIME encoding can begin anywhere in the Subject: header.

Other headers are then given a chance to be reviewed (line 24).

8.8.3. Other Uses for xxfi_header()

Following are other uses for the xxfi_header() handler function.

Recording the presence of a header so that the xxfi_eoh() handler function (see section 8.9) can detect its absence

Checking header values for adherence to standards and rejecting those that do not conform

Reviewing the sequence of Received: headers to detect bogus ones or ones that break a logical sequence

Flagging the presence of an unwanted header for later removal by the xxfi_eom() handler function (see section 8.11).

8.9. xxfi_eoh Reviews at End of Headers

After all recipients have been reviewed, the data portion of a message is reviewed. Recall that the data portion is composed of two parts: the message headers and the message body. These two parts are separated by a single blank line. The headers are always first, and the message body is always second. Each Milter reviews the data portion (header and body) before the next Milter is allowed to review the data portion.

The xxfi_header() handler function is called each time sendmail gathers a header. After all headers have been reviewed (and if that header review process did not already accept, reject, defer, or discard the envelope), the xxfi_eoh() handler function is called next (but only once).

Remember that other Milters, sendmail, and sendmail rule sets are also reviewing headers and may reject a message before your Milter gets a chance to perform any header reviews, including a post-header review with xxfi_eoh().

8.9.1. How to Use xxfi_eoh()

The xxfi_eoh() handler function is called this way:

sfsistat
xxfi_eoh(SMFICTX *ctx)

The return type is sfsistat, and thus xxfi_eoh() can return any of the values shown earlier in Table 8.2. We discuss the effect of each of the possible return values shortly.

The xxfi_eoh() handler function takes a single argument: the usual ctx private-context pointer (see section 8.1.2), which defines the connection that is being handled.

Before you can use an xxfi_eoh() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_eoh() handler function's position in that structure includes the name of a function, the function named will be called once after all headers have been reviewed:[14]

[14] Recall that we only use xxfi_eoh() to parallel the sendmail documentation. You may, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   xxfi_eoh,        /* this function performs post-header review */

If the position of the xxfi_eoh() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter to perform post-header processing:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   NULL,            /* don't call this milter to perform post-header processing */

Remember that the value returned by the xxfi_eoh() handler function is of type sfsistat. Table 8.8 shows the effect of each of these possible return values.

Table 8.8. Effect of Return Values for xxfi_eoh()

Return Value

Effect on sendmail

SMFIS_ACCEPT

The entire envelope is acceptable to the Milter, so the envelope is accepted. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope.

SMFIS_REJECT

Reject the current DATA phase with a 5yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_TEMPFAIL

Temporarily reject the DATA phase with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_DISCARD

The current envelope is discarded. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_CONTINUE

Acknowledge the information received. The Milter will be contacted with other requested information when and if that information becomes available. If the end-of-header part of the smfiDesc structure is NULL, this becomes the default return value.

If your Milter accepts, rejects, defers, or discards the envelope because of a header or missing header, you must remember to deallocate envelope-specific resources at the same time. Remember that this envelope rejection will prevent both your xxfi_eom() function (see section 8.11) and your xxfi_abort() function (see section 8.12) from being called for this envelope. But note that xxfi_close() will still be called when the connection is closed.

8.9.2. An Example of xxfi_eoh()

To illustrate one use for an xxfi_eoh() handler function, consider the need to reject a message that lacks a Message-Id: header. In the following example, we presume that the earlier xxfi_header() handler function may have set a flag if it found a Message-Id: header.

 1  sfsistat
 2  xxfi_eoh(SMFICTX *ctx)
 3  {
 4    struct priv_struct *priv;
 5
 6    priv = smfi_getpriv(ctx);
 7    if (priv == NULL)
 8      return SMFIS_TEMPFAIL;
 9
10    if (! priv->hasMID)
11    {
12      /* ... perform envelope deallocation cleanup here */
13      return SMFIS_REJECT;
14    }
15    /* ...other post-header review from here down */
16    return SMFIS_CONTINUE;
17  }

Here, we start the xxfi_eoh() handler function (line 4) by declaring a variable (priv) that holds as its value the address of our private data that was set earlier with smfi_setpriv() (see section 7.3.2). The smfi_getpriv() milter-library routine (see section 7.3.3) is called to fetch the pointer to our private data. If our private data does not exist (line 7), we return SMFIS_TEMPFAIL.

Next (line 10), the value of priv->hasMID is tested. If an earlier screening of headers with xxfi_header() discovered that a Message-Id: header was present, then the value in this variable will be trUE; otherwise, it will be FALSE. If it is FALSE, this message is rejected (line 13). Note (line 12) the need to perform envelope-specific deallocation and cleanup here, because message rejection prevents xxfi_eom() and xxfi_abort() from being called. However, if you can safely defer envelope-specific deallocation until connection close, then xxfi_close() will still be called. But as a general principle, you should clear and restart envelope-specific data inside xxfi_envfrom(), and then deallocate the last envelope and the connection inside xxfi_close().

8.9.3. Other Uses for xxfi_eoh()

Following are other uses for the xxfi_eoh() handler function.

Flagging missing headers so that a later call to xxfi_eom() (see section 8.11) can add them to the message

Comparing the number of envelope recipients to the number of header recipients and basing a decision on the ratio

Logging a statistical review of the headers in a message

8.10. xxfi_body Reviews Each Body Chunk

After all recipients have been reviewed, the data portion of the message is reviewed. Recall that the data portion is composed of two parts: the message headers and the message body. These two parts are separated by a single blank line. The headers are always first, and the message body is always second. Each Milter reviews the data portion (header and body) before the next Milter is allowed to review the data portion.

The xxfi_body() handler function is called after the xxfi_eoh() handler function (see section 8.9) but before the xxfi_eom() handler function (see section 8.11). Each time sendmail gathers a chunk of the message body, it passes that chunk to this xxfi_body() function, which can be called multiple times for a single envelope, once per body chunk. Note that no end-of-body semaphore is passed to xxfi_body(). Instead, after the entire body has been passed, the xxfi_eom() routine is called.

8.10.1. How to Use xxfi_body()

The xxfi_body() handler function is called this way:

sfsistat
xxfi_body(SMFICTX *ctx, unsigned char *chunk, size_t len)

The return type is sfsistat, and thus xxfi_body() can return any of the values shown earlier in Table 8.2. We discuss the effect of each of the possible return values shortly.

The xxfi_body() handler function takes three arguments. The first is the usual ctx private-context pointer (see section 8.1.2), which defines the connection context that is being handled.

The second argument is the address of a chunk of data of type unsigned char*. Note that this chunk of data is not a C language string (it does not end with a zero-value byte and thus may contain zero-value bytes), so you must not use C language string routines on it without first performing the appropriate conversion.

Line-oriented data in each chunk will have each line terminated with the normal carriage-return/linefeed combination. But note that any given line can begin in one chunk but end in the next chunk. For this reason you should plan to concatenate chunks together before screening for spam, or else maintain state inside your screening routines.

The third argument (len) is a count of the number of bytes in the chunk. Note that len can be zero, but zero does not indicate end of body.[15]

[15] . Although the chunk size will likely never be zero, defensive programming dictates that you should anticipate a zero size and do nothing foolish.

For MIME-encoded messages, the body can contain multiple MIME parts, each separated by a "boundary" and each with its own header/body section. The handling of MIME is up to your Milter; sendmail does not partition MIME parts into chunks.

Before you can use an xxfi_body() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_body() handler function's position in that structure includes the name of a function, the function named will be called each time a body chunk needs to be reviewed:[16]

[16] Recall that we only use xxfi_body() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   xxfi_eoh,        /* end of headers (see section 8.9) */
   xxfi_body,       /* this function reviews body chunks */

If the position of the xxfi_body() handler function in the smfiDesc structure is NULL, sendmail will not call the Milter to perform post-header processing:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the RCPT TO (see section 8.8) */
   xxfi_eoh,        /* end-of-headers (see section 8.9) */
   NULL,            /* don't call this milter to review body chunks */

Remember that the value returned by the xxfi_body() handler function is of type sfsistat. Table 8.9 shows the effect of each of the possible return values.

Table 8.9. Effect of Return Values for xxfi_body()

Return Value

Effect on sendmail

SMFIS_ACCEPT

The current body chunk under review marks the message as acceptable to the Milter, so the message (envelope) is accepted. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope for body chunk reasons.

SMFIS_REJECT

Reject the current DATA phase with a 5yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_TEMPFAIL

Temporarily reject the DATA phase with a 4yz SMTP reply code. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter may be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_DISCARD

The current envelope is discarded. The Milter will not be contacted again for this envelope (but will be contacted again when the connection is closed). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_CONTINUE

Acknowledge the information received. The Milter will be contacted with other requested information when and if that information becomes available. In the event that the body entry in the smfiDesc structure is NULL, this return code becomes the default.

If your Milter accepts, rejects, defers, or discards the envelope because of something found in the message body, you must remember to deallocate envelope-specific resources at the same time. Remember that this envelope rejection will prevent both your xxfi_eom() function (see section 8.11) and your xxfi_abort() function (see section 8.12) from being called for this envelope. In the event that you can safely deallocate upon connection close, you can defer envelope-specific deallocation until xxfi_close() is called.

8.10.2. An Example of xxfi_body()

To illustrate one use for an xxfi_body() handler function, consider the need to reject a message in which any body chunk contains too many characters with the high bit set.

 1  sfsistat
 2  xxfi_body(SMFICTX *ctx, unsigned char *chunk, size_t len)
 3  {
 4    unsigned char *cp, *ep;
 5    size_t hicount = 0;
 6
 7    if (len < 2)
 8      return SMFIS_CONTINUE;
 9    cp = chunk;
10    ep = chunk +len;
11    while (cp < ep)
12    {
13      if ((*cp++ & 0x80) != 0)
14        ++hicount;
15    }
16    if (hicount > (len / 2))
17    {
18      /* ... deallocate envelope resources here */
19      return SMFIS_REJECT;
20    }
21    return SMFIS_CONTINUE;
22  }

Here, we start the xxfi_body() handler function (line 4) by declaring two pointers (cp and ep) that will be used to scan through the body chunk. A counter (hicount) is declared (line 5) to count the number of high-bit-set bytes and is initialized to begin at zero. If the length of the chunk (in len) is less than 2 (line 7), then we return SMFIS_CONTINUE.

The first pointer (cp) is set to point at the first byte of the chunk (line 9), and the second pointer (ep) is set to point just after the last byte of the chunk (line 10). Then, in a loop (line 11), the byte pointed to by cp is checked (line 13) to see if it has the high bit set. If it does, the counter hicount is incremented (line 14). The pointer cp is also incremented (line 13) so that the next byte will be examined the next time through the loop.

After all the bytes have been examined, a check is made to see whether the count of high-bit-set bytes (hicount) is greater than half the total number of bytes (line 16). If it is, the message is rejected by returning SMFIS_REJECT (line 19); otherwise, the chunk is allowed by returning SMFIS_CONTINUE (line 21). Note (line 18) that we deallocate envelope resources before returning SMFIS_REJECT.

8.10.3. Other Uses for xxfi_body()

Following are other uses for the xxfi_body() handler function.

Saving (buffering) the body to a file or in memory so that xxfi_eom() can later rewrite it. This could be useful, for example, to MIME-decode the body for later Milters.

Screening the body for viruses, unacceptable website references, spam-indicating keywords, and the like.

Saving (buffering) all but attachments so that xxfi_eom() can later rewrite the body with attachments omitted.

Allowing only selected local recipients to receive messages over a maximum size, or deferring delivery of large messages until after normal business hours.

Archiving a copy of all outbound email for conformance or other reasons.

Storing attachments in a database and replacing each with a reference to its database location.

8.11. xxfi_eom Reviews at End of Envelope

The sending of the data portion of an SMTP message begins with the sending site issuing the DATA command and sendmail replying that it will accept multiple lines of data in return. The sending site sends that data and, when it is finished sending, sends a dot on a line by itself to signal that it is finished.

Recall that the data portion is composed of two parts: the message headers and the message body. Each Milter reviews the data portion (header and body) before the next Milter is allowed to review the data portion.

After sendmail receives the concluding dot from the sending site but before sendmail acknowledges that dot, the xxfi_eom() handler function is called. Thus, the xxfi_eom() function is in a unique position to review the entire envelope. When xxfi_eom() is called, the envelope has already had

The MAIL FROM envelope sender reviewed by xxfi_envfrom() (see section 8.6).

The RCPT TO envelope recipients reviewed by xxfi_envrcpt() (see section 8.7).

The DATA headers reviewed by xxfi_header() (see section 8.8) and xxfi_eoh() (see section 8.9).

The DATA body reviewed by xxfi_body() (see section 8.10).

The xxfi_eom() function is called only once per envelope, after the xxfi_body() function. But note that if the envelope was earlier rejected, tempfailed, or discarded by sendmail or another Milter, this xxfi_eom() function will not be called. Instead, the xxfi_abort() function will be called. That is, either xxfi_eom() or xxfi_abort() will be called, but not both.

Also note that if this Milter accepted, rejected, deferred, or discarded this envelope itself, then neither this xxfi_eom() function (see section 8.11) nor the xxfi_abort() function (see section 8.12) will be called for this envelope.

The xxfi_eom() function is also unique among all the xxfi handler functions, because it is the only function from which you are allowed to do the following:

Add, remove, and modify message headers (using the smfi_addheader() [see section 7.4.2] and smfi_chgheader() [see section 7.4.4] milter-library routines)

Delete, modify, or replace the message body (using the smfi_replacebody() [see section 7.4.7] milter-library routine)

Add and remove envelope recipients (using the smfi_addrcpt() [see section 7.4.5] and smfi_delrcpt() [see section 7.4.6] milter-library routines)

Quarantine a message (using the smfi_quarantine() [see section 7.4.9] milter-library routine)

Call the smfi_progress() [see section 7.4.8] milter-library routine to delay the timeout

Note that despite all the xxfi_eom() function can do, it is constrained by two time limits. One time limit is imposed by sendmail when its configuration file is set up (see section 3.3) to work with the Milter (the default limit is 5 minutes), and the second is imposed by the sending site, which is waiting for acknowledgment of its dot.

You determine the amount of time sendmail will wait by using the T parameter when you first declare your Milter:

MAIL_FILTER(`MyMilt´, ``S=inet6:999@localhost, T=E:2m´´)

Here, because we are running three Milters (we show only one), we set the timeout for the end-of-message phase to be 2 minutes, and this means that MyMilt must complete all the activities of its xxfi_eom() function within two minutes. If MyMilt needs more than two minutes (perhaps because of an overly large or complex message), it can request more time from sendmail by calling the smfi_progress() milter-library routine (see section 7.4.8).

The amount of time the sending site will wait for acknowledgment of its dot can vary. Some sites may wait for as little as five or ten minutes. Others will wait for an hour (sendmail's default).

Remember that any given Milter knows nothing about any other Milter or Milters. It doesn't know whether it is the only Milter that can take time or whether it is competing for time with other Milters. Therefore, you must strive to design each Milter to use as little time as necessary to perform all its end-of-message tasks.

8.11.1. End-of-Envelope Cleanup

Recall that a connection may contain multiple envelopes and that an envelope extends from the MAIL FROM envelope sender declaration to the end-of-message dot.

In general, envelope-specific resources that are created (memory allocated or files opened) during an envelope's processing should be deallocated during this end-of-message processing; or during xxfi_abort() if that is called instead; or as part of any envelope handler function if the current Milter accepted, rejected, deferred, or discarded the message itself.

If you fail to deallocate resources for one envelope, very likely you will never get a chance to deallocate them later, and failure to deallocate memory can create memory leaks that can slowly gobble up all available memory. Failure to close open files can result in an eventual exhaustion of available file descriptors.

All of your private connection information (envelope and connection) is contained in the private data portion of the ctx pointer. You fetch that private data by using the smfi_getpriv() milter-library routine (see section 7.3.3). For example:

struct my_priv_struct *priv;

priv = smfi_getpriv(ctx);

/* ... deallocate variables and close files in priv */

(void) smfi_setpriv(ctc, NULL);

After this, you should clean up by freeing allocated memory and closing open files that may be contained in that private data. Then (if all you are storing is envelope data) you call the smfi_setpriv() milter-library routine (see section 7.3.2) to store NULL, thus removing knowledge of your envelope-specific private data. But if your private data includes connection data, you should either skip this last step and defer it until your xxfi_close() function (see section 8.13), or just deallocate the envelope information and leave the connection information intact.

The details of how you clean up after your envelope processing will vary depending on the logic of your code. The underlying lesson here is simply to be certain that you clean up after yourself.

8.11.2. xxfi_eom and sendmail Macros

Beginning with sendmail V8.13, the xxfi_eom() handler function has access to selected sendmail macros in addition to its function arguments. With the default sendmail configuration, the following macro can be fetched, and its value used, by calling the smfi_getsymval() milter-library routine (see section 7.3.1):

The ${msg_id} macro holds the value of the Message-Id: header.

Another macro can also be of use to the xxfi_eom() function:

The ${nbadrcpts} macro (new with sendmail V8.13) holds a count of the number of bad recipients. This count includes those recipients rejected by sendmail and those rejected by this and other Milters.

You can add macros to the default macros given to xxfi_eom() by using an mc configuration-file line such as the following (new with sendmail V8.13):

define(`confMILTER_MACROS_EOM´, confMILTER_MACROS_EOM``, {nbadrcpts}´´)

Note the use of single-quote pairs, which are required because of the quoted comma.

8.11.3. How to Use xxfi_eom()

The xxfi_eom() handler function is called this way:

sfsistat
xxfi_eom(SMFICTX *ctx)

The return type is sfsistat, and thus xxfi_eom() can return any of the values shown earlier in Table 8.2. We will discuss the effect of each of these possible return values shortly.

The xxfi_eom() handler function takes a single argumentthe usual ctx private-context pointer (see section 8.1.2)which defines the connection context currently being handled.

Before you can use an xxfi_eom() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_eom() handler function's position in that structure includes the name of a function, the function named will be called at the end of the envelope (unless xxfi_abort() is called instead):[17]

[17] Recall that we only use xxfi_eom() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   xxfi_eoh,        /* end-of-headers (see section 8.9) */
   xxfi_body,       /* the body (see section 8.10) */
   xxfi_eom,        /* this function reviews end of message */

If the position of the xxfi_eom() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter to perform end-of-message processing:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   xxfi_eoh,        /* end of headers (see section 8.9) */
   xxfi_body,       /* the body (see section 8.9) */
   NULL,            /* don't call this milter to review end of message */

The value returned by the xxfi_eom() handler function is of type sfsistat. Table 8.10 shows the effect of each of the possible return values.

Table 8.10. Effect of Return Values for xxfi_eom()

Return Value

Effect on sendmail

SMFIS_ACCEPT

Accept the message. The Milter will not be contacted again for this envelope (but will be contacted at connection close). The Milter can be contacted for the next envelope if there is one on this same connection. A subsequent Milter can still reject the envelope.

SMFIS_REJECT

Reject the message. The Milter will not be contacted again for this envelope (but will be contacted at connection close). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_TEMPFAIL

Temporarily reject the message. The Milter will not be contacted again for this envelope (but will be contacted at connection close). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_DISCARD

Discard the message. The Milter will not be contacted again for this envelope (but will be contacted at connection close). The Milter can be contacted for the next envelope if there is one on this same connection. Subsequent Milters will not get a chance to review this envelope.

SMFIS_CONTINUE

The same as SMFIS_ACCEPT.

8.11.4. An Example of xxfi_eom()

To illustrate one use for an xxfi_eom() handler function, consider the need to add a header that shows this Milter reviewed the message and to deallocate prior envelope information.

 1  sfsistat
 2  xxfi_eom(SMFICTX *ctx)
 3  {
 4    struct my_priv_struct *priv;
 5    static char *name = "X-Milter";
 6    static char *value = "MyMilter";
 7
 8    (void) smfi_addheader(ctx, name, value);
 9    priv = smfi_getpriv(ctx);
10    if (priv != NULL)
11    {
12      (void) free(priv->envfrom);
13      (void) free(priv);
14      (void) smfi_setpriv(ctx, NULL);
15    }
16    return SMFIS_CONTINUE;
17  }

Here, the xxfi_eom() handler function begins (line 4) by declaring a pointer (priv) that will hold the address of the private data structure. Two static pointers are then declared that contain the name (line 5) and value (line 6) portions of the header that will be added.

The header is added (line 8) by calling the smfi_addheader() milter-library routine (see section 7.4.2) and ignoring any errors. Note that we really don't care if this Milter fails to add that header.

The address of the private data (here, it contains only envelope data and not connection data) is fetched (line 9) and that address is saved in the priv pointer. If that address isn't NULL (line 10), then the allocated item in it is freed (line 12). The smfi_setpriv() milter-library routine is then used to store NULL as the private data value (line 14).

8.11.5. Other Uses for xxfi_eom()

Following are other uses for the xxfi_eom() handler function.

Logging a summary of everything that the Milter did with the envelope

Adding headers found to be missing

Changing the value of selected headers, such as the value of a custom tracking header

Removing junk headers sometimes added by spam software

Screening the entire body earlier saved by xxfi_body() to detect spam, viruses, and unwanted attachments

Decoding (such as base64-decoding) the body for later Milters

Adding an envelope recipient, perhaps to enable a fax or archive copy

Removing an envelope recipient, such as a printer or an internal-only mailing list address

8.12. xxfi_abort Handles Envelope Abort

A connection can contain one or more envelopes. Processing of each envelope ends either with a call to xxfi_eom(), so that the Milter can conclude processing of the current envelope, or with a call to the xxfi_abort() function. Note that one or the other will be called, but never both.

The xxfi_abort() handler function is called if envelope processing by this Milter is interrupted by an outside process (such as sendmail or another Milter). Another Milter may have rejected the envelope because of a bad header, or sendmail may have rejected the envelope because of a bad envelope recipient. If the envelope was rejected, deferred, or discarded by an outside process, then the xxfi_abort() function will be called instead of the xxfi_eom() function.

But remember that if the envelope is accepted, rejected, discarded, or deferred by the current Milter, then neither xxfi_eom() nor xxfi_abort() will be called.

Finally, remember that the envelope begins with the calling of the xxfi_envfrom() handler function. If sendmail or an earlier Milter had rejected the envelope sender, then the current Milter's xxfi_envfrom() would not be called, so no envelope would be started. In this instance, xxfi_abort() would not be called.

8.12.1. How to Use xxfi_abort()

The xxfi_abort() handler function is called this way:

sfsistat
xxfi_abort(SMFICTX *ctx)

The return type is sfsistat, and thus xxfi_abort() can return any of the values shown earlier in Table 8.2. But remember that xxfi_abort() is called only if the message was rejected, deferred, or discarded by an outside process, so this return value will be ignored.

The xxfi_abort() handler function takes a single argumentthe usual ctx private-context pointer (see section 8.1.2)which defines the connection context that is being handled. Note, however, that xxfi_abort() is not given a reason for the abort.

Before you can use an xxfi_abort() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_abort() handler function's position in that structure includes the name of a function, the function named will be called if the processing of the envelope is aborted:[18]

[18] Recall that we only use xxfi_abort() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   xxfi_eoh,        /* end of headers (see section 8.9) */
   xxfi_body,       /* the body (see section 8.10) */
   xxfi_eom,        /* end of message (see section 8.11) */
   xxfi_abort,      /* this function handles envelope abort */

If the position of the xxfi_abort() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter to handle an abort caused by an outside process:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the RCPT TO (see section 8.8) */
   xxfi_eoh,        /* end of headers (see section 8.9) */
   xxfi_body,       /* the body (see section 8.10) */
   xxfi_eom,        /* end of message (see section 8.11) */
   NULL,            /* don't call this milter to handle envelope abort */

8.12.2. An Example of xxfi_abort()

To illustrate one use for an xxfi_abort() handler function, consider the need to close an open file.

 1  sfsistat
 2  xxfi_abort(SMFICTX *ctx)
 3  {
 4    struct my_priv_struct *priv;
 5
 6    priv = smfi_getpriv(ctx);
 7    if (priv != NULL)
 8    {
 9      if (priv->fp != NULL)
10      {
11        (void)fclose(fp);
12        priv->fp = NULL;
13      }
14    }
15    return SMFIS_CONTINUE;
16  }

Here, the xxfi_abort() handler function begins (line 4) by declaring a pointer that will hold the address of the private data.

First, the smfi_getpriv() milter-library routine (see section 7.3.3) is called to fetch the address of the private data. If that address is NULL (line 7), then the closing of the file is skipped.

If the file pointer for the open file is not NULL (line 9), then the file is closed (line 11) and the file pointer is set to be NULL (line 12).

Note that we don't free priv itself because, in this case, it also contains connection data that will be freed in the xxfi_close() handler function (see section 8.13).

8.12.3. Other Uses for xxfi_abort()

Following are other uses for the xxfi_abort() handler function.

Recording that this Milter was aborted by the action of an outside process

Calling a common subroutine to deallocate envelope data

8.13. xxfi_close Handles Connection Cleanup

Between the time the sending site connects to sendmail and the time the sending site concludes its connection (by giving the SMTP QUIT command), multiple envelopes may have been processed. Sending sites tend to keep a connection open for efficiency when more than one envelope is destined for an MTA. It is more expensive to set up and break down connections than it is to send an SMTP command notifying the receiving sendmail that another envelope is on the way.

Between the time sendmail receives the QUIT command and the time sendmail acknowledges that command, the xxfi_close() handler function is called. It is called for notification only, because the connection is already closing, and thus xxfi_close() may not return any meaningful status-changing values.

Note that xxfi_close() is called whenever the connection is closed. Connections can terminate because of SMTP errors, network errors, and Milter-caused early termination. Whenever the connection is closed, the xxfi_close() handler function is called.

8.13.1. End-of-Connection Cleanup

In general, connection-specific resources created during a connection's processing (memory allocated or files opened) should be deallocated during this end-of-connection processing.

If you fail to deallocate resources at this time, very likely you will never get a chance to deallocate them later. Failure to deallocate memory can create memory leaks that can slowly gobble up all available memory. Failure to close open files can result in an eventual exhaustion of available file descriptors.

8.13.2. How to Use xxfi_close()

The xxfi_close() handler function is called this way:

sfsistat
xxfi_close(SMFICTX *ctx)

The return type is sfsistat, and thus xxfi_close() can return any of the values shown earlier in Table 8.2. But remember that xxfi_close() is called only when the connection is in the process of being closed, so its return value makes no difference.

The xxfi_close() handler function takes a single argumentthe usual ctx private-context pointer (see section 8.1.2)which defines the connection being handled.

Before you can use an xxfi_close() handler function, you must first list it in an smfiDesc structure that is declared with the smfi_register() milter-library routine (see section 7.2.1). If the xxfi_close() handler function's position in that structure includes the name of a function, the function named will be called when the processing of the connection is finished:[19]

[19] Recall that we only use xxfi_close() to parallel the sendmail documentation. You can, of course, name your function whatever makes sense to you.

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the headers (see section 8.8) */
   xxfi_eoh,        /* end of headers (see section 8.9) */
   xxfi_body,       /* the body (see section 8.10) */
   xxfi_eom,        /* end of message (see section 8.11) */
   xxfi_abort,      /* abort the envelope (see section 8.12) */
   xxfi_close,      /* this function handles closing the connection */

If the position of the xxfi_close() handler function in the smfiDesc structure is NULL, then sendmail will not call the Milter to handle the close of the connection:

struct smfiDesc smfilter =
{
   "MyMilter",      /* filter name (see section 7.2.1) */
   SMFI_VERSION,    /* a constant (see section 7.2.1) */
   SMFIF_ADDHDRS,   /* flags (see section 7.2.2) */
   xxfi_connect,    /* the connection (see section 8.4) */
   xxfi_helo,       /* the HELO/EHLO (see section 8.5) */
   xxfi_envfrom,    /* the MAIL FROM (see section 8.6) */
   xxfi_envrcpt,    /* the RCPT TO (see section 8.7) */
   xxfi_header,     /* the RCPT TO (see section 8.8) */
   xxfi_eoh,        /* end of headers (see section 8.9) */
   xxfi_body,       /* the body (see section 8.10) */
   xxfi_eom,        /* end of message (see section 8.11) */
   xxfi_abort,      /* abort the envelope (see section 8.12) */
   NULL,            /* don't call this milter to handle close of the connection */

8.13.3. An Example of xxfi_close()

To illustrate one use for an xxfi_close() handler function, consider a need to log the total duration of the connection and the number of envelopes processed during that connection.

 1  sfsistat
 2  xxfi_close(SMFICTX *ctx)
 3  {
 4    struct my_priv_struct *priv;
 5    time_t now;
 6    char *ip;
 7
 8    priv = smfi_getpriv(ctx);
 9    if (priv == NULL)
10      return SMFIS_CONTINUE;
11    if (priv->ipnum == NULL)
12      ip = strdup("0.0.0.0");
13    else
14      ip = strdup(priv->ipnum);
15
16    /* priv->starttime and priv->ipnum both
17     * set in xxfi_connect. priv->numenvelopes
18     * incremented each xxfi_eom().
19     */
20    (void)time(&now);
21    syslog(LOG_INFO, "%s: %ld seconds, %d envelopes",
22      ip, now - priv->starttime,
23      priv->numenvelopes);
24
25    /* Clean up the allocated data */
26    if (priv->ipnum != NULL)
27      (void)free(priv->ipnum);
28    (void)free(priv);
29    (void)setpriv(ctx, NULL);
30    return SMFIS_CONTINUE;
31  }

Here, the xxfi_close() handler function begins (line 4) by declaring a pointer that will hold the address of the private data. A variable that will hold the current time (now) is also declared.

First, the smfi_getpriv() milter-library routine (see section 7.3.3) is called (line 8) to fetch the address of the private data. If that address is NULL (line 9), there is nothing to do.

If the text representation of the IP address of the connecting host is NULL (line 11), we record the IP number as 0.0.0.0 (line 12). Otherwise, we use the specified IP address.

The time(3) C library routine is called (line 20) to get the current time in seconds.

A summary is logged using syslog(3) (line 21) showing the IP number, how long it took to complete the connection, and how many envelopes were transmitted during the connection.

Because this is the last time access is available to the private data, that data must be deallocated at this time. First the allocated string containing the IP number is deallocated if not NULL (line 27), and then the private data structure itself is deallocated (line 28). Then, the record for the private data is set to NULL by calling the smfi_setpriv() milter-library routine (line 29).

This routine returns SMFIS_CONTINUE but could return anything because its return value means nothing.

8.13.4. Other Uses for xxfi_close()

Following are other uses for the xxfi_close() handler function.

Summarizing all actions taken by the connecting site for later statistical use

Ensuring that there is no allocated envelope data that still needs to be deallocated

Part IV: Nuts and Bolts

In this final part of the book we show you all the pieces that go into writing a Milter. When you finish this part, you will be able to design and implement very sophisticated Milters of your own.

Chapter 9, Milters and the Environment, examines how to code in relationship to your operating system's environment. Topics range from managing signals to architecting your Milter.

Chapter 10, User and Temporal Feedback, shows how to incorporate feedback into your Milter to maintain a history and accommodate user preferences.

Chapter 11, Handy Routines for the Message Body, provides a smorgasbord of routines for handling the message body. Routines range from base64 decoding to type identification using /etc/magic.

Chapter 9. Milters and the Environment

Milters do not exist in a vacuum. As you begin to develop your first Milter, you will find that you need to interact with the operating system, handle disk I/O, perform database updates, and provide user feedback.[1] In this chapter we illustrate useful techniques for interacting with this environment:

[1] Milters are also multithreaded. We don't cover multithreading in this book, but you can find many works on this subject by Googling "Pthread programming" and "posix thread programming."

Section 9.1 discusses the advantages of the various locations where Milter software can run.

Section 9.2 explains why a Milter should not run as root and provides alternatives.

Section 9.3 provides sample scripts for starting and stopping your Milters in relation to sendmail.

Section 9.4 shows how to run your Milter in the background as a daemon.

Section 9.5 shows which signals the Milter traps and which are available for your use.

Section 9.6 shows why a Milter must recognize its own MX servers.

Section 9.7 discusses ways to log your Milter's activities.

Section 9.8 describes the advantages of autoconf.

Section 9.9 shows how to avoid memory leaks in a Milter.

Each of these sections has code that will accomplish each task. These code fragments, where possible, are presented as subroutines so that you can test each in isolation before using it directly inside your Milter. For example, you can test the mx() routine (see section 9.6) by adding a simple main():

int
main(argc, argv)
        int argc;
        char **argv;
{
        if (argc > 1)
                mx(argv[1]);
        return 0;
}

Here, the mx() subroutine is called with whatever was passed on the command line as the program's single argument. Because the mx() subroutine looks up MX records for hosts, you should specify a host name on the command line. For example:

% ./a.out example.com
example.com. IN 3600 MX 100 mx1.example.com
example.com. IN 3600 MX 100 mx2.example.com

9.1. Where to Run Your Milter

When your Milter first starts, it should change directory either into a directory specially set up to accept core dumps (such as /var/run/yourmilter) or into the directory that contains the Milter's configuration and other files. If your Milter is entirely command-line driven and lacks database or other files, the choice to run in /var/run/yourmilter may be best. In the following, replace yourmilter with the name of your Milter:

1  if (chdir("/var/run/yourmilter") < 0)
2  {
3          (void) fprintf(stderr,
4       "milter: /var/run/yourmilter: %s\n",
5                 strerrno(errno));
6          exit(EX_OSFILE);
7  }

Here, if the chdir call fails, the reason for the error is printed (with strerror) and the Milter exits EX_OSFILE (line 6). Note that Milters should include <sysexits.h> and exit with the codes from that file. Those are the same exit codes used by sendmail so that the enclosing script can report mail-specific errors.

If your Milter uses a configuration file or if it uses a database or other status or startup files, all those files should be collected into a single place for ease of management. Two standards exist for mail-related software such as Milters. One places them in a special subdirectory under the /etc/mail directory (used by sendmail). Another places them in a "local" directory such as /usr/local or /opt.[2]

[2] FreeBSD recommends you place database files under /var/db, Processor ID (PID) files under /var/run, and UNIX domain sockets under /var/run. We instead recommend you keep all files related to a single Milter together under one common directory.

9.1.1. Run under /etc/mail

If you expect to install only a single Milter, run it in its own subdirectory directly under the /etc/mail directory.[3] In the following, for example, the name of our Milter is ourmilt:

[3] This is a bad expectation because once you get a taste for Milters, you will want to write and install more of them.

1  char *home = "/etc/mail/ourmilt";
2
3  if (chdir(home) < 0)
4  {
5          (void) fprintf(stderr, "ourmilt: %s: %s\n",
6                     home, strerrno(errno));
7          exit(EX_OSFILE);
8  }

Here, if the chdir call fails, the reason for the error is printed (with strerror) and the Milter exits. For this example, the directory to use is hard-coded. In your implementation, you may prefer to define the directory with a preprocessor #define directive or to parse it from a configuration file or the command line.

If you expect to run more than a single Milter, it might be better to run it under a general directory called /etc/mail/milters:

char *home = "/etc/mail/milters/ourmilt";

The location under /etc/mail that you choose is entirely up to you. It makes no difference to sendmail. The only Milter path sendmail cares about is the location of a Milter's UNIX domain socket (if one is used). If your Milter uses a UNIX domain socket, that socket should also reside in your Milter's home.

9.1.2. Run in /usr/local

Another common location for software is the /usr/local directory. As with /etc/mail, /usr/local can be set up either with a single directory for your single Milter, or with a general directory for all Milters:

char *home = "/usr/local/ourmilt";            Single Milter
char *home = "/usr/local/milters/ourmilt";    Many Milters

But no matter where you decide to locate your Milter or Milters, try to keep the location flexible so that you can easily change it during development or when your Milter is later displaced because of encroaching, inflexible commercially bought Milters.

9.2. Your Milter's User ID

First, note that you should never run a Milter as root. This is because it is very difficult to design a program that can run safely as root and because any errors when running as root can change or compromise your system. Instead of wasting your time by writing bulletproof code so that a Milter can run as root, decide ahead of time whom else it may run as. In this way, you can write much simpler code and still keep it secure.

Second, note that other programs also run as non-root, so they may have coopted preexisting user identities. Also note that system software uses many non-root identities. Thus, in general, it is best to avoid using the following identities:

Any user ID less than 100 because these are reserved for the system

Any user ID range reserved for actual users

nobody, noaccess, etc.

User IDs reserved for software (such as mysql, http, ftp, smmsp, etc.)[4]

[4] You might imagine that smmsp would be OK to use, because it is used by sendmail. But we discourage that overlapping use and, instead, encourage you to create a new user specially named for use with your Milter.

If, after you exclude these, there is no existing identity that is useful for your Milter, you will have to create a new one.

9.2.1. Prevent Running as root

A good precaution is to write a small bit of code to prevent your Milter from running as root. In the two sections that follow, we will expand this code to allow your Milter to run as a special user and group.

There are two ways a program can be run as root. First, the root user (or a system-level program) can execute the program, in which case it has a real user ID that is root's. Or it can be run by anyone (if the set user ID permission is set so that no matter who runs it runs as root), in which case its effective user ID is root's.

A correctly written routine to prevent running as rootwhether your Milter is run by root or with set user ID root permissionsmight look like this:

1  if (geteuid() == 0 || getuid() == 0)
2  {
3          (void)fprintf(stderr,
4                "ourmilt: Root execute forbidden\n");
5          exit(EX_NOPERM);
6  }

The geteuid function returns the effective user ID for your Milter. If that effective user ID is zero, your program was run with set user ID root permissions.

The getuid function returns the real user ID for your Milter. If that real user ID is zero, your program was run by root or by a program effectivly run as root.

In either instance, your Milter will print an error message and exit. Naturally this check should be made before your Milter opens any files or devices for writing or reads any file other than its configuration file. It is also wise to avoid coding any potentially harmful actions before this check.

9.2.2. Run as Your Own User

If your Milter has been run as root, you may prefer to change its real and effective user IDs to a non-root user and continue to run rather than exit. The following runas() function will do just that. But remember: It will work only if your Milter was run by, or as, root.

 1  int
 2  runas(user)
 3     char *user;
 4  {
 5     uid_t uid = 0;
 6
 7     if (user == NULL)
 8       return EINVAL;
 9
10     if (isdigit((int)*user))
11     {
12        uid = atoi(user);
13        if (uid < 0 || uid >= INT_MAX)
14           return ERANGE;
15     }
16     else
17     {
18        struct passwd *pw;
19
20        pw = getpwnam(user);
21        if (pw == NULL)
22           return EINVAL;
23        uid = pw->pw_uid;
24     }
25     if (uid == 0)
26        return EACCES;
27
28          if (setgroups(0, NULL) == -1)
29                  return errno;
30
31     if (setuid(uid) < 0 || seteuid(uid) < 0)
32        return errno;
33
34     return 0;
35  }

In this code sample, we presume that the variable user (line 2) is set outside this function, perhaps in a configuration file, as part of the command line, or as a default with a #define statement elsewhere in the code. If the value in that variable begins with a digit (line 10), we also presume it is numeric and save that value as the user ID into which to transform (line 12). Otherwise, we presume the value in user is a string that is the name of a user (such as "milter"). We look up that name in the system password database (line 20) and use the user ID found there (line 23). If the user name could not be found, we return an error (line 22). If the user specified was root, we also return an error (line 26).

If the user ID specified was nonzero (not root), we clear any list of groups that root belonged to (line 28) and change our real user ID and effective user ID into the value specified (line 31).[5] If either change fails, an error is returned (line 32).

[5] In addition, make sure that any file descriptors opened by root are closed (including standard input, standard output, and standard error) to avoid I/O redirection attacksanother reason to never enable set user ID root permissions on a Milter.

This runas() function can be tested by adding a small main() such as the following:

int
main(argc, argv)
        int argc;
        char **argv;
{
        if (argc > 1 && (errno = runas(argv[1])) != 0)
                perror(argv[1]);
        return 0;
}

Note that runas() only resets the user ID. If your Milter depends on a group ID, you may also need to reset the group ID with similar code. But even if your Milter does not depend on a group ID, you should reset the group ID to something benign in order to minimize risk.

9.3. How to Start and Stop Your Milter

One way to start your Milter is to copy a startup script from an existing rc.d directory and then edit it to suit your needs. For FreeBSD, copy (or create) a file in the /usr/local/etc/rc.d directory; on Solaris or Linux, copy (or create) it in the /etc/init.d directory. In general, try to start and stop only one Milter per file, thereby allowing you to deal with Milters individually. See your operating system's documentation for the details of how to install a boot-time script.

Commands to start a Milter in those files might look like the following, where the name of the Milter is ourmilt:

if [ -f /etc/mail/milters/ourmilt.conf ]
then
        /etc/mail/milters/ourmilt &
        echo "milter: ourmilt started"
fi

The disadvantage of this approach is that it does not take into account the likelihood that your Milter will fail and abort periodically. Remember that Milters are multithreaded, and writing bulletproof multithreaded code is difficult at best. No matter how careful you have been, your Milter is bound to fail every once and a while during its initial development.

It is better to run your Milter from a script that can monitor its life and that can restart your Milter should it ever fail. To accomplish that, we need to use two scripts. The first script sets up the environment and runs the second script. The second script runs the Milter and monitors its life.

But note that this approach will work only if your Milter does not put itself into the background (see section 9.4). If it does, add a command-line switch to turn off that ability so that this pair of scripts can run. Later, after your Milter proves itself reliable, you can replace this pair of scripts with a single, much simpler script that allows your Milter to put itself into the background.

In the following shell script we use shell macros to configure everything, so you can easily adapt this script to any Milter of your own design.

 1  #!/bin/sh
 2  #############################################
 3  # MILTER.INIT
 4  #       A script that starts, stops a milter.
 5  #
 6  # This script may be installed in /etc/init.d
 7  # under Solaris and the like. Or used as
 8  # an /etc/rc-style startup script.
 9  #############################################
10
11  export MILTER MILTERDIR MILTERARGS MILTERSLEEP MILTERSEMAPHORE
12
13  MILTER=ourmilt
14  MILTERDIR=/etc/mail/milters
15  MILTERARGS=
16  MILTERSLEEP=5
17  MILTERRUN=our.run
18  MILTERSEMAPHORE=ourmilt.semaphore
19
20  MILTERKILL="/usr/bin/pkill -x ${MILTER}"
21  #MILTERKILL="/usr/bin/killall ${MILTER}"
22
23  case "$1" in
24     'start')
25     if [ -f ${MILTERDIR}/${MILTER} ]
26     then
27       touch ${MILTERDIR}/${MILTERSEMAPHORE}
28       ${MILTERDIR}/${MILTERRUN} &
29       echo ${MILTER} started
30     else
31       echo Error: ${MILTERDIR}/${MILTER} missing
32     fi
33     ;;
34  'stop')
35     rm ${MILTERDIR}/${MILTERSEMAPHORE}
36     ${MILTERKILL}
37     echo ${MILTER} stopped
38     ;;
39  *)
40     echo "Usage: $0 { start | stop }"
41     exit 1
42     ;;
43  esac
44  exit 0

The first thing this script does (line 11) is to export all the shell macros that the baby-sitting script will later use. These same macros are given values in the six lines that follow. The MILTER macro (line 13) defines the name of the Milter program that will be started or stopped (for our example we use ourmilt). The MILTERDIR macro (line 14) defines the directory in which the Milter lives. The MILTERARGS macro (line 15) defines any command-line arguments that are needed by the Milter in order to start up (here there are none). The MILTERSLEEP macro (line 16) determines how long the baby-sitting script should sleep after the Milter dies and before restarting it. This should be zero for all but Solaris, where it should be five or more seconds (because reuse of its sockets does not reset swiftly enough to avoid the need for a sleep). The MILTERRUN macro (line 17) defines the name of the baby-sitting script. And the MILTERSEMAPHORE macro (line 18) defines the name of a file that will signal to the baby-sitting script that it should keep running (a file that acts as a semaphore).

The MILTERKILL macro determines the method used to halt the Milter. For Solaris, uncomment the first line (line 20). For BSD or Linux operating systems, uncomment the second line (line 21).

This script must be run with one of two command-line arguments: start or stop. If neither argument is given or if all arguments are omitted, then the script prints (line 40) the following message and exits:

Usage: ourmilt.init { start | stop }

To start the Milter, the start argument (line 24) is given when you execute the script. If the Milter exists (line 25), then the semaphore file is created (line 27), and the baby- sitting script is started in the background (line 28). If the Milter doesn't exist, the following error (line 31) is printed:

Error: /etc/mail/milters/ourmilt missing

To stop the Milter, the stop argument (line 34) is given when you execute the script. The semaphore file is removed (line 35), and the Milter is killed (line 36).

9.3.1. Baby-Sit the Milter

The second script needed to run your Milter is a baby-sitting script. Its job is to keep the Milter running. The key concept here is in the semaphore file. Recall that the semaphore file is created by the prior script's start argument and removed by its stop argument.

 1  #! /bin/sh
 2  #############################################
 3  # OURMILT.RUN
 4  #       A script that runs the milter in the
 5  #       foreground, for as long as the
 6  #       semaphore file exists. In the event
 7  #       that the milter dies unexpectedly,
 8  #       it is automatically restarted.
 9  #
10  # This script is called from ourmilt.init.
11  #############################################
12
13  while [ -f ${MILTERDIR}/${MILTERSEMAPHORE} ]
14  do
15    ${MILTERDIR}/${MILTER} ${MILTERARGS}
16    if [ -f ${MILTERDIR}/${MILTERSEMAPHORE} ]; then
17       logger -p mail.alert -t ${MILTER} "restarted";
18     fi
19     sleep $MILTERSLEEP
20  done
21  exit 0

All the work in this script is done in an indefinite while loop (line 13). As long as the semaphore file exists (line 13), the Milter will continue to run. Each time through the loop, this baby-sitting script first launches the Milter in the foreground (line 15). The script then waits patiently for the Milter to exit, but if the Milter exits abnormally (or if it is killed to make it stop), then control returns to this baby-sitting script (line 16).

As soon as control returns, a check is made to see whether the semaphore file still exists (line 16). If it does, the baby-sitting script logs a warning (line 17). If the semaphore file does not exist (if the earlier script removed it), the while loop exits (line 20), and the script (done) also exits (line 21).

If the semaphore file exists, then the while loop continues around to restart the Milter (line 13). On Solaris systems this script will sleep for a while (line 19) before restarting (because reuse of its sockets does not reset swiftly enough to avoid the need for a sleep).

Variations on these scripts are possible. They may be desirable if, for example, your operating system's default shell is not the Bourne shell or if your operating startup scripts do not follow the start/stop model.

9.4. Put Your Milter into the Background

A Milter is a process that runs in the background, unattached to any controlling terminal. There are two ways to put a Milter into the background. The first is by adding a trailing ampersand to the line in the script that launches the Milter:

if [ -f /etc/mail/milters/ourmilt ]; then
        /etc/mail/milters/ourmilt &
fi

Here, the trailing ampersand on the second line causes the Milter ourmilt to be executed in the background.

You can also place a program in the background by including code inside it. The following bg() function is an example of one way to do this.

 1  int
 2  bg()
 3  {
 4    int fd, i;
 5
 6    i = fork();
 7    if (i < 0)
 8      return errno;
 9
10    if (i != 0)
11      exit(0);
12
13    (void) freopen("/dev/null", "r", stdin);
14    fd = open("/dev/null", O_WRONLY, 0666);
15    (void) fflush(stdout);
16    (void) dup2(fd, fileno(stdout));
17    (void) setsid();
18    errno = 0;
19    return 0;
20  }

This bg() function takes no argument. It will return a nonzero integer if there is any error. It first performs a fork() (line 6) in which the original (called the parent) is copied (to a child). At this point the parent and child continue to run. But if the fork() fails, a nonzero error is returned (line 8). Note that if the fork() succeeds the child sees a zero value, whereas the parent sees a positive value (line 10). Then the parent exits (line 11), and only the child continues to run.

The next four lines, line 13 through line 16, connect the standard input and standard output to the file /dev/null. That file is special in that writes to it and reads from it always succeed. We do this so that any attempts by your Milter to read from stdin or write to stdout will silently succeed rather than fail.[6] Note that, although your Milter can still print to its standard error output (perhaps for additional startup errors), such printing is discouraged in favor of recording errors using syslog().

[6] Had we just closed stdin and stdout, your Milter might have produced errors while trying to use them and exited for the wrong reasons.

The call to setsid() (line 17) causes your Milter to become the leader of its own session. This means that signals from the shell that launched it will not be caught by it. Thus, because the prior lines were cast to void (errors were ignored), we need to set errno to zero on the chance that an actual error was ignored.[7]

[7] This is good coding practice, even though the sample main() does not use errno.

You can test this bg() function with a small main() that looks like this:

int
main()
{
        printf("will sleep 15 seconds in the background\n");
        if (bg() != 0)
        {
                perror("backgroundify");
                exit(EX_OSERR);
        }
        sleep(15);
        fprintf(stderr, "done.\n");
        return 0;
}

When you design your Milter, consider adding a debugging or other command-line switch that can be used to run it in the foreground for testing. In that way, you can use printf() statements freely and their output will be masked whenthe Milter is running in the background.

9.5. Handle Signals

The three signals used by the milter-library are SIGHUP, SIGTERM, and SIGINT, so these signals are not available for your use in your Milters. The milter-library also causes your Milter to ignore SIGPIPE.

SIGTERM causes an orderly shutdown of the Milter.

SIGHUP causes an orderly shutdown of the Milter (synonym for SIGTERM).

SIGINT causes an immediate abort of the Milter.

SIGPIPE is ignored.

If you wish to use a signal (perhaps to force a reread of the configuration file), we recommend you use SIGUSR1 or SIGUSR2.[8]

[8] You should never perform I/O inside a signal handler. Instead, set a flag that another thread will recognize so that the work is performed outside the signal handler.

When you're handling signals in a multithreaded program, it is important that you trap signals in only one thread (usually the main() thread) and mask those signals so that they are not seen by other threads. One way to do that is shown in the following sig() function. Note that sig() as shown here presumes you are trapping SIGUSR1 to mark that the configuration file should be reread. The signal-trapping function sigmarkreadconf() is shown first, and the signal setup routine sig() is shown second.

 1  void
 2  sigmarkreadconf(sig)
 3     int sig;
 4  {
 5     (void)signal(SIGUSR1, SIG_IGN);
 6     /*
 7      * set a flag here that will cause
 8      * another thread to reread the
 9      * configuration file
10      */
11     (void)signal(SIGUSR1, sigmarkreadconf);
12  }
13
14  int
15  sig()
16  {
17    sigset_t set;
18
19    if (signal(SIGUSR1, sigmarkreadconf) == SIG_ERR)
20       return errno;
21    (void) sigemptyset(&set);
22    (void) sigaddset(&set, SIGUSR1);
23    return pthread_sigmask(SIG_BLOCK, &set, NULL);
24  }

The sigmarkreadconf() function is very simple. It will be called whenever the program receives a SIGUSR1 signal. The argument to sigmarkreadconf() is sig (line 3), which will always be the value SIGUSR1. First, we arrange to ignore any additional SIGUSR1 signals (line 5). Then a flag is set so that another thread will reread the configuration file (line 6). Then, we arrange to trap SIGUSR1 just as before (line 11).

But before sigmarkrereadconf() can trap any signals, your main() function must make some prearrangements. It does so by calling the sig() routine (line 15). That routine takes no argument. As designed here, it only returns nonzero values to indicate errors and expects main() to print any error found. First, sig() arranges for the sigmarkrereadconf() function to be called if the program receives a SIGUSR1 signal (line 19). Then a mask is cleared (line 21) and filled to contain only SIGUSR1 (line 22). The pthread_sigmask() library routine is called last so that any threads launched from main() will be prevented from seeing that signal (line 23).

A simple way to test these functions is with a small main() such as the following.

int
main()
{
        if (sig() != 0)
                perror("signal");
        printf("%ld\n", getpid());
        sleep(30);
        return 0;
}

Note that when this program is run it prints its process ID and then sleeps for 30 seconds. In that way, you can test it by killing it with a -USR1, which will cause it to exit immediately.

% ./a.out &
29271
% kill -USR1 29271

9.6. Anticipate MX Servers

One way to block spam is by rejecting connections from selected IP numbers. This works well if all email is delivered directly to the machine or machines running your Milter. But if you are a smaller site or a geographically dispersed site, you may have MX servers that you have no direct control over. In such a case, you cannot install your Milter on those MX servers, so you must set up a method to detect when mail is received from them. To illustrate, consider the following simple MX records:

mail.example.com.    IN    MX 10 mail.example.com.
mail.example.com.    IN    MX 50 isp.mx.host.

The intention here is to have email delivered directly to mail.example.com unless mail.example.com is busy or down. In that case, email should instead be delivered to isp.mx.host (a big MX server at some ISP), which will hold the email until mail.example.com is ready to receive it. But spam email is often sent directly to the highest-numbered MX host, even if the lowest-numbered (most desirable) host is up and waiting for mail. Spammers do this because they know that you probably cannot run your email spam filters on the ISP's MX machines. They also know that you will probably reject connections based on the connecting host's IP number. Thus, if they deliver to the ISP's MX machine, that machine will connect to your host and you will see the IP number of the MX machine and not the IP number of the spammer's machine.

Normally, MX records are fairly static, so you can safely list them in an array and compare inbound connection addresses to the IP addresses in the array before rejecting any connection. There may be times, however, when you won't know MX servers ahead of time (such as when selling a Milter product), and you will have to look up that information when the program first starts to run.

The following mx() function illustrates one way to look up MX servers. Here, it prints only what it finds. In an actual program you would probably want to use getipnodebyname() to look up the IP numbers for each MX host found.[9]

[9] Dont use the gethostbyname() function because it is not thread safe.

 1  #include <stdio.h>
 2  #include <sys/types.h>
 3  #include <netdb.h>
 4  #include <arpa/nameser.h>
 5  #include <resolv.h>
 6
 7  void
 8  mx(host)
 9     char *host;
10  {
11     u_char *eob, *cp;
12     int k;
13     u_char buf[BUFSIZ];
14     HEADER *hp;
15     union {
16       HEADER  h;
17       u_char  u[PACKETSZ];
18     } q;
19     int acnt, qdcnt;
20     u_short pref;
21     u_short type;
22     u_long ttl;
23
24    if (host == NULL || *host == '\0')
25       return;
26
27    k = res_query(host, C_IN, T_MX, (u_char *) &q, sizeof(q));
28    if (k < 0)
29       return;

Because this function is so complex we will explain it in small bites rather than attempting to swallow it whole. The mx() function takes a single argument (line 9), the name of the host for which you wish to look up MX records. If the argument is NULL or an empty string, the function returns (line 24). Otherwise, that host name will have its MX records looked up (line 27). If the host has no MX records or does not exist, or if there is any other error, this function silently returns (line 29).

The rest of the code is complex because decoding the result of a res_query() call (line 27) is complex:

30    hp  = &(q.h);
31    cp  = q.u + HFIXEDSZ;
32    if (k >= sizeof q)
33       return;
34    eob = q.u + k;
35
36    for (qdcnt = ntohs(hp->qdcount); qdcnt--; cp += k + QFIXEDSZ)
37       if ((k = dn_skipname(cp, eob)) < 0)
38         return;

In a for loop, we first search the returned buffer (q) to find a satisfactory answer (line 36). If dn_skipname() finds any problems, we return (line 37). Otherwise, we continue by parsing the buffer:

39    acnt = ntohs(hp->ancount);
40    while (--acnt >= 0 && cp < eob)
41    {
42      if ((k = dn_expand(q.u, eob, cp, buf, (sizeof buf)-1)) < 0)
43        break;
44       cp += k;
45       GETSHORT(type, cp);
46       cp += INT16SZ;
47       GETLONG(ttl, cp);
48       GETSHORT(k, cp);
49       if (type != T_MX)
50       {
51          cp += k;
52          continue;
53       }
54       GETSHORT(pref, cp);

For as many MX records as there are to parse out of the buffer, we call dn_expand() to extract the information for the first part of each record (line 42). We do this because the buffer holds packed data that needs to be pulled apart with appropriate macros for each expected size. The type (an MX record, A record, PTR record, etc.) is represented by a two-byte short, so we use GETSHORT to extract it (line 45). The ttl (time to live) is an unsigned long, so we use GETLONG to extract it (line 47). If the type is not T_MX (not an MX record), we skip that record and try again (line 49).

Then we call dn_expand() again (line 55) to extract the host name associated with this record:

55       if ((k = dn_expand(q.u, eob, cp, buf, (sizeof buf)-1)) < 0)
56         break;
57       cp += k;
58       (void) printf("%s. %ld IN MX %d %s.\n", host, ttl, pref, buf);
59    }
60    return;
61  }

For each record found, we print the host name that was looked up (the argument given to this mx() function) and its ttl and MX records (including preference) to the standard output (line 58).

An easy way to test this mx() function is to compile it using a simple main() routine. Basically, we accept a host name on the command line and pass that host name to mx() as is.

int
main(argc, argv)
        int argc;
        char **argv;
{
           if (argc > 1)
                mx(argv[1]);
           return 0;
}

When this program is compiled, you can run it to print the MX records for any host you select:

% ./a.out aol.com
aol.com. 3600 IN MX 15 mailin-03.mx.aol.com.
aol.com. 3600 IN MX 15 mailin-04.mx.aol.com.
aol.com. 3600 IN MX 15 mailin-01.mx.aol.com.
aol.com. 3600 IN MX 15 mailin-02.mx.aol.com.

9.7. Status and Logging

The sendmail program logs the envelope sender record separate from each envelope recipient record, but it includes the queue identifier with each log record as a means to join those records. Because a Milter operates in conjunction with sendmail, it should also provide the queue identifier so that its log records can be easily joined with the log records produced by sendmail.

A sendmail log record looks (in part) like this:

Apr 30 08:00:53 host sendmail[29456]: i3UE0kpG029456: from=...

Here, each log record occupies a single line and begins with the date and time the record was made. The time is followed by the host name of the logging host and then by the name sendmail.[10] The name sendmail is followed by its process ID inside square brackets. A colon and a space are next, and then the queue identifier (shown here in bold). A colon follows that, and then one or more equates conveying information appropriate to the log record (here, from= documents the envelope sender).

[10] If sendmail was run with the -L command-line switch, this name may appear as something else, such as "smtp-mta" or "slow-sendmail."

The log record for Solaris contains additional information following the square-bracketed process ID (the [ID 801593 mail.info]) but is otherwise the same, also including the queue identifier.

Apr 30 08:32:57 host sendmail[2487]: [ID 801593 mail.info] i3UEWoLV002487: from=...

The queue identifier identifies envelopes and not connections, so the first time the queue identifier is available to your Milter it comes from inside your xxfi_envfrom() handler function (see section 8.6), which handles the envelope sender.[11] The queue identifier is passed to that function in the $i sendmail macro, which can be fetched this way:

[11] If you want to join connection information with envelope information in your logs, you will need to create your own connection-based identifier.

priv->qid = smfi_getsymval(ctx, "i");
if (priv->qid == NULL)
         priv->qid = "NOQUEUEID";

Here, the smfi_getsymval() milter-library function (see section 7.3.1) is called to fetch the value of $i. That value is stored as a string in the variable priv->qid, a part of your private data. If no queue identifier is available, the string NOQUEUEID is used instead.

Thereafter, whenever you log envelope information, you can do so with a syslog() expression something like this:

(void) syslog(LOG_INFO, "%s: sender=%s whitelisted",
        priv->qid == NULL ? "NOQUEUEID" : priv->qid,
        priv->mailfrom == NULL ? "<>" : priv->mailfrom);

Notice that we use defensive programming when logging. We never presume that a string will have a valid value, even if we placed that value there. Remember that the xxfi_ handler functions are not necessarily called in the order that the sending client sends SMTP commands, and it is possible that any handler function may be called with uninitialized strings because of an unexpected ordering.

9.8. Consider Portability Early

If you are writing a Milter for your own machine or for a homogeneous environment of machinesall Solaris 9 or all Linux, for exampleyou can skip this section. But if you are writing a Milter intended for a heterogeneous environment or for a commercial or free product, you need to consider portability early.

The best tools for achieving portability are those in the GNU autoconf suite:

autoheader create a template include file for use by autoconf.

automake create a makefile from a Makefile.am template file.

autoconf create a configuration script from a configure.ac template file.

We won't describe the autoconf suite here because that would require a book of its own. Instead, we recommend that you download and install it, and then learn to use it.[12] After you have set up your Milter to use autoconf, installing your Milter becomes as simple as this:

[12] Available from http://www.gnu.org/directory/devel/build/autoconf.html.

% ./configure
% make
# make install

9.9. Avoid Memory Leaks

The adage "avoid memory leaks" may seem trite, but when you're working with Milters you should take it seriously. Recall, for example, that the handler xxfi_eom() function (see section 8.11) is called only if all prior Milters (including your Milter) allowed the envelope to pass through. If any prior Milter rejects or discards the envelope, your xxfi_eom() handler function will not be called, but your xxfi_abort() function (see section 8.12) will be. But if your Milter rejects or discards the envelope itself, neither xxfi_eom() nor xxfi_abort() will be called.

If you are careless in how you code around these conditions, your Milter can easily leak memory. There are three ways to approach this situation:

If you know that your xxfi_abort() handler function won't otherwise be called, call it yourself.

Make the xxfi_abort() and xxfi_eom() handler functions idempotent so that they can be called repeatedly without harm and thus can be called when in doubt.[13]

[13] For example, if you free() a pointer, always immediately set it to NULL so that you can check for NULL and avoid freeing it a second time.

Establish deallocation routines that can be called from either xxfi_abort() or xxfi_eom(), or directly from within your handler functions.

We prefer the last technique because it is the most flexible. Consider the following two routines.

 1  sfsistat
 2  dealloc_connection(ctx, ret)
 3          SMFICTX *ctx;
 4          sfsistat ret;
 5  {
 6          PRIVDATA *priv = (PRIVDATA *)smfi_getpriv(ctx);
 7
 8          if (priv == NULL)
 9                  return ret;
10          /* deallocate connection stuff here */
11      (void) dealloc_envelope(ctx, 0);
12          (void) free(priv);
13          (void) smfi_setpriv(ctx, NULL);
14          return ret;
15  }
16
17  sfsistat
18  dealloc_envelope(ctx, ret)
19          SMFICTX *ctx;
20          sfsistat ret;
21  {
22          PRIVDATA *priv = (PRIVDATA *)smfi_getpriv(ctx);
23
24          if (priv == NULL)
25                  return ret;
26          /* deallocate envelope stuff here */
27          return ret;
28  }

There are two routines here. The dealloc_connection() routine (line 2) is called from your xxfi_close() handler function and deallocates connection-persistent information. Because it is always called on the close of the connection, it can safely deallocate the private data (line 12) after the connection-specific information has been deallocated. Note that the private data is changed to NULL by calling the smfi_setpriv() milter-library function (line 13) so that you cannot accidentally free what was already freed in a mistaken later call.

Here, the dealloc_envelope() routine (line 18) is called from your xxfi_abort() handler functions, but it can also be called directly when you know that these functions won't otherwise be called. Note that the dealloc_envelope() routine deallocates only envelope-specific information and does not deallocate the private structure (which may contain connection information) because there can be multiple envelopes during a single connection.

One way to use the dealloc_envelope() routine looks like this:

if (message_size > MAX_SIZE)
        return dealloc_envelope(ctx, SMFIS_REJECT);

Because you are rejecting the envelope (the SMFIS_REJECT), neither the xxfi_eom() nor the xxfi_abort() handler function will be called. Therefore, this code returns by calling dealloc_envelope() to make sure that any allocated envelope resources are deallocated.

When you deallocate items, be sure to replace each deallocated item with NULL so that none can later be mistakenly deallocated again:

if (priv->hostname != NULL)
{
        (void) free(priv->hostname);
        priv->hostname = NULL;
}

9.10. Final Words

There are many more ways a Milter can interact with its environment than we have shown here.

Milters can interact with databases. The access database used by sendmail, for example, might contain whitelisting information useful to a Milter. Or Lightweight Directory Access Protocol (LDAP) may contain user authentication information used sitewide.

Milters may also need to identify and recognize email whose destination is the Milter itself. In that way, for example, a Milter could be designed that learns from experience.

This means that, until you have architected your Milter, you cannot anticipate what further interactions it will have.

In this chapter we have given you a few of the most common tools you will need to develop all Milters. In the next chapter we show you how to architect a professional Milter.

Chapter 10. User and Temporal Feedback

Before you can code a Milter you must first model the user community it will serve and the interactive environment in which it will run. How, for example, will the customer (the email reader) interact with the Milter? And how much and what kind of history should be maintained?

In this chapter we explore these and related questions.

Section 10.1 tells why you should consider the architecture of your Milter first.

Section 10.2 shows how to identify your customers' needs.

Section 10.3 illustrates the value of keeping historical records when you're fighting spam.

Section 10.4 examines ways to insert feedback from humans into your system.

Section 10.5 discusses what whitelisting is and how to manage it.

Section 10.6 illustrates one way to perform "fuzzy" matching of sender addresses.

Section 10.7 examines what to do with detected spam.

Section 10.8 shows how to plan for future spamming techniques.

10.1. Consider Architecture

Before you design your Milter, there are several questions you need to ask yourself. By answering them ahead of time you reduce the risk of designing a Milter that will need to be redesigned after it is deployed.

If a user decides an email message is spam, how does the user communicate that decision to the Milter? Does the Milter recognize mail sent to itself? Is there a command-line interface for updating the Milter's knowledge? Is there a web CGI (Common Gateway Interface) script that can update the Milter's knowledge?

Do all users expect the same services from the Milter? Should the Milter allow whitelisting for some users and blacklisting for others? How do users communicate personal profile information to the Milter?

What databases and directory services already exist? If sendmail uses LDAP (Lightweight Directory Access Protocol), should your Milter use LDAP too? If your Milter uses a Berkeley database, should it support only one version or all versions? Will administration personnel need to perform complex lookups on the Milter's data, and, if so, should mySQL or another SQL (Structured Query Language) database be supported?

Will there be one master machine running the Milter, with other machines' sendmail connecting to that central Milter? Or will there be a Milter running on every machine? If you use the distributed (first) model, how will log and database data be unified and joined? How do you push new knowledge out to other machines?

Will the Milter have few options (thereby allowing it to be run only using command-line switches), or will a configuration file be necessary? Should the Milter be able to reread its configuration information while running? Should the configuration be in a flat file, in a database, or in a directory service?

How does a blocked sender get email past the Milter? If an innocent user is wrongly blocked, how does that user communicate with a local human who can correct the error? Does the Milter recognize and always accept certain addresses? Can an innocent person send mail to, for example, "postmaster" or "whitelist" and expect that message to always arrive in a human's mailbox?

Are policies for inbound and outbound email the same? Will there be instances when a local user legitimately needs to send something that is not allowed to come in (such as an executable program)? Should your architecture allow for exceptions?

What should you do with the spam you find? Do you reject the first message and discard all others from the same source? Do you archive all spam alongside each local recipient's mailbox so that it can be screened for false positives before being discarded? Do you simply annotate each message as spam or not, and pass all messages through to the local recipients?

10.2. Model the End User

Spam suppression serves two communities of users. For outbound email, it prevents your site from sending spam to the outside world. For inbound email, it prevents your users' mailboxes from filling with unwanted email. We examine each of these in turn.

10.2.1. The Outside World as the End User

Normally, you might not think of the outside user as benefiting from local spam control. But just as you don't want to receive spam email from the outside world, the outside world does not want to receive spam email from you.

At a minimum, this means that your Milter spam suppression should screen all email, inbound and outbound. Later in this chapter we discuss an asymmetric approach to bidirectional screening, but for now, just consider outbound hygiene to be as important as inbound.

In addition to spam filtering, you need to be aware of the risk imposed by internal PC customers. Unless you manage all internal PCs, there is a significant risk that some of your internal PCs will be hijacked by computer worms. Hijacked PCs can be made to send viruses and spam email without knowledge of the PC owner. It is your responsibility to ensure that hijacked PCs never send anything outward past your domain's borders.

One way to block PCs is to set up your router or firewall to allow only outbound connections to port 25 from your master mail-sending machine. This is an efficient safeguard in a business environment but (unfortunately) is impractical in public venues such as hotels and wireless-access airports. At a minimum, we recommend that you disallow outbound access to port 25 for all internal machines as a matter of policy and enable access only on a case-by-case basis.

10.2.2. Your Internal Customer as the End User

When we think of spam suppression, we normally think of preventing delivery of spam to our own internal customers. But not all customers are equal. Some read email using a web interface you provide, whereas others POP messages off your server and read the email on a PC. Yet others (although a minority) have shell accounts and read email directly on a server.

Clearly the way you handle spam email will vary depending on how a customer reads email. For web interface and Internet Message Access protocol (IMAP) email readers, for example, you can provide a spam folder that can be browsed to find possible false positives. A Post Office Protocol (POP) user, on the other hand, may prefer to have spam email discarded so that fewer messages need to be downloaded over a slow dial-up connection.

But before you can decide how to dispose of spam email, you must also understand the needs and desires of your user community. If you are a small business, you may be able to select a single solution and impose it downward across all users. But if you are a university or an ISP, for example, you will probably have to offer multiple solutions and provide them on a customer-by-customer basis.

If you expect your user community to need per-user customizing of spam filtering, you should first examine your use of existing databases. If, for example, you already use LDAP to record user preferences, you could customize your spam filter preferences around that same directory service. Because users rarely change spam preferences, the directory service will likely remain read intensive.

It is also important to use existing mechanisms to enable user preferences. If, for example, the vast bulk of your customers use a web mail interface, you should add spam-filtering preferences to that interface rather than design and add a whole new interface.

When you model the end user, always use the lowest common denominator. Strive to find solutions that fit the most users in the most comfortable manner, thereby limiting the number of exceptions you must handle.

10.3. Maintain a History

When spam email arrives for one user, it will very likely also arrive for many other users. After a piece of email has been identified as spam email, it should be recorded so that later arrivals of that same message are automatically recognized and handled.

To illustrate using a simple example, suppose a piece of email arrives with the subject "Thanks" and a simple two-line body:

Your mortgage application was approved. Please
complete the process at http://mortgage.example.com

Suppose a couple of users received this email and reported that they hadn't applied for a mortgage and that this was a scam. To prevent possible harm to other users, you decide to reject all email that contains this URL.

First, you add the URL's host (mortgage.example.com) to your local spam database. One way to do that, using Berkeley DB, might be to store the host name as the key and to store the number of occurrences as the value (remember there were two initial reporting users):

mortgage.example.com       2

To make this work, you will need to design your Milter to look up the host name and to reject any message that contains a URL containing that host name.

Recall, however (see section 2.3), that spammers often lie about the envelope sender so that bounced messages will not be returned to them. A handy rule of thumb, when you archive site occurrences in a database, is to bounce only the first few messages (on the off chance that a real person sent something that only looked like spam) and to discard any received after the initial few.[1]

[1] This prevents your site from sending endless bounce messages to (perhaps) an innocent person whose address was used by spammers as the envelope sender address.

But be aware that the next spam message may arrive looking like this:

Your mortgage application was approved. Please
complete the process at http://loans.example.com

Note that the prior URL host was mortgage.example.com, but in this message the host is loans.example.com. The two share the same domain part (the example.com), but can we really be sure that they came from the same host? To find out, you need to look up the IP numbers for each host and compare them.[2] One way to do that is shown in the following bit of code, where a function looks up host IP numbers and prints what it finds:

[2] But be careful. Sometimes unrelated sites can share a common IP number.

 1 void
 2 getips(host)
 3   char *host;
 4 {
 5    struct hostent *hp;
 6    int err;
 7    struct in_addr in;
 8    char **p;
 9
10    if (host == NULL)
11       return;
12
13    hp = getipnodebyname(host, AF_INET,
14      AI_DEFAULT , &err);
15    if (hp == NULL)
16      return;
17    if (hp->h_addr_list == NULL)
18    {
19      freehostent(hp);
20      return;
21    }
22    for (p = hp->h_addr_list; *p != 0; p++)
23    {
24      (void) memcpy(&in.s_addr, *p,
25        sizeof(in.s_addr));
26      (void) printf("%s%s\n", host, inet_ntoa(in));
27    }
28    freehostent(hp);
29    return;
30  }
31

This function is passed a string that is the host name to look up (line 2). If that string is NULL, nothing is done (line 10). Otherwise, the host is looked up using the getipnodebyname() C library function (line 13).[3] If that lookup fails, the function returns. If no IP numbers are found, freehostent() is called to free allocated memory, and the function returns. Otherwise, in a loop, each IP number is printed.

[3] Don't use the gethostbyname() function because it is not thread safe.

Note that this function is merely an example. In an actual Milter, you should return a list of IP numbers rather than print them, but printing works for now just to illustrate our point.

To test this function you can use a simple main() such as the following:

1 int
2 main(argc, argv)
3         int argc;
4         char **argv;
5 {
6         if (argc > 1)
7                 getips(argv[1]);
8         return 0;
9 }

Now, by using the finished program, you can compare the IP numbers for mortgage.example.com, loans.example.com, and example.com.

% ./a.out mortgage.example.com
mortgage.example.com   192.168.33.44
% ./a.out loans.example.com
loans.example.com      192.168.33.44
% ./a.out example.com
example.com      10.55.66.77

From these results, it is clear that, because mail with a URL host of mortgage.example.com is being rejected, we should also reject email with a URL host of loans.example.com. But the results do not clearly indicate that a URL host of example.com should also be rejected. Perhaps example.com is an ISP and mortgage and loans are vanity subdomains under it. If that is the case, we should allow email containing the URLs of the ISP and reject email containing only URLs from those particular subdomains.

When all this information is stored in a Berkeley database, it might look like this:

mortgage.example.com     2
loans.example.com        1
192.168.33.44            3

After this information is in the database, if a future message arrives with a URL host of, say, approved.home.loans.example.com, you can find the IP number of that host (suppose it is 192.168.33.44), then look up that address in the database, and determine that this spam message should also be rejected.

10.4. Possible Feedback Mechanisms

Although some characteristics of spam can be detected by machines, others can be detected only by humans. For example, gross abuse of the HTML standards, such as the following, is a sure sign of spam:

Our t<muzo>e<consultant>am of U.S. board c<diet>e<cupric>rt<befell>ifie<ne=
on>d p<nietzsche>hys<doleful>ic<amass>ians<sugar> and p<botfly>harmac<hyst=
eresis>is<gamma>t<loaf>s<blueberry>

But more subtle forms of email, such as the following, may not be so easy to detect:

Your application was approved. Please visit
<a href="http://discount.example.com">our
profile page</a> to set up your initial profile.

An email like this could be the result of a user signing up for some web service. On the other hand, it could be an attempt to gather profile information for nefarious purposes. There is no way to be certain of its true nature without human decision intervention.

Even worse, this message might be valid for one user (the one who signed up) but spam to other users (because that one user who signed up opened the floodgates to your site). We discuss decisions based on the desires of individual users shortly. Here we focus on how humans and Milters can work together to detect spam.

There are several ways a Milter can learn from human feedback:

Email sent to the Milter can be interpreted for indications of spam.[4]

[4] Alternatively, mail can be sent to a program with the use of sendmail's aliases file, which some may find an easier technique.

A web interface can be used to update a Milter's knowledge of what is and is not spam.

A system administrator can edit a configuration file to augment a Milter's spam knowledge.

Email review personnel can screen each message and advise the Milter about what is and is not spam.

Other spam filters can prescreen email, and a Milter can base its learning on the output of those other filters.

To illustrate, we show you one of these methods. If a user determines that a message is spam, the user forwards the message to the Milter so that the Milter can learn from human input. For this to work, the Milter must be able to read email addressed to itself and received from inside your domain. In the following code we presume that the email address ourmilt@your.domain was defined in an external configuration file or on the command line. Note that the numbers to the left are not a part of the code.

1 sfsistat
2 xxfi_envrcpt(SMFICTX *ctx, char **argv)
3 {
4         PRIVDATA *priv;
5         char *cp;
6         int len;

Here, we detect the addressee of the message inside your xxfi_envrcpt() handler function (see section 8.7). This is given each envelope recipient (line 2) as one of its arguments. Recall that the recipient address is passed in argv[0], with the rest of argv containing additional envelope recipient arguments.

7   if(argv[0] == NULL)
8      return SMFIS_CONTINUE;

Note that if argv[0] is NULL (line 7), the check for the envelope recipient is skipped.

 9   priv = (PRIVDATA *)smfi_getpriv(ctx);
10   if (priv == NULL)
11      return SMFIS_CONTINUE;

In line 9 the smfi_getpriv() milter-library routine (see section 7.3.3) fetches private data from the private-context pointer (line 2). If there is no private data, the envelope recipient check is skipped (line 11).

12    if (++(priv->numrcpts) > 1)
13    {
14      priv->tomilter = FALSE;
15      return SMFIS_CONTINUE;
16    }

Note that a part of the private data is a count of the number of envelope recipients (line 12). The idea here is that if the message is to the Milter (and if there is more than one envelope recipient), we should not allow the Milter to think the message contains spam notification (line 14).

17    cp = argv[0];
18    len = strlen(cp);
19    if (*cp == '<')
20    {
21      len -= 2;
22      ++cp;
23    }
24    if (len != strlen(MilterEmailAddress))
25      return SMFIS_CONTINUE;
26    if (strncasecmp(MilterEmailAddress, cp, len) != 0)
27      return SMFIS_CONTINUE;
28    priv->tomilter = TRUE;

Remember that the email address of the Milter is either stored in an external configuration file or specified on the command line. Here, it is stored in the variable named MilterEmailAddress. In line 19 we check to see whether the recipient address is surrounded in angle brackets and, if it is, we strip them.[5] Then the result is compared to the Milter's email address (line 26), and if there is a match, that fact is recorded in the private data (line 28).

[5] This simple technique would have to be more complex if you also desire to check for the presence of nested angle brackets.

29    return SMFIS_CONTINUE;
30  }

After the check for the Milter as a recipient, additional recipient checks can be made, or the function can return SMFIS_CONTINUE to continue (line 29). In this example we don't remove the Milter's address because it was aliased to /dev/null in the aliases file. You may of course prefer to just remove it using smfi_delrcpt() (see section 7.4.6) when your xxfi_eom() function (see section 8.11) is called.

Later in the Milter, the effect of the message being addressed to the Milter can be handled by the xxfi_eom() handler function (see section 8.11).

1    if (priv->tomilter)
2    {
3      addsite_database(url_host);
4      /* ... other actions, such as logging. */
5    }

In this code fragment, a check is made to determine whether the private data indicates that the message is to the Milter (line 1). If it is, the found URL host and its IP addresses are added to the database (line 3).

10.5. Whitelisting

Whitelisting is the method of allowing email to be delivered only if the sender and recipient match in an "allow" list. With whitelisting, for example, the local recipient bob might want to receive outside email only from his mother and brother. If anyone else sends bob email, it will be rejected.

But what happens when bob purchases this book from amazon.com? Email notification of that purchase will be sent to bob but, because amazon.com is neither bob's mother nor brother, that message will be (probably wrongly) rejected.

The hardest problem to solve in a whitelisting environment is not whom to allow in, but how to add new addresses to the existing whitelist. Several means are commonly used:

In the rejection, advise the user of a website to visit. There, the sender will see a graphic image of a number or word that must be typed in. If what the sender types matches the image, email from that sender will be allowed.

In the rejection, include instructions about whom to email or telephone to be added to a whitelist.

Provide the recipient with a list of senders who have been rejected. The recipient can then selectively add rejected senders to a whitelist.

No solution is perfect. Is it better, for example, to impose extra work on the sender or the recipient? A recipient that is a small business, for example, may want to impose the extra work on the sender, because email volume is low and because it needs to know exactly who is sending the email.

At a large ISP, however, the extra work of whitelisting should be imposed on the recipient because it is likely that the recipient selected whitelisting from a menu of options. Because the recipient has listed acceptable senders, it is the recipient's responsibility to maintain the list.

Designing a whitelisting environment is a complex undertaking, and such an environment requires many more component parts than does a Milter. For example, it may require the following:

A web interface, with a text statement describing how to become whitelisted, at a minimum, or the full processes and procedures to become whitelisted, at a maximum.

An underlying database, with the sender/recipient pairs that determine whitelisting policy, at a minimum, and with aliases, timeout intervals, and authentication information for each pair, at a maximum.

An exception process, in which an administrator periodically screens all senders, at a minimum, or screens each recipient to periodically detect unknown senders, at a maximum.

A sanity process, which ensures that senders are not also listed in other antispam databases, at a minimum, or automatically reviews all antispam systems for sanity (conflicting rules), at a maximum.

A phone number where clients or customers can telephone to become whitelisted, at a minimum, or an international toll-free number, at a maximum.

We don't show you how to create these other components (after all, this is a book about the sendmail Milter interface). Instead, we leave design and programming of these additional components as an exercise.

10.6. Graylisting

Whitelisting allows specific senders to send email to specific recipients. All other email is blocked (rejected or discarded). Whitelisting is a black-and-white approach that does not allow "fuzzy" rules.

Graylisting also allows email from specific senders, sending sites, sending mailing lists, and so on.[6] Graylisting is usually implemented with a form of fuzzy address matching. The procmail program uses one form of graylisting rules, whereas the slocal program uses another. Many HTML-enabled mail-reading programs also support fuzzy matching of addresses.

[6] See "The Next Step in the Spam Control War, Greylisting" by Evan Harris, at http://projects.puremagic.com/greylisting/.

In this section we show you one way to perform fuzzy matching inside your Milter using the regular-expression library.

1 #include <sys/types.h>
2 #include <stdio.h>
3 #include <regex.h>
4
5 #define TRUE  (1)
6 #define FALSE (0)

The regular-expression subroutine will return TRUE if a match was made. Otherwise, it will return FALSE.

 7 int
 8 fuzzy(expr, addr)
 9    char *expr;
10    char *addr;

The subroutine to do the fuzzy matching is called fuzzy(). It is passed two arguments: the regular expression char *addr; to use (line 9), and the address to match (line 17).

11 {
12    regex_t rx;
13    int err;
14    char buf[BUFSIZ];
15
16    if (expr == NULL || addr == NULL)
17      return FALSE;

If the regular expression is NULL or if the address is NULL, we return FALSE right away (line 17) to avoid dereferencing a NULL pointer later.

18    err = regcomp(&rx, expr, REG_EXTENDED | REG_ICASE | REG_NOSUB);
19    if (err == 0)
20      err = regexec(&rx, addr, 1, NULL, 0);

Performing a regular-expression evaluation requires two steps. First, the regular expression in text form (passed to this subroutine in expr) needs to be compiled into a special format (line 18). If that succeeds (line 19), the actual match is determined by a call to the regexec() C library routine (line 20).

21    if (err != 0)
22    {
23      if (err == REG_NOMATCH)
24        return FALSE;
25      (void) regerror(err, &rx, buf, sizeof buf);
26      (void) fprintf(stderr, "%s: %s\n",
27        expr, buf);
28      return FALSE;
29    }
30    return TRUE;
31  }

If no match was found, FALSE is returned (line 24). If there was any error (line 21), then regerror() is used to format the error text (line 25). Then that error text is printed (line 26), and the subroutine returns FALSE (line 28). Otherwise, a match was found, and the subroutine returns trUE (line 30).

You can test this routine with a small main() such as the following:

 1 int
 2 main(argc, argv)
 3    int argc;
 4    char **argv;
 5  {
 6    int r;
 7
 8    if (argc <= 2)
 9      return 0;
10    r = fuzzy(argv[1], argv[2]);
11    printf("%s: %s: %s\n", argv[1], argv[2],
12      r == TRUE ? "MATCHED" : "FAILED");
13    return 0;
14  }

You provide this program with two arguments: the regular expression to use and the address to match or not. Note that the regular expression should be quoted to prevent it from being interpreted by your shell. Consider, for example, the following tests:

% ./a.out '.*@.*amazon.com' sales@example.com
.*@.*amazon.com: sales@example.com: FAILED
% ./a.out '.*@.*amazon.com' sales@updates.amazon.com
.*@.*amazon.com: sales@updates.amazon.com: MATCHED

One use for such a routine might be to put regular-expression rules in a file that is read periodically by your Milter, or to allow individual users to put their own rules into (perhaps) a ~/.oksenders file.

10.7. Archive, Reject, or Pass Through Spam

After your Milter has reviewed the message, you must decide what to do with the result. There are probably more choices than you were aware of:

Reject the entire envelope, no matter how many recipients it has.

Reject the entire envelope only once and thereafter discard it.

Reject selected recipients while allowing delivery to proceed for other recipients.

Defer (tempfail) the entire envelope, no matter how many recipients it has.

Defer (tempfail) selected recipients while allowing delivery to proceed for other recipients.

Defer (tempfail) an envelope the first time it arrives, and then allow delivery on the second try.[7]

[7] This is a simplification of a graylisting technique described by Evan Harris. See previous footnote for more information.

Discard (accept and silently throw away) the envelope, no matter how many recipients it has.

Drop (remove) the selected recipients while allowing delivery to others.

Archive otherwise undelivered envelopes so that humans can review them to detect false positives.

Any well-rounded Milter can perform a combination of these possible outcomes. However, before you decide on any, you should be aware of potential pitfalls.

When you reject an envelope, that rejection can sometimes result in bounce mail being sent to the envelope sender. Be aware that a great deal of spam email is purposely deceptive about the identity of the envelope sender. Even worse, some spam uses real addresses for the envelope sender, causing bounce email to be sent to some innocent party. If you want to reject spam from a particular IP address or one that advertises a particular website, try to reject only the first copy and silently discard subsequent copies. In that way, the bounce of a real message will let the real user know that there is a problem, but repeated spam email will not wrongly bombard an innocent person with huge numbers of bounce notifications.

When you defer an envelope, it may be queued on the sending machine, or it may be queued on one of your MX servers (if you have any). Some spam-sending software discards a message if it is deferred, but you cannot count on that. If you have MX servers, you should probably never defer as an antispam technique unless all your MX servers share that same policy. On the other hand, if you lack MX servers, that technique could prove of some value.

When you discard an envelope, it appears to the sender that you accepted and delivered the message. From the outside, there is no way to detect that it was discarded. In effect, your Milter is lying to the outside world, but remember that it was driven to lying by the scourge of spam email.

When you archive copies of rejected email, be prepared to archive a huge amount of potentially large messages. Plan ahead to remove messages older than some comfortable interval. Also avoid allowing all users to read all rejected email, because that could lead to invasion of privacy problems. Instead, set up permissions on the archived email so that they can be read only by the intended recipient.

10.8. Dynamic Configurations

Spammers are constantly adapting to the means used to block them. Clearly, in such a dynamic world you should avoid a static design for your Milter. This is one reason to use a configuration file, especially one that is easily updated as your coding needs change. For this reason, we recommend that you begin your coding by designing and implementing a configuration file parser.

The simplest form of configuration file that can be parsed is line based, where each item is followed by an equal sign and then its value:

maxrcpts   = 20
maxsize    = 1000000

In addition, it would be beneficial if the routine that parsed this file could skip empty lines and lines that begin with a # comment character:

# This is a bare bones start
maxrcpts   = 20
maxsize    = 1000000

It would also be beneficial if comments could follow each line:

# This is a bare bones start
maxrcpts   = 20
maxsize    = 1000000 # one million

To begin coding such a parser, we start with a simple structure to hold the predefined item names and default values:

1 typedef struct {
2    char  *name;
3    char  *defval;
4    int   updated;
5 } CONF;

Each item that the program knows about will have an item name (line 2) and a default value represented as a string constant (line 3). With this setup, declaring the known items and values is easy:

 6 static CONF Cf[] = {
 7    /***************************************************
 8     * Maximum sizes and limits.
 9     ***************************************************/
10    {"maxsize","0",FALSE},
11    {"maxrcpts","20",FALSE},
12    {NULL,NULL,FALSE},
13 };

Here, we declare an array of the type CONF (line 6) and initialize it with two values. The maxsize item (line 10) specifies the maximum size (in bytes) of an email message that can be screened, but the default of zero negates that limit. The maxrcpts item (line 11) specifies the maximum number of envelope recipients to accept. Note that the default values are string constants, so care must be taken to never free them. The array ends (line 12) with NULL values so that we can detect its end.

Before we can actually parse the configuration file, we need to establish a modest subroutine to clean spaces from around strings:

14 /*
15 ** Remove leading and trailing white space from around
16 ** a string. This routine will core dump if given a
17 ** C language string constant. The original string is
18 ** modified.
19 */
20 char *
21 cleanup(str)
22    char *str;
23 {
24    register char *cp, *ep;
25
26    for (cp = str; ; ++cp)
27    {
28      if (!(isascii((int) *cp) && isspace((int)*cp)))
29        break;
30    }
31    for (ep = cp + strlen(cp) -1; ; --ep)
32    {
33      if (!(isascii((int) *ep) && isspace((int)*ep)))
34        break;
35    }
36    if (ep <= cp)
37      return cp;
38    *(ep+1) = '\0';
39    return cp;
40 }

This cleanup() routine takes a string as its only argument (line 21). Note that no check is made to ensure that the string is not NULL and is not a string constant. Two pointers are declared (line 24) that will scan through the string, where cp points to the start and ep points to the end of any enclosed text found. In two loops, first the leading spaces are removed (line 26), and then the trailing spaces are removed (line 31). The cleaned-up string is then returned (line 39).

Next, we add the routine that actually reads the configuration file. It is somewhat long, so we break its explanation into chunks:

41 /*
42 **  CONFIG_READ -- Read a configuration file.
43 ** Resets defaults based on what it finds.
44 **
45 ** Parameters:
46 **    fname -- The configuration file name
47 **
48 ** Returns:
49 **    zero on success, errno (!= 0) on failure
50 */
51 int
52 config_read(fname)
53    char *fname;
54 {
55    FILE *fp;
56    char buf[BUFSIZ];
57    char *cp, *ep, *lp;
58    CONF *c;
59    int len;
60
61    if (fname == NULL)
62      return EINVAL;

The config_read() routine is given the file name of the configuration file as its sole argument (line 52). If that name is NULL, this routine returns an error (line 62).

63    fp = fopen(fname, "r");
64    if (fp == NULL)
65      return errno;

We attempt to open the configuration file for reading (line 63). If that fails, an appropriate error is returned (line 65).

66    while (fgets(buf, BUFSIZ, fp) != NULL)
67    {
68      if ((cp = strchr(buf, '\r')) != NULL
69         || (cp = strchr(buf, '\n')) != NULL)
70        *cp = '\0';
71      cp = buf;

Then, in a loop (line 66), we read the configuration file one line at a time. If a line ends in a newline character (line 68), that newline character is removed prior to parsing (line 70). The pointer cp is set to start at the beginning of the line just read (line 71).

72      /*
73       * Skip comments.
74       */
75      if (*cp == '\0' || *cp == '#')
76        continue;

Here (line 75), lines that are empty and lines that begin with a comment character are skipped.

77      /*
78       * Line must be of form key=value
79       */
80      ep = strchr(cp, '=');
81      if (ep == NULL)
82        continue;

If a line lacks an equal sign, it too is skipped (line 80). This test allows item names to contain any character except an equal sign or white space characters (such as "is%50toolow?") and including comment characters (such as "bob#two").

83      *ep++ = '\0';
84      if (*ep == '\0' || *ep == '#')
85        continue;

If the value is missing (such as "maxrcpts=") the line is skipped (line 84). This might be a good place for an error message.

86      /*
87       * Strip any trailing comments.
88       */
89      if ((lp = strchr(ep, '#')) != NULL)
90        *lp = '\0';

After the item and the value have been isolated, any trailing comments are stripped (line 89).

91      cp = cleanup(cp);
92      if ((len = strlen(cp)) == 0)
93        continue;
94      ep = cleanup(ep);
95      if (strlen(ep) == 0)
96        continue;

The item name (line 91) and the value (line 94) are cleaned up with the cleanup() routine described earlier (line 21). If the item is a zero-length string, that line is skipped (line 92). If the value is a zero-length string, that line is also skipped (line 95).

 97      for (c = Cf; c->name != NULL; ++c)
 98      {
 99        if (strncasecmp(cp, c->name, len) != 0)
100          continue;
101        if (c->updated)
102          (void) free(c->defval);
103        c->defval = strdup(ep);
104        c->updated = TRUE;
105        break;
106      }
107    }
108    if (ferror(fp))
109      return errno;
110    (void)fclose(fp);
111    return 0;
112 }

In a loop (line 97), the item name is compared to the array of known item names. Note that this is a leftmost comparison, and the first match wins. For example, "aa" will match "aardvark" provided that "aardvark" is the first item in the array beginning with an "aa". If the item name is found in that list, its default value is replaced with the new value and is stored as a string in allocated memory. Clearly, this section of code could use additional error checking.

Reading a configuration file would be useless if there were no way for your program to look up values. To provide that ability, we use the config_getitem() routine:

113 /*
114 ** CONFIG_GETITEM -- get a config item's setting.
115 **
116 **      Parameters:
117 **              name -- the config item to find.
118 **      Returns:
119 **              A malloc'd copy of the value (all values
120 **              are stored as strings).
121 */
122 char *
123 config_getitem(name)
124    char *name;
125 {
126    size_t len;
127    CONF *c;
128
129    if (name == NULL)
130      return NULL;
131    if ((len = strlen(name)) == 0)
132      return NULL;
133
134    for (c = Cf; c->name != NULL; ++c)
135    {
136      if (strncasecmp(name, c->name, len) == 0)
137        return strdup(c->defval);
138    }
139    return NULL;
140 }

The config_getitem() routine takes an item name as its single argument (line 123). It then scans the list of known item names (line 128) to find one that matches (using the same logic as before). When it finds a match (line 130), it returns the current value associated with that item name. If the item is not found, NULL is returned (line 133). Note that NULL is also returned if the item name is empty (line 125) or NULL (line 123), or if strdup() cannot allocate memory.[8]

[8] If more complete error detection is desired, you can either precondition errno or pass a pointer to an error variable.

To test all this, you can add a simple main():

 1 int
 2 main(argc, argv)
 3    int argc;
 4    char **argv;
 5 {
 6    char *val;
 7    int err;
 8
 9    if (argc < 2)
10      return 0;
11    if ((val = config_getitem("maxrcpts")) != NULL)
12      printf("before: maxrcpts = %s\n", val);
13    if ((err = config_read(argv[1])) != 0)
14    {
15      fprintf(stderr, "%s: %s\n",
16        argv[1], strerror(err));
17      exit(EX_OSERR);
18    }
19    if ((val = config_getitem("maxrcpts")) != NULL)
20      printf("after:  maxrcpts = %s\n", val);
21    return 0;
22 }

After you compile this program, you can run it by specifying the name of a configuration file. Suppose, for example, your test configuration file contained this single line:

maxrcpts = 50 # max recipients

If this file were called /tmp/x.config, you would run this program this way:

% ./a.out /tmp/x.config
before: maxrcpts = 20
after:  maxrcpts = 50

10.9. In Summary

This chapter shows some of the ways to architect a good Milter, but these lessons are by no means complete. We suggest that you now examine your ideas so far and discuss them with others in your organization. From their feedback you may be surprised by assumptions you have made in your initial design that were just plain wrong. Don't code in a vacuum, especially when your design will impact the email of others.

Chapter 11. Handy Routines for the Message Body

In this chapter we conclude our description of Milters and spam email with an offering of handy routines. We begin with routines that perform the four basic forms of decoding and end with an assortment of helpful routines.

Section 11.1 shows how to parse MIME boundaries and describes the pitfalls to avoid.

Section 11.2 provides and explains code for decoding base64-encoded email and attachments.

Section 11.3 provides and explains code for decoding quoted-printable-encoded text.

Section 11.4 shows two ways to decode character entity-encoded HTML text.

Section 11.5 provides and explains code for decoding URL-encoded HTML text.

Section 11.6 provides and explains code for stripping formal and bogus HTML comments from text to simplify analysis.

Section 11.7 explains the nature of /etc/magic and discusses how it can be used to recognize file types despite spammers' attempts to hide them.

Section 11.8 shows how to use /usr/share/dict/words as a simple Bayesian filter.

Note that all examples presume that a buffer and length will be passed and the buffer rewritten in place. This may not be the best way to design a Milter for all circumstances, so be sure to revise each routine for your own needs. In particular, be aware that the current method requires the entire body or MIME part to be passed at once. But this can cause problems because the body is passed to the xxfi_body() handler function in chunks. Consider passing "state" so that a routine can pick up from where it left off. Alternatively, consider writing the chunks to a disk file (or into allocated memory if the data is not too large) to be parsed after all the chunks have arrived.

11.1. Parsing MIME-Encoded Boundaries

MIME provides the ability to send an email message that is composed of multiple parts, where each part is separated from the next by a boundary. Because each part may differ from the next, you must learn how to separate the parts before searching for spam content.

When MIME is used to encode such a message, you may find a Content-Type: header, in the header portion of the message, that indicates it is a multipart message. The Content-Type: header specifies the types of those parts and the boundary to use. For example, the following says that this is a multipart report (perhaps a bounce report) with the boundary "XXXyyy":

Content-Type: multipart/report;
        boundary="XXXyyy"

In the message body that follows this header, there are one or more parts, each divided from the next by the boundary text defined in the header (but with the quotation markes removed) and prefixed with two hyphen characters:

--XXXyy

The end of the last part of the message is indicated by the same boundary line used in the message body, but suffixed with two additional hyphen characters:

--XXXyy--

Each part of a multipart email message is like an individual miniature email message with MIME-specific headers of its own. For example, the following email message has only one part (for simplicity), and that one part is of the type HTML. Note that the numbers to the left are not part of the message.

 1  From: Bob@example.com
 2  To: you@your.site
 3  MIME-Version: 1.0
 4  Content-Type: multipart/alternative;
 5    boundary="5730887507964581281"
 6
 7  --5730887507964581281
 8  Content-Type: text/html
 9
10  <!-- ...the message in html is here -->
11  --5730887507964581281--

Note that even though this message has only one part, it still has a boundary definition, a start, and an end.

Also, because each part can have its own MIME headers (line 8), it is possible for a part to have subparts. Consider, for example, the following mail message:

 1  From: Bob@example.com
 2  To: you@your.site
 3  MIME-Version: 1.0
 4  Content-Type: multipart/alternative;
 5    boundary="5730887507964581281"
 6
 7  --5730887507964581281                    1st part starts
 8  Content-Type: multipart/alternative;
 9    boundary="4519549345002713423"
10
11  --4519549345002713423                    1st subpart starts
12  Content-Type: text/html
13
14  <!-- ...the message in html is here -->
15  --4519549345002713423                    2nd subpart starts
16  Content-Type: text/ascii
17
18  <!-- ...the message in plain text is here -->
19  --4519549345002713423--                  Subparts end
20  --5730887507964581281--                  Parts end

The message declares (line 5) that it will be divided into one or more parts separated by the boundary 5730887507964581281. But following the first boundary (line 7) are the MIME headers for that part, which declare that this part itself will be divided into still more subparts (line 9). Notice that these internal boundaries begin (line 11) and end (line 19) inside the originally defined boundary (line 20).

From this example you might expect that either recursive parsing or a push/pop approach might work to parse messages that employ MIME-encoded boundaries, but you would be mistaken. As we have mentioned throughout this book, spammers are fond of violating standards, and so the more obvious parsing techniques will eventually fail.

Consider methods used by spammers to break fancy parsing routines. They might reverse the terminating part boundaries, as in switching line 19 and line 20. Or they may omit the terminating part boundaries altogether, as in omitting line 19 or line 20 or both. Yet another spam method is to place random text following the terminating parts.

Because of these spam techniques, a better way to parse boundaries when you're screening for spam is to simply interpret each boundary as the start of a new part, whether it is a main part boundary, a subpart boundary, or even a terminating boundary. Although this approach would not be suitable if this were an HTML-compliant mail-reading program, it is more than sufficient for a Milter.

11.2. Decoding Base64 Encoding

Base64 encoding (described in RFC 2045) is used to convert binary data (such as images and sounds) into ordinary text that can be transmitted via email. A Milter must decode base64-encoded email before it can screen the message contents for spam.

The process of encoding converts 8-bit data into a stream of 6-bit bytes, where each 6-bit byte contains printable text. Because each group of three binary bytes is encoded into four text bytes, base64 encoding increases the size of that portion of an email message by 33 percent. Decoding base64 always reduces its size from four bytes to three for each group of encoded data.

Base64-encoded data is always in uniform line lengths of 72 characters each, except for the last line, which may be shorter.

The code to decode a base64-encoded portion of an email message employs two subroutines. The first subroutine decodes a single base64 encoded byte into a 6-bit byte. The second calls the first and shifts the resulting 6-bit bytes together to form the original message contents.

The first subroutine is somewhat long because each character of input is defined by RFC 2045 to represent a specific value. In addition to returning decoded values, this routine detects and reports end of file and illegal input characters. It begins by defining three macros that will be used by both subroutines:

1  # define BASE64EOF (-1)
2  # define BASE64EMPTY (-2)
3  # define BASE64WHITE (-3)

The first two macros define end-of-file (or end of buffer) as a negative 1, and any illegal input character or white space characters as negative 2. The last macro defines a negative 3 for use by the second subroutine.

 4  static int
 5  base64toval(ch)
 6    char ch;
 7  {
 8    int ret;
 9
10    switch ((int) ch)
11        {
12      case ' ': case '\t ': case '\r': case '\n':
13      case '\v': ret = BASE64WHITE; break;

Here, the white space characters are hard-coded. But note that an alternative would be to use the C language library routine isspace(). The white space characters are followed by 63 individual character definitions:

14       case 'A': ret = 0; break;
15       case 'B': ret = 1; break;
16      case 'C': ret = 2; break;
17       case 'D': ret = 3; break;
18      case 'E': ret = 4; break;
19       case 'F': ret = 5; break;
20       case 'G': ret = 6; break;
21       case 'H': ret = 7; break;
22       case 'I': ret = 8; break;
23       case 'J': ret = 9; break;
24       case 'K': ret = 10; break;
25       case 'L': ret = 11; break;
26       case 'M': ret = 12; break;
27       case 'N': ret = 13; break;
28       case 'O': ret = 14; break;
29       case 'P': ret = 15; break;
30       case 'Q': ret = 16; break;
31       case 'R': ret = 17; break;
32       case 'S': ret = 18; break;
33       case 'T': ret = 19; break;
34       case 'U': ret = 20; break;
35      case 'V': ret = 21; break;
36       case 'W': ret = 22; break;
37       case 'X': ret = 23; break;
38       case 'Y': ret = 24; break;
39       case 'Z': ret = 25; break;
40       case 'a': ret = 26; break;
41       case 'b': ret = 27; break;
42       case 'c': ret = 28; break;
43       case 'd': ret = 29; break;
44       case 'e': ret = 30; break;
45       case 'f': ret = 31; break;
46       case 'g': ret = 32; break;
47       case 'h': ret = 33; break;
48       case 'i': ret = 34; break;
49       case 'j': ret = 35; break;
50       case 'k': ret = 36; break;
51       case 'l': ret = 37; break;
52       case 'm': ret = 38; break;
53       case 'n': ret = 39; break;
54       case 'o': ret = 40; break;
55       case 'p': ret = 41; break;
56       case 'q': ret = 42; break;
57       case 'r': ret = 43; break;
58       case 's': ret = 44; break;
59       case 't': ret = 45; break;
60       case 'u': ret = 46; break;
61       case 'v': ret = 47; break;
62       case 'w': ret = 48; break;
63       case 'x': ret = 49; break;
64       case 'y': ret = 50; break;
65       case 'z': ret = 51; break;
66       case '0': ret = 52; break;
67       case '1': ret = 53; break;
68       case '2': ret = 54; break;
69       case '3': ret = 55; break;
70       case '4': ret = 56; break;
71       case '5': ret = 57; break;
72       case '6': ret = 58; break;
73       case '7': ret = 59; break;
74       case '8': ret = 60; break;
75       case '9': ret = 61; break;
76       case '+': ret = 62; break;
77       case '/': ret = 63; break;

Note that these characters are not in ASCII order, meaning that we cannot simply map their ASCII value to the correct decode value.

78      case '=': ret = BASE64EOF; break;
79      default:        ret = BASE64WHITE; break;
80         }
81         return ret;
82  }

Note also that the '=' character (line 78) is special because it marks the end of base64 data when that data does not end on a clean 4-byte boundary. Any undefined characters (line 79) are presumed to be white space.

Remember that base64 decoding always results in a reduction in the size of the decoded portion of the message. With that in mind, note that the base64decode() function, shown next, decodes base64 in place. You pass it a buffer and a length. It modifies the buffer and returns the new (shorter) length.

 1   /*
 2  ** Decode base64 in place.
 3  **
 4  **      buf -- a buffer of data to decode
 5  **      size -- the size of the data in the buffer.
 6  **
 7  **      Returns: size of the decoded buffer.
 8  **      Errors:
 9  **              If size is < 4, the buffer is
10  **              unchanged and an error of 0 is returned.
11  */
12
13  unsigned int
14  base64decode(buf, size)
15     char *buf;
16     unsigned int size;
17  {
18        char *op, *ip;
19        int buf4[4];
20        int buf3[3];

The decoding routine that your functions can call is named base64decode(). It begins (line 14) by accepting two arguments: buf (the address of the data to be decoded) and size (the length in bytes of that data). Inside base64decode() are four internal variables (line 18). The pointers ip and op point to the buffer from which the input will be read (buf) and where the output will be written to (in buf), respectively. Note that op will lag behind ip, meaning that the 4-byte buf4 holds the current base64-encoded bytes that will be decoded (into the 3-byte buffer buf3) before being written to where op points.

21         if (buf == NULL || size < 4)
22      return 0;

Before getting to the actual decoding we have provided a bit of defensive programming. If the buffer passed (line 21) is NULL or if the size of the buffer is too small (line 21) to hold a minimal base64-encoded datum, a length of zero is returned to indicate an error (line 22).

23      op = buf;
24      for (ip = buf; ip < (buf + size);)
25      {
26         int j;
27
28         buf4[0] = buf4[1] = buf4[2] = buf4[3] = BASE64EMPTY;
29         buf3[0] = buf3[1] = buf3[2] = 0;
30
31      for (j = 0; j < 4;)
32      {
33         buf4[j] = base64toval(*(ip++));
34         if (buf4[j] == BASE64WHITE)
35         {
36           if(ip >= (buf + size))
37           {
38             buf4[j] = BASE64EMPTY;
39             break;
40           }
41           continue;
42         }
43         if (buf4[0] == BASE64EOF)
44           break;
45      ++j;
46      }
47

For all bytes in the input buffer (buf), we loop (line 31) to copy four bytes of input data into buf4 for processing. Each copied byte is decoded in place from its ASCII character into its original 6-bit value (line 33) by calling base64toval() (the first subroutine). Note that the end of base64-encoded data (line 43) is indicated by an equal sign.

After the input bytes are decoded back into 6-bit values and properly stuffed into the 4-byte input buffer, we next perform a series of shifts to repack the 6-bit values into their three original 8-bit bytes:

48      if (buf4[0] == BASE64EOF || buf4[0] == BASE64EMPTY)
49        break;
50      buf3[0] = ((buf4[0] << 2));
51
52      if (buf4[1] == BASE64EOF)
53      {
54        *op++ = buf3[0];
55        break;
56      }
57      if (buf4[1] == BASE64EMPTY)
58        break;
59      buf3[0] |= ((buf4[1] >> 4));
60      *op++ = buf3[0];
61      buf3[1] = ((buf4[1] & 0xf) << 4);
62
63      if (buf4[2] == BASE64EOF)
64      {
65        *op++ = buf3[1];
66        break;
67      }
68      if (buf4[2] == BASE64EMPTY)
69        break;
70                 buf3[1] |= ((buf4[2] >> 2));
71                 *op++ = buf3[1];
72                 buf3[2] = ((buf4[2] & 0x3) << 6);
73                 if (buf4[3] == BASE64EOF)
74                 {
75                         *op++ = buf3[2];
76                         break;
77                 }
78                 if (buf4[3] == BASE64EMPTY)
79                         break;
80                 buf3[2] |= (buf4[3]);
81                 *op++ = buf3[2];
82      }

This is complex code, perhaps best explained by an example. Ignoring certain exceptions, the code shifts and bitwise ORs four 6-bit values into three bytes. Consider the following visual representation where the input is on the top:

In this illustration, the first 6-bit value is shifted left two bits and copied into the first 8-bit byte. Then the second 6-bit value is right-shifted four bits, and the remaining two bits are appended onto the end of the first 8-bit byte. This process is repeated until all 6-bit values have been shifted and copied into the correct 8-bit bytes.

The base64decode function ends after storing a zero byte (the end-of-string indicator) at the end of the rewritten buffer (line 83) and then returns the new length (line 85).

83    *op = '\0';
84
85    return (op - buf -1);
}

A simple way to test this function is with a small main():

 1  int
 2  main()
 3  {
 4    char *str = "UmU6QmVzdCBQcmljZSBPbiBWJmlhZ3JhIC0gTk8gUFJFU0NSSVBUSU9OIE5
 5  FRURFRA==";
 6     char buf[BUFSIZ];
 7     unsigned long l;
 8
 9     l = strlen(str);
10     strlcpy(buf, str, sizeof buf);
11
12     printf("length before=%lu\n", l);
13
14     if ((l = base64decode(buf, l)) > 0
15     {
16       printf("length after =%lu\n", l);
17       printf("Result='%s'\n", buf);
18     }
19     return 0;
20  }

Here, str is initialized to point to a string constant (line 4) containing some base64-encoded text that might appear as part of a Subject: header. The string constant is copied to a buffer (line 9) because string constants cannot be rewritten in place, but copies can.[1] The length of the original is printed (line 12). Then it is decoded (line 14), and the length and contents of the decoded buffer are printed (line 16).

[1] We use strlcpy() because it is the safest way to copy a string to a buffer. If your system lacks strlcpy(), you can instead use sm_strlcpy() from libsm/strl.c in the sendmail source.

./a.out
length before=68
length after =49
Result="Re:Best Price On V&iagra - NO PRESCRIPTION NEEDED"

Note that this also demonstrates that base64 decoding always reduces the size of the buffer.

11.3. Decoding Quoted-Printable Encoding

Quoted-printable encoding (described in RFC 2045) is the process of converting a character (binary or otherwise) into three printable ASCII characters. Recall from section 2.7 that spammers sometimes use quoted-printable encoding to obscure the true content of a message, and such a message must first be decoded before it can be screened for spam.

Each character to be converted to quoted-printable ends up being replaced with an equal sign followed by that character's ASCII value, which is represented by two hexadecimal characters. For example, the letter N would be converted into =4E. Note that the = character itself is always encoded as =3D, and lines longer than 72 characters (bytes) are always broken by the insertion of an = character. Any line that ends in a white space character has that character converted into quoted-printable, even if the line has fewer than 72 characters.

Decoding quoted-printable is fairly straightforward. Just look for any equal sign in the text, and perform the appropriate conversion of the hex value that follows it. Note that whenever quoted-printable is decoded, the size of the converted text is always reduced.

The following routine can be used to decode quoted-printable-encoded text.

 1  /*
 2  ** Decode quoted-printable in place.
 3  **
 4  **      buf -- a buffer of data to decode
 5  **      size -- the size of the data in the buffer.
 6  **
 7  **      Returns: size of the decoded buffer.
 8  **      Errors:
 9  **              If size is < 3, the buffer is
10  **              unchanged and an error of 0 is returned.
11  */
12  unsigned int
13  qpdecode(buf, size)
14    char *buf;
15    unsigned int size;
16  {
17    char *op, *ip, *ep;
18    int ch, save;
19
20    if (buf == NULL)
21      return 0;
22    if (size < 3)
23      return size;

Here, the qpdecode() function (line 13) is passed two arguments. The buf is the text buffer that contains the text to decode. The size is the number of bytes in the text. If buf is NULL, this function returns zero to indicate an error (line 20). If the size is smaller than a single quoted-printable object (an equal sign and two hexadecimal characters), the buffer is left unchanged (line 22).

24    op = buf;
25    ep = buf + size;
26    for (ip = buf; ip < ep;)
27    {

In line 26 through line 29, two pointers into the buffer are advanced at unequal rates in a loop. Input is read from ip, and output is written to op for as long as ip is less than the end of the buffer marked by ep.

28      if (*ip != '=')
29      {
30         *op++ = *ip++;
31         continue;
32       }

If the current input character is not an equal sign (line 28), the input character is copied to the output (line 30) and the loop repeats.

Next, in line 33 through line 47, we paste back together lines of text that were split.

33      if (*(ip+1) == '\r' && *(ip+2) == '\n')
34      {
35        ip += 3;
36        continue;
37      }
38      if (*(ip+1) == '\r')
39      {
40        ip += 2;
41        continue;
42      }
43      if (*(ip+1) == '\n')
44      {
45        ip += 2;
46        continue;
47      }

In line 33 we check for an equal sign followed by a carriage-return ('\r') and linefeed ('\n') pair. Then we check for an equal sign followed by a lone carriage return (line 38), and then for an equal sign followed by a lone newline (line 43). If any one of these is found, the input pointer is advanced and the loop continues.

48      if (isxdigit((int)*(ip+1)) &&
49        isxdigit((int)*(ip+2)))
50      {
51        save = *(ip+3);
52        *(ip+3) = '\0';
53        errno = 0;
54        ch = strtol(ip+1, NULL, 16);
55        *(ip+3) = save;
56        if (ch == 0 || errno != 0)
57        {
58          *op++ = *ip++;
59          continue;
60        }
61        *op++ = ch;
62        ip += 3;
63      }

In line 48, we look to see whether the equal sign is followed by two hexadecimal digits.[2] The character following the two hexadecimal digits is saved (line 51) and then converted to a zero byte (line 52). This prevents the strtol() C library function from interpreting past the two hexadecimal digits (as it would, for example, with =4E9B34).[3] The strol() function then converts the hexadecimal digits into the original (pre-quoted-printable) value (line 54). The zero byte is restored to its saved value (line 55), and the decoded value (if decoding was successful) is copied to the output (line 58).

[2] RFC2045 requires uppercase hexadecimal digits, so you can reasonably reject any encoding that uses lowercase hexadecimal digits.

[3] Some may find the code we use here unnecessarily complex. Note that you could also use sscanf() to accomplish this purpose, but we prefer avoiding it because it has the potential to cause a buffer overflow.

64       *op++ = *ip++;
65    }
66    *op = '\0';
67    return (op - buf);
68  }

If the equal sign is followed by anything else, the equal sign is simply copied from the input to the output (line 64).[4]

[4] Strictly speaking, the equal sign must be followed only by white space.

This qpdecode() function can be tested with the addition of a simple main() routine:

 1  int
 2  main()
 3  {
 4    char *str = "href=\"=77=77=77=2E=65=78=61=6D=70=6C=65=2E=63=6F=6D\"";
 5    char buf[BUFSIZ];
 6    unsigned long l;
 7
 8    (void) strlcpy(buf, str, sizeof buf);
 9    l = strlen(str);
10    printf("length before=%lu\n", l);
11
12    if ((l = qpdecode(buf, l)) > 0)
13    {
14      printf("length after =%lu\n", l);
15      printf("result: %s\n", buf);
16    }
17    return 0;
18  }

Here, a string constant holds a line of text that might appear in a spam message (line 4). That constant is copied to a buffer (line 8) because a string constant cannot be rewritten, but its copy can. The length of the string is printed (line 10), and then the string is quoted-printable-decoded (line 12). Its new length (line 14) and the decoded string are then printed (line 15).

% ./a.out
length before=52
length after =21
result: href="www.example.com"

Notice again that quoted-printable decoding always reduces the size of the text.

11.4. Decoding Character-Entity Encoding

Recall from section 2.5.2 that character-entity encoding provides the means to include special characters within HTML documents. Character-entity encoding allows all characters, and not just special characters, to be encoded and provides a handy way for spammers to hide otherwise obvious information. Consider this example:

[View full width]

<a href="mailto
:&#098;&#111;&#098;&#064;&#101;&#120;&#097;&#109;&#112;&#108;&#101;&#046;&#099;&#111;&#109;">

After decoding, this obscured email address is easily recognizable:

<a href="mailto:bob@example.com">

Clearly, it is necessary to decode character entities before attempting to parse HTML for clues about spam.

Character-entity encoding can be invoked in either of two forms. Both begin with an (ampersand);ampersand (&) xe"/>ampersand character and end with a semicolon character. The first form employs a keyword (such as lt). The second employs a literal #, which is followed by a three-digit decimal value. For example:

&lt;     A keyword that produces a < character
&#060;   # and 3 digits produces a < character

In this section we describe the amper program, which decodes character-entity-encoded text. It begins with two subroutines. The first converts word; xe"/>&word; expressions into the characters they represent. The list of words to be converted (line 5) is brief because spam screening is best performed on plain text and the words we exclude from this list produce non-plain-text characters.

 1 static struct words {
 2   char *word;
 3   int  letter;
 4   int  len;
 5 } Words[] = {
 6    {"quot;", '"', 5},
 7    {"lt;",   '<', 3},
 8    {"gt;",   '>', 3},
 9    {"nbsp;", ' ', 5},
10    {"shy;",  '-', 3},
11    {"amp;",  '&', 3},
12    {NULL,     0,  0},
13 };

Here, we first declare a list of words that can be converted into plain-text characters. In addition to the word (with the semicolon included) and its value, we store the length of the word (including the semicolon).

The amper() subroutine comes next:

14  static int
15  amper(p, len)
16     char *p;
17     int *len;
18  {
19     struct words *w;
20
21    *len = 0;
22
23    if (p == NULL)
24      return -1;
25    if (*p != '&')
26      return -1;

The amper() subroutine takes two arguments. The first is the substring to decode (line 16). The second is the address of an integer that will hold the length of that substring after it is decoded (line 17). If the substring is NULL (line 23) or does not begin with an ampersand (line 25), an error is returned.

27    ++p;
28    for (w = Words; w->word != NULL; w++)
29    {
30      if (strncasecmp(p, w->word, w->len) == 0)
31        break;
32    }

In line 27, the pointer is incremented to just past the ampersand in the word; xe"/>&word; expression, and then, in a loop (line 28), the word following the ampersand is compared (line 30) to our short list of known words (line 5).

If the word between the ampersand and the semicolon is not in the list of known words (line 33), we perform a couple of checks:

33    if (w->word == NULL)
34    {
35      char *cp;
36
37      if ((cp = strchr(p, ';')) == NULL)
38        *len = -1;
39      else if ((*len = cp - p) > 10)
40        *len = -1;
41
42      (*len) += 1;
43      return ' ';
44    }

If the unfound word lacks a semicolon (line 37), the word is taken as is (line 38). If the unfound word is more than nine characters in length (line 39), the word is also taken as is (line 40). Otherwise, the unfound word is replaced with a space.

45    *len = w->len;
46    return w->letter;
47  }

In line 46, the word which at this point must have been found in the list is replaced with the letter value that corresponds to it.

The decimal() subroutine is next. It converts #ddd; xe"/>&#ddd; expressions into a form suitable for screening to detect spam. Note that this subroutine would have been simpler if conversion could be arbitrary, but spam screening requires plain text only.

1  static int
2  decimal(p, len)
3     char *p;
4     int *len;
5  {
6     unsigned long l;
7     char *ep;

In line 3 and line 4, this decimal() subroutine takes two arguments: a string to decode (line 3) and a pointer to the length to use after conversion (line 4). Next, we do some error handling:

 8    *len = 0;
 9
10    if (*p == NULL)
11      return -1;
12
13    if (*p != '&')
14      return -1;
15    if (*++p != '#')
16      return -1;

Errors are returned if the passed string is NULL (line 10), or if the string does not start with an ampersand (line 13), or if the character following the ampersand is not a # character (line 15).

17    ++p;
18    if (!(isascii(p[0]) && isdigit(p[0]) &&
19        isascii(p[1]) && isdigit(p[1]) &&
20       isascii(p[2]) && isdigit(p[2]) &&
21       p[3] == ';'))
22      return -1;

In line 17, we advance the pointer to the first digit of the three-digit number. If the expression found lacks three decimal digits or if the character following the third digit is not a semicolon (line 18), an error is returned (line 22).

23    l = strtol(p, &ep, 10);
24    if (l < 0 || l > 999)
25      return -1;

In line 23, the three-digit number (in text form) is converted into an integer. If the conversion fails, an error is returned (line 25).

26    *len = 2 + (ep-p);
27
28    if (isascii(l) && isprint(l))
29      return (int)l;

In line 26, the length is adjusted to accommodate the conversion. If the integer value is that of a printable ASCII character, it is returned as is (line 28).

In the next chunk of code, integer values outside the printable ASCII range are converted (line 30) to printable or normal white space characters.

30    switch ((int)l)
31    {
32      case 8:  return ' ';
33      case 9:  return '\t ';
34      case 10:  return '\n';
35      case 13:  return '\r';
36      case 139:  return '<';
37      case 145:  return ''';
38      case 146:  return '';
39      case 147:  return '"';
40      case 148:  return '"';
41      case 149:  return '+';
42      case 150:  return '-';
43      case 151:  return '~';
44      case 153:  return '_';

45      case 154:  return '>';
46      default:  break;
47    }
48    return ' ';
49  }

In line 48, any remaining values are converted to a space character, and we are finished creating the two subroutines needed.

Now the actual decoding routine can be written. Called deamper(), it takes two arguments:

 1 unsigned int
 2 deamper(buf, size)
 3    char *buf;
 4    unsigned int size;
 5  {
 6    register char *op, *ip, *ep;
 7    int letter;
 8    int len;
 9
10    if (buf == NULL || size == 0)
11      return 0;

The first argument is the address of a text buffer that holds the text to be decoded (line 3). The second is the length of the text in that buffer (line 4). Note that if the buffer's address is NULL or if the length is zero, no conversion is done (line 10).

Next, a pointer is advanced through the buffer in a loop (line 15).

12    op = buf;
13    ep = buf + size;
14
15    for (ip = buf; ip < ep; ++ip)
16    {

Remember that because character-entity decoding never increases the size of the text, the buffer can be rewritten in place.

17    if (*ip != '&')
18    {
19         *op++ = *ip;
20         continue;
21       }

In line 17 we check to see whether the current character in the buffer is an ampersand. If it is not, this cannot be a character-entity-encoded expression, so that character is copied as is (line 19), and the loop continues to the next character. But if the current character is an ampersand, we do a little more checking:

22      if (*(ip+1) == '#' && (ep - ip) > 5)
23        letter = decimal(ip, &len);
24      else
25        letter = amp(ip, &len);

In line 22 through line 25, we look to see whether the character following the ampersand is a # character, and if the buffer has at least five characters left. If so, the decimal() subroutine is called. Otherwise, the amp() subroutine is called.

26                if (letter < 0)
27                {
28                        *op++ = *ip;
29                        continue;
30                }
31                *op++ = letter;
32                ip += len;
33    }

In line 26, we look to see whether the conversion failed. If so, the current character (the ampersand) is copied as is (line 28) and the loop continues. Otherwise, the converted character is copied (line 31), the current character pointer is advanced past the converted expression (line 32), and the loop continues (line 33).

34         *op = '\0';
35         return (op - buf);
36  }

After the contents of the original buffer have been decoded in place, the buffer is terminated with a zero value to make it into a string (line 34), and the new (possibly shorter) length is returned (line 35).

A simple main() can be used to test these routines. The following illustrates a program that takes no arguments:

[View full width]

 1  int
 2  main()
 3  {
 4    char *str = "<a href= \"http
:&#119;&#119;&#119;&#046;&#101;&#120;&#097;&#109;&#112;&#108;&#101;&#046;&#099;&#111;&#109;&#047;&#098;&#117;&#110;&#110;&#121;\">&lt;click
 here&gt;</a>";
 5    char buf[BUFSIZ];
 6    unsigned long l;
 7
 8    (void)strlcpy(buf, str, sizeof buf);
 9    l = strlen(str);
10    printf("length before=%lu\n", l);
11
12    if((l = deamper(buf, l)) > 0)
13    {
14      printf("length after =%lu\n", l);
15      printf("result=%s\n", buf);
16    }
17    return 0;
18   }

In line 4, a buffer to decode is declared as a string constant. Because string constants cannot be rewritten in place, that string constant is copied to a new buffer (line 8) and the copy is decoded (line 12). When run, this program prints the before and after lengths and the result of decoding:

% ./amper
length before=164
length after =53
result=<a href="http:www.example.com/bunny"><click here></a>

Consider modifying this main() so that the program reads from its standard output and writes the decoded result to its standard output. In that way, you can experiment with character-entity-encoded spam using this program as a stand-alone test program.

11.5. Decoding Url-Encoding

Recall from section 2.5.3 that URL encoding is the conversion of a character into its hexadecimal ASCII equivalent. URL encoding is often misused by spammers to hide message content. Consider, for example, the following obscured web reference:

href="%77%77%77%2E%65%78%61%6D%70%6C%65%2E%63%6F%6D"

A URL-encoded character consists of a % character followed by two hexadecimal digits that represent the value of the character that was encoded. For example, the two hexadecimal digits that represent the integer value of a w in ASCII are 77, so the encoded result is %77. Notice that the obscured web reference begins with three %77 expressions, and thus originally began with three w's (www).

To decode URL-encoded text you merely replace the digits with their corresponding character values and discard the % character.

1  unsigned int
2  percenthex(buf, size)
3    char *buf;
4    unsigned int size;
5  {
6    char *op, *ip, *ep;
7    unsigned l;
8    int save;
9

The percenthex() subroutine takes two arguments. The first is the address of a buffer (line 3) containing the text to decode. The second is the length of the text in that buffer (line 4).

Before moving to the next chunk of code, recall that URL decoding never increases the size of the buffer (allowing decoding to take place in a single buffer), but be aware that string constants cannot be decoded in this way.

10    if (buf == NULL || size == 0)
11      return size;

In line 10, if the passed buffer is NULL or if the size is zero, no decoding is performed. Next, a pointer into the buffer is advanced in a loop (line 15) to see whether conversion is required.

12    op = buf;
13    ep = buf + size;
14
15    for (ip = buf; ip < ep; ++ip)
16    {
17       if (*ip == '%' && (ep - ip) >= 3)
18       {
19         ++ip;
20         save = *(ip+2);
21         *(ip+2) = '\0';
22         l = strtol(ip, NULL, 16);
23         *(ip+2) = save;
24         if (l < 33 || l > 126)
25           *op++ = ' ';
26         else
27           *op++ = l;
28         ++ip;
29         continue;
30      }
31      *op++ = *ip;
32    }

In line 17, we check to see whether the current character is a percent and whether there are at least three characters left in the buffer. If so, decoding is performed. Then the pointer is advanced to the first hexadecimal character (line 19), and the two hexadecimal digits are terminated with a zero value (line 21).

Conversion from hexadecimal to integer is performed in line 22. The character that was given a zero value is restored (line 23). If the conversion failed or if the result of the conversion is a nonprintable character (line 24), the URL-encoded expression is replaced with a space (line 25). Otherwise, it is replaced with the converted value (line 27). If the current character was not a percent, it is copied as is (line 31) and the loop continues (line 32).

33        *op = '\0';
34        return (op - buf);
35  }

After the entire buffer has been decoded, it is terminated with a zero value (line 33) to ensure that it is a string. Finally, the new (possibly smaller) buffer size is returned (line 34).

A simple main() routine can be used to test this percenthex() function:

 1  int
 2  main()
 3  {
 4     char *str = "http://%77%77%77%2E%65%78%61%6D%70%6C%65%2E%63%6F%6D%2F%62%75%6E%6E%79";
 5     char buf[BUFSIZ];
 6     unsigned long l;
 7
 8     (void) strlcpy(buf, str, sizeof buf);
 9     l = strlen(str);
10     printf("length before=%lu\n", l);
11
12     l = percenthex(buf, l);
13     printf("length after =%lu\n", l);
14     printf("result='%s'\n", buf);
15     return 0;
16  }

Here, an example of URL-encoded text is declared as a string constant (line 4). But because a string constant cannot be rewritten in place, we copy it to a buffer (line 8) and convert that buffer (line 12). When run, this program prints the length before conversion, the length after conversion, and the contents of the converted buffer:

% ./a.out
length before=70
length after =28
result="http://www.example.com/bunny"

Note that you may wish to convert this program into a filter so that you can easily experiment with spam email.

11.6. Stripping Html Comments

There are two forms of HTML comments. The legal form begins with a <!-- expression and ends with a --> expression. For example:

<!-- this is a legal HTML comment. -->

Note that HTML-like expressions, such as <p> and <font color="blue">, are ignored when they are inside such comments.

<!-- <font size="+5"> is ignored. -->

The other form of HTML comment, which consists of simply placing random words inside angle brackets, is illegal, but it is frequently used by spammers to disguise text. For example:

v<bob>i<is>a<your>g<uncle>r<sothere>a

Here the letters of the word "viagra" are separated by angle-bracketed words that are not HTML keywords. Before you can screen this form of spam, you need to clean out the illegal HTML tags.

The following simple function only counts the unknown words between angle brackets, but it serves to illustrate the process. It begins by declaring many common HTML keywords:[5]

[5] A complete list can be found at http://werbach.com/barebones/barebones.html.

 1 static char *HTMLnames[] = {
 2   "a", "address", "applet", "area", "b", "base",
 3   "basefont", "bdo", "bgsound", "big", "blockquote",
 4   "body", "br", "caption", "center", "cite", "code",
 5   "col", "colgroup", "comment", "dd", "dfn",
 6   "dir", "div", "dl", "dt", "em", "embed", "font",
 7   "form", "frame", "frameset", "h1", "h2", "h3",
 8   "h4", "h5", "h6", "h7", "h8", "h9", "head",
 9   "hr", "html", "i", "iframe", "img", "input",
10   "isindex", "kbd", "li", "link", "listing", "map",
11   "marquee", "menu", "meta", "multicol", "nextid",
12   "nobr", "noframes", "noscript", "object", "ol",
13   "option", "p", "param", "plaintext", "pre", "q",
14   "s", "samp", "script", "select", "small", "spacer",
15   "span", "strike", "strong", "style", "sub",
16   "sup", "table", "tbody", "td", "textarea",
17   "tfoot", "th", "thead", "title", "tr",
18   "tt", "ul", "var", "wbr", "xmp",
19  };

Recall that an HTML keyword must immediately follow a left angle bracket, or a left angle bracket and slash combination:

<font      Font is an HTMLkeyword. 
</font     Font is an HTMLkeyword. 
< font     Font is not an HTMLkeyword. 
</ font    Font is not an HTMLkeyword. 

The code to detect non-HTML words between angle brackets begins this way:

20  int
21  ishtmlcmp(a, b)
22    void *a, **b;
23  {
24    return strcasecmp((char *)a, (char *)*b);
25  }

Here, we search the list of HTML keywords using the bsearch() C library routine. This ishtmlcmp() function is passed to the bsearch() routine to compare the word being looked up to a word in the list of words in HTMLnames (line 1).

Next comes the comments() function, which does all the work:

26  int
27  comments(buf, size)
28    char *buf;
29    unsigned int size;
30  {
31    register char *ip, *ep, *e;
32    int count = 0;
33    int save;
34
35    if (buf == NULL || size == 0)
36      return 0;

The comments() function takes two arguments. The first (line 28) is the address of the buffer that contains the text that we will scan for the bad angle-bracketed words. The second is the size of (the number of characters in) the buffer (line 29). Note that if the buffer is NULL or if the size is zero, nothing is done (line 35).

In a loop (line 38), we scan through the buffer one character at a time.

37    ep = buf + size;
38    for (ip = buf; ip < ep; ++ip)
39    {
40      if (*ip != '<')
41        continue;

We continue scanning until the whole buffer has been examined or until we find a left angle bracket (line 40).

42      ++ip;
43      if (strncmp(ip, "!--", 3) == 0)
44      {
45        /* an actual HTML comment */
46        ip += 3;
47        for (; ip < (ep-2); ++ip)
48           if (strncmp(ip, "-->", 3) == 0)
49                                        break;
50        ip += 2;
51        if (ip >= ep)
52          break;
53        continue;
54     }

In line 42, we advance to the character following the left angle bracket and then check to see whether the word that follows is a normal HTML comment (line 43). If it is, we skip to the end of the comment (line 48) and continue the loop (line 53).

55      if (*ip == '/')
56        ++ip;

In line 55, we look to see whether the character following the left angle bracket is a slash. If it is, we skip past the slash and look for an HTML keyword.

57      if (isalpha((int)*ip))
58      {
59        for (e = ip; e < ep; ++e)
60        {
61           if (! (isascii((int)*e) &&
62               isalnum((int)*e)))
63             break;
64         }
65         if (e >= ep)
66           break;
67         save = *e;
68         *e = '\0';

In line 57, we check to see whether the word following the angle bracket begins with a letter. If it does, we then check to see whether it is an HTML keyword. Note that HTML keywords must start with a letter and can contain only letters and digits (line 61). In a loop, we skip those letters and digits to find the end of the word (line 59). The character following the word is saved (line 67), and that character is changed to a zero value (line 68) to make comparisons easy.

Next, the bsearch() function is called to see whether the word just found is one of those listed as an HTML keyword:

69         if (bsearch(ip, HTMLnames,
70             sizeof(HTMLnames)/sizeof(char *),
71             sizeof(char *), ishtmlcmp) == NULL)
72               ++count;
73         *e = save;
74      }

If the word (searched for using the bsearch() function) is not a keyword (not in the words listed in line 2 through line 18), then it is counted (line 72) as a bogus keyword used to obscure spam. The saved character is then restored (line 73). Note, again, that because we rewrite the buffer in place, a string constant cannot be passed to this routine.

Next, if the word begins with anything other than a letter (line 75), we also count that as a bogus keyword (line 76). In a loop, everything is skipped up to the closing right angle bracket (line 77).

75      else
76        ++count;
77      for (; ip < ep; ++ip)
78      {
79         if (*ip == '>')
80           break;
81     }
82    }
83    return count;
84  }

Finally, when the buffer has been totally examined, the comments() function returns a count of the bogus keywords found (line 83).

Note that this function does not handle HTML tags that cross message chunks, meaning that you will need to pass it the entire message body or body part. Also note that tags that cross line boundaries may also be incorrectly handled.

A simple main() can be used to illustrate how this comments() function works:

 1  int
 2  main()
 3  {
 4    char *str = "<p>fred<bob><!-- <jill> --></bob><li></li>";
 5    char buf[BUFSIZ];
 6    int count;
 7    unsigned long l;
 8
 9    (void)strlcpy(buf, str, sizeof buf);
10    l = strlen(str);
11    count = comments(buf, l);
12    printf("count=%d\n", count);
13    return 0;
14  }

When this is run, the bogus words in line 4 will be counted. But note that there are only two of them. That is, <p> is a keyword, as are <li> and </li>, but <bob> and </bob> are bogus and are counted as such. Because <jill> appears inside a valid HTML comment, it is not counted.

% ./a.out
count=2

11.7. How to Use /etc/magic

The file /etc/magic (or /etc/share/magic) is used by the UNIX file program to identify file types based on the actual contents of files. For example, suppose you have a file that begins with the following eight characters:

%PDF-1.2

This file will be identified by the file program as a PDF (Adobe Portable Document Format) file.

Under UNIX, a PDF file can be viewed using the acroread program no matter what the file's name. Neither UNIX nor acroread cares what the file is called, only that it is a PDF file. You can call a PDF file "xxx" or "sample.pdf" or even "foo.jpg," and it will still display as a PDF file.

Microsoft Windows, on the other hand, requires the presence of a three-letter suffix that identifies the file type. A PDF file's name, under Microsoft Windows, must end in the following four characters (a dot and a three-letter suffix):

.pdf

If it ends in anything else, the file will not be recognized under Windows as a PDF file.

But when you're scanning email for spam, it is best to identify the actual file type of an attachment than to trust senders' naming conventions. Senders can lie, and when they do, viruses and worms can make their way into your system.

The information in /etc/magic can allow your Milter to correctly identify file types without regard to the file's name. Consider, for example, the following MIME header and start of an attachment:

------=_NextPart_000_0016----=_NextPart_000_0016
Content-Type: application/octet-stream;
        name="msg.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
        filename="msg.zip"

UEsDBAoAAAAAAOl0szCjiB3egHMAAIBzAABUAAAAZGV0YWlscy50

Here, the MIME headers describe this attachment as containing a file named msg.zip (a possible ZIP file). But instead of blindly trusting the sender to correctly identify the file, we will use /etc/magic to find out exactly what the attachment contains.

First, the attachment body is base64-decoded see section 11.2 to reveal the following values in part (spaced out for clarity):

P K 03 04 00 00 00 00 ...

Here, the P and K are characters, and the 03 and 04 are values. Using your favorite editor (or a paging program such as more), open the /etc/magic file and search for ZIP. You will find an entry like this:

0       string          PK\003\004      ZIP archive

Each line in the /etc/magic file specifies a test to perform. The test says to begin at an offset of zero into the file (at the P), look for a string, and compare that string to the four characters specified (a P and K followed by the values 3 and 4). If the test string matches the file's contents, it is identified (here as a ZIP archive).

We won't provide code to implement a full-blown fileprogram. Instead we show one way to implement a simplifiedfile program directly in your Milter code.

First, we declare two simple tests:

1  typedef struct {
2     char *test;
3     int len;
4     char *what;
5  } TESTS;
6  TESTS Tests[] = {
7    {"PK\003\004",  4, "ZIP archive"},
8    {"MZ",  2, "DOS executable (EXE)"},
9  };

Note that both of our tests start at an offset of zero by implication (because we exclude the offset from the structure). The first test (line 7) is the one you've just seen, a PK followed by the values 3 and 4. The second test (line 8) is the two-character test to detect a DOS executable file.

Next comes the magic() routine, which is very simple.

10  char *
11  magic(buf)
12    char *buf;
13  {
14    TESTS *t;
15
16    for (t = Tests; t->test != NULL; ++t)
17   {
18       if (strncmp(buf, t->test, t->len) == 0)
19         return t->what;
20    }
21    return "Unknown Type";
22  }

The magic() routine is passed the address of the buffer containing the file information to identify (line 12). In a loop (line 16), each test in the list of tests is compared (line 18) to the beginning of the buffer (with an implied offset of zero). If a match is found, the description for that found type is returned (line 19). Otherwise, "Unknown Type" is returned (line 21).

A simple main() routine can be used to test our simplified file program:

 1  int
 2  main(argc, argv)
 3          int argc;
 4          char **argv;
 5  {
 6          FILE *fp;
 7          char buf[BUFSIZ];
 8
 9          if (argc < 2)
10                 return 0;
11
12         fp = fopen(argv[1], "r");
13         if (fp == NULL)
14         {
15                 perror(argv[1]);
16                 exit(EX_OSERR);
17         }
18         (void) fread(buf, sizeof buf, 1, fp);
19         if (ferror(fp))
20         {
21                 perror(argv[1]);
22                 exit(EX_OSERR);
23         }
24         (void) fclose(fp);
25
26         printf("%s\n", magic(buf));
27         return 0;
28  }

When you run this main(), you specify the name of a file on the command line. The file is opened for reading (line 12), and the beginning of the file is read into a buffer (line 18). The magic() subroutine (line 26) is called, and its result is printed.

% ./a.out /tmp/test.pdf
ZIP archive

For additional information about the /etc/magic file and ideas for how to use it, read the UNIX online manual, page magic(5).

11.8. How to Use /usr/share/dict/words

The file /usr/share/dict/words (or /usr/dict/words on older systems) is a list of correctly spelled words in alphabetical order. It contains more than 25,000 words and can be used to form a simple Bayesian filter.

A Bayesian filter is most generally implemented with a database of words as keys and probabilities as values. A word such as which will have a very low probability of indicating a spam email message, whereas a word such as vi@gr@ will have a very high probability of indicating a spam email message.

With /usr/share/dict/words, a simple filter compares the number of known words in a message to the number of unknown words and bases the likelihood of spam on the resulting ratio.

Our filter begins with a simple routine called loadwords(), that loads the file called /usr/share/dict/words into memory:

 1  static char **Words;
 2  static int    Nwords;
 3
 4  void
 5  loadwords()
 6  {
 7     char *fname = "/usr/share/dict/words";
 8     struct stat st;
 9     char *buf, *cp;
10     FILE *fp;
11     int i;
12
13     if (stat(fname, &st) != 0
14       || (fp = fopen(fname, "r")) == NULL
15       || (buf = malloc(st.st_size + 1)) == NULL
16       || fread(buf, 1, st.st_size, fp) != st.st_size
17       || fclose(fp))
18    {
19       perror(fname);
20       exit(EX_OSERR);
21    }

This loadwords() function takes no arguments (line 5) and returns no value (line 4). It assigns values to two global variables: Words (line 1) is an array of pointers to the dictionary words we will load, and Nwords (line 2) is the number of pointers in the list.

In line 13 the stat() function fetches the size of the file. A buffer is allocated (line 15) to hold the file with its terminating zero byte, and the file /usr/share/dict/words is read into that buffer (line 16).

22    Nwords = 1;
23    buf[st.st_size] = '\0';
24    for (cp = buf; *cp; ++cp)
25       if (*cp == '\n')
26         ++Nwords;

In line 22 the number of words in Nwords is initialized as 1. The list of words in the buffer has a zero value appended to it to convert it into a string (line 23). The buffer is scanned (line 24), and, each time a newline character is found (line 25), the number of words is incremented (line 26).[6]

[6] The last word in the file will be overlooked if it does not end in a newline.

27    if ((Words = (char **) malloc(sizeof(char *)
28      * Nwords)) == NULL)
29    {
30      perror(fname);
31      exit(EX_OSERR);
32    }

In line 27, memory is allocated to hold the list of pointers pointing into the buffer that holds the actual words.

Next, the buffer is scanned again (line 34) looking for newline characters:

33    Words[i=0] = buf;
34    for (cp = buf; *cp; ++cp)
35    {
36       if (*cp != '\n')
37         continue;
38       *cp = '\0';
39       Words[++i] = ++cp;
40       if (i == Nwords)
41         break;
42    }
43  }

In line 38, each newline character is turned into a zero value to delimit each word, and the list of pointers is updated to point at the next word (line 39).

Next comes a simple function that will be used by bsearch() to search the list of dictionary words.

44  int
45  cmpwords(a, b)
46     void *a, **b;
47  {
48     return strcasecmp((char *)a, (char *)*b);
49  }

Note that we do a case-insensitive comparison (line 48) so that "which" will match "Which" and even "WHICH".

The next function performs the actual lookups. It is called nwords():

50  int
51  nwords(buf, size, bad)
52          char *buf;
53          unsigned int size;
54          int *bad;
55  {
56          register char *ip, *ep, *e;
57          int count = 0;
58          int save;
59
60          if (buf == NULL || size == 0)
61                  return 0;

The first argument (line 52) is the address of a buffer that contains the text to analyze. The second argument is the length of the text in that buffer (line 53). The last argument is the address of an integer to update with the count of bad words (line 54).

62    *bad = 0;
63    ep = buf + size;
64    for (ip = buf; ip < ep; ++ip)
65    {
66      if (! (isascii((int)*ip) &&
67          isalpha((int)*ip)))
68         continue;

In line 64, the buffer is scanned one character at a time looking for a letter that starts a word (line 66). Note that we presume that all words must begin with a letter.[7]

[7] /usr/share/dict/words contains entries such as 2nd, which do not begin with a letter.

69      for (e = ip; e < ep; ++e)
70      {
71         if (! (isascii((int)*e) &&
72             isalpha((int)*e)))
73           break;
74      }
75      if (e >= ep)
76        break;
77      save = *e;
78      *e = '\0';
79      if (bsearch(ip, Words, Nwords,
80          sizeof(char *), cmpwords) != NULL)
81         ++count;
82      else
83        ++*bad;
84      *e = save;
85      ip = e;
86    }

Having found the start of a word, we continue to scan (line 69) looking for the end of that word. The end will be marked by the first nonletter found (line 71). We save that nonletter (line 77) and replace it with a zero value (line 78).

The found word is then looked up in the list of words using the bsearch() C library function (line 79). If the word is in the list, the count of good words is incremented (line 81); otherwise, the count of bad words is incremented (line 83). The saved nonletter is restored (line 84) and the scan advanced to that nonletter (line 85). The scan of the buffer continues (line 86).

87         }
88         return count;
89  }

After the buffer has been completely scanned for words, the count of good words is returned (line 88). Recall that the address of an integer was passed to hold the count of bad words (line 54).

A simple main() routine can be added to test all this:

 1  int
 2  main()
 3  {
 4    char buf[BUFSIZ];
 5    int good, bad;
 6
 7    (void) loadwords();
 8    while (fgets(buf, sizeof buf, stdin) != NULL)
 9    {
10      good = nwords(buf, strlen(buf), &bad);
11      printf("%d good, %d bad\n", good, bad);
12    }
13    return 0;
14  }

This program takes no command-line arguments. You run it and then type in lines of text. The words in each line are looked up and reported:

% ./a.out
a fine fat Tuesday with that old viagra
7 good, 1 bad
Web Pay Phones are the future of Internet Communication
6 good, 3 bad

Note that /usr/share/dict/words lacks many valid words used in modern email. But if you find this approach of value, you can explore the web for public domain lists of words that might be more comprehensive and up-to-date. Note that although this technique requires a lot of tuning to be effective, you will quickly grasp how to improve it by adding likelihoods that any given word indicates spam. If you enjoy this challenge, you are encouraged to load words into a database for easier maintenance and an easier way to rate words.

11.9. More

Now that you have reached the end of this book, where do you go next? First, if you have not done so already, try to write a Milter of your own. There is no better way to learn about Milters than to stumble through the process, picking yourself up and starting over again when you fall.

The successful writing of a Milter is not a trivial task, but the completion of a successful one can be very rewarding. Why, even with just a "test" Milter (available via anonymous ftp see Appendix A), the authors have successfully screened out 99 percent of all spam received.[8]

[8] Of 600 messages received daily, 550 are screened out and 50 passed through, of which 15 per day were actual email.

Rather than write a Milter from scratch, we recommend that you obtain one whose purpose parallels your needs and then modify it. There is no need to reinvent the wheel when others have already done much of the same work.

Enjoy your fight against spam email, and thanks for your help in the war against the spammers, the scourge of the Internet.

Appendix A. Useful Source Code

The C source for the examples in this book, along with three examples of Milters, are available from two sites:

www.awprofessional.com/title/0321213335

http://spambook.bcx.org/

This source is organized into two directories:

milters contains three examples of milters.

src contains source for the examples in the chapters.

A.1. Examples of Milters

In the milters directory are three subdirectories, each of which contains the complete source for a Milter. They are as follows:

logmilter, a Milter that logs anything you want about the email that is being received or sent. It is highly configurable at compile time, allowing you to choose exactly what is logged. It is a very early Milter that contains major coding mistakes, but it is worth looking at for ideas.

slowmilt, an open source Milter that records to a database the bad behavior of email. A command-line interface allows you to query the database for summaries of this behavior. This Milter uses autoconf, so it is the most portable of the three.

test, the Milter used by the authors to test the concepts in this book. It is a real mess. It has been hacked and rehacked, with code commented out and routines rewritten without much thought other than the goal of coming up with examples. It actually works, but you should not use it as is. Instead, grab what you want from it and write your own Milter.

A.2. Chapter Examples

The src directory contains five subdirectories, one for each chapter that contains coding examples:

ch04 contains an example for posting bait news articles.

ch05 contains an example of a JavaScript email-address-obscuring routine.

ch09 contains examples of how to relate to the environment.

ch10 contains examples of fuzzy-matching and reading a configuration file.

ch11 contains examples of decoding encoded MIME and HTML parts.

Each subdirectory contains a Makefile, but be certain to edit each Makefile because they are Solaris-specific.

Bibliography

Books

Journals and Magazines

Requests for Comments

Websites

Books

Costales, Bryan, and Eric Allman . sendmail, 3rd edition. O'Reilly & Associates, 2002.

Eisenson, Marc, Nancy Castleman, and Marcy Ross . Stop Junk Mail Forever (Telemarketing & Spam, Too), 4th edition. Good Advice Pr, 2001.

Gladwell, Malcolm . The Tipping Point: How Little Things Can Make a Big Difference. Back Bay Books, 2002.

Hitchcock, Jayne A. Net Crimes & Misdemeanors: Outmaneuvering the Spammers, Swindlers, and Stalkers Who Are Targeting You Online. Information Today, 2002.

Honeynet Project. Know Your Enemy. Addison-Wesley, 2001.

Hunt, Craig . sendmail Cookbook. O'Reilly & Associates, 2003.

Land, Jonathan . The Spam Letters. No Starch Press, 2004.

Levine, John R., Ray Everett-Church, and Margaret Levine Young . Fighting Spam for Dummies. For Dummies, 2004.

Meade, Marie, and Joseph Vranich . Are We Allowed to Spam Them Back: Featuring ComputiousThe Impish Sage of the Computer Age. 1stBooks Library, 2003.

Mulligan, Geoff . Removing the Spam: Email Processing and Filtering. Addison-Wesley, 1999.

Poteet, Jeremy . Canning Spam: You've Got Mail (That You Don't Want). Pearson Education, 2004.

Schwartz, Alan, and Simson Garfinkel . Stopping Spam. O'Reilly & Associates, 1998.

Wolfe, Paul, Charlie Scott, and Mike Erwin . Anti-Spam Tool Kit. Osborne/McGraw-Hill, 2004.

Journals and Magazines

"A hidden menace." Economist. 5 June 2004:61.

Adegoke, Yinka . "Spam makes up two-thirds of all email worldwide." New Media Age. 11 March 2004:12.

"American spam." Telecom Asia. April 2004:6.

"ASAE comments on spam regulations." Association Management. May 2004:10.

Baumgartner, Jeff . "Stemming the tide: Spam is flooding e-mail inboxes faster than ever before, causing ISPs and cable operators to do whatever they can to stanch the flow well before it delivers a crippling blow." CED. July 2003:S6(3).

Bills, Steve . "As spam clogs networks, new defenses raised." American Banker. 31 Dec. 2003:12.

"Biting spam." Time. 29 Dec. 2003:164.

Brown, Jennifer . "Filtering out the true cost of spam." Computing Canada. 14 May 2004:20.

Butler, Mike . "Spamthe meat of the problem." Computer Law & Security Report. Sep. 2003:388.

Callaghan, Dennis . "Spam fight taps authentication." eWeek. 31 May 2004:16.

"CAN-SPAM Act of 2003Plan to comply now." Mondaq Business Briefing. 10 June 2004.

"Can-Spam isn't canning spam: Spam-filtering vendor Audiotrieve says most spammers are ignoring the new law's requirements." InformationWeek. 12 Jan. 2004.

Carr, Jim . "Putting the slam on spamSporting goods retailer The Sports Authority tries a variety of anti-spam techniques and finds a hosted service is least expensive and most effective." Network Magazine. 1 March 2004:46.

Chabrow, Eric . "Anti-Spam bill approved." InformationWeek. 23 June 2003:12.

Chaddock, Gail Russell . "Congress tries to 'can spam'again." Christian Science Monitor. 25 May 2004:2.

"Committee of the regions hopes to put spam ban plan into practice." European Report. 19 June 2004:412.

Cross, Lisa . "House spam measure to add more controls." Graphic Arts Monthly. June 2003:16.

Davidson, Paul . "Do-not-spam registry could result in more spam, FTC says." USA Today. 16 June 2004.

"'Do Not Spam' list pointless right now." Chain Store Age Executive Fax. 18 June 2004.

"Does can-spam act lead to more spam?" InformationWeek. 3 June 2004.

Dornan, Andy . "Change DNS, not SMTP." Network Magazine. 1 April 2004:20.

. "Who's counting?" Network Magazine. 1 Feb. 2004:22.

"'Dr. Spam' tracks down spam mail sources." GUI Program News. June 2004.

"E-mail postage." R & D. March 2004:37.

"E-Mail sanity: Take my boulder pledge." Yahoo! Internet Life. 1 June 2002.

Everett, Catherine . "Stronger laws needed to stem spam." Computer Fraud & Security. Jan. 2004:1.

Fahlman, Scott E. "Technical forum." IBM Systems Journal. 2002:759.

Farrow, Rik . "Blocking spam: ISPs could help prevent spammers from using broadband networks as spam relays by permitting e-mail from registered servers only." Network Magazine. 1 April 2004:69.

"Fighting spam, fueling growth: More productivity and increased sales will remain high priorities when choosing software in the coming year." InformationWeek. 5 Jan. 2004.

"Finland faces action from EU for failing to implement spam legislation." Nordic Business Report. 2 April 2004.

Forrester Research. "Spam and terrorism boost market for email security." New Media Age. 3 July 2003:13.

"FTC cans spam registry." eWeek. 21 June 2004:24.

"FTC seeks comments on spam regulation." The Computer & Internet Lawyer. June 2004:27(2).

"FTC shoots down spam registry, boosts authentication scheme." PC Magazine. 15 June 2004.

"FTC study finds deception in 66 percent of spam." The Computer & Internet Lawyer. July 2003:34.

"Gap in the law aids spammers." MicroScope. 1 March 2004:18.

Gilhooly, Kym . "Spam battle plans: Companies are relying on multilevel spam-fighting strategies that include e-mail filtering tools, blacklist services and employee education." Computerworld. 28 July 2003: 2122.

Gilmour, Kim . "Destroy all spam & spyware! Crush junk mail! Exterminate adware! (on your disk)." Internet Magazine. February 2004: 24-30.

Graves, Brad . "Stopping spammers as easy as winning arms race: Changing tactics, technology give bulk e-mailers an edge; 'they get trickier every day.'" San Diego Business Journal. 28 July 2003:6.

Grimes, Galen A. "Issues with spam." Computer Fraud & Security. May 2004:12.

Grossman, Wendy M. "A man, a plan, spam: A Stanford lawyer pits his job against junk e-mail." Scientific American. May 2003:24(2).

Hall, Mark . "Winning ways to stop spam: These IT organizations took different approaches, but they all successfully controlled spam." Computerworld. 1 March 2004:21(2).

Hansell, Saul . "4 rivals almost united on ways to fight spam." New York Times. 23 June 2004:C1.

Harbaugh, Logan G. "Strong spam combatants." InfoWorld. 31 May 2004:24.

Harris, Kaisha . "Taking the lid off the new CAN-SPAM law." ABA Banking Journal. March 2004:10(2).

Hayes, Frank . "New spam plan." Computerworld. 21 June 2004:54.

Heunemann, Charles . "Big bucks open a spam can of worms." Journal of Banking and Financial Services. August-Sept. 2003: 4446.

Hunter, Philip . "Hardware-based security: FPGA-based devices." Computer Fraud & Security. Feb. 2004:11.

"Internet major players team up to test anti-spam law." Internet Magazine. June 2004:11.

Johnson, Steven . "Winning the war on spam." Discover. June 2004:24.

Jones, Reneé Beasley . "Unwanted messages leave a bad taste." San Diego Business Journal. 28 July 2003: 12.

Kennedy, Charles H., and Christine E. Lyon . "The CAN-SPAM Act of 2003: A new regime for email advertising." The Computer & Internet Lawyer. March 2004:1(9).

Khong, DennisW. K. "The problem of spam law: A comment on the Malaysian communications and multimedia commission's discussion paper on regulating unsolicited commercial messages." Computer Law & Security Report. May 2004:206.

Kooser, Amanda C. "Putting the heat on spam." Entrepreneur. April 2004:34.

Livingston, Brian . "Protect your e-mail address." PC Magazine. 22 June 2004:78.

Mack, Ann M. "Maryland spam act awaits governor's signature." ADWEEK Southeast. 18 April 2004.

McAuliffe, Wendy . "Pay to get in: Attempts to block spam are hitting legitimate marketing emails too." New Media Age. 8 April 2004:23.

"Microsoft talks tough on spam: It's pushing for better industry self-regulation and lobbying legislators for tougher federal and state laws." InternetWeek. 30 May 2003.

Mullins, Robert . "Irked marketers trying to outwit spamming filters." Journal of Accounting and Public Policy. Jan.-Feb. 2003:3(2).

"New federal anti-spam law update." Mondaq Business Briefing. 21 May 2004.

"No change seen following federal anti-spam law introduction." eWeek. 11 Jan. 2004.

Oser, Kris . "Spam stings." Direct. 1 Nov. 2002.

"OUT, OUT, DAMNED SPAM: Junk e-mail accounts for roughly half of all network traffic." Business Week. 11 August 2003:54.

"Overview of current anti-spam regulations." Mondaq Business Briefing. 2 April 2004.

Plesser, Ronald L., James J. Halpert, and Stuart Ingis . "Spam a hot topic at FTC workshop and on the Hill." Mondaq Business Briefing. 8 May 2003.

Powell, William . "Spam wars." T&D. April 2003:22(3).

"Product of the year: Spam?" InformationWeek. 5 Jan. 2004.

Quain, John R. "Putting spammers in the pokey." PC Magazine. 22 June 2004:26.

Rabinovitch, Eddie . "Readers' comments on spam." IEEE Communications Magazine. Nov. 2002:20.

Richmond, Riva . "Subject: No more spam from fakes." Wall Street Journal, Eastern Edition. 9 June 2004:D9.

Roberts, John . "More than a nuisance: spam costs are high and rising." Computer Reseller News. 28 April 2003:20.

Schwartz, Evan I. "Spam wars." Technology Review. July-August 2003:32(8).

Seda, Catherine . "Stopped in your tracks? We've got 5 tips to make sure your e-mails don't get tagged as spam." Entrepreneur. May 2004:92(2).

"Sending spammers to the slammer." Security Management. May 2004:42.

"Should spam be on the menu?" Communications of the ACM. June 2004:59.

"Spam costs to hit $10 billion this year." InternetWeek. 28 April 2003.

"Spam fighters to gather at MIT: Conference comes on heels of reports saying spam increasing." InfoWorld.com. 16 January 2003.

"Spam filter features." Network Computing. 13 May 2004:66.

"Spam gets canned." Successful Meetings. Feb. 2004:18.

"Spam glossary." San Diego Business Journal. 28 July 2003:7.

"Spam rage." Communications of the ACM. April 2004:9.

"Spam's costs escalate." InformationWeek. 14 June 2004.

"Spam spoofers." eWeek. 14 June 2004:45.

"Spam summit reveals new weapons to beat junk email." New Media Age. 3 July 2003:6.

"SPIM gaining ground on spam." Insurance & Technology. June 2004:18.

Spring, Tom . "Spam wars rage: The new federal antispam law doesn't seem to be working." PC World. April 2004:24(3).

Stead, Bette Ann, and Jackie Gilbert . "Ethical issues in electronic commerce." Journal of Business Ethics. 15 Nov. 2001:75.

"SurfControl: 'Brand spoofing' spams up nearly 500 percent since January." Wireless News. 16 May 2004.

Swartz, Jon, and Byron Acohido . "E-mail carriers sign on to anti-spam efforts." USA Today. 23 June 2004.

Swartz, Nikki . "The worldwide war on spam continues." Information Management Journal. March-April 2004:16.

. "What spam law? Next up … Spim." Information Management Journal. May/June 2004:12.

"The can-spam act." Mondaq Business Briefing. 17 March 2004.

"The cost of spam." InformationWeek. 17 May 2004.

"The politics of spam." Wall Street Journal, Eastern Edition. 18 June 2004:A10.

Thilmany, Jean . "Who sent the spam?" Mechanical Engineering. June 2004:14.

"Urgent assistance needed." CFO. Winter 2003:14.

Valongo, Kye . "How to avoid junk email." Internet Magazine. Dec 2001: 99102.

"What does it mean for you?" Mondaq Business Briefing. 14 June 2004.

Willcox, Bill, and Rommert J. Casimir . "For responsible spam, pay as you go." Communications of the ACM. Nov. 2003:13.

Wollenberg, Yvonne Chilik . ". . . And the 'phish' attack like sharks." Medical Economics. 20 Feb. 2004:12.

. "Most e-mail spam lies." Medical Economics. 11 July 2003:22.

Woolley, Scott . "A new low: Spammers attack antispam crusader Ronald Guilmette." Forbes. 27 Oct. 2003:64.

Requests for Comments

RFC2822. Internet Message Format. P. Resnick, Editor, April 2001.

RFC2821. Simple Mail Transfer Protocol. J. Klensin, Editor, April 2001.

RFC2151. A Primer on Internet and TCP/IP Tools and Utilities. G. Kessler and S. Shepard , June 1997.

RFC1855. Netiquette Guidelines. S. Hambridge , October 1995.

RFC1521. MIME (Multipurpose Internet Mail Extensions): Mechanisms for Specifying and Describing the Format of Internet Message Bodies. N. Borenstein and N. Freed , September 1993.

RFC1505. Encoding Header Field for Internet Messages. D. Robinson and R. Ullmann , August 1993.

RFC1036. Standard for Interchange of USENET Messages. M. R. Horton and R. Adams , December 1987.

RFC1034. Domain NamesConcepts and Facilities. P. Mockapetris , November 1987.

RFC706. On the Junk Mail Problem. J. B. Postel , November 1975.

Websites

http://alt.kill.spammers/

http://alt.spam/

http://antispam.msexchange.org/

http://antispam.yahoo.com/domainkeys/

http://cvs.puremagic.com/viewcvs/greylisting/sendmail-milter/

http://eu.spamassassin.org/

http://lists.sans.org/pipermail/unisog/2001-November/000683.php

http://lxer.com/module/newswire/view/15364/

https://mailsift.com/

http://members.hostedscripts.com/antispam.html

http://news.admin.net-abuse.bulletins/

http://news.admin.net-abuse.email/

http://news.admin.net-abuse.misc/

http://news.admin.net-abuse.policy/

http://news.admin.net-abuse.sightings/

http://projects.puremagic.com/greylisting/

http://razor.sourceforge.net/

http://savannah.nongnu.org/projects/spamass-milt/

http://spam.abuse.net/

http://spamhaus.org/

http://spamlinks.openrbl.org/filter-server-addon.htm

http://spamsoftware.net/

http://werbach.com/barebones/barebones.html

http://wiki.apache.org/spamassassin/IntegratedInMta/

http://www.antispam.com/

http://www.appriver.com/

http://www.arachnoid.com/lutusp/antispam.html

http://www.auditmypc.com/freescan/antispam.html

http://www.azillionmonkeys.com/qed/ANTISPAM.HTML

http://www.barracudanetworks.com/

http://www.bmsi.com/python/milter.html

http://www.brightmail.com/

http://www.cauce.org/

http://www.claws-and-paws.com/spam-l/

http://www.cloudmark.com/

http://www.corriganinc.com/

http://www.cybersitter.com/antispampg.htm

http://www.dbsinet.com/

http://www.dcc-servers.net/

http://www.devguide.net/

http://www.dgl.com/docs/antispam.html

http://www.e-scrub.com/

http://www.eSafe.com/

http://www.esoft.com/

http://www.exit109.com/~jeremy/news/antispam.html

http://www.FilterMy.com/

http://www.frontbridge.com/

http://www.gfi.com/

http://www.giantcompany.com/

http://www.glenns.org/spam/sendmail.antispam.html

http://www.ietf.org/

http://www.iHateSpam.net/

http://www.ijs.si/software/amavisd/

http://www.inboxer.com/

http://www.LashBack.com/

http://www.miavia.com/

http://www.milter.org/

http://www.mozilla.org/mailnews/spam.html

http://www.onlamp.com/pub/a/onlamp/2004/06/10/milters.html

http://www.paulgraham.com/antispam.html

http://www.Postini.com/

http://www.PostiniForSmallBusiness.com/

http://www.qurb.com/

http://www.renaissoft.com/projects/maia/

http://www.samag.com/documents/s=7178/sam0206l/0206l.htm

http://www.scambusters.org/

http://www.sendmail.com/partner/resources/development/milter_api/

http://www.sendmail.net/links/

http://www.sendmail.org/antispam.html

http://www.snert.com/Software/sendmail.html

http://www.sophos.com/

http://www.spam.com/

http://www.spamalizer.com/

http://www.spambouncer.org/

http://www.spambully.com/

http://www.spambutcher.com/

http://www.spamcon.com/

http://vww.spamcop.com/

http://www.spamfighter.com/

http://www.spamlaws.com/

http://www.spammarshall.com/

http://www.SpamMonitor.com/

http://www.SpamReviews.com/

http://www.spamsoap.com/

http://www.STOPzilla.com/

http://www.symantec.com/antispam/

http://www.symantec.com/surfsafe/

http://www.throwmail.com/

http://www.turntide.com/

http://www.user-groups.net/news/editor/spam.html

http://www.winnetmag.com/

http://www.znet.com/blocked-domains.html

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

#define statement

$_ macro

$i macro 2nd

${_} macro

${auth_authen} macro 2nd

${auth_author} macro 2nd

${auth_ssf} macro 2nd

${auth_type} macro 2nd

${cert_issuer} macro 2nd

${cert_subject} macro 2nd

${cipher_bits} macro 2nd

${cipher} macro 2nd

${client_resolve} macro

${daemon_name} macro 2nd

${if_addr} macro 2nd

${if_name} macro 2nd

${i} macro

${mail_addr} macro 2nd

${mail_host} macro 2nd

${mail_mailer} macro 2nd

${msg_id} macro

${nbadrcpts} macro

${rcpt_addr} macro 2nd

${rcpt_host} macro 2nd

${rcpt_mailer} macro 2nd

${tls_version} macro 2nd

&#ddd; expressions

-D_REENTRANT 2nd

-l items 2nd

-lnsl 2nd

.forward file 2nd

/dev/null file 2nd 3rd

/etc/aliases file

     editing 2nd

     minimal 2nd

/etc/inetd.conf file, minimizing 2nd 3rd

/etc/init.d directory 2nd

/etc/init.d/apache file

/etc/magic file

     tests in

     usage 2nd 3rd 4th 5th

/etc/mail directory, running Milters under 2nd

/etc/mail/aliases file 2nd

/etc/mail/local-host-names file, setting up 2nd

/etc/mail/milters directory

/etc/rc* files

/etc/syslog.conf file

/root/bin/roll shell script

/usr/local directory

/usr/local/etc/rc.d directory

/usr/local/nmh/lib/slocal program

/usr/sbin/logrotate file

/usr/share/dict/words file

     loading 2nd

     usage 2nd 3rd 4th 5th 6th

/var/log/maillog file

/var/log/milter.log file

/var/log/syslog file

/var/run/ourmilter.sock socket

/var/run/yourmilter directory

220 SMTP success code 2nd 3rd

2yz SMTP return code 2nd 3rd

3yz return code

419 baiting

419 fraud

421 SMTP error code

4yz SMTP code 2nd 3rd

550 SMTP error

554 SMTP error

5yz SMTP reply code 2nd 3rd

<a command and web references

<base command

<script command

_domainkey domain

_srv._smtp.perm domain

~/.oksenders file

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Abort item

Abort phase

Abort section

Aborting envelopes 2nd 3rd 4th

Accept decision

Accept reply 2nd 3rd 4th 5th 6th

Advance Fee fraud

Advisory-oriented handler functions

AF_INET 2nd 3rd

Aliases

     hosts 2nd

     rebuilding database 2nd

     sendmail 2nd

Aliases file 2nd

Allocated memory, freeing up

alt.test newsgroup

amper program

amper() subroutine 2nd 3rd 4th

Apache HTTP server 2nd

Architecture 2nd

Archiving spam 2nd 3rd

argv argument 2nd

argv array 2nd

Arrays

     CONF type

     of pointers to strings

Articles

     Date: header

     Lines: header

     mandatory headers 2nd

     Message-Id: header

     posting 2nd

Asterisk (*) special character

Atkinson Caller-ID standard

Attachments

     base64-decoding 2nd

     base64-encoded 2nd

     binary

     MIME (Multipurpose Internet Mail Extensions) 2nd

     MIME headers

     quoted-printable encoded 2nd

Authentication

autoconf 2nd

autoheader

automake

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

ba.test newsgroup

Baby-sitting script 2nd 3rd

Background, running Milters in 2nd 3rd

backlog argument

Bait machine

     choosing platform 2nd 3rd 4th

     compiler choice 2nd

     configuring sendmail 2nd 3rd 4th 5th

     database support 2nd

     excluding non-email ports 2nd 3rd

     forwarding copies of good email to 2nd

     installing Milters library

     network connections

     posix threads 2nd

     rebooting 2nd

     scanning 2nd

     sendmail version

    setting up

         DNS records 2nd 3rd 4th

         logging 2nd 3rd

Bank card information theft 2nd

Base64 encoding

     decoding 2nd 3rd 4th 5th 6th 7th 8th

     marking end of data

Base64-decoding attachments 2nd

Base64-encoded data

Base64-encoding

     attachments 2nd

     Subject: headers 2nd

base64decode() function 2nd

base64total() subroutine

Bayesian filters 2nd 3rd 4th 5th 6th 7th

Berkeley database 2nd

bg() function 2nd 3rd

Binary attachments

Bitwise OR (|)

Blacklisting

Blocked senders

Body 2nd 3rd 4th

     deleting

     Milter replies per chunk

     modifying

     replacing 2nd 3rd 4th

     routines 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th 35th 36th 37th 38th 39th

Body item

Body section

Bounce address

Bounce email

Bounce information, multiline

Bounce messages 2nd

Bounce reply envelope sender address

Bouncing email 2nd

Boundary 2nd 3rd

BSD method used to halt Milter

bsearch() C library function 2nd

Buffers

     containing replacement body

     for incoming message 2nd

     length of

Bulletproof multithreaded code

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

C language compiler, keeping up-to-date 2nd

C language programs main() function 2nd

Cable accounts and nonfixed IP addresses

Camouflaging HTML body 2nd 3rd 4th 5th

Certificates

cf variable

cf/cf directory

Chapters source code

char *addr; regular expression

Character-entity encoding 2nd

     decoding 2nd 3rd 4th 5th 6th 7th 8th

     keywords 2nd 3rd

     literal # 2nd 3rd

     web references

Characters

     character-entity encoding 2nd 3rd 4th 5th 6th 7th 8th

     converting to hexadecimal ASCII equivalent 2nd 3rd

     quoted-printable encoding 2nd 3rd 4th 5th

     URL-encoding

chdir() function 2nd

check_mail rule set

check_rcpt rule set

check_relay rule set

Child

Chunks (of body)

     concatenating

     containing too many characters with high bit set 2nd

     counting number of bytes in

     Milter replies per

     reviewing 2nd 3rd 4th 5th

     unsigned char* type

     writing to disk file

Cleanup section

cleanup() routine 2nd 3rd

Clickable link

Close item

CNAME records

     adding 2nd

     infinite loops

     leading to other CNAME records

     URLs and 2nd

Commands, case insensitive

Comments

     .forward file

     breaking up words with 2nd

     HTML 2nd 3rd

     intervening newlines 2nd

     spam aliases

     unbalanced angle brackets

     unknown HTML keyword in angle brackets acting like

     URLs used as 2nd

comments() function 2nd

common characteristics

     ctx argument

     declaring

     types 2nd

Compilers, choosing 2nd

Complex data, storing

Concatenating chunks

config_getitem() routine 2nd

config_read() routine 2nd

Configuration files

     # comment character

     cleaning spaces around strings 2nd

     looking up values 2nd

     rereading

     routine actually reading

     running

     simplest form

Configurations

     dynamic 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

     static

configure script, creation of

configure.ac template file

confINPUT_MAIL_FILTERS mc macro

confMILTER_MACROS_CONNECT mc macro

confMILTER_MACROS_ENVFROM mc macro

confMILTER_MACROS_ENVRCPT mc macro

confMILTER_MACROS_EOM mc macro

confMILTER_MACROS_HELO mc macro

Connect item

Connect phase and Milter replies 2nd

Connect section

Connecting host

     name of

     result of lookup of name

Connection-context type

Connection-oriented handler functions

Connection-oriented resources, deallocating

Connection-persistent information, deallocating

Connection-specific macros

Connections

     affecting

     behavior 2nd

     cipher suite used for

     cleanup 2nd 3rd 4th

     deferring if host cannot be looked up

     defining context

     disconnecting by rejecting with tempfail

     initializing and timeout

     keeping track of

     listening for incoming 2nd

     logging

         number of envelopes processed during connection 2nd

         total duration 2nd

     Milters rejecting

     rejecting

     reviewing 2nd 3rd 4th 5th 6th

     skipping checks

     termination 2nd 3rd 4th

Content-Transfer-Encoding: header

Content-Type: header

Continue reply 2nd 3rd 4th 5th 6th 7th

cp pointer 2nd 3rd

Credit card information theft 2nd

ctx context pointer 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th

Custom-added headers

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Daemons

Data access routines 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th

DATA phase

Data portion

     body

     headers

     reviewing 2nd 3rd 4th 5th 6th

DATA SMTP command 2nd 3rd

Databases, support for 2nd

Date: header 2nd

dealloc envelope() handler function

dealloc_connection() routine

dealloc_envelope() routine 2nd

Deallocating connection-oriented resources

deamper() routine 2nd

     default level

decimal() subroutine 2nd 3rd 4th

Decoding

     base64 encoding 2nd 3rd 4th 5th 6th 7th 8th

     character-entity encoding 2nd 3rd 4th 5th 6th 7th 8th

     quoted-printable encoding 2nd 3rd 4th 5th

     URL-encoding 2nd 3rd

Default SMTP replies 2nd

Deferring envelopes

delay_checks FEATURE

df queue file, truncating

Dial-up accounts and nonfixed IP addresses

Dictionaries 2nd 3rd 4th 5th 6th

dig program

Directories

     accepting core dumps

     defining with preprocessor #define directive

     for Milters

Discard reply 2nd 3rd 4th 5th 6th

Discarding envelopes

Disguising Subject: header 2nd 3rd

Distributed model

dn_expand() function

dn_skipname() function

DNS (Domain Name Service)

     adding CNAME records 2nd

     TXT record

DNS (Domain Name Services)-based services

DNS and BIND (Albitz and Liu)

DNS records

     domain versus subdomain 2nd

     setting up 2nd 3rd 4th

     wildcard 2nd

Domain Keys standard

Domain names

     case insensitive

     registering for testing

Domain records, controlling

Domains

     adding new host 2nd

     enclosed in quotation marks 2nd

     versus subdomains 2nd

DSL (digital subscriber line) and nonfixed IP addresses

dsn argument

DSN reply code, changing

Dynamic configurations 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

EHLO command

     arriving at unexpected times

     requiring before MAIL FROM: command

     reviewing 2nd 3rd 4th 5th

     sending site

EHLO/HELO phases and Milter replies

Email

     composed of multiple parts 2nd 3rd

     delivered using order specified by MX records

     detecting when received from MX servers 2nd 3rd 4th 5th

     false positives

     fictional persons created to receive 2nd 3rd

     graylisting 2nd 3rd

     literal "+" character inserted in user part 2nd 3rd

     maximum size

     policies for inbound and outbound

     protecting good 2nd

     screening inbound and outbound

Email addresses

     as plain text outside mailto: command

     determining who may have sold

     encoded @ character

     expressing abstractly as possible 2nd

     innocent person's as bounce address

     JavaScript obscuring

     masking URLs

     posting to newsgroup 2nd 3rd 4th 5th 6th 7th 8th

     reader cutting and pasting

     setting up for spam 2nd 3rd

     showing ultimate recipient 2nd

     spam email

     Usenet

     verifying 2nd

Email fraud

     bank or credit card information theft 2nd

     Nigerian fraud

     password theft

     viruses and worms 2nd

Email readers

Empty addresses

Encryption key length

End of envelope 2nd 3rd 4th 5th 6th 7th 8th

End of headers section

End of message section

End users

     internal customer as 2nd

     modeling 2nd

     outside world as

End-of-body semaphore

End-of-envelope cleanup 2nd

End-of-message phase timeout

End-of-message routine and Milter replies 2nd

Envelope recipients

     accepting 2nd

     adding recipient 2nd 3rd

     addresses 2nd 3rd

     delivery agent name

     maximum number

     number of

     possible replies

     processing

     recipient address

     rejecting

     relay host

     removing 2nd 3rd 4th 5th

     removing address from list 2nd

     rule sets and aliasing modifying 2nd

Envelope senders

     accepting

     address 2nd 3rd 4th 5th

     authentication

     bounce messages

     deferring

     delivery agent name

     discarding 2nd

     error notification sent

     MTAs rejecting

     rejecting 2nd

     relay host part

     reviewing

     saving address 2nd

     source of spam email

Envelope-handling functions, deallocating resources

Envelope-specific information, deallocating

Envelope-specific macros

Envelope-specific resources, deallocating 2nd 3rd 4th

Envelopes

     aborting 2nd 3rd 4th

     accepting

     arbitrary number of recipients

     DATA headers

     DATA portion 2nd

     deferring

     direct access to raw information

     discarding 2nd

     end of 2nd 3rd 4th 5th 6th 7th 8th

     falsifying sender address 2nd

     headers

     identifying

     MAIL FROM envelope sender

     MAIL FROM: command

     private data

     RCPT TO envelope recipients

     rejecting 2nd 3rd 4th

Envfrom item

Envrcpt item

Eoh item

Eom item

ep pointer

errno variable 2nd

errno.h included file

Error messages, name of program for use in

Errors

     recording

     smfi_setconn() routine

EX_SOFTWARE

EX_UNAVAILABLE

Exception process, whitelisting

exit(2)

EXPN command 2nd

Exporting shell macros

Extended SMTP commands 2nd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Fake recipients

     automatic addresses

     creation of 2nd 3rd

     names corresponding to real services

     non-user names

     UNIX administrative names

Fallback hosts and mail 2nd

False positives

Falsifying envelope sender address

Fatal (nonrecoverable) errors

Feedback

     human 2nd 3rd

     possible mechanisms 2nd 3rd 4th

file (for viewing local files)

file program 2nd

     simplified

     testing

Files, identifying types by file contents 2nd 3rd 4th 5th

Financial institutions 2nd

finger program 2nd

Firewalls 2nd

Fixed IP addresses

Flags and smfiDesc structure 2nd

flags item

fork() function

FreeBSD

     /root/bin/roll shell script

     copying startup scripts

freehostent() function

ftp (File Transfer Protocol)

ftp daemon

ftp host name

Fudgenews

     missing command-line switches

     opening connection to news posting host

     post() subroutine 2nd

     switches

Functions and Milter phases

Fuzzy address matching 2nd 3rd

fuzzy() subroutine

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

g.msn.com websites

gethostbyname() function

getipnodebyname() C library function 2nd 3rd

GETLONG

getpeername(3)

GETSHORT

getuid function

GNU autoconf suite 2nd

Gorillas 2nd 3rd

Graylisting 2nd 3rd

greetpause FEATURE

Grokking site 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th

GROUP command

Group ID

Guerrillas 2nd 3rd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

haddr argument

Handle signals 2nd 3rd

Handler functions 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th 35th 36th 37th 38th 39th 40th 41st 42nd 43rd 44th 45th 46th 47th 48th 49th 50th 51st 52nd 53rd 54th

     advisory-oriented

     belonging to connection

     connection-oriented

     message-oriented (envelope-oriented)

     recipient-oriented

     smfi data access routines 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th

     xxfi_ prefix

     xxfi_ prefix for names

header item

Header sender address

Headers 2nd 3rd 4th 5th

     adding 2nd 3rd 4th 5th

     appearing multiple times

     case insensitive names

     changing 2nd 3rd 4th

     count of

     custom-added

     end of 2nd 3rd 4th 5th

     illegal values

     index into list of existing

     inserting in messages 2nd 3rd 4th

     Milters 2nd

     MIME

     missing

     modifying

     multiple lines 2nd

     name

         in form of string 2nd 3rd

     name portion

     ordering

     prefixed with literal X-

     recording presence

     rejecting 2nd 3rd

     removing 2nd 3rd 4th 5th

     reviewing 2nd 3rd 4th 5th 6th 7th

     RFC standards 2nd

     trace-type

     tracking of offset

     user-added

     value 2nd 3rd 4th

     value portion

Headers section

HELO command

     arriving at unexpected times

     requiring before MAIL FROM: command

     reviewing 2nd 3rd 4th 5th

     skipping

helo item

HELO/EHLO section

HELO/EHLO SMTP command 2nd

hicount counter

Hijacked PCs 2nd

host argument

Host names

     accepting

     adding to spam database 2nd

     case insensitive

     for posting to Usenet

     looking up

         for MX records

         records

     random word masquerading 2nd

     string containing

     validity

host.domain form

host.domain part, expressed as IP number

Hosts

     adding to existing domain 2nd

     comparing IP numbers 2nd 3rd

     disguising name

     enclosed in quotation marks 2nd

     IP number of connecting

     MX records 2nd

     names of

     redirecting site 2nd

     using other aliases 2nd

HTML

     bogus keywords 2nd

     camouflaging body 2nd 3rd 4th 5th

     character-entity encoding 2nd

     clickable link in code

     commands and URLs referenced case insensitive

     commands and web references

     comments 2nd 3rd

     declaring common keywords

     detecting non-HTML words

     documentation

     intervening newlines in comments 2nd

     keywords

     order of encoding

     unknown keywords acting like comments

     URL encoding 2nd

     valid keywords

HTML (Hypertext Markup Language)-enabled email readers

HTML comments

     illegal form

     legal form 2nd

     stripping 2nd 3rd 4th 5th 6th

HTML documents and special characters 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

HTML-capable mail programs 2nd

http (Hypertext Transport Protocol)

HTTP listener

https (HTTP with Secure Sockets Layer, or SSL)

Human feedback 2nd 3rd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

ident lookup

identd

Idle, Eric

if clause

IMAP (Internet Message Access protocol) email readers

include/milter.h file

inet6: prefix 2nd

inet: prefix 2nd 3rd 4th

inetd daemon 2nd

inetd.conf file, commenting out lines

INPUT_MAIL_FILTER mc command

Installing Milter library

Internal PCs, risks imposed by

IP addresses

     associated with receiving (listening) interface

     fixed

     nonfixed 2nd

     reverse look up

IP numbers

     assigning multiple to network interface

     comparing 2nd 3rd

     connecting host

     decimal or hexadecimal

     rejecting connections from

ip pointer 2nd

IPv4 socket 2nd

IPv6 socket 2nd

ishtmlcmp() function

ISPs 2nd

isspace() C language library routine

Items and zero-length string

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

JavaScript, obscuring email address

JavaScript.Encode URLs

Jones, Terry

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Keystroke logging 2nd

Keywords

     character-entity encoding 2nd

     valid HTML

Kill (Ctrl+C) keyboard shortcut 2nd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Large ISPs

     policies

     spam 2nd

     TXT record

     whitelisting

LDAP (Lightweight Directory Access Protocol)

Leftmost comparison

len argument

libmilter directory

libmilter RPM (Redhat Package Manager)

libmilter.h file

libmilter/docs documentation 2nd

libmilter/mfapi.h included file

Library routines, reporting errors to 2nd 3rd

Lines: header

Linux

     /usr/sbin/logrotate file

     copying startup scripts

     method used to halt Milter

listen(2) queue 2nd

listen(3)

listen(3) queue

Listeners

     daemons as

     eliminating unwanted

Listening connection, establishing

Listening daemon, name of

Literal character-entity encoding 2nd

loadwords() routine 2nd

local-host-names file 2nd

local3 logging facility

local: prefix, defining 2nd

localhost loopback interface 2nd

Log files. limiting size

logadm program

Logging

     connections

     defensive programming

     facilities available for nonsystem programs

     Milters 2nd

     number of envelopes processed during connection 2nd

     overview 2nd

     queue identifiers

     recording every connection

     rotating logs

     sendmail 2nd 3rd

     setting up 2nd 3rd

     setting up local# 2nd

     Solaris

     total connection duration 2nd

logmilter Milter

Logs, rotating

Lost productivity

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

m4 Build file

Macros

     adding to default list 2nd

     connection-specific

     defining 2nd 3rd

         end-of-file (or end of buffer)

         illegal input character or white space characters

         needed

     envelope-specific

     fetching values 2nd

     name whose value is looked up 2nd

     passing sendmail macros to Milter

     persisting

     xxfi handler function return values

     xxfi_connect() handler function 2nd

     xxfi_envfrom() handler function

     xxfi_envrcpt() handler function

     xxfi_eom() handler function

     xxfi_helo() handler function

magic() routine 2nd

mail facility

Mail fallback hosts 2nd

MAIL FROM: command 2nd 3rd 4th 5th 6th 7th

     calling Milters

     ESMTP (Extended SMTP) arguments

     Milter replies 2nd

     reviewing 2nd 3rd 4th 5th 6th 7th

mail host name

mailto: command 2nd

     searching for

main() function 2nd

     arguments

     changing default socket time out

     minimal 2nd 3rd 4th

     routing

Makefile 2nd

Makefile.am template file

malloc

Masking signals

Masking web addresses 2nd 3rd

maxrcpts item

maxsize item

mc configuration file

    adding

         macros to default list

         Milter support

     adding macros

     delay_checks FEATURE

     editing 2nd

     naming Milters 2nd

     smfi_setconn() routine

mc macros

Memory

     allocating for strings

     freeing allocated

Memory leaks, avoiding 2nd 3rd

Message-Id: header 2nd 3rd

Message-oriented (envelope-oriented) handler functions

Messages

     aborting

     accepting 2nd

     adding header 2nd 3rd 4th

     body

     bouncing 2nd

     changing

     data portion

     discarding

     headers 2nd 3rd 4th 5th

     lacking Message-Id: header

     large chunks of random text 2nd

     left with no recipients

     multiple Milters reviewing 2nd 3rd

     quarantining 2nd 3rd 4th

     rejecting 2nd

     reviewing data portion 2nd 3rd 4th 5th 6th

     to be logged

MI_FAILURE value 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th

MI_SUCCESS value 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

Microsoft Windows

Milter header file

MILTER macro

Milter-library

     declaring Milter phases

     installing

     overview 2nd 3rd

     registering smfiDesc structure with 2nd

     routines 2nd

     smfi_prefix 2nd

     version

     xxfi_prefix

milter.init script 2nd 3rd

MILTERARGS macro

MILTERDIR macro

MilterEmailAddress variable

MILTERKILL macro

MILTERRUN macro

Milters

     abort phase

     aborting

     adding support in sendmail 2nd 3rd

     architecture 2nd

     as recipient

     baby-sitting script sleep time

     beginning execution

     capabilities 2nd

     command-line arguments

     communicating with sendmail

     configuration file 2nd

     considering portability early 2nd

     database

     debugging level 2nd 3rd

     declaring phases acceptable or ignorable

     default wait

    defining

         macros

         name

     directory for

     distributed model

     dynamic configurations 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

     email address stored in variable

     failing 2nd

     functions for phases

     headers 2nd

     immediate abort

     interweaving calls to many 2nd

     killing 2nd

     learning from human input 2nd

     libraries needed 2nd

     listening

     logging 2nd

     macros for passing sendmail macros to

     main() function 2nd

     method used to halt

     multiple reviewing message 2nd 3rd

     multithreaded 2nd

     name of 2nd

     non-root user 2nd 3rd

     order called 2nd 3rd

     orderly shutdown

     phrases accepted or ignored 2nd 3rd 4th

     port numbers listening on

     post-connection cleanup

     preventing from running as root

     private variables

     process phases

     queue identifiers

     quitting 2nd

     real user ID

     registering with library

     regular-expression rules

     rejecting connection

     rejecting SMTP command 2nd 3rd

    replies

         at end-of-message routine 2nd

         for Connect and Ehlo/Helo 2nd

         per chunk

         to MAIL FROM: command 2nd

         to RCPT TO: command 2nd

     required initialization elements 2nd

     return values from multiple

     reviewing recipients 2nd

     role of 2nd

    running

         by root

         in /usr/local directory

         in background 2nd 3rd 4th

         in foreground for testing

         under /etc/mail directory 2nd

    sendmail

         point of view 2nd

         supporting

     SMTP DATA replies 2nd 3rd

     sockets

     source code examples 2nd

     starting 2nd 3rd 4th 5th

     startup script 2nd 3rd 4th 5th

     static configurations

     status or startup files

     stopping 2nd 3rd 4th 5th

     syslog records

     T parameter

     tempfailing SMTP command 2nd 3rd

     time before restarting

     timeout on amount of time

     UNIX domain socket

     updating knowledge

     use of multiple

     user ID 2nd 3rd 4th

     waiting for connection from sendmail 2nd

     where to run 2nd 3rd

milters directory 2nd

MILTERSEMAPHORE macro

MIME (Multipurpose Internet Mail Extensions)

     attachments 2nd

     Content-Type: header

     headers

     headers and attachments

MIME-encoded boundaries, parsing 2nd 3rd

MIME-encoded messages

Missing headers

Monty Python's Flying Circus,

msg argument 2nd

MTAMARK (Marking Mail Transfer Agents) x standard

MTAs (mail transfer agents)

     multiline reports

     rejecting envelope sender

Multiline replies 2nd

Multipart messages 2nd 3rd

Multithreaded mode, launching 2nd

Multithreaded operation 2nd

Multithreaded program

     deallocating resources

     signals

MX (mail exchange) servers

     anticipating 2nd 3rd 4th 5th

     deferring envelopes

     detecting when mail received from 2nd 3rd 4th 5th

     looking up 2nd 3rd

     relaying spam through 2nd 3rd

     unable to run spam filters on

MX host, spam email sent directly to highest-numbered

MX records

     adding 2nd

     controlling domain records

     extracting host name associated with

     looking up 2nd

     printing

     trapping IP number subterfuge

mx() function

     testing 2nd

mySQL

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

name argument

Name item

Named pipes 2nd 3rd

Named sockets 2nd

Network connections and bait machine

newaliases command

newaliases program, running

Newline characters 2nd

News server

     acknowledging posting

     allowing posting

     host sending greeting

Newsgroups

     posting to

     to post to

     validating existence

Nigerian fraud

nmap program 2nd

Non-root user 2nd 3rd

NOQUEUEID string

Nwords global variable

nwords() function 2nd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

okaymail user

Old MTA addresses 2nd

op pointer 2nd

Operating systems

     posix threads

     thread-safe C language library 2nd

Organized crime and Nigerian fraud

ourmilt.run script 2nd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Parent

Parsing MIME-encoded boundaries 2nd 3rd

Passing state

Passwords 2nd

Pdata structure

pdatap pointer

percenthex() subroutine 2nd 3rd

Phone numbers

     detection of

     whitelisting

Phonemes

Platform, choosing for bait machine 2nd 3rd 4th

Plus addressing 2nd 3rd

Pointers, storing single

POP (Post Office Protocol) email readers

Portability 2nd

Ports

     excluding non-email 2nd 3rd

     list of numbers

     unnecessary services listening

Posix threads 2nd

POST command

post() subroutine 2nd

Post-connection cleanup

Posting

     articles 2nd

     to Usenet news groups 2nd 3rd 4th 5th 6th 7th 8th

Preventive measures

     EXPN command 2nd

     telling users about plus addressing 2nd 3rd

Printer hosts name

printf() statements

Printing MX records

priv pointer

priv variable

priv->qid variable

Private data

     allocating memory to pointer

     fetching 2nd

     registering 2nd 3rd

Private variables

procmail program

Programs run as root

Protecting good email 2nd

Protocols

     default

     enclosed in quotation marks 2nd

     identifying in URL 2nd

     not actually present with each URL

pthread_sigmask() library routine

Pthreads [See posix threads]

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

qpdecode() function 2nd 3rd 4th

Quarantine reply

Quarantining messages 2nd 3rd 4th

Queue identifiers 2nd

Queued messages

     not seen by sendmail 2nd 3rd

     sendmail identifier

QUIT command

Quitting Milters 2nd

Quoted-printable encoded attachments 2nd

Quoted-printable encoding, decoding 2nd 3rd 4th 5th

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Random text 2nd

RCPT TO: command 2nd 3rd

     calling Milters

     Milter replies to 2nd

     reviewing 2nd 3rd 4th 5th 6th

RCTP TO: command

rd.yahoo.com website

README file

Real user IDs

Rebooting bait machine 2nd

Received: headers 2nd 3rd

Receiving (listening) interface

Recipient address @ character

Recipient-oriented handler functions

Recipients

     accepting

     counting number of 2nd

     number of bad

     rejecting

Recording errors

Redirect servers 2nd

Redirecting site 2nd

regerror() function

regexec() C library routine

Registering private data 2nd 3rd

Regular-expression evaluation

Reject decision

Reject reply 2nd 3rd 4th 5th 6th

Rejecting

     connections from IP numbers

     envelopes

     spam 2nd 3rd

Relaying through MX (mail exchange) 2nd 3rd

res_query() function

Resources

     deallocating

         connection-oriented

         envelope-handling functions

     failure to deallocate temporary

return keyword 2nd

Return values from multiple milters

Reverse DNS

Reverse lookup of IP address

Reviewing connections 2nd 3rd 4th 5th 6th

Reviewing SMTP HELO/EHLO 2nd 3rd 4th 5th

RFC1413 validation

Risks with internal PC customers

root user

     delivering mail for

     executing programs

     preventing Milters from running as

     programs run as

Rotating logs

Routers 2nd

Routines

     /etc/magic file usage 2nd 3rd 4th 5th

     body 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th 31st 32nd 33rd 34th 35th 36th 37th 38th 39th

    decoding

         base64 encoding 2nd 3rd 4th 5th 6th 7th 8th

         character-entity encoding 2nd 3rd 4th 5th 6th 7th 8th

         quoted-printable encoding 2nd 3rd 4th 5th

         URL-encoding 2nd 3rd

     parsing MIME-encoded boundaries 2nd 3rd

     passing state

     stripping HTML comments 2nd 3rd 4th 5th 6th

Rule sets

     disposing of message

     rejecting connection

runas() function 2nd 3rd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Sanity process and whitelisting

Semaphore file

Sending site

     connection cleanup 2nd 3rd 4th

     EHLO command

sendmail

     220 greeting 2nd 3rd

     220 SMTP code

     adding Milter support 2nd 3rd

     aliases 2nd

     buffer for incoming message 2nd

     configuring 2nd 3rd 4th 5th

     connection request from sending host

     getpeername(3)

     greetpause FEATURE

     header added by Milter

     host and RFC1413 validation

     interweaving calls to many Milters 2nd

     killing and restarting

     log records

     logging 2nd

     mail facility

     mc macros

     minimal aliases file

     multiple Milter programs

     plus addressing 2nd 3rd

     point of view on Milters 2nd

     queried files not seen by 2nd 3rd

     rejecting envelope recipient

     reverse lookup of IP address

     setting up local-host-names file 2nd

     SMART_HOST option

     source directory

     version supporting Milters

     where and how to deliver email

sendmail configuration file

     Milter.LogLevel option 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th

     order Milters called 2nd 3rd

sendmail macros

     fetching values 2nd

     xxfi_connect() handler function 2nd

     xxfi_envfrom() handler function

     xxfi_envrcpt() handler function

     xxfi_helo() handler function

sendmail Milters

sendmail website

sendmail.cf file

sendmail.mc file 2nd 3rd

Services

     screening URLs

     unnecessary listening on ports

setsid() function

setting level 2nd 3rd

sfsistat type 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th

Shell macros 2nd

Shutting down in stages

sig() function

SIGHUP signal 2nd 3rd

SIGINT signal 2nd 3rd

sigmarkreadconf() function

sigmarkrereadconf() function 2nd

Signals 2nd

Signature detectors, attempting to fool 2nd

SIGPIPE signal 2nd 3rd

SIGTERM signal 2nd 3rd

SIGUSR1 signal 2nd 3rd

SIGUSR2 signal

sizeof (3) integer

Sleepycat DB 2nd

slocal program 2nd

slowmilt open source

Small businesses and whitelisting

SMART_HOST option

smfi data access routines 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th

smfi modifier routines 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th

smfi routines 2nd

smfi_addheader() routine 2nd 3rd 4th 5th

     ctx connection-context pointer

smfi_addrcpt() routine 2nd 3rd 4th

smfi_chgheader() routine 2nd 3rd 4th 5th 6th

smfi_delrcpt() routine 2nd 3rd 4th

smfi_getpriv() routine 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

smfi_getsymval() routine 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th

smfi_insheader() routine 2nd 3rd 4th 5th

smfi_main() routine 2nd 3rd 4th 5th 6th

smfi_opensocket() routine 2nd 3rd 4th 5th 6th 7th 8th 9th

smfi_prefix 2nd

smfi_progress() routine 2nd 3rd 4th 5th

smfi_quarantine() routine 2nd 3rd 4th

smfi_register() routine 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th

smfi_replacebody() routine 2nd 3rd 4th

smfi_section() routine 2nd 3rd

smfi_setbacklog() routine 2nd 3rd

smfi_setconn() routine 2nd 3rd 4th 5th 6th 7th 8th 9th

smfi_setdbg() routine 2nd 3rd

smfi_setmlreply() routine 2nd 3rd 4th

smfi_setpriv() routine 2nd 3rd 4th 5th 6th 7th 8th

smfi_setreply() routine 2nd 3rd 4th 5th 6th

smfi_settimeout() routine 2nd 3rd 4th

smfi_stop() routine 2nd 3rd

SMFI_VERSION literal expression

SMFI_VERSION macro

smfiDesc structure 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th

     declaring xxfi_ functions

     flags 2nd

     global or local

     items 2nd

     position of xxfi_connect() handler function

     registering

         with milter-library 2nd

         with smfi_register() function

SMFIF_ADDHDRS flag 2nd 3rd 4th 5th 6th

SMFIF_ADDRCPT flag 2nd 3rd 4th 5th 6th

SMFIF_CHGBODY flag 2nd 3rd 4th

SMFIF_CHGHDRS flag 2nd 3rd 4th

SMFIF_DELRCPT flag 2nd 3rd 4th 5th 6th

SMFIF_NONE flag

SMFIF_QUARANTINE flag 2nd 3rd 4th

smfilter structure

SMFIS_ACCEPT return value 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

SMFIS_CONTINUE return value 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

SMFIS_DISCARD return value 2nd 3rd 4th 5th 6th 7th 8th 9th

SMFIS_REJECT return value 2nd 3rd 4th 5th 6th 7th 8th 9th

SMFIS_TEMPFAIL return value 2nd 3rd 4th 5th 6th 7th 8th 9th

SMTP (Simple Mail Transfer Protocol)

     changing reply 2nd 3rd 4th 5th

     DATA portion

     EHLO command

     envelopes

     EXPN command 2nd

     HELO command

     MAIL FROM part

     MAIL FROM: command

     Milter replies with data 2nd 3rd

     modifying messages 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd 24th

     reply code

     reviewing HELO/EHLO 2nd 3rd 4th 5th

smtp argument

SMTP commands

     extended 2nd

     rejecting 2nd 3rd

     tempfailing 2nd 3rd

Sockets

     changing default time out

     opening 2nd 3rd

     setting up 2nd

     smfi_setconn() routine

Solaris

     -lnsl switch

     baby-sitting script

     copying startup scripts

     log records

     logadm program

     method used to halt Milter

Source code Milter examples 2nd

Spam

     adding URL's host to database 2nd

     aliases and comments

     archiving 2nd 3rd

     attempting to fool signature detectors 2nd

     bouncing

     camouflaging HTML body 2nd 3rd 4th 5th

     clickable web reference (URL)

     connection behavior 2nd

     constantly changing and evolving

     disguising Subject: header 2nd 3rd

     disposing of

     email addresses

     evolution of

     exponential growth

     falsifying envelope sender address

     filtering

     full-blown war against

     grokking site 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th

     human view of

     internally structured

     ISPs 2nd

     large ISPs 2nd

     lost productivity

     maintaining history 2nd 3rd 4th

     method to contact spammer

     passing through 2nd 3rd

     phone numbers

     possible feedback mechanisms 2nd 3rd 4th

     rejecting 2nd 3rd

     relaying through MX (mail exchange) 2nd 3rd

     religiously or politically motivated

     selling something

     setting up addresses to be gathered 2nd 3rd

     speeding up process

     tracking source

     unnecessary encoding 2nd

     what to do with

Spam address-gathering software 2nd

Spam detection software

     envelope sender

     phonemes

"Spam Song,"

Spam suppression

     cost of 2nd

     software

Spam-blocking software recognizing spam

Spam-screening software

Spam-sending sites

     false envelope sender

Spammers

     hijacked PCs

     method for recipient to contract

     thinking like 2nd 3rd

Special characters

     HTML documents 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

     redirect servers 2nd

SPF (Sender Policy Framework) standard

Spyware

SQL (Structured Query Language) database

src directory

start argument

Startup script 2nd 3rd 4th 5th

stat() function

State, passing

Static configurations

Status information

stdin

stdio.h included file

stdout

stop argument

strerror() function 2nd 3rd

String constants

string.h included file

Strings

     allocating memory for

     cleaning spaces around 2nd

     longer than 980 characters 2nd

Stripping HTML comments 2nd 3rd 4th 5th 6th

strtol() C library function

struct keyword

struct priv_struct type

Structure

Subdomains 2nd

Subject: headers

     base64-encoding 2nd

     disguising 2nd 3rd

submit.cf file

Syntax error 2nd

sysexits.h included file 2nd

syslog records

syslog() function 2nd

syslog(3)

syslogd daemon, restarting

System password database

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

T_MX type

Target file

TCP sockets 2nd 3rd 4th 5th

TCP/IP (Transmission Control Protocol/Internet Protocol) 2nd

Telnet to known web server on port 2nd

Tempfail decision

Tempfail reply 2nd 3rd 4th 5th 6th

Template include file

Test machine [See bait machine]

test Milter

Text, computing value for 2nd

Theft

     of bank or credit card information 2nd

     of passwords

Thinking like spammers 2nd 3rd

Threads 2nd

time(3) C library routine

TLS encryption key length

TLS/SSL (Transaction Layer/Secure Socket Layer) version

Tomlinson, Fred

Trace-type headers

Trapping signals

ttl (time to live)

TXT record

Type

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

umask

Underlying database whitelisting

UNIX 2nd

UNIX domain socket 2nd

unix: prefix 2nd

Unnecessary encoding 2nd

unsigned char* type

Unwanted headers

URL detection

URL-encoding 2nd

     decoding 2nd 3rd

URLs (Uniform Resource Locators)

     case insensitive

     CNAME records and 2nd

     decoding

     email addresses masking

     encoding 2nd

     encountering @

     host.domain form

     hostname random word masquerading 2nd

     identifying protocol 2nd

     JavaScript.Encode

     quotation marks when pasting 2nd

     recording host names in database

     services that screen

     used as comments 2nd

Usenet

     email addresses

     plus addressing

     posting to news groups 2nd 3rd 4th 5th 6th 7th 8th

     spam risks

User IDs

     avoiding use of

     Milters 2nd 3rd 4th

     nonzero

     real

     resetting

     value as

User names 2nd

user variable

User-added headers

Users

     discovering if logged in

     modeling 2nd

     outside world as

     telling about plus addressing 2nd 3rd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Valid HTML keywords

value argument

Values and zero-length string

Version item

Vikings

Virtual Conspiracy website

Viruses 2nd

VRFY command 2nd

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Web addresses, masking 2nd 3rd

Web interface

     email readers

     whitelisting

Web references

     <a command

     character-entity encoding

     disguising

     HTML commands

Web servers, running 2nd 3rd

while loop

White space characters 2nd

Whitelisting 2nd

whitelisting 2nd

Whitelisting

Wildcard DNS records 2nd

Words global variable

Words, breaking up with HTML comments 2nd

Worms 2nd

write program

www host name

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

X-milter: header

xfi_eom() handler function

XMTP, VRFY command 2nd

xxfi_ handler functions

xxfi_abort() handler function 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th

     calling common subroutine to deallocate envelope data

     ctx private-context pointer

     deallocation routines called from

     example

     recording that Milter aborted

     usage 2nd 3rd

xxfi_body() handler function 2nd 3rd 4th 5th

     allowing selected local recipients to receive messages

     archiving copy of outbound email

     arguments 2nd

     calling

     ctx private-context pointer

     example 2nd

     len argument

     return values

     saving (buffering) body to file or in memory

     saving (buffering) body without attachments

     screening body for viruses

     storing attachments in database

     usage 2nd 3rd 4th

xxfi_close() function handler

xxfi_close() handler function 2nd 3rd 4th 5th 6th 7th 8th

     calling 2nd

     ctx private-context pointer

     ensuring allocated envelope data deallocated

     example 2nd

     summarizing actions taken by connecting site

     usage 2nd 3rd

xxfi_connect() handler function 2nd 3rd 4th 5th 6th 7th

     detecting if connection is on loopback interface

     example 2nd

     haddr argument

     keeping track of connections

     looking up host name and IP number

     return values 2nd

     sendmail macros 2nd

     sfsistat type

     usage 2nd

xxfi_envfrom() handler function 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th

     comparing IP address to list of rejected addresses

     envelope sender allowed to send mail in domain

     example of 2nd

     rejecting connections with small encryption key length

     return values 2nd

     sendmail macros

     usage 2nd 3rd 4th

xxfi_envrcpt() handler function 2nd 3rd 4th 5th 6th 7th 8th

     addressees of message inside

     arguments

     argv argument

     calling

     counting number of good recipients

     ctx private-context pointer

     example 2nd

     list of honey-pot (bait) recipients

     missing recipients can be found

     return values

     sendmail macros

     usage 2nd 3rd 4th

     validating whitelisting pairs

xxfi_eoh() handler function 2nd 3rd

     calling

     comparing number of envelope recipients to number of header recipients

     ctx private-context pointer

     example

     flagging missing headers

     logging statistical review of headers

     return values

     usage 2nd

xxfi_eom() handler function 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th 21st 22nd 23rd

     adding envelope recipient

     adding headers found to be missing

     argument

     calling

     changing value of headers

     constrained by time limits 2nd

     ctx private-context pointer

     deallocation routines called from

     decoding body

     example

     logging summary of everything Milter did with envelope

     removing envelope recipient

     removing junk headers

     return values

     screening body to detect spam, viruses, and unwanted attachments

     sendmail macros

     usage 2nd

xxfi_header() handler function 2nd 3rd 4th 5th 6th 7th

     arguments 2nd

     calling

     checking header values for adherence to standards

     ctx private-context pointer 2nd

     detecting bogus Received: headers

     detecting spamming software patterns in HELO/EHLO string

     example 2nd 3rd 4th

     host argument

     name argument

     recording presence of header

     return values 2nd 3rd

     sendmail macros

     sfsistat type

     unwanted headers

     usage 2nd 3rd 4th 5th 6th

     value argument

     verifying correct cipher suite

     xxfi_helo() handler function 2nd 3rd 4th

xxfi_prefix

xxfi_rcpt() handler function

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Zero-length file for logging messages

Zero-length string

Zombie mail machine