SPF is the Sender Policy Framework. It used to stand for Sender Permitted From, which is probably a more descriptive title, but was generally considered too cheesy for mainstream use.
What SPF is not
SPF is not an anti-spam tool, despite what it still says on spf.org. If you read one of the many diatribes against SPF on the Internet that rant about SPF's lack of effect against spam - well, guess what? That's not what it does. SPF should not be expected to stop spam for the same reason that your car should not be expected to cook a decent vindaloo.
What SPF is
SPF is an anti-forgery system. Its purpose is to make sure that a domain owner only takes the blame for email that his domain really has sent, rather than what some fool has sent whilst pretending to be a part of that domain. This is typically - but not exclusively - tied to spam runs, where a spammer will forge a domain name he doesn't own, in an attempt to gain credibility and/or cover his tracks.
How does SPF work?
SPF boils down to an expression of "it's my domain, and I'm going to say who's allowed to send mail from it". There are two parts to this :-
- the SPF records, which detail who is allowed to send mail, who isn't, and who we're not entirely sure about.
- the SPF filter, which checks incoming mail against the published records, and deals with it according to the result.
It is often claimed that, because SPF will not be 100% effective until 100% of mail servers implement SPF filters, SPF is not worth the effort. This is clearly bunkum; although SPF's effectiveness increases as people start to use it, it is already effective enough to be useful, even if some mails will slip through the net.
SPF records are transmitted through the DNS system. There is a recently ratified DNS resource record for SPF, but it is not widely available yet (it should be available from Bind 9.4). Therefore, the TXT record type is currently used. To ensure that other TXT records don't interfere with SPF (and vice-versa), SPF records always start
Following this come the clauses that make up the policy.
Each clause can have a modifier before it:
;"+" : "allowed" (the default if there is no modifier) ;"-" : "not allowed" ;"?" : "not sure - pretend this record didn't exist" ;"~" : "not allowed, but don't reject this".
The first two of these are obvious in their meaning; the second two are a little more obscure. "?" is used for shared systems, which may send legitimate mail for this domain, but may also send on behalf of people who may not send for this domain. The last - "~" is known as "softfail", and is primarily used for testing, so as not to ditch large amounts of email by putting in a wrong record into a heavily-cached DNS record.
The actual clauses modified by the above come down to the following important types :-
;a : the address specified by the A record for this domain ;mx : the address specified by the MX record for this domain ;ip4 : an explicit IPv4 address ;ip6 : an explicit IPv6 address ;all : matches everything
A simple, one-server domain could build its SPF record as
"v=spf1 a -all"
This permits mail from the A record machine (i.e. the one host running this network). If that clause is matched, then processing stops there - the mail is allowed. If that clause doesn't match, the next one is tried. This is "-all", whilch means "disallow from everywhere". Thus only the machine designated by the A record will get a PASS result from this record.
More complex policies can be built by stringing clauses together; suppose, for example, you wanted to allow the machine designated by the MX record to send as well as receive mail. You also want to allow a certain block of IP addresses to send mail, and you want to specifically disable the machine with the A record. You could use the record :-
"v=spf1 -a mx ip4:a.b.c.d/24 ~all"
This disallows the A record machine, allows the MX record machine, allows any machine with an IP address described by ip4:a.b.c.d/24, and returns SOFTFAIL (i.e. it's a FAIL, but don't bin the mail) for anything else.
There are yet other games to be played with SPF records - for example, there is an "include" directive, which allows you to permit any host permitted by someone else's SPF record. This is most useful when you send mail via an ISP's mail servers, and that ISP publishes SPF records - by including your ISP's SPF policy, you can reflect their mail structure in your record, without having to know it explicitly and react to every change they make. This is one place where the "?" modifier comes in handy - by adding something like "?include:myisp.com" in your record, you add your ISP's servers to your policy without claiming responsibility for all your ISP's other customers.
The actual filtering of emails occurs at the receiving mailserver. The most common mechanism is to check the envelope MAIL FROM: address. The SPF record for the domain claimed in the envelope is looked up, and the IP address of the sending machine is checked against that record. The receiving mailserver can then determine whether this IP address is permitted to send mail on behalf of that domain or not. If it isn't, the connection can be terminated - this is an attempted forgery.
SPF is not a perfect system. In particular, it cannot be 100% effective until 100% of domains have SPF records and 100% of mailservers have SPF filters. This will probably never happen. Nevertheless, adding SPF records to your domain is already effective; in my case it has reduced the bounce rate by 3 to 4 orders of magnitude. This is almost certainly because spammers don't bother to forge domains which have SPF records.
An SPF PASS doesn't mean that the email is worth receiving; there's nothing at all to stop a spammer or other ne'er-do-well from publishing SPF records for his own domain. As I said at the top, this technology isn't about stopping spam, it's about stopping forgeries. SPF means that a domain owner won't get the blame (and the bounces!) for thousands of mails purportedly sent from his domain, but actually sent from a forger.
Perhaps the biggest problem presented by SPF is the case of email reflectors - mailing lists and the like. Because SPF deliberately restricts the hosts that may send on behalf of this domain, email reflectors will typically be prevented from doing so - rendering them useless. It is impractical to list every reflector that a domain will use in the SPF record. The answer is to use the Sender Rewriting Scheme (SRS). This is a method by which the envelope is modified in a known, cryptographically-secure way so that the email may be forwarded using the mail reflector's SPF record instead of the originator's, but without losing the original envelope information. This is a little work for the SysAdmins running mail reflectors, but it permits the system to work across such instances, without losing the ability to bounce email to the originator if necessary. SRS might be the subject of a future article, if I ever get round to it.
SPF is a fairly new technology, and is still subject to substantial changes and evolution even if the basic groundwork is frozen. In particular, migration from TXT RRs to SPF's own RRs may be slow.
Another issue is that Microsoft's Sender-ID scheme has appropriated SPF as an added check within that framework (by IETF, not MS, just to apportion blame correctly), but this is widely thought to be a faulty system because Sender-ID works on a complex system of mail headers and SPF is a basic check on SMTP envelopes. Efforts to merge the two systems over could result in neither system being as effective as originally conceived.
This article isn't yet complete. The things that need expanding that I can think of so far are :-
- HELO checking (used always when there is no envelope from address and optionally at other times)
- ptr records
- DNS loading
Record size limits & nested includes
- Links to filter implementations for various MTAs
SPF is now an RFC [http://www.ietf.org/rfc/rfc4408.txt]