Changes in spam handling for Sept 22, 2008

As of Sept 22, 2008, we will be using much of the same mail processing as RCI. RCI has recently improved their spam control. Most of you will find the changes to be an improvement. But since it could result in losing some mail, you need to know what the changes are:

All mail meeting the following criteria will be dropped , no matter what options you have chosen for spam control.

• Mail containing a virus
• Mail containing a file that appears to be a Windows program or Javascript file. These are almost always viruses. For a list of file types, see Banned file types.
• The hostname used in the SMTP FROM command is not registered with the Internet name servers. (This test is not as strict as the hostname checks on RCI. Some users have reported that legitimate mail is being rejected for this reason on RCI. We will generally not reject it.)
• There are signs of forgery, e.g. a from line saying "VIAGRA mail site <hedrick@rutgers.edu>" or "admin@microsoft.com <president@rutgers.edu>".
• Mail involved in "phishing" (attempts to get users to give out passwords or other personal information), if we are really sure.
• Mail where the spam checks give very high values (30 or more on a scale where anything over 10 is certainly spam).

If you read mail on rulink, you can control how we handle spam. We give you two options, one using the RCI spam controls, and one a clone of the computer science department's spam controls. If you don't make any choice, you get RCI's controls.

Our 9/22 version of RCI's controls will be much more effective than the RCI controls we have now. In order to catch most spam, many Rulink users have chosen very low cutoff values, e.g. 1 or 2. In the past this was necessary, because the recommended value of 5 didn't catch a lot of the spam. The new controls are much more sensitive. A value below 5 is likely to reject a lot of good mail. Thus when we make the conversion, we are going to reset all cutoffs that are below 5 to 5. This only affects people who are using the RCI spam controls. There will be no change to the computer science-based controls.

If you have chosen the computer science (LCSR) spam processing, you might want to try RCI's.

As of 9/22 we will be using the same spam-handling system as RCI, so any further changes in RCI will be occur on RULink at the same time.

Spam processing for RULink

Spam is an increasing headache for all of us. For some people, it's a bad enough problem to make mail almost unusable. RULink has several features to control spam. This document will cover the following items:

• Spam control for people who use addresses @rutgers.edu, but don't read mail on RULink. This is the section that is relevant to most people.
• Spam control for people who read mail on rulink .
• How to avoid getting spam-- for everyone.

In addition there is a section at the end on Technical Details.

Spam control for people who use addresses @rutgers.edu, but don't read mail on RULink

RULink provides support for addresses ending in @rutgers.edu. Many people use this without realizing that they are using RULink at all. E.g. if you have a netid of jsmith, and read mail at RCI, you can have people send mail to jsmith@rutgers.edu. As long as you have registered your RCI email address, RULink will forward mail sent to jsmith@rutgers.edu to your real email address, which is jsmith@rci.rutgers.edu.

It is also possible to register "friendly addresses", such as john.smith@rutgers.edu. Again, RULink handles those addresses, but forwards mail to RCI, Gmail, or any other system where you actually read your mail.

When RULink forwards mail to another system, it automatically provides spam control. RULink uses a system called "Spamassassin". Our copy is configured to check a variety of databases of known spammers. It also checks mail for words and phrases that are commonly used in spam. If there are enough signs that the message is spam, it will be discarded or saved in the user's mailbox on rulink.

The user can control how sensitive the test is. If you go to http://rulink.rutgers.edu/admin, and choose "Spam Control", you'll see a screen that lets you choose a value from 5 to 10. We recommend 5. However it will sometimes reject real mail as spam. 10 should be safe, but it will be less effective at catching spam. The default, which applies if you don't do anything, is 10.

If you are forwarding to an address within Rutgers, the default is not to do any spam checks. Most people find it confusing if spam is handled by rulink, since they expect the system where they are reading mail to do their spam handling. However you can set a cutoff between 5 and 10 if you prefer.

[We don't let you turn off spam checking if you forward to systems outside Rutgers, because many ISPs blame us for spam, even though we're just forwarding the mail. If we don't do at least some spam checking, ISPs will blacklist us.]

There is one feature that you can control for messages forwarded to another system: "Greylisting." This is a feature that will significantly reduce the amount of spam you get, but has some disadvantages. Please see the Greylisting section for details.

Spam control for people who read mail on rulink

For most people, RULink simply forwards mail to the system where they actually read mail. However RULink is a full-featured mail system, so you can actually read mail directly on RULink. To do this, you connect to http://rulink.rutgers.edu/admin, select "Manage Mail Delivery", and then select "I will read mail on this system".

Some departments use RULink as a departmental mail system. Users in those departments normally read mail on RULink, so they are covered by this section.

If you read mail on RULink, you control what type of spam processing is done. By default, there is no spam checking.

To enable spam processing, connect to http://rulink.rutgers.edu/admin, and select "Spam Control." If you have not set up spam processing there will be a link, "Turn on spam filtering." Once you have done that, you'll see a menu that lets you control details of how spam processing is done, if you want to.

At this point I recommend clicking the box "Use LCSR's Spamassassin", and setting the score to 5 (instead of the default 7). The options are explained later on that page.

There is special feature that you can control: "Greylisting." This is a feature that will significantly reduce the amount of spam you get, but has some disadvantages. Please see the Greylisting section for details.

Greylisting

Greylisting is a technique intended to fight spam. It uses a database containing entries for everyone who has sent you mail in the past. The first time you get mail from someone who hasn't sent it to you before, the message is rejected with an error message requesting the sender to try again later. Most spammers use software that doesn't retry. So this will reduce the amount of spam you get by 1/2 to 2/3. In addition, by delaying the email, it gives time for the spam "black lists" to discover this particular spammer. So even if they do retry, there's a good chance that they will now be listed in a black list.

While Greylisting is a fairly effective precaution, it has one serious problem: it causes the first email from a given person to be delayed up to an hour. This can be a problem in some situations. E.g. if you register with a site, they often confirm your registration by email. This will probably be the first email you get from the site, so it will be delayed.

For this reason, we recommend that you set greylisting only if other types of filtering aren't good enough.

Note that greylisting works for forwarded mail as well as mail delivered on rulink.

To enable or disable greylisting, connect to http://rulink.rutgers.edu/admin, and select "Spam Control." Down the page, under "Advanced Options" you will find a link to "Set greylisting" (or remove it if it's currently set).

How to avoid getting spam

Spammers make some attempt to guess email addresses. However in most cases, we believe that you get spam because your email address appears in public. The most common place is web pages. Spammers often search web pages for email addresses. Other web-based services are also possible targets.

We don't think simply having an email address at Rutgers leads to spam. I have a test user on both rulink and RCI, set up to look exactly like a faculty member. It gets no spam.

We recommend against having departmental directories with email addresses. It is probably OK to have a web page that says at the top "all addresses on this page are at RCI" and then list just the part before the @. Some people have tried to make addresses harder to recognize by writing out the @, e.g. "user at rci.rutgers.edu". We believe spammers are onto this trick.

Technical details

Rutgers has recently gotten in trouble in the past with various ISPs because they believe we are the source of spam.

This doesn't mean that our systems were actually the source of spam. However spam is increasing, and ISPs are getting progressively more hair-trigger in the things they do to cope with spam. Some of their efforts affect innocent third parties more than the spammers, in my opinion.

For example, if a user at Rutgers forwards their email to an ISP, the ISP sees any spam received by the user as being due to Rutgers, because the mail arrives at the ISP from Rutgers. Thus if the user complains about getting spam, Rutgers may get blacklisted.

For this reason, a number of ISPs are demanding that Rutgers not forward email. We've gotten requests not to let anyone forward, to prohibit vacation messages, and even not to send error messages.

This has affected the way email is processed by RULink:

• We do spam control for all mail forwarded by RULink to other systems. We can't afford to send significant amounts of spam.

• We are dropping all email whose from address comes from a non-existent host. This will get rid of a fair amount of the spam.

• One of the major causes of concern from ISPs is "backscatter". Because of the fact that spam uses invalid addresses, spam tends to generate large numbers of bounce messages to invalid addresses. ISPs often consider a host that sends lots of messages to non-existent users to be a spammer itself. There's no really good solution to this, but we are current disabling many types of messages to non-Rutgers addresses. This includes the report that normally went back to the sender saying that their message had been rejected as spam, as well as vacation messages and various kinds of delivery errors. Users will still see simple "no such user" errors. These changes affect only sites outside of Rutgers. We will still give vacation messages and the full set of error messages to other mail systems within Rutgers.

• We are disabling forwarding for users on rulink when we get an error message back saying that the user does not exist on the remote system. We are very careful about this: we only act on messages that clearly indicate that the user is permanently gone. Because the mail standards do not define a precise error message for this situation, we match certain English text that is commonly used in these messages. That means we will not always be able to detect this error.