Where Websites Come To Life
Spam Issue Troubleshooting and Guidelines
Posted: May 14th, 2018
What is Spam?
Bulk email is an email that is sent in batches where each email is the same. Commercial email is an email that is regarding a commercial product or service. Unsolicited email that is sent without permission or without a request to do so that is both bulk and commercial is spam.
Unsolicited Bulk Email (UBE) + Unsolicited Commercial Email = SPAM
While bulk email that is also commercial is generally not spam if it is solicited there is an exception to this rule. In USA/Canada it is illegal to send commercial email without permission even if it is not bulk.
In addition, Spam is flooding the Internet with many copies of the same message, in an attempt to force the message on people who would not otherwise choose to receive it. Most spam is commercial advertising, often for dubious products, get-rich-quick schemes, or quasi-legal services. Spam costs the sender very little to send -- most of the costs are paid for by the recipient or the carriers rather than by the sender.
There are two main types of spam, and they have different effects on Internet users. Cancellable Usenet spam is a single message sent to 20 or more Usenet newsgroups. (Through long experience, Usenet users have found that any message posted to so many newsgroups is often not relevant to most or all of them.) Usenet spam is aimed at "lurkers", people who read newsgroups but rarely or never post and give their address away.
Usenet spam robs users of the utility of the newsgroups by overwhelming them with a barrage of advertising or other irrelevant posts. Furthermore, Usenet spam subverts the ability of system administrators and owners to manage the topics they accept on their systems.
Sources of Spam
Server administrators have a responsibility to ensure their infrastructure is not being used to send Spam. There are several common sources of Spam (Email spam or otherwise) on the Internet directly related to the administration of a server.
Botnets: Some types of malware infecting a computer make it a member of a 'botnet'. The bot (or zombie) is then under remote control by a malicious person. Examples of botnets that are responsible for massive spam attacks around the world include the Cutwail and Rustock botnets among others.
Open Relays: A mis-configured server might actually relay mail on behalf of others without checking to see if that person is authorized to use the server to send mail.
Compromised Server/Accounts: Spammers sometimes scan entire networks for mail services and then attempt thousands of usernames and passwords until they find accounts with weak passwords that they can use to send spam. Systems that also provide webmail are especially vulnerable because if the spammer chooses to send the spam using the webmail interface it can hide their IP address entirely and make it more difficult to track down the source.
Backscatter: Most spam email is sent from one or more spoofed (fake) address and if the spam email bounces because it cannot be delivered the non-delivery-report is sent back to the fake address(es). Poorly configured non-delivery systems on mail servers can send mail back to spoofed sources but if the spoofed source of a message is an email address of an actual person then the bounced email message will reach that real person. This is sometimes an intentional way of sending spam since anti-spam systems do not block backscatter spam very well. The downside is that the messages look like delivery failures but the recipient may read them nonetheless.
Unsecured Networks: A spammer might physically locate themselves in such a place that they can access an unsecured wireless network or a data port with Internet connectivity and send a deluge of spam messages from that network using their own computer. They might physically leave the area and then come back again and again in order to continue to send spam in this way. A variation of this is allowing spam mail to be sent through a VPN from a remote computer where the VPN provider does not keep track of who is using which user is using which private IP address on the private network at any point in time. This makes it difficult if not impossible to find out who the responsible user is.
Email Marketers: Not all email sent by email marketers is spam. There are some legitimate mass mail marketing companies out there with good business practices that not only observe applicable laws but also observe proper mailing list management practices. Some email marketers actually but incorrectly believe they are doing a good job at managing their mailing lists. It's possible that a recipient might forget genuinely providing permission to send them marketing related email and regarding it as spam. It is up to the marketer to ensure their mailing lists are managed properly and that they have explicit permission to send the email in the first place, remove subscribers when delivery fails and unsubscribe users promptly and permanently when they request it. Never buy a mailing list from a third party even if it is said to be opt-in, double-opt-in or even triple-opt-in (whatever that means).
Instant Messaging: More commonly considered a trusted channel for sending messages spam sent through instant messages can be very effective for spammers. People commonly communicate through instant messaging with people that they have never met when it relates to their business. Spammers who use instant messaging usually attempt to blast as many messages out as fast as possible until the providers involved block them.
Email Marketing and Properly Managed Mailing Lists
Managing a mailing list properly and increasing it's business value while ensuring less wasted resources during delivery is all about permission. A properly managed mailing list is built using the Confirmed Opt In (COI) method. Upon the addition of a new subscriber a process obtains permission from that subscriber using the email address provided such as a link to confirm the subscription.
The COI process works like this:
> The subscriber provides their email address on a form or during an order for a product or service
> The mailing list system sees this new subscriber address and sends them an email containing a special link that marks the address OK to send to in the future
> The subscriber receives the email and clicks the link
> The subscriber's address is marked as willing to receive future emails about the products and services for which they are subscribed.
Failure to obtain permission directly from the user receiving the email is the difference between email marketing and spam.
More information about this process and it's benefits can be found on Spamhaus's website
Suggestions for Resolution
1) Identify the MTA (Exim, native Postfix or Plesk postfix, qmail, Mailenable etc.)
2) Identify the spam source by inspecting the MTA logs and the mailqueue content. In some uncommon cases, the spam can be sent via another component (such as a rogue proxy or rootkit/backdoor on a compromised server, or a compromised VPN account) and the MTA will not show trace of the spam in the logs.
3) Stop the spam activity cause by fixing the spam origin issue:
1) In case of compromised email/user account: change the account password
2) In case of compromised website exploited to perform spam activity, refer to our Malicious URL Troubleshooting and Guidelines procedure
3) In case of bad configuration causing spam activities, the settings need to be fixed.
4) Clean the remaining spam from the mail queue.
5) Delist the IP addresses from the blacklists
6) Avoid a similar issue by securing/hardening the websites/accounts/server
Identifying Spam Sources Under Linux
1) Analyze Headers
1) Obtain one or more spam complaints and analyze the headers of an actual spam message that was sent
2) Look at the IP addresses used in each of the "Received:" header lines. They are listed in reverse-chronological order. If your server's IP address is not the oldest (listed last) "Received:" header then it means someone is likely relaying mail through your mail server with a username/password.
3) Look at any of the X- headers (like X-Spam) in case they are include any information that might have been added by your server while the email was sent such as the authenticated username or possibly a website domain.
4) Check the "From" header in case it's an actual valid address on your server since this may be an indication that the email was sent through a webmail application.
2) Check the your mail queue
1) Some mail is sent by a process that is 'direct to MX' which means it doesn't use your server's MTA (mail transfer agent). For mail that does user your MTA it will be logged and there will be spam queued to go out in the MTA's mail queue.
3) LINUX: Check for running processes with the 'lsof' utility that have connections going out to port 25 or port 587 on other machines
1) lsof -Pni tcp:25
2) Isof -Pni tcp:587
3) Note down any process IDs (PIDs) as well as the username that owns the process where the 'COMMAND' is suspicious and the destination IP address is not yours. You are looking for connections out of the server.
4) Don't necessarily ignore processes that have names like smtpd or sendmail because the spammer could have intentionally gave them common names to make them difficult to identify as spam related. Be sure to investigate them as well but use caution before killing those processes.
5) Although rare - this might show an active connective in to tcp port 25 or tcp port 287 from the spammer relaying through your SMTP service using a legitimate but compromised user account.
4) Inspect the details of suspicious processes with 'lsof':
1) lsof -Pnp <PID>
2) Pay close attention to any lines with 'cwd' (current working directory) or 'txt' in the FD column. These may contain clues as to where the responsible scripts are located.
3) You may want to kill the process once it is found before or after any files related to the process are investigated more thorougly but use caution not to stop processes that are legitimate mail software delivering real mail.