Mystery of the Chinese Spammer That Could Not Be Blocked
I have sendmail configured to limit the number of daemon processes and new connections as an anti-spam measure. It helps to prevent a certain type of spam attack involving large volume delivery attempts testing for the existence of email addresses sometimes referred to as "list probing." However, sometimes when a lot of spam is coming though the system, the maximum number of processes is reached, blocking local users from sending email via SMTP. I've aggressively firewalled out common sources of spam to prevent this. Lately, I have been fighting a problem I thought was a spam source that had figured out a way to circumvent my firewall. It was causing serious mail outage problems and I was challenged to figure out why I wasn't able to successfully block the source.
Customers were calling me reporting that they were having problems sending email. I would check the mail server and discover that the maximum number of sendmail processes had been reached, all open connections to the same place. When I would kill the processes or restart sendmail, they would just come back again.
It was coming up China, again and again, the maximum number of sendmail processes attempting to communicate with a remote mail server and causing serious problems. The process list kept showing sendmail trying to communicate with domains ending in .cn, which is the SO for China. Here are a couple examples:
7817 ? S 0:00 sendmail: ./m4MNlw410036 mail.sausagechannel.cn.: user open 9032 ? S 0:00 sendmail: ./m4P6RCR15475 mail.hebrewmassachusetts.cn.: user open
Except there would be many all at the same time. The host was always "mail," the domain name seemed random (or from a dictionary word list), and the extension was always "cn" for China. So I used netstat with the -p switch to show the process id with the connection and -n to show numeric IP addresses. I was able to match that against the process list and each of those random domains ending in .cn all resolved to the same IP address:
89.149.229.48
So I determined it was a spam source and blocked the entire CIDR network range going off of the AP-NIC info. Normally this stops the problem. Yet days went by and I kept seeing the same stuff appearing in the process list. Again, random domains with the .cn extension resolving back to an IP address in a network that is supposed to be blocked by my firewall.
All of the regular problem solving approaches yielded no solution, including checking firewall rule syntax and performing system security audits. Then it occurred to me. I remembered a similar security issue with a mail server I had solved a few years back.
Rather than the sendmail processes being SMTP connections from the IP address trying to deliver mail though my mail server, could it be some sort of process on my server attempting to deliver mail out to that Chinese remote system?
Taking a closer look at netstat and then looking at the mail queue provided evidence supporting my new theory. Apache owned all of the undelivered messages in the mail queue "mqueue"!
Killing Apache caused the sendmail processes to disappear. As soon as I started Apache, the sendmail processes would come back. I located a large number of undelivered messages in the mail queue that were relatively the same size.
Apparently, there was some web script installed on my server that was trying to deliver these messages. No wonder the firewall rule wasn't solving the problem. I analyzed the message headers and discovered that one of my customer's had installed a web based PHP forum with a user registration component. That component was attempting to deliver confirmation email to a bunch of bogus email addresses in China.
Normally, when a new user registers to an online forum, the forum program will generate a message and mail it to the new user via the email addresses supplied by that person during the signup process. That is the way most online forum applications work. Many actually email the new user their password. The particular PHP forum my customer installed had a security flaw allowing it to be exploited. Someone in the Asia pacific was basically doing list probing of a Chinese mail server though the forum.
This particular customer of mine has been a problem in the past. He is a non-paying customer that I have tolerated over the years for various reasons, mostly out of kindness and charity. This customer has installed insecure web software on his user account before, and has been warned before. Well this time I removed his account and deleted the insecure PHP forum he had installed for his web site.
Stopping sendmail for a moment allowed me to go though the mail queue and delete all of the undelivered messages generated by the insecure PHP forum addressed to the mail server in China and delete them using a little REGEX. Now, with the messages gone from the mail queue, and the PHP forum deleted from my server, I restarted sendmail. This ended the mystery of the Chinese mail server spam stuffing up my system and causing all of my other paying customers to complain.