410 Gone

The Backstory:

I’ve spent the past 6 years working as an engineer at a company whose client-base is primarily made up of SMB (Small to Medium Business). In a perfect world, every business would have an IT staff, IDS/IPS, and well-trained employees who are savvy enough to avoid common infection vectors. As we all know, this certainly isn’t the case, and we have to be prepared to deal with outbreak situations using a minimal amount of time, money, and equipment.

By far the most common infection I see hitting these environments are the mass-mailer worms. Someone opens a harmless-looking email attachment, and within minutes their computer is sending out an endless stream of email with bogus headers. Quite often the infection will spread to other computers on the network. The worst part is that nobody notices anything until the ISP shuts down the account due to abuse claims, and business grinds to a halt. That’s when I receive the panicked phone call.

What Would MacGyver Do?

The first instinct many technicians have is to run around to every single computer, scan/remove any virii/malware, then call the ISP and have the connection re-activated. My advice: don’t do this unless you hate efficiency and want the problem to come back. Here’s why:

1. You’re flying blind. You have no idea which or how many computers are infected. The first step in any good diagnostic process should be to clearly define the problem.
2. It probably wont work. If the infection got past the resident shield of the antivirus software, a manual scan probably won’t find it either.
3. You can’t fight a wildfire with a single bucket. Even if you successfully remove the infection from one PC, it can get re-infected while you’re working on other machines.
4. Nobody can work, even the uninfected. In a company with a 50% infection ratio, the business is losing twice the amount of productivity that they could be. Time is expensive.
5. Your ISP could blacklist you. If you think you’ve removed the worm and you re-activate your internet connection, a single overlooked infection could get you shut down again in short order. Try doing that 2 or 3 times in a row and you’ll find yourself with a very angry Abuse Department on the other end of the line. Permanent blacklisting will be the result and you really don’t want that.

Most small businesses are reluctant to invest in network security until after they get hit, so you’re going to find yourself staring at a couple of unmanaged switches. The router will be consumer-grade (probably a Linksys that someone picked up at the local Big Box) and if there is any kind of firewall in place it will only be capable of blocking inbound traffic. A paperclip and chewing gum might work on TV, but you’re going to need some bigger guns. Upgrading the infrastructure will have to be dealt with later, but there are some simple steps you can take immediately. Please keep in mind that this is just my advice, and not meant to be a definitive guide, but it has worked well in my experience.

First of all, you’ll want to disconnect the data line between the modem and router. Once this is done, call the ISP and tell them that you’ve isolated the infected machines and would like your account re-activated. You want to do this first because it can take a few hours for them to respond to your request, and you want an active connection on standby.

Remember hubs? Now relegated to the dustbin of antiquated hardware, one of these cheap devices can help you isolate the infection in a few minutes flat. When you don’t have access to high-end equipment, putting a hub between the gateway router and internal switches can effectively become a poor man’s port mirroring. I like to use Wireshark (see above image,) and I simply connect my laptop to the hub and tell the software to start monitoring traffic. It’s best to use OS X or some distribution of Linux for this to minimize the chance of catching the virus yourself. Keep an eye on the traffic for things that shouldn’t be there. In the event of a mass-mailer, you’re going to see a lot of outbound SMTP traffic with all kinds of false header info. To zero in on that particular traffic you can apply the following filter:

This will only show packets that are being sent to port 25, which is the default port for SMTP. Now you’ve got an exact list of infected IP addresses, with no ambiguity of the scope of the infection. Use your favorite network scanner (I use Angry IP Scanner) to locate those hosts and literally unplug them from the network. Clear the filter so that you’re once again looking at all network traffic, and watch for anymore weirdness. When it looks good, go ahead and plug the modem back into the router. This part of the process shouldn’t take more than an hour, and you’ll have a good chunk of the company back up and running.

At this point you’ll able to deal with the infected machines on an individual basis, disinfecting and immunizing as necessary. This process varies greatly depending on the infection, so you’ll have to do a little research. It’s important that you keep an eye on Wireshark as you introduce more and more machines to the network so that nothing slips through; you’ll have to start all over again if that happens.

The final step is the most important: don’t stop monitoring after you’ve finished removing the worm. I recommend logging all network traffic for at least 24 hours afterward; if someone turns on an infected machine that you weren’t aware of (or brings in an infected laptop) you must be able to quarantine it immediately.

I’ve followed this process many times and I’ve never had a relapse. You can use all the time and money saved to convince the powers that be to invest in more robust security, and hopefully prevent the situation from recurring the next time someone clicks on something they shouldn’t.

This past Saturday was the second annual NewBCamp - an ‘unconference’ that brings people together to learn from each other about technology - in Providence, RI. I put together a presentation on how I use Virtual Machines to improve my development workflow, and since there was a large ‘blogging’ presence at the event, I focused on how this can help when designing for WordPress. This is a condensed version, so some basic knowledge of how web-hosting works is assumed.

A larger version is available here.

A client of ours recently had us upgrade their old workgroup environment to Active Directory. Part of the project involved a handful of new Dell laptops for the mobile workers, and we implemented offline file synchronization of their “My Documents” folder to a location on the server.

Almost immediately after the deployment we began getting phone calls from the laptop users about failed syncs, bizarre error messages, and most importantly, disappearing documents. We first looked at the wireless connectivity, since that was the only apparent difference between the laptops and the perfectly functioning desktops. Everything looked fine there, and for a while we were totally baffled. I have since lost the URL, but after a few days of researching the subject I stumbled upon a possible solution. Someone had experienced similar issues with Dell laptops that had the Wave Systems’ Embassy Trust Suite installed. I checked, and sure enough, every one of the laptops had been shipped to us with that piece of software installed. Removing it immediately solved the problem, and the missing files even came back, much to the relief of our client. I am not entirely clear on what the purpose of the Trust Suite is, but a quick search yields several unhappy users. Hopefully this issue is resolved in future updates, but until then I recommend simply removing the software.

Have you ever played Katamari Damacy?  It’s a fun little game for the PlayStation 2, and without spending too much time on the plot, the game mechanics work in much the same way as building a snowman.  You start out with a small ball which you roll around the level, picking up objects that you run over.  As more objects stick to the ball,  you’re able to pick up larger and larger objects. When time runs out, you are judged based on how large your ball (known as a Katamari,) has grown.

If you’re a regular Twitter user, you can probably see where I’m going with this.  In the early days of the service, spammers and spam-bots would follow thousands of people, sometimes tens of thousands, and only a very small percentage of those users would reciprocate.  Twitter caught on, and eventually put in some safeguards that would suspend accounts with a very low followers/following ratio.

Of course, the spammers figured this out and have recently adopted a new tactic, which I have aptly named after the video game:

What’s happening now is that when a spammer follows me, I look at their followers/following ratio and am surprised to see that it is usually only slightly below 1.  This means that they follow only a small amount more users than the amount of users that are following them.  If I don’t reciprocate by following the spammer, I am invariably unfollowed within a few days to a week at most.

This is where the Katamari Damacy analogy comes in.  It seems like spammers are now indiscriminately following dozens of users at a time, but only enough so that their ratio doesn’t drop below the suspension trigger level.  After a few days, they simply unfollow those who have not reciprocated, but now they’ve increased their “Katamari” of followers, and are able to follow an even larger group of people without triggering a suspension.  I’m willing to bet that this process is largely automated by scripts.

Finding spammers used to be as simple as calculating a ratio, so how can Twitter catch this new breed?  The detection algorithms could be adjusted so that users with unusually high numbers of new follows/followers over a period of time (say a month) also triggered a suspension, but this would likely require some fine tuning.  What do you think?

While I love online backup services like Mozy and IDrive, I already pay for storage space elsewhere, and wanted to roll my own scheduled backup for all of my Documents and Photos.  Since I use a Mac, a simple AppleScript was all I needed.  I’ve been using this for months now with no problems, so I’m posting it in the hope that other geeks will find it useful and maybe save some money.

You will need to install RsyncX, which can be found here for free.  Also, this script assumes you already have SSH access to a server somewhere.  Most cheap web-hosting companies frown upon using your space to store backups, so be sure to check the Terms of Service before using this.

It’s important to note that this script lacks certain features that you’ll find in the paid arena:

• If your computer is off when this backup is scheduled to run, you will not get a warning the next time you turn it on.  I used Lingon to schedule the script to execute at 6pm daily, but I do find myself having to manually trigger the backup via terminal when I don’t have a network connection at that time.
• You will get a Growl notification when the backup completes, but there is no way to monitor “% complete” or things like that.
• If you delete something from your Mac, it will not be deleted from the remote server; you’ll have to be comfortable managing that yourself.

Here’s the script, make sure to replace the “username@domain.com:~/remote-directory/” sections (there are 2) with your username, domain, and remote directory.

tell application "GrowlHelperApp"
set the allNotificationsList to {"Backup Started", "Backup Finished"}
set the enabledNotificationsList to {"Backup Started", "Backup Finished"}

register as application "RSync Notifier" all notifications allNotificationsList default notifications enabledNotificationsList icon of application "RsyncX"
end tell

tell application "GrowlHelperApp"
notify with name "Backup Started" title "Backup Started" description "Sending files to remote." application name "RSync Notifier" icon of application "RsyncX"
end tell

property resultDocs : ""
property resultPics : ""

set resultDocs to do shell script "rsync -avz ~/Documents/ username@domain.com:~/remote-directory/Documents/"

tell application "GrowlHelperApp"
notify with name "Backup Finished" title "Documents Complete" description resultDocs application name "RSync Notifier" icon of application "RsyncX" with sticky
end tell

set resultPics to do shell script "rsync -avz ~/Pictures/ username@domain.com:~/remote-directory/Pictures/"

tell application "GrowlHelperApp"
notify with name "Backup Finished" title "Pictures Complete" description resultPics application name "RSync Notifier" icon of application "RsyncX" with sticky
end tell