Last month, I had a massive bill from rackspace for my servers. Apparently the server acquired a trojan that was constantly downloading new versions of itself. At rackspace, pricing is $0.12 per GB of transfer. My server, after slowing to a crawl for a few days, sending out data, did about 3 TB worth of data in December, racking up my charge for that month to over $400.

After I found it I reimaged the server that was infected, after moving all of my databases and everything over to my backup next gen server, and it was clear. For a month.

Yesterday, I found the same virus running on both of my servers. This time, dedicated to tracking it down, and also having time to, I found it. More importantly, I found out how to get rid of it, but I haven't found out how exactly it found its way onto my server, or how to protect against it.

It is the Linux / DDOS trojan. It has an embedded rootkit. It is impossible to find by googling "Linux virus" as part of your search, as the only results returned are in relation to how Linux doesn't get viruses...

That article wasn't written at the time of the previous infection, and luckily I came across it this time. Otherwise, I would have had to re-image both servers, causing lots of down time. That was a pain, but at least now I know what was causing it, if that bastard finds his way onto my servers again.

After adding some better logging and stuff in the webserver, I determined that no error was causing the server to close. While I was starting the service with node webserver &, I think that I still needed to do nohup.

We shall see. My nohup'ed webserver is now running, and will run always in the future, whether it dies randomly or not. It's just the better way to start it.

After upgrading to Node.js 0.10.21 and implementing domains in Node.js, I was still getting errors. Since then, I've added code to listen for the response 'close' event, as well as the server 'clientError' event, in an effort to track down the issue. I think I may be on to something. At least then I'll be able to go to stack overflow or github with the issue in hopes that it will be fixed, or the act of listening for those events and logging the errors will lead me to the fix myself. Here's hoping.

Every morning I have an email from pingdom that my site went down overnight. I've upgraded to node.js 0.10.20 a few weeks ago to take advantage of some other bug fixes and optimizations, and it shows in the Google Analytics page load times (they're all 0 seconds)

I have had no luck tracking down the cause, but I read about how to prevent a server error from bringing down the site. I've implemented the suggested solution using node.js domains, and we will see what's happening tomorrow.

It might be the same thing because when I go into the server, node is still running my webserver, it just appears the socket was destroyed... We shall see. Wish me luck!