Thank You Joe C - The Planet Fire
No, not Joe C the midget that hung around Kid Rock, Joe C that saved me from leaving the Planet this morning.
I’ve been running on fully managed dedicated servers for the last 4-5 years. A few months ago, I decided to move on to a self managed host. I save 30% and have much better hardware this way. The problem is that I have to learn as I go on some issues. Some issues, I just pay someone to help with. It’s a trade off that will be worth it in the long run. So, lets move to the Planet’s Explosion.
It’s unfortunate to have such a disaster. This is costing the Planet a great deal of money to repair, honor SLA agreements, and the income from customers they’ve lost and will lose. I was going to be one of them if I wasn’t online this morning.
So, the disaster happened around 4:30pm on Saturday. I noticed at about 6pm and since I couldn’t access my server in any form, I put in a manual reboot ticket. It was promptly closed and a ticket was opened on my account stating the outage. Ok, sorry to hear this… I’ll be patient. I was patient. Sunday night, I started getting a bit fidgety. I woke up Monday morning to an email saying we should be up. I wasn’t. The funny thing was that some sites I knew were affected as well were, such as Jason Dunn’s sites. I opened another ticket.
I am not going to slam staff and cite each and every little thing that pissed me off this week, but I will summarize the experience…
Most of Monday, I was patient. I was at work and there was little I could do anyhow. Monday evening, I started to beat on doors in the form of tickets and calls. Sure, we see your problem, but we can’t fix it… huh, wtf?
Anyhow, Monday night and almost all day Tuesday. I called in and complained but nobody was listening. I had some that were giving cookie cutter answers. I got put back into the queue for another 15 minute hold a couple times. Nobody seemed to give a shit. They all assumed that my issue was something that would be fixed when everyone else was fixed. Well, most everyone was already fixed. “Sir, it’s an internal network problem”. When I asked how to get it fixed, I heard answers like “I don’t know” and “we just have to wait”.
This crap went on for almost a day and a half. I made my choice up last night that I was going to try one more time and if I wasn’t up at 7am this morning, I was going to move on to another company. I called tech support again at 11:30pm. I got a hold of Joe C in support. He actually took the time to actually look at my server to see what the problem was rather than just assume it was x, y, or z.
After 20 minutes, Joe had my email flowing and my webpages flying up. Since I am not a linux guru, I couldn’t really explain a lot about it but it was something to do with a resolver config setting or something. He also found that were were being pounded by 20 or so IP addresses using a huge number of connections. The slamming is not what was causing the dns issue, but it wasn’t helping the situation.
Anyhow, Joe took the time to teach me a couple things as well. He is a true asset to the Planet. He single handedly saved their company a customer. Thanks Joe C.
