About Me:
I'm a 32-year old Bronx livin' sarcastic bastard. If you cross me, I'll shred you. I have no problems sharing my opinion whether you want to hear it or not, so get used to it. There's a lot of it going on here. Hang around if you'd like and comment if you dare.
Meta
Recent Comments
Blogroll
Archives
Disclaimer
All e-mails sent to Vincent Ferrari or to any address at insignificantthoughts.com are considered for publication regardless of any disclaimers placed in the e-mail. It is automatically assumed that an e-mail sent is for publication purposes. Sending an e-mail with a disclaimer does not bind this site or its owners / moderators to adhere to your request. Thank you for playing!

What a week…

I really don’t know where to start with this, to be honest, so I’m just going to turn this into the narrative to end all narratives and hope it makes some kind of sense when it’s all done.

Two weeks ago, Beth was registering for classes, and it forced me to take a look at my expenses and try to trim the fat as much as possible. I cancelled a bunch of extraneous bills and consolidated others. In other words, I did what I needed to do to save a buck. One of many things I needed to cancel was AOL. Frankly, the only reason I kept it was to troubleshoot some of the suits’ AOL installs at work. I hadn’t used it in an eternity, so the time had come. The account was going to be closed.

Before I did it, I thought, “Ya know, I’ve heard about the horror stories with cancelling AOL, let’s record this convo.” I hooked up my recorder and made the call. I never expected what I got. Honestly. I knew it was going to be bad. In fact, I was expecting it to be extremely difficult. What I wasn’t expecting, however, was the utter assault I got the second the call started. In the back of my mind, though, I kept thinking “stay polite as long as you can; it’ll make a stronger point.” So I did. I pretty much kept my composure through most of the call.

Up until the point where the guy started harassing me about what I used the computer for. I mean, in the end, what difference did it make? I don’t want the account. As Anthony from the Opie and Anthony show put it, “Who gives a shit if I use the computer specifically to cancel AOL?” (Yes, Opie and Anthony ran my audio on their show and I didn’t even send it to them!)

Anyway, the conversation got progressively more testy at that point, ending with Jon, the AOL rep telling me the following after I told him I wasn’t interested in another sales pitch:

‘Kay. If you want me to cancel this account, you’re going to let me speak and, and give this paragraph. ‘Kay? ‘Cause if not, we can just argue all day, I d- I really don’t care, to be honest with you. But, you’re going to listen to me if you want this turned off. So: can I speak now?

At that point I gave in just to get the call over with, got the shpiel, got transferred to the recording that gave me the confirmation number and got the hell out of there. Then I sat there at my desk in utter disbelief. I had not only gotten an interesting call, I had just recorded what could commonly be called gold. Real gold, too. Not the kind that turns green when you sweat.

I didn’t do anything with it. I knew I was going to post it on here, I just wanted to wait a bit and see if I changed my mind. I didn’t, and let it fly on Tuesday morning.

Brilliant move on my part. From that point on, things just started moving REALLY quickly.

First, I had the bright idea to send it to Consumerist, one of my all-time favorite consumer advocacy arenas. Of course Ben posted it immediately. Hell, Joel Johnson even loved it, and we know how powerful he is (if you don’t, you’re an idiot; go read some other site you don’t belong here). When Ben posted it, he threw a direct link to the MP3 on this site in there. Within mere minutes, IT was down. As was Slobokan, LV Soda Pop, and three other sites on our server. I got 272 inbound referrals, all of which simultaneously downloaded the 3 meg MP3. In other words, server go boom. We were all down for about an hour.

When we got back up, I jokingly told Slobokan I should submit the story to Digg.com. His exact words at that point are recorded below for all posterity:

Slobokan: what the heck, digg it man

And with that, I made the post that changed the life of IT forever. Now, I should preface this quite clearly. I’m not a big fan of Digg. I make no bones about it, and have written a million posts bagging on them. I don’t like the mob mentality. I don’t like that the site has strayed from tech news. I don’t like a lot of things. One thing you can’t argue with, though, is that Digg is hardcore when it comes to consumer advocacy, and that’s why this post went there.

Then it started.

IT got slower.

And slower…

And then it was down…

And I mean down. Not just down, down down. The server load, at its peak hit 89%. I kid you not. We were getting blasted off the internet and convincingly so. I’ve been DDOS’ed before. This was a thousand times worse. Everything just stopped still.

We tried some triage stuff. Getting in, clearing out some WordPress plugins, disabling the anti-spam stuff I was running. Turning on WP-Cache. Nothing did anything. We were getting blasted and it was only getting worse. Finally, at around 6:30, we replaced the main page of insignificant thoughts with a banner you probably saw that said we’d be back when the server calmed down.

Around 10:30 I tried to compile the logs to check the damage. No dice. AW Stats sat on the logs for almost two hours before I gave up. In Cpanel, I got a hint of what was coming. I burned through 31 gigabytes of bandwidth in approximately 8 hours. To put that number in perspective, I go through about 8 a month. Hitwise, I made about 510,000 hits in that time period, compared to 360,000 for a typical month.

Yep. I got hammered.

Since IT was down, and I had nothing better to do, I started reading the comments on Digg. Typically, people were behind me, but there were large chunks of people who thought I was a fraud… I was wrong… The call was staged. Feh. Whatever. Two thousand diggs later, they were obviously in the minority so screw ‘em. I went to bed, and woke up the next morning. Checking the server logs, I saw we just had error after error after error. HTTPD (the http daemon, basically the service that serves up the pages) was up, down, up, down. My account was suspended (we thought it would help) then unsuspended (it didn’t). Basically, we wrecked the server in a profound way. But, with a static front page, things were calming down. The audio was gone, but were still getting bombed with hits from mirrors put up the day before by digg readers who wanted to help (I’m grateful to each and everyone of you by the way, if any of you read this).

Unfortunately, the problems didn’t end. Wednesday saw the audio picked up by Gear Live and his podcast (he did a special podcast of just my audio!), Wizbang and their podcast (Thanks for the support Charlie!), and more sites than I’ve ever even seen. Looking through my referral logs, we were getting hundreds of hits each from Anandtech, Metafilter, Digg, Boing Boing, TotalFark, Arstechnica and Digg, not to mention people leaving continuous streams of comments on the site through the mirrors. Basically, even with the site down, I was still getting hammered. At least things were calming down, though.

I went to work.

I was in one of my friends’ office and I was telling a few people what had happened the day before. We were joking around about it and such. Then, I went back to my office, and there was a voicemail on my mobile. It was my friend Paul. “10:30. You’re on Opie and Anthony.” I damn near fainted. I had just missed them talking about me. My audio, despite me not even submitting it to them had made it onto XM Satellite Radio, and was beamed to between 2 and 3 million listeners.

And the hits just kept on coming. In the early afternoon, I got an email from Dru Sefton wanting to discuss my story. We exchanged a few e-mails and she and I scheduled an interview for Monday night at 8pm. I was quite pleased. A mainstream media interview for little old me!

The day rolled on. IT was still down, but so be it. People were still seeing the story.

Nothing really eventful happened the rest of yesterday.

Today was the big end.

I put IT back up this morning, this time putting the MP3 on Putfile (thank GOD for Putfile!) to conserve bandwidth. Of course, server load has been pretty high all day, but I was managing. I also noticed that one of my heroes embarked on a new project this morning. Jason Calacanis (I bow before the in a completely manly way, Jason) decided that Digg was okay, but that they needed a touch of real journalism, so he launched Netscape.com, a brilliant mashup of the social news networking site Digg and real pavement-pounding journalism. In other words, Digg, but for people who don’t just want a buzz story and would like some followup and investigation. I was hooked immediately.

Then I saw it. Second story from the top was my cancellation story. Apparently, some diggers submitted it to test the integrity of Jason and his team. They ended up wearing lots of egg on their faces because of the two stories that bashed AOL, Jason left both of them. In other words, he has every bit the integrity I always knew he had.

Imagine launching a site, and on launch day, you’re getting assaulted by people who hate you, and you have enough confidence to just leave it all in place?

Later in the day, I got an e-mail from one of the anchors at Netscape. Dakota Smith was looking for a comment from me on the story. I thought it was kinda cool that she found me, so of course I volunteered. In her e-mail, though, was the following:

I’m an anchor for the new Netscape.com…your story about your dealings with AOL has been voted up on our site and I was hoping I could ask you a few quick questions about the whole ordeal.

We’ve already gotten a statement from AOL about the matter (did you know the guy you talked to was fired?) but want to follow-up with you.

WOW! A social site that follows up and pursues stories! AWESOME! And wait. Was that what I think it was? Jon was FIRED?

Sure enough, on that same story appeared the statement from AOL via spokesperson Nicholas Graham.

“At AOL, we have zero-tolerance for customer care incidents like this - which is deeply regrettable and also absolutely inexcusable. The employee in question violated our customer service guidelines and practices, and everything that AOL believes to be important in customer care - chief among them being respect for the member, and swiftly honoring their requests. This matter was dealt with immediately and appropriately, and the employee cited here is no longer with the Company.”

I had nothing to say at that point. I was basically stunned speechless. I had never actually complained, although I did bag Jon really badly in a survey about my experience with him, but somehow it got back to AOL and they shitcanned him. Makes you wonder how many other times he did this kind of thing, huh?

Then came the time to do my interview for Netscape:

Netscape Anchor Dakota Smith conducted an IM interview with Vincent Ferrari, the blogger behind Insignificant Thoughts and a 30-year-old Bronx resident, this morning. Here’s an excerpt of their IM exchange:

Dakota: What led you to post this?

Ferrari: Well, I had heard the horror stories about canceling AOL, and decided to post it. Honestly, I’m always looking for something interesting on my site, and this seemed to be something my readers would be interested in.

Dakota: For you, what was the most surprising part of the whole exchange?

Ferrari: Well first, the part about asking for my dad. I mean, I’m 30 years old, and both the card and the account are in my name…. Secondly…

Ferrari: When he told me he could “stall me” all day. I couldn’t believe he was going to make me wait just so he could cancel my account.

Dakota: What happened after? Did AOL reach out to you after or anything?

Ferrari: All I got from them afterward was a survey that asked (ironically enough) if Jonathan met my needs. Other than that, I’ve not heard a single word from AOL.

Dakota: Did you know the guy got fired? How do you feel about that?

Ferrari: Honestly, before your e-mail, I had no idea. I don’t know how I feel. Honestly? I’m not going to shed any tears for the guy. It sucks that he got fired, but maybe he’ll learn from it.

Dakota: Is it just AOL, you think, or just the nature of customer service in general?

Ferrari: Well, I think it is actually the nature of customer service to be honest. I think retention people are particularly bad because they can never accept that someone wants to cancel…

Ferrari: If I’m predisposed to canceling, just let me do it. I’m not staying no matter what. Retention people are the worst.

Ferrari: I don’t mind a token effort, but aggressive pursuing is really not appropriate.

Ferrari: Jonathan was definitely aggressive about it and wasn’t listening to me at all.

Dakota: They must be so bored with their jobs that they lash out…

Ferrari: Well, I know what it’s like. Their performance is basically “how many people did you keep today?” Under that kind of pressure, we’d all probably crack.

Ferrari: Doesn’t make him a bad person, just a guy in the wrong position for his skills or whatever.”

I’d say I was infinitely fair.

Look. I had no intention of getting this guy fired. Honestly, I was waiting for this to blow over a bit before I bitched to AOL. Really, all I wanted was an apology and a serious reprimand. I’m not totally comfortable with the fact that I cost this guy his job, but in reality, I’m not broken up over it either. Bummed is probably the closest word for it. Come to think of it, he cost himself his own job. Maybe in his next job, he’ll be more careful who he tells off.

So that’s the end, so far. Today, I came home from work to a call from AOL. We’ll see what they wanted tomorrow. Of course, I’ll record it, and of course that recording won’t be online (not doing that again any time soon, the other folks on this server would kick my ass!).

There were people who thought the story was fake. Obviously it wasn’t seeing as the guy got sacked.

There are people who think I got the guy fired. Screw each and every last one of you. I went in there completely polite. Instead of making it possible to cancel, he insisted on telling me how much my father used AOL when he doesn’t even have the software on his computer!

To the people who put up with me borking the server, sorry! What more can I say!?

And that ends that.

For now.

If anything else happens, I’ll let you know.

Technorati Tags: ,

This entry was posted on Thursday, June 15th, 2006 at 11:59 pm and is filed under Me Stuff. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

Viewing 98 Comments