What Happened Today?

Calvin Yu, 9:48 pm, 2 Jun 2008

Some of you might have noticed that we had some uptime problem today.  The cause of the interruption was that one of our Skribit bloggers had a post that got dugg heavily today:

Here’s where we are at the end of the day:

  • First, congrats to Jobmob on getting their post dugg.  Great post.
  • Despite the downtime on our site and our widget, this did not affect the page load times of any of our user’s blogs.  We consider that a very (very) minor victory.  I wish I can promise you that we’ll never suffer from downtime again, but that is unrealistic.   What is realistic is to expect that your site will not be negatively affected when we have problems on ours, and that we’ll fail as gracefully as possible when something on our side does go wrong.
  • About an hour ago, we made a patch to Skribit which should withstand the level of activity we saw today, and we’re continuing to explore other ways of squeezing more performance out of the system.

Lastly, I want to express that we don’t consider any downtime from a blog being dugg as being acceptable level of service.  We should be able to withstand many diggs without any noticeable affect on site performance, and are taking whatever action necessary to make that happen.

Tags: , ,

9 Responses


  1. Calvin, thanks for the shout out. Much appreciated.

    I did notice at one point that Skribit suggestions were no longer coming up. Although I was thankful for the iframe not being able to block page loads (THAT would have sucked yesterday in a big way) Firefox would still throw an error once the server connection timed out, which definitely didn’t help in any way.

    Hopefully your widget will be able to fail more gracefully in the future but it does sound like you guys know what you need to do and will keep pushing to run a top-notch service.

    Jacob from JobMob, June 3rd, 2008 at 3:34 am

  2. Congratulations on the Digg suggess though - I think most websites would struggle with a Digg homepage ‘rush’ :)

    Tom, June 3rd, 2008 at 3:58 am

  3. any screenshots of what the widget looked like on jobmob? How did it fail and what did it look like?

    Congrats though Jacob- very funny post!

    titanium_geek, June 3rd, 2008 at 7:37 am

  4. Most people will just see a blank white area where the widget should be. If they’re on that page long enough, they will get the browser specific error page or a very plain server error page if they have the browser specific one turned off.

    Calvin Yu, June 3rd, 2008 at 7:08 pm

  5. Thanks Tom and titanium_geek!

    There were 2 keys to my site not going down - suffering the Digg Effect, if you will:

    1) It’s WordPress-based, and I use the caching plugin Wp-cache2. The vast majority of visitors were basically seeing static html coming from the cache directory instead of the usual pages generated by WP from MySql. Static html loads quick and barely taxes the web server.

    2) Although not always terrific, I’m hosted by MediaTemple, same as Skribit. They’ve had many uptime issues recently but when a site gets a massive burst of traffic, their Grid can take it. This is the 3rd time I’ve seen that, albeit the biggest by far.

    Jacob from JobMob, June 4th, 2008 at 5:22 am

  6. Just out of curiosity Jacob, what told you that you were being dugg? I noticed the first comment on that post was from November 2007, so the post has been out for awhile. Did you just find out during a routine daily check, or did you have a special tool that does that?

    Calvin Yu, June 4th, 2008 at 5:32 am

  7. Yes, the article wasn’t new. I rarely submit my stuff to social media because I don’t have power user status anywhere, so I prefer to just sit tight until someone who does comes along. It’s happened a few times to me already but this was the biggest.

    The digger enjoyed the post so much that he emailed me about the digging. That was almost 12 hours before it went popular.

    Depending on what specifically you’d like to know, you may want to check the stuff in the Digg Labs. Also, there are lots of digg Yahoo Widgets and Google Gadgets and there might already be one that will watch for when someone diggs your site. Otherwise, there’s probably a way to use Google Alerts for this, and there might be analytics tools that will allow you to create alerts via filters on your logs. Worst case - not SO terrible but somewhat annoying - would be to code a little filter yourself that compares with past digg referrers to tell you when something new pops up.

    Jacob from JobMob, June 5th, 2008 at 9:12 am

  8. Btw, you guys really need to use the Subscribe to Comments plugin for WordPress. Personally, I don’t usually keep coming back to see if people have responded to my comments.

    Jacob from JobMob, June 5th, 2008 at 9:13 am

  9. I have tried repeatedly to contact the Skribit staff about an urgent problem (unrelated to the uptime), but I keep getting the following error when trying to send the contact form:

    Error: Hidden form fields were submitted that should not have been. Please try again. This usually happens because an automated script attempted to submit this form.

    Please can someone from the staff contact me as soon as possible.

    Thank you,
    Silla

    Silla, June 5th, 2008 at 5:43 pm

Leave a Reply