Flakey Flickr goes down. Again

Flickr, the popular photo sharing web site that spends more time on the canvas than a pulverized Frank Bruno, is down again.

It's the third serious outage this year. In January the site notified users with this message,

"The Flickr database is at the spa: Treatment includes a full-body exfoliation scrub, followed by an avocado cleansing mask and finishes with an invigorating steam bath. We'll be back as soon as we can!"

In March, following the company's $18m acquisition by Yahoo!, another outage hit North American users in peak-time. Flickr explained,

"Being Owned By Yahoo Fails to Prevent Harddrive Failures."

Ho, ho!

Now sounding like a useless Uncle who's fused the lights on the family Christmas tree, this time Flickr explained, "We didn't read the contract carefully enough. We assumed that Yahoo had some deal with ... magnets ... and, uh, electricity ...  where harddrives would never die once we signed."

Stop, guys, you're killing us! Hard drives fail and so administrators need to ensure redundancy sees no user downtime at all.

"... But it turns out that that was wrong and the ill-paired failure of two different drives within an hour of each other and some not-as-advertised raid controller flakiness in one of the database masters meant that we had to take the site down to reconfigure. Since it is down, we are running some power maintenance as well and plugging various servers into other slots. Should be back up well before 8pm pacific."

Then the Flickr system maintainers discovered even more problems.

"Oy. We were back up before 8pm, but had some serious database replication issues. Working on it now and will report back with an estimate soon."

Today's outage hit east coast users early evening, and the cute excuse sounded rather more desperate and pleading.

"Is there a chiropractor in the house?" was the title of a post on the Flkr blog, which somehow survived the downtime. (While Flickr users couldn't get at their photographs, Flckr staff could keep blogging!)

"If our master database experiences a hardware hiccup, it's like you're stuck on your loungeroom floor unable to move because you've slipped a disc. We're fixing it now, and will be as quick as we can," wrote one George Oates.

But there's a silver lining to every cloud. The message promised a small miracle - a few hours without any downtime at all:

"But! As an extra special treat, while the site was paralysed, we were able to do the database updates we were going to tonight at 7pm in the meantime, so no downtime tonight!" promised George.

No downtime tonight? So far the promise has held true... but it's not yet midnight Pacific Time, so we let's not get premature.

Quality is Flickr's nuclear weapon

Flickr's success and stellar press coverage has been well deserved.

Flickr makes posting and sharing pictures trivial, and it isn't hard to see how it has grown a community of enthusiastic photo browsers. Unlike many similar efforts over the years, which feature goofed-out ravers, dull holiday snaps and eventually, porn wannabees, Flickr's secret lies in very subtly promoting good editorial judgement. It's a pleasure to browse not because of some obscure techie APIs, or the overhyped and mostly useless "tags", but because of the site owners' gentle and unobtrusive promotion of the best photographs, which in itself, makes the many amazing images on the site easy to find. [All these were no more than three clicks away from the home page].

Quality seems to breed quality, and at its best, Flickr an enchanting window onto the world.

But at the same time, photographs are not a novelty. To trust your family album to the internet requires more than a funky UI and a bit of New Age press coverage. The system must be durable, reliable, and as a user, you must have a guarantee that you'll be able to retrieve them in forty or sixty years time: just as the shoebox of family photos under the bed. In other words digital must prove itself to be permanent.

Can internet people think in such time spans? It's hard to imagine when digital culture is so transitory and self-obsoleting. And especially when sites such as Flickr pay such a cavalier disregard to fault tolerant systems.

The best engineers, David Rosenthal reminded us recently, are pessimistic. They imagine the worst that can happen, and prepare for it. They don't spent the day dreaming up amusing analogies for a hard disc failure.

If Yahoo!'s Flickr wants to avoid being dubbed Flakr, it needs to take this seriously. Recently, we suggested mandatory federal prison terms for flakey photo web sites that lose our photographs - a reader popular with Register readers.®

Bootnote: Flakr's downtime may be as much due to laziness as it is to incompetence, thinks one user. In the January outage, one fan grumbled -

"Most popular sites schedule downtime on weekends and late at night.  It's too bad if your developers and admins would like to be home asleep at midnight.  Tough.  To grow a great system like Flickr takes sacrifice.  You should be sacrificing your nights instead of pissing off your valuable users.  I've spent many late nights working by candlelight so that my users would be happy in the morning:
This should have been done OFF PEAK. [His emphasis].

So sending the BOFH's Guide to Learn Yourself Uptime over to these emergent people, may not be enough.

Related stories

Digital memories: we can forget them for you wholesale!
Digital memories: cheap to take, cheaper to lose
Strength through pessimism! Keeping your stuff safe