Thursday afternoon, August 2nd: Fotolog operations team sent me the following photo of a shipment of a Sun 6540 disk array. Essentially our new backup system:
The equipment on the bottom was damaged, the boxes were stacked (inspite of Sun stickers saying "do not stack") and the entire package was wet, from what I gather not soaked, just wet.
Backup is important to Fotolog — with 280M photos, billions of guest book messages and a vendor mess up a few weeks ago that corrupted our guestbook database, this wasn't what we needed. But what started as a mess turned into a customer service story from the gods. The Sun team went right to work, through the weekend — by 9.24pm on Monday (2 working days) we had this sorted out and the shipping details for the replacement boxes.
Shortly after getting the picture and alert from my CTO I forwarded it to Jonathan Schwartz at Sun, asked for his help in sorting this mess out. Within hours I got the following note back:
From: Jonathan Schwartz .
Date: August 3, 2007 12:30:10 AM EDT
To: John Borthwick .
Subject: Re: Contact info
Instantly – stay tuned.
Let me know if there's anything I can do to help out – we will get this right asap.
Instantly is a tall order but by Friday morning things were moving, they moved all weekend and by 9.24pm on Monday we had shipment details for the replacement boxes. Great service, no finger pointing, no discussion of why and who let this happen — the organization and everything I saw was focussed on execution and getting the client (us) the drives.
Communication was excellent the team sent frequent updates and Schwartz checked in periodically to make sure things were on track. I know Schwartz from college days, but we haven't spoken since I left Time Warner — this was essentially an email sent into the dark morass of corporate America, which yielded fantastic results, way better than I could have expected. I dont have the time to cull through the emails but I noted one from Schwartz cc'd a core customer service group email — suggests he has setup a mechanism to respond to issues like this, building processes like this is hard, particularly as you transition a huge business.
Good stuff is happening over there at Sun.