Over ten years ago I vividly remember the first moment I saw ‘Safe to fail.’ I was working for a large scale telecommunications organisation. Most of us working there were pretty young. With our youth came inexperience. I learnt that through a project re-assignment I was going to have a new manager and was anxious of what he was like. This was until a valued friend chipped in and regaled a story.
He told me that he had once reported to this manager and that I had nothing to fear. He recalled where he was once working on a email server and had setup a rule on the server to bounce an automated reply to not contact the server directly if an email was directly sent to it. Much to his horror he walked into work one day to find that the whole telecommunications network was crippled and this his server had been a strong part of the cause. It turned out that another automated server sent his server an email and the two of them began a bouncing war to the tune of over 10,000 emails being bounced per second between the two of them.
His manager once removed wanted heads to roll but his manager stood firm, took the fallout and accountability and refused to let that happen.
This manager knew what safe to fail was and stood up for it.
There have been many big stories lately of companies making gross errors, but organisations don’t have to hide behind them as savvy customers are more than willing to stay loyal to companies that stand tall and say ‘We are sorry, it was our fault.’
Atlassian had such an incident in April 2010 when a security breach exposed thousands of passwords. There were many responses to their openness, both positive and negative.
Zappos had a different kind of incident when in May 2010 they incorrectly priced their stock at one of their sister sites 6pm.com. They found the glitch a few hours later but to much surprise they honored the prices at a cost of 1.6 million (if you take their retail and not cost price). Word spread like wildfire and subsequent media resulted in a significant increase in their sales.
This week I found another major example crop up from one of my favourite pastimes – MMORPGs (Massive Multiplayer Online Role Playing Games).
Trion recently released a MMORPG called Rift that has been boasted as the potential replacement to Blizzards successful World of Warcraft.
The release itself went quite flawlessly barring the fact that the game was so popular that the servers were flooded and queue times were staggeringly high. But the servers stayed up despite the load and the quality of the gameplay was high with very few bugs evident.
And then the security issues began. It started out fairly lightly as you would expect in retrospect, lurkers waiting for the players to actually accumulate virtual currency. There was a story or two of people having their accounts ‘hacked’ where someone else logged in as them and their virtual currency was re-directed elsewhere. Complaints on the forums resulted in attacks on those unfortunate enough to loose control of their account with citings of ‘poor security’ as the cause.
Within weeks it reached significant proportions. Although supposedly less than 1% of accounts were compromised it was rare to find a person who didn’t have a friend that has hacked. The forums became a mess of witch hunts. Players, sister fan sites, downloadable combat parsers, keyloggers and mostly the users were blamed as the cause.
On the 19th the Executive Producer of the game, Scott Hartsman, released a statement to say whilst some of the hacking was caused by traditional security holes such as keyloggers, and bad/re-used passwords from other games, there was indeed a defect that caused a man in the middle vulnerability that meant neither an account nor a password was required in order to hijack accounts. Kudos was even given to the user that found the breach. But importantly the breach was fixed within two hours (in MMORPG time this is like a heartbeat).
Replies from customers were almost unanimously positive with most boasting appreciation over the honesty that the company provided. My favourite reply was,
“I had cancelled my sub over this even though my account was never attacked, I cancelled out of fear I would be hit sooner or later. After this and the impressive response from the company I have since resubbed my account and will spend some time on my toons to enjoy the beautiful game that it is.”
by Mauvelence, demonstrating that being open and honest can lead to a more loyal customer base.
Will a security incident like this ever happen again in an MMORPG? Not likely – all future companies will look at this and give it the focus and attention it truly did deserve prior to a product going live.
So what is safe to fail? It is having the guts to say you were wrong (though ironically only Atlassian said ‘We’re sorry!’). Safe to fail not only applies on a business to customer context but it also applies internally.
I would keenly love to know how the employee at Trion who caused the oversight in security was treated. Were they fired? Were they reprimanded? Or were they praised?
Human beings are completely imperfect. It is what makes each and every one of us so special. Creative human beings probably more so as creativity breeds innovation. So next time someone you know does something really wrong laugh it off with them and look for the positive learnings and results that have come out of it.
Agile Forest 