On the evening of Sept. 17, one of the biggest hosting providers in the US, Layered Technologies in Plano, Texas, suffered a major security breech. Hackers managed to access the company's support database and download client data on something between 5,000 and 6,000 user accounts.
The company's notification to it customers stated that "Due to the significant amount of uncertainty in determining which accounts may have been impacted ... we are asking all of our clients to change the login credentials for all host details they have submitted in the past 2 years." Essentially, what Layered is asking their clients to do is to change every password they have.
Now nothing is known about how the hackers got in, and my guess is that we will probably never find out which is not unreasonable - why would Layered want to tell us if their staff or their system screwed up? Obviously full disclosure would be preferable because we might all learn something that would protect more people, but the realities of branding and marketing have to be dealt with ...
What this even made me wonder is how well you are prepared for such an eventuality. First, if you have customers that fall under any kind of data protection legislation then you're going to have to notify them.
This would be fine, but you've got Web applications that drive scores of business processes in your organization and you've got user accounts, database passwords, e-mail accounts, management accounts, literally hundreds of passwords that are your first line of security defense and when you change them many of your customers won't be contactable!
And there's the problem: You HAVE to change all the passwords yourself -- you can't notify all account holders and have them do it because that will take too long and it only needs hackers to gain access to one account with any management privileges to make it game over for your security. The worst situation is a compromised account that you don't know about that the hackers don't use until months after a security problem - by that time you've stopped looking for problems and if they are cunning you might not have any idea of what they are up to.
So if you have to change all of these passwords, notify all stakeholders, employees, customers, and partners do you have any idea how long it will take you? Do you have any idea of how you'll actually go about doing it?
Here's an interesting part of the problem: Many of the account holders will only be contactable in the short term via e-mail so when you reset their passwords they are going to call you. Do you have enough support staff on the phones to handle the support tsunami?
One strategy I can think of is to change your corporate Web site's home page to announce the problem and try to structure who calls you on what number: customers to one number, staff to another, and so on and try to put support effort where it is needed to try to minimize the business impact.
Of course that does suppose that you actually know where in your business processes the effort is actually needed, which is something that isn't necessarily obvious if you haven't studied it. For example, it may be that your supply line management is the key to ensuring that your business can function so those accounts need to be addressed immediately while fixing the customer accounts can be shelved for 24 hours. Or is that no more than six hours? Do you know?
Oh and how do you validate all of the people calling for password resets? Unless you have some kind of lost password reset system that you are absolutely certain isn't compromised then you are going to have to do something like ask for credit card data and social security numbers to validate the callers. Of course the hackers may have that data and "social engineering" is easy.
Obviously, what I'm talking about is disaster planning, but I'd suggest that having a Web application disaster plan is a very different context from your usual disaster plan (which, of course, you do have ... don't you?).
Whatever the cause of the breech, the scale of the problems Layered and its customers are facing is monumental and I'm betting that hardly anyone had a disaster plan in place. I wish Layered and its customers the best of luck - they're going to need it.