Rapid growth over the six years since Ladbrokes Australia launched has meant that the wagering company is left dealing with a not insignificant level of technical debt, according to Alistair Roberts
“Coming from a startup environment, you can make mistakes early on,” Roberts, Ladbrokes’ head of technology operations, yesterday told the PagerDuty Connect event in Sydney. “Sometimes you make a mistake on day one and it comes back to bite you later on.”
A particular setting or a table in a database that receives tens of queries might not be a big issue — but a few years later when the database is handling thousands of queries a second, those decisions have a habit of coming back to haunt you.
“As we’ve grown so big, if we have a small issue — a couple of years ago it might have been a little bit of a blip.” Now that same issue might cause “major problems” for the company, Roberts said.
A minute of downtime can be worth more than $100,000 to Ladbrokes — and that figure increases every year.
“It’s not just the trade we lose; it’s getting customers back,” Roberts said.
As a result, incident reporting and management are a key concern for the company’s tech team.
For a number of years the company has been using PagerDuty for incident management. Now when buying off the shelf tools, integration with PagerDuty is one of the factors the team considers, Roberts said.
All of Ladbrokes’ tech support, devs and sysadmins use the system. “It’s making our teams smarter by having them develop and set their alerts for what their environments are,” Roberts said.
“If everything is critical — nothing is,” he explained. “If there are hundreds of alerts going off and they’re all just for nothing then it’s [not a] real alert, so we’re trying to get smarter with our alerting.”
The PagerDuty alerts include relevant stakeholders, such as executives, Roberts said. “So executives know what’s going on – they see the alert, see it’s been actioned, see who’s dealing with it. It gives them that peace of mind.”
As part of addressing the issue of technical debt the company is breaking down its “quite large” code base into microservices. “Anything new that goes out is a microservice now,” Roberts said.
The company is increasingly moving to an Agile framework with a DevOps culture, he said, with dev teams shifting to a “you built it, you support it” mode, he said.
“It’s no longer ‘it goes out; I forget about it’,” Roberts said. “No — if it breaks at three o’clock in the morning it’s those devs that are getting called.”