Ontario Company Saw Rogers Outage Coming Weeks Ago: Downtime Alerts Surged 50x
Packetworks, a Waterloo company that builds and maintains networks for cities, hospitals, and businesses, said that disruption alerts on the Rogers network went from maybe one per week to a whopping 50 per week in the days leading up to the outage.
“When you get too many notices from your monitoring systems it is telling you: you are on the edge, something is going to happen,” said Packetworks president John Fagg.
“It was the Rogers monitoring system that is sending us notices saying, ‘Hey, we see a problem here and we see a problem there.’ They are small, a computer can see it, but most people don’t see those problems,” he added.
The monitoring system is there to detect problems in the first place, but it was the sheer volume of alerts Packetwork was seeing on a weekly basis that, in hindsight, was alarming.
“If you get one a year that is not a problem, but if you start getting one a week you know that equipment is going to be failing pretty soon,” said Fagg. “Once a week we were getting several, 50 at a time, that sort of thing.”
Rogers CEO Tony Staffieri said on Saturday that the service breakdown was caused by “a network system failure following a maintenance update.”
According to Fagg, Rogers even got in touch with Packetworks before Friday’s outage. “They contacted us a couple of times wanting to do upgrades because their optics are showing signs of deterioration,” he said.
Fagg believes a network crash of this magnitude should be a wake-up call for the Canadian government, regulators, and telecom giants alike to invest more in network infrastructure and improve the country’s telecom landscape.
After all, last week’s incident hit everyone from citizens and businesses to government agencies, smaller telecom operators, and even banking systems like INTERAC and Visa
Consumer advocacy groups, internet organizations, and more are calling for public inquiries into the disruption. Canada’s telecom authority, the CRTC, has demanded answers from Rogers over the “unacceptable” outage.
Fortunately, Packetworks and its customers were unaffected by last week’s debacle thanks to the company maintaining a redundant system. Packetworks simply switched over to the Bell network when Rogers went down, keeping customers online.