CRTC Report Explains the ‘Catastrophic’ Rogers Outage from 2022 [Update]

The CRTC has released its commissioned report by Xona Partners that has summarized the “catastrophic” 2022 Rogers outage that affected both wireless and wireline services that impacted over 12 million personal and business customers, that lasted over 26 hours, before services started to be restored.

“In this report we detail the findings for the period before and during the outage and outline the measures that Rogers has since implemented to address deficiencies in its network design and processes. This report is primarily based on an extensive independent review of the Rogers responses to multiple rounds of questions and meetings with the Rogers technical and management staff during this assessment, as well as information Rogers provided in response to the CRTC’s request for information (RFI) after the outage,” wrote Xona Partners.

So what happened and caused the infamous outage?

Xona Partners explained the Rogers’ wireless and wireline networks use a shared IP core network to manage and route data traffic both internally and externally. In July 2022, Rogers was upgrading this IP core network in a seven-phase process. The outage occurred during the sixth phase of this upgrade. This core network handles all data traffic, so the outage disrupted both wireless and wireline services. The upgrade process was intended to improve network performance but inadvertently led to a significant service disruption, ironically.

Specifically, the outage was caused by an error in configuring the distribution routers within the Rogers IP network. Rogers staff removed the Access Control List policy filter from the distribution routers, leading to an overwhelming flood of IP routing information into the core network routers. This surge exceeded the core routers’ capacity, causing them to crash within minutes.

As a result, user traffic could not be routed correctly, leading to a complete halt in services such as mobile, home phone, internet, business wireline connectivity, and critical 9-1-1 calling.

“The July 2022 outage exposed the absence of overload protection on the core network routers. The network failure could have been prevented had the core network routers been configured with an overload limit that specifies the maximum acceptable number of IP routing data the router can support,” reads the report.

Describing a “deficiency in the change management process,” The report says the configuration error that caused the outage was due to a change management oversight by Rogers staff. In an effort to clean up configuration files, staff removed the policy filter that prevented IP route flooding.

The change management process did not flag this error. This configuration change was part of the sixth phase in a seven-phase network upgrade. Although the risk was initially assessed as high, it was downgraded to low after successful prior phases, leading to insufficient scrutiny and testing, which contravenes industry norms requiring high scrutiny for such changes, said Xona.

As for why recovery efforts took so long? The report says the Rogers’ management network relied on its IP core network, which failed during the outage, preventing remote access for troubleshooting. Lack of redundant connectivity from alternative providers forced staff to physically visit remote sites, delaying recovery.

Not only that, but Rogers employees relied on the company’s mobile and Internet services. When both networks failed, internal communication was disrupted. Rogers had to distribute SIM cards from rival telecoms to reestablish connectivity, slowing response times.

There was also delayed access to error logs from failed routers and confusion over multiple configuration changes during maintenance that stalled the initial diagnosis. It took Rogers a whopping 14 hours to identify the root cause of the outage, writes Xona Partners. Once the root cause was found, recovery proceeded as expected.

What changes did Rogers implement to prevent a future outage of this magnitude?

Rogers made critical changes to router configurations to prevent IP routing data floods. A separate management network was created for troubleshooting, and backup connectivity from third-party providers was added to the most important sites. Tools were also introduced to validate router configuration changes.

The company also is separating its IP core networks for wireless and wireline services. This move aims to ensure that if one core network fails, the other remains online.

Rogers also improved its change management process with a new risk assessment algorithm, improved collaboration, and automation. Incident management was strengthened with better guidelines, leadership roles, and prioritized alarms. Backup communications from third-party providers were provided to ensure connectivity during outages.

As for some tech recommendations for Rogers moving forward, the report says the company should consider “low earth orbit satellite constellations to provide remote sites with backup connectivity and consider emerging direct-to-cell constellations for emergency 9-1-1 calling.” Since Rogers is already a partner with SpaceX and its Starlink Direct to Cell network, this should be easy enough to implement.

The CRTC paid $230,000 for Xona Partners for a “forensic level technical review” of the Rogers outage.

The 2022 Rogers outage disrupted over 12 million Canadians and also took down some payment networks that relied on the mobile network for debit and credit card processing.

How were you affected when the Rogers outage took place two years ago? Where were you?

Update July 5, 2024, 2:52pm PDT: A Rogers spokesperson told iPhone in Canada in a statement, “As part of issuance of report, the CRTC released two letters confirming that all the recommendations were actions and our response.”

The CRTC sent a letter to Rogers dated July 4, 2024, addressed to Jason Ball, Rogers Communications Canada Inc., stating, “Based on Xona’s findings, the measures taken by Rogers have addressed the cause of the outage. Xona also made additional recommendations to Rogers to further enhance the reliability and resilience of their network, and Rogers has confirmed the implementation of all measures.”

Earlier this year, a Rogers letter to the CRTC, dated January 17, 2024 in response to recommendations cited CRTC report, explained, “Rogers has responded below to each of the Report’s recommendations for additional measures that could be undertaken by Rogers to further improve network resiliency. All of these recommendations were previously identified as part of Rogers’ earlier review and we welcome the opportunity to provide an update. Each of the proposed recommendations has been implemented.”