Rogers has detailed what caused its nationwide network outage yesterday, according to its CEO in an shared this afternoon:
I also want to share what we know about what happened yesterday. We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction early Friday morning. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
We know how much our customers rely on our networks and I sincerely apologize. We’re particularly troubled that some customers could not reach emergency services and we are addressing the issue as an urgent priority.
Rogers CEO Tony Staffieri also reiterated automatic bill credits are coming, saying “We will proactively credit all customers automatically for yesterday’s outage. This credit will be automatically applied to your account and no action is required from you.”
The CEO concluded, “We let you down yesterday. You have my personal commitment that we can, and will, do better.”
Last April, when Rogers suffered another daylong outage, the company blamed the cause on an Ericsson update.
Want to see more of our stories on Google?
P.S. Want to keep this site truly independent? Support us by buying us a beer, treating us to a coffee, or shopping through Amazon here. Links in this post are affiliate links, so we earn a tiny commission at no charge to you. Thanks for supporting independent Canadian media!
Didn’t they say the exact same thing when it happened in April 2021?
and probably say the same thing next time…
It’s the easy way out
Yes and this is part of their apology back then in 2021:
“In the letter, Staffieri said that the company is working to “fully understand the root cause of this outage” and to ensure that it won’t happen again.”
You would have thought they could have reversed the maintenance update? Most IT companies usually have rollback steps if an update causes issues.
It’s called rollback. If you apply an update to a system or network, always take notes and save configuration so you can roll back to the previous version if this happens in the future.
There should always be a basic phone and 3G data system that Rogers can fall back on if something happens to the 5G network, instead what we got was zero access to phone and data for nearly 24 hours, and that’s ridiculous.
Seems like they were trying to do a fix and go.
Seems like excuse making. It wasn’t just network services that were down. Their tiers weren’t transmitting at all. That implies more than just the network backhaul going down.
Nah, this was a core routers meltdown… everything is IP these days and BGP peering was down all this time.
Even if cell towers didn’t provide signal (and not just lack of IP connectivity) it’s because of backhaul feedback. If you have signal the phone won’t switch networks.
What I find interesting is that they didn’t dare take down towers to allow people to roam on Bhellus – with which they have a nation wide agreement. Maybe they were told not to as it would have had a snowball effect.
Redundancy is wonderful….but expensive. This is why you don’t get a CFO to run a telecom company.
I see.
I had tried manually selecting Rogers-EXT but it never succeeded.
I was automatically pushed onto Fido-EXT a number of times during the outage, but still kept disconnecting every few minutes.
I don’t think the connection was stable enough for you to do a manual network selection to grab on to a specific broadcast. These broadcast profiles are defined and prioritized on your SIM any way, so it probably would have been better to let your phone automatically latch on to whatever was available.
i wouldn’t be surprised if it was a stalled windows update lmfao
it is always a windows /server update that causes this. can they learn to test out the updates prior to implementing in production???
Gee…if only Rogers would be known for their thorough lab testing of software patches or for the precautions taken to upgrade a node at the time and soak the new software oir a few days/weeks.
But heck…this is 2022, neurons are overrated and apps get daily untested updates. Fok network reliability, who cares.
Note they didn’t dare mention the vendor name…law suits must be scary for the former CFO. BTW..where is Ted Rogers?
Dear Rogers,
please continue giving your boring core network upgrade to your coop students.
Signed: Vidéotron, Bell and Telus.
If they provide refund for a day out of your plan, they deserve to be dropped. Everyone should get at least 1/2 or a full month off for July. This is the 2nd time. It is just embarrassing. They should fire at least one executive.
Rogers 2021, outage after a maintenance update: “You have the commitment of our entire team, and our network partner Ericsson, that we will learn from what happened yesterday, to help ensure that this never happens again.
“Rogers 2022, outage after a maintenance update: “We let you down yesterday. You have my personal commitment that we can, and will, do better.”