One Ontario-based networking company says it saw last week’s Rogers outage brewing as it noticed downtime alerts grow exponentially in the weeks prior — reports The Record.
Packetworks, a Waterloo company that builds and maintains networks for cities, hospitals, and businesses, said that disruption alerts on the Rogers network went from maybe one per week to a whopping 50 per week in the days leading up to the outage.
“When you get too many notices from your monitoring systems it is telling you: you are on the edge, something is going to happen,” said Packetworks president John Fagg.
“It was the Rogers monitoring system that is sending us notices saying, ‘Hey, we see a problem here and we see a problem there.’ They are small, a computer can see it, but most people don’t see those problems,” he added.
The monitoring system is there to detect problems in the first place, but it was the sheer volume of alerts Packetwork was seeing on a weekly basis that, in hindsight, was alarming.
“If you get one a year that is not a problem, but if you start getting one a week you know that equipment is going to be failing pretty soon,” said Fagg. “Once a week we were getting several, 50 at a time, that sort of thing.”
Rogers CEO Tony Staffieri said on Saturday that the service breakdown was caused by “a network system failure following a maintenance update.”
According to Fagg, Rogers even got in touch with Packetworks before Friday’s outage. “They contacted us a couple of times wanting to do upgrades because their optics are showing signs of deterioration,” he said.
Fagg believes a network crash of this magnitude should be a wake-up call for the Canadian government, regulators, and telecom giants alike to invest more in network infrastructure and improve the country’s telecom landscape.
After all, last week’s incident hit everyone from citizens and businesses to government agencies, smaller telecom operators, and even banking systems like INTERAC and Visa
Consumer advocacy groups, internet organizations, and more are calling for public inquiries into the disruption. Canada’s telecom authority, the CRTC, has demanded answers from Rogers over the “unacceptable” outage.
Fortunately, Packetworks and its customers were unaffected by last week’s debacle thanks to the company maintaining a redundant system. Packetworks simply switched over to the Bell network when Rogers went down, keeping customers online.
Want to see more of our stories on Google?
P.S. Want to keep this site truly independent? Support us by buying us a beer, treating us to a coffee, or shopping through Amazon here. Links in this post are affiliate links, so we earn a tiny commission at no charge to you. Thanks for supporting independent Canadian media!
it was the sheer volume of alerts Packetwork was seeing on a weekly basis that, in hindsight, was alarming.
——
Wouldn’t it be satisfying to see Rogers face criminal charges for shunning their responsibilities by ignoring imminent catastrophe indications?
Rogers fully knows how businesses, government institutions as well as private users depend and rely on their services.
Perhaps a full inquiry should be explored due to the level of how vital the internet has become in our modern society.
Unfortunately we’re talking a private company here, no matter what the impact was. Unless the law mandates a 5-nines reliability and sets out penalties for not meeting that target, it will always going to be “best efforts” (or worst).
Definitely a wake-up call to expand network redundancies and improve private network competition!
Perhaps one, IMO a new dedicated public data-centric crown corporation Tier 1 network with traditional services (voice calls, texting, TV, etc.) using only new data-based means (IPTV, VoIP (includes Wi-Fi Calling, VoLTE, etc.), etc.) should be created with only recent modern telecommunication technologies including fibre optics and wireless cellular data tech. No support for legacy technologies (includes fibre internet gateway WAN connections) including copper wire telecommunication lines (except for only providing power, converting between wireless and wired signals, LAN connections, etc.) such as coaxial TV cable lines, telephone and DSL landlines, etc. Instead, public network should provide incentives and deals to users to upgrade their legacy devices (e.g. provide consumers discounts for trading legacy phone without native VoIP support with a new data-capable VoIP phone), let private networks support legacy tech since they already have the equipment and infrastructure, and provide a backup redundancy for private networks. Public entities and agencies, and every government level be mandated to support the public network as their primary telecom provider for their telecommunication needs including internal secure network and public telecommunications so government’s and public entities’ traffic be primarily separate from private networks, freeing up traffic for more private entities and individual use. Backup private providers can be optionally supported. For every new device issued, at least government contract and carrier-locked devices be issued with eSIM programmed with appropriate service plan to minimize physical SIM issuance. Physical SIMs still be issued at least for unlocked devices and BYODs. Unlocked device holders and BYOD users be still given the option to use eSIM. Every eSIM and physical SIM be equipped with various applications and developer tools including e-Wallet and card emulation tools such as Mobile FeliCa, Calypso, MIFARE, EMV, ISO/IEC 14443, etc. Also, every physical SIM be equipped with SWP and an integrated NFC antenna for NFC Contactless mobile transactions of any kind (payment, transit, access control, etc.) including NFC payments via non-standard Contactless means (i.e. not EMV methods such as Interac, Visa, Mastercard, etc.) such as MIFARE (PRESTO, Compass, store rewards gift card, etc.), Calypso (OPUS, etc.), FeliCa (e.g. Suica, PASMO, iD, QUICPay, nanaco, etc.), etc. If SIM detects a functional NFC in existing device, then SIM would automatically switch to functional device NFC.
Two, government should work with schools to assist startups in creating their own Tier 1 telecom network. This must be primarily prioritized over the next aspect so there is fresh new competition and minimizes problems with large monopolies regardless of national origin. Three, some legislation must be revised or even abolished so foreign telecoms of any size can compete with principled restrictions (e.g. not allow telecom companies from regimes such as PRC’s CCP) against local establishments.
Call me a cynic, but I don’t think that’s the best solution. Imagine the scenario of having a leader of the government with dictatorial leanings in charge (eg: some trust fund kid from Quebec having grown up with a Marxist leaning father for example) leaves the country vulnerable to a single, highly censored, government run autocratic system at the stroke of a pen.
Agreed. I understand the sentiment of thinking a new crown corp might improve things, but governments never run anything well. And, as we’ve seen in Canada, an incompetent government can turn an already poorly run bureaucracy into a complete dumpster fire. Add in one that holds Canadians in disdain, doesn’t understand basic economics, destroys the good in a vain attempt to reach a perverted perfection and feels they know what’s best for you and will impose that on the citizenry regardless of the cost and you have the recipe for us to be Venezuela or Cuba of the north. Not a great idea to then give them even more levers to control the flow of discourse and information in Canada. They’re busily trying to seize as much control as possible now, no reason to look for more ways for them.
Imagine the screw-ups at every change of gov’t..
I don’t think my thoughts on the public network is fully understood. I never said anything about PRIVATE individuals and entities being mandated in any way to use the public network. Private citizens, businesses, and other entities and individuals CANNOT be mandated to use the public network. ONLY PUBLIC entities be mandated to use the public network as their primary provider. I should also add that private networks DON’T have to use the public network in the event of IT issues.
But what would be the point of spending billions and billions to build, maintain, and constantly upgrade a public network if only a handful of public enterprises would use it? I would never put my company on such a network, and it wouldn’t be financially sustainable for the government to operate. It makes no sense to me to build it.
Legacy technologies, as you call them, are far, far more prevalent than you think, including copper. I dont really know what you mean by ‘fiber internet’. If you mean hybrid fiber coax networks like the ones Rogers operates, that is not at all a legacy or outdated internet technology in any way. What you are proposing, to lay another fibre and another wireless network all over, would never be practical.
I see.
Regarding the fibre network, I was thinking more along the lines of Telus PureFibre. Their Wi-Fi Hub’s WAN connection is fibre. No WAN coax, telephone/DSL, etc.
If such new dedicated public network has a negative net benefit versus the latter two means I discussed in the original post (i.e. assisting post-secondary grad’s telecom startups in creating their own Tier 1 network, and easing restrictions on private telecom competition with allowing foreign telecoms), then at least government should focus on those latter two fronts in the short term.
But what would be the point of spending billions and billions to build, maintain, and constantly upgrade a public network if only a handful of public enterprises would use it? I would never put my company on such a network, and it wouldn’t be financially sustainable for the government to operate. It makes no sense to me to build it.
Legacy technologies, as you call them, are far, far more prevalent than you think, including copper. I dont really know what you mean by ‘fiber internet’. If you mean hybrid fiber coax networks like the ones Rogers operates, that is not at all a legacy or outdated internet technology in any way. What you are proposing, to lay fibre and another wireless network all over, would never be practical.
But what would be the point of spending billions and billions to build, maintain, and constantly upgrade a public network if only a handful of public enterprises would use it? I would never put my company on such a network, and it wouldn’t be financially sustainable for the government to operate. It makes no sense to me to build it.
Legacy technologies, as you call them, are far, far more prevalent than you think, including copper. I dont really know what you mean by ‘fiber internet’. If you mean hybrid fiber coax networks like the ones Rogers operates, that is not at all a legacy or outdated internet technology in any way. What you are proposing, to lay fibre and another wireless network all over, would never be practical.
Putting aside my observations of Trudeau’s tendencies toward authoritarianism, I still don’t think government ownership is the way to go.
Can you name one industry that is better when the government runs parts of it, even if that’s limited to parts used by government? Add to that the graft, ethical lapses and corruption on the scale of a banana republic that we’ve all grown to expect from the Trudeau’s and boondoggle would probably not be enough to describe the outcome.
I see. Anyway, government should still in the short to medium term should improve private telecom competition through working with various schools to assist graduates with their telecom startups in creating their Tier 1 (NOT 2, 3, etc.) Network (top priority), and loosening restrictions for competition from foreign telecoms (lowest priority as it can invite foreign monopolies similar to current local monopolies) as I discussed in the smaller last paragraph of the original comment. IMO the telecom competition issue should be solved from multiple aspects.
I don’t think my thoughts regarding the public network are understood properly here. I should clarify that PRIVATE entities and individuals (e.g. private banks (e.g. TD, RBC, etc.), credit unions, schools, properties, etc.) WON’T be mandated to use the public network as their primarily provider UNLIKE PUBLIC entities (e.g. Metrolinx, TTC, STM, TransLink, BC Transit, ETS, Calgary Transit, WorkBC, WorkSafeBC, SFU, BCIT, UBC, National Bank of Canada, etc.). Also, private networks WON’T be mandated to use public network as a backup so they can be primarily left alone. I was thinking of increasing the quantity and number of types of network redundancies which includes creation of a dedicated public network so every public entity (NOT private entities) won’t primarily clog the private networks so PRIVATE individuals and entities can have all the bandwidth and freedom on PRIVATE networks. Furthermore, no government of any level can mandate every PRIVATE individual and entity to use the public network so bandwidth and reliability won’t be a major issue.
Call me a cynic, but I don’t think that’s the best solution. Imagine the scenario of having a leader of the government with dictatorial leanings in charge (eg: some trust fund kid from Quebec having grown up with a Marxist leaning father for example) leaves the country vulnerable to a single, highly censored, government run autocratic system at the stroke of a pen.
I call BS on this one…unless the router upgrades were rushed to fix an ongoing problem (service degradation, not outage) how could one “predict” that some knucklehead at Robbers would botch the upgrade and take the routers down (or at least BGP).
Depends on if you believe it was a “maintenance update” failure.
“John Fagg”
Jeez, what a name to grow up with, I can’t imagine the grief he was put through as a kid.
Clickbait title: “Ontario Company Saw Rogers Outage Coming Weeks Ago”
Actual story: “the sheer volume of alerts Packetwork was seeing on a weekly basis that, in hindsight, was alarming.”
Pretty easy to predict “ in hindsight ” lol!
Clickbait title: “Ontario Company Saw Rogers Outage Coming Weeks Ago”
Actual story: “the sheer volume of alerts Packetwork was seeing on a weekly basis that, in hindsight, was alarming.”
Pretty easy to predict “ in hindsight ” lol!
I just happened to be a friends shop the day this Article came out. He had to close his location friday 10am day of Outage because Packetworks was down. Later that night I sent him link to article. His response: “f****** liars”.