By Corey Moss
Jonathan Rosenberg, former VP & CTO at Cisco Collaboration – presented at Cisco Live 2018 talking about innovation behind the new Cisco Webex platform (its 30 minutes). And it’s essentially a base statement for what was to come for Webex… Collaboration Reimagined. Rosenberg talks about Cisco building out an IP backbone network over the last ten years to support lots of users all over the world for one thing – great video quality meetings, supporting 33 million meetings, 6 billion minutes of meetings each month.
Rosenberg did interject at that point though: “by the way just to be clear, this is a marketing slide, this is not like datacenter locations, so don’t get yourself excited about that… it’s a beautiful marketing picture. But we have a backbone network like this.”
(Go ahead and watch the beginning).
Yes — a beautiful marketing picture detailing (or so those in the audience thought) what Webex users could expect to support them and the 33 million meetings taking place around the world. In the AV business, we have a name for something that’s there but doesn’t really “exist” – you’ll normally see it at a trade show.
The header image is in fact that slide shown by Rosenberg – and I’ll apply the term here the only way I see fit…
This is The Register (Biting the hand that feeds IT) headline the morning of September 25th: Ooof! Cisco Webex has been down for 7 hours – and counting
And to begin here, a message from the CEO at Cisco from September 25th:
“The @webex outage today is unacceptable, and we apologize for the disruption caused to you, our customers. Webex Meetings is now functional. Our engineers are working to restore Webex Teams and ensure this doesn’t happen again. Thank you for your patience & trust.”
This was the text of Chuck Robbins’ tweet, as Cisco Collaboration people and end users alike scrambled to figure out just what was happening with Cisco’s front-line collaboration application Webex Teams (formerly Spark), touted on the website: “Experience a more intuitive way to work.”
What was taking place, as it began on the (rolling) Cisco support page titled: Webex Teams: Services are currently unavailable on the morning of September 25th
…was anything but intuitive.
Investigating – Webex Teams services are currently impacted by an ongoing service outage. Engineering resources are online and working to restore services. We apologize for the impact and all hands are on deck to restore Teams, Meetings, Calling, Care and Context services.
Updates will be provided as they become available.
Sep 25, 01:22 UTC
I immediately subscribed to the updates which began to come through my e-mail during the day (and I watched my phone prompt each one). What would it be – a few hours?
Along with following the support site throughout the day as this was happening, I also went on social media to alert those in the business (and beyond) of the updates, and the continuous impact on Webex services. For me, press extends beyond the news and stories on the website to social media, where up to the minute reporting can take place – this to go along with posting the news items, stories along with Convergent AV Radio podcasts. All of this being posted on my personal Twitter, LinkedIn and Facebook accounts – which I prefer to report from in these situations.
This followed at 15:32 UTC (which I began reporting on Facebook and LinkedIn):
Update – Service restoration activities continue to be ongoing. All engineering teams are laser focused to complete their remediation tasks and restore access to services. Meeting services are being restored, and Video Platform 2.0 users may join Webex Meetings from some video devices while we continue to restore the Webex devices. We are working as quickly as possible and apologize for the delays. As updates are available, or any estimates for service restoration are available, we will provide them.
Sep 25, 15:32 UTC
Laser focused. Now did that mean the sum total of Webex engineering stopped all other activities to devote full attention to this outage which was (literally) taking the Cisco world by storm? And if this team could be so laser focused, why did this utterly incredible outage occur in the first place – and as many who were impacted would witness, continue throughout the whole day and into the next?
On September 26th:
Update – Engineering has restored service to many of the Cisco Webex Teams Messaging services. At this time, the services are available for users to log in and use the services. As we are continuing restoration activities, some intermittent latency or failures may be experienced when using the platform. Users that experience a failure should retry their operations.
Thank you for your continued patience. As additional services are restored, updates will be published.
Sep 26, 07:50 UTC
Followed later by:
Update – Service restoration activities are ongoing. Currently, Webex Teams connectivity and room creation is experiencing latency or errors due to additional service onboarding. We appreciate your continued patience as we continue to work on the service recovery efforts.
Sep 26, 15:33 UTC
Messaging services restored with some latency issues, yet almost eight hours later connectivity and room creation latency issues exist.
A day and a half from original outage impact, and still ongoing.
I’ll end there as there are still certain ongoing activities – you can track them (and the rest) here.
Prior to the outage, I had posted an article I had recently written The Spark/Webex Re-brand and Acquisitions – A Short History of Cisco Cloud and Collaboration Confusion, and it was shared on numerous social media platforms. I do happen to be active on all the social media platforms that I use, and am always willing to discuss anything that I’ve written, or shared. Sometimes people will even ask me how to purchase a product I may have posted news about or shared information on in social. I always respond, whether e-mailed or on social, to contact the manufacturer.
However this time, it was much different.
I had begun a private LinkedIn chat with a Cisco employee about the article, actually someone I know. A private chat is always good for me, however this one started winding in directions I wasn’t at all expecting. In time I found a comment from the person then went public on the platform, which had actually occurred earlier. Now being called out is one thing, being called out as misinformed — different ballgame.
I have long been one, going back to my commercial integration sales days, to as they say dot all the i’s and cross the t’s – and I’ll admit that an i or t might have gone missed at times. and I would appreciate if someone else caught it.
But definitely not this way.
And the one thing I was challenged about – was that it was never called a Webex Teams Board. That’s it.
Though I did give the Spark Board, which is now the Webex Board (and possibly was the Webex Teams Board which I was 99% sure of) its due in the article in a way noting the Enterprise Connect Best of Show Award in 2017.
To go with this, the article had eyes at Cisco on it at this point. It has become the most viewed story or article on Convergent AV to date – this to go with validated proof of details from higher sources.
A dilemma and a challenge.
At least the challenge was resolved quickly.
More to come, and this will be about trending innovation at Cisco (I hope).
With over 20 years in audio visual integration and IT/computer sales and consulting, Corey Moss is the owner of Convergent AV. Corey writes for the publication and hosts/produces podcasts – The AV Life, The Collaboration Factor and Convergent Tech Talk. He has written for numerous industry publications about AV, IT, unified communications and collaboration (UCC), cloud and software, IoT, cybersecurity and more. He has also conducted interviews with AV and IT executives and global influencers. Find him talking about a whole lot of things, tech and otherwise. On LinkedIn https://www.linkedin.com/in/mosscorey/.