By Andrew Davis
A message to the industry about the state of cyber security in AV.
I want you to imagine a 46″ flat panel display. This display is in a staff cafeteria on the 30th floor of an office building. Wall mounted. It is not an extravagant display, probably 1080p..not even smart (gasp!). You’ve seen one just like it, looping special events and other corporate propaganda. Now I want you to imagine stealing it. This display has a black-market value of, we’ll say for argument’s sake $100…heck, $200 if you are a smooth-talking con. Never mind the fact that the elevators all have cameras. Disregard that you are on the 30th floor. Don’t worry that you have to pass the security guard who was already skeptical of your “AV” story. You could also choose to set off a fire exit alarm after running down 29 flights of stairs. Lastly, and maybe most importantly, you will undoubtedly walk by 100 other items on your way out the door that are magnitudes smaller, and significantly more valuable.
All of these seemingly insurmountable obstacles, but you have committed yourself to stealing this particular TV. I bet you dollars to donuts that you won’t even be able to lift that TV off the mount because the technician that installed it used the security screws… and I’m not talking about the screws that keep it from easily falling off the mount. I am talking about the locking screws that require a two foot long hex key (the security one with the stupid hole in the middle). That is, unless you also want to steal 6 to 8 sq. ft of drywall and an indeterminate length of mangled tin studs.
Why did they install it this way? Why go through the trouble? Because it is pretty much muscle memory for most installers and the screws come in the box. If you are like me, you probably agree that extra degree of security might seem maybe…”situationally overkill” but it takes a few seconds, so why not?
Why is it that after fifteen years (or more) of AV being “on the network” that we remain awesome at something like that, but so completely inept when it comes to network security when there is so much more at stake? The average cost of a data breach in North America is $1.23 million for enterprises and $120,000 for SMB according to a study done by Kaspersky Labs. Let those numbers sink in and then tell me why our industry is putting more care into network security than the 46″ TV on the 30th floor.
I just read an article this week about a data breach at NASA Jet Propulsion Lab where some at large individual managed to steal 1/2 GB of state secrets with a small computer, about the size of a deck of cards (raspberry pi*) that was installed and forgotten about by NASA staff. Undoubtedly they either didn’t change the root password or changed it to something that could be unlocked with a brute force attack in an hour or two. Maybe NASA person downloaded some code from the internet, not even sure the name of the person who wrote the code, but “it does what it’s supposed to do and that’s good enough”. Despite the fact that this story is removed from the AV industry, I couldn’t help but feel like it had commercial AV written all over it.
While there are varying degrees of threat level when it comes to security vulnerabilities in AV systems, if you really want to know how terrible we are as an industry. Do what I do and actually take the time on one project and (try to) plug all the holes. Turn off the network services that aren’t being used. Password lock every device with a different password and don’t forget the special characters.
Just kidding about the special characters, we have lazy coders so we only accept alphanumeric passwords of 4-8 characters. “Password1!” need not apply.
…Turn on encryption. Turn off the ports that aren’t being used. Turn off cloud updates, but definitely load the updates. Once you are done, you will have probably read about 5 product manuals, 10 web forums (6 of which were just a question with no response), typed “help” or “?” at a command line about 30 times….and forgot to pick up the kids at daycare.
Fast forward a day and we were successful in plugging most of the holes. We feel pretty good that this system will not and can not be hacked without physically gaining access to it, factory wiping it with a paper clip, and writing some malicious code that will also result in the AV system not doing what it’s supposed to. If you were truly successful at raising the security bar on this commercial AV system, there is a fantastic chance that your AV rack will be blinking a sea of error lights so bright that we must be seconds from catastrophic failure.
“Hey! Did you know that you turned this off?” “Dude! I can’t talk to the mother-ship!” “uhh yeah..this feature actually doesn’t work with encryption yet.” Is that a skull and crossbones!?**.
Some manufactures have published secure deployment guides which are a good start. Kudos to your initiative; but if it takes 20 pages to explain how to keep your product from being a public menace, maybe, just maybe you should think about the security screws. A minute task that anyone could perform, turns 5 seconds of work, into a near comical level of prevention with zero impact on the end-user experience. That’s where we need to get to.
Maybe it is going to take some serious innovation… Back to the drawing board so we can look at cyber-security a different way. Here is food for thought, the IT manufacturers aren’t even there yet. Heaven forbid we beat them at their own game.
I need to write a custom script for my thousand dollar AV network switch to run when the install work is done to shut down unused switch ports, protecting from easy intrusion. Given the price of the switch, it seems like it should be an easier process, but why wouldn’t I do it? Yes the switch is in the IT closet, so it’s on the lowish end of the threat spectrum but I am also not going to leave my car keys sitting on the dash. Meanwhile there are AV products in the market that are easily accessible while seated at the board room table with a patch cable that don’t give me the same option (USB ports are safe right?!). Come on people…
If you are already working on rebuttals to the effect of, “nobody wants all that security” or “we don’t want to add extra complexity to the installation” or “we aren’t cybersecurity experts”, or the worst and most likely, “our software devs have assured us that our product is inherently safe so there is no need to waste any resources on this…it’s fine”, please feel free to address the responses to me and send them right to that magical place where discarded drafts go because this is not something you can pass on. The Titanic was unsinkable and Enigma was un-crackable, yet here we are.
Dear product managers and executives. The reality is this:
The alternative to fixing what some might consider ignorance or negligence not to fix, is only going to get your employers and us the sales people, installers, and programmers in a whole boat load of trouble. It is not enough to add the features that IT departments and DND are asking for so you can meet bid spec. We are talking about risking huge money, and jobs, and livelihoods for what amounts to, in some cases, a couple lines of code. Worst case scenario, front page news tomorrow is an enterprise data attack worth millions that is traced back to AV equipment. All the AV equipment you installed will be ripped out, and the prophecy from my last article about the end of AV hardware will happen in the span of a week.
For the rest of us. The IT noobs trying horribly to fake it through a conversation about certificates, firewalls, etc. Make sure your teams at the very least understand the importance of secure deployments. Like extras on an insurance policy, it’s one of those things that’s easy enough to shrug off now, but you can be sure that we are all going to regret inaction when things go awry.
*No knock against SBC’s. They are as safe as
you make them.
**You know who you are.
(Note: Reprinted with permission from Andrew Davis – this article originally appeared here. Header: Royalty free image. No attribution required).