Sysadmin blog Security flaws are a great source of inter-company marketing FUD, but it is how a company responds to them that determines how trustworthy they are. Can you bet your business – or your personal data – on a company that simply brushes flaws under a rug? Where does the vendor's responsibility end and that of the customer begin?
As the "internet of things" becomes the new reality there are an increasing number of "unmanaged" computers connected to the internet. These range from home automation, to Google's Nest, to a diverse array of industrial sensors – and even the baseband management controllers that provide lights out management for our servers.
This last is an important canary for the problems the Internet of Things will present. A BMC is a computer in its own right. These small embedded computers allow administrators to remotely access the larger, more powerful servers they serve at a level "below" the operating system. This allows administrators to remotely update the larger server's BIOS, change firmware settings or install operating systems.
BMCs typically adhere to the IPMI standard, often with unique twists, features or functionality depending on the manufacturer. They go by different names, depending on the manufacturer: HP calls their implementation ILIO; Dell has DRAC; Supermicro simply uses IPMI.
BMCs blur the lines between managed and unmanaged computers. The servers that the BMCs are designed to augment are typically actively maintained. Unfortunately, while the server operating systems and applications often receive regular patching, security scans and so forth, the BMCs are all too often neglected.
Vulnerabilities and how you handle them
The most basic response that any company provide is to issue patches for known issues. A security researcher detects and issue, raises it with the company in question and – in a perfect world – that company creates a patch and releases it for customers to install.
This doesn't always happen. There are innumerable vulnerable home routers still in service that will never see patches. These serve as examples of just how lax many companies are about these sorts of issues.
Supermicro has recently been in the news regarding security vulnerabilities in their BMC implementations. They are by no means the only company to have BMCs with security vulnerabilities, but their response to the issue is worthy of deeper consideration.
Fortunately, unlike many of the companies cranking out home routers, Supermicro do issue patches. What makes them worthy of interest is that instead of holding to this basic reactionary stance, Supermicro chooses to go that little bit beyond.
According to Zachary Wikholm, senior security engineer for Cari.net, the Security Incident Response Team (CARISIRT) has been cooperative. When asked about specific BMC security issues, they don't simply provide some pre-canned marketing statements, but help researchers dig into other issues, even when they know that information about those security problems will be published.
I had a chance to talk to Arun Kalluri, senior product manager for Supermicro's Software Solutions division, and asked some hard questions in the hopes of getting a better idea of Supermicro's approach to security. Considering that Supermicro is often portrayed as "nothing more than a whitebox vendor", I wanted to dig into what Supermicro could – and is – doing that goes beyond simply reacting. How companies choose to respond to these issues is always of great interest to me.
Supermicro competitors HP, Dell, IBM and so forth all have massive R&D departments. Their resources vastly outstrip anything Supermicro can bring to bear. Acknowledging this, it appears that Supermicro's approach is to forgo the typical vendor secrecy and reach out to both the security community and the academic research community.
How Supermicro responds
Supermicro's approach to security is three pronged. It is determined to work hand in glove with security researchers to fix vulnerabilities they identify. It has an internal security team working to find vulnerabilities and patch them and is investing in trying to find completely novel ways to solve security problems.
If you're reading this sort of thing online, you best have a firewall installed
As part of the latter, Kalluri introduced me to Dr Alex Halderman from University of Michigan. Supermicro has been working very closely with Halderman in the hopes of finding new solutions to the problems presented by BMC vulnerabilities. The hope is that if they can come up with a better way to deal with BMC vulnerabilities these techniques can be applied to other unattended computers and eventually help firm up security for the internet of things as a whole.
I threw a lot of questions and ideas at Halderman. He proved both patient with my lack of knowledge and passionate about the topic. There were limits to what information I could work out of him, but ultimately he does seem to know his stuff.
Some of my ideas – like that of embedded application layer gateways, centralised command and control and the need some form of automatic update mechanism – seemed to trigger in Halderman a desire to really leap into conversations more in depth than could be discussed before they are ready to publish. It's unfortunate, but I was asked to pick the conversation back up in September, so I suspect we won't have to wait long before seeing the early results of Supermicro tie-up with the University of Michigan.
There are many problems faced by companies trying to secure small embedded computers. Like most embedded devices, BMCs have limited room for applications, so there are limits to how many security toys you can pack in. They are limited in compute power, so on and so forth.
That Supermicro is putting resources into tackling these problems head on is encouraging. We are in the middle of the transition from a world where security was the sole province of the customer to one where vendors of all sizes and market niches are accept that security must be a fundamental part of both the design of systems they sell and their ongoing research and development efforts. While we see that transition through, it's still up to us as systems administrators to hold down the fort.
Vendor, meet sysadmin
Right now, today, it's foolish for any systems administrator to put an embedded device – be it a BMC, a sensor or an internet of things branded and marketed superwidget – naked to the internet. It doesn't matter who makes the system in question: put a firewall between that device and the all-seeing eye of the Shodan search engine.
More critically, patch your systems. It does us all no good for vendors to busy themselves issuing patches if we aren't going to take advantage of them.
If Supermicro's efforts are anything to go by – and I do hope that other vendors are engaged in similar avenues of research – the next generation of embedded devices will be a lot more secure and a lot more able to take care of themselves. That said, it will be at least a decade – maybe more – before most of the existing embedded systems, sensors, BMCs and early internet of things widgetry is retired.
We need to do our part. We need to secure our systems and to educate others. We also need to talk with our vendors and ask the hard questions. What are their policies regarding security? How do they engage with the security community? What plans and/or programs do they have in place to push the frontiers of security and ensure that tomorrow's systems are more secure than today's?
A server isn't just a server anymore, and it hasn't been for some time. Your hardware vendor is responsible for selling you systems with embedded operating systems, firmware and other bits of software. Long term support matters.
If Supermicro is willing to commit to seven year life cycles, then I am sure as all hell going to expect at least that from those companies charging more. It's easy to wave those expectations around for enterprise hardware. We need to be doing so for consumer gear as well.
Price and performance aren't good enough metrics anymore. Security – both current and planned – matters at all levels. ®