After three years of modest or no gains, the number of publicly reported vulnerabilities jumped in 2005, boosted by easy-to-find bugs in web applications. Yet, questions remain about the value of analyzing current databases, whose data rarely correlates easily.
A survey of four major vulnerability databases found that the number of flaws counted by each in the past five years differed significantly. However, three of the four databases exhibited a relative plateau in the number of flaws publicly disclosed in 2002 through 2004. And, every database saw a significant increase in their count of the flaws disclosed in 2005.
A few common themes emerged from the data as well. In 2005, easy-to-find flaws in web applications were likely responsible for the majority of the increase, the database managers said in interviews with SecurityFocus. However, some of the increase came from a doubling in the number of flaws released by large software companies.
The most important, and perhaps obvious, lesson is that the software flaws are here to stay, said Peter Mell, a senior computer scientist for the National Institute of Standards and Technology (NIST) and the creator of the National Vulnerability Database (NVD), one of the four databases surveyed.
"The problem of people breaking into computers is not going away any time soon," Mell said. "There is certainly more patches every year that system administrators need to install, but the caveat is that more vulnerabilities seem to apply to less important software."
Vulnerability databases are coming of age. In 2005, NIST created the National Vulnerability Database and software makers and security service providers have cooperated to create the Common Vulnerability Scoring System (CVSS), a standardized measure of the severity of software flaws. The National Vulnerability Database completed scoring flaws in its database using the CVSS in late November. While auctions of vulnerability research have not taken off, two companies now buy vulnerability information from flaw finders.
Four databases were surveyed: The Computer Emergency Response Team (CERT) Coordination Center's database, the National Vulnerability Database (NVD), the Open-Source Vulnerability Database (OSVDB), and the Symantec Vulnerability Database. (SecurityFocus is owned by Symantec.)
The number of flaws cataloged by each database in 2005 varied widely, because of differing definitions of what constitutes a vulnerability and differing editorial policy. The OSVDB - which counted the highest number of flaws in 2005 at 7,187 - breaks down vulnerabilities into their component parts, so what another database might classify as one flaw might be assigned multiple entries. SecurityFocus had the lowest count of the vulnerabilities at 3,766.
The variations in editorial policy and lack of cross-referencing between databases as well as unmeasurable biases in the research community and disclosure policy mean that the databases - or refined vulnerability information (RVI) sources - do not produce statistics that can be meaningfully compared, Steve Christey, the editor of the Common Vulnerability and Exposures (CVE), wrote in an e-mail to security mailing lists on Thursday. The CVE is a dictionary of security issues compiled by The MITRE Corp., a government contractor and nonprofit organization.
"In my opinion, RVI sources are still a year or two away from being able to produce reliable, repeatable, and comparable statistics," he wrote. "In general, consumers should treat current statistics as suggestive, not conclusive."
Recent numbers produced by the U.S. Computer Emergency Readiness Team (US-CERT) revealed some of the problems with refined vulnerability sources. Managed by the CERT Coordination Center, the US-CERT's security bulletins outline security issues but are updated each week. In a year end list published last week, the US-CERT announced that 5,198 vulnerabilities had been reported in 2005. Some mainstream media outlets noted the number, compared it to the CERT Coordination Center's previous data - which is compiled from a different set of vulnerability reports - and concluded there was a 38 per cent increase in vulnerabilities in 2005 over the previous year.
In fact, discounting the updated reports resulted in a 41 per cent decrease to 3,074 vulnerabilities, according to an analysis done by Alan Wylie, an independent computer programmer. If the data point could be compared with statistics from CERT/CC, that would have placed the number of flaws reported in line with the previous three years.
Yet, while the data is significantly flawed, the original story told by US-CERT's list seems to be the right one. The number of vulnerabilities reported in 2005 increased, mainly due to researchers looking into the security of Web applications. The National Vulnerability Database noted the largest increase of 96 percent from 2004 to 2005, while the Symantec Vulnerability Database saw the smallest increase of 40 percent.
While publicly reported flaws jumped, that does not necessarily mean dire prospects for home users' or businesses' security, said David Ahmad, manager for development at Symantec's Security Response team.
"Web-based vulnerabilities are all over the place and they are really easy to find--they are the low-hanging fruit," Ahmad said." We have had high-profile vulnerabilities, but that is not what is driving this increase."
Finding those flaws does not require much skills, said Brian Martin, content manager for the OSVDB.
"We are seeing people discover vulnerabilities in software with tiny distribution and low installed base--free guestbooks that are written left and right, available by the thousands," he said. "And we are seeing that it takes no skill to find vulnerabilities in these applications."
The number of vulnerabilities entered into four major databases vary widely over the past five years, but seem to indicate that 2005 was a banner year for bugs.
Sources: Computer Emergency Response Team Coordination Center (CERT/CC), National Vulnerability Database, Open-Source Vulnerability Database, and the Symantec Vulnerability Database.
Yet, the entire focus should not be on the rash of Web application flaws, Mell said.
The computer scientist conducted an informal survey of entries for flaws in products from well-known companies and found that six of 14 software makers had seen a doubling in the number of vulnerability reports, while another four firms saw a decrease in the number of reports. The remaining four companies reported a similar number of flaws as the year before.
"I find it amazing that large and reputable software companies are seeing a large number more flaws this year (2005) than last year," Mell said.
The database managers also cautioned that the vulnerability counts for any particular year generally do not reflect the state of secure software development, only where the research community's interests lie.
"These numbers are showing the state of practice from a few years ago, rather than what the current state of practice is today," said Jeff Havrilla, team leader of vulnerability analysis at the CERT Coordination Center.
Making the issue more difficult, several software vendors move to release patches on a specific day has resulted in most security bulletins detailing multiple vulnerabilities, a situation that makes the true number of flaws harder to count, Havrilla said.
This article was originally published at SecurityFocus.