Anti-spam developer Cloudmark claims its high-volume content-filtering email gateway could prevent almost all unwanted email from reaching a network's mail servers.
Many carriers and large enterprises run basic filters and throttles at the network edge, because if they can reject spam there, it takes a pile of dirt off their infrastructure and saves bandwidth, processing and storage. They then run the remainder through content filters such as Cloudmark in the core, before delivering it to users.
Neil Cook, Cloudmark Europe's technology chief, claimed the new gateway (or message transfer agent, MTA) could increase the effectiveness of edge filtering from perhaps 60 or 70 per cent to over 90 per cent, taking yet more load off the network.
He added that with junk now representing as much as 95 per cent of all US email traffic, and maybe 85 per cent of European email traffic, it is essential to block it as early as possible - but until now, content filtering has required too much processing power to do it at the network edge.
According to Cloudmark, its MTA can process messages 20 to 40 times faster than other content filters, making it feasible to do at the edge. Each MTA can scan more than three million messages an hour for spam, viruses and phishing attacks, the company said.
Cook said that the secret of its speed is that it runs messages through just eight single-pass 'fingerprinting' algorithms, rather than using techniques such as pattern-matching.
"Most of our competitors work on heuristics, and have manual centres around the world with people writing new rules to block spam as it is reported," he said. "The problem is that spam is randomised now and changes very rapidly.
"Cloudmark's intelligent fingerprinting algorithms extract the essence of the message - its characteristics and structure, not the text - so they are resistant to mutation and are not language-dependent."
He added that the next thing is for carriers to use these gateways on outbound traffic too, throttling or blocking spam from leaving their infrastructures, and alerting abuse teams to unusual traffic patterns that might indicate zombie PCs.
"There's so much going on in your network that's not through your mail system, and if you can't detect that, you get blacklisted. Blacklisting is a huge incentive - ISPs hate being blacklisted, because of the customer service calls and costs," he said, pointing to Tiscali's recent woes.
Cloudmark's technology is based on Vipul's Razor, an open source spam-filtering scheme invented by its founder, Vipul Ved Prakash. Cook said the Cloudmark version is also available for smaller organisations to use as a SpamAssassin plug-in, replacing Vipul's Razor. ®