Dynamic application-layer protocol analysis for network intrusion detection

Many network intrusion detection systems (NIDS) rely on protocol-specific analyzers to extract the higher-level semantic context from a traffic stream. To select the cor-rect kind of analysis, traditional systems exclusively de-pend on well-known port numbers. However, based on our experience, increasingly significant portions of to-day’s traffic are not classifiable by such a scheme. Yet for a NIDS, this traffic is very interesting, as a primary rea-son for not using a standard port is to evade security and policy enforcement monitoring. In this paper, we dis-cuss the design and implementation of a NIDS extension to perform dynamic application-layer protocol analysis. For each connection, the system first identifies potential protocols in use and then activates appropriate analyz-ers to verify the decision and extract higher-level seman-tics. We demonstrate the power of our enhancement with three examples: reliable detection of applications not us-ing their standard ports, payload inspection of FTP data transfers, and detection of IRC-based botnet clients and servers. Prototypes of our system currently run at the border of three large-scale operational networks. Due to its success, the bot-detection is already integrated into a dynamic inline blocking of production traffic at one of the sites. 1