Thursday, June 5, 2014

NetFlow Version 9 module for Node.JS

I am writing some small automation scripts to help me in my work from time to time. I needed a NetFlow collector and I wanted to write it in javascript for Node.JS because of my general desire to support this platform enabling JavaScript language in generic application programming and system programming.

Node.JS is having probably the best in the market package manager (for a framework) named npm. It is extremely easy to install and maintain a package, to keep dependencies or even "scoping" it on a local installation avoiding the need of having root permissions for your machine. This is great. However, most of the packages registered in the npm database are junk. A lot of code is left without any development or having generic bugs or is simply incomplete. I am strongly suggesting to the nodejs community to introduce a package statuses based on public voting marking each module in "production", "stable", "unstable", "development" quality and to set by default the npm search searching in "production" and "stable". Actually, npm already have a way to do that, but leaves the marking decision to the package owner.

Anyway, I was looking for Netflow v9 module that could allow me to capture netflow traffic with this version. Unfortunately the only module supporting NetFlow was node-Netflowd. It does support Netflow version 5 but has a lot of issues with NetFlow v9, to say at least. After few hours testing it at the end I decided to write one on my own.

So please welcome the newest Node.JS module that support collecting and decoding of NetFlow version 9 flows named "node-netflowv9"

This module supports only Netflow v9 and has to be used only for it.
The library is very very simple, having about 250 lines of code and supports all of the publicly defined Cisco properties, including variable length numbers and IPv6 addressing.

It is very easy to use it. You just have to do something like this:
var Collector = require('node-netflowv9');

Collector(function(flow) {
The flow will be represented in JavaScript object in a format very similar to this:
{ header: 
  { version: 9,
     count: 25,
     uptime: 2452864139,
     seconds: 1401951592,
     sequence: 254138992,
     sourceId: 2081 },
  { address: '',
     family: 'IPv4',
     port: 29471,
     size: 1452 },
  { in_pkts: 3,
     in_bytes: 144,
     ipv4_src_addr: '',
     ipv4_dst_addr: '',
     input_snmp: 27,
     output_snmp: 16,
     last_switched: 2452753808,
     first_switched: 2452744429,
     l4_src_port: 61538,
     l4_dst_port: 62348,
     out_as: 0,
     in_as: 0,
     bgp_ipv4_next_hop: '',
     src_mask: 32,
     dst_mask: 24,
     protocol: 17,
     tcp_flags: 0,
     src_tos: 0,
     direction: 1,
     fw_status: 64,
     flow_sampler_id: 2 } }
There will be a callback per each flow, not only one for each packet. If the packet contain 10 flows, there will be 10 callbacks containing each different flow. This simplifies the Collector code as you don't have to loop on your own trough the flows.

Keep in mind that Netflow v9 does not have a fixed structure (in difference to NetFlow v1/v5) and it is based on templates. It depends on the platform which properties it will set in the temlpates and what will be the order of it. You always have to test you netflow v9 collector configuration. This library is trying to simplify it as much as possible, but it cannot compensate it.

My general feeling is that S-Flow is much better defined and much more powerful than NetFlow in general. NetFlow v9 is the closest Cisco product that can provide (but is not necessary providing) similar functionality. However, the behavior and the functionality of NetFlow v9 differ between the different Cisco products. On some - you can define aggregations and templates on your own. On some (IOS XR) you can't and you use NetFlow v9 as a replacement to NetFlow v5. On some other Cisco products (Nexus 7000) there is no support of NetFlow at all, but there is S-Flow :)

In all of the Cisco products, the interfaces are sent as SNMP interface index. However, this index may not be persistent (between device reboots) and to associate it with an interface name you have to implement cached SNMP GET to the interface table OID on your own.

Because of the impressive performance of the modern JavaScript this little module performs really fast in Node.JS. I have a complex collector implemented with configurable and evaluated aggregations that uses on average less than 2% CPU on a virtual machine, processing about 100 packets with flows and about 1000 flow statistics per second.


No comments:

Post a Comment