Things yet to do: softflowd --------- - Use strtonum() Flow tracking engine - Calculate hash over flow and use it as a key to avoid lots of cache-trashing comparisons - Verify checksums (maybe. perhaps bad for accounting, good for flow tracking) - Fragment processing - We don't handle fragments right - This shouldn't be too hard or too memory intensive. We just need to keep a tree of fragment entries. Each entry would need to contain enough information to reconstruct the flow (source/dest addr, etc), but also fragment related info: IP id, list of fragment offsets. etc. - When we receive a new fragment, we add an entry to this tree (keyed by source IP, protocol, IP id) - Each new fragment matched in the tree gets its offset added to the list, until all fragments have been seen - Must be careful, as later fragments may arrive before inital one - When does accounting occur? - Upon receipt of inital fragment? (and thus for ever frag thereafter) - When we have seen all fragments? (what if we don't?) - Must limit size of tree - Must have fragment timeout (what happens then, apart from removal?) - Timeouts - Timeout for unanswered TCP connection - Ditto orphaned connections (one packet in one direction only) - Track ICMP generated by TCP/UDP session (painful, probably unecessary) - More datalink types - Improve fast-expiry of TCP session by tracking FIN sequence numbers - Multiple interface support - Requires some way to avoid duplicate recording of flows (IP ID) - Track IPsec SPIs - Track ToS / DSCP - Make counters 64 bits - We can report these directly for NetFlow v.9 - For older NetFlow, report by sending multiple flows until counter < 2^32 Misc features - Ability to open more than one interface (maybe) - Ability to read more than one pcap file (maybe) - Fork for ctlsock actions? (don't block mainloop) - Remote control over network (requires SSL) Performance - Profile and see where the hot spots are - Fast "new flow" test using a bloom filter - See if we can reduce per-packet overhead more - Cost of expiry remove and re-add per packet - Stop run-time malloc (maybe) - Preallocate a pool of expiry events and flow entries - keep a queue, pick/push first from head Exporter features - sflow support (www.sflow.org) - Needs XDR encoding - Ability to export to multiple hosts - Partly done, just need to keep a list of targets instead of a single one - Ability to directly write to file (maybe. If so, reuse flowd store code) - NetFlow v.9 field selection - Get AS numbers from bgpd and fill in to Netflow packets Statistics code - Collect more statistics (maybe) - Advanced packet analysis: store hash of packet payload, keep statistics on traffic similarity - Bloom filter? - Option to record histograms of - Flow lifetime and size, packet size - Flow bandwidth - Per well-known-port - How to do this quicky? Memory efficiently? - Per IP address/range - How to do this quicky? Memory efficiently? - Moving averages - Track traffic over lifetime of flow - Maintain linked list traffic counts, keyed by time interval - E.g. key by (now / 300) - Or (now - start_time) / 300 (better) - When new packet comes in: - If timestamp of HEAD of list == (now / xxx), then counter += octets - Otherwise create new traffic counter at HEAD and update it - Then trim tail if the list length is too big - Maybe store "hunks" of data, rather than individual counts in the list, as storing a single int is a huge waste of space - Maybe a rrdtool-like heirarchy of timespans - 300 seconds (5 minutes) (2400 bytes) - 360 1-minute blocks (6 hours) (2880 bytes) - 288 10-minute blocks (2 days) (2304 bytes) - 336 1-hour blocks (2 weeks) (2688 bytes) - Total 10kb worst-case per-flow (scary, probably overkill) softflowctl ----------- - Extend interface - Query for specific flows (e.g. by address) - Do this in softflowd or softflowctl? - Expire/delete specific flows (maybe) - Runtime respecify timeouts - Real-time binary dump of flowtable (shm/mmap fd pass?) - ntop like view - Spiffy GUI (separate tool)