softflowd/TODO

Things yet to do:

softflowd
---------

 - Use strtonum()

 Flow tracking engine
  - Calculate hash over flow and use it as a key to avoid lots of
    cache-trashing comparisons
  - Verify checksums (maybe. perhaps bad for accounting, good for flow tracking)
  - Fragment processing
    - We don't handle fragments right
      - This shouldn't be too hard or too memory intensive. We just need to 
	keep a tree of fragment entries. Each entry would need to contain 
	enough information to reconstruct the flow (source/dest addr, etc), 
	but also fragment related info: IP id, list of fragment offsets. etc.
      - When we receive a new fragment, we add an entry to this tree (keyed 
	by source IP, protocol, IP id)
      - Each new fragment matched in the tree gets its offset added to the 
	list, until all fragments have been seen
      - Must be careful, as later fragments may arrive before inital one
      - When does accounting occur? 
	- Upon receipt of inital fragment? (and thus for ever frag thereafter)
	- When we have seen all fragments? (what if we don't?)
      - Must limit size of tree
      - Must have fragment timeout (what happens then, apart from removal?)
   - Timeouts
    - Timeout for unanswered TCP connection
    - Ditto orphaned connections (one packet in one direction only)
    - Track ICMP generated by TCP/UDP session (painful, probably unecessary)
    - More datalink types
    - Improve fast-expiry of TCP session by tracking FIN sequence numbers
  - Multiple interface support
    - Requires some way to avoid duplicate recording of flows (IP ID)
  - Track IPsec SPIs
  - Track ToS / DSCP
  - Make counters 64 bits
    - We can report these directly for NetFlow v.9
    - For older NetFlow, report by sending multiple flows until counter < 2^32

 Misc features
  - Ability to open more than one interface (maybe)
  - Ability to read more than one pcap file (maybe)
  - Fork for ctlsock actions? (don't block mainloop)
  - Remote control over network (requires SSL)

 Performance
  - Profile and see where the hot spots are
  - Fast "new flow" test using a bloom filter
  - See if we can reduce per-packet overhead more
    - Cost of expiry remove and re-add per packet
  - Stop run-time malloc (maybe)
    - Preallocate a pool of expiry events and flow entries
      - keep a queue, pick/push first from head

 Exporter features
  - sflow support (www.sflow.org)
    - Needs XDR encoding
  - Ability to export to multiple hosts
    - Partly done, just need to keep a list of targets instead of a single one
  - Ability to directly write to file (maybe. If so, reuse flowd store code)
  - NetFlow v.9 field selection
  - Get AS numbers from bgpd and fill in to Netflow packets

 Statistics code
  - Collect more statistics (maybe)
    - Advanced packet analysis: store hash of packet payload, keep 
      statistics on traffic similarity
       - Bloom filter?
    - Option to record histograms of
      - Flow lifetime and size, packet size
    - Flow bandwidth
    - Per well-known-port
      - How to do this quicky? Memory efficiently?
    - Per IP address/range
      - How to do this quicky? Memory efficiently?
    - Moving averages
  - Track traffic over lifetime of flow
    - Maintain linked list traffic counts, keyed by time interval
    - E.g. key by (now / 300)
      - Or (now - start_time) / 300 (better)
    - When new packet comes in:
      - If timestamp of HEAD of list == (now / xxx), then counter += octets
      - Otherwise create new traffic counter at HEAD and update it
        - Then trim tail if the list length is too big
    - Maybe store "hunks" of data, rather than individual counts in the 
      list, as storing a single int is a huge waste of space
    - Maybe a rrdtool-like heirarchy of timespans
      - 300 seconds (5 minutes)       (2400 bytes)
      - 360 1-minute blocks (6 hours) (2880 bytes)
      - 288 10-minute blocks (2 days) (2304 bytes)
      - 336 1-hour blocks (2 weeks)   (2688 bytes)
        - Total 10kb worst-case per-flow (scary, probably overkill)

softflowctl
-----------

  - Extend interface
    - Query for specific flows (e.g. by address)
      - Do this in softflowd or softflowctl?
    - Expire/delete specific flows (maybe)
  - Runtime respecify timeouts
  - Real-time binary dump of flowtable (shm/mmap fd pass?)
    - ntop like view
    - Spiffy GUI (separate tool)