Now that the FlowWorker handles the TCP Stream directly, having
the TCP engine as a thread module is no longer needed.
This patch removes the registration.
Instead of handling the packet update during flow lookup, handle
it in the stream/detect threads. This lowers the load of the
capture thread(s) in autofp mode.
The decoders now set a flag in the packet if the packet needs a
flow lookup. Then the workers will take care of this. The decoders
also already calculate the raw flow hash value. This is so that
this value can be used in flow balancing in autofp.
Because the flow lookup/creation is now done in the worker threads,
the flow balancing can no longer use the flow. It's not yet
available. Autofp load balancing uses raw hash values instead.
In the same line, move UDP AppLayer out of the DecodeUDP module,
and also into the stream/detect threads.
Handle TCP session reuse inside the flow engine itself. If a looked up
flow matches the packet, but is a TCP stream starter, check if the
ssn needs to be reused. If that is the case handle it within the
lookup function. Simplies the locking and removes potential race
conditions.
The option sets in bytes the value at which segment data is passed to
the app layer API directly. Data sizes equal to and higher than the
value set are passed on directly.
Default is 128.
Stream pseudo packets are taken from the packet pool, which can be empty.
In this case a pseudo packet will not be created and processed.
This patch adds a counter "tcp.pseudo_failed" to track this.
Instead of error phrone externs with macro's, use functions with a local
static enum var instead.
- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode
To set the modes:
- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode
Bug #1177.
Raw reassembly is used only by the detection engine. For users only
caring about logging it's a significant overhead, both in cpu and
memory usage.
The option is called 'raw' and lives under the stream.reassembly
options.
stream:
memcap: 32mb
checksum-validation: yes # reject wrong csums
inline: auto # auto will use inline mode in IPS mode, yes or no set it statically
reassembly:
memcap: 64mb
depth: 1mb # reassemble 1mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
#randomize-chunk-range: 10
raw: false # <- new option
If a session only has data in one direction, like ftp data sessions,
protocol detection will only run in one direction. This led to a
situation where reassembly would hold all the segments as proto
detection was never flagged as complete.
This patch introduces a limit for protocol detection in this case.
If the limit is reached, detection will give up.
Use per thread pools to store and retrieve SSN's from. Uses PoolThread
API.
Remove max-sessions setting. Pools are set to unlimited, but TCP memcap
limits the amount of sessions.
The prealloc_session settings now applies to each thread, so lowered the
default from 32k to 2k.
Until now, when processing the TCP 3 way handshake (3whs), retransmissions
of SYN/ACKs are silently accepted, unless they are different somehow. If
the SEQ or ACK values are different they are considered wrong and events
are set. The stream events rules will match on this.
In some cases, this is wrong. If the client missed the SYN/ACK, the server
may send a different one with a different SEQ. This commit deals with this.
As it is impossible to predict which one the client will accept, each is
added to a list. Then on receiving the final ACK from the 3whs, the list
is checked and the state is updated according to the queued SYN/ACK.
Set event on overlapping data segments that have different data.
Add stream-events option stream-event:reassembly_overlap_different_data and
add an example rule.
Issue 603.
Added stream counters:
- tcp.reassembly_memuse -- current memory use by reassembly in bytes
- tcp.memuse -- current memory use by stream tracking in bytes
- tcp.reused_ssn -- ssn reused by new session with identical tuple
- tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits
This patch introduces a function called StreamMsgForEach which
can be used to run a callback on all segments of a stream. This
is currently only supported for TCP as this is the only streaming
aware protocol.
The stream engine memory handling needed updating as it didn't scale. Changes:
- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
added.
- all reassembly errors are converted to debug msgs.
Victor Julien
Ken Steele
Anoop Saldanha
Anoop Saldanha
Eric Leblond
Gurvinder Singh
Pablo Rincon
William Metcalf