Make stream engine use the streaming buffer API for it's data storage.
This means that the data is stored in a single reassembled sliding
buffer. The subleties of the reassembly, e.g. overlap handling, are
taken care of at segment insertion.
The TcpSegments now have a StreamingBufferSegment that contains an
offset and a length. Using this the segment data can be retrieved
per segment.
Redo segment insertion. The insertion code is moved to it's own file
and is simplified a lot.
A major difference with the previous implementation is that the segment
list now contains overlapping segments if the traffic is that way.
Previously there could be more and smaller segments in the memory list
than what was seen on the wire.
Due to the matching of in memory segments and on the wire segments,
the overlap with different data detection (potential mots attacks)
is much more accurate.
Raw and App reassembly progress is no longer tracked per segment using
flags, but there is now a progress tracker in the TcpStream for each.
When pruning we make sure we don't slide beyond in-use segments. When
both app-layer and raw inspection are beyond the start of the segment
list, the segments might not be freed even though the data in the
streaming buffer is already gone. This is caused by the 'in-use' status
that the segments can implicitly have. This patch accounts for that
when calculating the 'left_edge' of the streaming window.
Raw reassembly still sets up 'StreamMsg' objects for content
inspection. They are set up based on either the full StreamingBuffer,
or based on the StreamingBufferBlocks if there are gaps in the data.
Reworked 'stream needs work' logic. When a flow times out the flow
engine checks whether a TCP flow still needs work. The
StreamNeedsReassembly function is used to test if a stream still has
unreassembled segments or uninspected stream chunks.
This patch updates the function to consider the app and/or raw
progress. It also cleans the function up and adds more meaningful
debug messages. Finally it makes it non-inline.
Unittests have been overhauled, and partly moved into their own files.
Remove lots of dead code.
Add list of 'blocks'. This list contains offsets and lengths to
continuous data blocks. This is useful for TCP tracking where we
can have data gaps.
The blocks don't contain any data themselves, instead they contain
lenght and offsets. This way no extra copying is needed.
On inserting new data, existing blocks are expanded instead of
having multiple neighbouring blocks.
Implement common code to easily add more per HTTP header detection
keywords.
Implement http_accept sticky buffer. It operates on the HTTP Accept
header.
We assume session resumption has occurred if the Client Hello message
included a session id, we have not seen the server certificate, but
we have seen a Change Cipher Spec message from the server.
Previously, these transactions were not logged at all because the
server cert was never seen.
Ticket: https://redmine.openinfosecfoundation.org/issues/1969
eve: detects libevent for async redis at configure
eve: moves redis output code to new file - util-log-redis.{c,h}
eve: redis ECHO and QUIT commands for async mode
eve: redis output defaults if conf is missing
If running against a pcap there is no reason to drop events,
a blocking socket is fine here. So only do non-blocking writes
when running off a live device.
Writing to a unix socket can cause Suricata to block in the
packet path. This could happen if the read-endpoint of the
unix socket stays connected, but stops reading, or simply
can't read fast enough as part of its event processing.
To choose packets over events, do non-blocking socket
writes and drop the event if the write would block and
update a dropped counter.
Recursively create new log directories when needed. This makes it
possible to use date modifiers in the file path to create
directories based on date, e.g.:
/var/log/suricata/2017/02/14/
Rotate log file based on time. Support both rotating based on a timer (XXs,
XXm, XXd, XXw) and rotating based on a absolute time, like each minute,
hour or day.
If a subsequent fragment has a lower offset than a previous
one and overlaps, trim off the beginning of the previous
fragment.
Based on an issue reported privately.
Migrate to the new Travis container build system. This build system does
not allow use of sudo, so required packages must be done declaratively
which required reworking how we install packages that are conditional
based on the build being done.
Mac builds are still done with sudo=true.
UnixManagerThreadInit needs to return a failure code if the socket
fails to initialize to avoid entering the UnixManager loop which
will continuously fail on the call to bind, as no listening
socket was setup.
This can occur when the socket fails to initialize due to a
permissions error and fatal init errors is not on.
Victor Julien
Ray Ruvinskiy
fooinha
psanders240
Jason Ish
Mats Klepsland