Don't assume that the next block after the sbb head is after the
requested offset.
If the next block was before the offset, the returned data_len
would underflow and return a nonsense value to the app-layer.
Bug #2993.
When a TCP session is picked up from the response the flow is
reversed by the protocol detection code.
This would lead to duplicate logging of the response. The reason this
happened was that the per stream app progress tracker was not handled
correctly by the direction reversing code. While the streams were
swapped the stream engine would continue to use a now outdated pointer
to what had become the wrong direction.
This patches fixes this by making the stream a ptr to ptr that can be
updated by the protocol detection as well.
In addition, the progress tracking was cleaned up and the GAP error
handling in this case was improved as well.
In low memory or memcap reached conditions a crash could happen in
inline stream detection.
The crash had the following path:
A packet would come in and it's data was added to the stream. Due
to earlier packet loss, the stream buffer uses a stream buffer block
tree to track the data blocks. When trying to add the current packets
block to the tree, the memory limit was reached and the add fails.
A bit later in the pipeline for the same packet, the inline stream
mpm inspection function gets the data to inspect. For inline mode
this is the current packet + stream data before and after the packet,
if available.
The code looking up the packets data in the stream would not
consider the possibility that the stream block returned wasn't
the right one. The tree search returns either the correct or the
next block. In adjusting the returned block to add the extra stream
data it would miscalculate offsets leading to a corrupt pointer to the
data.
This patch more carefully checks the result of the lookup, and
falls back to simply inspecting the packet payload if the lookup
didn't produce the expected result.
Bug 2842.
Reported-by: Ad Schellevis <ad@opnsense.org>
This deep in the stream engine packets are valid, so don't check
for the tcph header in a packet as it confuses scan-build.
Do add a DEBUG_VALIDATE_BUG_ON so in QA we double check.
Some rules need to inspect both raw stream data and higher level
buffers together. When this higher level buffer is a streaming
buffer itself, the risk of mismatch exists.
This patch allows an app-layer parser to set a 'min inspect depth'.
The value is used by the stream engine to keep at least this
depth worth of data, so that the detection engine can request
all of it for inspection.
For rules that have the SIG_FLAG_FLUSH flag set, data is inspected
not from offset raw_progress, but from raw_progress minus
min_inspect_depth.
At this time this is only used for sigs that have their fast_pattern
in a HTTP body and have raw stream match as well.
Use this in places where we need to use the outer right
edge of our sequence space.
This way we can avoid walking the tree to find this, which
is a potentially expensive operation.
Switch StreamBufferBlocks implementation to use RBTREE instead of
a list. This makes inserts/removals and lookups a lot cheaper if
the number of data gaps is large.
Use separate compare functions for inserts and regular lookups.
Inserts care about the offset, while lookups care about the blocks
right edge as well.
To improve worst case performance turn the segments list into a rbtree.
This greatly improves inserts, lookups and removals if the number of
segments gets very large.
The tree is sorted by the segment sequence number as its primary key.
If 2 segments have the same seq, the payload_len (segment length) is
used. Then the larger segment will be places after the smaller segment.
Exact matches are not added to the tree.
Trigger app layer reassembly in both directions as soon as we've set
the TCP state to closed.
In IDS mode, if a toserver packet would close the state, the app layer
would not get updated until the next toclient packet. However, in
detection, the raw stream inspection would already use all available
stream data in detection and move the 'raw stream progress' tracker
forward. When in later (a) packet(s) the app layer was updated and
inspection ran on the app layer, the stream progress was already
moved too far forward. This would lead to signatures that matched
on both stream and app layer to not match.
By triggering the app layer reassembly as soon as the TCP state is
set to closed, the inspection as both the stream and app layer data
available at the same time so these rules can match.
Bug: #2570
Bug: #2554
The stream depth computation was partly done with the stream_config
depth instead of using the value in the TCP session. As a result,
some configuration were resulting in abnormal behavior.
In particular, when stream depth was 0 and the file store depth was
not 0, Suricata was stopping the streaming on the flow as soon as
the filestore was started.
Reported-by: Pascal Delalande <pdl35@free.fr>
This adds new functions that will be called
through unix-socket and permit to update
and show memcap value.
The memcap value needs to be handled in a
thread safe way, so for this reason it is
declared as atomic var.
According to all checking memcap functions,
the size passed as argument is declared as uint64_t
except for StreamTcpReassembleCheckMemcap where it's
defined as uint32_t.
With large number of threads the default memcaps lead to pool setup
failures. Make sure these are reported properly so that the user
knows what is going on.
Bug: #2226
A parser can now set a flag that will tell the application
layer that it is capable of handling gaps. If enabled, and a
gap occurs, the app-layer needs to be prepared to accept
input that is NULL with a length, where the length is the
number of bytes lost. It is up to the app-layer to
determine if it can sync up with the input data again.
For logging streaming TCP data so far the individual segments where
used. However since the last big stream changes, the segments are
no longer the proper place for this. Segments can now have overlaps
etc.
This patch introduces a new tracker. Next to the existing 'app' and
'raw' trackers, the new tracker is 'log'. When the TCP logging is
used, a flag in the config is set and the log tracker is used to
determine how much of the stream window can be moved.
Move all GAP checks into CheckGap. Remove seg_list based check.
Also remove seg_list == NULL check to make sure the Gap check is
done on an empty list as well.
Improve next_seq < last_ack check, but add data beyond gap check.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
Instead of killing all reassembly instantly do things slightly more
gracefully:
1. disable app-layer reassembly immediately
2. flag raw reassembly not to accept new data
This will allow the current data to be inspected still.
After detect as run the raw reassembly will be fully disabled and
thus all reassembly will be as well.
If raw reassembly falls behind, for example because no raw mpm is
active, then we need to sync up to the app progress if that is
available, or to the generic tcp tracking otherwise.
Victor Julien
Jeff Lucovsky
Eric Leblond
Giuseppe Longo
Jason Ish