In midstream mode we may encounter a case where the data we is beyond
the isn, but below last_ack. This means we're missing some data, that
is already acked so it won't be retransmitted. Therefore, we can
conclude it's a data gap.
Until now a PoolInit failure for the segment pools would result in
an abort() through BUG_ON(). This patch adds a proper error message,
then exits.
Bug #1108.
When TcpSegmentPoolInit fails (e.g. because of a too low memcap),
it would free the segment. However, the segment memory is managed
by the Pool API, which would also free the same memory location.
This patch fixes that.
Also, memset the structure before any checks are done, as the segment
memory is passed to TcpSegmentPoolCleanup in case of error as well.
Bug #1108
The stream chunk pool contains preallocating stream chunks (StreamMsg).
These are used for raw reassembly, used in raw content inspection by
the detection engine. The default setting so far has been 250, which
was hardcoded. This meant that in setups that needed more, allocs and
frees would be happen constantly.
This patch introduces a yaml option to set the 'prealloc' value in the
pool. The default is still 250.
stream.reassembly.chunk-prealloc
Related to feature #1093.
The stream reassembly engine uses a set of pools in which preallocated
segments are stored. There are various pools each with different packet
sizes. The goal is to lower memory presure. Until now, these pools were
hardcoded.
This patch introduces the ability to configure them fully from the yaml.
There can be at max 256 of these pools.
Yaml layout is as follows:
stream:
reassemble:
segments:
- size: 2048
prealloc: 3000
- size: 4
prealloc: 1000
- size: 1024
prealloc: 2000
The size is the packet size. The prealloc value indicates how many
segments are set up at startup.
The pools have no limit wrt how many segments can be used of a certain
size. If the engine needs more than the prealloc size, segments are
malloc'd and free'd. The only limit here is the stream.reassemble.memcap.
If the yaml part if omitted, the default values are the same as before.
Feature #1093
Move app layer event handling into app-layer-event.[ch].
Convert 'Set' macro's to functions.
Get rid of duplication in Set and SetRaw. Set now calls SetRaw.
Fix potentential int overflow condition in the event storage.
Update callers.
To be able to register counters from AppLayerGetCtxThread, the
ThreadVars pointer needs to be available in it and thus in it's
callers:
- AppLayerGetCtxThread
- DecodeThreadVarsAlloc
- StreamTcpReassembleInitThreadCtx
StreamMsgs would be stored in a per thread queue before being
attached to the tcp ssn. This is unnecessary, so this patch
removes this queue and puts the smsgs into the ssn directly.
Large patch as it affects a lot of tests.
StreamMsg' flow reference was used mostly to make sure a flow would
not get removed from the hash before inspection. For this it needed
to reference the flow use_cnt reference counter. Nowadays we have
more advanced flow timeout handling. This will make sure that if
there still are pending smsgs' in a flow, these will still be
processed.
Preparation for removing flow pointer from StreamMsg. Instead of
getting the ssn indirectly through StreamMsg->flow, we pass it
directly as all callers have it already.
StreamSmgs are used for raw stream reassembly only. They could also
be used to tell the rest of the engine about sequence gaps. This was
a left over from the older implementation, where the app layer used
the smsgs as well.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
stream-tcp-reassemble.c:2569:17: warning: Value stored to 'seg' is never read
seg = seg->next;
^ ~~~~~~~~~
stream-tcp-reassemble.c:2587:17: warning: Value stored to 'seg' is never read
seg = seg->next;
The uint8_t *pkt in the Packet structure always points to the memory
immediately following the Packet structure. It is better to simply
calculate that value every time than store the 8 byte pointer.
Raw reassembly is used only by the detection engine. For users only
caring about logging it's a significant overhead, both in cpu and
memory usage.
The option is called 'raw' and lives under the stream.reassembly
options.
stream:
memcap: 32mb
checksum-validation: yes # reject wrong csums
inline: auto # auto will use inline mode in IPS mode, yes or no set it statically
reassembly:
memcap: 64mb
depth: 1mb # reassemble 1mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: yes
#randomize-chunk-range: 10
raw: false # <- new option
When checking the reassembly limit for raw reassembly, consider the
STREAMTCP_STREAM_FLAG_NOREASSEMBLY a trigger immediately. We won't
process any more segments in the reassembly engine anyway.
When multiple segments were put into a smsg, the seq would be updated
each time a segment was added. Because of this, the seq wasn't pointing
to the start of the data.
This caused some false negatives when the fast_pattern was in the raw
stream, but another part of the inspection was in the state. Because of
the wrong seq, the inspection of the smsg could be delayed. This in turn,
could make the inspection engine consider a TX inspected, even if it wasn't
fully yet.
Now that we call stream reassembly directly from proto detection, we will
need to check if reassembly has been disabled inside the stream reassembly
callback.
This prevents any calls to bypass and re-enter proto detection, despite
having reassembly disabled.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
Set event on overlapping data segments that have different data.
Add stream-events option stream-event:reassembly_overlap_different_data and
add an example rule.
Issue 603.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
Victor Julien
Anoop Saldanha
Ken Steele
Eric Leblond
Ludovico Cavedon
Last G