Commit Graph

228 Commits (165f129c61c74d18deff78449636c5880e997bcc)

Author SHA1 Message Date
Victor Julien 165f129c61 stream: detect data gap at stream start
In midstream mode we may encounter a case where the data we is beyond
the isn, but below last_ack. This means we're missing some data, that
is already acked so it won't be retransmitted. Therefore, we can
conclude it's a data gap.
11 years ago
Victor Julien f9213d7cc6 stream-tcp: proper error if segment pool init fails
Until now a PoolInit failure for the segment pools would result in
an abort() through BUG_ON(). This patch adds a proper error message,
then exits.

Bug #1108.
12 years ago
Victor Julien 7e38347d99 stream-tcp: fix error handling in segment pool
When TcpSegmentPoolInit fails (e.g. because of a too low memcap),
it would free the segment. However, the segment memory is managed
by the Pool API, which would also free the same memory location.
This patch fixes that.

Also, memset the structure before any checks are done, as the segment
memory is passed to TcpSegmentPoolCleanup in case of error as well.

Bug #1108
12 years ago
Victor Julien 7450f32351 stream: add performance output for stream pools
Add info messages at shutdown that give an indication of pool use
for the various segment and chunk pools.
12 years ago
Victor Julien 84696ebe2a stream: configurable stream chunk prealloc
The stream chunk pool contains preallocating stream chunks (StreamMsg).
These are used for raw reassembly, used in raw content inspection by
the detection engine. The default setting so far has been 250, which
was hardcoded. This meant that in setups that needed more, allocs and
frees would be happen constantly.

This patch introduces a yaml option to set the 'prealloc' value in the
pool. The default is still 250.

stream.reassembly.chunk-prealloc

Related to feature #1093.
12 years ago
Victor Julien b5f8f386a3 stream: configurable segment pools
The stream reassembly engine uses a set of pools in which preallocated
segments are stored. There are various pools each with different packet
sizes. The goal is to lower memory presure. Until now, these pools were
hardcoded.

This patch introduces the ability to configure them fully from the yaml.
There can be at max 256 of these pools.

Yaml layout is as follows:

stream:
  reassemble:
    segments:
      - size: 2048
        prealloc: 3000
      - size: 4
        prealloc: 1000
      - size: 1024
        prealloc: 2000

The size is the packet size. The prealloc value indicates how many
segments are set up at startup.

The pools have no limit wrt how many segments can be used of a certain
size. If the engine needs more than the prealloc size, segments are
malloc'd and free'd. The only limit here is the stream.reassemble.memcap.

If the yaml part if omitted, the default values are the same as before.

Feature #1093
12 years ago
Victor Julien 347c0df9c4 app-layer-event: refactor
Move app layer event handling into app-layer-event.[ch].
Convert 'Set' macro's to functions.
Get rid of duplication in Set and SetRaw. Set now calls SetRaw.
Fix potentential int overflow condition in the event storage.
Update callers.
12 years ago
Victor Julien bf6ab333ff stream: use reassembly.memcap for stream chunks
Use the stream.reassembly.memcap for stream chunks (StreaMsg) as well.
12 years ago
Victor Julien 7a0649f9c4 pool: rename data structure fields to stack
Rename the following fields:
 -    uint32_t alloc_list_size;
 +    uint32_t alloc_stack_size;

 -    PoolBucket *alloc_list;
 +    PoolBucket *alloc_stack;

 -    PoolBucket *empty_list;
 -    uint32_t empty_list_size;
 +    PoolBucket *empty_stack;
 +    uint32_t empty_stack_size;

To reflect that these are in fact, stacks.
12 years ago
Victor Julien 3c7f6ed876 stream: improve memcap checking
Only the TcpSegment structure would be checked for fitting in the
memcap, not the actual data.
12 years ago
Victor Julien feedb45770 stream: cast memcap checks to uint64_t 12 years ago
Victor Julien 5f307acace Pass ThreadVars ptr to various thread init funcs
To be able to register counters from AppLayerGetCtxThread, the
ThreadVars pointer needs to be available in it and thus in it's
callers:

- AppLayerGetCtxThread
- DecodeThreadVarsAlloc
- StreamTcpReassembleInitThreadCtx
12 years ago
Victor Julien 6e389a1fbb stream: don't send empty streammsg at stream end
No longer send an empty StreamMsg through the engine on stream end,
the messages were ignored anyway.
12 years ago
Victor Julien c801ef3515 stream: remove flags from StreamMsg 12 years ago
Victor Julien 261881fce2 stream: remove per thread queue for stream msgs
StreamMsgs would be stored in a per thread queue before being
attached to the tcp ssn. This is unnecessary, so this patch
removes this queue and puts the smsgs into the ssn directly.

Large patch as it affects a lot of tests.
12 years ago
Victor Julien b159c1714c stream: remove flow reference from StreamMsg
StreamMsg' flow reference was used mostly to make sure a flow would
not get removed from the hash before inspection. For this it needed
to reference the flow use_cnt reference counter. Nowadays we have
more advanced flow timeout handling. This will make sure that if
there still are pending smsgs' in a flow, these will still be
processed.
12 years ago
Victor Julien 1d08a3ff26 stream: pass TcpSession to StreamTcpReassembleProcessAppLayer
Preparation for removing flow pointer from StreamMsg. Instead of
getting the ssn indirectly through StreamMsg->flow, we pass it
directly as all callers have it already.
12 years ago
Victor Julien 0ec375d95a stream msg: remove structure 12 years ago
Victor Julien 3804f3f1b3 stream: no longer process STREAM_GAP smsgs
StreamSmgs are used for raw stream reassembly only. They could also
be used to tell the rest of the engine about sequence gaps. This was
a left over from the older implementation, where the app layer used
the smsgs as well.
12 years ago
Victor Julien 106e1c7d19 profiling: fix compilation
Stream engine can't access app layer proto detection datatypes
anymore, so moved some of the logic into app-layer.c
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Victor Julien 1252ee3f04 stream: suppress minor scan-build warnings
stream-tcp-reassemble.c:2569:17: warning: Value stored to 'seg' is never read
                seg = seg->next;
                ^     ~~~~~~~~~
stream-tcp-reassemble.c:2587:17: warning: Value stored to 'seg' is never read
                seg = seg->next;
12 years ago
Ken Steele f9705377ae Remove pkt variable from Packet structure.
The uint8_t *pkt in the Packet structure always points to the memory
immediately following the Packet structure. It is better to simply
calculate that value every time than store the 8 byte pointer.
12 years ago
Victor Julien 8522da8ea5 stream: add option to disable raw reassembly
Raw reassembly is used only by the detection engine. For users only
caring about logging it's a significant overhead, both in cpu and
memory usage.

The option is called 'raw' and lives under the stream.reassembly
options.

stream:
  memcap: 32mb
  checksum-validation: yes      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 64mb
    depth: 1mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    raw: false # <- new option
12 years ago
Victor Julien 3f8947ff3d app layer: set event if proto detect disabled for a stream, but we see data anyway. 12 years ago
Victor Julien abccbe13f3 stream: add size debug code 12 years ago
Victor Julien daedb6c557 stream: wait for protocol detection to complete
Wait for protocol detection to complete before removing segments
from the list.
12 years ago
Victor Julien ab7677638e stream: improve raw reassembly
When checking the reassembly limit for raw reassembly, consider the
STREAMTCP_STREAM_FLAG_NOREASSEMBLY a trigger immediately. We won't
process any more segments in the reassembly engine anyway.
12 years ago
Victor Julien 0c12ad6bd0 stream: fix sequence number on smsg
When multiple segments were put into a smsg, the seq would be updated
each time a segment was added. Because of this, the seq wasn't pointing
to the start of the data.

This caused some false negatives when the fast_pattern was in the raw
stream, but another part of the inspection was in the state. Because of
the wrong seq, the inspection of the smsg could be delayed. This in turn,
could make the inspection engine consider a TX inspected, even if it wasn't
fully yet.
12 years ago
Anoop Saldanha 95ed53c590 Fix for #1003.
Now that we call stream reassembly directly from proto detection, we will
need to check if reassembly has been disabled inside the stream reassembly
callback.

This prevents any calls to bypass and re-enter proto detection, despite
having reassembly disabled.
12 years ago
Anoop Saldanha 836bad85a4 Reset app layer processed flag for segments that have been sent for proto
detection, but we failed to figure out the proto.

Updated a unittest to reflect the above change.
12 years ago
Anoop Saldanha d76a5bedbc Update stream inline to use the improved app proto detection. 12 years ago
Anoop Saldanha 2cb5bdd3fa Cosmetic changes to code. Introduce human readabel flag values for some constants. Here the parameter in question is "data_first_seen_dir" for session context. 12 years ago
Anoop Saldanha 976a86def4 Introduce convenience macro to set Stream app proto completion flag. 12 years ago
Anoop Saldanha 0d7159b525 App layer protocol detection updated and improved. We now use
confirmation from both directions and set events if there's a mismatch
between the 2 directions.

FPs from corrupt flows have disappeared with this.
12 years ago
Anoop Saldanha 22c05da3cd Replace ssn appproto_detection_completed flag with individual stream ones. 12 years ago
Eric Leblond cd3e32ce19 unittests: some functions needs a flow lock.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
12 years ago
Victor Julien 92b7ffad69 Improve memory cleanup in some unittests 12 years ago
Ken Steele 9c7b411a5d More PacketGetFromMalloc() to allocate packets. 12 years ago
Victor Julien 28ea129d9b stream: remove unused 'pause' feature 12 years ago
Victor Julien abecef5d82 stream: send eof to app layer from stream end pkt if necessary 13 years ago
Ludovico Cavedon ac8b087717 Wait until both sides close the TCP connection before initiating cleanup 13 years ago
Eric Leblond 09b79cb5bf stream-tcp: fix double call to debug print function 13 years ago
Last G 8ae11f73b2 Added parentheses to fix Eclipse static code analysis
Fixed bug in action priority (REJECT_DST had lowest prio)
13 years ago
Victor Julien 6f76ac176d stream: add option to match on overlapping data
Set event on overlapping data segments that have different data.

Add stream-events option stream-event:reassembly_overlap_different_data and
add an example rule.

Issue 603.
13 years ago
Anoop Saldanha 3d74fa964a Update all flow referencing to use the new FlowReference and FlowDeReference
macros.
13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Victor Julien 20d2db085e reintroduce pool free func for cases where block alloc is not used. 13 years ago
Victor Julien 8f337a3904 stream: never resend reassembled data to app layer. 13 years ago
Eric Leblond 619014a280 pool: rename Free function to Cleanup
This patch renames Free functions to Cleanup as the free is made
by the pool system.
13 years ago