Commit Graph

191 Commits (4749420f325acc5965cd942d78ba9ce28c686eda)

Author SHA1 Message Date
Victor Julien 92b7ffad69 Improve memory cleanup in some unittests 12 years ago
Ken Steele 9c7b411a5d More PacketGetFromMalloc() to allocate packets. 12 years ago
Victor Julien 28ea129d9b stream: remove unused 'pause' feature 12 years ago
Victor Julien abecef5d82 stream: send eof to app layer from stream end pkt if necessary 13 years ago
Ludovico Cavedon ac8b087717 Wait until both sides close the TCP connection before initiating cleanup 13 years ago
Eric Leblond 09b79cb5bf stream-tcp: fix double call to debug print function 13 years ago
Last G 8ae11f73b2 Added parentheses to fix Eclipse static code analysis
Fixed bug in action priority (REJECT_DST had lowest prio)
13 years ago
Victor Julien 6f76ac176d stream: add option to match on overlapping data
Set event on overlapping data segments that have different data.

Add stream-events option stream-event:reassembly_overlap_different_data and
add an example rule.

Issue 603.
13 years ago
Anoop Saldanha 3d74fa964a Update all flow referencing to use the new FlowReference and FlowDeReference
macros.
13 years ago
Eric Leblond e176be6fcc Use unlikely for error treatment.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.

This patch has been obtained via coccinelle. The transformation
is the following:

@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@

x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
13 years ago
Victor Julien 20d2db085e reintroduce pool free func for cases where block alloc is not used. 13 years ago
Victor Julien 8f337a3904 stream: never resend reassembled data to app layer. 13 years ago
Eric Leblond 619014a280 pool: rename Free function to Cleanup
This patch renames Free functions to Cleanup as the free is made
by the pool system.
13 years ago
Eric Leblond fa079c1da0 pool: realize a block allocation for preallocated item.
This patch required a evolution of Pool API as it is needed to
proceed to alloc or init separetely. The PoolInit has been changed
with a new Init function parameter.
13 years ago
Victor Julien 869109a6a0 stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP. 13 years ago
Victor Julien 9d2e17fa98 stream: don't NULL dereference p->flow->protoctx in StreamTcpReassembleDepthReached 13 years ago
Victor Julien 43c7fd7585 file inspection: improve logging when stream.depth limit is reached. #493. 13 years ago
Eric Leblond 0227a87fcb cleaning: fix warning when building with clang.
clang was issuing some warnings related to unused return in function.
This patch adds some needed error treatment and ignore the rest of the
warnings by adding a cast to void.
13 years ago
Anoop Saldanha c5cc9d454d stream raw reassembly fix 13 years ago
Victor Julien ed3599b3d8 stream: improve error checking. 13 years ago
Victor Julien 88a21456e3 stream: keep segments in memory until we are sure the stream/state is inspected. 13 years ago
Anoop Saldanha 64625675ce set stream_eof flag per stream, only when the stream initiates a close. Fix htp parser to close connection per direction based on this 13 years ago
Anoop Saldanha 225b917e93 remove unused stream ssn flag - STREAMTCP_FLAG_TOSERVER_REASSEMBLY_STARTED 13 years ago
Victor Julien fddaca6e8b Implement stream memcap enforcements using atomics instead of spinlocked counters. 14 years ago
Victor Julien 0150e66ede flow engine: improve scalability
Major redesign of the flow engine. Remove the flow queues that turned
out to be major choke points when using many threads. Flow manager now
walks the hash table directly. Simplify the way we get a new flow in
case of emergency.
14 years ago
Victor Julien e237841a8e Fix compilation with profiling enabled. Minor unittest fixes. 14 years ago
Victor Julien c4b34e6ef7 Fix various minor clang/scan-build warnings. 14 years ago
Victor Julien 16cfae2f51 Trigger raw stream reassembly on receiving a full HTTP request or response. 14 years ago
Victor Julien ddfa5c49c6 Stream engine: gap handling
Set a stream event for stream gaps.
Add a (disabled by default) signature to the stream-event.rules.
14 years ago
Victor Julien 425294f912 stream reassembly: account stream gaps
Add counter to the stream reassembly engine to count stream gaps. Stream gaps
are the result of missing packets (usually due to packet loss). This missing
data stops the reassembly for the app layer.
14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien a0b532dc45 stream reassembly: simplify base_seq tracking for protocol detection. Shrinks TcpStream structure. 14 years ago
Anoop Saldanha 4130c5e2b8 if flow has disabled app layer inspection, disable buffering the segments unnecessarily in inline reassembly 14 years ago
Anoop Saldanha 43cbed8c92 enable toclient alproto detection for inline reassembly 14 years ago
Anoop Saldanha f684b60127 if flow has disabled app layer inspection, disable buffering the segments unnecessarily 14 years ago
Victor Julien c9960473bb Fix stream reassembly engine rejecting valid packet for reassembly. 14 years ago
Anoop Saldanha 4650bf7170 minor code cleanup. remove commented out code 14 years ago
Anoop Saldanha 78e6a7f713 enable toclient alproto detection. Detection all current alproto toclient PMP patterns 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Victor Julien 9b437caaea Fix stream unittests. 14 years ago
Victor Julien b8659daef7 Add stream engine counters
Added stream counters:
- tcp.reassembly_memuse -- current memory use by reassembly in bytes
- tcp.memuse -- current memory use by stream tracking in bytes
- tcp.reused_ssn -- ssn reused by new session with identical tuple
- tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits
14 years ago
Victor Julien 8208eacd79 Convert stream memcaps to u64. Bug #332. 14 years ago
Victor Julien fca541f40e Add per app layer parser profiling
Per packet per app layer parser profiling. Example summary output:

Per App layer parser stats:

App Layer              IP ver   Proto   cnt        min      max          avg
--------------------   ------   -----   ------     ------   ----------   -------
ALPROTO_HTTP            IPv4       6    163394        126     38560320     42814
ALPROTO_FTP             IPv4       6       644        117        26100      2566
ALPROTO_TLS             IPv4       6       670        117         7137       799
ALPROTO_SMB             IPv4       6    114794        126       225270       957
ALPROTO_DCERPC          IPv4       6      5207        126        25596      1266

Also added to the csv out.

In the csv out there is a new column "stream (no app)" that removes the
app layer parsers from the stream tracking. So raw stream engine performance
becomes visible.
14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Victor Julien 862b708a70 Fix stream unittest. 14 years ago
Victor Julien 047b19d271 Fix a reassembly bug that in some cases could lead to a crash. 14 years ago
Gerardo Iglesias Galvan dd5e438d6f Make all access to memory tracking counters in stream engine lock protected 14 years ago
Gerardo Iglesias Galvan bd6d1bfac4 Fix potential crash in classtype parsing code 14 years ago
Victor Julien be5ad4402d Fix stream reassembly engine compilation on Windows. 14 years ago
Anoop Saldanha 000ce98cd1 push all proto detection code into their respective app parser register functions for every alproto 14 years ago