Commit Graph

180 Commits (98484ffdccb3b9b0564e5dca0ceb91a5f1d9e1c8)

Author SHA1 Message Date
Victor Julien 8f337a3904 stream: never resend reassembled data to app layer. 13 years ago
Eric Leblond 619014a280 pool: rename Free function to Cleanup
This patch renames Free functions to Cleanup as the free is made
by the pool system.
13 years ago
Eric Leblond fa079c1da0 pool: realize a block allocation for preallocated item.
This patch required a evolution of Pool API as it is needed to
proceed to alloc or init separetely. The PoolInit has been changed
with a new Init function parameter.
13 years ago
Victor Julien 869109a6a0 stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP. 13 years ago
Victor Julien 9d2e17fa98 stream: don't NULL dereference p->flow->protoctx in StreamTcpReassembleDepthReached 13 years ago
Victor Julien 43c7fd7585 file inspection: improve logging when stream.depth limit is reached. #493. 13 years ago
Eric Leblond 0227a87fcb cleaning: fix warning when building with clang.
clang was issuing some warnings related to unused return in function.
This patch adds some needed error treatment and ignore the rest of the
warnings by adding a cast to void.
13 years ago
Anoop Saldanha c5cc9d454d stream raw reassembly fix 13 years ago
Victor Julien ed3599b3d8 stream: improve error checking. 13 years ago
Victor Julien 88a21456e3 stream: keep segments in memory until we are sure the stream/state is inspected. 13 years ago
Anoop Saldanha 64625675ce set stream_eof flag per stream, only when the stream initiates a close. Fix htp parser to close connection per direction based on this 13 years ago
Anoop Saldanha 225b917e93 remove unused stream ssn flag - STREAMTCP_FLAG_TOSERVER_REASSEMBLY_STARTED 13 years ago
Victor Julien fddaca6e8b Implement stream memcap enforcements using atomics instead of spinlocked counters. 14 years ago
Victor Julien 0150e66ede flow engine: improve scalability
Major redesign of the flow engine. Remove the flow queues that turned
out to be major choke points when using many threads. Flow manager now
walks the hash table directly. Simplify the way we get a new flow in
case of emergency.
14 years ago
Victor Julien e237841a8e Fix compilation with profiling enabled. Minor unittest fixes. 14 years ago
Victor Julien c4b34e6ef7 Fix various minor clang/scan-build warnings. 14 years ago
Victor Julien 16cfae2f51 Trigger raw stream reassembly on receiving a full HTTP request or response. 14 years ago
Victor Julien ddfa5c49c6 Stream engine: gap handling
Set a stream event for stream gaps.
Add a (disabled by default) signature to the stream-event.rules.
14 years ago
Victor Julien 425294f912 stream reassembly: account stream gaps
Add counter to the stream reassembly engine to count stream gaps. Stream gaps
are the result of missing packets (usually due to packet loss). This missing
data stops the reassembly for the app layer.
14 years ago
Victor Julien 262a7300d7 flow: shrink Flow datatype
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.

Add macro's to check the family, copy the address, etc.

Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.

On 64 bit this shrinks the flow with 8 bytes.
14 years ago
Victor Julien a0b532dc45 stream reassembly: simplify base_seq tracking for protocol detection. Shrinks TcpStream structure. 14 years ago
Anoop Saldanha 4130c5e2b8 if flow has disabled app layer inspection, disable buffering the segments unnecessarily in inline reassembly 14 years ago
Anoop Saldanha 43cbed8c92 enable toclient alproto detection for inline reassembly 14 years ago
Anoop Saldanha f684b60127 if flow has disabled app layer inspection, disable buffering the segments unnecessarily 14 years ago
Victor Julien c9960473bb Fix stream reassembly engine rejecting valid packet for reassembly. 14 years ago
Anoop Saldanha 4650bf7170 minor code cleanup. remove commented out code 14 years ago
Anoop Saldanha 78e6a7f713 enable toclient alproto detection. Detection all current alproto toclient PMP patterns 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Victor Julien 9b437caaea Fix stream unittests. 14 years ago
Victor Julien b8659daef7 Add stream engine counters
Added stream counters:
- tcp.reassembly_memuse -- current memory use by reassembly in bytes
- tcp.memuse -- current memory use by stream tracking in bytes
- tcp.reused_ssn -- ssn reused by new session with identical tuple
- tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits
14 years ago
Victor Julien 8208eacd79 Convert stream memcaps to u64. Bug #332. 14 years ago
Victor Julien fca541f40e Add per app layer parser profiling
Per packet per app layer parser profiling. Example summary output:

Per App layer parser stats:

App Layer              IP ver   Proto   cnt        min      max          avg
--------------------   ------   -----   ------     ------   ----------   -------
ALPROTO_HTTP            IPv4       6    163394        126     38560320     42814
ALPROTO_FTP             IPv4       6       644        117        26100      2566
ALPROTO_TLS             IPv4       6       670        117         7137       799
ALPROTO_SMB             IPv4       6    114794        126       225270       957
ALPROTO_DCERPC          IPv4       6      5207        126        25596      1266

Also added to the csv out.

In the csv out there is a new column "stream (no app)" that removes the
app layer parsers from the stream tracking. So raw stream engine performance
becomes visible.
14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Victor Julien 862b708a70 Fix stream unittest. 14 years ago
Victor Julien 047b19d271 Fix a reassembly bug that in some cases could lead to a crash. 14 years ago
Gerardo Iglesias Galvan dd5e438d6f Make all access to memory tracking counters in stream engine lock protected 14 years ago
Gerardo Iglesias Galvan bd6d1bfac4 Fix potential crash in classtype parsing code 14 years ago
Victor Julien be5ad4402d Fix stream reassembly engine compilation on Windows. 14 years ago
Anoop Saldanha 000ce98cd1 push all proto detection code into their respective app parser register functions for every alproto 14 years ago
Victor Julien 96c2f2c877 Fix 2 stream reassembly unittests 14 years ago
Victor Julien 3b40b02a1b Stream reassembly fixes. 14 years ago
Victor Julien d0374ced38 Implement SACK in the stream engine. 14 years ago
Victor Julien 892a8a4985 Make stream inline use the chunk size settings. 15 years ago
Victor Julien 4f5aad1476 Enforce configurable minimum chunk size in raw stream reassembly. Minor stream cleanups, unittest updates. 15 years ago
Victor Julien 936b34ddf6 Remove minimum init chunk length code, set a default limit of 2560 to the minimum chunk size, allow toclient raw reassembly to start even if toserver hasn't started yet. 15 years ago
Victor Julien 8faacb727d Account for seg list not always being empty when stream closes. 15 years ago
Victor Julien ecfa2d0176 Only remove segments from segment list if they are completely before ra_base_seq. 15 years ago
Victor Julien 5bdf16380d Make sure we actually remove no longer required segments. 15 years ago
Victor Julien 7f45a4fd58 Fix missing segment flag, fix 2 unittests broken after previous stream changes. 15 years ago
Victor Julien 4a7f6079d5 Change segment removal in stream engine to not discard segments right away. Now they are only removed if they are fully before ra_base_seq. 15 years ago