aiden
/
suricata
mirror of https://github.com/OISF/suricata
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,557 Commits
7 Branches
161 Tags
169 MiB
main-7.0.x
master
master-6.0.x
master-5.0.x
master-4.1.x
master-4.0.x
master-3.2.x
suricata-8.0.0
suricata-7.0.11
suricata-8.0.0-rc1
suricata-8.0.0-beta1
suricata-7.0.10
suricata-7.0.9
suricata-7.0.8
suricata-7.0.7
suricata-7.0.6
suricata-6.0.20
suricata-7.0.5
suricata-6.0.19
suricata-6.0.18
suricata-7.0.4
suricata-6.0.17
suricata-7.0.3
suricata-6.0.16
suricata-7.0.2
suricata-6.0.15
suricata-7.0.1
suricata-6.0.14
suricata-7.0.0
suricata-7.0.0-rc2
suricata-6.0.13
suricata-6.0.12
suricata-6.0.11
suricata-7.0.0-rc1
suricata-6.0.10
suricata-6.0.9
suricata-7.0.0-beta1
suricata-6.0.8
suricata-6.0.7
suricata-6.0.6
suricata-5.0.10
suricata-6.0.5
suricata-5.0.9
suricata-6.0.4
suricata-5.0.8
suricata-6.0.3
suricata-5.0.7
suricata-6.0.2
suricata-5.0.6
suricata-6.0.1
suricata-5.0.5
suricata-4.1.10
suricata-4.1.9
suricata-5.0.4
suricata-6.0.0
suricata-6.0.0-rc1
suricata-6.0.0-beta1
suricata-5.0.3
suricata-4.1.8
suricata-4.1.7
suricata-5.0.2
suricata-4.1.6
suricata-5.0.1
suricata-5.0.0
suricata-5.0.0-rc1
suricata-4.1.5
suricata-5.0.0-beta1
suricata-4.1.4
suricata-4.1.3
suricata-4.0.7
suricata-4.1.2
suricata-4.1.1
suricata-4.1.0
suricata-4.0.6
suricata-4.1.0-rc2
suricata-4.1.0-rc1
suricata-4.0.5
suricata-4.1.0-beta1
suricata-4.0.4
suricata-4.0.3
suricata-4.0.2
suricata-3.2.5
suricata-4.0.1
suricata-3.2.4
suricata-4.0.0
suricata-4.0.0-rc2
suricata-3.2.3
suricata-4.0.0-rc1
suricata-4.0.0-beta1
suricata-3.2.2
suricata-3.1.4
suricata-3.2.1
suricata-3.2
suricata-3.2RC1
suricata-3.1.3
suricata-3.2beta1
suricata-3.1.2
suricata-3.1.1
suricata-3.1
suricata-3.0.2
suricata-3.1RC1
suricata-3.0.1
suricata-3.0.1RC1
suricata-3.0
suricata-2.0.11
suricata-3.0RC3
suricata-3.0RC2
suricata-3.0RC1
suricata-2.0.10
suricata-2.0.9
suricata-2.1beta4
suricata-2.0.8
suricata-2.0.7
suricata-2.1beta3
suricata-2.0.6
suricata-2.0.5
suricata-2.1beta2
suricata-2.0.4
suricata-2.1beta1
suricata-2.0.3
suricata-2.0.2
suricata-2.0.1
suricata-2.0.1rc1
suricata-2.0
suricata-2.0rc3
suricata-2.0rc2
suricata-2.0rc1
suricata-2.0beta2
suricata-1.4.7
suricata-1.4.6
suricata-1.4.5
suricata-2.0beta1
suricata-1.4.4
suricata-1.4.3
suricata-1.4.2
suricata-1.4.1
suricata-1.3.6
suricata-1.4
suricata-1.3.5
suricata-1.4rc1
suricata-1.3.4
suricata-1.4beta3
suricata-1.3.3
suricata-1.4beta2
suricata-1.3.2
suricata-1.4beta1
suricata-1.3.1
suricata-1.3
suricata-1.3rc1
suricata-1.3beta2
suricata-1.3beta1
suricata-1.2.1
suricata-1.2
suricata-1.2rc1
suricata-1.2beta1
suricata-1.1.1
suricata-0.8.2
suricata-1.0.0
suricata-1.0.1
suricata-1.0.2
suricata-1.0.3
suricata-1.0.4
suricata-1.0.5
suricata-1.1
suricata-1.1beta1
suricata-1.1beta2
suricata-1.1beta3
suricata-1.1rc1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '2913a4a860'
${ noResults }
Commit Graph

207 Commits (2913a4a86084441a2e9504bcd32a373f15aa65a9)

Author SHA1 Message Date
Anoop Saldanha f684b60127 if flow has disabled app layer inspection, disable buffering the segments unnecessarily 14 years ago
Victor Julien c9960473bb Fix stream reassembly engine rejecting valid packet for reassembly. 14 years ago
Anoop Saldanha 4650bf7170 minor code cleanup. remove commented out code 14 years ago
Anoop Saldanha 78e6a7f713 enable toclient alproto detection. Detection all current alproto toclient PMP patterns 14 years ago
Victor Julien 2d16abcf8b Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode. 14 years ago
Victor Julien 9b437caaea Fix stream unittests. 14 years ago
Victor Julien b8659daef7 Add stream engine counters 
Added stream counters:
- tcp.reassembly_memuse -- current memory use by reassembly in bytes
- tcp.memuse -- current memory use by stream tracking in bytes
- tcp.reused_ssn -- ssn reused by new session with identical tuple
- tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits
 14 years ago
Victor Julien 8208eacd79 Convert stream memcaps to u64. Bug #332. 14 years ago
Victor Julien fca541f40e Add per app layer parser profiling 
Per packet per app layer parser profiling. Example summary output:

Per App layer parser stats:

App Layer              IP ver   Proto   cnt        min      max          avg
--------------------   ------   -----   ------     ------   ----------   -------
ALPROTO_HTTP            IPv4       6    163394        126     38560320     42814
ALPROTO_FTP             IPv4       6       644        117        26100      2566
ALPROTO_TLS             IPv4       6       670        117         7137       799
ALPROTO_SMB             IPv4       6    114794        126       225270       957
ALPROTO_DCERPC          IPv4       6      5207        126        25596      1266

Also added to the csv out.

In the csv out there is a new column "stream (no app)" that removes the
app layer parsers from the stream tracking. So raw stream engine performance
becomes visible.
 14 years ago
Victor Julien 820b0ded82 Add per packet profiling. 
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
 14 years ago
Victor Julien 862b708a70 Fix stream unittest. 14 years ago
Victor Julien 047b19d271 Fix a reassembly bug that in some cases could lead to a crash. 14 years ago
Gerardo Iglesias Galvan dd5e438d6f Make all access to memory tracking counters in stream engine lock protected 14 years ago
Gerardo Iglesias Galvan bd6d1bfac4 Fix potential crash in classtype parsing code 14 years ago
Victor Julien be5ad4402d Fix stream reassembly engine compilation on Windows. 14 years ago
Anoop Saldanha 000ce98cd1 push all proto detection code into their respective app parser register functions for every alproto 14 years ago
Victor Julien 96c2f2c877 Fix 2 stream reassembly unittests 15 years ago
Victor Julien 3b40b02a1b Stream reassembly fixes. 15 years ago
Victor Julien d0374ced38 Implement SACK in the stream engine. 15 years ago
Victor Julien 892a8a4985 Make stream inline use the chunk size settings. 15 years ago
Victor Julien 4f5aad1476 Enforce configurable minimum chunk size in raw stream reassembly. Minor stream cleanups, unittest updates. 15 years ago
Victor Julien 936b34ddf6 Remove minimum init chunk length code, set a default limit of 2560 to the minimum chunk size, allow toclient raw reassembly to start even if toserver hasn't started yet. 15 years ago
Victor Julien 8faacb727d Account for seg list not always being empty when stream closes. 15 years ago
Victor Julien ecfa2d0176 Only remove segments from segment list if they are completely before ra_base_seq. 15 years ago
Victor Julien 5bdf16380d Make sure we actually remove no longer required segments. 15 years ago
Victor Julien 7f45a4fd58 Fix missing segment flag, fix 2 unittests broken after previous stream changes. 15 years ago
Victor Julien 4a7f6079d5 Change segment removal in stream engine to not discard segments right away. Now they are only removed if they are fully before ra_base_seq. 15 years ago
Victor Julien b233105cc2 Fix a issue in stream reassembly causing the segment list getting into a inconsistent state. 15 years ago
Victor Julien a8db8b334b Remove debug stream testing code from non-debug builds. 15 years ago
Victor Julien 48c7f18453 Fix bug in the segment insert code causing an inconsistent segment list in some overlap conditions. 15 years ago
Victor Julien 05539d7357 Fix a reassembly overlap issue. Fix a inline reassembly gap handling issue. 15 years ago
Victor Julien e92ab40d39 Fix compilation for non-DEBUG case. 15 years ago
Victor Julien 2db06cc79e Improve Inline reassembly wrt to GAP handling. Add more tests. 15 years ago
Victor Julien 4c82c0e750 Improve RawInline reassembly: remove unnecessary segments from the stream in an earlier stage. Test this properly. 15 years ago
Victor Julien 3a774165fa Initial version of a inline raw reassembly function that reassembles in a sliding window. Introduce new unittest helpers for stream reassembly. 15 years ago
Victor Julien abdffadc1c Add a new app layer reassembly function that is for inline use, and use it when the stream engine is in inline mode. 15 years ago
Victor Julien bff70eed6d Update to depth code. Get segment from the correct pool when a payload is truncated. 15 years ago
Victor Julien 66c40f782c Have reassembly errors also set a stream event. 15 years ago
Victor Julien 0f072648e6 Another iteration of the reassembly depth enforcement, now considering retransmissions. 15 years ago
Victor Julien 16cd31a408 Remove unused pseudo packet reassembly code. 15 years ago
Victor Julien bf88a6de09 Add depth comment. 15 years ago
Victor Julien a26768ce7a Change the way the reassembly depth is enforced. Ignore retransmissions, get rid of per session counter. 15 years ago
Victor Julien aa04d9eefb Improve stream gap handling. Instead of giving up as soon as we see a gap we now wait much longer before we decide it's a gap. 15 years ago
Victor Julien 2849d2b1d3 Initial code for stream 'inline' mode: packets that are (partly) overlapping with already accepted packets (meaning in the streams seg list) are rewritten to make sure they contain the exact same data. 15 years ago
Victor Julien 8b5f553a35 Inspect a pseudo packet upon receiving a RST so that we are sure both sides of the TCP session are inspected. 15 years ago
Gurvinder Singh 55a863359c support for pseudo packet creation from reassembled stream segments 15 years ago
Victor Julien 44e678b86b Comment out disabled unittests. 15 years ago
Victor Julien 1d971b53a6 Update all unittests 15 years ago
Victor Julien b0901ab30d Fix compilation with --enable-debug 15 years ago
Victor Julien 61a9936d55 Inspect a pseudo packet upon receiving a RST so that we are sure both sides of the TCP session are inspected. 15 years ago
Gurvinder Singh 00f21f34e8 support for pseudo packet creation from reassembled stream segments 15 years ago
Victor Julien 8fa5a2c025 Split applayer and raw stream reassembly 
Split stream reassembly in 2 parts: a part that sends ack'd data to the app
layer parsers as soon as it's available, and another part that queues up
data into larger chunks for raw inspection.
 15 years ago
Victor Julien dda6d3e07b Add error counters. 15 years ago
Victor Julien 4ae7144876 Fix 2 cases where overlapping data in the stream engine wouldn't be properly handled potentially causing the wrong data being used in stream reassembly. 15 years ago
Eric Leblond 1db4aadd16 Supress usage of Packet declaration in tests. 
For convenience, a massive usage of 'Packet p;' declaration has
been done in the tests function. Although this was completely
legal, this is not possible anymore because of the new Packet
allocation structure. This massive patch modifies all suricata
files to use a SCMalloc allocated pointer to Packet instead.

This patch has been done using coccinelle (http://coccinelle.lip6.fr)
which is a semantic patching tool. This ensures that things like call
to SCFree() should have not been forget because the semantic patch
explicitly forces the call to SCFree(p) before each return. With this
patch all unittests are running fine with a small and a big default
packet size.
 15 years ago
Gurvinder Singh 892dea31e4 added the counter for tcp.segment_memcap_drop to show the dropped segments count due to memory limit 15 years ago
Gurvinder Singh 6a5bc52461 support for several tcp evasion attacks. Thanks to Judy Novak and G2 Inc for reporting them 15 years ago
Victor Julien 1d73e1fb7e Small update to the ssh module: fix a valgrind warning and a couple of compiler warnings. Do a few small style updates. 15 years ago
Pablo Rincon 9d7baa7a9f Adding ssh app layer module with two new keywords: ssh.protoversion and ssh.softwareversion 15 years ago
Gurvinder Singh 0dab0e3935 fix the reassembly depth test (bug 216) 15 years ago
Victor Julien a9e78871fe Really fix bug 205 this time, repair a broken unittest. 15 years ago
Victor Julien 196e572daa Make sure holding up to_client reassembly stops after the proto is detected or we're sure we'll never detect it. Fixes issues related to bug 205. 15 years ago
Pablo Rincon 4c94a27b71 Fix bug 205 (at stream-tcp-reassemble) 15 years ago
Gurvinder Singh f0928a4555 support for enforcing the depth until when the reassembly will be performed 15 years ago
Victor Julien 66dee577d7 Force stream reassembly on streams where we didn't yet detect the protocol if the stream is closing. 15 years ago
Gurvinder Singh 8b0ca4f628 support for seperate memcaps for reassembly and stream engine 15 years ago
Victor Julien 92858a211d Fix STREAM_EOF flag overwriting STREAM_START flag on short streams. This made us miss short HTTP sessions. 15 years ago
Victor Julien a3ff0e7210 Don't scan TCP packet payload if it was added to the stream. Inspect the tcp stream with the correct packet. Should fix #184 and #185. 15 years ago
Pablo Rincon 8cc525c939 UDP support at AppLayer message handling 15 years ago
William Metcalf cc76aa4bc6 properly init flows inside of unit-tests caused lock-up when falling back to using mutex locks 15 years ago
Victor Julien 9a08d6c11c Fixes to stream pattern matching. 15 years ago
Victor Julien 81f2499834 Store stream msgs processed by the app layer in the tcp session so they can be inspected by the detection module as well. The detection module returns them to the pool. 15 years ago
Victor Julien c26434fef1 Move flow use cnt to atomic and outside of the flow mutex protection. 15 years ago
William Metcalf 2eef905c07 GPL and Copyright header updates. 15 years ago
Gurvinder Singh 5fe1dc1d24 support for sslv2/sslv3 their unit tests and better stream no reassembly flag handling 15 years ago
Victor Julien 70b32f7380 First stab at creating a stateful detection engine. 
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:

- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.

This commit fixes bug #124.
 15 years ago
Gerardo Iglesias Galvan 9f4fae5b1a Fix inconsistent use of dynamic memory allocation 15 years ago
Victor Julien ecf5a8fbcc Fix broken stream unittest. 15 years ago
Victor Julien 7a427ec7f4 Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach. 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Victor Julien b6a45fd1a4 Hack around cornercase in debug code in stream engine. Works around #140. 15 years ago
Victor Julien e0c2c86342 Make HTTP proto detection more reliable. Add HEAD keyword. Thanks rmkml for the report/request. 16 years ago
Victor Julien 59c5e819b1 Fix compilation if debugging is disabled. 16 years ago
Victor Julien ad02732907 Properly cleanup stream engine spinlocks and mutexes at shutdown. Fixes drd errors in unittests. 16 years ago
Victor Julien 156b844ccb Fix tcp segment list corruption bug 16 years ago
Pablo Rincon b708d7f65d Adding Uricontent inspection with spm. Modifiers for uricontent are now supported 16 years ago
Gurvinder Singh 8e444f1772 stream and application layer improvements 16 years ago
Victor Julien 3d7b882bde Make sure all smsgs are handled every time, even in case or error. The fuzzer found an issue where unhandled messages remained in the queue leading to threading issues. 16 years ago
Gurvinder Singh 7438f981da stream memory leaks fixed and unit tests added 16 years ago
Pablo Rincon 25a3a5c6d8 Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks. 16 years ago
Victor Julien f96511a8b1 Check reassembly limits against correct stream direction. Set proper direction flag in stream msgs. 16 years ago
Gurvinder Singh ed99e73622 bug 78 16 years ago
Victor Julien 53977fded6 Small compilation fixes when debugging is disabled. 16 years ago
Victor Julien 6a53ab9c5a Stream engine memory handling update 
The stream engine memory handling needed updating as it didn't scale. Changes:

- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
  of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
  added.
- all reassembly errors are converted to debug msgs.
 16 years ago
Gurvinder Singh d9677c7e2a bug 76 patch 16 years ago
Victor Julien fcb03099a3 Fix reassembly updating the wrong stream on ACK 
The stream reassembly updated the wrong stream on received ACK packets. Instead
of the opposing stream it updated the stream in packet direction. This caused
issues in the app layer handling.

Updated the unittests as well.
 16 years ago
Victor Julien c352bff6fb Remove unused conditional locking code from the app layer parsing code. 16 years ago
Victor Julien 9b4f3f918b Fix broken debug code in stream reassembly 16 years ago
Victor Julien cae8e06cb9 Properly lock app layer result pool and add some debugging code for memory tracking. 16 years ago
Gurvinder Singh 5c8d90afc8 memory leak fixes 16 years ago