Commit Graph

168 Commits (2c8e8c2516742a100875a4b9392bd889e4506a00)

Author SHA1 Message Date
Anoop Saldanha f2dd61868d variable names global vars, global no more. Moved to detection engine ctx, a place it belongs 13 years ago
Anoop Saldanha 55d4e9518e Kill engine during init stage if it fails to load valid value for sgh-mpm-context 13 years ago
Anoop Saldanha 09ec7ec728 bug 456 fix for byte_extract to have array of the right size to update values with 13 years ago
Victor Julien d378b76c04 http: body inspection improvement
Improve http_client_body and file_data performance when request and
response body limits are set to high values.
13 years ago
Victor Julien da3c5bf84d Minor error message cleanups 14 years ago
Nikolay Denev 139768dd58 Do not use underscored config vars internally. 14 years ago
Anoop Saldanha db859cc56e treate ac-bs auto as single context 14 years ago
Anoop Saldanha 4d38a571cc smtp reply code mpm phase support added 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Anoop Saldanha 35f3eafa5e byte extract added to the engine. Detection support added for packet payload, uri and dce detection engines 14 years ago
Eric Leblond 49adc264bc Don't print message after SCMalloc failure.
This patch generated via coccinelle is getting rid of logging
message after a SCMalloc failure. They were useless as SCMalloc
already displays a message.
15 years ago
Victor Julien b24ccf8c80 Clean up stream pmqs in the detect thread ctx. 15 years ago
Victor Julien 6ebe7b7cd3 Change the way the request body limit is enforced. 15 years ago
Anoop Saldanha 778ec0939c make client body buffer limit configurable. Also some minor changes 15 years ago
Victor Julien 07ec1ee10e Slightly cleanup detect-engine.sgh-mpm-context option parsing. 15 years ago
Anoop Saldanha c89507836b if sgh-mpm-context is not available in conf, alias the auto case inside the engine 15 years ago
Victor Julien 275bd3b7d7 Switch back to defaulting to full for detect-engine.sgh-mpm-context as it broke many tests. 15 years ago
Victor Julien 7e6f01765f Change default of detect-engine.sgh-mpm-context to auto. 15 years ago
Anoop Saldanha 59923316bc change the default recursion limit in the code to 3000, the value which we currently have in the conf file. Also change print modifier for printing timeval 15 years ago
Anoop Saldanha bc99328ec8 define a new conf paramter detect-engine:inspection-recursion-limit; Defines a recursion limit for content inspection code 15 years ago
Victor Julien 3bd7441ea5 Default to 'single' ctx for ac-gfbs as well. 15 years ago
Anoop Saldanha a2d04a94b5 selecting auto for detect-engine.sgh_mpm_context now uses single if the mpm is ac, full otherwise 15 years ago
Anoop Saldanha 0ef684705c support single mpm context distribution across sghs in staging. Also see to it that ac works fine with this setup 15 years ago
Anoop Saldanha b367c37ae6 suricata.yaml conf update to support single mpm context distribution over multiple sghs + code to parse this conf 15 years ago
Victor Julien 87f88867f4 Further improve B2gc. Add B2gm. Improve memory layout. 15 years ago
Anoop Saldanha 33f4beb0bc batching of packets support for cuda b2g mpm. Supported for both 32 and 64 bit platforms 15 years ago
Anoop Saldanha 9ecade76b9 in case of duplicate signatures used the one with the latest revision 15 years ago
Gurvinder Singh 8852b83fa7 flowbits, flowvars, pktvars, flow flags and app layer info added to alert-debug.log 15 years ago
Pablo Rincon eed0ef6e69 Adding tag keyword support 15 years ago
Anoop Saldanha bbb5bf5c51 allow counters clubbing for detect TM 15 years ago
Victor Julien 83b2c8abdb Improve stateful uri detection code. 15 years ago
Victor Julien a0c1209a44 Inspect the reassembled stream together with the packet payload in the same direction. 15 years ago
Victor Julien 2fd31a1a11 Remove dsize grouping from detection engine grouping reducing memory usage. Store sgh in flow to reduce lookups. Reduce locking in alert handling. Increase default grouping values as we use less memory. 15 years ago
Gurvinder Singh cda664a8c4 memroy leaks fixes in detection module, app layer and counters 15 years ago
William Metcalf 2eef905c07 GPL and Copyright header updates. 15 years ago
Victor Julien 70b32f7380 First stab at creating a stateful detection engine.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:

- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.

This commit fixes bug #124.
15 years ago
Gerardo Iglesias Galvan 9f4fae5b1a Fix inconsistent use of dynamic memory allocation 15 years ago
Victor Julien 7a427ec7f4 Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach. 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Anoop Saldanha 47037ef9ec fix for bug 115 15 years ago
Anoop Saldanha c54b91ed94 fix for bug 113 16 years ago
Victor Julien 50e41817a7 Share content id's between identical patterns. 16 years ago
Pablo Rincon 25a3a5c6d8 Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks. 16 years ago
Victor Julien 60685f8b3c Make unittests run more quiet. 16 years ago
Pablo Rincon 38dc7ffebc Adding settings for detect engine group config 16 years ago
Anoop Saldanha 8cf60d6645 Changed the way cuda dispatcher passes back results. Now each detection thread has it's own queue to which the dispatcher can pump packets back to the detect thread. Also, with cuda enabled and a non-cuda mpm being used, we won't create a dispatcher and instead call the b2g scan/search funtions directly instead of using the dispatcher. 16 years ago
Gurvinder Singh fea277b2aa memory leak fixes 16 years ago
Anoop Saldanha 011b74df63 Modify the classification config tests to use the buffer than a temp file and also fix an invalid free 16 years ago
Breno Silva 69eb869cc9 Threshold Rule 16 years ago
Victor Julien ecf86f9c23 Rename to Suricata. 16 years ago
Gerardo Iglesias Galvan 7642e53f06 Fix warnings from previous patch. Add info to usage output. 16 years ago
Gerardo Iglesias Galvan b254719ffc Add fatal failures on unittest and siginit failure (using Conf API) 16 years ago
Anoop Saldanha ceb7e495ae refactoring perf stats code 16 years ago
Anoop Saldanha 7a10ddc07b Fixes for the fast-pattern tests and a couple of other minor changes 16 years ago
Victor Julien 2d0e9658f8 Speed up per sgh content maxlen calc. Remove mpm ptrs from mpm ctx. Add unittests testing the detection engine internals. 16 years ago
Victor Julien e4c98c562c Merge DetectAddressData and DetectAddressGroup 16 years ago
Victor Julien 15ab5d7003 More engine init memleaks fixed. HashListTable remove function fixed. 16 years ago
Victor Julien 7a7bb7a390 Get rid of global mpm_ctx. 16 years ago
Victor Julien fbe87a3ad5 Bunch of mostly unittest related memleak fixes. 16 years ago
Gurvinder Singh 40b8afdd56 support for thread exit constants 16 years ago
Anoop Saldanha f658ffbc9c Order the signatures based on certain rule parameters like actions, flowbits, flowvar, pktvar, priority etc 16 years ago
Victor Julien b9972a9d2c Cleanups 16 years ago
Brian Rectanus fa5939ca91 64 bit cleanup part2 16 years ago
Victor Julien 51a9e36e10 Remove vips references. Rename to eidps. 16 years ago
Victor Julien ff4b5a5db7 Add support for flowbits. 16 years ago
Victor Julien 5df5b35e90 Put all globals in the detection engine ctx. Add HashList type, a hash that also stores the items ina list to it can be traversed. Many cleanups. 16 years ago
Victor Julien 54ffe2053e Large detection engine update. 16 years ago
Victor Julien dc48c58473 Switch to using a detection engine ctx. 16 years ago