Commit Graph

32 Commits (419cdc8558d13f0bc59c54c89917f067c4acb68b)

Author SHA1 Message Date
Anoop Saldanha 419cdc8558 support splitting mpm ctxs based on direction v2 14 years ago
Victor Julien 89f83e714c Introduce http_server_body keyword.
The http_server_body content modifier modifies the previous content to inspect
the normalized (dechunked, unzipped) http_server_body. The workings are similar
to http_client_body. Additionally, a new pcre flag was introduced "/S".

To facilitate this change the signature flags field was changed to be 64 bit.
14 years ago
Anoop Saldanha 17f3f36d38 packet keywords only added for packet mpm. Rest in stream mpm. Update detection engine to handle the same 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Anoop Saldanha 966119b6aa support for http_raw_uri keyword + mpm engine 14 years ago
Anoop Saldanha c9897a44a4 fast pattern support for http_cookie. Also support relative modifiers 15 years ago
Anoop Saldanha bbbedaf963 fast pattern support for http_method. Also support relative modifiers 15 years ago
Anoop Saldanha d241e51b32 adapt fast pattern engine analysis to reflect the new changes made to your mpm design 15 years ago
Anoop Saldanha 72b0fcf419 modify detection engine to carry out uri mpm run before build match array if alproto is http and if sgh has atleast one sig with uri mpm set 15 years ago
Anoop Saldanha 7ec0382774 support fast pattern for http raw header. Also support relative modifiers for http raw header 15 years ago
Anoop Saldanha c61c68fd36 mpm and fast pattern support for http_header. Also support relative modifiers for http_header 15 years ago
Anoop Saldanha 0aa5cffb12 fast pattern support for http_client_body keyword added. Also mpm support for http_client_body added 15 years ago
Anoop Saldanha 4883efd0f6 unifying content structure - uricontent now uses DetectContentData 15 years ago
Anoop Saldanha 0c5b82d891 provide separate ids for content, uricontent, http_(client_body_data|cookie|header|method|uri), when they share the same pattern 15 years ago
Victor Julien a3ff0e7210 Don't scan TCP packet payload if it was added to the stream. Inspect the tcp stream with the correct packet. Should fix #184 and #185. 15 years ago
Victor Julien 9dd753b5f3 Scan uricontent mpm on demand. 15 years ago
Victor Julien a24f288074 Moving the stream content scanning to have it's own mpm ctx. 15 years ago
Victor Julien a0c1209a44 Inspect the reassembled stream together with the packet payload in the same direction. 15 years ago
Victor Julien 7a427ec7f4 Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach. 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Victor Julien ec47f840f3 Remove more scan references. 16 years ago
Victor Julien dd846c9b0e Remove all search code from the pattern matchers, cleanup mpm api, remove unused http code, more cleanups. 16 years ago
Gurvinder Singh 356a8bf385 applayer uri match and modified http handling 16 years ago
Victor Julien 2d0e9658f8 Speed up per sgh content maxlen calc. Remove mpm ptrs from mpm ctx. Add unittests testing the detection engine internals. 16 years ago
Victor Julien 7a7bb7a390 Get rid of global mpm_ctx. 16 years ago
Gurvinder Singh 40b8afdd56 support for thread exit constants 16 years ago
Victor Julien b9972a9d2c Cleanups 16 years ago
Brian Rectanus fa5939ca91 64 bit cleanup part2 16 years ago
Victor Julien 657be002d1 Big detection engine update: scan improvements, b2g/b3g updates, bloom fixes, iponly detection implementation, dsize/flow grouping. 16 years ago
Victor Julien 5df5b35e90 Put all globals in the detection engine ctx. Add HashList type, a hash that also stores the items ina list to it can be traversed. Many cleanups. 16 years ago
Victor Julien dce2c12915 Add Scan before Search to the detection engine. 16 years ago
Victor Julien 8b3d06fd92 Rename some detection engine related files. 16 years ago