Commit Graph

181 Commits (fca541f40ecf08b83e1ac4ed5ae198150be29618)

Author SHA1 Message Date
Victor Julien fca541f40e Add per app layer parser profiling
Per packet per app layer parser profiling. Example summary output:

Per App layer parser stats:

App Layer              IP ver   Proto   cnt        min      max          avg
--------------------   ------   -----   ------     ------   ----------   -------
ALPROTO_HTTP            IPv4       6    163394        126     38560320     42814
ALPROTO_FTP             IPv4       6       644        117        26100      2566
ALPROTO_TLS             IPv4       6       670        117         7137       799
ALPROTO_SMB             IPv4       6    114794        126       225270       957
ALPROTO_DCERPC          IPv4       6      5207        126        25596      1266

Also added to the csv out.

In the csv out there is a new column "stream (no app)" that removes the
app layer parsers from the stream tracking. So raw stream engine performance
becomes visible.
14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Eric Leblond a0b4068041 autotools: fix duplicate check command in Makefile.
It seems that check target can not be used in Makefile.am. Using
check-am fix a make failure.
15 years ago
Anoop Saldanha 576ec7da66 smtp parser support 15 years ago
Martin Beyer 2f1262b446 fixed cuda build: portability issues and nvcc version check 15 years ago
Martin Beyer 49d66430bc build cuda modules with make 15 years ago
Anoop Saldanha 35f3eafa5e byte extract added to the engine. Detection support added for packet payload, uri and dce detection engines 15 years ago
Victor Julien 7e128176d2 Add Vector datatype for SSE operations. 15 years ago
Victor Julien d0374ced38 Implement SACK in the stream engine. 15 years ago
Anoop Saldanha 966119b6aa support for http_raw_uri keyword + mpm engine 15 years ago
Anoop Saldanha 6fceeda8c5 move erf dag runmode into its own file runmode-erf-dag.[ch] 15 years ago
Anoop Saldanha f51cf34210 move erf file runmode into its own file runmode-erf-file.[ch] 15 years ago
Anoop Saldanha 86eabbc2f5 move ipfw runmode into its own file runmode-ipfw.[ch] 15 years ago
Anoop Saldanha 036015d6b9 move nfq runmode into its own file runmode-nfq.[ch] 15 years ago
Anoop Saldanha 9affa39b29 move pfring runmode into its own file runmode-pfring.[ch] 15 years ago
Anoop Saldanha e7ac1d7c4c move pcap file runmode into its own file runmode-pcap-file.[ch] 15 years ago
Anoop Saldanha f6af567ce0 move pcap live runmode into its own file runmode-pcap.[ch] 15 years ago
Eric Leblond 9beebf621a Add support for 'nfq_set_mark' keyword
This patch introduces 'nfq_set_mark' which is new rules option. If a packet
matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask
specified in the option during the verdict.
It is thus possible to trigger different behaviour on the packet inside
Linux/Netfilter.
15 years ago
William Metcalf 023a0f94a2 first stab at pcap logging no rotating buff etc 15 years ago
Anoop Saldanha c105a739e9 support for ssl_state keyword added 15 years ago
Anoop Saldanha 4c570777c4 delete files app-layer-tls.[ch] 15 years ago
Eric Leblond e1d966eaf6 Makefile: add sctp files to build
This patch simply adds decode-sctp files to the compilation.
15 years ago
Gurvinder Singh 7d0781b349 added support to log dropped packet as netfilter logs while in inline mode 15 years ago
Gurvinder Singh 8f8b1212af support for ssl_version keyword 15 years ago
Victor Julien 35b938a8db Don't pass config to unittests run in make check. 15 years ago
Eric Leblond 0044bb221b Add suricata unittests to 'make check'
This patch adds a run of suricata's unittests to 'make check'
15 years ago
Gurvinder Singh e5edc6e8e3 add the support to log the fast.log alerts type to syslog 15 years ago
Eric Leblond 37ee483b75 Add affinity util function and related files
This patch adds two new files which implement advanced affinity
settings.

Signed-off-by: Eric Leblond <eric@regit.org>
15 years ago
Victor Julien 3a774165fa Initial version of a inline raw reassembly function that reassembles in a sliding window. Introduce new unittest helpers for stream reassembly. 15 years ago
Victor Julien 2849d2b1d3 Initial code for stream 'inline' mode: packets that are (partly) overlapping with already accepted packets (meaning in the streams seg list) are rewritten to make sure they contain the exact same data. 15 years ago
Anoop Saldanha c9897a44a4 fast pattern support for http_cookie. Also support relative modifiers 15 years ago
Anoop Saldanha bbbedaf963 fast pattern support for http_method. Also support relative modifiers 15 years ago
Eric Leblond 3eada85ff8 Add interface setting discovery via ioctl
This patch adds support for MTU discovery of link following idea
of go.ph1g. It also adds some function to give a approximation of
link header length.
15 years ago
Anoop Saldanha eecf2d7e13 Add the makefile.am addition that I forgot to add in the previous commit for http_raw_header 15 years ago
Anoop Saldanha c61c68fd36 mpm and fast pattern support for http_header. Also support relative modifiers for http_header 15 years ago
Anoop Saldanha 5c6a65dc58 support relative modifiers for http_client_body. Introduce body processing engine in detect-engine-hcbd.[ch] 15 years ago
Gurvinder Singh b7da115e6d support for http_stat_code keyword has been added to detection module 15 years ago
Gurvinder Singh 1deae70cf7 added http_stat_msg keyword support for detection module 15 years ago
Anoop Saldanha 88d94b136d Support for reference.config file 15 years ago
Anoop Saldanha 658ff5753d aho-corasick for the cpu. We have 2 versions of ac. The first MPM_AC uses the delta table and the secone one MPM_AC_GFBS uses the goto-failure table 15 years ago
Gurvinder Singh 3eab715153 support for printing protocol names for known protocol 15 years ago
Victor Julien 1859ed54c7 Add memcmp api with a plain memcmp function and a SSE3 accelerated memcmp. 15 years ago
Victor Julien 87f88867f4 Further improve B2gc. Add B2gm. Improve memory layout. 15 years ago
Victor Julien 9dfbab42f8 WIP B2gc 15 years ago
Pablo Rincon 9d7baa7a9f Adding ssh app layer module with two new keywords: ssh.protoversion and ssh.softwareversion 15 years ago
Anoop Saldanha 33f4beb0bc batching of packets support for cuda b2g mpm. Supported for both 32 and 64 bit platforms 16 years ago
Victor Julien e685579231 Add optional structure validation code. 16 years ago
Pablo Rincon eed0ef6e69 Adding tag keyword support 16 years ago
Kirby Kuehl c3b9305259 dcerpc udp support 16 years ago
Anoop Saldanha 45ea0d914e dce stub content keywords support using dcepayload.c support for all dce related content keywords 16 years ago