Commit Graph

37 Commits (b4887943fb00998c9a5c1cebce9fd05dc899400c)

Author SHA1 Message Date
Anoop Saldanha 4ef3679b13 Remove the macro for pktacqloop which is now replaced by an inline function 14 years ago
Anoop Saldanha 8363533a02 support for forced stream reassembly for to be pruned flows 14 years ago
Anoop Saldanha 15359dc47e Slot structure now holds the TV it belongs to 14 years ago
Anoop Saldanha c365bafbf6 We now inspect timed out streams + streams not processed as yet, at engine shutdown 14 years ago
Anoop Saldanha a844eecb0e - Updated all runmodes to use synchronization points, right before each thread(slot function) tries to de-init the thread. - Main thread now first disables receive thread(s) before it kills receive and rest of the threads. 14 years ago
Anoop Saldanha e567c2d002 Introduce master-slave synchronization support for ThreadVars 14 years ago
Anoop Saldanha 94c5ecb069 introduce inline function version of TmThreadsSlotProcessPkt macro. Retain the macro as well 14 years ago
Anoop Saldanha fd6faac196 update TmThreadsSlotProcessPkt with better error handling + post pq processing 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Victor Julien 7e1d911215 Small optimizations to pkt acq loop code. 14 years ago
Victor Julien b753ecce50 Implement a pkt acq loop infra with support for pcap-file. 14 years ago
Anoop Saldanha 4f7df1029d Unify the use of slots to a single struct for threading API. Remove separate slot append functions for 1slot and varslot 14 years ago
Eric Leblond f9e453e14c affinity: Use configured 'threads' value if set
This patch modifies runmodes to make them use the new 'threads'
variable.

Signed-off-by: Eric Leblond <eric@regit.org>
15 years ago
Eric Leblond 0809deafc4 Implement function needed for affinity in tm-threads
This patch features the implementation of affinity related
changes in tm-threads. In place code has been used but some
refactoring has been done to avoid code duplication.

Signed-off-by: Eric Leblond <eric@regit.org>
15 years ago
Victor Julien 3c1ae607cf Fix cuda compilation. 15 years ago
Anoop Saldanha 33f4beb0bc batching of packets support for cuda b2g mpm. Supported for both 32 and 64 bit platforms 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Victor Julien 23cb020493 Fix thread flag code to compile with gnu99 15 years ago
Victor Julien 63cb1818ed Move TmThread*Flag functions to the header so they can be inline with gnu99 as well. 15 years ago
Pablo Rincon 7719216575 Setting thread priorities with nice 16 years ago
Anoop Saldanha 41e6735b92 mpm b2g cuda support added 16 years ago
Pablo Rincon b482471a7d Adding auto runmodes based on available core/cpu's. Setting thread priorities 16 years ago
Pablo Rincon e26833be3f Changing mutex/spinlocks/conditions naming types 16 years ago
Pablo Rincon 769022f4be Adding support for Mac OS X, FreeBSD, centrailizing mutex/spins/conditions in a macro API, and some unittests 16 years ago
Gurvinder Singh 48c94bbf03 initial thread code support 16 years ago
Anoop Saldanha 157d5e8113 Implementation of the logging module 16 years ago
Victor Julien 597d0e9a20 Fix detection of failed thread startup. Cleanup startup output a bit. 16 years ago
Victor Julien cfb605aa8a Put the precooked runmodes in a separate file. 16 years ago
Victor Julien 1858be7a2f Lock threadvars flags using spinlocks. 16 years ago
Anoop Saldanha 3a45b2711c Fixed the Perf API startup issue 16 years ago
Anoop Saldanha f35d9f0437 threading improvements. Replaced the use of slot(2/3) with varslot. Improve error handling in slot functions. Additional helper functions for thread creation 16 years ago
Anoop Saldanha 53c21410b6 improve the threading api for the ids 16 years ago
Anoop Saldanha a5fb240a4a Changes added for the Performance Counter API 16 years ago
Victor Julien 657be002d1 Big detection engine update: scan improvements, b2g/b3g updates, bloom fixes, iponly detection implementation, dsize/flow grouping. 16 years ago
Victor Julien 5df5b35e90 Put all globals in the detection engine ctx. Add HashList type, a hash that also stores the items ina list to it can be traversed. Many cleanups. 16 years ago
Victor Julien efb10fc0d6 big update 16 years ago
Victor Julien bab4b62376 Initial add of the files. 16 years ago