Commit Graph

66 Commits (dec34afa4039aee40cc765a4699787ebce2da16f)

Author SHA1 Message Date
Anoop Saldanha 5133098bd6 Accomodate pcap-file mode to signal flow mgr to wakeup when it exceeds a certain time interval. This let's the flow mgr keep in sync with pcap timestamp changes 14 years ago
Victor Julien 67cea09911 Handle failing thread modules that are called by the Pcap file callback. 14 years ago
Anoop Saldanha d68f182ebd introduce SCPerfSyncCounters/SCPerfSyncCounters macro to synchronize counters 14 years ago
Anoop Saldanha f7b1972263 update broken stats.log. Use pktacqloop funcs in pcap-file, pfring, pcap-live, af-pkt to sync counters - bug #343 14 years ago
Anoop Saldanha 3f1c4efceb Add new flags var to tm module. TMs can now set flags to identify special properties. Also use these to identify receive TMs 14 years ago
Eric Leblond de1d002ea6 Return OK when leaving cleanly. 14 years ago
Victor Julien 820b0ded82 Add per packet profiling.
Per packet profiling uses tick based accounting. It has 2 outputs, a summary
and a csv file that contains per packet stats.

Stats per packet include:
 1) total ticks spent
 2) ticks spent per individual thread module
 3) "threading overhead" which is simply calculated by subtracting (2) of (1).

A number of changes were made to integrate the new code in a clean way:
a number of generic enums are now placed in tm-threads-common.h so we can
include them from any part of the engine.

Code depends on --enable-profiling just like the rule profiling code.

New yaml parameters:

profiling:
  # packet profiling
  packets:

    # Profiling can be disabled here, but it will still have a
    # performance impact if compiled in.
    enabled: yes
    filename: packet_stats.log
    append: yes

    # per packet csv output
    csv:

      # Output can be disabled here, but it will still have a
      # performance impact if compiled in.
      enabled: no
      filename: packet_stats.csv

Example output of summary stats:

IP ver   Proto   cnt        min      max          avg
------   -----   ------     ------   ----------   -------
 IPv4       6     19436      11448      5404365     32993
 IPv4     256         4      11511        49968     30575

Per Thread module stats:

Thread Module              IP ver   Proto   cnt        min      max          avg
------------------------   ------   -----   ------     ------   ----------   -------
TMM_DECODEPCAPFILE          IPv4       6     19434       1242        47889      1770
TMM_DETECT                  IPv4       6     19436       1107       137241      1504
TMM_ALERTFASTLOG            IPv4       6     19436         90         1323       155
TMM_ALERTUNIFIED2ALERT      IPv4       6     19436        108         1359       138
TMM_ALERTDEBUGLOG           IPv4       6     19436         90         1134       154
TMM_LOGHTTPLOG              IPv4       6     19436        414      5392089      7944
TMM_STREAMTCP               IPv4       6     19434        828      1299159     19438

The proto 256 is a counter for handling of pseudo/tunnel packets.

Example output of csv:

pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading
1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337

First line of the file contains labels.

2 example gnuplot scripts added to plot the data.
14 years ago
Eric Leblond 88559901d4 pcap-file: Allocated packet must be free if there's error 14 years ago
Victor Julien b753ecce50 Implement a pkt acq loop infra with support for pcap-file. 14 years ago
Eric Leblond dd038c1906 Modify files to avoid direct pckt payload access
This patch implements the needed modification of payload access
in a Packet structure to support the abstraction introduced by
the extended data system.
15 years ago
Victor Julien 4cacb1e970 Disable adding to unregistered mbit/s counter. 15 years ago
Victor Julien 6943a7eb8c Move updating the time from the pcap callback to the decoding stage in file mode. 15 years ago
Victor Julien 634b328d38 In case of error in pcap file reading mode, we shut the engine down hard instead of gracefully. 15 years ago
Victor Julien 1d74797b17 Attempt to work around NULL packets we're seeing ending up in queues when the compiler has optimized our code. 15 years ago
Victor Julien 6519a86ec7 Move packet pool to ringbuffer, update packet pool api and ringbuffer api. Remove memset usage from PACKET_RECYCLE, add proper cleanup macros. 15 years ago
Victor Julien dce323b1f4 Fix SCondWait -> SCCondWait typo. 15 years ago
Pablo Rincon 8cc525c939 UDP support at AppLayer message handling 15 years ago
Victor Julien b657705d8c Fix pcap file mode not shutting down on ctrl-c 15 years ago
Victor Julien e3fc53ecb9 Fix a corner case where the pcap receive modules could alloc packets at line rate until memory was depleted. 15 years ago
Victor Julien 53acf08996 Add multi packet reading for pcap live mode. Add a partly lock free multi writer, multi reader ringbuffer. 15 years ago
Victor Julien 4e7df60b2f Make pcap file mode read multiple packets per 'read'. Update threading model to deal with this. 15 years ago
William Metcalf 2eef905c07 GPL and Copyright header updates. 15 years ago
Gerardo Iglesias Galvan 9f4fae5b1a Fix inconsistent use of dynamic memory allocation 15 years ago
Victor Julien ccf22cf563 Move to different way of enforcing max_pending. Should require less locks. 15 years ago
William Metcalf 41172f0024 increment packet count before assigning value, tshark/wireshark starts with pkt no 1 so should we 15 years ago
Victor Julien b90ebc1ce5 Add a packet count var for pcap file mode to the Packet structure to ease debugging. 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Victor Julien 070ed778b8 Libcap-ng support by Gurvinder Singh and myself. Basic support for per thread caps is added, but not activated as it doesn't seem to work yet. Work around for incompatibility between libnet 1.1 and libcap-ng added. 15 years ago
Victor Julien eeb98c6900 Move SCSetThreadName to proper functions. 15 years ago
Gerardo Iglesias Galvan 9f35a24a1f Set threads name. Fix bug #83 15 years ago
Victor Julien 7dcc1daa89 Pcap eof msg can be informational 15 years ago
Victor Julien c5e15213f7 Improve pcap file mode EOF message. Fixes #123. Small cleanups to pcap file code. 15 years ago
Pablo Rincon 25a3a5c6d8 Adding mem wrapper to debug runtime alloc()/free() functions. Fixing some memory leaks. 16 years ago
Pablo Rincon 29d51a6182 Using the loggin API in source-pcap and source-pcap-file 16 years ago
Gurvinder Singh 5293681860 b86 16 years ago
Jason Ish 7142fdb780 quick way to make max_pending configurable. 16 years ago
Victor Julien 53c9276d51 Cleanup pcap output. 16 years ago
Victor Julien e0aacac4c6 Move bpf string retrieval to it's own function. Clean up pcap sourcres a bit. 16 years ago
William Metcalf ba46c16aac bpf support for pcap modes 16 years ago
Steve Grubb c95cd2e80a memory leak cleanups in misc places
Hello,

This is all the rest of the memory leaks I found.

*In src/source-pcap-file.c at line 152, ptv is not being freed.
*In src/util-unittest-helper.c at line 152, p was not being freed.
*In src/log-httplog.c at line 195, aft was not being freed
*In src/counters.c at line 51, log_filename was not being freed. At line 1188
pctx is being tested to see if its NULL. However, at 1173 it exits the
function if it were NULL. This test is not needed and should be deleted.
*In src/defrag.c at line 351, tracker was not being freed. At line 390, dc is
being checked for NULL but this was already done at line 384. Probably what
was meant was checking the value of dc->frag_table which was just assigned.

The patch below makes the above described changes.

-Steve
16 years ago
William Metcalf 8a64321340 raw pcap support additionl ipv4/6 validation 16 years ago
Victor Julien 9ececacda3 Fix packet timestamp handling for encapsulated packets. 16 years ago
Victor Julien b7bac14040 Fixup code to compile with -Wall -Werror -Wextra -Wno-unused-parameter compiler options. 16 years ago
Victor Julien ecf86f9c23 Rename to Suricata. 16 years ago
Pablo Rincon e26833be3f Changing mutex/spinlocks/conditions naming types 16 years ago
Pablo Rincon 769022f4be Adding support for Mac OS X, FreeBSD, centrailizing mutex/spins/conditions in a macro API, and some unittests 16 years ago
Anoop Saldanha 8beef4a9fc stats upgrade. Added interval counters to the decoder module 16 years ago
Anoop Saldanha ceb7e495ae refactoring perf stats code 16 years ago
Gurvinder Singh ec75946e21 one more change 16 years ago
Gurvinder Singh 40b8afdd56 support for thread exit constants 16 years ago