21 Commits (ef7cd043cc285d31df7942f74aa7e4ca330db6b8)

Author	SHA1	Message	Date
Victor Julien	2c3a92a1c9	profiling: conditional rule profiling ... Add support for conditional rule profiling. Currently only simple rate limiting is supported, but hardcoded to inspecting rules for each packet.	12 years ago
Victor Julien	13d491f577	profiling: lower overhead when disabled ... Instead of a large (6k+) structure in the Packet, make the profiling storage dynamic. To do this the Packet->profile is now a pointer. Initial support for selective sampling, e.g. only profile every 1000th packet.	12 years ago
Victor Julien	5686c673ec	profiling: per buffer profiling	12 years ago
Victor Julien	97bfcac444	profiling: introduce per keyword profiling ... Initial version of per keyword profiling. Prints stats about how ofter a keyword was checked and what the costs were.	12 years ago
Victor Julien	c91c359692	profiling: fix build on older systems	13 years ago
Victor Julien	1f92307517	profiling: minor cleanup	13 years ago
Victor Julien	ec7e79c748	Rule profiling update ... - Remove usage of counters api. - Store stats in detect engine thread ctx to remove locking - Support rule reloads	13 years ago
Victor Julien	e874a5a3de	Fix error in per packet detection engine profiling.	14 years ago
Victor Julien	d908e707d7	profiling: add per lock location profiling ... Add profiling per lock location in the code. Accounts how often a lock is requested, how often it was contended, the max number of ticks spent waiting for it, avg number of ticks waiting for it and the total ticks for that location. Added a new configure flag --enable-profiling-locks to enable this feature.	14 years ago
Victor Julien	41e9dba20b	Profile pcap file callback.	14 years ago
Victor Julien	979edf0b97	Add way to profile mutex/spin locks per thread module.	14 years ago
Victor Julien	cc4e89fbe1	Profiling: convert all packet profile counters/variables to u64. Improve output for larger numbers.	14 years ago
Victor Julien	3693a7a9ee	Profiling: add accounting for several detection phases.	14 years ago
Victor Julien	e8e392fb1f	Profiling: add per packet accounting of how much ticks are spend in protocol detection.	14 years ago
Victor Julien	fca541f40e	Add per app layer parser profiling ... Per packet per app layer parser profiling. Example summary output: Per App layer parser stats: App Layer IP ver Proto cnt min max avg -------------------- ------ ----- ------ ------ ---------- ------- ALPROTO_HTTP IPv4 6 163394 126 38560320 42814 ALPROTO_FTP IPv4 6 644 117 26100 2566 ALPROTO_TLS IPv4 6 670 117 7137 799 ALPROTO_SMB IPv4 6 114794 126 225270 957 ALPROTO_DCERPC IPv4 6 5207 126 25596 1266 Also added to the csv out. In the csv out there is a new column "stream (no app)" that removes the app layer parsers from the stream tracking. So raw stream engine performance becomes visible.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Victor Julien	1bd1a62526	Rename profile macro's and variables to reflect that they are for rule profiling.	14 years ago
William	85643fe780	Convert to logging perf stats to file by default. Add a few columns to output avg ticks per match, avg ticks non match, allow sorting on based on them.	14 years ago
Jason Ish	ea4b7cc33b	add profiling to stateful detection engine + other fixups.	15 years ago
William Metcalf	2eef905c07	GPL and Copyright header updates.	15 years ago
Jason Ish	18e5ac8cde	Basic rule profiling even though the results may be skewed by a bad rule in a grouping of rules.	15 years ago