159 Commits (7e3c15e54a293a66e59a22ace9ea05f513086187)

Author	SHA1	Message	Date
Anoop Saldanha	4130c5e2b8	if flow has disabled app layer inspection, disable buffering the segments unnecessarily in inline reassembly	14 years ago
Anoop Saldanha	43cbed8c92	enable toclient alproto detection for inline reassembly	14 years ago
Anoop Saldanha	f684b60127	if flow has disabled app layer inspection, disable buffering the segments unnecessarily	14 years ago
Victor Julien	c9960473bb	Fix stream reassembly engine rejecting valid packet for reassembly.	14 years ago
Anoop Saldanha	4650bf7170	minor code cleanup. remove commented out code	14 years ago
Anoop Saldanha	78e6a7f713	enable toclient alproto detection. Detection all current alproto toclient PMP patterns	14 years ago
Victor Julien	2d16abcf8b	Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode.	14 years ago
Victor Julien	9b437caaea	Fix stream unittests.	14 years ago
Victor Julien	b8659daef7	Add stream engine counters ... Added stream counters: - tcp.reassembly_memuse -- current memory use by reassembly in bytes - tcp.memuse -- current memory use by stream tracking in bytes - tcp.reused_ssn -- ssn reused by new session with identical tuple - tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits	14 years ago
Victor Julien	8208eacd79	Convert stream memcaps to u64. Bug #332.	14 years ago
Victor Julien	fca541f40e	Add per app layer parser profiling ... Per packet per app layer parser profiling. Example summary output: Per App layer parser stats: App Layer IP ver Proto cnt min max avg -------------------- ------ ----- ------ ------ ---------- ------- ALPROTO_HTTP IPv4 6 163394 126 38560320 42814 ALPROTO_FTP IPv4 6 644 117 26100 2566 ALPROTO_TLS IPv4 6 670 117 7137 799 ALPROTO_SMB IPv4 6 114794 126 225270 957 ALPROTO_DCERPC IPv4 6 5207 126 25596 1266 Also added to the csv out. In the csv out there is a new column "stream (no app)" that removes the app layer parsers from the stream tracking. So raw stream engine performance becomes visible.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Victor Julien	862b708a70	Fix stream unittest.	14 years ago
Victor Julien	047b19d271	Fix a reassembly bug that in some cases could lead to a crash.	14 years ago
Gerardo Iglesias Galvan	dd5e438d6f	Make all access to memory tracking counters in stream engine lock protected	14 years ago
Gerardo Iglesias Galvan	bd6d1bfac4	Fix potential crash in classtype parsing code	14 years ago
Victor Julien	be5ad4402d	Fix stream reassembly engine compilation on Windows.	14 years ago
Anoop Saldanha	000ce98cd1	push all proto detection code into their respective app parser register functions for every alproto	14 years ago
Victor Julien	96c2f2c877	Fix 2 stream reassembly unittests	14 years ago
Victor Julien	3b40b02a1b	Stream reassembly fixes.	14 years ago
Victor Julien	d0374ced38	Implement SACK in the stream engine.	14 years ago
Victor Julien	892a8a4985	Make stream inline use the chunk size settings.	15 years ago
Victor Julien	4f5aad1476	Enforce configurable minimum chunk size in raw stream reassembly. Minor stream cleanups, unittest updates.	15 years ago
Victor Julien	936b34ddf6	Remove minimum init chunk length code, set a default limit of 2560 to the minimum chunk size, allow toclient raw reassembly to start even if toserver hasn't started yet.	15 years ago
Victor Julien	8faacb727d	Account for seg list not always being empty when stream closes.	15 years ago
Victor Julien	ecfa2d0176	Only remove segments from segment list if they are completely before ra_base_seq.	15 years ago
Victor Julien	5bdf16380d	Make sure we actually remove no longer required segments.	15 years ago
Victor Julien	7f45a4fd58	Fix missing segment flag, fix 2 unittests broken after previous stream changes.	15 years ago
Victor Julien	4a7f6079d5	Change segment removal in stream engine to not discard segments right away. Now they are only removed if they are fully before ra_base_seq.	15 years ago
Victor Julien	b233105cc2	Fix a issue in stream reassembly causing the segment list getting into a inconsistent state.	15 years ago
Victor Julien	a8db8b334b	Remove debug stream testing code from non-debug builds.	15 years ago
Victor Julien	48c7f18453	Fix bug in the segment insert code causing an inconsistent segment list in some overlap conditions.	15 years ago
Victor Julien	05539d7357	Fix a reassembly overlap issue. Fix a inline reassembly gap handling issue.	15 years ago
Victor Julien	e92ab40d39	Fix compilation for non-DEBUG case.	15 years ago
Victor Julien	2db06cc79e	Improve Inline reassembly wrt to GAP handling. Add more tests.	15 years ago
Victor Julien	4c82c0e750	Improve RawInline reassembly: remove unnecessary segments from the stream in an earlier stage. Test this properly.	15 years ago
Victor Julien	3a774165fa	Initial version of a inline raw reassembly function that reassembles in a sliding window. Introduce new unittest helpers for stream reassembly.	15 years ago
Victor Julien	abdffadc1c	Add a new app layer reassembly function that is for inline use, and use it when the stream engine is in inline mode.	15 years ago
Victor Julien	bff70eed6d	Update to depth code. Get segment from the correct pool when a payload is truncated.	15 years ago
Victor Julien	66c40f782c	Have reassembly errors also set a stream event.	15 years ago
Victor Julien	0f072648e6	Another iteration of the reassembly depth enforcement, now considering retransmissions.	15 years ago
Victor Julien	16cd31a408	Remove unused pseudo packet reassembly code.	15 years ago
Victor Julien	bf88a6de09	Add depth comment.	15 years ago
Victor Julien	a26768ce7a	Change the way the reassembly depth is enforced. Ignore retransmissions, get rid of per session counter.	15 years ago
Victor Julien	aa04d9eefb	Improve stream gap handling. Instead of giving up as soon as we see a gap we now wait much longer before we decide it's a gap.	15 years ago
Victor Julien	2849d2b1d3	Initial code for stream 'inline' mode: packets that are (partly) overlapping with already accepted packets (meaning in the streams seg list) are rewritten to make sure they contain the exact same data.	15 years ago
Victor Julien	8b5f553a35	Inspect a pseudo packet upon receiving a RST so that we are sure both sides of the TCP session are inspected.	15 years ago
Gurvinder Singh	55a863359c	support for pseudo packet creation from reassembled stream segments	15 years ago
Victor Julien	44e678b86b	Comment out disabled unittests.	15 years ago
Victor Julien	1d971b53a6	Update all unittests	15 years ago