180 Commits (98484ffdccb3b9b0564e5dca0ceb91a5f1d9e1c8)

Author	SHA1	Message	Date
Victor Julien	8f337a3904	stream: never resend reassembled data to app layer.	13 years ago
Eric Leblond	619014a280	pool: rename Free function to Cleanup ... This patch renames Free functions to Cleanup as the free is made by the pool system.	13 years ago
Eric Leblond	fa079c1da0	pool: realize a block allocation for preallocated item. ... This patch required a evolution of Pool API as it is needed to proceed to alloc or init separetely. The PoolInit has been changed with a new Init function parameter.	13 years ago
Victor Julien	869109a6a0	stream/app layer: add Truncate app layer callback that is called if stream depth is reached. Use it to trunc open files in HTTP.	13 years ago
Victor Julien	9d2e17fa98	stream: don't NULL dereference p->flow->protoctx in StreamTcpReassembleDepthReached	13 years ago
Victor Julien	43c7fd7585	file inspection: improve logging when stream.depth limit is reached. #493.	13 years ago
Eric Leblond	0227a87fcb	cleaning: fix warning when building with clang. ... clang was issuing some warnings related to unused return in function. This patch adds some needed error treatment and ignore the rest of the warnings by adding a cast to void.	13 years ago
Anoop Saldanha	c5cc9d454d	stream raw reassembly fix	13 years ago
Victor Julien	ed3599b3d8	stream: improve error checking.	13 years ago
Victor Julien	88a21456e3	stream: keep segments in memory until we are sure the stream/state is inspected.	13 years ago
Anoop Saldanha	64625675ce	set stream_eof flag per stream, only when the stream initiates a close. Fix htp parser to close connection per direction based on this	13 years ago
Anoop Saldanha	225b917e93	remove unused stream ssn flag - STREAMTCP_FLAG_TOSERVER_REASSEMBLY_STARTED	13 years ago
Victor Julien	fddaca6e8b	Implement stream memcap enforcements using atomics instead of spinlocked counters.	14 years ago
Victor Julien	0150e66ede	flow engine: improve scalability ... Major redesign of the flow engine. Remove the flow queues that turned out to be major choke points when using many threads. Flow manager now walks the hash table directly. Simplify the way we get a new flow in case of emergency.	14 years ago
Victor Julien	e237841a8e	Fix compilation with profiling enabled. Minor unittest fixes.	14 years ago
Victor Julien	c4b34e6ef7	Fix various minor clang/scan-build warnings.	14 years ago
Victor Julien	16cfae2f51	Trigger raw stream reassembly on receiving a full HTTP request or response.	14 years ago
Victor Julien	ddfa5c49c6	Stream engine: gap handling ... Set a stream event for stream gaps. Add a (disabled by default) signature to the stream-event.rules.	14 years ago
Victor Julien	425294f912	stream reassembly: account stream gaps ... Add counter to the stream reassembly engine to count stream gaps. Stream gaps are the result of missing packets (usually due to packet loss). This missing data stops the reassembly for the app layer.	14 years ago
Victor Julien	262a7300d7	flow: shrink Flow datatype ... Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address that doesn't have the family in it like the Address structure. Instead, the family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6. Add macro's to check the family, copy the address, etc. Update many unittests to reflect these changes. Introduce unittest helper functions for creating and initializing a flow and freeing it again. On 64 bit this shrinks the flow with 8 bytes.	14 years ago
Victor Julien	a0b532dc45	stream reassembly: simplify base_seq tracking for protocol detection. Shrinks TcpStream structure.	14 years ago
Anoop Saldanha	4130c5e2b8	if flow has disabled app layer inspection, disable buffering the segments unnecessarily in inline reassembly	14 years ago
Anoop Saldanha	43cbed8c92	enable toclient alproto detection for inline reassembly	14 years ago
Anoop Saldanha	f684b60127	if flow has disabled app layer inspection, disable buffering the segments unnecessarily	14 years ago
Victor Julien	c9960473bb	Fix stream reassembly engine rejecting valid packet for reassembly.	14 years ago
Anoop Saldanha	4650bf7170	minor code cleanup. remove commented out code	14 years ago
Anoop Saldanha	78e6a7f713	enable toclient alproto detection. Detection all current alproto toclient PMP patterns	14 years ago
Victor Julien	2d16abcf8b	Minor code cleanups fixing all GCC 4.6 compiler warnings for default, debug and unittests mode.	14 years ago
Victor Julien	9b437caaea	Fix stream unittests.	14 years ago
Victor Julien	b8659daef7	Add stream engine counters ... Added stream counters: - tcp.reassembly_memuse -- current memory use by reassembly in bytes - tcp.memuse -- current memory use by stream tracking in bytes - tcp.reused_ssn -- ssn reused by new session with identical tuple - tcp.no_flow -- TCP packets with no flow - indicating flow engine memory at its limits	14 years ago
Victor Julien	8208eacd79	Convert stream memcaps to u64. Bug #332.	14 years ago
Victor Julien	fca541f40e	Add per app layer parser profiling ... Per packet per app layer parser profiling. Example summary output: Per App layer parser stats: App Layer IP ver Proto cnt min max avg -------------------- ------ ----- ------ ------ ---------- ------- ALPROTO_HTTP IPv4 6 163394 126 38560320 42814 ALPROTO_FTP IPv4 6 644 117 26100 2566 ALPROTO_TLS IPv4 6 670 117 7137 799 ALPROTO_SMB IPv4 6 114794 126 225270 957 ALPROTO_DCERPC IPv4 6 5207 126 25596 1266 Also added to the csv out. In the csv out there is a new column "stream (no app)" that removes the app layer parsers from the stream tracking. So raw stream engine performance becomes visible.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Victor Julien	862b708a70	Fix stream unittest.	14 years ago
Victor Julien	047b19d271	Fix a reassembly bug that in some cases could lead to a crash.	14 years ago
Gerardo Iglesias Galvan	dd5e438d6f	Make all access to memory tracking counters in stream engine lock protected	14 years ago
Gerardo Iglesias Galvan	bd6d1bfac4	Fix potential crash in classtype parsing code	14 years ago
Victor Julien	be5ad4402d	Fix stream reassembly engine compilation on Windows.	14 years ago
Anoop Saldanha	000ce98cd1	push all proto detection code into their respective app parser register functions for every alproto	14 years ago
Victor Julien	96c2f2c877	Fix 2 stream reassembly unittests	14 years ago
Victor Julien	3b40b02a1b	Stream reassembly fixes.	14 years ago
Victor Julien	d0374ced38	Implement SACK in the stream engine.	14 years ago
Victor Julien	892a8a4985	Make stream inline use the chunk size settings.	15 years ago
Victor Julien	4f5aad1476	Enforce configurable minimum chunk size in raw stream reassembly. Minor stream cleanups, unittest updates.	15 years ago
Victor Julien	936b34ddf6	Remove minimum init chunk length code, set a default limit of 2560 to the minimum chunk size, allow toclient raw reassembly to start even if toserver hasn't started yet.	15 years ago
Victor Julien	8faacb727d	Account for seg list not always being empty when stream closes.	15 years ago
Victor Julien	ecfa2d0176	Only remove segments from segment list if they are completely before ra_base_seq.	15 years ago
Victor Julien	5bdf16380d	Make sure we actually remove no longer required segments.	15 years ago
Victor Julien	7f45a4fd58	Fix missing segment flag, fix 2 unittests broken after previous stream changes.	15 years ago
Victor Julien	4a7f6079d5	Change segment removal in stream engine to not discard segments right away. Now they are only removed if they are fully before ra_base_seq.	15 years ago