72 Commits (94571c5dd28858ff68c44b648fd41c5d87c0e28d)

Author	SHA1	Message	Date
Eric Leblond	18c7eeefca	packet handling: fix release function ... Extended data were freed before the release function was called. The result was that, in AF_PACKET IPS mode, the release function was only sending void data because it the content of the extended data is the content of the packet. This patch updates the code to have the freeing of extended data done in the cleaning function for a packet which is called by the release function. This improves consistency of the code and fixes the bug.	11 years ago
Victor Julien	d26ceb2356	decode: introduce DecodeThreadVarsFree ... As a mirror of DecodeThreadVarsAlloc, DecodeThreadVarsFree is used to free the memory that DecodeThreadVarsAlloc alloc'd, including AppLayer storage.	11 years ago
Victor Julien	f791d0f5c5	vlan/QinQ: add vlan_qinq counter ... This patch introduces a new counter "decoder.vlan_qinq". It counts packets that have more than two stacked vlan layers. Packets with 2 vlan layers will both increment "decoder.vlan" and "decoder.vlan_qinq".	12 years ago
Victor Julien	91f8c33b0a	defrag: fix reassembly with vlan ... When creating a pseudo packet with the reassembled IP packet, the parent's vlan id or id's are also needed. The defrag packet is run through decode and the flow engine, where the vlan id is necessary for connecting the packet to the correct flow.	12 years ago
Anoop Saldanha	29fb9b099a	CUDA: Fix header file resolution issues.	12 years ago
Victor Julien	5f307acace	Pass ThreadVars ptr to various thread init funcs ... To be able to register counters from AppLayerGetCtxThread, the ThreadVars pointer needs to be available in it and thus in it's callers: - AppLayerGetCtxThread - DecodeThreadVarsAlloc - StreamTcpReassembleInitThreadCtx	12 years ago
Anoop Saldanha	429c6388f6	App layer API rewritten. The main files in question are: ... app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch]. Things addressed in this commit: - Brings out a proper separation between protocol detection phase and the parser phase. - The dns app layer now is registered such that we don't use "dnstcp" and "dnsudp" in the rules. A user who previously wrote a rule like this - "alert dnstcp....." or "alert dnsudp....." would now have to use, alert dns (ipproto:tcp;) or alert udp (app-layer-protocol:dns;) or alert ip (ipproto:udp; app-layer-protocol:dns;) The same rules extend to other another such protocol, dcerpc. - The app layer parser api now takes in the ipproto while registering callbacks. - The app inspection/detection engine also takes an ipproto. - All app layer parser functions now take direction as STREAM_TOSERVER or STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the functions. - FlowInitialize() and FlowRecycle() now resets proto to 0. This is needed by unittests, which would try to clean the flow, and that would call the api, AppLayerParserCleanupParserState(), which would try to clean the app state, but the app layer now needs an ipproto to figure out which api to internally call to clean the state, and if the ipproto is 0, it would return without trying to clean the state. - A lot of unittests are now updated where if they are using a flow and they need to use the app layer, we would set a flow ipproto. - The "app-layer" section in the yaml conf has also been updated as well.	12 years ago
Ken Steele	f9705377ae	Remove pkt variable from Packet structure. ... The uint8_t *pkt in the Packet structure always points to the memory immediately following the Packet structure. It is better to simply calculate that value every time than store the 8 byte pointer.	12 years ago
Eric Leblond	f76448c1e6	decode: fix failure in layered tunnel ... If we have multiple layer of tunnel, the decoding of initial Packet will recurse in DecodeTunnel function called in PacketTunnelPktSetup. If we are not setting the pseudo packet root before calling DecodeTunnel (as done in previous code), then the tunnel root will no be correct for the lower layer packets. This result in an counter problem and a suricata failure after some time.	12 years ago
Eric Leblond	0460b194b1	decode: clean DecodeThreadVars counter ... Speed counters are not compute anymore and can be removed from the structure definition.	12 years ago
Eric Leblond	3088b6ac34	Add invalid pkt counter. ... This patch adds and increments a invalid packet counter. It does this by introducing PacketDecodeFinalize function This function is incrementing the invalid counter and is also signalling the packet to CUDA.	12 years ago
Victor Julien	277fb61c1d	defrag: clean up ... Rename PacketDefragPktFinishSetup to PacketDefragPktSetupParent to better refect it's function.	12 years ago
Eric Leblond	3fdf52239d	defrag: don't modify packet if defrag fails ... If defrag fails dur to an invalid decoding, we are not modifying the origin packet anymore.	12 years ago
Eric Leblond	c611b258a5	decode: PacketTunnelPktSetup replaces PacketPseudoPktSetup ... This patch replaces PacketPseudoPktSetup by a better named PacketTunnelPktSetup function which is also in charge of doing the decoding of the tunneled packet. This allow to clean the code. But it also fixes an issue. Previously, if the DecodeTunnel function was failling (cause of an invalid packet mainly), the result was that the original packet to be considered as a tunnel packet (and not inspected by payload detection).	12 years ago
Eric Leblond	d4b7ecfbe3	decode: update API to return error ... In some cases, the decoding is not possible and some really invalid packet can be created. This is in particular the case of tunnel. In that case, it is more interesting to forget about the tunneled packet and only consider the original packet. DecodeTunnel function is maked as warn_unused_result because it is meaningful for the decoder to know if the underlying data were not correct. And in this case, only focus detection on the content.	12 years ago
Victor Julien	49087f21e4	Optimizations to reduce branch misses	12 years ago
Victor Julien	698ff4e4aa	Counters: remove all unused parts of the API	12 years ago
Victor Julien	8080494e9a	counters: consolidate counters after all ThreadInit functions of a thread have run. This prevents duplicate and overwriting memory allocations.	12 years ago
Eric Leblond	3dbf6c6fee	solaris: fix compilation failure ... This patch fixes a compilation failure on Solaris. Compiler does not support when a function returning void is used in return of an other function returning void.	12 years ago
Victor Julien	d82ce3f50c	Fix compiler warning due to missing include ... decode.c: In function 'DecodeThreadVarsAlloc': decode.c:437: error: implicit declaration of function 'ConfGetBool'	12 years ago
Victor Julien	16c3487444	Add yaml option to disable vlan ids hashing ... In some cases using the vlan id(s) in flow hashing is problematic. Cases of broken routers have been reported. So this option allows for disabling the use of vlan id(s) while calculating the flow hash, and in the future other hashes. Vlan tracking for flow is enabled by default.	12 years ago
Ken Steele	b076a26cdc	Replace ReleaseData function on Packet Structure with ReleasePacket. ... This commit allows handling Packets allocated by different methods. The ReleaseData function pointer in the Packet structure is replaced with ReleasePacket function pointer, which is then always called to release the memory associated with a Packet. Currently, the only usage of ReleaseData is in AF Packet. Previously ReleaseData was only called when it was not NULL. To implement the same functionality as before in AF Packet, a new function is defined in AF Packet to first call the AFP specific ReleaseData function and then releases the Packet structure. Three new general functions are defined for releasing packets in the default case: 1) PacketFree() - To release a packet alloced with SCMalloc() 2) PacketPoolReturnPacket() - For packets allocated from the Packet Pool. Calls RECYCLE_PACKET(p) 3) PacketFreeOrRelease() - Calls PacketFree() or PacketPoolReturnPacket() based on the PKT_ALLOC flag. Having these functions removes the need to check the PKT_ALLOC flag when releasing a packet in most cases, since the ReleasePacket function encodes how the Packet was allocated. The PKT_ALLOC flag is still set and is needed when AF Packet releases a packet, since it replaces the ReleasePacket function pointer with its own function and then calls PacketFreeOfRelease(), which uses the PKT_ALLOC flag.	12 years ago
Victor Julien	4a0050b9ea	Print pkt src to alert-debug log	12 years ago
Ken Steele	699d9e01f1	Move memset() out of PACKET_INITIALIZE() ... The memset() inside PACKET_INITIALIZE() is redundant in some cases and it is cleaner to do as part of the memory allocation. This simplifies changes for integrating Tilera mPIPE support because the size of memory cleared in that case is different from SIZE_OF_PACKET. For the cases where Packets are directly allocated and then call PACKET_INITIALIZE() without memset() first, this patch adds memset() calls. A further change would use GetPacketFromAlloc() directly.	12 years ago
Eric Leblond	e176be6fcc	Use unlikely for error treatment. ... When handling error case on SCMallog, SCCalloc or SCStrdup we are in an unlikely case. This patch adds the unlikely() expression to indicate this to gcc. This patch has been obtained via coccinelle. The transformation is the following: @istested@ identifier x; statement S1; identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)"; @@ x = func(...) ... when != x - if (x == NULL) S1 + if (unlikely(x == NULL)) S1	13 years ago
Eric Leblond	f241312a36	defrag: don't use message for repetitive error ... When nothing can be fetch from the pool, this can repeat frequently. Thus displaying a message in the log will not help. This patch uses a counter instead of a log message. As this is a sort of memcap this is conformed to what is done for other issues of the same type.	13 years ago
Eric Leblond	e6e339aacf	Add counters for IPv4 in IPv6 and IPv6 in IPv6	13 years ago
Eric Leblond	f9046d8284	Add teredo counter.	13 years ago
Victor Julien	48da3bb48b	Make sure all fake packets have datalink type DLT_RAW. Make sure stream end packets set pkt size.	13 years ago
Eric Leblond	59057e542e	Openbsd: Fix some warning related to inline usage. ... gcc on OpenBSD does not support C99 inline functions. This patch modify the build system to handle this. It also change the order of declaration of some functions to avoid to use them before declaring them as inline.	13 years ago
Victor Julien	79691f675a	defrag: don't increment recursion level for reassembled packets. Fixes defragged packets not seeing the same flow.	13 years ago
Eric Leblond	3593cb051e	decode: add PacketSetData funtion ... This patch adds a function which can be used to set the payload of a packet when a zero copy mode is used.	14 years ago
Anoop Saldanha	7c9d1b80fd	Update size parsing API with new calls for returing u8, u16, u32 and u64 values. Make updates in the codebase to use these new calls	14 years ago
Eric Leblond	acf10525f6	doc: add decode group and related documentation.	14 years ago
Eric Leblond	324986694a	decode: improve and fix comments.	14 years ago
Anoop Saldanha	54f6e4ff4d	Merge thread kill functions. Merge slot's tm_id with the one used by packet profiling. Remove some junk unused code from ms sync pts. Timeout setup cleanup as well. packet q dbg_maxlen now u32 var.	14 years ago
Anoop Saldanha	99207c718d	Avoid possiblity of potential engine idling from consumption of all packetpool packets - v1 ... - Now forced reassembly uses only malloced packets.	14 years ago
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Eric Leblond	6b9d1012ff	Transform inet_ntop call into PrintInet one.	14 years ago
Victor Julien	54cd3552e1	Remove tunnel_proto field from Packet structure.	14 years ago
Victor Julien	f5674eff74	Fix a copy issue in PacketCopyDataOffset.	14 years ago
Eric Leblond	482991ad6d	decode: add support for SCTP protocol ... This patch adds a new counter for SCTP and defines some macros needed for SCTP support.	15 years ago
Eric Leblond	49adc264bc	Don't print message after SCMalloc failure. ... This patch generated via coccinelle is getting rid of logging message after a SCMalloc failure. They were useless as SCMalloc already displays a message.	15 years ago
Eric Leblond	e802e1ed16	Modify Packet structure and prepare accessor. ... This patch modifies decode.c and decode.h to avoid the usage by default of a bigger than 65535 bytes array in Packet structure. The idea is that the packet are mainly under 1514 bytes size and a bigger size must be supported but should not be the default. If the packet length is bigger than DFLT_PACKET_SIZE then the data are stored in a dynamically allocated part of the memory. To ease the modification of the rest of the code, functions to access and set the payload/length in a Packet have been introduced. The default packet size can be set at runtime via the default-packet-size configuration variable.	15 years ago
Victor Julien	587a53b904	Disable per second counters as they are unreliable.	15 years ago
Victor Julien	6519a86ec7	Move packet pool to ringbuffer, update packet pool api and ringbuffer api. Remove memset usage from PACKET_RECYCLE, add proper cleanup macros.	15 years ago
Victor Julien	ba12f3c109	Applayer to flow fixes and cleanups.	15 years ago
Pablo Rincon	8cc525c939	UDP support at AppLayer message handling	15 years ago
William Metcalf	2eef905c07	GPL and Copyright header updates.	15 years ago
Victor Julien	bb685751d9	Fix NFQ receive/verdict race condition in cases where the packetpool is empty.	15 years ago