170 Commits (4749420f325acc5965cd942d78ba9ce28c686eda)

Author	SHA1	Message	Date
Victor Julien	820b0ded82	Add per packet profiling. ... Per packet profiling uses tick based accounting. It has 2 outputs, a summary and a csv file that contains per packet stats. Stats per packet include: 1) total ticks spent 2) ticks spent per individual thread module 3) "threading overhead" which is simply calculated by subtracting (2) of (1). A number of changes were made to integrate the new code in a clean way: a number of generic enums are now placed in tm-threads-common.h so we can include them from any part of the engine. Code depends on --enable-profiling just like the rule profiling code. New yaml parameters: profiling: # packet profiling packets: # Profiling can be disabled here, but it will still have a # performance impact if compiled in. enabled: yes filename: packet_stats.log append: yes # per packet csv output csv: # Output can be disabled here, but it will still have a # performance impact if compiled in. enabled: no filename: packet_stats.csv Example output of summary stats: IP ver Proto cnt min max avg ------ ----- ------ ------ ---------- ------- IPv4 6 19436 11448 5404365 32993 IPv4 256 4 11511 49968 30575 Per Thread module stats: Thread Module IP ver Proto cnt min max avg ------------------------ ------ ----- ------ ------ ---------- ------- TMM_DECODEPCAPFILE IPv4 6 19434 1242 47889 1770 TMM_DETECT IPv4 6 19436 1107 137241 1504 TMM_ALERTFASTLOG IPv4 6 19436 90 1323 155 TMM_ALERTUNIFIED2ALERT IPv4 6 19436 108 1359 138 TMM_ALERTDEBUGLOG IPv4 6 19436 90 1134 154 TMM_LOGHTTPLOG IPv4 6 19436 414 5392089 7944 TMM_STREAMTCP IPv4 6 19434 828 1299159 19438 The proto 256 is a counter for handling of pseudo/tunnel packets. Example output of csv: pcap_cnt,ipver,ipproto,total,TMM_DECODENFQ,TMM_VERDICTNFQ,TMM_RECEIVENFQ,TMM_RECEIVEPCAP,TMM_RECEIVEPCAPFILE,TMM_DECODEPCAP,TMM_DECODEPCAPFILE,TMM_RECEIVEPFRING,TMM_DECODEPFRING,TMM_DETECT,TMM_ALERTFASTLOG,TMM_ALERTFASTLOG4,TMM_ALERTFASTLOG6,TMM_ALERTUNIFIEDLOG,TMM_ALERTUNIFIEDALERT,TMM_ALERTUNIFIED2ALERT,TMM_ALERTPRELUDE,TMM_ALERTDEBUGLOG,TMM_ALERTSYSLOG,TMM_LOGDROPLOG,TMM_ALERTSYSLOG4,TMM_ALERTSYSLOG6,TMM_RESPONDREJECT,TMM_LOGHTTPLOG,TMM_LOGHTTPLOG4,TMM_LOGHTTPLOG6,TMM_PCAPLOG,TMM_STREAMTCP,TMM_DECODEIPFW,TMM_VERDICTIPFW,TMM_RECEIVEIPFW,TMM_RECEIVEERFFILE,TMM_DECODEERFFILE,TMM_RECEIVEERFDAG,TMM_DECODEERFDAG,threading 1,4,6,172008,0,0,0,0,0,0,47889,0,0,48582,1323,0,0,0,0,1359,0,1134,0,0,0,0,0,8028,0,0,0,49356,0,0,0,0,0,0,0,14337 First line of the file contains labels. 2 example gnuplot scripts added to plot the data.	14 years ago
Eric Leblond	a8ae1c42c3	Fix macro about default packet size ... Being pessimistic about packet default size has side effect in some module. Falling back to the sane correct value.	14 years ago
Victor Julien	ad175c8aec	Fix complition on OS/archs that don't support atomic variables.	14 years ago
Victor Julien	e1d4e16645	Simplify packet decoding macro's.	14 years ago
Victor Julien	52eb8d2be0	Convert mutex protected tunnel counters to lockless atomic counters.	14 years ago
Victor Julien	54cd3552e1	Remove tunnel_proto field from Packet structure.	14 years ago
Victor Julien	3d22713b09	Convert Packet tunnel variables to bit flag checks.	14 years ago
Victor Julien	75439863ed	Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15.	14 years ago
Victor Julien	9ca0658a6e	Clear pcap_cnt variable on packet recycle.	15 years ago
Victor Julien	864c8718e1	Store matching stream msg (ptr) in packets alert structure so it's available to the output plugins.	15 years ago
Victor Julien	bc7e21aee6	Add special sguil mode to log-pcap to support logging into date based directory structure and rotate when the day passes. Also do not log packets beyond stream reassembly depth and encrypted traffic.	15 years ago
Eric Leblond	4495efcb62	Add macro for direct access ... In some case, this is needed to have a direct access to the pkt field. This patch adds macro for this usage.	15 years ago
Eric Leblond	9beebf621a	Add support for 'nfq_set_mark' keyword ... This patch introduces 'nfq_set_mark' which is new rules option. If a packet matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask specified in the option during the verdict. It is thus possible to trigger different behaviour on the packet inside Linux/Netfilter.	15 years ago
Pablo Rincon	ce3b76a102	Fix compilation on Mac OS X (it was missing IPPROTO_SCTP definition)	15 years ago
Eric Leblond	482991ad6d	decode: add support for SCTP protocol ... This patch adds a new counter for SCTP and defines some macros needed for SCTP support.	15 years ago
Eric Leblond	8be92fdd99	SCTP support: add parsing of sctp ... This patch adds support of SCTP in all part of the code in charge of decoding packets.	15 years ago
Victor Julien	477bc1d050	Set DROP flag on a packet in addition to the REJECT flags. This makes sure we not only send a reject, but also drop the offending packet. Closes #248.	15 years ago
Victor Julien	addab7b5ee	Don't test the several packet detection checks against pseudo packets as the matches would not be meaningful anyway. Prevents a segv in the csum detection.	15 years ago
Eric Leblond	d0faa6c96e	Fix some spacing. ... This trivial patch fixes some indentation problems. Signed-off-by: Eric Leblond <eric@regit.org>	15 years ago
Victor Julien	25f5589078	First round of adding 'stream events'. Basic stream tracking events added.	15 years ago
Victor Julien	2849d2b1d3	Initial code for stream 'inline' mode: packets that are (partly) overlapping with already accepted packets (meaning in the streams seg list) are rewritten to make sure they contain the exact same data.	15 years ago
Victor Julien	f606621e8c	Fix the pseudo packet having the wrong proto set, causing massive fp's. Flag packets to be part of the established phase of a tcp session, so we won't prematurely inspect the app layer state.	15 years ago
Victor Julien	61a9936d55	Inspect a pseudo packet upon receiving a RST so that we are sure both sides of the TCP session are inspected.	15 years ago
Eric Leblond	e802e1ed16	Modify Packet structure and prepare accessor. ... This patch modifies decode.c and decode.h to avoid the usage by default of a bigger than 65535 bytes array in Packet structure. The idea is that the packet are mainly under 1514 bytes size and a bigger size must be supported but should not be the default. If the packet length is bigger than DFLT_PACKET_SIZE then the data are stored in a dynamically allocated part of the memory. To ease the modification of the rest of the code, functions to access and set the payload/length in a Packet have been introduced. The default packet size can be set at runtime via the default-packet-size configuration variable.	15 years ago
Victor Julien	1112e103a8	Disable DBG_PERF by default except for when DEBUG is enabled.	15 years ago
Anoop Saldanha	88d94b136d	Support for reference.config file	15 years ago
Victor Julien	fc248ca7a1	Many small performance updates.	15 years ago
Victor Julien	57b098e98c	Add padding to commonly used data structures.	15 years ago
Pablo Rincon	5c43db85ce	Drop streams on inline mode when a drop rule match from a reassembled stream and/or app layer inspection	15 years ago
Pablo Rincon	70bda6506d	Fix for bug 180 (check proto specified at the IP hdr)	15 years ago
Victor Julien	1fb11e939a	Improve configure messages. Make sure CUDA doesn't try to process packets that are too big.	15 years ago
Victor Julien	e14331cbb2	Fix PACKET_RECYCLE not cleaning all of the packet.	15 years ago
Victor Julien	3c1ae607cf	Fix cuda compilation.	15 years ago
Anoop Saldanha	33f4beb0bc	batching of packets support for cuda b2g mpm. Supported for both 32 and 64 bit platforms	15 years ago
Victor Julien	017b95f9ef	More thoroughly cleanup a Packet when we recycle it. Fixes a corner case where we'd have a invalid tcp packet but p->proto would still say IPPROTO_TCP because of a previous run. Fixes bug #187.	15 years ago
Victor Julien	580b09c2b8	Make sure we inspect all outstanding reassembled stream chunks (smsg) if the stream is shutting down. Make sure to do inspect signatures that use dsize against the tcp packet payload, even if that payload was already added to the stream. Likewise, the dsize signatures are not inspected against the reassembled stream.	15 years ago
Victor Julien	a3ff0e7210	Don't scan TCP packet payload if it was added to the stream. Inspect the tcp stream with the correct packet. Should fix #184 and #185.	15 years ago
Pablo Rincon	eed0ef6e69	Adding tag keyword support	15 years ago
Victor Julien	6519a86ec7	Move packet pool to ringbuffer, update packet pool api and ringbuffer api. Remove memset usage from PACKET_RECYCLE, add proper cleanup macros.	15 years ago
Victor Julien	37442a8a84	Prefilter signatures before fully scanning them.	15 years ago
Pablo Rincon	8cc525c939	UDP support at AppLayer message handling	15 years ago
Victor Julien	fdd0f3939e	Reduce size of event bit array in the packet structure.	15 years ago
Anoop Saldanha	f4120ff5d5	wrap multi line macros in do while	15 years ago
William Metcalf	2eef905c07	GPL and Copyright header updates.	15 years ago
Victor Julien	b8641f300d	Rename asn1 files, fix an invalid free, fix improper init of vars in one unittest.	15 years ago
Pablo Rincon	3fa3229e01	ASN1 decoder and keyword implementation	15 years ago
Victor Julien	bb685751d9	Fix NFQ receive/verdict race condition in cases where the packetpool is empty.	15 years ago
Victor Julien	e741bd0202	Cleanup packet recycling code. Fix issues in the packet tunnel/pseudo code.	15 years ago
Victor Julien	19584d0416	Fix tunneled and defrag reassembled packets with the new pending limits.	15 years ago
Victor Julien	46831e0f8f	Fix signature grouping bug for protocols without ports. Add debugging code.	15 years ago
Victor Julien	7a427ec7f4	Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach.	15 years ago
Victor Julien	a372c1d14e	Fix/workaround a strange detection issue.	15 years ago
Pablo Rincon	1238668961	Adding actions order and suport for rule action "pass"	15 years ago
Victor Julien	b90ebc1ce5	Add a packet count var for pcap file mode to the Packet structure to ease debugging.	15 years ago
William Metcalf	ce01927515	Import of GPLv2 Header 050410	15 years ago
Victor Julien	be5b547803	Convert DecodeSetNoPayloadInspectionFlag and DecodeSetNoPacketInspectionFlag to macro's.	15 years ago
Victor Julien	fe7ece997a	Different approach to the reference keyword. Lots of cleanups, bug fixes in reference keyword code and tests.	16 years ago
Breno Silva	89baf93a40	Reference Support	16 years ago
Victor Julien	dd846c9b0e	Remove all search code from the pattern matchers, cleanup mpm api, remove unused http code, more cleanups.	16 years ago
Anoop Saldanha	8cf60d6645	Changed the way cuda dispatcher passes back results. Now each detection thread has it's own queue to which the dispatcher can pump packets back to the detect thread. Also, with cuda enabled and a non-cuda mpm being used, we won't create a dispatcher and instead call the b2g scan/search funtions directly instead of using the dispatcher.	16 years ago
Anoop Saldanha	c26e92733d	handle the cuda cleanup at shutdown. should get rid of any errors from the call to SigGroupCleanup	16 years ago
Anoop Saldanha	41e6735b92	mpm b2g cuda support added	16 years ago
Breno Silva	b02bb6b6b4	VLAN Support	16 years ago
Jason Ish	e87d4f8a28	Split the defrag counters into ipv4 and ipv6.	16 years ago
Jason Ish	6b562f7aa6	Issue 82 - fragment counters. - number of fragments - number reassembled - number of timeouts	16 years ago
Pierre Chifflier	4515ae13e4	Add Prelude output plugin ... Add support for reporting alerts to the Prelude SIEM system, using libprelude to send IDMEF (RFC4765) messages. Each message contains the alert description and reference (using the SID/GID), and a normalized description (assessment, impact, sources etc.) libprelude handles the connection with the manager (collecting component), spooling and sending the event asynchronously. It also offers transport security (using TLS and trusted certificates) and reliability (events are retransmitted if not sent successfully). This modules requires a Prelude profile to work (see man prelude-admin and the Prelude Handbook for help). Signed-off-by: Pierre Chifflier <chifflier@edenwall.com>	16 years ago
Pablo Rincon	51dc773eec	Changing the veredict actions to flags to allow simultaneous veredict	16 years ago
Jason Ish	7d920a1254	Fix issue 65. ... - Update unit test to trigger the failure found in the issue 65 pcap. - Increase pkt buffer to account for the IPv6 header, as a maximum size IPv6 datagram is 40 + 0xffff. - Account for IPv4 header when checking where end of fragment lies. - Second sanity check during re-assembly to check for writing past the end of the pkt buffer.	16 years ago
Jason Ish	5076452707	Potential fix for issue 60. ... - Increase the packet buffer so it can hold the link header when we have maximum size IP datagrams. - Fix ip header length and pkt length calculation for re-assembled packets.	16 years ago
Eric Leblond	db2d483d11	convert action_type to enum ... This patch converts packet action type to an enum. This will provide some facilities and ease bad value detection by gcc.	16 years ago
William Metcalf	8a64321340	raw pcap support additionl ipv4/6 validation	16 years ago
Victor Julien	9b90c553b5	Clean ip fields from packet as well when the packet is reused. Prevents issues with malformed packets that are rejected by the decoders before ipaddresses are set.	16 years ago
Victor Julien	9ececacda3	Fix packet timestamp handling for encapsulated packets.	16 years ago
Anoop Saldanha	bc4df59414	Support for Classtype keyword and Classification Config file	16 years ago
Victor Julien	10cc9d5b6a	Add icmp flow handling.	16 years ago
Victor Julien	a8116f65c8	Fix packet flags field not being cleared properly when the packet is being reused. Add some debug statements and cleanup some.	16 years ago
Victor Julien	4824868766	Application layer detection improvements ... - improve locking of application layer handling, making sure that the flow cannot be freed/cleared when the detection engine is still working with it. - add a check to the app layer detection to make sure that a match function will only inspect an app layer state if it's of the right type.	16 years ago
Victor Julien	493715c0d2	Implement alert sid storage in the flow so we can check previous alerts in the flow.	16 years ago
Gerardo Iglesias	991d421394	Changed printf's to logging API functions	16 years ago
Pablo Rincon	e26833be3f	Changing mutex/spinlocks/conditions naming types	16 years ago
Pablo Rincon	769022f4be	Adding support for Mac OS X, FreeBSD, centrailizing mutex/spins/conditions in a macro API, and some unittests	16 years ago
Anoop Saldanha	8beef4a9fc	stats upgrade. Added interval counters to the decoder module	16 years ago
Pablo Rincon	292a7e47ef	ICMPv6 Decoder and unittests	16 years ago
Breno Silva	15a8f34d36	Gid Keyword ... Signed-off-by: Brian Rectanus <brectanu@gmail.com>	16 years ago
Gurvinder Singh	71da019734	some minor changes	16 years ago
Gurvinder Singh	bbb385422d	function to set packet flags	16 years ago
Gurvinder Singh	cc0b4f7181	initial version to support detection byepass	16 years ago
Victor Julien	db9e693054	Small fixes to pfring patch.	16 years ago
Victor Julien	a4fe971897	Fix datalink retrieval for pcap file mode and nfq mode for use in unified2.	16 years ago
Victor Julien	0d0ffb9963	Reorganize header inclusions.	16 years ago
Jamie	a297225157	victor must be getting sick of PPPoE and ICMP	16 years ago
Victor Julien	b6deadd2b4	Fixed a few missing places for tv->flags access. Changed mutex init for packets slightly.	16 years ago
William Metcalf	6095b8f2a1	fix for older libc's missing def for IPPROTO_DCCP	16 years ago
Anoop Saldanha	22c0ec2bc5	Added support for the csum-<protocol> rules keyword to the detection engine. Keywords added are ipv4-csum, tcpv4-csum, tcpv6-csum, udpv4-csum, udpv6-csum, icmpv4-csum and icmpv6-csum	16 years ago
Jamie	9adfe54620	more unit tests for pppoe - part I	16 years ago
Victor Julien	e7f8a02484	Improve GRE counting. Actually use the GRE decoder. Register the GRE counter.	16 years ago
Breno Silva	9528e02e46	GRE support	16 years ago
Victor Julien	57f71f7e4b	Pass the DecodeThreadVars to all Decoder functions properly. Improve the error handling.	16 years ago
Anoop Saldanha	244f5d547a	new registration functions for the stats api, with local thread storage for counter ids	16 years ago
Brian Rectanus	fa5939ca91	64 bit cleanup part2	16 years ago