6627 Commits (bac37fc9ae5e3469652fda2ef268de617de485dd)

Author	SHA1	Message	Date
Victor Julien	9195708d58	detect analyzer: give minimal prefilter info	9 years ago
Victor Julien	065d9bceae	detect-dsize: enable prefilter support ... Enable prefilter support for the dsize keyword.	9 years ago
Victor Julien	9ccd0c0f90	prefilter: implement fragbits	9 years ago
Victor Julien	3b4aa06377	prefilter: engine for ack rules ... Rules for the 'ack' keyword are uncommon, but if used inspected against almost every packet.	9 years ago
Victor Julien	31ad0a133b	prefilter: engine for tcp flags keyword ... If there are many rules for TCP flags these rules would be inspected against each TCP packet. Even though the flags check is not expensive, the combined cost of inspecting multiple rules against each and every packet is high. This patch implements a prefilter engine for flags. If a rule group has rules looking for specific flags and engine for that flag or flags combination is set up. This way those rules are only inspected if the flag is actually present in the packet.	9 years ago
Victor Julien	8798bf48b2	profiling: support prefilter engines	9 years ago
Victor Julien	ea26ee906f	prefilter: intro common engine for u8 matches	9 years ago
Victor Julien	99b9896bd7	prefilter: common funcs for packet header prefilters	9 years ago
Victor Julien	f80623fd73	prefilter: show prefilter capability in --list-keywords	9 years ago
Victor Julien	56239690d0	prefilter: implement prefilter keyword ... Introduce prefilter keyword to force a keyword to be used as prefilter. e.g. alert tcp any any -> any any (content:"A"; flags:R; prefilter; sid:1;) alert tcp any any -> any any (content:"A"; flags:R; sid:2;) alert tcp any any -> any any (content:"A"; dsize:1; prefilter; sid:3;) alert tcp any any -> any any (content:"A"; dsize:1; sid:4;) In sid 2 and 4 the content keyword is used in the MPM engine. In sid 1 and 3 the flags and dsize keywords will be used.	9 years ago
Victor Julien	85cb749e8b	detect cleanup: remove sgh mpm_ctx pointers	9 years ago
Victor Julien	82d3c0b520	sgh: remove unused flags	9 years ago
Victor Julien	08407b6d47	tls: mpm prefilter engines	9 years ago
Victor Julien	7acdc66061	smtp file_data: mpm prefilter engine	9 years ago
Victor Julien	0019a7bd9f	http_raw_header: mpm prefilter engine ... Register for both regular headers and trailer.	9 years ago
Victor Julien	cef12ed80f	http_server_body / file_data: mpm prefilter engine	9 years ago
Victor Julien	5646dd9ecf	http_client_body: mpm prefilter engine	9 years ago
Victor Julien	9b6fd6bb48	http_headers: mpm prefilter engines ... Register for both regular headers and trailers.	9 years ago
Victor Julien	9cab3ea2cd	http_stat_code: mpm prefilter engine	9 years ago
Victor Julien	4d57b2fc63	http_stat_msg: mpm prefilter engine	9 years ago
Victor Julien	86d303e32b	http_raw_host: mpm prefilter engine	9 years ago
Victor Julien	5218849213	http_host: mpm prefilter engine	9 years ago
Victor Julien	61c3748fc4	http_user_agent: mpm prefilter engine	9 years ago
Victor Julien	a43a69305d	http_cookie: mpm prefilter engine	9 years ago
Victor Julien	7a46364e42	http_raw_uri: mpm prefilter engine	9 years ago
Victor Julien	746a169127	dns_query: mpm prefilter engine	9 years ago
Victor Julien	9ff5703c49	packet/stream: mpm prefilter engine	9 years ago
Victor Julien	72f2a78b1f	http_method: mpm prefilter engine	9 years ago
Victor Julien	b62c4cc359	http_uri: mpm prefilter engine ... Inspect partial request line as well.	9 years ago
Victor Julien	5bcdbe3922	prefilter: introduce prefilter engines ... Introduce abstraction layer for prefilter engines.	9 years ago
Victor Julien	3dad824fb2	detect: rename SignatureNonMpmStore ... New name is SignatureNonPrefilterStore to reflect that it's not just about MPM anymore.	9 years ago
Victor Julien	17bc0299fe	detect: rename non_mpm lists/vars to non_pf ... Rename to non_pf: non prefilter.	9 years ago
Victor Julien	bb0cd0e883	prefilter: rename PatternMatcherQueue datatype ... In preparation of the introduction of more general purpose prefilter engines, rename PatternMatcherQueue to PrefilterRuleStore. The new engines will fill this structure a similar way to the current mpm prefilters.	9 years ago
Victor Julien	4c0ab681f2	mpm: remove Cleanup API call ... It's unused by all of the implementations.	9 years ago
Victor Julien	7c47016913	detect-fragoffset: minor cleanup	9 years ago
Victor Julien	a41695f29f	uricontent: remove left over func decl	9 years ago
Victor Julien	ff70e0cca0	mpm tls: remove unused function args	9 years ago
Victor Julien	ad3a55d938	mpm dns query: remove unused function args	9 years ago
Victor Julien	d647db1775	mpm stat code: remove unused function args	9 years ago
Victor Julien	bd03307921	mpm stat msg: remove unused function args	9 years ago
Victor Julien	6d54b70db4	mpm ua: remove unused function args	9 years ago
Victor Julien	704afeb078	mpm cookie: remove unused function args	9 years ago
Victor Julien	4229e603f0	mpm raw host: remove unused function args	9 years ago
Victor Julien	1380853ee8	mpm host: remove unused function args	9 years ago
Victor Julien	b40ecb7356	mpm method: remove unused function args	9 years ago
Victor Julien	3d5807ba44	mpm raw uri: remove unused function args	9 years ago
Victor Julien	d461c7888a	mpm uri: remove unused function args	9 years ago
Victor Julien	c4dcb20522	detect-parse: add new func to get last sigmatch ... Add SigMatchGetLastSM which simply returns the very last SM added to the signature. Minor cleanups.	9 years ago
Eric Leblond	3ca663d7ff	output-json-flow: display bypass method ... In the case of a bypassed flow we add a 'bypass' key that can be 'local' or 'capture'. This will allow the user to know if capture bypass method is failing by looking at the 'bypass' key.	9 years ago
Giuseppe Longo	e6bac998d9	flow: add timeout for local bypass ... This adds a new timeout value for local bypassed state. For user simplication it is called only `bypassed`. The patch also adds a emergency value so we can clean bypassed flows a bit faster.	9 years ago
Eric Leblond	51bfe4960a	flow: discard packets belonging to bypassed flows	9 years ago
Eric Leblond	724069626d	flow: downgrade to local bypass if we see packets ... If we see packets for a capture bypassed flow after some times, it means that the capture method is not handling correctly the bypass so it is better to switch to local bypass method.	9 years ago
Eric Leblond	4cf887b4f7	flow: update lastts in FlowHandlePacketUpdate ... This allows to make it conditional to the state of packet and then trigger modified behavior.	9 years ago
Giuseppe Longo	5b71b5834f	filestore: avoid conflict with bypass keyword ... If a packet triggers a rule which contains both bypass and filestore keywords, it won't be stored since it's not inspected. To avoid that, when a rule containing filestore keyword we make sure that also bypass keyword is present.	9 years ago
Giuseppe Longo	07564c4e41	detect: add bypass keyword ... This adds a new keyword which permits to call the bypass callback when a sig is matched. The callback must be called when the match of the sig is complete.	9 years ago
Eric Leblond	c19cd12620	flow: bypass encrypted and after stream depth flow ... This patch activates bypass for encrypted flow and for flow that have reached stream depth on both side. For encrypted flow , suricata is stopping the inspection so we can just get it out via bypass. The same logic apply for flow that have reached the stream depth. For a basic test of feature, use the following ruleset: ``` table ip filter { chain output { type filter hook output priority 0; policy accept; ct mark 0x1 counter accept oif lo counter queue num 0 } chain connmark_save { type filter hook output priority 1; policy accept; mark 0x1 ct mark set mark counter ct mark 0x1 counter } } ``` And use bypass mark and mask of 1 in nfq configuration. Then you can test the system by scp big file to 127.0.0.1. You can also use iperf to measure the performance on localhost. It is recommended to lower the MTU to 1500 to get something more realistic by increasing the number of packets..	9 years ago
Giuseppe Longo	177df305d4	stream-tcp: enable bypass setting ... This permits to enable/disable in suricata.yaml and the bypass function will be called when stream.depth is reached.	9 years ago
Giuseppe Longo	97783f8142	nfq: introduce bypass function	9 years ago
Eric Leblond	285b4dd981	decode: implement bypass function ... Call the packet bypass callback if necessary and update the flow state. In case of failure we switch to local bypassed state and set capture bypassed state if the callback is successful.	9 years ago
Eric Leblond	68d9677eea	flow: force reassembly for bypassed flows ... As capture method like nfq will cut both side of the flow instantly we will not get the hack for most data which have been received. So it is better to force reassembly to be sure to get the timeout of the entry.	9 years ago
Eric Leblond	39c8786a8e	flow: get bypass info in get used flow function	9 years ago
Eric Leblond	07ef451c2b	flow: add pruned bypassed flow counter	9 years ago
Eric Leblond	745dad9809	flow: display info about bypass in log	9 years ago
Eric Leblond	e88555caf9	flow: add bypassed states ... This patch adds two new states to the flow: * local bypass: for suricata only bypass, packets belonging to a flow in this state will be discard fast * capture bypass: capture method is handling the bypass and suricata will discard packets that are currently queued A bypassed state to flow that will be set on flow when a bypass decision is taken. In the case of capture bypass this will allow to remove faster the flow entry from the flow table instead of waiting for the "established" timeout.	9 years ago
Giuseppe Longo	616782aa98	packet: add API for bypass	9 years ago
Jason Ish	1f4725fcab	detect-tls: make check on fingerprint directional	9 years ago
Jason Ish	44c846f2f8	tls-json: make tls events direction sensitive ... Previously the src/dest ips in TLS events would differ between IDS and IPS modes. Make the header creation direction sensitive so they are identical in both modes.	9 years ago
Mats Klepsland	c0f93503b7	util-decode-der-get: fix coverity warning ... *** CID 1373380: Control flow issues (DEADCODE) /src/util-decode-der-get.c: 126 in UtctimeToTime() 120 year = strtol(yy, NULL, 10); 121 if (year >= 50) 122 snprintf(buf, sizeof(buf), "%i%s", 19, utctime); 123 else if (year < 50) 124 snprintf(buf, sizeof(buf), "%i%s", 20, utctime); 125 else >>> CID 1373380: Control flow issues (DEADCODE) >>> Execution cannot reach this statement: "goto error;". 126 goto error; 127 128 time = GentimeToTime(buf); 129 if (time == -1) 130 goto error; 131	9 years ago
Victor Julien	d6f051cdf9	http: removed unused flags	9 years ago
Eric Leblond	a194dfbd5b	app-layer: tx counter implementation ... This patch adds a transaction counter for application layers supporting it. Analysis is done after the parsing by the different application layers. This result in new data in the stats output, that looks like: ``` "app-layer": { "tx": { "dns_udp": 21433, "http": 12766, "smtp": 0, "dns_tcp": 0 } }, ```	9 years ago
Giuseppe Longo	675fa56497	app-layer: add ThreadVars to AppLayerParserParse ... To be able to add a transaction counter we will need a ThreadVars in the AppLayerParserParse function. This function is massively used in unittests and this result in an long commit.	9 years ago
Giuseppe Longo	5908dd0804	app-layer: add flow counters ... This adds per flow counters for all supported protocols. This results in new data in stats output that looks like: ``` "app-layer": { "flow": { "http": 9310, "ftp": 0, "smtp": 0, "tls": 71, "ssh": 0, "imap": 0, "msn": 0, "smb": 170, "dcerpc_udp": 0, "dns_udp": 870, "dcerpc_tcp": 2, "dns_tcp": 0 }, }, ```	9 years ago
Eric Leblond	398489e6df	stream: fix depth reached detection ... When a segment only partially fit in streaming depth, the stream depth reached flag was not set resulting in a continuous inspection of the rest of the session. By setting the stream depth reached flag when the segment partially fit we avoid to reenter the code and we don't take anymore a code path resulting in the flag not to be set.	9 years ago
Mats Klepsland	dc8e0b3cf2	detect: add detect engine for tls validity keywords ... Add detect engine for tls validity keywords (tls_cert_notbefore and tls_cert_notafter).	9 years ago
Mats Klepsland	d91664d67a	detect-dns: move DetectEngineInspectGenericList to detect-engine.c ... Move DetectEngineInspectGenericList from detect-engine-dns.c to detect-engine.c to enable it to be used other places as well.	9 years ago
Mats Klepsland	cad638697d	lua: add lua functions for certificate validity dates ... Add functions TlsGetCertNotBefore and TLSGetCertNotAfter to get notBefore and notAfter fields from TLS certificate in lua scripts.	9 years ago
Mats Klepsland	67ea821521	util-lua: add (wrapper) function to push integer to lua scripts	9 years ago
Mats Klepsland	ee24949065	log-tls: add notBefore and notAfter fields to extended output ... Add notBefore and NotAfter fields from TLS certificate to extended tls log output.	9 years ago
Mats Klepsland	5b230bbce5	output-json-tls: add notBefore and notAfter fields to extended output ... Add notBefore and notAfter fields from TLS certificate to extended JSON output.	9 years ago
Mats Klepsland	ac4e308140	util-time: add function to create a UTC time string ... Add function CreateUtcIsoTimeString to create a UTC time string.	9 years ago
Mats Klepsland	ea5696812f	detect: add tls_cert_notbefore and tls_cert_notafter keywords ... Detection plugin for TLS certificate fields notBefore and notAfter. Supports equal to, less than, greater than, and range operations for both keywords. Dates can be represented as either ISO 8601 or epoch (Unix time). Examples: alert tls [...] tls_cert_notafter:1445852105; [...] alert tls [...] tls_cert_notbefore:<2015-10-22T23:59:59; [...] alert tls [...] tls_cert_notbefore:>2015-10-22; [...] alert tls [...] tls_cert_notafter:2000-10-22<>2020-05-15; [...]	9 years ago
Mats Klepsland	c49cb05399	util-time: add function to parse a date string based on patterns ... Add function SCStringPatternToTime to parse a date string based on an array of pattern strings.	9 years ago
Mats Klepsland	bfd16dc74e	app-layer-ssl: add validity dates from certificate ... Parsing of certificate validity dates to get notBefore and notAfter fields.	9 years ago
Mats Klepsland	6c1c53b5a1	util-time: add function to convert tm to time_t ... Add function SCMkTimeUtc to convert broken-down time to Unix epoch in UTC.	9 years ago
Mats Klepsland	03cda74b95	util-decode-der: decode GeneralizedTime ... Decode ASN.1 element type GeneralizedTime in DER-encoded structures.	9 years ago
Mats Klepsland	b914861692	app-layer-ssl: use new unit test macros	9 years ago
Mats Klepsland	12356d1fca	detect-ssl-version: use new unit test macros	9 years ago
Mats Klepsland	1503ac97a6	detect-tls-version: use new unit test macros	9 years ago
Mats Klepsland	d9e2cde585	detect-tls-sni: use new unit test macros	9 years ago
Mats Klepsland	8e77d0c312	detect: fix faulty tls_sni unittests	9 years ago
Mats Klepsland	9d23ad9d25	tls: fix faulty unittests	9 years ago
Mats Klepsland	b74f3fd978	coverty: fix CID 1361873	9 years ago
Mats Klepsland	c36595eb35	tls: set event if input buffer overflows ... Set HANDSHAKE_INVALID_LENGTH event if input buffer overflows while decoding client_hello/server_hello.	9 years ago
Mats Klepsland	1f7b813080	app-layer-tls: add name to authors	9 years ago
Mats Klepsland	12da0e8681	tls: add function for decoding client_hello ... Add function TLSDecodeHandshakeHello() to enable using the same code for decoding both client_hello and server_hello.	9 years ago
Jason Ish	04da43d65d	rule parsing: check for balanced double quotes ... If a rule option value starts with a double quote, ensure it ends with a double quote, exclusive of white space which gets trimmed anyways. Catches errors like 'filemagic:"picture" sid:5555555;' reporting that a missing semicolon may be the error.	9 years ago
Victor Julien	48b3cb0492	unittests: fix tests	9 years ago
Victor Julien	6530c3d0d8	unittests: replace SCMutex* calls by FLOWLOCK_*	9 years ago
Victor Julien	682459d640	file: remove dead code	9 years ago
Victor Julien	70c16f50e7	flow-manager: optimize hash walking ... Until now the flow manager would walk the entire flow hash table on an interval. It would thus touch all flows, leading to a lot of memory and cache pressure. In scenario's where the number of tracked flows run into the hundreds on thousands, and the memory used can run into many hundreds of megabytes or even gigabytes, this would lead to serious performance degradation. This patch introduces a new approach. A timestamp per flow bucket (hash row) is maintained by the flow manager. It holds the timestamp of the earliest possible timeout of a flow in the list. The hash walk skips rows with timestamps beyond the current time. As the timestamp depends on the flows in the hash row's list, and on the 'state' of each flow in the list, any addition of a flow or changing of a flow's state invalidates the timestamp. The flow manager then has to walk the list again to set a new timestamp. A utility function FlowUpdateState is introduced to change Flow states, taking care of the bucket timestamp invalidation while at it. Empty flow buckets use a special value so that we don't have to take the flow bucket lock to find out the bucket is empty. This patch also adds more performance counters: flow_mgr.flows_checked | Total | 929 flow_mgr.flows_notimeout | Total | 391 flow_mgr.flows_timeout | Total | 538 flow_mgr.flows_removed | Total | 277 flow_mgr.flows_timeout_inuse | Total | 261 flow_mgr.rows_checked | Total | 1000000 flow_mgr.rows_skipped | Total | 998835 flow_mgr.rows_empty | Total | 290 flow_mgr.rows_maxlen | Total | 2 flow_mgr.flows_checked: number of flows checked for timeout in the last pass flow_mgr.flows_notimeout: number of flows out of flow_mgr.flows_checked that didn't time out flow_mgr.flows_timeout: number of out of flow_mgr.flows_checked that did reach the time out flow_mgr.flows_removed: number of flows out of flow_mgr.flows_timeout that were really removed flow_mgr.flows_timeout_inuse: number of flows out of flow_mgr.flows_timeout that were still in use or needed work flow_mgr.rows_checked: hash table rows checked flow_mgr.rows_skipped: hash table rows skipped because non of the flows would time out anyway The counters below are only relating to rows that were not skipped. flow_mgr.rows_empty: empty hash rows flow_mgr.rows_maxlen: max number of flows per hash row. Best to keep low, so increase hash-size if needed. flow_mgr.rows_busy: row skipped because it was locked by another thread	9 years ago