934 Commits (8547d113bfe6b44f188ff72194c31e407d5ae42c)

Author	SHA1	Message	Date
Victor Julien	f370e88135	detect: move init only Signature members to init_data	9 years ago
Victor Julien	0a5ae415b8	detect: shrink Signature::sm_arrays ... Signature::sm_arrays now only contains 'built-in' lists, and so is sized appropriately.	9 years ago
Victor Julien	4978a7a133	detect: reorganize id's in prep of dynamic lists	9 years ago
Victor Julien	59303d1fbb	threshold: fix and redo tests	9 years ago
Victor Julien	6f7e4adbe8	detect: improve memory handling & comments	9 years ago
Victor Julien	8edc954e82	detect: get rid of Signature::sm_lists ... Instead use the lists in init_data during setup and the SigMatchData arrays during runtime.	9 years ago
Victor Julien	715ff60087	detect: remove unused SIG_FLAG_INIT_PAYLOAD init_flag	9 years ago
Victor Julien	bfd4bc8233	detect: constify Signature/SigMatch use at runtime	9 years ago
Victor Julien	86222428dd	detect: don't run IP inspection on non-IP packets ... The code to get the rule group (sgh) would return the group for IP proto 0 instead of nothing. This lead to certain types of rules unintentionally matching (False Positive). Since the packets weren't actually IP, the logged alert records were missing the IP header. Bug #2017.	9 years ago
Victor Julien	f91d490d25	detect: remove unused flow_locked hint	9 years ago
Victor Julien	810e43f373	magic: make optional ... Make libmagic optional. If installed it will be enabled by default in configure. Use --disable-libmagic to disable.	9 years ago
Victor Julien	3da7dad514	lua: luajit improvements ... Luajit has a strange memory requirement, it's 'states' need to be in the first 2G of the process' memory. This patch improves the pool approach by moving it to the front of the start up. A new config option 'luajit.states' is added to control how many states are preallocated. It defaults to 128. Add a warning when more states are used then preallocated. This may fail if flow/stream/detect engines use a lot of memory. Add hint at exit that gives the max states in use if it's higher than the default.	9 years ago
Victor Julien	1f670837ac	detect: add missing break (CID 1374301)	9 years ago
Jason Ish	1a31bded4a	DNP3: dnp3_data, dnp3_func, dnp3_ind, dnp3_obj rule keywords	9 years ago
Victor Julien	321fb6463e	vars: small cleanups	9 years ago
Victor Julien	0ed119068d	detect-app-layer-protocol: implement prefilter ... Introduce 'Protocol detection'-only rules. These rules will only be fully evaluated when the protocol detection completed. To allow mixing of the app-layer-protocol keyword with other types of matches the keyword can also inspect the flow's app-protos per packet. Implement prefilter for the 'PD-only' rules.	9 years ago
Victor Julien	43aed70976	detect: during detection sgh is read only so turn into const	9 years ago
Victor Julien	0e31124609	detect: add util func for post-inspect tasks on first sgh	9 years ago
Victor Julien	d3fb4de1b5	detect: move file flags update into it's own function	9 years ago
kwong	a3ffebd835	Adding SCADA EtherNet/IP and CIP protocol support ... Add support for the ENIP/CIP Industrial protocol This is an app layer implementation which uses the "enip" protocol and "cip_service" and "enip_command" keywords Implements AFL entry points	9 years ago
Victor Julien	8321f04ef3	prefilter: clean up setup code	9 years ago
Victor Julien	d36c0c15ea	detect: reshuffle keyword registration order ... The order of keyword registration currently affects inspect engine registration order and ultimately the order of inspect engines per rule. Which in turn affects state keeping. This patch makes sure the ordering is the same as with older releases.	9 years ago
Victor Julien	5f994756e6	detect-engine: improved inspect engines ... Inspect engines are called per signature per sigmatch list. Most wrap around DetectEngineContentInspection, but it's more generic. Until now, the inspect engines were setup in a large per ipproto, per alproto, per direction table. For stateful inspection each engine needed a global flag. This approach had a number of issues: 1. inefficient: each inspection round walked the table and then checked if the inspect engine was even needed for the current rule. 2. clumsy registration with global flag registration. 3. global flag space was approaching the need for 64 bits 4. duplicate registration for alprotos supporting both TCP and TCP (DNS). This patch introduces a new approach. First, it does away with the per ipproto engines. This wasn't used. Second, it adds a per signature list of inspect engine containing only those engines that actually apply to the rule. Third, it gets rid of the global flags and replaces it with flags assigned per rule per engine.	9 years ago
Victor Julien	f1e3840516	http_response_body: implement keyword with mpm ... Implemented as 'stickybuffer'.	9 years ago
Victor Julien	4c98b6cef3	http_request_line: implement keyword and mpm ... Implemented as 'stickybuffer'. Move all logic into the keyword file and remove bad tests that tested URI instead of request line.	9 years ago
Victor Julien	5b2e36a1b0	mpm: add App Layer MPM registery ... Register keywords globally at start up. Create a map of the registery per detection engine. This we need because the sgh_mpm_context value is set per detect engine. Remove APP_MPMS_MAX.	9 years ago
Victor Julien	e1eb481647	prefilter: cleanup and optimization	9 years ago
Victor Julien	dba14b676c	profiling: more prefilter profiling	9 years ago
Victor Julien	125603871b	detect: config opt to enable keyword prefilters	9 years ago
Victor Julien	9bb12ccb27	prefilter: move payload engines into separate list	9 years ago
Victor Julien	14b0537f95	prefilter: implement basic prefilter priority order	9 years ago
Victor Julien	8798bf48b2	profiling: support prefilter engines	9 years ago
Victor Julien	f80623fd73	prefilter: show prefilter capability in --list-keywords	9 years ago
Victor Julien	56239690d0	prefilter: implement prefilter keyword ... Introduce prefilter keyword to force a keyword to be used as prefilter. e.g. alert tcp any any -> any any (content:"A"; flags:R; prefilter; sid:1;) alert tcp any any -> any any (content:"A"; flags:R; sid:2;) alert tcp any any -> any any (content:"A"; dsize:1; prefilter; sid:3;) alert tcp any any -> any any (content:"A"; dsize:1; sid:4;) In sid 2 and 4 the content keyword is used in the MPM engine. In sid 1 and 3 the flags and dsize keywords will be used.	9 years ago
Victor Julien	08407b6d47	tls: mpm prefilter engines	9 years ago
Victor Julien	7acdc66061	smtp file_data: mpm prefilter engine	9 years ago
Victor Julien	0019a7bd9f	http_raw_header: mpm prefilter engine ... Register for both regular headers and trailer.	9 years ago
Victor Julien	cef12ed80f	http_server_body / file_data: mpm prefilter engine	9 years ago
Victor Julien	5646dd9ecf	http_client_body: mpm prefilter engine	9 years ago
Victor Julien	9b6fd6bb48	http_headers: mpm prefilter engines ... Register for both regular headers and trailers.	9 years ago
Victor Julien	9cab3ea2cd	http_stat_code: mpm prefilter engine	9 years ago
Victor Julien	4d57b2fc63	http_stat_msg: mpm prefilter engine	9 years ago
Victor Julien	86d303e32b	http_raw_host: mpm prefilter engine	9 years ago
Victor Julien	5218849213	http_host: mpm prefilter engine	9 years ago
Victor Julien	61c3748fc4	http_user_agent: mpm prefilter engine	9 years ago
Victor Julien	a43a69305d	http_cookie: mpm prefilter engine	9 years ago
Victor Julien	7a46364e42	http_raw_uri: mpm prefilter engine	9 years ago
Victor Julien	746a169127	dns_query: mpm prefilter engine	9 years ago
Victor Julien	9ff5703c49	packet/stream: mpm prefilter engine	9 years ago
Victor Julien	72f2a78b1f	http_method: mpm prefilter engine	9 years ago
Victor Julien	b62c4cc359	http_uri: mpm prefilter engine ... Inspect partial request line as well.	9 years ago
Victor Julien	5bcdbe3922	prefilter: introduce prefilter engines ... Introduce abstraction layer for prefilter engines.	9 years ago
Victor Julien	17bc0299fe	detect: rename non_mpm lists/vars to non_pf ... Rename to non_pf: non prefilter.	9 years ago
Victor Julien	ff70e0cca0	mpm tls: remove unused function args	9 years ago
Victor Julien	ad3a55d938	mpm dns query: remove unused function args	9 years ago
Victor Julien	d647db1775	mpm stat code: remove unused function args	9 years ago
Victor Julien	bd03307921	mpm stat msg: remove unused function args	9 years ago
Victor Julien	6d54b70db4	mpm ua: remove unused function args	9 years ago
Victor Julien	704afeb078	mpm cookie: remove unused function args	9 years ago
Victor Julien	4229e603f0	mpm raw host: remove unused function args	9 years ago
Victor Julien	1380853ee8	mpm host: remove unused function args	9 years ago
Victor Julien	b40ecb7356	mpm method: remove unused function args	9 years ago
Victor Julien	3d5807ba44	mpm raw uri: remove unused function args	9 years ago
Victor Julien	d461c7888a	mpm uri: remove unused function args	9 years ago
Giuseppe Longo	07564c4e41	detect: add bypass keyword ... This adds a new keyword which permits to call the bypass callback when a sig is matched. The callback must be called when the match of the sig is complete.	9 years ago
Giuseppe Longo	675fa56497	app-layer: add ThreadVars to AppLayerParserParse ... To be able to add a transaction counter we will need a ThreadVars in the AppLayerParserParse function. This function is massively used in unittests and this result in an long commit.	9 years ago
Mats Klepsland	ea5696812f	detect: add tls_cert_notbefore and tls_cert_notafter keywords ... Detection plugin for TLS certificate fields notBefore and notAfter. Supports equal to, less than, greater than, and range operations for both keywords. Dates can be represented as either ISO 8601 or epoch (Unix time). Examples: alert tls [...] tls_cert_notafter:1445852105; [...] alert tls [...] tls_cert_notbefore:<2015-10-22T23:59:59; [...] alert tls [...] tls_cert_notbefore:>2015-10-22; [...] alert tls [...] tls_cert_notafter:2000-10-22<>2020-05-15; [...]	9 years ago
Victor Julien	6530c3d0d8	unittests: replace SCMutex* calls by FLOWLOCK_*	9 years ago
Duarte Silva	89eb935f73	file-hashing: added support for SHA-256 file hashing	9 years ago
Duarte Silva	a6d928e269	file-hashing: added support for SHA-1 file hashing	9 years ago
Mats Klepsland	4172c4c8ac	tls: add (mpm) keyword tls_cert_subject ... This keyword is a replacement for tls.subject.	9 years ago
Mats Klepsland	9b2717799c	tls: add (mpm) keyword tls_cert_issuer ... This keyword is a replacement for tls.issuerdn.	9 years ago
Victor Julien	99dce740ef	detect: mark alproto in keyword reg deprecated ... No existing code uses it, and it had been useless for some time.	9 years ago
Victor Julien	519b2970ec	detect: don't print (null) in --list-keywords=all	9 years ago
Victor Julien	6b078e4f51	detect: fix ICMP error handling issue ... The first packet in both directions of a flow looks up the rule group (sgh) and stores it in the flow. This makes sure the lookup doesn't have to be performed for each packet. ICMPv4 error messages are connected to the TCP or UDP flow they apply to. In the case of such an ICMP error being the first packet in a flow's direction, this would lead to an issue. The packet would look up the rule group based on the ICMP protocol, not based on the embedded TCP/UDP. This makes sense, as the ICMP packet is inspected as ICMP packet. The consequence however, was that this rule group pointer (sgh) would be stored in the flow. This is wrong, as TCP/UDP packets that follow the ICMP packet would have no sgh or the wrong sgh. In normal traffic this shouldn't normally happen, but it could be used to evade Suricata's inspection.	9 years ago
Victor Julien	305b1b90fd	detect: minor cleanup	9 years ago
Victor Julien	b79d9cda3a	detect: minor cleanups	9 years ago
Victor Julien	6045420812	detect: reduce verbosity, don't warn on empty files	9 years ago
Victor Julien	b3bf7a5729	output: introduce config and perf output levels ... Goal is to reduce info output	9 years ago
Victor Julien	4b9a62d1fe	profiling: fix compilation if libjansson is missing	9 years ago
Victor Julien	a8f257e05f	detect: no longer a thread module ... Like stream, detect is now invoked directly by the FlowWorker.	9 years ago
Victor Julien	52d500c670	flowworker: initial support ... Initial version of the 'FlowWorker' thread module. This module combines Flow handling, TCP handling, App layer handling and Detection in a single module. It does all flow related processing under a single flow lock.	9 years ago
Victor Julien	408948815f	detect: simplify flow locking ... To simplify locking, move all locking out of the individual detect code. Instead at the start of detection lock the flow, and at the end of detection unlock it. The lua code can be called without a lock still (from the output code paths), so still pass around a lock hint to take care of this.	9 years ago
Victor Julien	a81766c046	detect: split detect entry into flow/noflow ... This is a preparation for flow locking updates.	9 years ago
Mats Klepsland	a13df67864	detect: add (mpm) keyword for tls_sni ... Match on server name indication (SNI) extension in TLS using tls_sni keyword, e.g: alert tls any any -> any any (msg:"SNI test"; tls_sni; content:"example.com"; sid:12345;)	9 years ago
Mats Klepsland	e4f03b18ba	tls: make TX aware	9 years ago
Zachary Rasmor	f211fa48f4	Update unit test to account for 'name' type change.	9 years ago
maxtors	9d3fd82849	Removed duplicate include statements.	9 years ago
Jason Ish	796dd5223b	tests: no longer necessary to provide successful return code ... 1 pass, 0 is fail.	9 years ago
Victor Julien	2ee3e726a9	detect: don't set unused variable ... detect.c:3801:13: warning: Value stored to 'tmplist2_tail' is never read tmplist2_tail = joingr; ^ ~~~~~~ detect.c:3804:13: warning: Value stored to 'tmplist2_tail' is never read tmplist2_tail = joingr; ^ ~~~~~~ 2 warnings generated.	9 years ago
Victor Julien	d5c6d08bc8	detect-port: remove debug mem counters	9 years ago
Victor Julien	9bd8197009	detect-address: remove debug mem counters	9 years ago
Victor Julien	e0111fbb90	detect grouping: remove debug mem counters	9 years ago
Victor Julien	5f676167a3	detect grouping: make json dump configurable ... Make the rule grouping dump to rule_group.json configurable. detect: profiling: grouping: dump-to-disk: false include-rules: false # very verbose include-mpm-stats: false	9 years ago
Victor Julien	d6ba01b1b7	detect: make port whitelisting configurable ... Make the port grouping whitelisting configurable. A whitelisted port ends up in it's own port group. detect: grouping: tcp-whitelist: 80, 443 udp-whitelist: 53, 5060 No portranges are allowed at this point.	9 years ago
Victor Julien	ac2c206359	mpm: clean up builtin mpm setup, enable single/full	9 years ago
Victor Julien	6ef27c9f92	mpm: allow app buffer shared/unique ... Allow setting of shared or unique setting per app buffer type: e.g. detect.mpm.http_uri.shared=true	9 years ago
Victor Julien	79a96b2b90	mpm: refactor 'single' setup handling	9 years ago
Victor Julien	157ca89dd7	mpm: remove useless flag from factory	9 years ago
Victor Julien	fdd05e8fb4	mpm: remove unused app proto factory	9 years ago
Victor Julien	2b84387ea4	detect: work around cocci limitation	9 years ago
Victor Julien	0311f01b97	rule grouping: speed up port based grouping ... Create a hash table of unique DetectPort objects before trying to create a unique list of these objects. This safes a lot of cycles in the creation of the list.	9 years ago
Victor Julien	30755265ee	http_raw_header: improve mpm progress handling	9 years ago
Victor Julien	de273d88cc	detect: remove unused content minlen tracking	9 years ago
Victor Julien	5857660568	http_uri: mpm cleanup. Use mpm_ctx's minlen	9 years ago
Victor Julien	caea596ce5	profiling: output post-prefilter matches ... Dump a json record containing all sigs that need to be inspected after prefilter. Part of profiling. Only dump if threshold is met, which is currently set by: --set detect.profiling.inspect-logging-threshold=200 A file called packet_inspected_rules.json is created in the default log dir.	9 years ago
Victor Julien	92c571b26d	detect: move sm_list to string funcs to parser code	9 years ago
Victor Julien	722e2dbf7c	profiling: initial rulegroup tracking ... Per rule group tracking of checks, use of lists, mpm matches, post filter counts. Logs SGH id so it can be compared with the rule_group.json output. Implemented both in a human readable text format and a JSON format.	9 years ago
Victor Julien	e310a033be	detect: assign id to sgh	9 years ago
Victor Julien	ba9d43cce5	mpm: improve negated mpm ... The idea is: if mpm is negated, it's both on mpm and nonmpm sid lists and we can kick it out in that case during the merge sort. It only works for patterns that are 'independent'. This means that the rule doesn't need to only match if the negated mpm pattern is limited to the first 10 bytes for example. Or more generally, an negated mpm pattern that has depth, offset, distance or within settings can't be handled this way. These patterns are not added to the mpm at all, but just to to non-mpm list. This makes sense as they will *always* need manual inspection. Similarly, a pattern that is 'chopped' always needs validation. This is because in this case we only inspect a part of the final pattern.	9 years ago
Victor Julien	c1ad08d11e	detect: remove stream pmq array	9 years ago
Victor Julien	c87fcb29ff	detect mpm: fast_pattern assignment cleanup	9 years ago
Victor Julien	a96fa0fc2f	detect: remove unused dport sgh hash	9 years ago
Victor Julien	e6248b0dbe	detect: clean up sgh's at detect engine free	9 years ago
Victor Julien	1f7e33a4b0	detect: remove unused flag	9 years ago
Victor Julien	2ab20d0b9b	detect: free lookup structures	9 years ago
Victor Julien	4ba1ac55f0	detect: output sgh stats ... Output stats for the rule groups into a json format.	9 years ago
Victor Julien	810d2d3ec6	detect: add list id to string funcs	9 years ago
Victor Julien	a34be23002	detect: simplify negated mpm handling	9 years ago
Victor Julien	b84d6d402f	detect grouping: multiple whitelist conditions ... Instead of the binary yes/no whitelisting used so far, use different values for different sorts of whitelist reasons. The port list will be sorted by whitelist value first, then by rule count. The goal is to whitelist groups that have weak sigs: - 1 byte pattern groups - SYN sigs Rules that check for SYN packets are mostly scan detection rules. They will be checked often as SYN packets are very common. e.g. alert tcp any any -> any 22 (flags:S,12; sid:123;) This patch adds whitelisting for SYN-sigs, so that the sigs end up in as unique groups as possible. - negated mpm sigs Currently negated mpm sigs are inspected often, so they are quite expensive. For this reason, try to whitelist them. These values are set during 'stage 1', rule preprocessing.	9 years ago
Victor Julien	3c184c19cd	detect grouping: port based group whitelisting ... Whitelist some ports in grouping to make sure they get their own group.	9 years ago
Victor Julien	5772f526dc	detect grouping: warn on and fix up bad sigs ... Only inspect directionless SYN scan sigs toserver. Issue a warning for those rules.	9 years ago
Victor Julien	2ce03fbabb	detect: split non-mpm list into syn/nosyn ... Since SYN inspecting rules are expensive, this patch splits the 'non-mpm' list (i.e. the rules that are always considered) into a 'syn' and 'non-syn' list. The SYN list is only inspected if the packet has the SYN flag set, otherwise the non-syn list is used. The syn-list contains _all_ rules. The non-syn list contains all minus the rules requiring the SYN bit in a packet.	9 years ago
Victor Julien	26517b8b61	detect: mpm store frees mpm_ctx' it owns	9 years ago
Victor Julien	102a82fc7b	detect: use mpm store for app layer mpms ... Rework app-layer mpm setup and registration to make this possible.	9 years ago
Victor Julien	fac2cc0560	detect: mpm deduplication ... Create hash for mpm's that we can reuse. Have packet/stream mpms use this.	9 years ago
Victor Julien	f0ba00e51d	detect: remove old unused code	9 years ago
Victor Julien	2ee9bf2aef	detect: rename groupings vars	9 years ago
Victor Julien	ae80ed5964	detect: make port grouping use config limits	9 years ago
Victor Julien	df529b13ce	detect: change port grouping ... Update port grouping logic. Previously it would create one consistent list w/o overlap. It largely still does this, except for the 'catch all' port group at the end of the list. This port group contains all the sigs that didn't fit into the other groups.	9 years ago
Victor Julien	a392812302	detect: sort/group port sigs	9 years ago
Victor Julien	e570b10abe	detect: display unique sgh count	9 years ago
Victor Julien	eda9552e95	detect: group proto sghs	9 years ago
Victor Julien	d82df4eb8b	detect-mpm: make sgh setup proto aware ... Allow multi-proto, multi-direction sgh's.	9 years ago
Victor Julien	4223ce9aba	detect: remove obsolete grouping code	9 years ago
Victor Julien	9ae4cb9e02	detect: debug output	9 years ago
Victor Julien	fd5a06017d	detect: per port and proto rule grouping ... Replace tree based approach for rule grouping with a per port (tcp/udp) and per protocol approach. Grouping now looks like: +----+ |icmp+---> +----+ |gre +---> +----+ |esp +---> +----+ other|... | +----->-----+ | |N +---> | +----+ | | tcp +----+ +----+ +----->+ 80 +-->+ 139+--> | +----+ +----+ | | udp +----+ +----+ +---+----->+ 53 +-->+ 135+--> | +----+ +----+ |toserver +---> |toclient | +---> So the first 'split' in the rules is the direction: toserver or toclient. Rules that don't have a direction, are in both branches. Then the split is between tcp/udp and the other protocols. For tcp and udp port lists are used. For the other protocols, grouping is simply per protocol. The ports used are the destination ports for toserver sigs and source ports for toclient sigs.	9 years ago
Victor Julien	27e63a1e11	detect: track direction and ipproto of sgh ... Each SGH has a unique ipproto and direction.	9 years ago
Victor Julien	e75a93b125	detect: pass ipproto to rule grouping funcs	9 years ago
Victor Julien	c71c991669	detect: delay sgh cleanup	9 years ago
Victor Julien	bb662a65f8	detect: delay sgh mpm setup	9 years ago
Victor Julien	18dd54dfa7	Start rule inspect with mask check	9 years ago
Victor Julien	66b3dba676	detect: remove dead code	9 years ago
Victor Julien	4f8e1f59a6	mpm: remove obsolete mpm algos ... Remove: ac-gfbs, wumanber, b2g, b3g.	9 years ago
Victor Julien	8394b38941	cppcheck: work around snprintf warning ... Cppcheck 1.72 gives a warning on the following code pattern: char blah[32] = ""; snprintf(blah, sizeof(blah), "something"); The warning is: (error) Buffer is accessed out of bounds. While this appears to be a FP, in most cases the initialization to "" was unnecessary as the snprintf statement immediately follows the variable declaration.	9 years ago
Victor Julien	a74a8c874f	smtp: reset inspection buffers	10 years ago
Victor Julien	d7d76e7b27	multi-detect: use default tenant ... The default detect engine can be used as 'default tenant'.	10 years ago
Jason Ish	6b15686fd1	base64_decode, base64_data: decode and match base64	10 years ago
Victor Julien	6b2f831a70	mpm: SGH maxlen was actually minlen, so rename	10 years ago
Victor Julien	2716c78628	mpm: improve SGH content len tracking ... SGH's track content length for rule grouping. This patch changes the logic to only consider the pattern that is used in the mpm for a sig.	10 years ago