399 Commits (f1117ba2dcdf5ef4a8fcd6ec010b6ea1bb98622f)

Author	SHA1	Message	Date
Victor Julien	215d0d54c7	detect: optimize rule address parsing ... Many rules have the same address vars, so instead of parsing them each time use a hash to store the string and the parsed result. Rules now reference the stored result in the hash table.	9 years ago
Victor Julien	99dce740ef	detect: mark alproto in keyword reg deprecated ... No existing code uses it, and it had been useless for some time.	9 years ago
Victor Julien	e43ce0a9ec	file: switch to streaming buffer API ... Make the file storage use the streaming buffer API. As the individual file chunks were not needed by themselves, this approach uses a chunkless implementation.	9 years ago
Victor Julien	46e55f1e34	http body handling: use streaming buffer API ... Convert HTTP body handling to use the Streaming Buffer API. This means the HtpBodyChunks no longer maintain their own data segments, but instead add their data to the StreamingBuffer instance in the HtpBody structure. In case the HtpBodyChunk needs to access it's data it can do so still through the Streaming Buffer API. Updates & simplifies the various users of the reassembled bodies: multipart parsing and the detection engine.	9 years ago
Mats Klepsland	a13df67864	detect: add (mpm) keyword for tls_sni ... Match on server name indication (SNI) extension in TLS using tls_sni keyword, e.g: alert tls any any -> any any (msg:"SNI test"; tls_sni; content:"example.com"; sid:12345;)	9 years ago
Justin Viiret	cce2d114e8	spm: add and use new SPM API ... This new API allows for different SPM implementations, using a function pointer table like that used for MPM. This change also switches over the paths that make use of DetectContentData (which previously used BoyerMoore directly) to the new API.	9 years ago
Justin Viiret	ce408c4d10	spm: add SinglePatternMatchDefaultMatcher ... Allows selecting SPM algorithm with the 'spm-algo' value in the YAML config file.	9 years ago
Victor Julien	f1ee125810	detect-flowvar: shrink mem structure by 8 bytes	9 years ago
Victor Julien	e2789a87c2	detect: shrink IPOnlyCIDRItem with 8 bytes	9 years ago
Victor Julien	adc453eec6	detect-port: improve comment about sgh pointer	9 years ago
Victor Julien	ef6f347f84	detect-address: remove sgh pointer as it's unused	9 years ago
Victor Julien	d6ba01b1b7	detect: make port whitelisting configurable ... Make the port grouping whitelisting configurable. A whitelisted port ends up in it's own port group. detect: grouping: tcp-whitelist: 80, 443 udp-whitelist: 53, 5060 No portranges are allowed at this point.	9 years ago
Victor Julien	fdd05e8fb4	mpm: remove unused app proto factory	9 years ago
Victor Julien	0311f01b97	rule grouping: speed up port based grouping ... Create a hash table of unique DetectPort objects before trying to create a unique list of these objects. This safes a lot of cycles in the creation of the list.	9 years ago
Victor Julien	87f3adbe4c	detect/mpm: unify packet/stream mpm_ctx pointers ... SGH's for tcp and udp are now always only per proto and per direction. This means we can simply reuse the packet and stream mpm pointers. The SGH's for the other protocols already used a directionless catch all mpm pointer.	9 years ago
Victor Julien	af3bf3dc7a	detect: optimize sgh layout	9 years ago
Victor Julien	de273d88cc	detect: remove unused content minlen tracking	9 years ago
Victor Julien	5857660568	http_uri: mpm cleanup. Use mpm_ctx's minlen	9 years ago
Victor Julien	e57e7d1b96	mpm: cleanup, remove unused structs and prototypes	9 years ago
Victor Julien	caea596ce5	profiling: output post-prefilter matches ... Dump a json record containing all sigs that need to be inspected after prefilter. Part of profiling. Only dump if threshold is met, which is currently set by: --set detect.profiling.inspect-logging-threshold=200 A file called packet_inspected_rules.json is created in the default log dir.	9 years ago
Victor Julien	722e2dbf7c	profiling: initial rulegroup tracking ... Per rule group tracking of checks, use of lists, mpm matches, post filter counts. Logs SGH id so it can be compared with the rule_group.json output. Implemented both in a human readable text format and a JSON format.	9 years ago
Victor Julien	e310a033be	detect: assign id to sgh	9 years ago
Victor Julien	c880b79f45	detect: shrink sgh ... Turn list of mpm_ctx pointers into a union so that we don't waste space. The sgh's for tcp and udp are in one direction only, so the ts and tc ones are now in the union.	9 years ago
Victor Julien	c804102a9a	detect: move app_mpms array to init data	9 years ago
Victor Julien	9e71ef4c3b	detect: remove signature pattern id reference	9 years ago
Victor Julien	c1ad08d11e	detect: remove stream pmq array	9 years ago
Victor Julien	7c94077892	detect mpm: remove unused mpm flags	9 years ago
Victor Julien	a7d126738a	detect address: remove unused features	9 years ago
Victor Julien	cbf80de6fe	detect-port: cleanup	9 years ago
Victor Julien	a96fa0fc2f	detect: remove unused dport sgh hash	9 years ago
Victor Julien	1f7e33a4b0	detect: remove unused flag	9 years ago
Victor Julien	e8c9598035	detect sgh: remove unused field	9 years ago
Victor Julien	a34be23002	detect: simplify negated mpm handling	9 years ago
Victor Julien	b84d6d402f	detect grouping: multiple whitelist conditions ... Instead of the binary yes/no whitelisting used so far, use different values for different sorts of whitelist reasons. The port list will be sorted by whitelist value first, then by rule count. The goal is to whitelist groups that have weak sigs: - 1 byte pattern groups - SYN sigs Rules that check for SYN packets are mostly scan detection rules. They will be checked often as SYN packets are very common. e.g. alert tcp any any -> any 22 (flags:S,12; sid:123;) This patch adds whitelisting for SYN-sigs, so that the sigs end up in as unique groups as possible. - negated mpm sigs Currently negated mpm sigs are inspected often, so they are quite expensive. For this reason, try to whitelist them. These values are set during 'stage 1', rule preprocessing.	9 years ago
Victor Julien	3c184c19cd	detect grouping: port based group whitelisting ... Whitelist some ports in grouping to make sure they get their own group.	9 years ago
Victor Julien	2ce03fbabb	detect: split non-mpm list into syn/nosyn ... Since SYN inspecting rules are expensive, this patch splits the 'non-mpm' list (i.e. the rules that are always considered) into a 'syn' and 'non-syn' list. The SYN list is only inspected if the packet has the SYN flag set, otherwise the non-syn list is used. The syn-list contains _all_ rules. The non-syn list contains all minus the rules requiring the SYN bit in a packet.	9 years ago
Victor Julien	e48d745ed7	mpm: constify search func args	9 years ago
Victor Julien	102a82fc7b	detect: use mpm store for app layer mpms ... Rework app-layer mpm setup and registration to make this possible.	9 years ago
Victor Julien	fac2cc0560	detect: mpm deduplication ... Create hash for mpm's that we can reuse. Have packet/stream mpms use this.	9 years ago
Victor Julien	f0ba00e51d	detect: remove old unused code	9 years ago
Victor Julien	2ee9bf2aef	detect: rename groupings vars	9 years ago
Victor Julien	1f70ccfc23	detect: remove unused grouping settings	9 years ago
Victor Julien	d82df4eb8b	detect-mpm: make sgh setup proto aware ... Allow multi-proto, multi-direction sgh's.	9 years ago
Victor Julien	fd5a06017d	detect: per port and proto rule grouping ... Replace tree based approach for rule grouping with a per port (tcp/udp) and per protocol approach. Grouping now looks like: +----+ |icmp+---> +----+ |gre +---> +----+ |esp +---> +----+ other|... | +----->-----+ | |N +---> | +----+ | | tcp +----+ +----+ +----->+ 80 +-->+ 139+--> | +----+ +----+ | | udp +----+ +----+ +---+----->+ 53 +-->+ 135+--> | +----+ +----+ |toserver +---> |toclient | +---> So the first 'split' in the rules is the direction: toserver or toclient. Rules that don't have a direction, are in both branches. Then the split is between tcp/udp and the other protocols. For tcp and udp port lists are used. For the other protocols, grouping is simply per protocol. The ports used are the destination ports for toserver sigs and source ports for toclient sigs.	9 years ago
Victor Julien	27e63a1e11	detect: track direction and ipproto of sgh ... Each SGH has a unique ipproto and direction.	9 years ago
Jason Ish	3c3fe4b47d	detect: denote the max detection list; fix issue 1674. ... Denotes the max detection list so that rule validation can allow post-detection lists to come after base64_data, but disallow detection lists to come after it.	9 years ago
Victor Julien	c0b16fa2bb	detect: allow for more than 64k mpm rules	10 years ago
Jason Ish	6b15686fd1	base64_decode, base64_data: decode and match base64	10 years ago
Victor Julien	6b2f831a70	mpm: SGH maxlen was actually minlen, so rename	10 years ago
Victor Julien	2716c78628	mpm: improve SGH content len tracking ... SGH's track content length for rule grouping. This patch changes the logic to only consider the pattern that is used in the mpm for a sig.	10 years ago