Commit Graph

354 Commits (e75a93b1256e604d2528baff76b9c236057dba64)

Author SHA1 Message Date
Victor Julien 575c87aeba engine events: prefilter sigs that need a event 13 years ago
Victor Julien c91c359692 profiling: fix build on older systems 13 years ago
Victor Julien ec7e79c748 Rule profiling update
- Remove usage of counters api.
- Store stats in detect engine thread ctx to remove locking
- Support rule reloads
13 years ago
Victor Julien ba3260ed38 Thread local ctx for detection keywords
Some detection keywords need thread local ctx storage. Example is the
filemagic keyword that has a ctx that is modified with each call. That
is not thread safe. This functionality allows registration of thread
local ctxs so that each detect thread works on it's own copy.
13 years ago
Victor Julien f58e828c5e luajit: stub detection keyword 13 years ago
Eric Leblond 71b4257bc2 sig: add l3_proto keyword
This patch adds a l3_proto keyword to the signature language. It
can be used to specify if the signature has to match on IPv4, IPv6
or both. For example, one can write:
  alert http any any -> any 22 (msg: "HTTP v6"; l3_proto:ip6; sid:14;)

This should close #494.
13 years ago
Victor Julien 8f71333e12 file: implement filesize keyword. #489. 13 years ago
Victor Julien 3849588c61 Create separate detect API call (FileMatch) for file detection keywords. #531. 13 years ago
Eric Leblond 7e09cdc265 Delay Detect threads initialization
This patch modifies the init of Detect threads. They are now started
with a dummy function and their initialisation is done after the
signatures are loaded. Just after this, the dummy function is switched
to normal one.

In IPS mode, this permit to route packets without waiting for the
signature to start and should fix #488.

Offline mode such as pcap file don't use this mode to be sure to
analyse all packets in the file.

The patch introduces a "delayed-detect" configuration variable
under detect-engine. It can be used to activate the feature
(set to "yes" to have signature loaded after capture is started).
13 years ago
Jean-Paul Roliers c4df7a45ae tls: adding store option for TLS
This patch adds a TLS store option to save certificate in PEM format.
Each time the store action is met, a file and a metafile are created.

Reworked-by: Eric Leblond <eric@regit.org>
13 years ago
Jean-Paul Roliers 00d4357362 tls: adding support for fingerprint rule matching.
Add the support for tls.fingerprint keyword in rules.
13 years ago
Victor Julien c9e93ec52c filemd5: add support code for md5 handling for signatures. 13 years ago
Anoop Saldanha 0c24bbab0c code cleanup for live swap 13 years ago
Anoop Saldanha ecad4a24fa live rule support added
To reload ruleset during engine runtime, send the USR2 signal to the engine, and the ruleset would be reloaded from the same yaml file supplied at engine startup
13 years ago
Anoop Saldanha 5878d83174 byte_extract_id var now a non-global de_ctx specific var 13 years ago
Anoop Saldanha f4ce9011d2 make mpm ctx container de_ctx specific. Also introduce global variable in mpm_ctx. this is a workaround for cleaning non global mpm_ctx's since we now don't supply the de_ctx around the detection engine API 13 years ago
Anoop Saldanha f2dd61868d variable names global vars, global no more. Moved to detection engine ctx, a place it belongs 13 years ago
Anoop Saldanha f9612f3b83 mpm engine cleanup. Remove unnecessary flags 13 years ago
Anoop Saldanha 988c92f71c http user agent keyword + mpm + inspection + fast pattern support added 13 years ago
Victor Julien 9f7588a756 Add filemd5 keyword that loads a list of md5's to match a file's md5 against. 13 years ago
Victor Julien cae46ab5eb pcre: print filename and line number for JIT warning. 13 years ago
Anoop Saldanha 493c3db413 fix FNs for flow- only_stream and no_stream options 13 years ago
Victor Julien d378b76c04 http: body inspection improvement
Improve http_client_body and file_data performance when request and
response body limits are set to high values.
13 years ago
Eric Leblond ad0e05a112 TLS app layer: Add tls.issuerdn keyword. 14 years ago
Pierre Chifflier 4be65fd016 TLS handshake: decode the SERVER_CERTIFICATE message
Add a decoder for the SERVER_CERTIFICATE during a TLS handshake, extracts the
certificates and keep the subject name.
Add the tls.subject keyword for substring match in rules (TLS layer).

Signed-off-by: Pierre Chifflier <pierre.chifflier@ssi.gouv.fr>
14 years ago
Victor Julien c0a2cbd478 Move over src and dst thresholding to use host table. Fix a bug in threshold 'both' handling. 14 years ago
Victor Julien a05df345de Introduce host table, make tag use it
Add a host table similar to the flow table. A hash using fine grained
locking. Flow manager for now takes care of book keeping / garbage
collecting.

Tag subsystem now uses this for host based tagging instead of the
global tag hash table. Because the latter used a global lock and the
new code uses very fine grained locking this patch should improve
scalability.
14 years ago
Anoop Saldanha e682796d03 feature #414 - support listing supported keywords. Remove support for dummy keywords __address__, __proto__, __port__. Remove support for recursive keyword and all references to it 14 years ago
Anoop Saldanha 603d4a719a remove det_ctx->payload_offset and use det_ctx->buffer_offset. Update hscd and hsmd to use the new generic content inspection engine 14 years ago
Anoop Saldanha 35f1f7e8d9 unify payload detection engines + fix other bugs in pcre init 14 years ago
Anoop Saldanha 09313cf9bd Support http stat code detection engine, fast pattern(mpm engine included). Fix http stat code setup function. Fix pcre option for stat msg keyword. With this the pcre options for server_body is Q, for stat_msg is Y and for stat_code is S 14 years ago
Anoop Saldanha 2007c2711c Support http stat msg detection engine, fast pattern(mpm engine included). Fix http stat msg setup function. Fix pcre option for stat msg keyword 14 years ago
Anoop Saldanha 419cdc8558 support splitting mpm ctxs based on direction v2 14 years ago
Victor Julien c8c4a76dc6 Move threshold to it's own sig match list. 14 years ago
Victor Julien 6eeab37ab3 Add post-match list, move flowbits set, etc functions to it. Move flowint set, etc functions to it as well. 14 years ago
Victor Julien 21ee59e6f3 Add signature direction (flow:toserver/flow:toclient) as a signature flag. 14 years ago
Anoop Saldanha eea5ab4a7a Support for app layer decoder events added + app_layer_event keyword added 14 years ago
Anoop Saldanha 92643f6110 introduce separate mpm ctxs for tcp/udp/other_protos 14 years ago
Victor Julien 298289f43f Let flow:only_stream and flow:no_stream set the require packet and require stream flags. Toss out sigs with conflicting settings. Rename flow:stream_only to flow:only_stream. Fixes #261. 14 years ago
Victor Julien 18d79c4215 file store: respect flowbits and other keywords
The filestore keyword until now flagged a file, tx or ssn for storage as soon
as the keyword was inspected. This happens before flowbits and some other
keywords, so files were stored that weren't supposed to.

This patch makes the filestore keyword fill an array in the detect engine
thread ctx. Then if the full signature matches, a post-match filestore
function makes the store final.
14 years ago
Victor Julien 07e560b137 file-data: initial file_data support
Support file_data for: content, pcre (relative), byte_test, byte_jump,
byte_extract, isdataat.

File_data support is handled at signature parsing time, all matches
occurring after the file_data in the rule are converted to http_server_body
matches.

Content matches relative to the file_data are converted. Within to depth,
distance to offset. Relative to the start of the body buffer.
14 years ago
Victor Julien 7adac3048d file-data: create initial keyword registration. 14 years ago
Victor Julien fa0152fa80 Shrink signature flags field to 32 bits. 14 years ago
Victor Julien dd9da1a56f Merge all http mpm related signature flags into a single set: SIG_FLAG_MPM_HTTP and SIG_FLAG_MPM_HTTP_NEG. 14 years ago
Victor Julien d5ed28b065 Remove SIG_FLAG_MPM flag. 14 years ago
Victor Julien fe48920514 Remove per sgh mpm_streamcontent_maxlen variable. It was checked but never set. 14 years ago
Victor Julien 4992f7c417 Remove SIG_FLAG_MPM_URI flag. It was checked but never set. 14 years ago
Victor Julien 2650551192 Rename signature init flags to indicate they are init flags. 14 years ago
Victor Julien 6ebd71545b Fix signature flag definitions on 32 bit. 14 years ago
Victor Julien 291ddd95f2 Detection engine -- mpm
Each signature is in one mpm ctx at max, but there were 3 separate
id's in use: packet, stream, http. Merged them all into one.

Could shrink the SignatureHeader structure with 8 bytes because of this,
should lead to better caching performance.
14 years ago
Victor Julien 7db72bce75 Optimize detection engine prefiltering logic. 14 years ago
Victor Julien 89f83e714c Introduce http_server_body keyword.
The http_server_body content modifier modifies the previous content to inspect
the normalized (dechunked, unzipped) http_server_body. The workings are similar
to http_client_body. Additionally, a new pcre flag was introduced "/S".

To facilitate this change the signature flags field was changed to be 64 bit.
14 years ago
Victor Julien 04ea70ccf7 file extract: pruning
Add pruning of files in memory so we keep only memory what we really need.
Fix magic logic.
Reset file part of the de_state on receiving another file in the same tx.
14 years ago
Victor Julien 9b62ec65ab Make sure filemagic works properly regardless of filestore being in use for a flow. 14 years ago
Victor Julien 5945e652d6 Initial implementation of filemagic keyword. 14 years ago
Victor Julien 23e01d23d3 Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored. 14 years ago
Pablo Rincon 6d60b3a747 filename and fileext keywords 14 years ago
Anoop Saldanha 2b356dadff Support for tos keyword added 14 years ago
Victor Julien 8cc82c7241 Add -S commandline option that loads a rule file exclusively. Issue #338. 14 years ago
Anoop Saldanha c7b9d3fecb Remove broken dsize_sm in SigMatch used by dsize in detection engine 14 years ago
Anoop Saldanha ba6bada155 change rev field in Signature to u32 and use strotoul to extract the value. Cleanup some dead code/comments 14 years ago
Eric Leblond a85dc9b0e2 Add support for replace keyword.
This patch adds support for the replace keyword. It is used with
content to change selected part of the payload. The major point
with this patch is that having a replace keyword made necessary
to avoid all stream level check because we need to access to the
could-be-modified packet payload.

One of the main difficulty is to handle complex signature. If there is
other content check, we must do the substitution when we're sure all
match are valid. The patch adds an attribute to the thread context
variable to be able to deal with recursivity of the match function.

Replace is only activated in IPS mode and apply only to raw match.
14 years ago
Eric Leblond 0c34a1c5e7 rewrite constants and add flag for replace
This patch make use of bit shift to rewrite some of the mask constants.

It also delete an unused flag value and suppress the associated dead code.
The numeric value of the flag is now used by the flag needed for replace
code.
14 years ago
Eric Leblond 3f153fb0da Add 'stream-event' keyword.
This patch adds an alias to the 'engine-event' keyword. It is now
possible to access to the stream events via the 'stream-event'
keyword. A simple transformation is done:
    stream-event:reassembly_segment_before_base_seq
is a shortcut for:
    engine-event:stream.reassembly_segment_before_base_seq
14 years ago
Eric Leblond e3a6d8955e Introduce engine-event keyword
This patch renames the 'decode-event' keyword to 'engine-event' and
keep it for backword compatibility of rulesets. All *DecodeEvent*
references in the code are replaced by EngineEvent version.
14 years ago
Eric Leblond 03c185a3ad threshold: add suppress keyword
This patch adds the suppress keyword to the threshold.config file.
The alerts are suppressed but the other elements like flowbits are
maintained.
14 years ago
Anoop Saldanha 35f3eafa5e byte extract added to the engine. Detection support added for packet payload, uri and dce detection engines 14 years ago
Victor Julien 1174df9712 Fix passing a uint8_t as an int. Breaks on some args. 14 years ago
Victor Julien 1e0b050a54 Add more mask flags. 14 years ago
Victor Julien 2dbfdd40af Clean up new SIMD mask checking code, improve non-SIMD checks. 14 years ago
Victor Julien b421019cef Match packet mask against 16 signature masks at once using SIMD instructions for SSE3 and up. 14 years ago
Victor Julien 75439863ed Shrink PacketAlerts structure so that Packet structure is a lot smaller. Reduce max events per packet from 256 to 15. 14 years ago
Anoop Saldanha 966119b6aa support for http_raw_uri keyword + mpm engine 14 years ago
Eric Leblond 9beebf621a Add support for 'nfq_set_mark' keyword
This patch introduces 'nfq_set_mark' which is new rules option. If a packet
matches a rule using nfq_set_mark in NFQ mode, it is marked with the mark/mask
specified in the option during the verdict.
It is thus possible to trigger different behaviour on the packet inside
Linux/Netfilter.
15 years ago
Anoop Saldanha c105a739e9 support for ssl_state keyword added 15 years ago
Gurvinder Singh 8f8b1212af support for ssl_version keyword 15 years ago
Anoop Saldanha c9897a44a4 fast pattern support for http_cookie. Also support relative modifiers 15 years ago
Anoop Saldanha bbbedaf963 fast pattern support for http_method. Also support relative modifiers 15 years ago
Victor Julien 39dea56a84 Remove flowbits as a mask prefilter as they are dynamic. Add a dynamic check. 15 years ago
Anoop Saldanha 8c21511c99 don't buffer raw headers. Retrieve them individually from htp_state during mpm stage and content valiadation stage 15 years ago
Victor Julien 435d0fb327 Clean up signature flags creating room for merging flags and mpm_flags. Merge flags and mpm_flags. Move new mpm id's into signature header. Get rid of full signature access in signature prefiltering. 15 years ago
Victor Julien 169aa5581a Rename SIG_FLAG_AMATCH flag to SIG_FLAG_STATE_MATCH to better reflects its purpose. 15 years ago
Victor Julien d7b92d9bfe Consolidate several signature flags into one. 15 years ago
Anoop Saldanha 25588b6910 comment out hrhd flags that we were using previously. Also remove the de_mpm_ based flags inside detect.h used by uri|hcbd|hhd|hrhd mpms. indentation fix as well 15 years ago
Anoop Saldanha 72b0fcf419 modify detection engine to carry out uri mpm run before build match array if alproto is http and if sgh has atleast one sig with uri mpm set 15 years ago
Anoop Saldanha 7ec0382774 support fast pattern for http raw header. Also support relative modifiers for http raw header 15 years ago
Anoop Saldanha c61c68fd36 mpm and fast pattern support for http_header. Also support relative modifiers for http_header 15 years ago
Anoop Saldanha 778ec0939c make client body buffer limit configurable. Also some minor changes 15 years ago
Anoop Saldanha 0aa5cffb12 fast pattern support for http_client_body keyword added. Also mpm support for http_client_body added 15 years ago
Anoop Saldanha bbd0c5056b store the content added for mpm inside Signature. also carry out an unconditional cleanup of packet pattern matcher pmq det_ctx->pmq 15 years ago
Anoop Saldanha 6df051321f fix fp when content is negated and also added to mpm 15 years ago
Anoop Saldanha 5c6a65dc58 support relative modifiers for http_client_body. Introduce body processing engine in detect-engine-hcbd.[ch] 15 years ago
Anoop Saldanha eade60f0fd make some name changes. break PopulateMpm(). Set the avoid mpm double check flags 15 years ago
Anoop Saldanha 6eaba8941c Use new flags to indicate uricontent has a mpm set 15 years ago
Anoop Saldanha ede7be34b5 replace all Signature->tmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_TMATCH] 15 years ago
Anoop Saldanha 3d2f81d978 replace all Signature->dmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_DMATCH] 15 years ago
Anoop Saldanha a7353be20d replace all Signature->amatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_AMATCH] 15 years ago
Anoop Saldanha e0476242c6 replace all Signature->umatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_UMATCH] 15 years ago
Anoop Saldanha e54358a9e1 replace all Signature->pmatch instances in the engine with Signature->sm_lists[DETECT_SM_LIST_PMATCH] 15 years ago
Anoop Saldanha 82fd581b64 replace all sm lists (match, pmatch, dmatch, umatch, amatch, tmatch) with an array Signature->sm_lists[]. Replace all Signature->match instances in the engine with Signature->sm_lists[DETECT_SM_LIST_MATCH] 15 years ago
Victor Julien 001f91056e Add http_raw_header as an alias to the http_header keyword as that actually inspects the raw headers (see issue #243). Closes issue #242. 15 years ago
Gurvinder Singh b7da115e6d support for http_stat_code keyword has been added to detection module 15 years ago
Gurvinder Singh 1deae70cf7 added http_stat_msg keyword support for detection module 15 years ago
Anoop Saldanha bc99328ec8 define a new conf paramter detect-engine:inspection-recursion-limit; Defines a recursion limit for content inspection code 15 years ago
Anoop Saldanha 88d94b136d Support for reference.config file 15 years ago
Anoop Saldanha 0ef684705c support single mpm context distribution across sghs in staging. Also see to it that ac works fine with this setup 15 years ago
Anoop Saldanha b367c37ae6 suricata.yaml conf update to support single mpm context distribution over multiple sghs + code to parse this conf 15 years ago
Victor Julien cbd4c298ed Initial version of a new bitmask based signature pre-filtering method. 15 years ago
Victor Julien fc248ca7a1 Many small performance updates. 15 years ago
Victor Julien 57b098e98c Add padding to commonly used data structures. 15 years ago
Pablo Rincon f225bd1428 Adding modifiers /C /H and /M to pcre (http cookie, header and method) 15 years ago
Pablo Rincon 9d7baa7a9f Adding ssh app layer module with two new keywords: ssh.protoversion and ssh.softwareversion 15 years ago
Victor Julien 9ba11dbfbd Clean up detection engine mpm initialization phase. 15 years ago
Anoop Saldanha b94eaec7c2 implement relative pcre matching in detect-engine-(payload|uri|dcepayload).c. Also fix within/distance handling of RELATIVE_NEXT flag for uricontent 15 years ago
Anoop Saldanha ae3148aded fix false positives for a negated content case 15 years ago
Victor Julien 102092a89c Make signature address matching more cache efficient. 15 years ago
Victor Julien 1eec149f5e Use Address structure in DetectAddress struct. 15 years ago
Pablo Rincon 693d4f54eb Load signatures with incompatible fast_pattern option (due to design differences for optimization) 15 years ago
Victor Julien d41b5645ef Make sure decoder event rules are inspected even if the packet is invalid and has no addesses or proto. Update fast log and alert debug log to display the alerts. Fixes #179. 15 years ago
Victor Julien 67429e523f Fix compiler warning about incomplete prototype. 15 years ago
Victor Julien 7454336ef5 Make SigWrapper private to detect-parse.c and rename to SigDuplWrapper to reflect it's use and purpose. 15 years ago
Anoop Saldanha 9ecade76b9 in case of duplicate signatures used the one with the latest revision 15 years ago
Gurvinder Singh 8852b83fa7 flowbits, flowvars, pktvars, flow flags and app layer info added to alert-debug.log 15 years ago
Victor Julien 580b09c2b8 Make sure we inspect all outstanding reassembled stream chunks (smsg) if the stream is shutting down. Make sure to do inspect signatures that use dsize against the tcp packet payload, even if that payload was already added to the stream. Likewise, the dsize signatures are not inspected against the reassembled stream. 15 years ago
Pablo Rincon eed0ef6e69 Adding tag keyword support 15 years ago
Pablo Rincon 6950537a76 Move rate_filter rule tracking data from Signature to Threshold context 15 years ago
Victor Julien 37442a8a84 Prefilter signatures before fully scanning them. 15 years ago
Anoop Saldanha 45ea0d914e dce stub content keywords support using dcepayload.c support for all dce related content keywords 15 years ago
Victor Julien 83b2c8abdb Improve stateful uri detection code. 15 years ago
Victor Julien 9dd753b5f3 Scan uricontent mpm on demand. 15 years ago
Victor Julien a24f288074 Moving the stream content scanning to have it's own mpm ctx. 15 years ago
Victor Julien a0c1209a44 Inspect the reassembled stream together with the packet payload in the same direction. 15 years ago
Victor Julien 2fd31a1a11 Remove dsize grouping from detection engine grouping reducing memory usage. Store sgh in flow to reduce lookups. Reduce locking in alert handling. Increase default grouping values as we use less memory. 15 years ago
Victor Julien dff6795df5 Detect cleanups. 15 years ago
Gerardo Iglesias Galvan 55dfa36963 Add support for http_uri keyword 15 years ago
Victor Julien 7a6d4b57f0 Add support for class id in classification code. Submitted by firnsy@securixlive.com, thanks. 15 years ago
Pablo Rincon 3fa3229e01 ASN1 decoder and keyword implementation 15 years ago
Victor Julien 70b32f7380 First stab at creating a stateful detection engine.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:

- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.

This commit fixes bug #124.
15 years ago
Jason Ish 18e5ac8cde Basic rule profiling even though the results may be skewed by a bad rule in a grouping of rules. 15 years ago
Victor Julien e27cefa6f7 Complete conversion of pattern id mpm storage vs sig id storage. 15 years ago
Victor Julien 46831e0f8f Fix signature grouping bug for protocols without ports. Add debugging code. 15 years ago
Victor Julien 7a427ec7f4 Switch to pattern id based results checking in the mpm. Move app layer proto detection towards a more signature based approach. 15 years ago
Victor Julien a372c1d14e Fix/workaround a strange detection issue. 15 years ago
Pablo Rincon e18e2ec998 Changing threshold logic 15 years ago
William Metcalf ce01927515 Import of GPLv2 Header 050410 15 years ago
Pablo Rincon ab02ab9ead adding http_header keyword support 15 years ago
Anoop Saldanha 47037ef9ec fix for bug 115 15 years ago
Victor Julien fe7ece997a Different approach to the reference keyword. Lots of cleanups, bug fixes in reference keyword code and tests. 15 years ago
Breno Silva 89baf93a40 Reference Support 15 years ago
Pablo Rincon e7a989e305 IP Only Engine using radix trees 16 years ago