aiden
/
suricata
mirror of https://github.com/OISF/suricata
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,723 Commits
7 Branches
161 Tags
210 MiB
master
main-7.0.x
master-6.0.x
master-5.0.x
master-4.1.x
master-4.0.x
master-3.2.x
suricata-8.0.0
suricata-7.0.11
suricata-8.0.0-rc1
suricata-8.0.0-beta1
suricata-7.0.10
suricata-7.0.9
suricata-7.0.8
suricata-7.0.7
suricata-7.0.6
suricata-6.0.20
suricata-7.0.5
suricata-6.0.19
suricata-6.0.18
suricata-7.0.4
suricata-6.0.17
suricata-7.0.3
suricata-6.0.16
suricata-7.0.2
suricata-6.0.15
suricata-7.0.1
suricata-6.0.14
suricata-7.0.0
suricata-7.0.0-rc2
suricata-6.0.13
suricata-6.0.12
suricata-6.0.11
suricata-7.0.0-rc1
suricata-6.0.10
suricata-6.0.9
suricata-7.0.0-beta1
suricata-6.0.8
suricata-6.0.7
suricata-6.0.6
suricata-5.0.10
suricata-6.0.5
suricata-5.0.9
suricata-6.0.4
suricata-5.0.8
suricata-6.0.3
suricata-5.0.7
suricata-6.0.2
suricata-5.0.6
suricata-6.0.1
suricata-5.0.5
suricata-4.1.10
suricata-4.1.9
suricata-5.0.4
suricata-6.0.0
suricata-6.0.0-rc1
suricata-6.0.0-beta1
suricata-5.0.3
suricata-4.1.8
suricata-4.1.7
suricata-5.0.2
suricata-4.1.6
suricata-5.0.1
suricata-5.0.0
suricata-5.0.0-rc1
suricata-4.1.5
suricata-5.0.0-beta1
suricata-4.1.4
suricata-4.1.3
suricata-4.0.7
suricata-4.1.2
suricata-4.1.1
suricata-4.1.0
suricata-4.0.6
suricata-4.1.0-rc2
suricata-4.1.0-rc1
suricata-4.0.5
suricata-4.1.0-beta1
suricata-4.0.4
suricata-4.0.3
suricata-4.0.2
suricata-3.2.5
suricata-4.0.1
suricata-3.2.4
suricata-4.0.0
suricata-4.0.0-rc2
suricata-3.2.3
suricata-4.0.0-rc1
suricata-4.0.0-beta1
suricata-3.2.2
suricata-3.1.4
suricata-3.2.1
suricata-3.2
suricata-3.2RC1
suricata-3.1.3
suricata-3.2beta1
suricata-3.1.2
suricata-3.1.1
suricata-3.1
suricata-3.0.2
suricata-3.1RC1
suricata-3.0.1
suricata-3.0.1RC1
suricata-3.0
suricata-2.0.11
suricata-3.0RC3
suricata-3.0RC2
suricata-3.0RC1
suricata-2.0.10
suricata-2.0.9
suricata-2.1beta4
suricata-2.0.8
suricata-2.0.7
suricata-2.1beta3
suricata-2.0.6
suricata-2.0.5
suricata-2.1beta2
suricata-2.0.4
suricata-2.1beta1
suricata-2.0.3
suricata-2.0.2
suricata-2.0.1
suricata-2.0.1rc1
suricata-2.0
suricata-2.0rc3
suricata-2.0rc2
suricata-2.0rc1
suricata-2.0beta2
suricata-1.4.7
suricata-1.4.6
suricata-1.4.5
suricata-2.0beta1
suricata-1.4.4
suricata-1.4.3
suricata-1.4.2
suricata-1.4.1
suricata-1.3.6
suricata-1.4
suricata-1.3.5
suricata-1.4rc1
suricata-1.3.4
suricata-1.4beta3
suricata-1.3.3
suricata-1.4beta2
suricata-1.3.2
suricata-1.4beta1
suricata-1.3.1
suricata-1.3
suricata-1.3rc1
suricata-1.3beta2
suricata-1.3beta1
suricata-1.2.1
suricata-1.2
suricata-1.2rc1
suricata-1.2beta1
suricata-1.1.1
suricata-0.8.2
suricata-1.0.0
suricata-1.0.1
suricata-1.0.2
suricata-1.0.3
suricata-1.0.4
suricata-1.0.5
suricata-1.1
suricata-1.1beta1
suricata-1.1beta2
suricata-1.1beta3
suricata-1.1rc1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'f2fc5a255f'
${ noResults }
Commit Graph

475 Commits (f2fc5a255f09c5287e9a6baf4cbeadab6f35a82b)

Author SHA1 Message Date
Victor Julien 798ba010ca prefilter: use array of engines per sgh 
Instead of the linked list of engines setup an array
with the engines. This should provide better locality.

Also shrink the engine structure so that we can fit
2 on a cacheline.

Remove the FreeFunc from the runtime engines. Engines
now have a 'gid' (global id) that can be used to look
up the registered Free function.
 9 years ago
Victor Julien 5f994756e6 detect-engine: improved inspect engines 
Inspect engines are called per signature per sigmatch list. Most
wrap around DetectEngineContentInspection, but it's more generic.

Until now, the inspect engines were setup in a large per ipproto,
per alproto, per direction table. For stateful inspection each
engine needed a global flag.

This approach had a number of issues:
1. inefficient: each inspection round walked the table and then
   checked if the inspect engine was even needed for the current
   rule.
2. clumsy registration with global flag registration.
3. global flag space was approaching the need for 64 bits
4. duplicate registration for alprotos supporting both TCP and
   TCP (DNS).

This patch introduces a new approach.

First, it does away with the per ipproto engines. This wasn't used.

Second, it adds a per signature list of inspect engine containing
only those engines that actually apply to the rule.

Third, it gets rid of the global flags and replaces it with flags
assigned per rule per engine.
 9 years ago
Victor Julien f1e3840516 http_response_body: implement keyword with mpm 
Implemented as 'stickybuffer'.
 9 years ago
Victor Julien 4c98b6cef3 http_request_line: implement keyword and mpm 
Implemented as 'stickybuffer'.

Move all logic into the keyword file and remove bad tests that tested
URI instead of request line.
 9 years ago
Victor Julien 5b2e36a1b0 mpm: add App Layer MPM registery 
Register keywords globally at start up.

Create a map of the registery per detection engine. This we need because
the sgh_mpm_context value is set per detect engine.

Remove APP_MPMS_MAX.
 9 years ago
Victor Julien 125603871b detect: config opt to enable keyword prefilters 9 years ago
Victor Julien 9bb12ccb27 prefilter: move payload engines into separate list 9 years ago
Victor Julien 14b0537f95 prefilter: implement basic prefilter priority order 9 years ago
Victor Julien 31ad0a133b prefilter: engine for tcp flags keyword 
If there are many rules for TCP flags these rules would be inspected
against each TCP packet. Even though the flags check is not expensive,
the combined cost of inspecting multiple rules against each and every
packet is high.

This patch implements a prefilter engine for flags. If a rule group
has rules looking for specific flags and engine for that flag or
flags combination is set up. This way those rules are only inspected
if the flag is actually present in the packet.
 9 years ago
Victor Julien 8798bf48b2 profiling: support prefilter engines 9 years ago
Victor Julien 56239690d0 prefilter: implement prefilter keyword 
Introduce prefilter keyword to force a keyword to be used as prefilter.

e.g.
alert tcp any any -> any any (content:"A"; flags:R; prefilter; sid:1;)
alert tcp any any -> any any (content:"A"; flags:R; sid:2;)
alert tcp any any -> any any (content:"A"; dsize:1; prefilter; sid:3;)
alert tcp any any -> any any (content:"A"; dsize:1; sid:4;)

In sid 2 and 4 the content keyword is used in the MPM engine.
In sid 1 and 3 the flags and dsize keywords will be used.
 9 years ago
Victor Julien 85cb749e8b detect cleanup: remove sgh mpm_ctx pointers 9 years ago
Victor Julien 82d3c0b520 sgh: remove unused flags 9 years ago
Victor Julien 9ff5703c49 packet/stream: mpm prefilter engine 9 years ago
Victor Julien 5bcdbe3922 prefilter: introduce prefilter engines 
Introduce abstraction layer for prefilter engines.
 9 years ago
Victor Julien 3dad824fb2 detect: rename SignatureNonMpmStore 
New name is SignatureNonPrefilterStore to reflect that it's not just
about MPM anymore.
 9 years ago
Victor Julien 17bc0299fe detect: rename non_mpm lists/vars to non_pf 
Rename to non_pf: non prefilter.
 9 years ago
Victor Julien bb0cd0e883 prefilter: rename PatternMatcherQueue datatype 
In preparation of the introduction of more general purpose prefilter
engines, rename PatternMatcherQueue to PrefilterRuleStore. The new
engines will fill this structure a similar way to the current mpm
prefilters.
 9 years ago
Giuseppe Longo 07564c4e41 detect: add bypass keyword 
This adds a new keyword which permits to call the
bypass callback when a sig is matched.

The callback must be called when the match of the sig
is complete.
 9 years ago
Mats Klepsland dc8e0b3cf2 detect: add detect engine for tls validity keywords 
Add detect engine for tls validity keywords (tls_cert_notbefore and
tls_cert_notafter).
 9 years ago
Mats Klepsland ea5696812f detect: add tls_cert_notbefore and tls_cert_notafter keywords 
Detection plugin for TLS certificate fields notBefore and notAfter.

Supports equal to, less than, greater than, and range operations
for both keywords. Dates can be represented as either ISO 8601 or
epoch (Unix time).

Examples:
alert tls [...] tls_cert_notafter:1445852105; [...]
alert tls [...] tls_cert_notbefore:<2015-10-22T23:59:59; [...]
alert tls [...] tls_cert_notbefore:>2015-10-22; [...]
alert tls [...] tls_cert_notafter:2000-10-22<>2020-05-15; [...]
 9 years ago
Duarte Silva 89eb935f73 file-hashing: added support for SHA-256 file hashing 9 years ago
Duarte Silva a6d928e269 file-hashing: added support for SHA-1 file hashing 9 years ago
Duarte Silva 188b382c46 file-hashing: common code added 
Moved and adapted code from detect-filemd5 to util-detect-file-hash,
generalised code to work with SHA-1 and SHA-256 and added necessary
flags and other constants.
 9 years ago
Mats Klepsland 4172c4c8ac tls: add (mpm) keyword tls_cert_subject 
This keyword is a replacement for tls.subject.
 9 years ago
Mats Klepsland 9b2717799c tls: add (mpm) keyword tls_cert_issuer 
This keyword is a replacement for tls.issuerdn.
 9 years ago
Victor Julien 215d0d54c7 detect: optimize rule address parsing 
Many rules have the same address vars, so instead of parsing them
each time use a hash to store the string and the parsed result.

Rules now reference the stored result in the hash table.
 9 years ago
Victor Julien 99dce740ef detect: mark alproto in keyword reg deprecated 
No existing code uses it, and it had been useless for some time.
 9 years ago
Victor Julien e43ce0a9ec file: switch to streaming buffer API 
Make the file storage use the streaming buffer API.

As the individual file chunks were not needed by themselves, this
approach uses a chunkless implementation.
 9 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API 
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
 9 years ago
Mats Klepsland a13df67864 detect: add (mpm) keyword for tls_sni 
Match on server name indication (SNI) extension in TLS using tls_sni
keyword, e.g:

alert tls any any -> any any (msg:"SNI test"; tls_sni;
        content:"example.com"; sid:12345;)
 9 years ago
Justin Viiret cce2d114e8 spm: add and use new SPM API 
This new API allows for different SPM implementations, using a function
pointer table like that used for MPM.

This change also switches over the paths that make use of
DetectContentData (which previously used BoyerMoore directly) to the new
API.
 9 years ago
Justin Viiret ce408c4d10 spm: add SinglePatternMatchDefaultMatcher 
Allows selecting SPM algorithm with the 'spm-algo' value in the YAML
config file.
 9 years ago
Victor Julien f1ee125810 detect-flowvar: shrink mem structure by 8 bytes 9 years ago
Victor Julien e2789a87c2 detect: shrink IPOnlyCIDRItem with 8 bytes 9 years ago
Victor Julien adc453eec6 detect-port: improve comment about sgh pointer 9 years ago
Victor Julien ef6f347f84 detect-address: remove sgh pointer as it's unused 9 years ago
Victor Julien d6ba01b1b7 detect: make port whitelisting configurable 
Make the port grouping whitelisting configurable. A whitelisted port
ends up in it's own port group.

detect:
  grouping:
    tcp-whitelist: 80, 443
    udp-whitelist: 53, 5060

No portranges are allowed at this point.
 9 years ago
Victor Julien fdd05e8fb4 mpm: remove unused app proto factory 9 years ago
Victor Julien 0311f01b97 rule grouping: speed up port based grouping 
Create a hash table of unique DetectPort objects before trying to
create a unique list of these objects. This safes a lot of cycles
in the creation of the list.
 9 years ago
Victor Julien 87f3adbe4c detect/mpm: unify packet/stream mpm_ctx pointers 
SGH's for tcp and udp are now always only per proto and per direction.
This means we can simply reuse the packet and stream mpm pointers.

The SGH's for the other protocols already used a directionless catch
all mpm pointer.
 9 years ago
Victor Julien af3bf3dc7a detect: optimize sgh layout 9 years ago
Victor Julien de273d88cc detect: remove unused content minlen tracking 9 years ago
Victor Julien 5857660568 http_uri: mpm cleanup. Use mpm_ctx's minlen 9 years ago
Victor Julien e57e7d1b96 mpm: cleanup, remove unused structs and prototypes 9 years ago
Victor Julien caea596ce5 profiling: output post-prefilter matches 
Dump a json record containing all sigs that need to be inspected after
prefilter. Part of profiling. Only dump if threshold is met, which is
currently set by:

 --set detect.profiling.inspect-logging-threshold=200

A file called packet_inspected_rules.json is created in the default
log dir.
 9 years ago
Victor Julien 722e2dbf7c profiling: initial rulegroup tracking 
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.

Logs SGH id so it can be compared with the rule_group.json output.

Implemented both in a human readable text format and a JSON format.
 9 years ago
Victor Julien e310a033be detect: assign id to sgh 9 years ago
Victor Julien c880b79f45 detect: shrink sgh 
Turn list of mpm_ctx pointers into a union so that we don't waste
space. The sgh's for tcp and udp are in one direction only, so the
ts and tc ones are now in the union.
 9 years ago
Victor Julien c804102a9a detect: move app_mpms array to init data 9 years ago
Victor Julien 9e71ef4c3b detect: remove signature pattern id reference 9 years ago
Victor Julien c1ad08d11e detect: remove stream pmq array 9 years ago
Victor Julien 7c94077892 detect mpm: remove unused mpm flags 9 years ago
Victor Julien a7d126738a detect address: remove unused features 9 years ago
Victor Julien cbf80de6fe detect-port: cleanup 9 years ago
Victor Julien a96fa0fc2f detect: remove unused dport sgh hash 9 years ago
Victor Julien 1f7e33a4b0 detect: remove unused flag 9 years ago
Victor Julien e8c9598035 detect sgh: remove unused field 9 years ago
Victor Julien a34be23002 detect: simplify negated mpm handling 9 years ago
Victor Julien b84d6d402f detect grouping: multiple whitelist conditions 
Instead of the binary yes/no whitelisting used so far, use different
values for different sorts of whitelist reasons. The port list will
be sorted by whitelist value first, then by rule count.

The goal is to whitelist groups that have weak sigs:

 - 1 byte pattern groups

 - SYN sigs

    Rules that check for SYN packets are mostly scan detection rules.
    They will be checked often as SYN packets are very common.

    e.g. alert tcp any any -> any 22 (flags:S,12; sid:123;)

    This patch adds whitelisting for SYN-sigs, so that the sigs end up
    in as unique groups as possible.

 - negated mpm sigs

    Currently negated mpm sigs are inspected often, so they are quite
    expensive. For this reason, try to whitelist them.

These values are set during 'stage 1', rule preprocessing.
 9 years ago
Victor Julien 3c184c19cd detect grouping: port based group whitelisting 
Whitelist some ports in grouping to make sure they get their own group.
 9 years ago
Victor Julien 2ce03fbabb detect: split non-mpm list into syn/nosyn 
Since SYN inspecting rules are expensive, this patch splits the
'non-mpm' list (i.e. the rules that are always considered) into
a 'syn' and 'non-syn' list. The SYN list is only inspected if the
packet has the SYN flag set, otherwise the non-syn list is used.

The syn-list contains _all_ rules. The non-syn list contains all
minus the rules requiring the SYN bit in a packet.
 9 years ago
Victor Julien e48d745ed7 mpm: constify search func args 9 years ago
Victor Julien 102a82fc7b detect: use mpm store for app layer mpms 
Rework app-layer mpm setup and registration to make this possible.
 9 years ago
Victor Julien fac2cc0560 detect: mpm deduplication 
Create hash for mpm's that we can reuse. Have packet/stream mpms
use this.
 9 years ago
Victor Julien f0ba00e51d detect: remove old unused code 9 years ago
Victor Julien 2ee9bf2aef detect: rename groupings vars 9 years ago
Victor Julien 1f70ccfc23 detect: remove unused grouping settings 9 years ago
Victor Julien d82df4eb8b detect-mpm: make sgh setup proto aware 
Allow multi-proto, multi-direction sgh's.
 9 years ago
Victor Julien fd5a06017d detect: per port and proto rule grouping 
Replace tree based approach for rule grouping with a per port (tcp/udp)
and per protocol approach.

Grouping now looks like:

               +----+
               |icmp+--->
               +----+
               |gre +--->
               +----+
               |esp +--->
               +----+
          other|... |
        +----->-----+
        |      |N   +--->
        |      +----+
        |
        | tcp  +----+   +----+
        +----->+ 80 +-->+ 139+-->
        |      +----+   +----+
        |
        | udp  +----+   +----+
    +---+----->+ 53 +-->+ 135+-->
    |          +----+   +----+
    |toserver
+--->
    |toclient
    |
    +--->

So the first 'split' in the rules is the direction: toserver or toclient.
Rules that don't have a direction, are in both branches.

Then the split is between tcp/udp and the other protocols. For tcp and
udp port lists are used. For the other protocols, grouping is simply per
protocol.

The ports used are the destination ports for toserver sigs and source
ports for toclient sigs.
 9 years ago
Victor Julien 27e63a1e11 detect: track direction and ipproto of sgh 
Each SGH has a unique ipproto and direction.
 9 years ago
Jason Ish 3c3fe4b47d detect: denote the max detection list; fix issue 1674. 
Denotes the max detection list so that rule validation can
allow post-detection lists to come after base64_data, but
disallow detection lists to come after it.
 10 years ago
Victor Julien c0b16fa2bb detect: allow for more than 64k mpm rules 10 years ago
Jason Ish 6b15686fd1 base64_decode, base64_data: decode and match base64 10 years ago
Victor Julien 6b2f831a70 mpm: SGH maxlen was actually minlen, so rename 10 years ago
Victor Julien 2716c78628 mpm: improve SGH content len tracking 
SGH's track content length for rule grouping.

This patch changes the logic to only consider the pattern that is
used in the mpm for a sig.
 10 years ago
Victor Julien e529ebb50e mpm: redo uri maxlen logic 
The mpm_uricontent_maxlen logic was meant to track the shortest
possible pattern in the MPM of a SGH. So a minlen more than a maxlen.

This patch replaces the complicated tracking logic by a simpler
scheme. When the SGH's are finalize, the minlen is calculated.

It also fixes a small corner case where the calculated "maxlen" could
be wrong. This would require a smaller pattern in a rule to be forced
as fast pattern.
 10 years ago
Victor Julien df95d375bb detect: improve comments on mpm 10 years ago
Victor Julien 496f9800ac mpm: remove used counter 10 years ago
Victor Julien c53c9b4b20 mpm: remove bloated counting logic 
Counters were only used to print debug info.
 10 years ago
Jason Ish 06beca62f5 app-layer: template for application layer content inspection 10 years ago
Victor Julien cb060f16fd detect plugin: add template 10 years ago
Victor Julien eb09118d64 detect: create loader threads 
To speed up startup with many tenants, tenant loading will be parallelized.
As no tempary threads should be used for these memory allocation heavy
tasks, this patch adds new type of 'command' thread that can be used to
load and reload tenants.

This patch hardcodes the number of loaders to 4. Future work will make it
dynamic.

The loader thread essentially sleeps constantly. When a tasks is sent to
it, it will wake up and execute it.
 10 years ago
Victor Julien e19c41a807 multi-detect: hash lookup for tenants 
Use hash for storing and looking up det_ctxs.
 10 years ago
Victor Julien 1893c5edb1 multi-detect: initial selectors for tenants 
The Detection Thread has the TenantGetId pointer which allows it
to select a tenant id based on the packet.
 10 years ago
Victor Julien b653479815 detect: make multi tenancy a global switch 
At start up we will set this flag based on "multi-detect.enabled".
 10 years ago
Victor Julien def2b58725 detect: initial MT lookup logic 
In the DetectEngineThreadCtx, store another DetectEngineThreadCtx per
tenant.

Currently it's just a simple array indexed by the tenant id.
 10 years ago
Victor Julien 147a6d2bfd multi-detect: (un)register-tenant unix socket commands 
Make available to live mode and unix socket mode.

register-tenant:
    Loads a new YAML, does basic validation.
    Loads a new detection engine
    Loads rules
    Add new de_ctx to master store and stores tenant id in the de_ctx so
        we can look it up by tenant id later.

unregister-tenant:
    Gets the de_ctx, moves it to the freelist
    Removes config

Introduce DetectEngineGetByTenantId, which gets a reference to the
detect engine by tenant id.
 10 years ago
Victor Julien a0899cbe85 detect: optimize Signature layout 10 years ago
Victor Julien 110e23964e detect: add AppLayerTxMatch call 10 years ago
Victor Julien e6129f7b47 dns: generic request/response detect lists 10 years ago
Victor Julien 2c8e8c2516 dns: rename type so it's purpose is more clear 10 years ago
Victor Julien ef7cd043cc detect: various header cleanups 10 years ago
Victor Julien 5483b800c5 detect: remove struct/union tricks from Signature 10 years ago
Alexander Gozman f11e237d77 Feature #1440: support wildcards in rule filenames 10 years ago
Giuseppe Longo 84dc73d9de mpm: implement prefiltering for smtp 10 years ago
Giuseppe Longo f0c54d4764 Detect engine for smtp file_data file_data: inspecting smtp attachments 
Create a buffer to store reassembled file chunks,
and inspect the content.
 10 years ago
Giuseppe Longo 41a1a9f4af find and replace HSBDMATCH by FILEDATA 
This commit do a find and replace of the following:

- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
  sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*

- HSBD by FILEDATA:
  sed -i 's/HSBDMATCH/FILEDATA/g' src/*
 10 years ago
Victor Julien 61cb2abc8d ippair: xbit implementation 
detect xbits for ippair: initial implementation
 10 years ago
Victor Julien 5c880377ae detect: hostbits keyword 
Per host bit similar to 'flowbits'.

Initial code that uses just the 'src' ip for the operations.
 10 years ago