aiden
/
suricata
mirror of https://github.com/OISF/suricata
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,805 Commits
7 Branches
161 Tags
169 MiB
main-7.0.x
master
master-6.0.x
master-5.0.x
master-4.1.x
master-4.0.x
master-3.2.x
suricata-8.0.0
suricata-7.0.11
suricata-8.0.0-rc1
suricata-8.0.0-beta1
suricata-7.0.10
suricata-7.0.9
suricata-7.0.8
suricata-7.0.7
suricata-7.0.6
suricata-6.0.20
suricata-7.0.5
suricata-6.0.19
suricata-6.0.18
suricata-7.0.4
suricata-6.0.17
suricata-7.0.3
suricata-6.0.16
suricata-7.0.2
suricata-6.0.15
suricata-7.0.1
suricata-6.0.14
suricata-7.0.0
suricata-7.0.0-rc2
suricata-6.0.13
suricata-6.0.12
suricata-6.0.11
suricata-7.0.0-rc1
suricata-6.0.10
suricata-6.0.9
suricata-7.0.0-beta1
suricata-6.0.8
suricata-6.0.7
suricata-6.0.6
suricata-5.0.10
suricata-6.0.5
suricata-5.0.9
suricata-6.0.4
suricata-5.0.8
suricata-6.0.3
suricata-5.0.7
suricata-6.0.2
suricata-5.0.6
suricata-6.0.1
suricata-5.0.5
suricata-4.1.10
suricata-4.1.9
suricata-5.0.4
suricata-6.0.0
suricata-6.0.0-rc1
suricata-6.0.0-beta1
suricata-5.0.3
suricata-4.1.8
suricata-4.1.7
suricata-5.0.2
suricata-4.1.6
suricata-5.0.1
suricata-5.0.0
suricata-5.0.0-rc1
suricata-4.1.5
suricata-5.0.0-beta1
suricata-4.1.4
suricata-4.1.3
suricata-4.0.7
suricata-4.1.2
suricata-4.1.1
suricata-4.1.0
suricata-4.0.6
suricata-4.1.0-rc2
suricata-4.1.0-rc1
suricata-4.0.5
suricata-4.1.0-beta1
suricata-4.0.4
suricata-4.0.3
suricata-4.0.2
suricata-3.2.5
suricata-4.0.1
suricata-3.2.4
suricata-4.0.0
suricata-4.0.0-rc2
suricata-3.2.3
suricata-4.0.0-rc1
suricata-4.0.0-beta1
suricata-3.2.2
suricata-3.1.4
suricata-3.2.1
suricata-3.2
suricata-3.2RC1
suricata-3.1.3
suricata-3.2beta1
suricata-3.1.2
suricata-3.1.1
suricata-3.1
suricata-3.0.2
suricata-3.1RC1
suricata-3.0.1
suricata-3.0.1RC1
suricata-3.0
suricata-2.0.11
suricata-3.0RC3
suricata-3.0RC2
suricata-3.0RC1
suricata-2.0.10
suricata-2.0.9
suricata-2.1beta4
suricata-2.0.8
suricata-2.0.7
suricata-2.1beta3
suricata-2.0.6
suricata-2.0.5
suricata-2.1beta2
suricata-2.0.4
suricata-2.1beta1
suricata-2.0.3
suricata-2.0.2
suricata-2.0.1
suricata-2.0.1rc1
suricata-2.0
suricata-2.0rc3
suricata-2.0rc2
suricata-2.0rc1
suricata-2.0beta2
suricata-1.4.7
suricata-1.4.6
suricata-1.4.5
suricata-2.0beta1
suricata-1.4.4
suricata-1.4.3
suricata-1.4.2
suricata-1.4.1
suricata-1.3.6
suricata-1.4
suricata-1.3.5
suricata-1.4rc1
suricata-1.3.4
suricata-1.4beta3
suricata-1.3.3
suricata-1.4beta2
suricata-1.3.2
suricata-1.4beta1
suricata-1.3.1
suricata-1.3
suricata-1.3rc1
suricata-1.3beta2
suricata-1.3beta1
suricata-1.2.1
suricata-1.2
suricata-1.2rc1
suricata-1.2beta1
suricata-1.1.1
suricata-0.8.2
suricata-1.0.0
suricata-1.0.1
suricata-1.0.2
suricata-1.0.3
suricata-1.0.4
suricata-1.0.5
suricata-1.1
suricata-1.1beta1
suricata-1.1beta2
suricata-1.1beta3
suricata-1.1rc1
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '1d9f37a60e'
${ noResults }
Commit Graph

964 Commits (1d9f37a60e7fde720768f41e5680ee2e02b78ffc)

Author SHA1 Message Date
Victor Julien 8321f04ef3 prefilter: clean up setup code 9 years ago
Victor Julien d36c0c15ea detect: reshuffle keyword registration order 
The order of keyword registration currently affects inspect engine
registration order and ultimately the order of inspect engines per
rule. Which in turn affects state keeping.

This patch makes sure the ordering is the same as with older
releases.
 9 years ago
Victor Julien 5f994756e6 detect-engine: improved inspect engines 
Inspect engines are called per signature per sigmatch list. Most
wrap around DetectEngineContentInspection, but it's more generic.

Until now, the inspect engines were setup in a large per ipproto,
per alproto, per direction table. For stateful inspection each
engine needed a global flag.

This approach had a number of issues:
1. inefficient: each inspection round walked the table and then
   checked if the inspect engine was even needed for the current
   rule.
2. clumsy registration with global flag registration.
3. global flag space was approaching the need for 64 bits
4. duplicate registration for alprotos supporting both TCP and
   TCP (DNS).

This patch introduces a new approach.

First, it does away with the per ipproto engines. This wasn't used.

Second, it adds a per signature list of inspect engine containing
only those engines that actually apply to the rule.

Third, it gets rid of the global flags and replaces it with flags
assigned per rule per engine.
 9 years ago
Victor Julien f1e3840516 http_response_body: implement keyword with mpm 
Implemented as 'stickybuffer'.
 9 years ago
Victor Julien 4c98b6cef3 http_request_line: implement keyword and mpm 
Implemented as 'stickybuffer'.

Move all logic into the keyword file and remove bad tests that tested
URI instead of request line.
 9 years ago
Victor Julien 5b2e36a1b0 mpm: add App Layer MPM registery 
Register keywords globally at start up.

Create a map of the registery per detection engine. This we need because
the sgh_mpm_context value is set per detect engine.

Remove APP_MPMS_MAX.
 9 years ago
Victor Julien e1eb481647 prefilter: cleanup and optimization 9 years ago
Victor Julien dba14b676c profiling: more prefilter profiling 9 years ago
Victor Julien 125603871b detect: config opt to enable keyword prefilters 9 years ago
Victor Julien 9bb12ccb27 prefilter: move payload engines into separate list 9 years ago
Victor Julien 14b0537f95 prefilter: implement basic prefilter priority order 9 years ago
Victor Julien 8798bf48b2 profiling: support prefilter engines 9 years ago
Victor Julien f80623fd73 prefilter: show prefilter capability in --list-keywords 9 years ago
Victor Julien 56239690d0 prefilter: implement prefilter keyword 
Introduce prefilter keyword to force a keyword to be used as prefilter.

e.g.
alert tcp any any -> any any (content:"A"; flags:R; prefilter; sid:1;)
alert tcp any any -> any any (content:"A"; flags:R; sid:2;)
alert tcp any any -> any any (content:"A"; dsize:1; prefilter; sid:3;)
alert tcp any any -> any any (content:"A"; dsize:1; sid:4;)

In sid 2 and 4 the content keyword is used in the MPM engine.
In sid 1 and 3 the flags and dsize keywords will be used.
 9 years ago
Victor Julien 08407b6d47 tls: mpm prefilter engines 9 years ago
Victor Julien 7acdc66061 smtp file_data: mpm prefilter engine 9 years ago
Victor Julien 0019a7bd9f http_raw_header: mpm prefilter engine 
Register for both regular headers and trailer.
 9 years ago
Victor Julien cef12ed80f http_server_body / file_data: mpm prefilter engine 9 years ago
Victor Julien 5646dd9ecf http_client_body: mpm prefilter engine 9 years ago
Victor Julien 9b6fd6bb48 http_headers: mpm prefilter engines 
Register for both regular headers and trailers.
 9 years ago
Victor Julien 9cab3ea2cd http_stat_code: mpm prefilter engine 9 years ago
Victor Julien 4d57b2fc63 http_stat_msg: mpm prefilter engine 9 years ago
Victor Julien 86d303e32b http_raw_host: mpm prefilter engine 9 years ago
Victor Julien 5218849213 http_host: mpm prefilter engine 9 years ago
Victor Julien 61c3748fc4 http_user_agent: mpm prefilter engine 9 years ago
Victor Julien a43a69305d http_cookie: mpm prefilter engine 9 years ago
Victor Julien 7a46364e42 http_raw_uri: mpm prefilter engine 9 years ago
Victor Julien 746a169127 dns_query: mpm prefilter engine 9 years ago
Victor Julien 9ff5703c49 packet/stream: mpm prefilter engine 9 years ago
Victor Julien 72f2a78b1f http_method: mpm prefilter engine 9 years ago
Victor Julien b62c4cc359 http_uri: mpm prefilter engine 
Inspect partial request line as well.
 9 years ago
Victor Julien 5bcdbe3922 prefilter: introduce prefilter engines 
Introduce abstraction layer for prefilter engines.
 9 years ago
Victor Julien 17bc0299fe detect: rename non_mpm lists/vars to non_pf 
Rename to non_pf: non prefilter.
 9 years ago
Victor Julien ff70e0cca0 mpm tls: remove unused function args 9 years ago
Victor Julien ad3a55d938 mpm dns query: remove unused function args 9 years ago
Victor Julien d647db1775 mpm stat code: remove unused function args 9 years ago
Victor Julien bd03307921 mpm stat msg: remove unused function args 9 years ago
Victor Julien 6d54b70db4 mpm ua: remove unused function args 9 years ago
Victor Julien 704afeb078 mpm cookie: remove unused function args 9 years ago
Victor Julien 4229e603f0 mpm raw host: remove unused function args 9 years ago
Victor Julien 1380853ee8 mpm host: remove unused function args 9 years ago
Victor Julien b40ecb7356 mpm method: remove unused function args 9 years ago
Victor Julien 3d5807ba44 mpm raw uri: remove unused function args 9 years ago
Victor Julien d461c7888a mpm uri: remove unused function args 9 years ago
Giuseppe Longo 07564c4e41 detect: add bypass keyword 
This adds a new keyword which permits to call the
bypass callback when a sig is matched.

The callback must be called when the match of the sig
is complete.
 9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse 
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
 9 years ago
Mats Klepsland ea5696812f detect: add tls_cert_notbefore and tls_cert_notafter keywords 
Detection plugin for TLS certificate fields notBefore and notAfter.

Supports equal to, less than, greater than, and range operations
for both keywords. Dates can be represented as either ISO 8601 or
epoch (Unix time).

Examples:
alert tls [...] tls_cert_notafter:1445852105; [...]
alert tls [...] tls_cert_notbefore:<2015-10-22T23:59:59; [...]
alert tls [...] tls_cert_notbefore:>2015-10-22; [...]
alert tls [...] tls_cert_notafter:2000-10-22<>2020-05-15; [...]
 9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Duarte Silva 89eb935f73 file-hashing: added support for SHA-256 file hashing 9 years ago
Duarte Silva a6d928e269 file-hashing: added support for SHA-1 file hashing 9 years ago
Mats Klepsland 4172c4c8ac tls: add (mpm) keyword tls_cert_subject 
This keyword is a replacement for tls.subject.
 9 years ago
Mats Klepsland 9b2717799c tls: add (mpm) keyword tls_cert_issuer 
This keyword is a replacement for tls.issuerdn.
 9 years ago
Victor Julien 99dce740ef detect: mark alproto in keyword reg deprecated 
No existing code uses it, and it had been useless for some time.
 9 years ago
Victor Julien 519b2970ec detect: don't print (null) in --list-keywords=all 9 years ago
Victor Julien 6b078e4f51 detect: fix ICMP error handling issue 
The first packet in both directions of a flow looks up the rule group
(sgh) and stores it in the flow. This makes sure the lookup doesn't
have to be performed for each packet.

ICMPv4 error messages are connected to the TCP or UDP flow they apply
to. In the case of such an ICMP error being the first packet in a
flow's direction, this would lead to an issue.

The packet would look up the rule group based on the ICMP protocol,
not based on the embedded TCP/UDP. This makes sense, as the ICMP
packet is inspected as ICMP packet. The consequence however, was that
this rule group pointer (sgh) would be stored in the flow. This is
wrong, as TCP/UDP packets that follow the ICMP packet would have no sgh
or the wrong sgh.

In normal traffic this shouldn't normally happen, but it could be
used to evade Suricata's inspection.
 9 years ago
Victor Julien 305b1b90fd detect: minor cleanup 9 years ago
Victor Julien b79d9cda3a detect: minor cleanups 9 years ago
Victor Julien 6045420812 detect: reduce verbosity, don't warn on empty files 9 years ago
Victor Julien b3bf7a5729 output: introduce config and perf output levels 
Goal is to reduce info output
 9 years ago
Victor Julien 4b9a62d1fe profiling: fix compilation if libjansson is missing 9 years ago
Victor Julien a8f257e05f detect: no longer a thread module 
Like stream, detect is now invoked directly by the FlowWorker.
 9 years ago
Victor Julien 52d500c670 flowworker: initial support 
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
 9 years ago
Victor Julien 408948815f detect: simplify flow locking 
To simplify locking, move all locking out of the individual detect
code. Instead at the start of detection lock the flow, and at the
end of detection unlock it.

The lua code can be called without a lock still (from the output
code paths), so still pass around a lock hint to take care of this.
 9 years ago
Victor Julien a81766c046 detect: split detect entry into flow/noflow 
This is a preparation for flow locking updates.
 9 years ago
Mats Klepsland a13df67864 detect: add (mpm) keyword for tls_sni 
Match on server name indication (SNI) extension in TLS using tls_sni
keyword, e.g:

alert tls any any -> any any (msg:"SNI test"; tls_sni;
        content:"example.com"; sid:12345;)
 9 years ago
Mats Klepsland e4f03b18ba tls: make TX aware 9 years ago
Zachary Rasmor f211fa48f4 Update unit test to account for 'name' type change. 9 years ago
maxtors 9d3fd82849 Removed duplicate include statements. 9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code 
1 pass, 0 is fail.
 9 years ago
Victor Julien 2ee3e726a9 detect: don't set unused variable 
detect.c:3801:13: warning: Value stored to 'tmplist2_tail' is never read
            tmplist2_tail = joingr;
            ^               ~~~~~~
detect.c:3804:13: warning: Value stored to 'tmplist2_tail' is never read
            tmplist2_tail = joingr;
            ^               ~~~~~~
2 warnings generated.
 9 years ago
Victor Julien d5c6d08bc8 detect-port: remove debug mem counters 9 years ago
Victor Julien 9bd8197009 detect-address: remove debug mem counters 9 years ago
Victor Julien e0111fbb90 detect grouping: remove debug mem counters 9 years ago
Victor Julien 5f676167a3 detect grouping: make json dump configurable 
Make the rule grouping dump to rule_group.json configurable.

detect:
  profiling:
    grouping:
      dump-to-disk: false
      include-rules: false      # very verbose
      include-mpm-stats: false
 9 years ago
Victor Julien d6ba01b1b7 detect: make port whitelisting configurable 
Make the port grouping whitelisting configurable. A whitelisted port
ends up in it's own port group.

detect:
  grouping:
    tcp-whitelist: 80, 443
    udp-whitelist: 53, 5060

No portranges are allowed at this point.
 9 years ago
Victor Julien ac2c206359 mpm: clean up builtin mpm setup, enable single/full 9 years ago
Victor Julien 6ef27c9f92 mpm: allow app buffer shared/unique 
Allow setting of shared or unique setting per app buffer type:
e.g. detect.mpm.http_uri.shared=true
 9 years ago
Victor Julien 79a96b2b90 mpm: refactor 'single' setup handling 9 years ago
Victor Julien 157ca89dd7 mpm: remove useless flag from factory 9 years ago
Victor Julien fdd05e8fb4 mpm: remove unused app proto factory 9 years ago
Victor Julien 2b84387ea4 detect: work around cocci limitation 9 years ago
Victor Julien 0311f01b97 rule grouping: speed up port based grouping 
Create a hash table of unique DetectPort objects before trying to
create a unique list of these objects. This safes a lot of cycles
in the creation of the list.
 9 years ago
Victor Julien 30755265ee http_raw_header: improve mpm progress handling 9 years ago
Victor Julien de273d88cc detect: remove unused content minlen tracking 9 years ago
Victor Julien 5857660568 http_uri: mpm cleanup. Use mpm_ctx's minlen 9 years ago
Victor Julien caea596ce5 profiling: output post-prefilter matches 
Dump a json record containing all sigs that need to be inspected after
prefilter. Part of profiling. Only dump if threshold is met, which is
currently set by:

 --set detect.profiling.inspect-logging-threshold=200

A file called packet_inspected_rules.json is created in the default
log dir.
 9 years ago
Victor Julien 92c571b26d detect: move sm_list to string funcs to parser code 9 years ago
Victor Julien 722e2dbf7c profiling: initial rulegroup tracking 
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.

Logs SGH id so it can be compared with the rule_group.json output.

Implemented both in a human readable text format and a JSON format.
 9 years ago
Victor Julien e310a033be detect: assign id to sgh 9 years ago
Victor Julien ba9d43cce5 mpm: improve negated mpm 
The idea is: if mpm is negated, it's both on mpm and nonmpm sid lists
and we can kick it out in that case during the merge sort.

It only works for patterns that are 'independent'. This means that the
rule doesn't need to only match if the negated mpm pattern is limited
to the first 10 bytes for example.

Or more generally, an negated mpm pattern that has depth, offset,
distance or within settings can't be handled this way. These patterns
are not added to the mpm at all, but just to to non-mpm list. This
makes sense as they will *always* need manual inspection.

Similarly, a pattern that is 'chopped' always needs validation. This
is because in this case we only inspect a part of the final pattern.
 9 years ago
Victor Julien c1ad08d11e detect: remove stream pmq array 9 years ago
Victor Julien c87fcb29ff detect mpm: fast_pattern assignment cleanup 9 years ago
Victor Julien a96fa0fc2f detect: remove unused dport sgh hash 9 years ago
Victor Julien e6248b0dbe detect: clean up sgh's at detect engine free 9 years ago
Victor Julien 1f7e33a4b0 detect: remove unused flag 9 years ago
Victor Julien 2ab20d0b9b detect: free lookup structures 9 years ago
Victor Julien 4ba1ac55f0 detect: output sgh stats 
Output stats for the rule groups into a json format.
 9 years ago
Victor Julien 810d2d3ec6 detect: add list id to string funcs 9 years ago
Victor Julien a34be23002 detect: simplify negated mpm handling 9 years ago
Victor Julien b84d6d402f detect grouping: multiple whitelist conditions 
Instead of the binary yes/no whitelisting used so far, use different
values for different sorts of whitelist reasons. The port list will
be sorted by whitelist value first, then by rule count.

The goal is to whitelist groups that have weak sigs:

 - 1 byte pattern groups

 - SYN sigs

    Rules that check for SYN packets are mostly scan detection rules.
    They will be checked often as SYN packets are very common.

    e.g. alert tcp any any -> any 22 (flags:S,12; sid:123;)

    This patch adds whitelisting for SYN-sigs, so that the sigs end up
    in as unique groups as possible.

 - negated mpm sigs

    Currently negated mpm sigs are inspected often, so they are quite
    expensive. For this reason, try to whitelist them.

These values are set during 'stage 1', rule preprocessing.
 9 years ago
Victor Julien 3c184c19cd detect grouping: port based group whitelisting 
Whitelist some ports in grouping to make sure they get their own group.
 9 years ago
Victor Julien 5772f526dc detect grouping: warn on and fix up bad sigs 
Only inspect directionless SYN scan sigs toserver. Issue a warning for
those rules.
 9 years ago
Victor Julien 2ce03fbabb detect: split non-mpm list into syn/nosyn 
Since SYN inspecting rules are expensive, this patch splits the
'non-mpm' list (i.e. the rules that are always considered) into
a 'syn' and 'non-syn' list. The SYN list is only inspected if the
packet has the SYN flag set, otherwise the non-syn list is used.

The syn-list contains _all_ rules. The non-syn list contains all
minus the rules requiring the SYN bit in a packet.
 9 years ago
Victor Julien 26517b8b61 detect: mpm store frees mpm_ctx' it owns 9 years ago
Victor Julien 102a82fc7b detect: use mpm store for app layer mpms 
Rework app-layer mpm setup and registration to make this possible.
 9 years ago
Victor Julien fac2cc0560 detect: mpm deduplication 
Create hash for mpm's that we can reuse. Have packet/stream mpms
use this.
 9 years ago
Victor Julien f0ba00e51d detect: remove old unused code 9 years ago
Victor Julien 2ee9bf2aef detect: rename groupings vars 9 years ago
Victor Julien ae80ed5964 detect: make port grouping use config limits 9 years ago
Victor Julien df529b13ce detect: change port grouping 
Update port grouping logic. Previously it would create one consistent
list w/o overlap. It largely still does this, except for the 'catch
all' port group at the end of the list. This port group contains all
the sigs that didn't fit into the other groups.
 9 years ago
Victor Julien a392812302 detect: sort/group port sigs 9 years ago
Victor Julien e570b10abe detect: display unique sgh count 9 years ago
Victor Julien eda9552e95 detect: group proto sghs 9 years ago
Victor Julien d82df4eb8b detect-mpm: make sgh setup proto aware 
Allow multi-proto, multi-direction sgh's.
 9 years ago
Victor Julien 4223ce9aba detect: remove obsolete grouping code 9 years ago
Victor Julien 9ae4cb9e02 detect: debug output 9 years ago
Victor Julien fd5a06017d detect: per port and proto rule grouping 
Replace tree based approach for rule grouping with a per port (tcp/udp)
and per protocol approach.

Grouping now looks like:

               +----+
               |icmp+--->
               +----+
               |gre +--->
               +----+
               |esp +--->
               +----+
          other|... |
        +----->-----+
        |      |N   +--->
        |      +----+
        |
        | tcp  +----+   +----+
        +----->+ 80 +-->+ 139+-->
        |      +----+   +----+
        |
        | udp  +----+   +----+
    +---+----->+ 53 +-->+ 135+-->
    |          +----+   +----+
    |toserver
+--->
    |toclient
    |
    +--->

So the first 'split' in the rules is the direction: toserver or toclient.
Rules that don't have a direction, are in both branches.

Then the split is between tcp/udp and the other protocols. For tcp and
udp port lists are used. For the other protocols, grouping is simply per
protocol.

The ports used are the destination ports for toserver sigs and source
ports for toclient sigs.
 9 years ago
Victor Julien 27e63a1e11 detect: track direction and ipproto of sgh 
Each SGH has a unique ipproto and direction.
 9 years ago
Victor Julien e75a93b125 detect: pass ipproto to rule grouping funcs 9 years ago
Victor Julien c71c991669 detect: delay sgh cleanup 9 years ago
Victor Julien bb662a65f8 detect: delay sgh mpm setup 9 years ago
Victor Julien 18dd54dfa7 Start rule inspect with mask check 9 years ago
Victor Julien 66b3dba676 detect: remove dead code 9 years ago
Victor Julien 4f8e1f59a6 mpm: remove obsolete mpm algos 
Remove: ac-gfbs, wumanber, b2g, b3g.
 9 years ago
Victor Julien 8394b38941 cppcheck: work around snprintf warning 
Cppcheck 1.72 gives a warning on the following code pattern:

    char blah[32] = "";
    snprintf(blah, sizeof(blah), "something");

The warning is:

    (error) Buffer is accessed out of bounds.

While this appears to be a FP, in most cases the initialization to ""
was unnecessary as the snprintf statement immediately follows the
variable declaration.
 10 years ago
Victor Julien a74a8c874f smtp: reset inspection buffers 10 years ago
Victor Julien d7d76e7b27 multi-detect: use default tenant 
The default detect engine can be used as 'default tenant'.
 10 years ago
Jason Ish 6b15686fd1 base64_decode, base64_data: decode and match base64 10 years ago
Victor Julien 6b2f831a70 mpm: SGH maxlen was actually minlen, so rename 10 years ago
Victor Julien 2716c78628 mpm: improve SGH content len tracking 
SGH's track content length for rule grouping.

This patch changes the logic to only consider the pattern that is
used in the mpm for a sig.
 10 years ago
Victor Julien e529ebb50e mpm: redo uri maxlen logic 
The mpm_uricontent_maxlen logic was meant to track the shortest
possible pattern in the MPM of a SGH. So a minlen more than a maxlen.

This patch replaces the complicated tracking logic by a simpler
scheme. When the SGH's are finalize, the minlen is calculated.

It also fixes a small corner case where the calculated "maxlen" could
be wrong. This would require a smaller pattern in a rule to be forced
as fast pattern.
 10 years ago
Victor Julien c53c9b4b20 mpm: remove bloated counting logic 
Counters were only used to print debug info.
 10 years ago
Victor Julien e67188e437 detect: fix issue with smsg and seq wraps 
Due to a broken sequence number check, detect could fail to process
smsgs in case of a sequence wrap. This could lead to excessive use
of smsg's but also of segments, since these aren't cleared until the
smsg containing them is.
 10 years ago
Jason Ish 06beca62f5 app-layer: template for application layer content inspection 10 years ago
Victor Julien cb060f16fd detect plugin: add template 10 years ago
Victor Julien 82aa419431 multi-detect: set tenant id on pseudo packets 
Store the tenant id in the flow and use the stored id when setting
up pesudo packets.

For tunnel and defrag packets, get tenant from parent. This will only
pass tenant_id's set at capture time.

For defrag packets, the tenant selector based on vlan id will still
work as the vlan id(s) are stored in the defrag tracker before being
passed on.
 10 years ago
Victor Julien e19c41a807 multi-detect: hash lookup for tenants 
Use hash for storing and looking up det_ctxs.
 10 years ago
Victor Julien f4c9915066 multi-detect: store tenant id in packet 
Store tenant id in the packet so that the output API's can log it.
 10 years ago
Victor Julien 93f856a1b3 detect: don't error out on no de_ctx 
This can happen on a multi-detect setup with no registered
engines yet.
 10 years ago
Victor Julien 1893c5edb1 multi-detect: initial selectors for tenants 
The Detection Thread has the TenantGetId pointer which allows it
to select a tenant id based on the packet.
 10 years ago
Victor Julien 0ff6d3dcfd detect: select detect engine at Detect entry 
Limited to Pcap only currently.
 10 years ago
Victor Julien def2b58725 detect: initial MT lookup logic 
In the DetectEngineThreadCtx, store another DetectEngineThreadCtx per
tenant.

Currently it's just a simple array indexed by the tenant id.
 10 years ago
Victor Julien 2ef0ebb24b detect: fix pass transaction handling 
If a flow was 'pass'd, it means that no packet of it will flow be handled
by the detection engine. A side effect of this was that the per flow
inspect_id would never be moved forward. This in turn lead to a situation
where transactions wouldn't be freed.

This patch addresses this case by incrementing the inspect_id anyway for
the pass case.
 10 years ago
Victor Julien 814f0b4094 detect: make http prefilter use disrupt flags 10 years ago
Victor Julien b592f98727 detect: optimize http prefilter handing 10 years ago
Victor Julien ea571add73 app-layer: disruption flags 
Stream GAPs and stream reassembly depth are tracked per direction. In
many cases they will happen in one direction, but not in the other.

Example:
HTTP requests a generally smaller than responses. So on the response
side we may hit the depth limit, but not on the request side.

The asynchronious 'disruption' has a side effect in the transaction
engine. The 'progress' tracking would never mark such transactions
as complete, and thus some inspection and logging wouldn't happen
until the very last moment: when EOF's are passed around.

Especially in proxy environments with _very_ many transactions in a
single TCP connection, this could lead to serious resource issues. The
EOF handling would suddenly have to handle thousands or more
transactions. These transactions would have been stored for a long time.

This patch introduces the concept of disruption flags. Flags passed to
the tx progress logic that are and indication of disruptions in the
traffic or the traffic handling. The idea is that the progress is
marked as complete on disruption, even if a tx is not complete. This
allows the detection and logging engines to process the tx after which
it can be cleaned up.
 10 years ago
Victor Julien 8125e04b39 detect: clean up flag usage 10 years ago
Victor Julien 6e2c90a83b detect: constify some DetectMpmPrefilter args 10 years ago
Victor Julien 62fa9f09d4 Sync alversion/appversion types 
The app layer state 'version' field is incremented with each update
to the state. It is used by the detection engine to see if the current
version of the state has already been inspected. Since app layer and
detect always run closely together there is no need for a big number
here. The detect code really only checks for equal/not-equal, so wrap
arounds are not an issue.
 10 years ago
Alexander Gozman d36eba4e5e Fix issues #1493 and #1494 10 years ago
Victor Julien 752f03e7a4 counters: remaining s/SCPerf/Stats/g 10 years ago
Victor Julien 8992275b0c counters: s/SCPerfCounterAddUI64/StatsAddUI64/g 10 years ago
Victor Julien 74ab84c194 counters: introduce SCPerfSetupPrivate for thread setup 10 years ago
Victor Julien 799640f906 counters: make threadvars::perf_private_ctx static 
Update SCPerfGetAllCountersArray and add a UT workaround.
 10 years ago
Victor Julien 55cfab89e4 counters: SCPerfGetLocalCounterValue cleanup 
Return u64, update arguments.
 10 years ago
Victor Julien e9b067c1eb counters: make increment call take threadvars 
This hides the implementation from the caller.
 10 years ago
Victor Julien 9a8bff7d96 counters: threadvars s/sc_perf_pca/perf_private_ctx/g 10 years ago
Victor Julien 50bb995458 counters: rename threadvars public counters 10 years ago
Alexander Gozman f11e237d77 Feature #1440: support wildcards in rule filenames 10 years ago
Victor Julien c7bc9ae6a8 detect: minor cleanups 10 years ago
Victor Julien c1558f5ac4 stream: remove FLOW_NO_APPLAYER_INSPECTION flag 
Instead, intruduce StreamTcpDisableAppLayer to disable app layer
tracking and reassembly. StreamTcpAppLayerIsDisabled can be used
to check it.

Replace all uses of FlowSetSessionNoApplayerInspectionFlag and
the FLOW_NO_APPLAYER_INSPECTION.
 10 years ago
Victor Julien 37f0bd57b6 detect-state: handle duplicate inspect/match 
If for a packet we have a TX N that has detect state and a TX N+1 that
has no detect state, but does have 'progress', we have a corner case
in stateful detection.

ContinueDetection inspects TX N, but cannot flag the rule in the
de_state_sig_array as the next (TX N+1) has already started and needs
to be inspected. 'StartDetection' however, is then unaware of the fact
that ContinueDetection already inspected the rule. It uses the per
session 'inspect_id' that is only moved forward at the end of the
detection run.

This patch adds a workaround. It uses the DetectEngineThreadCtx::
de_state_sig_array to store an offset between the 'base' inspect_id
and the inspect_id that StartDetection should use. The data type is
limited, so if the offset would be too big, a search based fall back
is implemented as well.
 10 years ago
Victor Julien 62e937672d detect-events: set SIG_MASK_REQUIRE_*_STATE for events 
Set SIG_MASK_REQUIRE_*_STATE for event rules to earlier discard
them.
 10 years ago
Giuseppe Longo 84dc73d9de mpm: implement prefiltering for smtp 10 years ago
Giuseppe Longo 41a1a9f4af find and replace HSBDMATCH by FILEDATA 
This commit do a find and replace of the following:

- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
  sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*

- HSBD by FILEDATA:
  sed -i 's/HSBDMATCH/FILEDATA/g' src/*
 10 years ago
Eric Leblond d98060410c list keywords: restore 2.0 similar output 
Unset was previously displayed if the application layer was not
set. Code before this patch was displaying '(null)' which is not
user friendly.
 10 years ago
Victor Julien dea8dda3c1 vars: redo var name <-> idx handling 
Can't use sm type anymore as multiple var carriers (host/flow/etc) will
all have xbits.
 10 years ago
Victor Julien 61cb2abc8d ippair: xbit implementation 
detect xbits for ippair: initial implementation
 10 years ago
Victor Julien 5c880377ae detect: hostbits keyword 
Per host bit similar to 'flowbits'.

Initial code that uses just the 'src' ip for the operations.
 10 years ago
Victor Julien 0704ece4d7 detect-reload: enable unconditionally 
Reloading is available unconditionally now.
 11 years ago
Victor Julien 7108085d33 detect: initialize detection engine by prefix 
Initalize detection engine by configuration prefix.

    DetectEngineCtxInitWithPrefix(const char *prefix)

Takes the detection engine configuration from:
<prefix>.<config>

If prefix is NULL the regular config will be used.

Update sure that DetectLoadCompleteSigPath considers the prefix when
retrieving the configuration.
 11 years ago
Victor Julien e250040b72 detect-state: implement tx state reset for reload 
In case of Detect Reload, we need to reset active tx' state.
 11 years ago
Victor Julien da3e8ad8f6 detect-state: split flow and tx state 
Use separate data structures for storing TX and FLOW (AMATCH) detect
state.

- move state storing into util funcs
- remove de_state_m
- simplify reset state logic on reload
 11 years ago
Victor Julien e9857200b3 detect: set action from utility function 
Set actions that are set directly from Signatures using the new
utility function DetectSignatureApplyActions. This will apply
the actions and also store info about the 'drop' that first made
the rule drop.
 11 years ago
Jason Ish 6ed246c041 Don't attempt to load the rule files if the rule-files configuration 
node is not a sequence.  Instead log a warning as this is usually
a configuration error.
 11 years ago
Victor Julien f1f5428faa detect: expand mask checking 
Change mask to u16, and add checks for various protocol states
that need to be present for a rule to be considered.
 11 years ago
Victor Julien ca59eabca3 detect: introduce DetectPrefilterBuildNonMpmList 
Move building of non-mpm list into a separate function, that is inlined
for performance reasons.
 11 years ago
Victor Julien cc4f7a4b96 detect: add profiling for non-mpm list build & filter 11 years ago
Victor Julien 4c10635dc1 detect: optimize non-mpm mask checking 
Store id and mask in a single array of type SignatureNonMpmStore so
that both are loaded into the same cache line.
 11 years ago
Victor Julien b5a3127151 detect: add mask check prefilter for non mpm list 
Add mask array for non_mpm sigs, so that we can exclude many sigs before
we merge sort.

Shows 50% less non mpm sigs inspected on average.
 11 years ago
Ken Steele 904441327c Conditionalize SigMatch performance counters. 
Only include the counters when PROFILING.
 11 years ago
Victor Julien 30b7fdcb49 Detect perf counters 11 years ago
Victor Julien ef6875d583 detect: Disable unused SignatureHeader code 11 years ago
Ken Steele 4bd280f196 Indentation clean up 11 years ago
Ken Steele 403b5a4645 Further optimize merging mpm and non-mpm rule ID lists. 
When reaching the end of either list, merging is no longer required,
simply walk down the other list.

If the non-MPM list can't have duplicates, it would be worth removing
the duplicate check for the non-MPM list when it is the only non-empty list
remaining.
 11 years ago
Ken Steele 86f4c6c47b Custom Quick Sort for Signature IDs 
Use an in place Quick Sort instead of qsort(), which does merge sort and
calls memcpy().

Improves performance on my tests.
 11 years ago
Ken Steele 736ac6a459 Use SigIntId as the type for storing signature IDs (Internal) 
Previously using uint32_t, but SigIntId is currently uint16_t, so arrays
will take less memory.
 11 years ago
Ken Steele 1874784c10 Create optimized sig_arrays from sig_lists 
Create a copy of the SigMatch data in the sig_lists linked-lists and store
it in an array for faster access and not next and previous pointers. The
array is then used when calling the Match() functions.

Gives a 7.7% speed up on one test.
 11 years ago
Ken Steele 923a77e952 Change Match() function to take const SigMatchCtx* 
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.

Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.

The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
 11 years ago
Ken Steele 1c76fa50b1 Prefetch the next signature pointer 
Read one signature pointer ahead to prefetch the value.
Use a variable, sflags, for s->flags, since it is used many times and the
compiles doesn't know that the signatures structure doesn't change, so it
will reload s->flags.
 11 years ago
Ken Steele d31db4ed1c Fix clang warning 
Clang doesn't seem to like defining a function within a function.
 11 years ago
Ken Steele 23d2a1c422 Optimize DetectPrefilterMergeSort 
Fixup rebase changes to remove debug code
 11 years ago
Victor Julien 7876277119 detect: move checks from prefilter to rule detect 
Move the prefilter checks to the main detect loop.
 11 years ago
Victor Julien d1d895a884 Replace build match array with new filter logic 
Use MPM and non-MPM lists to build our match array. Both lists are
sorted, and are merged and sorted into the match array.

This disables the old match array building code and thus also bypasses
the mask checking.
 11 years ago
Victor Julien f5df526f9b Detect: create per sgh non-MPM rule array 
Array of rule id's that are not using MPM prefiltering. These will be
merged with the MPM results array. Together these should lead to a
list of all the rules that can possibly match.
 11 years ago
Victor Julien e49d0a5924 MPM: build sid list from MPM matches 
Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.

AC: store sids in the pattern list, append to Pmq::rule_id_array on match.

Detect: sort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.
 11 years ago
Victor Julien 43a1007788 detect: add test for memcmp issue 11 years ago
Victor Julien d951de2f19 Bug 977: -T / --init-errors-fatal to process all rules 
Have -T / --init-errors-fatal process all rules so that it's easier
to debug problems in ruleset. Otherwise it can be a lengthy fix, test
error cycle if multiple rules have issues.

Convert empty rulefile error into a warning.

Bug #977
 11 years ago
DIALLO David b3bf2f9939 Detect: Add Modbus keyword management 
Add the modbus.function and subfunction) keywords for public function match in rules (Modbus layer).
Matching based on code function, and if necessary, sub-function code
or based on category (assigned, unassigned, public, user or reserved)
and negation is permitted.

Add the modbus.access keyword for read/write Modbus function match in rules (Modbus layer).
Matching based on access type (read or write),
and/or function type (discretes, coils, input or holding)
and, if necessary, read or write address access,
and, if necessary, value to write.
For address and value matching, "<", ">" and "<>" is permitted.

Based on TLS source code and file size source code (address and value matching).

Signed-off-by: David DIALLO <diallo@et.esia.fr>
 11 years ago
Anoop Saldanha b334b8a6e9 CUDA: Update the inspection engine to inform the cuda module that it 
doesn't need the gpu results and to release the packet for the next run.

Previously the inspection engine wouldn't inform the cuda module, if it
didn't need the results.  As a consequence, when the packet is next taken
for re-use, and if the packet is still being processed by the cuda module,
the engine would wait till the cuda module frees the packet.

This commits updates this functionality to inform the cuda module to
release the packet for the afore-mentioned case.
 11 years ago