Commit Graph

934 Commits (8547d113bfe6b44f188ff72194c31e407d5ae42c)

Author SHA1 Message Date
Victor Julien b62c4cc359 http_uri: mpm prefilter engine
Inspect partial request line as well.
9 years ago
Victor Julien 5bcdbe3922 prefilter: introduce prefilter engines
Introduce abstraction layer for prefilter engines.
9 years ago
Victor Julien 17bc0299fe detect: rename non_mpm lists/vars to non_pf
Rename to non_pf: non prefilter.
9 years ago
Victor Julien ff70e0cca0 mpm tls: remove unused function args 9 years ago
Victor Julien ad3a55d938 mpm dns query: remove unused function args 9 years ago
Victor Julien d647db1775 mpm stat code: remove unused function args 9 years ago
Victor Julien bd03307921 mpm stat msg: remove unused function args 9 years ago
Victor Julien 6d54b70db4 mpm ua: remove unused function args 9 years ago
Victor Julien 704afeb078 mpm cookie: remove unused function args 9 years ago
Victor Julien 4229e603f0 mpm raw host: remove unused function args 9 years ago
Victor Julien 1380853ee8 mpm host: remove unused function args 9 years ago
Victor Julien b40ecb7356 mpm method: remove unused function args 9 years ago
Victor Julien 3d5807ba44 mpm raw uri: remove unused function args 9 years ago
Victor Julien d461c7888a mpm uri: remove unused function args 9 years ago
Giuseppe Longo 07564c4e41 detect: add bypass keyword
This adds a new keyword which permits to call the
bypass callback when a sig is matched.

The callback must be called when the match of the sig
is complete.
9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
9 years ago
Mats Klepsland ea5696812f detect: add tls_cert_notbefore and tls_cert_notafter keywords
Detection plugin for TLS certificate fields notBefore and notAfter.

Supports equal to, less than, greater than, and range operations
for both keywords. Dates can be represented as either ISO 8601 or
epoch (Unix time).

Examples:
alert tls [...] tls_cert_notafter:1445852105; [...]
alert tls [...] tls_cert_notbefore:<2015-10-22T23:59:59; [...]
alert tls [...] tls_cert_notbefore:>2015-10-22; [...]
alert tls [...] tls_cert_notafter:2000-10-22<>2020-05-15; [...]
9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Duarte Silva 89eb935f73 file-hashing: added support for SHA-256 file hashing 9 years ago
Duarte Silva a6d928e269 file-hashing: added support for SHA-1 file hashing 9 years ago
Mats Klepsland 4172c4c8ac tls: add (mpm) keyword tls_cert_subject
This keyword is a replacement for tls.subject.
9 years ago
Mats Klepsland 9b2717799c tls: add (mpm) keyword tls_cert_issuer
This keyword is a replacement for tls.issuerdn.
9 years ago
Victor Julien 99dce740ef detect: mark alproto in keyword reg deprecated
No existing code uses it, and it had been useless for some time.
9 years ago
Victor Julien 519b2970ec detect: don't print (null) in --list-keywords=all 9 years ago
Victor Julien 6b078e4f51 detect: fix ICMP error handling issue
The first packet in both directions of a flow looks up the rule group
(sgh) and stores it in the flow. This makes sure the lookup doesn't
have to be performed for each packet.

ICMPv4 error messages are connected to the TCP or UDP flow they apply
to. In the case of such an ICMP error being the first packet in a
flow's direction, this would lead to an issue.

The packet would look up the rule group based on the ICMP protocol,
not based on the embedded TCP/UDP. This makes sense, as the ICMP
packet is inspected as ICMP packet. The consequence however, was that
this rule group pointer (sgh) would be stored in the flow. This is
wrong, as TCP/UDP packets that follow the ICMP packet would have no sgh
or the wrong sgh.

In normal traffic this shouldn't normally happen, but it could be
used to evade Suricata's inspection.
9 years ago
Victor Julien 305b1b90fd detect: minor cleanup 9 years ago
Victor Julien b79d9cda3a detect: minor cleanups 9 years ago
Victor Julien 6045420812 detect: reduce verbosity, don't warn on empty files 9 years ago
Victor Julien b3bf7a5729 output: introduce config and perf output levels
Goal is to reduce info output
9 years ago
Victor Julien 4b9a62d1fe profiling: fix compilation if libjansson is missing 9 years ago
Victor Julien a8f257e05f detect: no longer a thread module
Like stream, detect is now invoked directly by the FlowWorker.
9 years ago
Victor Julien 52d500c670 flowworker: initial support
Initial version of the 'FlowWorker' thread module. This module
combines Flow handling, TCP handling, App layer handling and
Detection in a single module. It does all flow related processing
under a single flow lock.
9 years ago
Victor Julien 408948815f detect: simplify flow locking
To simplify locking, move all locking out of the individual detect
code. Instead at the start of detection lock the flow, and at the
end of detection unlock it.

The lua code can be called without a lock still (from the output
code paths), so still pass around a lock hint to take care of this.
9 years ago
Victor Julien a81766c046 detect: split detect entry into flow/noflow
This is a preparation for flow locking updates.
9 years ago
Mats Klepsland a13df67864 detect: add (mpm) keyword for tls_sni
Match on server name indication (SNI) extension in TLS using tls_sni
keyword, e.g:

alert tls any any -> any any (msg:"SNI test"; tls_sni;
        content:"example.com"; sid:12345;)
9 years ago
Mats Klepsland e4f03b18ba tls: make TX aware 9 years ago
Zachary Rasmor f211fa48f4 Update unit test to account for 'name' type change. 9 years ago
maxtors 9d3fd82849 Removed duplicate include statements. 9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
9 years ago
Victor Julien 2ee3e726a9 detect: don't set unused variable
detect.c:3801:13: warning: Value stored to 'tmplist2_tail' is never read
            tmplist2_tail = joingr;
            ^               ~~~~~~
detect.c:3804:13: warning: Value stored to 'tmplist2_tail' is never read
            tmplist2_tail = joingr;
            ^               ~~~~~~
2 warnings generated.
9 years ago
Victor Julien d5c6d08bc8 detect-port: remove debug mem counters 9 years ago
Victor Julien 9bd8197009 detect-address: remove debug mem counters 9 years ago
Victor Julien e0111fbb90 detect grouping: remove debug mem counters 9 years ago
Victor Julien 5f676167a3 detect grouping: make json dump configurable
Make the rule grouping dump to rule_group.json configurable.

detect:
  profiling:
    grouping:
      dump-to-disk: false
      include-rules: false      # very verbose
      include-mpm-stats: false
9 years ago
Victor Julien d6ba01b1b7 detect: make port whitelisting configurable
Make the port grouping whitelisting configurable. A whitelisted port
ends up in it's own port group.

detect:
  grouping:
    tcp-whitelist: 80, 443
    udp-whitelist: 53, 5060

No portranges are allowed at this point.
9 years ago
Victor Julien ac2c206359 mpm: clean up builtin mpm setup, enable single/full 9 years ago
Victor Julien 6ef27c9f92 mpm: allow app buffer shared/unique
Allow setting of shared or unique setting per app buffer type:
e.g. detect.mpm.http_uri.shared=true
9 years ago
Victor Julien 79a96b2b90 mpm: refactor 'single' setup handling 9 years ago
Victor Julien 157ca89dd7 mpm: remove useless flag from factory 9 years ago
Victor Julien fdd05e8fb4 mpm: remove unused app proto factory 9 years ago
Victor Julien 2b84387ea4 detect: work around cocci limitation 9 years ago
Victor Julien 0311f01b97 rule grouping: speed up port based grouping
Create a hash table of unique DetectPort objects before trying to
create a unique list of these objects. This safes a lot of cycles
in the creation of the list.
9 years ago
Victor Julien 30755265ee http_raw_header: improve mpm progress handling 9 years ago
Victor Julien de273d88cc detect: remove unused content minlen tracking 9 years ago
Victor Julien 5857660568 http_uri: mpm cleanup. Use mpm_ctx's minlen 9 years ago
Victor Julien caea596ce5 profiling: output post-prefilter matches
Dump a json record containing all sigs that need to be inspected after
prefilter. Part of profiling. Only dump if threshold is met, which is
currently set by:

 --set detect.profiling.inspect-logging-threshold=200

A file called packet_inspected_rules.json is created in the default
log dir.
9 years ago
Victor Julien 92c571b26d detect: move sm_list to string funcs to parser code 9 years ago
Victor Julien 722e2dbf7c profiling: initial rulegroup tracking
Per rule group tracking of checks, use of lists, mpm matches,
post filter counts.

Logs SGH id so it can be compared with the rule_group.json output.

Implemented both in a human readable text format and a JSON format.
9 years ago
Victor Julien e310a033be detect: assign id to sgh 9 years ago
Victor Julien ba9d43cce5 mpm: improve negated mpm
The idea is: if mpm is negated, it's both on mpm and nonmpm sid lists
and we can kick it out in that case during the merge sort.

It only works for patterns that are 'independent'. This means that the
rule doesn't need to only match if the negated mpm pattern is limited
to the first 10 bytes for example.

Or more generally, an negated mpm pattern that has depth, offset,
distance or within settings can't be handled this way. These patterns
are not added to the mpm at all, but just to to non-mpm list. This
makes sense as they will *always* need manual inspection.

Similarly, a pattern that is 'chopped' always needs validation. This
is because in this case we only inspect a part of the final pattern.
9 years ago
Victor Julien c1ad08d11e detect: remove stream pmq array 9 years ago
Victor Julien c87fcb29ff detect mpm: fast_pattern assignment cleanup 9 years ago
Victor Julien a96fa0fc2f detect: remove unused dport sgh hash 9 years ago
Victor Julien e6248b0dbe detect: clean up sgh's at detect engine free 9 years ago
Victor Julien 1f7e33a4b0 detect: remove unused flag 9 years ago
Victor Julien 2ab20d0b9b detect: free lookup structures 9 years ago
Victor Julien 4ba1ac55f0 detect: output sgh stats
Output stats for the rule groups into a json format.
9 years ago
Victor Julien 810d2d3ec6 detect: add list id to string funcs 9 years ago
Victor Julien a34be23002 detect: simplify negated mpm handling 9 years ago
Victor Julien b84d6d402f detect grouping: multiple whitelist conditions
Instead of the binary yes/no whitelisting used so far, use different
values for different sorts of whitelist reasons. The port list will
be sorted by whitelist value first, then by rule count.

The goal is to whitelist groups that have weak sigs:

 - 1 byte pattern groups

 - SYN sigs

    Rules that check for SYN packets are mostly scan detection rules.
    They will be checked often as SYN packets are very common.

    e.g. alert tcp any any -> any 22 (flags:S,12; sid:123;)

    This patch adds whitelisting for SYN-sigs, so that the sigs end up
    in as unique groups as possible.

 - negated mpm sigs

    Currently negated mpm sigs are inspected often, so they are quite
    expensive. For this reason, try to whitelist them.

These values are set during 'stage 1', rule preprocessing.
9 years ago
Victor Julien 3c184c19cd detect grouping: port based group whitelisting
Whitelist some ports in grouping to make sure they get their own group.
9 years ago
Victor Julien 5772f526dc detect grouping: warn on and fix up bad sigs
Only inspect directionless SYN scan sigs toserver. Issue a warning for
those rules.
9 years ago
Victor Julien 2ce03fbabb detect: split non-mpm list into syn/nosyn
Since SYN inspecting rules are expensive, this patch splits the
'non-mpm' list (i.e. the rules that are always considered) into
a 'syn' and 'non-syn' list. The SYN list is only inspected if the
packet has the SYN flag set, otherwise the non-syn list is used.

The syn-list contains _all_ rules. The non-syn list contains all
minus the rules requiring the SYN bit in a packet.
9 years ago
Victor Julien 26517b8b61 detect: mpm store frees mpm_ctx' it owns 9 years ago
Victor Julien 102a82fc7b detect: use mpm store for app layer mpms
Rework app-layer mpm setup and registration to make this possible.
9 years ago
Victor Julien fac2cc0560 detect: mpm deduplication
Create hash for mpm's that we can reuse. Have packet/stream mpms
use this.
9 years ago
Victor Julien f0ba00e51d detect: remove old unused code 9 years ago
Victor Julien 2ee9bf2aef detect: rename groupings vars 9 years ago
Victor Julien ae80ed5964 detect: make port grouping use config limits 9 years ago
Victor Julien df529b13ce detect: change port grouping
Update port grouping logic. Previously it would create one consistent
list w/o overlap. It largely still does this, except for the 'catch
all' port group at the end of the list. This port group contains all
the sigs that didn't fit into the other groups.
9 years ago
Victor Julien a392812302 detect: sort/group port sigs 9 years ago
Victor Julien e570b10abe detect: display unique sgh count 9 years ago
Victor Julien eda9552e95 detect: group proto sghs 9 years ago
Victor Julien d82df4eb8b detect-mpm: make sgh setup proto aware
Allow multi-proto, multi-direction sgh's.
9 years ago
Victor Julien 4223ce9aba detect: remove obsolete grouping code 9 years ago
Victor Julien 9ae4cb9e02 detect: debug output 9 years ago
Victor Julien fd5a06017d detect: per port and proto rule grouping
Replace tree based approach for rule grouping with a per port (tcp/udp)
and per protocol approach.

Grouping now looks like:

               +----+
               |icmp+--->
               +----+
               |gre +--->
               +----+
               |esp +--->
               +----+
          other|... |
        +----->-----+
        |      |N   +--->
        |      +----+
        |
        | tcp  +----+   +----+
        +----->+ 80 +-->+ 139+-->
        |      +----+   +----+
        |
        | udp  +----+   +----+
    +---+----->+ 53 +-->+ 135+-->
    |          +----+   +----+
    |toserver
+--->
    |toclient
    |
    +--->

So the first 'split' in the rules is the direction: toserver or toclient.
Rules that don't have a direction, are in both branches.

Then the split is between tcp/udp and the other protocols. For tcp and
udp port lists are used. For the other protocols, grouping is simply per
protocol.

The ports used are the destination ports for toserver sigs and source
ports for toclient sigs.
9 years ago
Victor Julien 27e63a1e11 detect: track direction and ipproto of sgh
Each SGH has a unique ipproto and direction.
9 years ago
Victor Julien e75a93b125 detect: pass ipproto to rule grouping funcs 9 years ago
Victor Julien c71c991669 detect: delay sgh cleanup 9 years ago
Victor Julien bb662a65f8 detect: delay sgh mpm setup 9 years ago
Victor Julien 18dd54dfa7 Start rule inspect with mask check 9 years ago
Victor Julien 66b3dba676 detect: remove dead code 9 years ago
Victor Julien 4f8e1f59a6 mpm: remove obsolete mpm algos
Remove: ac-gfbs, wumanber, b2g, b3g.
9 years ago
Victor Julien 8394b38941 cppcheck: work around snprintf warning
Cppcheck 1.72 gives a warning on the following code pattern:

    char blah[32] = "";
    snprintf(blah, sizeof(blah), "something");

The warning is:

    (error) Buffer is accessed out of bounds.

While this appears to be a FP, in most cases the initialization to ""
was unnecessary as the snprintf statement immediately follows the
variable declaration.
9 years ago
Victor Julien a74a8c874f smtp: reset inspection buffers 10 years ago
Victor Julien d7d76e7b27 multi-detect: use default tenant
The default detect engine can be used as 'default tenant'.
10 years ago
Jason Ish 6b15686fd1 base64_decode, base64_data: decode and match base64 10 years ago
Victor Julien 6b2f831a70 mpm: SGH maxlen was actually minlen, so rename 10 years ago
Victor Julien 2716c78628 mpm: improve SGH content len tracking
SGH's track content length for rule grouping.

This patch changes the logic to only consider the pattern that is
used in the mpm for a sig.
10 years ago
Victor Julien e529ebb50e mpm: redo uri maxlen logic
The mpm_uricontent_maxlen logic was meant to track the shortest
possible pattern in the MPM of a SGH. So a minlen more than a maxlen.

This patch replaces the complicated tracking logic by a simpler
scheme. When the SGH's are finalize, the minlen is calculated.

It also fixes a small corner case where the calculated "maxlen" could
be wrong. This would require a smaller pattern in a rule to be forced
as fast pattern.
10 years ago
Victor Julien c53c9b4b20 mpm: remove bloated counting logic
Counters were only used to print debug info.
10 years ago
Victor Julien e67188e437 detect: fix issue with smsg and seq wraps
Due to a broken sequence number check, detect could fail to process
smsgs in case of a sequence wrap. This could lead to excessive use
of smsg's but also of segments, since these aren't cleared until the
smsg containing them is.
10 years ago
Jason Ish 06beca62f5 app-layer: template for application layer content inspection 10 years ago
Victor Julien cb060f16fd detect plugin: add template 10 years ago
Victor Julien 82aa419431 multi-detect: set tenant id on pseudo packets
Store the tenant id in the flow and use the stored id when setting
up pesudo packets.

For tunnel and defrag packets, get tenant from parent. This will only
pass tenant_id's set at capture time.

For defrag packets, the tenant selector based on vlan id will still
work as the vlan id(s) are stored in the defrag tracker before being
passed on.
10 years ago
Victor Julien e19c41a807 multi-detect: hash lookup for tenants
Use hash for storing and looking up det_ctxs.
10 years ago
Victor Julien f4c9915066 multi-detect: store tenant id in packet
Store tenant id in the packet so that the output API's can log it.
10 years ago
Victor Julien 93f856a1b3 detect: don't error out on no de_ctx
This can happen on a multi-detect setup with no registered
engines yet.
10 years ago
Victor Julien 1893c5edb1 multi-detect: initial selectors for tenants
The Detection Thread has the TenantGetId pointer which allows it
to select a tenant id based on the packet.
10 years ago
Victor Julien 0ff6d3dcfd detect: select detect engine at Detect entry
Limited to Pcap only currently.
10 years ago
Victor Julien def2b58725 detect: initial MT lookup logic
In the DetectEngineThreadCtx, store another DetectEngineThreadCtx per
tenant.

Currently it's just a simple array indexed by the tenant id.
10 years ago
Victor Julien 2ef0ebb24b detect: fix pass transaction handling
If a flow was 'pass'd, it means that no packet of it will flow be handled
by the detection engine. A side effect of this was that the per flow
inspect_id would never be moved forward. This in turn lead to a situation
where transactions wouldn't be freed.

This patch addresses this case by incrementing the inspect_id anyway for
the pass case.
10 years ago
Victor Julien 814f0b4094 detect: make http prefilter use disrupt flags 10 years ago
Victor Julien b592f98727 detect: optimize http prefilter handing 10 years ago
Victor Julien ea571add73 app-layer: disruption flags
Stream GAPs and stream reassembly depth are tracked per direction. In
many cases they will happen in one direction, but not in the other.

Example:
HTTP requests a generally smaller than responses. So on the response
side we may hit the depth limit, but not on the request side.

The asynchronious 'disruption' has a side effect in the transaction
engine. The 'progress' tracking would never mark such transactions
as complete, and thus some inspection and logging wouldn't happen
until the very last moment: when EOF's are passed around.

Especially in proxy environments with _very_ many transactions in a
single TCP connection, this could lead to serious resource issues. The
EOF handling would suddenly have to handle thousands or more
transactions. These transactions would have been stored for a long time.

This patch introduces the concept of disruption flags. Flags passed to
the tx progress logic that are and indication of disruptions in the
traffic or the traffic handling. The idea is that the progress is
marked as complete on disruption, even if a tx is not complete. This
allows the detection and logging engines to process the tx after which
it can be cleaned up.
10 years ago
Victor Julien 8125e04b39 detect: clean up flag usage 10 years ago
Victor Julien 6e2c90a83b detect: constify some DetectMpmPrefilter args 10 years ago
Victor Julien 62fa9f09d4 Sync alversion/appversion types
The app layer state 'version' field is incremented with each update
to the state. It is used by the detection engine to see if the current
version of the state has already been inspected. Since app layer and
detect always run closely together there is no need for a big number
here. The detect code really only checks for equal/not-equal, so wrap
arounds are not an issue.
10 years ago
Alexander Gozman d36eba4e5e Fix issues #1493 and #1494 10 years ago
Victor Julien 752f03e7a4 counters: remaining s/SCPerf/Stats/g 10 years ago
Victor Julien 8992275b0c counters: s/SCPerfCounterAddUI64/StatsAddUI64/g 10 years ago
Victor Julien 74ab84c194 counters: introduce SCPerfSetupPrivate for thread setup 10 years ago
Victor Julien 799640f906 counters: make threadvars::perf_private_ctx static
Update SCPerfGetAllCountersArray and add a UT workaround.
10 years ago
Victor Julien 55cfab89e4 counters: SCPerfGetLocalCounterValue cleanup
Return u64, update arguments.
10 years ago
Victor Julien e9b067c1eb counters: make increment call take threadvars
This hides the implementation from the caller.
10 years ago
Victor Julien 9a8bff7d96 counters: threadvars s/sc_perf_pca/perf_private_ctx/g 10 years ago
Victor Julien 50bb995458 counters: rename threadvars public counters 10 years ago
Alexander Gozman f11e237d77 Feature #1440: support wildcards in rule filenames 10 years ago
Victor Julien c7bc9ae6a8 detect: minor cleanups 10 years ago
Victor Julien c1558f5ac4 stream: remove FLOW_NO_APPLAYER_INSPECTION flag
Instead, intruduce StreamTcpDisableAppLayer to disable app layer
tracking and reassembly. StreamTcpAppLayerIsDisabled can be used
to check it.

Replace all uses of FlowSetSessionNoApplayerInspectionFlag and
the FLOW_NO_APPLAYER_INSPECTION.
10 years ago
Victor Julien 37f0bd57b6 detect-state: handle duplicate inspect/match
If for a packet we have a TX N that has detect state and a TX N+1 that
has no detect state, but does have 'progress', we have a corner case
in stateful detection.

ContinueDetection inspects TX N, but cannot flag the rule in the
de_state_sig_array as the next (TX N+1) has already started and needs
to be inspected. 'StartDetection' however, is then unaware of the fact
that ContinueDetection already inspected the rule. It uses the per
session 'inspect_id' that is only moved forward at the end of the
detection run.

This patch adds a workaround. It uses the DetectEngineThreadCtx::
de_state_sig_array to store an offset between the 'base' inspect_id
and the inspect_id that StartDetection should use. The data type is
limited, so if the offset would be too big, a search based fall back
is implemented as well.
10 years ago
Victor Julien 62e937672d detect-events: set SIG_MASK_REQUIRE_*_STATE for events
Set SIG_MASK_REQUIRE_*_STATE for event rules to earlier discard
them.
10 years ago
Giuseppe Longo 84dc73d9de mpm: implement prefiltering for smtp 10 years ago
Giuseppe Longo 41a1a9f4af find and replace HSBDMATCH by FILEDATA
This commit do a find and replace of the following:

- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
  sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*

- HSBD by FILEDATA:
  sed -i 's/HSBDMATCH/FILEDATA/g' src/*
10 years ago
Eric Leblond d98060410c list keywords: restore 2.0 similar output
Unset was previously displayed if the application layer was not
set. Code before this patch was displaying '(null)' which is not
user friendly.
10 years ago
Victor Julien dea8dda3c1 vars: redo var name <-> idx handling
Can't use sm type anymore as multiple var carriers (host/flow/etc) will
all have xbits.
10 years ago
Victor Julien 61cb2abc8d ippair: xbit implementation
detect xbits for ippair: initial implementation
10 years ago
Victor Julien 5c880377ae detect: hostbits keyword
Per host bit similar to 'flowbits'.

Initial code that uses just the 'src' ip for the operations.
10 years ago
Victor Julien 0704ece4d7 detect-reload: enable unconditionally
Reloading is available unconditionally now.
11 years ago
Victor Julien 7108085d33 detect: initialize detection engine by prefix
Initalize detection engine by configuration prefix.

    DetectEngineCtxInitWithPrefix(const char *prefix)

Takes the detection engine configuration from:
<prefix>.<config>

If prefix is NULL the regular config will be used.

Update sure that DetectLoadCompleteSigPath considers the prefix when
retrieving the configuration.
11 years ago
Victor Julien e250040b72 detect-state: implement tx state reset for reload
In case of Detect Reload, we need to reset active tx' state.
11 years ago
Victor Julien da3e8ad8f6 detect-state: split flow and tx state
Use separate data structures for storing TX and FLOW (AMATCH) detect
state.

- move state storing into util funcs
- remove de_state_m
- simplify reset state logic on reload
11 years ago
Victor Julien e9857200b3 detect: set action from utility function
Set actions that are set directly from Signatures using the new
utility function DetectSignatureApplyActions. This will apply
the actions and also store info about the 'drop' that first made
the rule drop.
11 years ago
Jason Ish 6ed246c041 Don't attempt to load the rule files if the rule-files configuration
node is not a sequence.  Instead log a warning as this is usually
a configuration error.
11 years ago
Victor Julien f1f5428faa detect: expand mask checking
Change mask to u16, and add checks for various protocol states
that need to be present for a rule to be considered.
11 years ago
Victor Julien ca59eabca3 detect: introduce DetectPrefilterBuildNonMpmList
Move building of non-mpm list into a separate function, that is inlined
for performance reasons.
11 years ago
Victor Julien cc4f7a4b96 detect: add profiling for non-mpm list build & filter 11 years ago
Victor Julien 4c10635dc1 detect: optimize non-mpm mask checking
Store id and mask in a single array of type SignatureNonMpmStore so
that both are loaded into the same cache line.
11 years ago
Victor Julien b5a3127151 detect: add mask check prefilter for non mpm list
Add mask array for non_mpm sigs, so that we can exclude many sigs before
we merge sort.

Shows 50% less non mpm sigs inspected on average.
11 years ago
Ken Steele 904441327c Conditionalize SigMatch performance counters.
Only include the counters when PROFILING.
11 years ago
Victor Julien 30b7fdcb49 Detect perf counters 11 years ago
Victor Julien ef6875d583 detect: Disable unused SignatureHeader code 11 years ago
Ken Steele 4bd280f196 Indentation clean up 11 years ago
Ken Steele 403b5a4645 Further optimize merging mpm and non-mpm rule ID lists.
When reaching the end of either list, merging is no longer required,
simply walk down the other list.

If the non-MPM list can't have duplicates, it would be worth removing
the duplicate check for the non-MPM list when it is the only non-empty list
remaining.
11 years ago
Ken Steele 86f4c6c47b Custom Quick Sort for Signature IDs
Use an in place Quick Sort instead of qsort(), which does merge sort and
calls memcpy().

Improves performance on my tests.
11 years ago
Ken Steele 736ac6a459 Use SigIntId as the type for storing signature IDs (Internal)
Previously using uint32_t, but SigIntId is currently uint16_t, so arrays
will take less memory.
11 years ago
Ken Steele 1874784c10 Create optimized sig_arrays from sig_lists
Create a copy of the SigMatch data in the sig_lists linked-lists and store
it in an array for faster access and not next and previous pointers. The
array is then used when calling the Match() functions.

Gives a 7.7% speed up on one test.
11 years ago
Ken Steele 923a77e952 Change Match() function to take const SigMatchCtx*
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.

Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.

The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
11 years ago
Ken Steele 1c76fa50b1 Prefetch the next signature pointer
Read one signature pointer ahead to prefetch the value.
Use a variable, sflags, for s->flags, since it is used many times and the
compiles doesn't know that the signatures structure doesn't change, so it
will reload s->flags.
11 years ago
Ken Steele d31db4ed1c Fix clang warning
Clang doesn't seem to like defining a function within a function.
11 years ago
Ken Steele 23d2a1c422 Optimize DetectPrefilterMergeSort
Fixup rebase changes to remove debug code
11 years ago
Victor Julien 7876277119 detect: move checks from prefilter to rule detect
Move the prefilter checks to the main detect loop.
11 years ago
Victor Julien d1d895a884 Replace build match array with new filter logic
Use MPM and non-MPM lists to build our match array. Both lists are
sorted, and are merged and sorted into the match array.

This disables the old match array building code and thus also bypasses
the mask checking.
11 years ago
Victor Julien f5df526f9b Detect: create per sgh non-MPM rule array
Array of rule id's that are not using MPM prefiltering. These will be
merged with the MPM results array. Together these should lead to a
list of all the rules that can possibly match.
11 years ago
Victor Julien e49d0a5924 MPM: build sid list from MPM matches
Pmq add rule list: Array of uint32_t's to store (internal) sids from the MPM.

AC: store sids in the pattern list, append to Pmq::rule_id_array on match.

Detect: sort rule_id_array after it was set up by the MPM. Rule id's
(Signature::num) are ordered, and the rule's with the lowest id are to
be inspected first. As the MPM doesn't fill the array in order, but instead
'randomly' we need this sort step to assure proper inspection order.
11 years ago
Victor Julien 43a1007788 detect: add test for memcmp issue 11 years ago
Victor Julien d951de2f19 Bug 977: -T / --init-errors-fatal to process all rules
Have -T / --init-errors-fatal process all rules so that it's easier
to debug problems in ruleset. Otherwise it can be a lengthy fix, test
error cycle if multiple rules have issues.

Convert empty rulefile error into a warning.

Bug #977
11 years ago
DIALLO David b3bf2f9939 Detect: Add Modbus keyword management
Add the modbus.function and subfunction) keywords for public function match in rules (Modbus layer).
Matching based on code function, and if necessary, sub-function code
or based on category (assigned, unassigned, public, user or reserved)
and negation is permitted.

Add the modbus.access keyword for read/write Modbus function match in rules (Modbus layer).
Matching based on access type (read or write),
and/or function type (discretes, coils, input or holding)
and, if necessary, read or write address access,
and, if necessary, value to write.
For address and value matching, "<", ">" and "<>" is permitted.

Based on TLS source code and file size source code (address and value matching).

Signed-off-by: David DIALLO <diallo@et.esia.fr>
11 years ago
Anoop Saldanha b334b8a6e9 CUDA: Update the inspection engine to inform the cuda module that it
doesn't need the gpu results and to release the packet for the next run.

Previously the inspection engine wouldn't inform the cuda module, if it
didn't need the results.  As a consequence, when the packet is next taken
for re-use, and if the packet is still being processed by the cuda module,
the engine would wait till the cuda module frees the packet.

This commits updates this functionality to inform the cuda module to
release the packet for the afore-mentioned case.
11 years ago
Ken Steele 60c46170b0 Check replist is not NULL inline before doing any processing.
The replist is often NULL, so it is worth checking that case before making
the function call do perform work on the list.
11 years ago
Victor Julien f23399d672 Rename Lua code to just Lua
As we support regular Lua as well as LuaJIT, it makes more sense to call
it all Lua.
11 years ago
Ken Steele 8f1d75039a Enforce function coding standard
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
11 years ago
Victor Julien 96adcf6829 refactor IDS/IPS engine mode logic
Instead of error phrone externs with macro's, use functions with a local
static enum var instead.

- EngineModeIsIPS(): in IPS mode
- EngineModeIsIDS(): in IDS mode

To set the modes:

- EngineModeSetIDS(): IDS mode (default)
- EngineModeSetIPS(): IPS mode

Bug #1177.
11 years ago
Victor Julien fc559ce227 detect: fix alstate handling
Previously, the alstate use in the main detect loop was unsafe. The
alstate pointer would be set duing a lock, but it would again be used
after one or more lock/unlock cycles. If the data pointed to would
disappear, a dangling pointer would be the result.

Due to they way flows are cleaned up using reference counting and
such, changes of this happening were very small. However, at least
one path can lead to this situation. So it had to be fixed.
11 years ago
Victor Julien b6e2a6f525 detect: locking update continued
Make DeStateDetectContinueDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
11 years ago
Victor Julien cf31e2cc74 detect: locking update
Make DeStateDetectStartDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
11 years ago
Victor Julien 5e1bc99e5b detect: cleanup
Remove unused alstate and app layer flags arguments from
DetectEngineInspectPacketPayload()
11 years ago
Victor Julien f00703acc1 Fix False Positive of rules with ports on portless protocols
In case of 'alert ip' rules that have ports, the port checks would
be bypassed for non-port protocols, such as ICMP. This would lead to
a rule matching: a false positive.

This patch adds a check. If the rule has a port setting other than
'any' and the protocol is not TCP, UDP or SCTP, then we rule won't
match.

Rules with 'alert ip' and ports are rare, so the impact should be
minimal.

Bug #611.
12 years ago
Victor Julien b5afe2b51f ssh: reenable ssh.protoversion keyword 12 years ago
Victor Julien 6c0162bf26 ssh: reenable ssh.softwareversion keyword 12 years ago
Victor Julien df927f7ea8 unittest: fix mutex unlock w/o a lock
Fixes an error in a test. SCMutexUnlock was called w/o a prior
SCMutexLock.
12 years ago
Victor Julien 2c3a92a1c9 profiling: conditional rule profiling
Add support for conditional rule profiling. Currently only simple
rate limiting is supported, but hardcoded to inspecting rules for
each packet.
12 years ago
Victor Julien 54d64a1237 detect: use pflow pointer
Use pflow pointer in SigMatchSignatures consistently. Also, when
needing access to the ipproto, use p->proto, not p->flow->proto.
12 years ago
Victor Julien 0ec375d95a stream msg: remove structure 12 years ago
Victor Julien 8dbf7a0d78 Update tests to use AppLayerParserThreadCtx ptr instead of void. Fix a few bugs uncovered by this. 12 years ago
Victor Julien fdefb65be4 app-layer: rename AppLayerThreadCtx funcs
AppLayerParserGetCtxThread -> AppLayerParserThreadCtxAlloc
AppLayerParserDestroyCtxThread -> AppLayerParserThreadCtxFree
12 years ago
Victor Julien f5f148805c app layer: uint16_t alproto -> AppProto alproto
This conversion was missing in a couple of places.
12 years ago
Victor Julien 8527b8e08e App Layer: cleanup state func naming
Rename functions related to AppLayerState to be more consistent.
12 years ago
Victor Julien e111401c10 detect unittests: clang build fix and cleanups
A number of unittests would lead to clang build errors because
of unsafe det_ctx ptr usage. This patch fixes these and inits
det_ctx to NULL in the other detect tests.
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Anoop Saldanha abded4200a Disabling the ssh parser temporarily, since we are moving away from some
of the archaic features we use in the app layer. We will reintroduce this
parser shortly. Also do note that keywords that rely on the ssh parser
would now be disabled.
12 years ago
Victor Julien 2913a4a860 debug: fix realloc error checking on flowbit print
detect.c:1074:17: warning: Potential leak of memory pointed to by \
                                field 'debuglog_flowbits_names'
                return;

Bug #1062.
12 years ago
Ken Steele 882d98733f Fix pfring so that zero-copy mode can work.
Detect when default_packet_size is zero, which enables zero-copy mode for
pfring and in that case, do what AF Packet does and set pkt_ext pointer to
the data and set PKT_ZERO_COPY flag.
12 years ago
Ken Steele f9705377ae Remove pkt variable from Packet structure.
The uint8_t *pkt in the Packet structure always points to the memory
immediately following the Packet structure. It is better to simply
calculate that value every time than store the 8 byte pointer.
12 years ago
Ken Steele 5a1bc025e5 Mark pflow as a constant pointer.
Address review comment from Victor that the pflow pointer is constant, so
it can be marked as such.
12 years ago
Ken Steele bee24d8909 Use pflow variable in place of p->flow to prevent reloading.
In SigMatchSignatures, the value p->flow doens't change, but GCC can't
figure that out, so it reloads p->flow many times during the function.
When p->flow is loaded into the variable pflow once at the start of the
function, the compile then doesn't need to reload it.
12 years ago
Victor Julien 0b0e9340dc rule setup: cleanup
Remove rule preparation logic that ran, but it's results were not
used.
12 years ago
Victor Julien d397ed94c5 detect: use macro for max rule size 12 years ago
Victor Julien e7f6107e79 signature address parsing improvements and tests
Fix sigatures not supporting [10.0.0.0/24, !10.1.1.1] notation when
used directly in a rule instead of through a variable.

Add tests for Bugs #815 and #920.
12 years ago