Commit Graph

925 Commits (7a96d18f36817ec8280546b248ef0d0afe16f88a)

Author SHA1 Message Date
Victor Julien af51e0f5a1 detect: rewrite of the detect engine
Use per tx detect_flags to track prefilter. Detect flags are used for 2
things:
1. marking tx as fully inspected
2. tracking already run prefilter (incl mpm) engines

This supercedes the MpmIDs API for directionless tracking
of the prefilter engines.

When we have no SGH we have to flag the txs that are 'complete'
as inspected as well.

Special handling for the stream engine:

If a rule mixes TX inspection and STREAM inspection, we can encounter
the case where the rule is evaluated against multiple transactions
during a single inspection run. As the stream data is exactly the same
for each of those runs, it's wasteful to rerun inspection of the stream
portion of the rule.

This patch enables caching of the stream 'inspect engine' result in
the local 'RuleMatchCandidateTx' array. This is valid only during the
live of a single inspection run.

Remove stateful inspection from 'mask' (SignatureMask). The mask wasn't
used in most cases for those rules anyway, as there we rely on the
prefilter. Add a alproto check to catch the remaining cases.

When building the active non-mpm/non-prefilter list check not just
the mask, but also the alproto. This especially helps stateful rules
with negated mpm.

Simplify AppLayerParserHasDecoderEvents usage in detection to only
return true if protocol detection events are set. Other detection is done
in inspect engines.

Move rule group lookup and handling into it's own function. Handle
'post lookup' tasks immediately, instead of after the first detect
run. The tasks were independent of the initial detection.

Many cleanups and much refactoring.
8 years ago
Victor Julien 40986b1f61 detect: put inspect code for MATCH-list into func
Introduce DetectRunInspectRulePacketMatches to inspect the signatures
match list.
8 years ago
Victor Julien 70597066e0 detect: move detect cleanup into util func 8 years ago
Victor Julien 01f9d0076c detect: move packet hdr inspect into util func 8 years ago
Victor Julien 746638b220 cuda: remove
Remove CUDA support as it has been broken for a long time.

Ticket #2382.
8 years ago
Victor Julien ac0ae2dcd1 file_data: smtp file_data to generic file_data
Generalize the SMTP file_data inspection into a 'files'
file_data inspection that can be used for any protocol
that uses the File API.
8 years ago
Victor Julien ccf202a4f0 detect: minor cleanup 8 years ago
Victor Julien 948dee9a98 app-layer: use bool for 'HasDecoderEvents' 8 years ago
Victor Julien 8b8f911600 detect: move rule loading into loader files 8 years ago
Victor Julien c374324916 detect: move keyword registration into own file 8 years ago
Victor Julien 90569d5fd6 detect: move grouping/building code into own file
Clean up main detect.c file by moving things related to rule
grouping out.
8 years ago
Victor Julien 622d60b74d detect: move unittests into tests/ 8 years ago
Victor Julien 9e37e266b6 detect: content limits propagation
Propagate inspection limits from anchered keywords to the rest of
a rule.

Examples:

content:"A"; depth:1; is anchored, it can only match in the first byte

content:"A"; depth:1; content:"BC"; distance:0; within:2;
"BC" can only be in the 2nd and 3rd byte of the payload. So effectively
it has an implicite offset of 1 and an implicit depth of 3.

content:"A"; depth:1; content:"BC"; distance:0; can assume offset:1; for
the 2nd content.

content:"A"; depth:1; pcre:"/B/R"; content:"C"; distance:0; can assume
at least offset:1; for content "C". We can't analyzer the pcre pattern
(yet), so we assume it matches with 0 bytes.

Add lots of test cases.
8 years ago
Giuseppe Longo 1328ecb8f6 detect: save invalid rules
This keeps the invalid rules in string format into a list,
added in DetectEngineCtx.
8 years ago
Giuseppe Longo 56000acefb detect-engine: add reload time/rules stats
This patch adds the following stats for
the detect engine:
- time of the last reload
- number of rules loaded
- number of rules failed
8 years ago
Victor Julien 40a819d5a6 detect/flowint: only check if packet has flow
Fixed bug #2288.
8 years ago
Victor Julien 2a237bdfca detect: make glob.h optional
glob.h is not available on MinGW.

Simply use the input on the rule list as a literal pattern.
8 years ago
Victor Julien e1d1a7f2ac detect: fix flow bypass flag handling 8 years ago
Victor Julien e86c3f0a40 detect: constify rule group lookup 8 years ago
Victor Julien a9ee041984 detect: minor profiling cleanup 8 years ago
Victor Julien 26abf5337c detect/mpm: minor cleanup: remove unused function arg 8 years ago
Victor Julien 63291d0f01 detect: style cleanup 8 years ago
Victor Julien ecfdd57ef8 detect: minor cleanups 8 years ago
Victor Julien ac57bd8149 detect: run buffer setup callback before validate 8 years ago
Mats Klepsland 5e6b8c47df detect: add (mpm) keyword tls_cert_fingerprint
Reimplement keyword to match on SHA-1 fingerprint of TLS
certificate as a mpm keyword.

alert tls any any -> any (msg:"TLS cert fingerprint test";
       tls_cert_fingerprint;
       content:"4a:a3:66:76:82:cb:6b:23:bb:c3:58:47:23:a4:63:a7:78:a4:a1:18";
       sid:12345;)
8 years ago
Victor Julien 0f65257a50 detect: don't register http_*_line twice 8 years ago
Victor Julien dbd2d7c058 detect: more gracefully handle mpm prepare failure
Exit with error instead of using the detection engine in a broken state.

Bug #2187
8 years ago
Victor Julien e087d93883 detect: reject dsize rules that can't match
Rules can contain conflicting statements and lead to a unmatchable rule.

2 examples are rejected by this patch:

1. dsize < content
2. dsize < content@offset

Bug #2187
8 years ago
Victor Julien 3c05379cbd detect: fix mix of pass and noalert
Noalert rules did not apply pass logic to the flow.

Bug #1888.
8 years ago
Victor Julien e0c6565e68 nfs: nfs_version keyword
Store nfs version in tx and add keyword to match on it.
8 years ago
Victor Julien 0d79181d78 nfs: rename nfs3 to nfs
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
8 years ago
Eric Leblond 97b89c0a54 detect-target: introduce new keyword
The target keyword allows rules writer to specify information about
target of the attack. Using this keyword in a signature causes
some fields to be added in the EVE output. It also fixes ambiguity
in the Prelude output.
8 years ago
Victor Julien d6592211d0 rust/nfs: NFSv3 parser, logger and detection 8 years ago
Victor Julien 3148ff34b6 app-layer API optimizations and cleanups 8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien 84b97ca155 detect: content-inspection tests
Add tests for the content inspection engine that count the number
of steps it takes to eval a rule.
8 years ago
Victor Julien 1bbf555318 detect: improve stateful detection
Now that MPM runs when the TX progress is right, stateful detection
operates differently.

Changes:

1. raw stream inspection is now also an inspect engine

   Since this engine doesn't take the transactions into account, it
   could potentially run multiple times on the same data. To avoid
   this, basic result caching is in place.

2. the engines are sorted by progress, but the 'MPM' engine is first
   even if the progress is higher

   If MPM flags a rule to be inspected, the inspect engine for that
   buffer runs first. If this step fails, the rule is no longer
   evaluated. No state is stored.
8 years ago
Victor Julien d1b7a83905 detect: change mask logic
Previously the MPM/Prefilter engines would suggest the same rule
candidates multiple times.

For example, while processing the request body, the http headers
would be inspected by MPM multiple times.

The mask check was one way to quickly decide which rules could be
skipped.

Now that the MPM engines normally return a rule just once, this
mask check no longer makes sense. If the rule meets the ip/port/
direction based conditions, it needs to be evaluated if the MPM
said so. Even if not all conditions are yet true.

WIP disable mask as it no longer makes sense

WIP redo mask match
8 years ago
Victor Julien e3bd5f371d detect: more detailed state profiling 8 years ago
Victor Julien aba9cd7d02 stream inspection: add debug counters 8 years ago
Victor Julien e1aba7d6c2 detect: only do flow dependent cleanup if a flow is present 8 years ago
Victor Julien 61c35d3c39 detect: make SigMatchSignatures void
None of the callers cared for it's retval, so get rid of it.
8 years ago
Victor Julien f49150ddb9 detect: turn single detect flag into bool 8 years ago
Victor Julien 6f76cbb870 detect: remove unused detect flag 8 years ago
Victor Julien 7c56c9ada0 stream: allow raw reassembly catch up
If raw reassembly falls behind, for example because no raw mpm is
active, then we need to sync up to the app progress if that is
available, or to the generic tcp tracking otherwise.
8 years ago
Victor Julien b099008b94 stream: handle no stream scanning case
Now that detect moves the raw progress forward, it's important
to deal with the case where detect don't consider raw inspection.

If no 'stream' rules are active, disable raw. For this the disable
raw flag is now per stream.
8 years ago
Victor Julien 971ab18b95 detect / stream: new 'raw' stream inspection
Remove the 'StreamMsg' approach from the engine. In this approach the
stream engine would create a list of chunks for inspection by the
detection engine. There were several issues:

1. the messages had a fixed size, so blocks of data bigger than ~4k
   would be cut into multiple messages

2. it lead to lots of data copying and unnecessary memory use

3. the StreamMsgs used a central pool

The Stream engine switched over to the streaming buffer API, which
means that the reassembled data is always available. This made the
StreamMsg approach even clunkier.

The new approach exposes the streaming buffer data to the detection
engine. It has to pay attention to an important issue though: packet
loss. The data may have gaps. The streaming buffer API tracks the
blocks of continuous data.

To access the data for inspection a callback approach is used. The
'StreamReassembleRaw' function is called with a callback and data.
This way it runs the MPM and individual rule inspection code. At
the end of each detection run the stream engine is notified that it
can move forward it's 'progress'.
8 years ago
Victor Julien 8c9f521707 tcp: streaming implementation
Make stream engine use the streaming buffer API for it's data storage.

This means that the data is stored in a single reassembled sliding
buffer. The subleties of the reassembly, e.g. overlap handling, are
taken care of at segment insertion.

The TcpSegments now have a StreamingBufferSegment that contains an
offset and a length. Using this the segment data can be retrieved
per segment.

Redo segment insertion. The insertion code is moved to it's own file
and is simplified a lot.

A major difference with the previous implementation is that the segment
list now contains overlapping segments if the traffic is that way.
Previously there could be more and smaller segments in the memory list
than what was seen on the wire.

Due to the matching of in memory segments and on the wire segments,
the overlap with different data detection (potential mots attacks)
is much more accurate.

Raw and App reassembly progress is no longer tracked per segment using
flags, but there is now a progress tracker in the TcpStream for each.

When pruning we make sure we don't slide beyond in-use segments. When
both app-layer and raw inspection are beyond the start of the segment
list, the segments might not be freed even though the data in the
streaming buffer is already gone. This is caused by the 'in-use' status
that the segments can implicitly have. This patch accounts for that
when calculating the 'left_edge' of the streaming window.

Raw reassembly still sets up 'StreamMsg' objects for content
inspection. They are set up based on either the full StreamingBuffer,
or based on the StreamingBufferBlocks if there are gaps in the data.

Reworked 'stream needs work' logic. When a flow times out the flow
engine checks whether a TCP flow still needs work. The
StreamNeedsReassembly function is used to test if a stream still has
unreassembled segments or uninspected stream chunks.

This patch updates the function to consider the app and/or raw
progress. It also cleans the function up and adds more meaningful
debug messages. Finally it makes it non-inline.

Unittests have been overhauled, and partly moved into their own files.

Remove lots of dead code.
8 years ago
Victor Julien 88cfb99910 detect: http_accept sticky buffer + common code
Implement common code to easily add more per HTTP header detection
keywords.

Implement http_accept sticky buffer. It operates on the HTTP Accept
header.
8 years ago
Jason Ish 9ff8882cbd detect: don't consider an empty rule file an error 8 years ago