Commit Graph

954 Commits (f30c05e68431dce92ea01aa8c665a275ff2507a4)

Author SHA1 Message Date
Victor Julien 63291d0f01 detect: style cleanup 8 years ago
Victor Julien ecfdd57ef8 detect: minor cleanups 8 years ago
Victor Julien ac57bd8149 detect: run buffer setup callback before validate 8 years ago
Mats Klepsland 5e6b8c47df detect: add (mpm) keyword tls_cert_fingerprint
Reimplement keyword to match on SHA-1 fingerprint of TLS
certificate as a mpm keyword.

alert tls any any -> any (msg:"TLS cert fingerprint test";
       tls_cert_fingerprint;
       content:"4a:a3:66:76:82:cb:6b:23:bb:c3:58:47:23:a4:63:a7:78:a4:a1:18";
       sid:12345;)
8 years ago
Victor Julien 0f65257a50 detect: don't register http_*_line twice 8 years ago
Victor Julien dbd2d7c058 detect: more gracefully handle mpm prepare failure
Exit with error instead of using the detection engine in a broken state.

Bug #2187
8 years ago
Victor Julien e087d93883 detect: reject dsize rules that can't match
Rules can contain conflicting statements and lead to a unmatchable rule.

2 examples are rejected by this patch:

1. dsize < content
2. dsize < content@offset

Bug #2187
8 years ago
Victor Julien 3c05379cbd detect: fix mix of pass and noalert
Noalert rules did not apply pass logic to the flow.

Bug #1888.
8 years ago
Victor Julien e0c6565e68 nfs: nfs_version keyword
Store nfs version in tx and add keyword to match on it.
8 years ago
Victor Julien 0d79181d78 nfs: rename nfs3 to nfs
Since the parser now also does nfs2, the name nfs3 became confusing.
As it's still in beta, we can rename so this patch renames all 'nfs3'
logic to simply 'nfs'.
8 years ago
Eric Leblond 97b89c0a54 detect-target: introduce new keyword
The target keyword allows rules writer to specify information about
target of the attack. Using this keyword in a signature causes
some fields to be added in the EVE output. It also fixes ambiguity
in the Prelude output.
8 years ago
Victor Julien d6592211d0 rust/nfs: NFSv3 parser, logger and detection 8 years ago
Victor Julien 3148ff34b6 app-layer API optimizations and cleanups 8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien 84b97ca155 detect: content-inspection tests
Add tests for the content inspection engine that count the number
of steps it takes to eval a rule.
8 years ago
Victor Julien 1bbf555318 detect: improve stateful detection
Now that MPM runs when the TX progress is right, stateful detection
operates differently.

Changes:

1. raw stream inspection is now also an inspect engine

   Since this engine doesn't take the transactions into account, it
   could potentially run multiple times on the same data. To avoid
   this, basic result caching is in place.

2. the engines are sorted by progress, but the 'MPM' engine is first
   even if the progress is higher

   If MPM flags a rule to be inspected, the inspect engine for that
   buffer runs first. If this step fails, the rule is no longer
   evaluated. No state is stored.
8 years ago
Victor Julien d1b7a83905 detect: change mask logic
Previously the MPM/Prefilter engines would suggest the same rule
candidates multiple times.

For example, while processing the request body, the http headers
would be inspected by MPM multiple times.

The mask check was one way to quickly decide which rules could be
skipped.

Now that the MPM engines normally return a rule just once, this
mask check no longer makes sense. If the rule meets the ip/port/
direction based conditions, it needs to be evaluated if the MPM
said so. Even if not all conditions are yet true.

WIP disable mask as it no longer makes sense

WIP redo mask match
8 years ago
Victor Julien e3bd5f371d detect: more detailed state profiling 8 years ago
Victor Julien aba9cd7d02 stream inspection: add debug counters 8 years ago
Victor Julien e1aba7d6c2 detect: only do flow dependent cleanup if a flow is present 8 years ago
Victor Julien 61c35d3c39 detect: make SigMatchSignatures void
None of the callers cared for it's retval, so get rid of it.
8 years ago
Victor Julien f49150ddb9 detect: turn single detect flag into bool 8 years ago
Victor Julien 6f76cbb870 detect: remove unused detect flag 8 years ago
Victor Julien 7c56c9ada0 stream: allow raw reassembly catch up
If raw reassembly falls behind, for example because no raw mpm is
active, then we need to sync up to the app progress if that is
available, or to the generic tcp tracking otherwise.
8 years ago
Victor Julien b099008b94 stream: handle no stream scanning case
Now that detect moves the raw progress forward, it's important
to deal with the case where detect don't consider raw inspection.

If no 'stream' rules are active, disable raw. For this the disable
raw flag is now per stream.
8 years ago
Victor Julien 971ab18b95 detect / stream: new 'raw' stream inspection
Remove the 'StreamMsg' approach from the engine. In this approach the
stream engine would create a list of chunks for inspection by the
detection engine. There were several issues:

1. the messages had a fixed size, so blocks of data bigger than ~4k
   would be cut into multiple messages

2. it lead to lots of data copying and unnecessary memory use

3. the StreamMsgs used a central pool

The Stream engine switched over to the streaming buffer API, which
means that the reassembled data is always available. This made the
StreamMsg approach even clunkier.

The new approach exposes the streaming buffer data to the detection
engine. It has to pay attention to an important issue though: packet
loss. The data may have gaps. The streaming buffer API tracks the
blocks of continuous data.

To access the data for inspection a callback approach is used. The
'StreamReassembleRaw' function is called with a callback and data.
This way it runs the MPM and individual rule inspection code. At
the end of each detection run the stream engine is notified that it
can move forward it's 'progress'.
8 years ago
Victor Julien 8c9f521707 tcp: streaming implementation
Make stream engine use the streaming buffer API for it's data storage.

This means that the data is stored in a single reassembled sliding
buffer. The subleties of the reassembly, e.g. overlap handling, are
taken care of at segment insertion.

The TcpSegments now have a StreamingBufferSegment that contains an
offset and a length. Using this the segment data can be retrieved
per segment.

Redo segment insertion. The insertion code is moved to it's own file
and is simplified a lot.

A major difference with the previous implementation is that the segment
list now contains overlapping segments if the traffic is that way.
Previously there could be more and smaller segments in the memory list
than what was seen on the wire.

Due to the matching of in memory segments and on the wire segments,
the overlap with different data detection (potential mots attacks)
is much more accurate.

Raw and App reassembly progress is no longer tracked per segment using
flags, but there is now a progress tracker in the TcpStream for each.

When pruning we make sure we don't slide beyond in-use segments. When
both app-layer and raw inspection are beyond the start of the segment
list, the segments might not be freed even though the data in the
streaming buffer is already gone. This is caused by the 'in-use' status
that the segments can implicitly have. This patch accounts for that
when calculating the 'left_edge' of the streaming window.

Raw reassembly still sets up 'StreamMsg' objects for content
inspection. They are set up based on either the full StreamingBuffer,
or based on the StreamingBufferBlocks if there are gaps in the data.

Reworked 'stream needs work' logic. When a flow times out the flow
engine checks whether a TCP flow still needs work. The
StreamNeedsReassembly function is used to test if a stream still has
unreassembled segments or uninspected stream chunks.

This patch updates the function to consider the app and/or raw
progress. It also cleans the function up and adds more meaningful
debug messages. Finally it makes it non-inline.

Unittests have been overhauled, and partly moved into their own files.

Remove lots of dead code.
8 years ago
Victor Julien 88cfb99910 detect: http_accept sticky buffer + common code
Implement common code to easily add more per HTTP header detection
keywords.

Implement http_accept sticky buffer. It operates on the HTTP Accept
header.
8 years ago
Jason Ish 9ff8882cbd detect: don't consider an empty rule file an error 8 years ago
Jason Ish f56428d996 tcp/udp: fix checksum validation when 0xffff
Issue:
https://redmine.openinfosecfoundation.org/issues/2041

One approach to fixing this issue to just validate the
checksum instead of regenerating it and comparing it. This
method is used in some kernels and other network tools.

When validating, the current checksum is passed in as an
initial argument which will cause the final checksum to be 0
if OK. If generating a checksum, 0 is passed and the result
is the generated checksum.
8 years ago
Victor Julien 4f8eacdc69 disable-detect: fix needless file hashing
When detection is running flags are set on flows to indicate if file
hashing is needed. This is based on global output settings and rules.

In the case of --disable-detection this was not happening, so all
files where hashed with all methods. This has a significant
performance impact.

This patch adds logic to set the flow flags in --disable-detect mode.
8 years ago
Victor Julien c066f043a5 detect: remove unused alversion logic 8 years ago
Victor Julien 6c6802b957 detect: simplify state detect code: remove unused params 8 years ago
Victor Julien c961da2dce detect: clean up test 8 years ago
Victor Julien 15d26f14e1 file-store: fix force store 8 years ago
Victor Julien 2a5b9eb5fa stream: remove unused variable 8 years ago
Mats Klepsland d6508e640a detect: add (mpm) keyword tls_cert_serial
Match on TLS certificate serial number using tls_cert_serial
keyword, e.g.:

alert tls any any -> any any (msg:"TLS cert serial test";
        tls_cert_serial; content:"5C:19:B7:B1:32:3B:1C:A1";
        sid:12345;)
9 years ago
Victor Julien 0f708d427b pkt-var: abuse flowvar postmatch logic for pktvars
Flowvars were already using a temporary store in the detect thread
ctx.

Use the same facility for pktvars. The reasons are:

1. packet is not always available, e.g. when running pcre on http
   buffers.

2. setting of vars should be done post match. Until now it was also
   possible that it is done on a partial match.
9 years ago
Victor Julien 5e39486399 pkt-var: use id instead of name pointer 9 years ago
Victor Julien e95a0c1344 alert-debug: print flowbit names from VarNameStore 9 years ago
Victor Julien 22f3205664 var-names: expose outside of detect engine
Until now variable names, such as flowbit names, were local to a detect
engine. This made sense as they were only ever used in that context.

For the purpose of logging these names, this needs a different approach.
The loggers live outside of the detect engine. Also, in the case of
reloads and multi-tenancy, there are even multiple detect engines, so
it would be even more tricky to access them from the outside.

This patch brings a new approach. A any time, there is a single active
hash table mapping the variable names and their id's. For multiple
tenants the table is shared between tenants.

The table is set up in a 'staging' area, where locking makes sure that
multiple loading threads don't mess things up. Then when the preparing
of a detection engine is ready, but before the detect threads are made
aware of the new detect engine, the active varname hash is swapped with
the staging instance.

For this to work, all the mappings from the 'current' or active mapping
are added to the staging table.

After the threads have reloaded and the new detection engine is active,
the old table can be freed.

For multi tenancy things are similar. The staging area is used for
setting up until the new detection engines / tenants are applied to
the system.

This patch also changes the variable 'id'/'idx' field to uint32_t. Due
to data structure padding and alignment, this should have no practical
drawback while allowing for a lot more vars.
9 years ago
Victor Julien 43cc06eabe detect: use engine version instead of id
Use engine version based on global detect engine master. This is
incremented between reloads.
9 years ago
Victor Julien 920709fe6f detect: ssh_software sticky buffer 9 years ago
Victor Julien f1ab6a6153 detect: ssh_proto stickybuffer 9 years ago
Victor Julien dfac5276b8 detect: remove unused SIGMATCH_PAYLOAD flag 9 years ago
Victor Julien fe415ae518 detect: remove DMATCH list 9 years ago
Victor Julien 1c02cf4542 flow: remove unused Flow::de_state 9 years ago
Victor Julien ad238121e3 detect: remove the AMATCH list 9 years ago
Victor Julien 1087495d6d detect: http_start sticky buffer
Matches on the start of a HTTP request or response.

Uses a buffer constructed from the request line and normalized request
headers, including the Cookie header.

Or for the response side, it uses the response line plus the
normalized response headers, including the Set-Cookie header.

Both buffers are terminated by an extra \r\n.
9 years ago
Victor Julien 36535efa04 detect: add http_protocol sticky buffer
Matches on protocol field in HTTP.
9 years ago
Victor Julien f2fc5a255f http_header: convert to use common code 9 years ago
Victor Julien 5ee68ca2b2 http_header: remove old files 9 years ago
Victor Julien aaeeae0722 detect: http_header_names sticky buffer keyword
A sticky buffer that allows content inspection on a contructed buffer
of HTTP header names. The buffer starts with \r\n, the names are
separated by \r\n and the end of the buffer contains an extra \r\n.

E.g. \r\nHost\r\nUser-Agent\r\n\r\n

The leading \r\n is to make sure one can match on a full name in all
cases.
9 years ago
Victor Julien cf9678d926 detect: global registery for keyword thread data
Some keywords need a scratch space where they can do store the results
of expensive operations that remain valid for the time of a packets
journey through the detection engine.

An example is the reconstructed 'http_header' field, that is needed
in MPM, and then for each rule that manually inspects it. Storing this
data in the flow is a waste, and reconstructing multiple times on
demand as well.

This API allows for registering a keyword with an init and free function.

It it mean to be used an initialization time, when the keyword is
registered.
9 years ago
Victor Julien 49fbd28ceb detect: cleanup built-in list id's 9 years ago
Victor Julien 815120896b app-layer-events: dynamic list 9 years ago
Victor Julien 6f867c3c73 http_raw_uri: dynamic buffer 9 years ago
Victor Julien ee55aefa1c http_client_body: dynamic buffer 9 years ago
Victor Julien e7d5e845c7 http_header / http_raw_header: dynamic buffers 9 years ago
Victor Julien b694d96e22 http_stat_msg: dynamic buffer 9 years ago
Victor Julien 7e3ab4f5ea http_stat_code: dynamic buffer 9 years ago
Victor Julien 128b59d4f6 http_raw_host: dynamic buffer 9 years ago
Victor Julien 296c275e23 http_host: dynamic buffer 9 years ago
Victor Julien 67b7d9734e http_cookie: dynamic buffer 9 years ago
Victor Julien 54604c7bf2 http_user_agent: dynamic buffer 9 years ago
Victor Julien 6346a074a7 http_uri: dynamic buffer
Clean up tests
9 years ago
Victor Julien e34102d67a http_method: make list dynamic 9 years ago
Victor Julien 6bd37611ee file_data: dynamic buffer 9 years ago
Victor Julien 779d40cedf detect: remove hardcoded sm_list logic from setup
Introduce utility functions to aid this.
9 years ago
Victor Julien 04592efb76 detect: buffer type API
To replace the hardcoded SigMatch list id's, use this API to register
and query lists by name.

Also allow for registering descriptions and whether mpm is supported.

Registration is only allowed at startup.
9 years ago
Victor Julien f370e88135 detect: move init only Signature members to init_data 9 years ago
Victor Julien 0a5ae415b8 detect: shrink Signature::sm_arrays
Signature::sm_arrays now only contains 'built-in' lists, and so is
sized appropriately.
9 years ago
Victor Julien 4978a7a133 detect: reorganize id's in prep of dynamic lists 9 years ago
Victor Julien 59303d1fbb threshold: fix and redo tests 9 years ago
Victor Julien 6f7e4adbe8 detect: improve memory handling & comments 9 years ago
Victor Julien 8edc954e82 detect: get rid of Signature::sm_lists
Instead use the lists in init_data during setup and the SigMatchData
arrays during runtime.
9 years ago
Victor Julien 715ff60087 detect: remove unused SIG_FLAG_INIT_PAYLOAD init_flag 9 years ago
Victor Julien bfd4bc8233 detect: constify Signature/SigMatch use at runtime 9 years ago
Victor Julien 86222428dd detect: don't run IP inspection on non-IP packets
The code to get the rule group (sgh) would return the group for
IP proto 0 instead of nothing. This lead to certain types of rules
unintentionally matching (False Positive).

Since the packets weren't actually IP, the logged alert records
were missing the IP header.

Bug #2017.
9 years ago
Victor Julien f91d490d25 detect: remove unused flow_locked hint 9 years ago
Victor Julien 810e43f373 magic: make optional
Make libmagic optional. If installed it will be enabled by default in
configure. Use --disable-libmagic to disable.
9 years ago
Victor Julien 3da7dad514 lua: luajit improvements
Luajit has a strange memory requirement, it's 'states' need to be in the
first 2G of the process' memory.

This patch improves the pool approach by moving it to the front of the
start up.

A new config option 'luajit.states' is added to control how many states
are preallocated. It defaults to 128.

Add a warning when more states are used then preallocated. This may fail
if flow/stream/detect engines use a lot of memory. Add hint at exit that
gives the max states in use if it's higher than the default.
9 years ago
Victor Julien 1f670837ac detect: add missing break (CID 1374301) 9 years ago
Jason Ish 1a31bded4a DNP3: dnp3_data, dnp3_func, dnp3_ind, dnp3_obj rule keywords 9 years ago
Victor Julien 321fb6463e vars: small cleanups 9 years ago
Victor Julien 0ed119068d detect-app-layer-protocol: implement prefilter
Introduce 'Protocol detection'-only rules. These rules will only be
fully evaluated when the protocol detection completed. To allow
mixing of the app-layer-protocol keyword with other types of matches
the keyword can also inspect the flow's app-protos per packet.

Implement prefilter for the 'PD-only' rules.
9 years ago
Victor Julien 43aed70976 detect: during detection sgh is read only so turn into const 9 years ago
Victor Julien 0e31124609 detect: add util func for post-inspect tasks on first sgh 9 years ago
Victor Julien d3fb4de1b5 detect: move file flags update into it's own function 9 years ago
kwong a3ffebd835 Adding SCADA EtherNet/IP and CIP protocol support
Add support for the ENIP/CIP Industrial protocol

This is an app layer implementation which uses the "enip" protocol
and "cip_service" and "enip_command" keywords

Implements AFL entry points
9 years ago
Victor Julien 8321f04ef3 prefilter: clean up setup code 9 years ago
Victor Julien d36c0c15ea detect: reshuffle keyword registration order
The order of keyword registration currently affects inspect engine
registration order and ultimately the order of inspect engines per
rule. Which in turn affects state keeping.

This patch makes sure the ordering is the same as with older
releases.
9 years ago
Victor Julien 5f994756e6 detect-engine: improved inspect engines
Inspect engines are called per signature per sigmatch list. Most
wrap around DetectEngineContentInspection, but it's more generic.

Until now, the inspect engines were setup in a large per ipproto,
per alproto, per direction table. For stateful inspection each
engine needed a global flag.

This approach had a number of issues:
1. inefficient: each inspection round walked the table and then
   checked if the inspect engine was even needed for the current
   rule.
2. clumsy registration with global flag registration.
3. global flag space was approaching the need for 64 bits
4. duplicate registration for alprotos supporting both TCP and
   TCP (DNS).

This patch introduces a new approach.

First, it does away with the per ipproto engines. This wasn't used.

Second, it adds a per signature list of inspect engine containing
only those engines that actually apply to the rule.

Third, it gets rid of the global flags and replaces it with flags
assigned per rule per engine.
9 years ago
Victor Julien f1e3840516 http_response_body: implement keyword with mpm
Implemented as 'stickybuffer'.
9 years ago
Victor Julien 4c98b6cef3 http_request_line: implement keyword and mpm
Implemented as 'stickybuffer'.

Move all logic into the keyword file and remove bad tests that tested
URI instead of request line.
9 years ago
Victor Julien 5b2e36a1b0 mpm: add App Layer MPM registery
Register keywords globally at start up.

Create a map of the registery per detection engine. This we need because
the sgh_mpm_context value is set per detect engine.

Remove APP_MPMS_MAX.
9 years ago
Victor Julien e1eb481647 prefilter: cleanup and optimization 9 years ago
Victor Julien dba14b676c profiling: more prefilter profiling 9 years ago
Victor Julien 125603871b detect: config opt to enable keyword prefilters 9 years ago
Victor Julien 9bb12ccb27 prefilter: move payload engines into separate list 9 years ago