Commit Graph

404 Commits (72829dbaea1d17370c486fa8c7504fb399efb534)

Author SHA1 Message Date
Victor Julien a806445abf mpm factory: include alproto
In preparation of spliting out mpm's for keywords shared by
multiple protocols, like file.data.
4 years ago
Victor Julien b31ffde6f4 output: remove error codes from output 4 years ago
Victor Julien 39f5c7f56a error: use SC_EINVAL for invalid input 4 years ago
Victor Julien cdd52dc8cd detect/mpm: output cleanups 4 years ago
Juliana Fajardini 5f4bcfe313 detect/engine: fix typos in assorted files 4 years ago
Philippe Antoine c5cf2967b3 detect: fix integer warnings
Ticket: #4516
4 years ago
Philippe Antoine 45d1a9ae77 detect: faster linked list copy
In DetectAppLayerInspectEngineCopyListToDetectCtx
Avoid quadratic complexity by remembering last element
of the linked list we are inserting into
5 years ago
Philippe Antoine f30975fb16 app-layer: fix integer warnings
Ticket: 4516
5 years ago
Victor Julien 087151ddc3 detect/mpm: initialization micro optimization 5 years ago
Victor Julien 54a6dd09dd detect: pattern id assignment through hash table
Only consider active part of the pattern for mpm (so consider chop).

Move data structure to hash list table over the custom array logic.
5 years ago
Victor Julien 9e6370ae2e detect: optimize mpm-engine setup
Instead of a loop over the rules in a group *per engine* do a single
loop in which all the engines are prepared in parallel.
5 years ago
Victor Julien 3352c0bee4 detect: initialization optimization
A lot of time was spent in `SigMatchListSMBelongsTo` for the `mpm_sm`.

Optimize this by keeping the value at hand during Signature parsing and
detection engine setup.
5 years ago
Philippe Antoine 529678d501 dns: wrap with HAVE_LUA
This is just code style, to minimize the compiled code.
5 years ago
Victor Julien f6f124f283 detect/engine: support frames
Implement the low level detect engine support for inspecting frames,
including MPM, transforms and inspect API's.
5 years ago
Victor Julien ecce116117 detect/fast_pattern: allow for rule time registration
In preparation of more dynamic logic in rule loading also doing
some registration, allow for buffers to be registered as fast_patterns
during rule parsing.

Leaves the register time registrations mostly as-is, but copies the
resulting list into the DetectEngineCtx and works with that onwards.
This list can then be extended.
5 years ago
Victor Julien 5bcaae0a01 detect: use hashes for all buffer to id
Instead of a map that is constantly realloc'd, use 2 hash tables for
DetectBufferType entries: one by name (+transforms), the other by
id. Use these everywhere.
5 years ago
Victor Julien ed87784907 detect: enforce max app-layer progress
Allow progress values in the range 0-47 so we have 48 bits to track
prefilter engines.

Mark bits 48-62 as reserved explicitly.

Add debug validation checks to make sure the reserved space isn't used.
5 years ago
Victor Julien ce18f4b8e2 detect/mpm: micro optimization for initialization
Do less expensive check first.
5 years ago
Philippe Antoine bbbb816ed6 detect: debug validation for list ids overflows 5 years ago
Victor Julien 86681c9d7c detect: move init only array to init data 5 years ago
Victor Julien 975062cf40 detect: track base id for xform buffers
Buffers with transforms are based on the non-transformed "base"
buffer, with a new ID assigned and the transform callbacks added.

This patch stores the id of the original buffer in the new buffer
inspect and prefilter structures. This way the buffers with and
without transforms can share some of the logic are progression
of file and body inspection trackers.

Related tickets: #4361 #4199 #3616
6 years ago
Victor Julien e7b5201016 detect/mpm: remove usused cleanup function 6 years ago
Victor Julien 82d7f64630 detect/mpm: improve transforms handling
Make sure keywords with transforms get their own mpm ctx, instead of
sharing it with the 'pure' version of the keyword.
6 years ago
Victor Julien ee15bd8076 detect/mpm: rename variable to ease code review 6 years ago
Victor Julien 9831839388 detect/mpm: fix hs check 6 years ago
Victor Julien 4726d7027c detect/mpm: 'mpm-algo' parsing cleanups 6 years ago
Shivani Bhardwaj 6f7d8e50c8 src: use FatalError whenever possible
Replaces all patterns of SCLogError() followed by exit() with
FatalError(). Cocci script to do this:

@@
constant C;
constant char[] msg;
@@

- SCLogError(C,
+ FatalError(SC_ERR_FATAL,
  msg);
- exit(EXIT_FAILURE);

Closes redmine ticket 3188.
6 years ago
Jeff Lucovsky fb409664d2 detect: byte_math support 6 years ago
Shivani Bhardwaj 537fb7a1c6 hyperscan: better error message if not compiled 6 years ago
Jeff Lucovsky 5d10db88bc detect/transform: Support transform options
This commit adds support for transform-specific options. During Setup,
transforms have the signature string available for options detection.
When a transform detects an option, it should convert the option into an
internal format and supply a pointer to this format as the last argument
to DetectSignatureAddTransform.

Transforms that support options must provide a function in their
Sigmatch table entry. When the transform is freed, a pointer to the
internal format of the option is passed to this function.
6 years ago
Victor Julien f903766849 detect/mpm: don't process empty store 7 years ago
Jeff Lucovsky 218a5c4345 mpm: Fix typos and spelling errors 7 years ago
Jeff Lucovsky db8527e7b3 detect/mpm: Improved handling of variable values
This commit removes the offset and depth if either of these values are
dependent upon a byte-extract operation.
7 years ago
Victor Julien bdd74d413b detect/mpm: put transform into 'profile name'
So that profiling gives more info about cost of the mpm
engines when they use transforms.
7 years ago
Victor Julien d6323ae33d detect/mpm: improve stats reporting 7 years ago
Victor Julien 4dff903b35 detect: introduce pkt mpm engines
Instead of the hardcode L4 matching in MPM that was recently introduced,
add an API similar to the AppLayer MPM and inspect engines.

Share part of the registration code with the AppLayer.

Implement for the tcp.hdr and udp.hdr keywords.
7 years ago
Victor Julien ac694b089a detect: add udp.hdr sticky buffer 7 years ago
Victor Julien 35be8385eb detect: tcp.hdr sticky buffer
Sticky buffer to inspect the TCP header.
7 years ago
Victor Julien 47ef8f5822 detect/mpm: add l4-header support
Prepare MPM part of the detection engine for a new type of per
packet matching, where the L4 header will be inspected.

Preparation for TCP header inspection keyword.
7 years ago
Victor Julien 636fb8d387 mpm: track maxdepth
Track max depth setting per MpmCtx.

To make sure the data structure doesn't increase in size change global
bool to use a flags field.
8 years ago
Victor Julien 6fcd2db043 tile: remove files 8 years ago
Victor Julien 3f253cb8c0 mpm: rename internal id for ac-tile to ac-ks 8 years ago
Victor Julien 998f32f88c detect/tls: consolidate validity code 8 years ago
Victor Julien ccdafe6697 detect/http-server-body: move tests to tests/ 8 years ago
Victor Julien 9a8092249e detect/http-client-body: move tests into tests/ 8 years ago
Victor Julien 76fd666cad detect/http_raw_header: move tests into tests/ 8 years ago
Victor Julien ab027cb481 detect/http_cookie: move tests into tests/ 8 years ago
Victor Julien 2f342da048 detect/http_stat_code: move tests into tests/ 8 years ago
Victor Julien 5dfba01b2e detect/http_stat_msg: move tests to tests/ 8 years ago
Victor Julien b469938998 detect/http_raw_host: move raw into regular host logic 8 years ago
Victor Julien dc43f35427 detect/http_host: move tests into tests/ 8 years ago
Victor Julien cb332b4cda detect/http_method: move all tests into tests/ 8 years ago
Victor Julien 0a405e27a0 detect/http_raw_uri: code reorganization
Move registration into http_uri logic, move tests into the other uri
tests. Switch to v2 mpm/inspect APIs.
8 years ago
Victor Julien 10e2731f18 detect/http-uri: move tests into tests/ 8 years ago
Victor Julien 3111910fc6 detect/http_user_agent: move tests into tests/ 8 years ago
Victor Julien 6f2b974f9c detect/mpm: fix fast_pattern handling of len >255
The fast pattern selection logic would truncate a patterns len to
255 leading to assigning the same pid to different patterns.

This in turn would be caught by the hyperscan setup code which would
abort.

Bug #2714.
8 years ago
Victor Julien 3a0eca9fde detect/mpm: minor code cleanup 8 years ago
Victor Julien 7abb8745bf detect/mpm: clean up setup code 8 years ago
Victor Julien 38b698c50f detect/analyzer: show pattern that is used by mpm
Set a new DETECT_CONTENT_MPM flag on the pattern that is selected
during setup.
8 years ago
Victor Julien 91296d1eec detect/prefilter: add de_ctx to registration 9 years ago
Victor Julien efbd901385 detect: move mpm engines into detect engine ctx
This allows safe registration at runtime.
9 years ago
Victor Julien 0de86211c6 detect: register dynamic buffers into de_ctx
Register buffers that are created during rule parsing. Currently
this means an existing buffer with one or more transformations.
9 years ago
Victor Julien a499a44f7a detect: move buffer type map into detect ctx
Move previously global table into detect engine ctx. Now that we
can register buffers at rule loading time we need to take concurrency
into account.

Move DetectBufferType to detect.h and update DetectBufferCtx API calls
to include a detect engine ctx reference.
9 years ago
Victor Julien f6e5cb1db6 detect: prefilter/inspect API v2, with transforms
Introduce InspectionBuffer a structure for passing data between
prefilters, transforms and inspection engines.

At rule parsing time, we'll register new unique 'DetectBufferType's
for a 'parent' buffer (e.g. pure file_data) with its transformations.
Each unique combination of buffer with transformations gets it's
own buffer id.

Similarly, mpm registration and inspect engine registration will be
copied from the 'parent' (again, e.g. pure file_data) to the new id's.

The transforms are called from within the prefilter engines themselves.

Provide generic MPM matching and setup callbacks. Can be used by
keywords to avoid needless code duplication. Supports transformations.

Use unique name for profiling, to distinguish between pure buffers
and buffers with transformation.

Add new registration calls for mpm/prefilters and inspect engines.

Inspect engine api v2: Pass engine to itself. Add generic engine that
uses GetData callback and other registered settings.

The generic engine should be usable for every 'simple' case where
there is just a single non-streaming buffer. For example HTTP uri.

The v2 API assumes that registered MPM implements transformations.

Add util func to set new transform in rule and add util funcs for rule
parsing.
9 years ago
Victor Julien ac0ae2dcd1 file_data: smtp file_data to generic file_data
Generalize the SMTP file_data inspection into a 'files'
file_data inspection that can be used for any protocol
that uses the File API.
9 years ago
Victor Julien 0b97fbbc13 detect/mpm: micro optimization in setup 9 years ago
Victor Julien 26abf5337c detect/mpm: minor cleanup: remove unused function arg 9 years ago
Victor Julien dbd2d7c058 detect: more gracefully handle mpm prepare failure
Exit with error instead of using the detection engine in a broken state.

Bug #2187
9 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
10 years ago
Victor Julien b099008b94 stream: handle no stream scanning case
Now that detect moves the raw progress forward, it's important
to deal with the case where detect don't consider raw inspection.

If no 'stream' rules are active, disable raw. For this the disable
raw flag is now per stream.
10 years ago
Victor Julien 5ee68ca2b2 http_header: remove old files 10 years ago
Victor Julien a1465bc4fa detect-engine-mpm: api cleanup 10 years ago
Victor Julien 779d40cedf detect: remove hardcoded sm_list logic from setup
Introduce utility functions to aid this.
10 years ago
Victor Julien f370e88135 detect: move init only Signature members to init_data 10 years ago
Victor Julien 4978a7a133 detect: reorganize id's in prep of dynamic lists 10 years ago
Victor Julien 8edc954e82 detect: get rid of Signature::sm_lists
Instead use the lists in init_data during setup and the SigMatchData
arrays during runtime.
10 years ago
Sascha Steinbiss e6044aaf1c mpm/spm: check for SSSE3 and enable/disable HS
The new Hyperscan 4.4 API provides a function to check for SSSE3
presence at runtime. This allows us to fall back to non-Hyperscan
matchers on systems without SSSE3 even when the suricata executable
is built with Hyperscan support. Addresses Redmine issue #2010.

Signed-off-by: Sascha Steinbiss <sascha@steinbiss.name>
Tested-by: Arturo Borrero Gonzalez <arturo@debian.org>
10 years ago
Victor Julien 879c3d8ad7 detect: fix scan-build 0-size alloc warnings 10 years ago
Victor Julien 9a0bbd6239 detect mpm: small optimization 10 years ago
Victor Julien ad3c97f470 detect-mpm: cleanup 10 years ago
Victor Julien 960461f4db fast_pattern: register app layer mpms automatically
Allow for duplicate registrations for the same list. After the first
registration new calls will be ignored.
10 years ago
Victor Julien 6dd4dff7b2 mpm: remove empty app_mpms table 10 years ago
Victor Julien e68b2214e5 tls: register mpm from keywords 10 years ago
Victor Julien 57ae3c43e5 dns_query: register mpm from keyword 10 years ago
Victor Julien a1a2187a0c http_cookie: register mpm from keyword 10 years ago
Victor Julien 74661449e0 http_raw_host: register mpm from keyword 10 years ago
Victor Julien b5cd4889ae http_host: register mpm from keyword 10 years ago
Victor Julien 91695c81aa http_client_body: register mpm from keyword 10 years ago
Victor Julien 644d4dc61b http_stat_code: register mpm from keyword 10 years ago
Victor Julien cf96db095a http_stat_msg: register mpm from keyword 10 years ago
Victor Julien 43b281a510 file_data: register mpm from keyword 10 years ago
Victor Julien 6d0632a9c6 http_method: register mpm from keyword 10 years ago
Victor Julien e4ea38a8de http_raw_header: register mpm from keyword 10 years ago
Victor Julien 7813a834d0 http_user_agent: register mpm from keyword 10 years ago
Victor Julien 7b98c0073f http_header: register mpm from keyword 10 years ago
Victor Julien 38e018e2d3 http_raw_uri: register mpm from keyword 10 years ago
Victor Julien 7289d12f1b http_uri: register mpm from keyword 10 years ago
Victor Julien 5b2e36a1b0 mpm: add App Layer MPM registery
Register keywords globally at start up.

Create a map of the registery per detection engine. This we need because
the sgh_mpm_context value is set per detect engine.

Remove APP_MPMS_MAX.
10 years ago
Victor Julien 732921922a detect mpm: consider sgh direction when adding rules 10 years ago
Victor Julien 9187c20782 detect mpm: negated setup fix 10 years ago