Commit Graph

81 Commits (2fd0025edeb6be5747d2439571b4a68c34dfbbf9)

Author SHA1 Message Date
Victor Julien 2cac440f7d detect/filemagic: fix thread ctx registration; reloads
Make sure thread ctx registration happens and id remains correct
in case of reloads.

To do so, move id var into the detect ctx.
2 years ago
Victor Julien 1f22b51848 detect/filemagic: switch to file.magic implementation
Replace implementation of the legacy `filemagic` keyword by the
implementation for the `file.magic` variant. This leads to better
performance and hooks the rules into the detection engine better.

Remove unittests that depended on the old logic.

Ticket: #6194.
2 years ago
Philippe Antoine e75956717d detect/files: centralize definition of protocols
Protocols supporting files are only defined in one place, which
gets used by all keywords, which can handle some exceptions
(like HTTP2 not having file names)
2 years ago
Victor Julien da89164ba1 detect: spelling: DetectBufferMpmRegistry 2 years ago
Victor Julien 5890a8a8ab detect: register multi-buffer keywords
Register the keywords that use multi buffer support as such, so that
rule parsing can set them up with multi-instance support.

Ticket: #5784.
2 years ago
Victor Julien 2c0e434bb8 detect: pass de_ctx to DetectBufferSetActiveList 2 years ago
Victor Julien 977c5ea719 detect: initialize empty buffers 2 years ago
Victor Julien 576bfc6bf0 detect/file.magic: update buffer initialization logic 2 years ago
Victor Julien 71bc9e75f5 app-layer: get sbconfg with files 2 years ago
Victor Julien b31ffde6f4 output: remove error codes from output 3 years ago
Victor Julien e042cd785e error: use SC_ENOMEM for alloc errors 3 years ago
Alice Akaki 8614bff017 detect-filemagic: convert unittests to FAIL/PASS APIs
Task: #4034
3 years ago
Victor Julien 41265a859e detect/files: optimize file.data by skipping non-file txs
As well as 'file' txs not in our direction.

Implement the same logic for file.name and file.magic prefilter engines.
3 years ago
Victor Julien 3263202094 detect/tx: add AppLayerTxData to PrefilterTx
In preparation of some file inspection optimizations, for which we need the
tx data.

Update all users.
3 years ago
Victor Julien 79499e4769 app-layer: move files into transactions
Update APIs to store files in transactions instead of the per flow state.

Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.

Update all protocol implementations to support this.

Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.

Update the filestore keyword to consider a match if filestore output not
enabled.
3 years ago
Victor Julien 040404b093 detect/profiling: track bytes scanned by prefilter engines 3 years ago
Victor Julien e250ef6402 debug: remove empty header 3 years ago
Victor Julien af145ad125 detect/file: reduce scope of keyword data structures 3 years ago
Victor Julien 73eb7744d8 detect/file: update copyright years 3 years ago
Philippe Antoine 02f2602dde src: rework includes as per cppclean 3 years ago
Philippe Antoine b8524e70d9 detect: change InspectEngineFuncPtr2 to return uint8_t 3 years ago
Victor Julien 4feb0529a4 detect/file: minor code cleanup
Reduce scope where possible. Suggested by cppcheck.
3 years ago
Victor Julien 47629b7aeb detect/filemagic: don't pass unused pointer 3 years ago
Victor Julien d8d1fbe443 detect/files: fix buffer tracking with multiple files 4 years ago
Victor Julien 3dc50322db detect: fix multi inspect buffer issue; clean up
Fix multi inspect buffer API causing cleanup logic in the single
inspect buffer paths. This could lead to a buffer overrun in the
"to clear" logic.

Multi buffers now use InspectionBufferSetupMulti instead of
InspectionBuffer. This is enforced by a check in debug validation.

Simplify the multi inspect buffer setup code and update the callers.
4 years ago
Philippe Antoine 707f027231 protos: renaming ALPROTO_HTTP* constants
Having now ALPROTO_HTTP1, ALPROTO_HTTP2 and ALPROTO_HTTP

Run with 3 sed commands
git grep ALPROTO_HTTP | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP/ALPROTO_HTTP1/g'
git grep ALPROTO_HTTP12 | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP12/ALPROTO_HTTP2/g'
git grep ALPROTO_HTTP1_ANY | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP1_ANY/ALPROTO_HTTP/g'

and then running clang-format
4 years ago
Victor Julien 13cebb1857 detect: fix heap overflow issue with buffer setup
In some cases, the InspectionBufferGet function would be followed by
a failure to set the buffer up, for example due to a HTTP body limit
not yet being reached. Yet each call to InspectionBufferGet would lead
to the matching list_id to be added to the
DetectEngineThreadCtx::inspect.to_clear_queue. This array is sized to
add each list only once, but in this case the same id could be added
multiple times, potentially overflowing the array.
4 years ago
Angelo Mirabella c72069c3b2 detect/magic: fix crash on rule reloading
This changseset fixes a bug causing a segmentation fault.
When rules are reloaded and a rule using libmagic matches, suricata
crashes due to an improper reinitialization of the thread contexts.

Bug: #3726
5 years ago
Philippe Antoine 1422b18a99 http2: initial support 5 years ago
Victor Julien 6ab323d323 detect: hide RegisterTests behind ifdef UNITTESTS
Update all callers to more aggressively use UNITTESTS guards as well.
5 years ago
Eric Leblond ae5650d443 magic: get rid of global lock
Global magic context was involving a lock that appear to be really
costly for some traffic.
5 years ago
Victor Julien 26bcc97515 detect/keywords: dynamic version part of doc URL 5 years ago
Jeff Lucovsky d3a65fe156 detect: Provide `de_ctx` to free functions
This commit makes sure that the `DetectEngineCtx *` is available
to each detector's "free" function.
5 years ago
Victor Julien a4a4d17ad0 app-layer/files: optimize GetFiles calls
Remove FlowGetProtoMapping calls from the GetFiles wrapper and
get the alstate from the flow directly.
5 years ago
Victor Julien 4dff903b35 detect: introduce pkt mpm engines
Instead of the hardcode L4 matching in MPM that was recently introduced,
add an API similar to the AppLayer MPM and inspect engines.

Share part of the registration code with the AppLayer.

Implement for the tcp.hdr and udp.hdr keywords.
6 years ago
Victor Julien 14896365ef detect: remove Threadvars argument from API calls
Remove it as it's (almost) never used. If it is really needed it can
be accessed through DetectEngineThreadCtx::tv as well.
6 years ago
Victor Julien 32fb7d773a detect/content-inspect: turn void arg into Packet
Replace the 'void *data' argument by a 'Packet *p' as this was
the only user left of the data pointer.
6 years ago
Victor Julien 84881bf1b8 detect/file.magic: add sticky buffer
Add sticky buffer to inspect file magic. Includes mpm support.
6 years ago
Eric Leblond 63f2032ed6 doc: fix way to build URL 6 years ago
Victor Julien 164252e381 detect/file: fix minor scan-build warnings 7 years ago
Andreas Herz 2e8678a5ff docs: replace redmine links and enforce https on oisf urls 7 years ago
Victor Julien a636d96b15 detect/file: cleanups
TX id is enfored in the engine, so the keywords don't need to.

Unify detect file engines.
8 years ago
Victor Julien 9b1f74409b magic: fix compile warnings 8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien 342059835f detect-parse: improve common parser
In preparation of turning input to keyword parsers to const add
options to the common rule parser to enforce and strip double
quotes and parse negation support.

At registration, the keyword can register 3 extra flags:

    SIGMATCH_QUOTES_MANDATORY: value to keyword must be quoted

    SIGMATCH_QUOTES_OPTIONAL: value to keyword may be quoted

    SIGMATCH_HANDLE_NEGATION: leading ! is parsed

In all cases leading spaces are removed. If the 'quote' flags are
set, the quotes are removed from the input as well.
8 years ago
Victor Julien 5bafc64c08 detect: unify FileMatch API with other calls 9 years ago
Victor Julien b68343e372 files: use dynamic list 9 years ago
Victor Julien bd456076a8 detect: pass SigMatchData to inspect functions 9 years ago
Victor Julien bfd4bc8233 detect: constify Signature/SigMatch use at runtime 9 years ago
Eric Leblond 569cc5d238 util-file: introduce new functions for file size
This patch introduces the FileDataSize and FileTrackedSize functions.
The first one is just a renaming of the initial FilSize function
whereas the other one is using the newly introduced size field as
value.
9 years ago