Commit Graph

210 Commits (3205a8789b418dc601e85bc90b92d0091834fd7c)

Author SHA1 Message Date
Victor Julien 5781c8fc78 pcre: fix \xHH issue for http_host
The http_host keyword checks if the regex contains uppercase characters.
This check was rejecting valid syntax in the following format:

    content:"|2E|suricata"; http_host; pcre:"/\x2Esuricata$/W";

This patch addresses this case.

Bug #1957.
8 years ago
Victor Julien 5c01b40931 tests: update tests for app-layer changes 8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien 342059835f detect-parse: improve common parser
In preparation of turning input to keyword parsers to const add
options to the common rule parser to enforce and strip double
quotes and parse negation support.

At registration, the keyword can register 3 extra flags:

    SIGMATCH_QUOTES_MANDATORY: value to keyword must be quoted

    SIGMATCH_QUOTES_OPTIONAL: value to keyword may be quoted

    SIGMATCH_HANDLE_NEGATION: leading ! is parsed

In all cases leading spaces are removed. If the 'quote' flags are
set, the quotes are removed from the input as well.
8 years ago
Victor Julien d31cb083e9 detect: update tests that mix state/stream inspect 8 years ago
Victor Julien 996112edf5 pktvars: same name pktvars, key-value vars 8 years ago
Victor Julien 1ba8c2fe3a pcre: new way of specifying var names
Until now the way to specify a var name in pcre substring capture
into pkt and flow vars was to use the pcre named substring support:
e.g. /(?P<pkt_somename>.*)/

This had 2 drawbacks:

1. limitations of the name. The name could be max 32 chars, only have
   alphanumeric and the underscore characters. This imposed limitations
   that are not present in flowbits/ints.

2. we didn't actually use the named substrings in pcre through the
   API. We parsed the names separately. So putting the names in pcre
   would actually be wasteful.

This patch introduces a new way of mapping captures with names:

  pcre:"/(.*)/, pkt:somename";
  pcre:"/([A-z]+) ([0-9]+)/, pkt:somename,flow:anothername";

The order of the captures and the order of the names are mapped 1 on 1.
This method is no longer limited by the pcre API's naming limits. The
'flow:' and 'pkt:' prefixes indicate what the type of variable is. It's
mandatory to specify one.

The old method is still supported as well.
8 years ago
Victor Julien 0f708d427b pkt-var: abuse flowvar postmatch logic for pktvars
Flowvars were already using a temporary store in the detect thread
ctx.

Use the same facility for pktvars. The reasons are:

1. packet is not always available, e.g. when running pcre on http
   buffers.

2. setting of vars should be done post match. Until now it was also
   possible that it is done on a partial match.
8 years ago
Victor Julien 5e39486399 pkt-var: use id instead of name pointer 8 years ago
Victor Julien a0bd15a1c4 pcre: support multiple captures
Support up to 8 substring captures into pkt or flow vars.
8 years ago
Victor Julien 017b16d421 detect-pcre: small cleanups 8 years ago
Victor Julien 22f3205664 var-names: expose outside of detect engine
Until now variable names, such as flowbit names, were local to a detect
engine. This made sense as they were only ever used in that context.

For the purpose of logging these names, this needs a different approach.
The loggers live outside of the detect engine. Also, in the case of
reloads and multi-tenancy, there are even multiple detect engines, so
it would be even more tricky to access them from the outside.

This patch brings a new approach. A any time, there is a single active
hash table mapping the variable names and their id's. For multiple
tenants the table is shared between tenants.

The table is set up in a 'staging' area, where locking makes sure that
multiple loading threads don't mess things up. Then when the preparing
of a detection engine is ready, but before the detect threads are made
aware of the new detect engine, the active varname hash is swapped with
the staging instance.

For this to work, all the mappings from the 'current' or active mapping
are added to the staging table.

After the threads have reloaded and the new detection engine is active,
the old table can be freed.

For multi tenancy things are similar. The staging area is used for
setting up until the new detection engines / tenants are applied to
the system.

This patch also changes the variable 'id'/'idx' field to uint32_t. Due
to data structure padding and alignment, this should have no practical
drawback while allowing for a lot more vars.
8 years ago
Victor Julien dfac5276b8 detect: remove unused SIGMATCH_PAYLOAD flag 8 years ago
Victor Julien fe415ae518 detect: remove DMATCH list 8 years ago
Victor Julien 775e182531 detect: remove AppLayerMatch API call 8 years ago
Victor Julien 6f867c3c73 http_raw_uri: dynamic buffer 8 years ago
Victor Julien ee55aefa1c http_client_body: dynamic buffer 8 years ago
Victor Julien e7d5e845c7 http_header / http_raw_header: dynamic buffers 8 years ago
Victor Julien b694d96e22 http_stat_msg: dynamic buffer 8 years ago
Victor Julien 7e3ab4f5ea http_stat_code: dynamic buffer 8 years ago
Victor Julien 128b59d4f6 http_raw_host: dynamic buffer 8 years ago
Victor Julien 296c275e23 http_host: dynamic buffer 8 years ago
Victor Julien 67b7d9734e http_cookie: dynamic buffer 8 years ago
Victor Julien 54604c7bf2 http_user_agent: dynamic buffer 8 years ago
Victor Julien 6346a074a7 http_uri: dynamic buffer
Clean up tests
8 years ago
Victor Julien e34102d67a http_method: make list dynamic 8 years ago
Victor Julien 6bd37611ee file_data: dynamic buffer 8 years ago
Victor Julien 779d40cedf detect: remove hardcoded sm_list logic from setup
Introduce utility functions to aid this.
8 years ago
Victor Julien f370e88135 detect: move init only Signature members to init_data 8 years ago
Victor Julien bd456076a8 detect: pass SigMatchData to inspect functions 8 years ago
Victor Julien bfd4bc8233 detect: constify Signature/SigMatch use at runtime 8 years ago
Eric Leblond a2e2f50fb9 documentation: fix list keywords URLs
Update URLs in keyword definition to point to sphinx documentation.
8 years ago
Victor Julien 4a190e07a6 pcre: disable JIT if RWX pages not supported 9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Jason Ish 7ce196e3bf detect-pcre: use new unit test macros 9 years ago
Jason Ish 4cdcada397 pcre: fix missing quote in pcre unit test 9 years ago
Victor Julien 9030e89c94 detect: don't set alproto while registering keyword
The field is not used except for some printing, and is wrong for
many keywords.
9 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
9 years ago
maxtors 9d3fd82849 Removed duplicate include statements. 9 years ago
Victor Julien e67ae0f174 detect keywords: use parse regex util func 9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
9 years ago
Victor Julien 4f8e1f59a6 mpm: remove obsolete mpm algos
Remove: ac-gfbs, wumanber, b2g, b3g.
9 years ago
Victor Julien ecb79391a0 pcre: use pcre_free_study to free the study data 9 years ago
Victor Julien 154891cf26 pcre: work around harmless coverity warnings 9 years ago
Jason Ish 4185f3ee92 detect-pcre: allow upper cases metachars for host (/W).
Redmine issue 1490.
10 years ago
Victor Julien 2c8e8c2516 dns: rename type so it's purpose is more clear 10 years ago
Giuseppe Longo 41a1a9f4af find and replace HSBDMATCH by FILEDATA
This commit do a find and replace of the following:

- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
  sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*

- HSBD by FILEDATA:
  sed -i 's/HSBDMATCH/FILEDATA/g' src/*
10 years ago
Victor Julien dea8dda3c1 vars: redo var name <-> idx handling
Can't use sm type anymore as multiple var carriers (host/flow/etc) will
all have xbits.
10 years ago
Ken Steele 923a77e952 Change Match() function to take const SigMatchCtx*
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.

Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.

The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
10 years ago