Commit Graph

200 Commits (017b16d42174f3f7597a41088f1b00e830c2ecbd)

Author SHA1 Message Date
Victor Julien 017b16d421 detect-pcre: small cleanups 9 years ago
Victor Julien 22f3205664 var-names: expose outside of detect engine
Until now variable names, such as flowbit names, were local to a detect
engine. This made sense as they were only ever used in that context.

For the purpose of logging these names, this needs a different approach.
The loggers live outside of the detect engine. Also, in the case of
reloads and multi-tenancy, there are even multiple detect engines, so
it would be even more tricky to access them from the outside.

This patch brings a new approach. A any time, there is a single active
hash table mapping the variable names and their id's. For multiple
tenants the table is shared between tenants.

The table is set up in a 'staging' area, where locking makes sure that
multiple loading threads don't mess things up. Then when the preparing
of a detection engine is ready, but before the detect threads are made
aware of the new detect engine, the active varname hash is swapped with
the staging instance.

For this to work, all the mappings from the 'current' or active mapping
are added to the staging table.

After the threads have reloaded and the new detection engine is active,
the old table can be freed.

For multi tenancy things are similar. The staging area is used for
setting up until the new detection engines / tenants are applied to
the system.

This patch also changes the variable 'id'/'idx' field to uint32_t. Due
to data structure padding and alignment, this should have no practical
drawback while allowing for a lot more vars.
9 years ago
Victor Julien dfac5276b8 detect: remove unused SIGMATCH_PAYLOAD flag 9 years ago
Victor Julien fe415ae518 detect: remove DMATCH list 9 years ago
Victor Julien 775e182531 detect: remove AppLayerMatch API call 9 years ago
Victor Julien 6f867c3c73 http_raw_uri: dynamic buffer 9 years ago
Victor Julien ee55aefa1c http_client_body: dynamic buffer 9 years ago
Victor Julien e7d5e845c7 http_header / http_raw_header: dynamic buffers 9 years ago
Victor Julien b694d96e22 http_stat_msg: dynamic buffer 9 years ago
Victor Julien 7e3ab4f5ea http_stat_code: dynamic buffer 9 years ago
Victor Julien 128b59d4f6 http_raw_host: dynamic buffer 9 years ago
Victor Julien 296c275e23 http_host: dynamic buffer 9 years ago
Victor Julien 67b7d9734e http_cookie: dynamic buffer 9 years ago
Victor Julien 54604c7bf2 http_user_agent: dynamic buffer 9 years ago
Victor Julien 6346a074a7 http_uri: dynamic buffer
Clean up tests
9 years ago
Victor Julien e34102d67a http_method: make list dynamic 9 years ago
Victor Julien 6bd37611ee file_data: dynamic buffer 9 years ago
Victor Julien 779d40cedf detect: remove hardcoded sm_list logic from setup
Introduce utility functions to aid this.
9 years ago
Victor Julien f370e88135 detect: move init only Signature members to init_data 9 years ago
Victor Julien bd456076a8 detect: pass SigMatchData to inspect functions 9 years ago
Victor Julien bfd4bc8233 detect: constify Signature/SigMatch use at runtime 9 years ago
Eric Leblond a2e2f50fb9 documentation: fix list keywords URLs
Update URLs in keyword definition to point to sphinx documentation.
9 years ago
Victor Julien 4a190e07a6 pcre: disable JIT if RWX pages not supported 9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Jason Ish 7ce196e3bf detect-pcre: use new unit test macros 9 years ago
Jason Ish 4cdcada397 pcre: fix missing quote in pcre unit test 9 years ago
Victor Julien 9030e89c94 detect: don't set alproto while registering keyword
The field is not used except for some printing, and is wrong for
many keywords.
9 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
9 years ago
maxtors 9d3fd82849 Removed duplicate include statements. 9 years ago
Victor Julien e67ae0f174 detect keywords: use parse regex util func 10 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
10 years ago
Victor Julien 4f8e1f59a6 mpm: remove obsolete mpm algos
Remove: ac-gfbs, wumanber, b2g, b3g.
10 years ago
Victor Julien ecb79391a0 pcre: use pcre_free_study to free the study data 10 years ago
Victor Julien 154891cf26 pcre: work around harmless coverity warnings 10 years ago
Jason Ish 4185f3ee92 detect-pcre: allow upper cases metachars for host (/W).
Redmine issue 1490.
10 years ago
Victor Julien 2c8e8c2516 dns: rename type so it's purpose is more clear 10 years ago
Giuseppe Longo 41a1a9f4af find and replace HSBDMATCH by FILEDATA
This commit do a find and replace of the following:

- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
  sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*

- HSBD by FILEDATA:
  sed -i 's/HSBDMATCH/FILEDATA/g' src/*
10 years ago
Victor Julien dea8dda3c1 vars: redo var name <-> idx handling
Can't use sm type anymore as multiple var carriers (host/flow/etc) will
all have xbits.
11 years ago
Ken Steele 923a77e952 Change Match() function to take const SigMatchCtx*
The Match functions don't need a pointer to the SigMatch object, just the
context pointer contained inside, so pass the Context to the Match function
rather than the SigMatch object. This allows for further optimization.

Change SigMatch->ctx to have type SigMatchCtx* rather than void* for better
type checking. This requires adding type casts when using or assigning it.

The SigMatch contex should not be changed by the Match() funciton, so pass it
as a const SigMatchCtx*.
11 years ago
Victor Julien ee8da21e36 pcre: fix var capture for non relative matches
Var capture setup depended on the match being relative due to a logic
error.
11 years ago
Ken Steele 8f1d75039a Enforce function coding standard
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
11 years ago
Victor Julien 9de536efdb Bug 1098: improve invalid pcre/R handling
When not using a file_data or similar 'sticky buffer', a pcre/R option
needs a content in the same buffer.
11 years ago
Victor Julien 2f14d1e94e Fix compiler warning:
array subscript has type ‘char’ [-Werror=char-subscripts]
12 years ago
Victor Julien 8dbf7a0d78 Update tests to use AppLayerParserThreadCtx ptr instead of void. Fix a few bugs uncovered by this. 12 years ago
Victor Julien fdefb65be4 app-layer: rename AppLayerThreadCtx funcs
AppLayerParserGetCtxThread -> AppLayerParserThreadCtxAlloc
AppLayerParserDestroyCtxThread -> AppLayerParserThreadCtxFree
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Victor Julien 3e604b8703 pcre: parsing cleanup
Remove all flags indicating the buffer type. They were only used
at parse time.

Because of this the DetectPcreData_ structure could shrink to 32
bytes.
12 years ago
Victor Julien dcc75acdec Convert pcre keyword parsing to use pcre_copy_substring 12 years ago
Anoop Saldanha 56143131da Fix unittests that use chunked encoding. 12 years ago