Commit Graph

137 Commits (50bb995458d49d01c0a13c0947edeb6b9d82c818)

Author SHA1 Message Date
Victor Julien f536099a67 app-layer: de_state optimization
Add API to bypass expensive TX list walks. This API call is optional.

Implement it for HTTP and DNS.
10 years ago
Victor Julien 5f0678120d detect-state: update test to check state storing 10 years ago
Victor Julien 37f0bd57b6 detect-state: handle duplicate inspect/match
If for a packet we have a TX N that has detect state and a TX N+1 that
has no detect state, but does have 'progress', we have a corner case
in stateful detection.

ContinueDetection inspects TX N, but cannot flag the rule in the
de_state_sig_array as the next (TX N+1) has already started and needs
to be inspected. 'StartDetection' however, is then unaware of the fact
that ContinueDetection already inspected the rule. It uses the per
session 'inspect_id' that is only moved forward at the end of the
detection run.

This patch adds a workaround. It uses the DetectEngineThreadCtx::
de_state_sig_array to store an offset between the 'base' inspect_id
and the inspect_id that StartDetection should use. The data type is
limited, so if the offset would be too big, a search based fall back
is implemented as well.
10 years ago
Victor Julien bc6e4140be detect: add de_state duplication check
Add test to check if no duplicate destate is created.

Only enabled with DEBUG_VALIDATION.
10 years ago
Victor Julien 9d198e6662 detect-state: fix state storing
Fix storing state and bypassing detection. Previously we'd store
on a match only, meaning that StartDetection would rerun often.

Make sure StartDetection only stores if there is something to store.
10 years ago
Victor Julien 4e177bc9d6 detect-state: cleanups and comments 10 years ago
Victor Julien 304c711a27 detect-state: use f->protomap instead of FlowGetProtoMapping(f->proto) 10 years ago
Victor Julien 5111aa2ec0 detect-state: handle 'post match' locking
The post match list was called with an unlocked flow until now.
However, recent de_state handling updates changed this. The stateful
detection code can now call the post match functions while keeping
the flow locked. The normal detection code still calls it with an
unlocked flow.

This patch adds a hint to the DetectEngineThreadCtx called
'flow_locked' that is set to true if the caller has already locked
the flow.
10 years ago
Victor Julien 18f0351d9b detect-state: fix profiling 10 years ago
Victor Julien e250040b72 detect-state: implement tx state reset for reload
In case of Detect Reload, we need to reset active tx' state.
11 years ago
Victor Julien da3e8ad8f6 detect-state: split flow and tx state
Use separate data structures for storing TX and FLOW (AMATCH) detect
state.

- move state storing into util funcs
- remove de_state_m
- simplify reset state logic on reload
11 years ago
Victor Julien 866d9684ea detect-state: fix profiling 11 years ago
Victor Julien 7e75279977 detect-state: various cleanups 11 years ago
Victor Julien eec22ce19b detect-state: rip per sig detect out of ContinueDetect 11 years ago
Victor Julien bf818b8fb2 detect-state: remove DeStateResetFileInspection
It was effectively unused.
11 years ago
Victor Julien 206f9d4010 detect-state: remove redundant code 11 years ago
Victor Julien e390e24a7c detect-state: add helper to test state
Add little helper function StateIsValid() to test if the state
can be inspected safely.

Cleans up stateful detection loops.
11 years ago
Victor Julien 072ae12771 detect-state: add helper to indicate last tx
Add little helper to indicate current tx is that last we have.
11 years ago
Victor Julien b710f2dd59 detect-state: cleanup ContinueDetection
Only lock f->de_state->m when we start to access it. So after
declaration and initialization of local vars.
11 years ago
Victor Julien 54cb2b6877 detect-state: cleanup retvals
Use DETECT_ENGINE_INSPECT_SIG_* instead of 0, 1, 2 and 3.
11 years ago
Victor Julien e9857200b3 detect: set action from utility function
Set actions that are set directly from Signatures using the new
utility function DetectSignatureApplyActions. This will apply
the actions and also store info about the 'drop' that first made
the rule drop.
11 years ago
Victor Julien 6720496324 detect: fix continue detection with amatch and tx
When using AMATCH, continue detection would fail if the tx part
had already run. This lead to start detection rerunning, causing
multiple alerts for the same issue.
11 years ago
Victor Julien 3b98a1ce66 detect: track current tx_id in det_ctx
When using the inspection engines, track the current tx_id in the
thread storage the detect thread uses. As 0 is a valid tx_id, add
a simple bool that indicates if the tx_id field is set.
11 years ago
Ken Steele 8f1d75039a Enforce function coding standard
Functions should be defined as:

int foo(void)
{
}

Rather than:
int food(void) {
}

All functions where changed by a script to match this standard.
11 years ago
Victor Julien 0564a8da3c detect: add more defensive checks for flow handling
Don't unconditionally deref f->alparser in detection through
DeStateFlowHasInspectableState(). In very rare cases it can
be NULL.
11 years ago
Victor Julien fc559ce227 detect: fix alstate handling
Previously, the alstate use in the main detect loop was unsafe. The
alstate pointer would be set duing a lock, but it would again be used
after one or more lock/unlock cycles. If the data pointed to would
disappear, a dangling pointer would be the result.

Due to they way flows are cleaned up using reference counting and
such, changes of this happening were very small. However, at least
one path can lead to this situation. So it had to be fixed.
11 years ago
Victor Julien b6e2a6f525 detect: locking update continued
Make DeStateDetectContinueDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
11 years ago
Victor Julien cf31e2cc74 detect: locking update
Make DeStateDetectStartDetection get it's own alstate pointer instead
of using the one that was passed to it. We now get and use it only
inside a flow lock.
11 years ago
Victor Julien 6e0112d737 detect: modify AMATCH locking
This is an intrusive change. This patch modifies the way AMATCH
inspection uses locking.

So far, each keyword did it's own locking. This lead to a situation
where a 'alstate' pointer was passed around that was not always
protected by a lock.

This patch moves the locking to the Stateful detection functions.
11 years ago
Victor Julien a96446d39e detect state: fix indent
AMATCH block was indented too far.
11 years ago
Victor Julien 6c8ca76942 detect state: remove alproto check for AMATCH
Not all AMATCHes set a alproto.
11 years ago
Victor Julien c9436a6aef Fix app-layer-protocol FP on multi TX flow
In case of multiple transactions, the stored AMATCH list would not have
been reset, but it would still be reconsidered. Even though none would
match, the engine would still conclude that the rule matched.
11 years ago
Victor Julien 2c3a92a1c9 profiling: conditional rule profiling
Add support for conditional rule profiling. Currently only simple
rate limiting is supported, but hardcoded to inspecting rules for
each packet.
12 years ago
Victor Julien 8dbf7a0d78 Update tests to use AppLayerParserThreadCtx ptr instead of void. Fix a few bugs uncovered by this. 12 years ago
Victor Julien fdefb65be4 app-layer: rename AppLayerThreadCtx funcs
AppLayerParserGetCtxThread -> AppLayerParserThreadCtxAlloc
AppLayerParserDestroyCtxThread -> AppLayerParserThreadCtxFree
12 years ago
Victor Julien f5f148805c app layer: uint16_t alproto -> AppProto alproto
This conversion was missing in a couple of places.
12 years ago
Anoop Saldanha 429c6388f6 App layer API rewritten. The main files in question are:
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].

Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
  parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
  "dnsudp" in the rules.  A user who previously wrote a rule like this -

  "alert dnstcp....." or
  "alert dnsudp....."

  would now have to use,

  alert dns (ipproto:tcp;) or
  alert udp (app-layer-protocol:dns;) or
  alert ip (ipproto:udp; app-layer-protocol:dns;)

  The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
  callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
  STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
  functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0.  This is
  needed by unittests, which would try to clean the flow, and that would
  call the api, AppLayerParserCleanupParserState(), which would try to
  clean the app state, but the app layer now needs an ipproto to figure
  out which api to internally call to clean the state, and if the ipproto
  is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
  they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
12 years ago
Victor Julien 7b687da726 profiling: add tracking of missing keywords 12 years ago
Victor Julien 5686c673ec profiling: per buffer profiling 12 years ago
Victor Julien 97bfcac444 profiling: introduce per keyword profiling
Initial version of per keyword profiling. Prints stats about
how ofter a keyword was checked and what the costs were.
12 years ago
Victor Julien edeeb7ed44 Store TX id with alerts
When generating an alert and storing it in the packet, store the tx_id
as well. This way the output modules can log the tx_id and access the
proper tx for logging.

Issue #904.
12 years ago
Anoop Saldanha c5cd3562d0 Stateful detection inspection continuation API call should update per
signature's Sigmatch entry as well.
12 years ago
Anoop Saldanha 94e40907e2 feature #727 - Add support for app-layer-protocol:<protocol> keyword 12 years ago
Eric Leblond cd3e32ce19 unittests: some functions needs a flow lock.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
12 years ago
Anoop Saldanha 48cf0585fb Suricata upgrade to libhtp 0.5.x.
Remove the support for now unsupported personalities from libhtp -
TOMCAT_6_0, APACHE and APACHE_2_2.  We instead use the APACHE_2
personality.
12 years ago
Victor Julien 43ba5a677e DNS: enable mpm/fast_pattern support for dns_query 12 years ago
Victor Julien f10dd603ff DNS: adding dns_request content modifier 12 years ago
Eric Leblond 3f107fa130 decode: Packet action start with PACKET
Rename all Packet action macro to have them prefixed by PACKET.
12 years ago
Eric Leblond efaa9a7302 action handling: define and use macros
The action field in Packet structure should not be accessed
directly as the tunneled packet needs to update the root packet
and not the initial packet.

This patch is fixing issue #819 where suricata was not able to
drop fragmented packets in AF_PACKET IPS mode. It also fixes
drop capability for tunneled packets.
12 years ago
Victor Julien 6e18ed0489 luajit flowvar support
This patch adds flowvar support to luajit. It does so by exposing two special
C functions to the luajit scripts: ScFlowvarGet and ScFlowvarSet.
12 years ago