Also, update StateTransactionFree to take an u64 tx id, so it's
consistant with the rest of the engine.
To reflect these changes, AppLayerRegisterTransactionIdFuncs has
been renamed to AppLayerRegisterTxFreeFunc.
HTP, DNS, SMB, DCERPC parsers updated.
Per TX decoder events resulted in significant overhead to the
detection engine, as it walked all TX' all the time to check
if decoder events were available.
This commit introduces a new API call StateHasEvents, which speeds
up this process, at the expense of keeping a counter in the state.
Implement this for DNS as well.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
The output of the list-keyword is modified to include the url to
the keyword documentation when this is available. All documented
keywords should have their link set.
list-keyword can be used with an optional value:
no option or short: display list of keywords
csv: display a csv output on info an all keywords
all: display a human readable output of keywords info
$KWD: display the info about one keyword.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
This patch required a evolution of Pool API as it is needed to
proceed to alloc or init separetely. The PoolInit has been changed
with a new Init function parameter.
In some cases AppLayerTransactionGetInspectId can return -1, which is
now handled by all it's callers.
Improve logic of selecting which transactions are inspected by the various
HTTP keywords.
In stateful detection only inspect the file portion of the rule after all
other conditions matched. This to prevent "filestore" from tagging files
for storage during a partial match.
Add a couple of unittests to test the behaviour change.
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.
Add macro's to check the family, copy the address, etc.
Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.
On 64 bit this shrinks the flow with 8 bytes.
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
Split stream reassembly in 2 parts: a part that sends ack'd data to the app
layer parsers as soon as it's available, and another part that queues up
data into larger chunks for raw inspection.
This patch generated via coccinelle is getting rid of logging
message after a SCMalloc failure. They were useless as SCMalloc
already displays a message.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:
- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.
This commit fixes bug #124.
The stream engine memory handling needed updating as it didn't scale. Changes:
- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
added.
- all reassembly errors are converted to debug msgs.
- improve locking of application layer handling, making sure that the flow cannot be freed/cleared when the detection engine is still working with it.
- add a check to the app layer detection to make sure that a match function will only inspect an app layer state if it's of the right type.
- Implement "closing" state in flow.
- Add protocol specific timeouts.
- Lots of stream tracking updates, fixing a lot of out of window issues.
- Stream reassembly fixes.
- Implement a new IDS runmode with 4 stream and detect threads.
- Added a BUG_ON macro that aborts the engine if the expression is true.
- Better balance the flow queue handler for traffic that doesn't have flow (like icmp currently).
- Simplify application level protocol in the Tcp Session.
- Add some debugging memory counters.
First version of a simple pattern based L7 proto detection engine. Currently just works by matching a single pattern in the initial data. Implemented HTTP, SSL, MSN, JABBER, SMTP and a few more.
Couple of pattern matcher cleanups.