AppLayerParserProtocolIsTxEventAware would check if a proto is tx
event aware by checking if it had registered a StateHasEvents function.
However, this is an optimization function. This patch changes it to
use the StateGetEvents function instead, which is a better indicator.
The new API call:
int AppLayerParserProtocolHasLogger(uint8_t ipproto,
AppProto alproto)
Returns TRUE if a logger is registered on the ip/alproto pair, and
FALSE otherwise.
If a protocol parser is active without a logger when detection is
disabled, the transaction handling logic would fail. Now it will
return the proper tx id so we can clean up the complete transactions.
When running w/o detect, TX cleanup handling needs to ignore the
inspect_id as it's only updated by detect.
This patch introduces a new ActiveTx handler for logging only:
AppLayerTransactionGetActiveLogOnly
If --disable-detection is passed on the commandline, this function
is registered.
Move app layer event handling into app-layer-event.[ch].
Convert 'Set' macro's to functions.
Get rid of duplication in Set and SetRaw. Set now calls SetRaw.
Fix potentential int overflow condition in the event storage.
Update callers.
In preparation of a patchset that will allow for disabling the detect
module, this patch introduces a way to register a function for getting
the lowest active tx id. This is used by the app layer for cleaning up
transactions that already fully inspected, and by the flow timeout code
to determine if a flow is fully inspected and logged at timeout.
The registration function RegisterAppLayerGetActiveTxIdFunc allows for
registration of a custom function of type:
uint64_t (*GetActiveTxIdFunc)(Flow *f, uint8_t flags);
If no function is called, AppLayerTransactionGetActiveDetectLog is used,
which implements the existing behaviour of considering both the
inspect_id's and the log_id.
In AppLayerTransactionsCleanup instead of figuring out 'done' tx id's
itself, now call AppLayerTransactionGetActive for both directions to
figure out the completed TX id's.
For AppLayerThreadCtx, AppLayerParserState, AppLayerParserThreadCtx
and AppLayerProtoDetectThreadCtx, use opaque pointers instead of
void pointers.
AppLayerParserState is declared in flow.h as it's part of the Flow
structure.
AppLayerThreadCtx is declared in decode.h, as it's part of the
DecodeThreadVars structure.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
of the archaic features we use in the app layer. We will reintroduce this
parser shortly. Also do note that keywords that rely on the ssh parser
would now be disabled.
Flow timeout code worked by luck when checking if a flow still needed
reassembly for app layer inspection or logging. It would check for a
part of raw reassembly (smsg list) to determine if detection was
needed. In this case it would also process app layer cleanup,
including logging.
Introduced AppLayerTransactionGetActive which returns the lowest tx_id
in a direction that still needs some work.
FlowForceReassemblyNeedReassmbly now uses it to determine if the
applayer still needs work.
Converted FlowForceReassemblyForHash to use the checking function
FlowForceReassemblyNeedReassmbly as well, so that checking if a flow
needs work is now unified.
When logging is disabled, the app layer would still be flagged
as logging. This caused transactions not to be freed until the
end of the flow as the logged tx id would never increment.
This fix postpones the setting of the app layer parser "logger"
flag to the point where we know the logger is enabled.
This patch is a result of applying the following coccinelle
transformation to suricata sources:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc|SCMallocAligned|SCRealloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
We don't support jabber protocol detection atm. Disable the code check
inside suricata to check if jabber protocol detection is enabled in the
yaml file.
Also updated an error log message for app layer.
app-layer-parser.c: In function ‘AppLayerPPTestData’:
app-layer-parser.c:2525:9: error: variable ‘dir’ set but not used [-Werror=unused-but-set-variable]
int dir = 0;
^
Removed a flag parameter introuced earlier to indicate the data
that is first acceptable by the parser. We now use a differently
named parameter to carry out the same activity.
1. Proto detection
2. Parsers
For app layer protocols.
libhtp has now been moved to the section under app-layer.protocols.http,
but we still provide backward compatibility with older conf files.
In debug validation mode, it is required to call application layer
parsing and other functions with a lock on flow. This patch updates
the code to do so.
Also, update StateTransactionFree to take an u64 tx id, so it's
consistant with the rest of the engine.
To reflect these changes, AppLayerRegisterTransactionIdFuncs has
been renamed to AppLayerRegisterTxFreeFunc.
HTP, DNS, SMB, DCERPC parsers updated.
Per TX decoder events resulted in significant overhead to the
detection engine, as it walked all TX' all the time to check
if decoder events were available.
This commit introduces a new API call StateHasEvents, which speeds
up this process, at the expense of keeping a counter in the state.
Implement this for DNS as well.
Improved accuracy, improved performance. Performance improvement
noticeable with http heavy traffic and ruleset.
A lot of other cosmetic changes carried out as well. Wrappers introduced
for a lot of app layer functions.
Failing dce unittests disabled. Will be reintroduced in the updated dce
engine.
Cross transaction matching taken care of. FPs emanating from these
matches have now disappeared. Double inspection of transactions taken
care of as well.
The output of the list-keyword is modified to include the url to
the keyword documentation when this is available. All documented
keywords should have their link set.
list-keyword can be used with an optional value:
no option or short: display list of keywords
csv: display a csv output on info an all keywords
all: display a human readable output of keywords info
$KWD: display the info about one keyword.
When handling error case on SCMallog, SCCalloc or SCStrdup
we are in an unlikely case. This patch adds the unlikely()
expression to indicate this to gcc.
This patch has been obtained via coccinelle. The transformation
is the following:
@istested@
identifier x;
statement S1;
identifier func =~ "(SCMalloc|SCStrdup|SCCalloc)";
@@
x = func(...)
... when != x
- if (x == NULL) S1
+ if (unlikely(x == NULL)) S1
This patch required a evolution of Pool API as it is needed to
proceed to alloc or init separetely. The PoolInit has been changed
with a new Init function parameter.
In some cases AppLayerTransactionGetInspectId can return -1, which is
now handled by all it's callers.
Improve logic of selecting which transactions are inspected by the various
HTTP keywords.
In stateful detection only inspect the file portion of the rule after all
other conditions matched. This to prevent "filestore" from tagging files
for storage during a partial match.
Add a couple of unittests to test the behaviour change.
Introduce a separate FlowAddress structure for holding the ipv4 or ipv6 address
that doesn't have the family in it like the Address structure. Instead, the
family is stored in the flow as a flag: FLOW_IPV4 and FLOW_IPV6.
Add macro's to check the family, copy the address, etc.
Update many unittests to reflect these changes. Introduce unittest helper
functions for creating and initializing a flow and freeing it again.
On 64 bit this shrinks the flow with 8 bytes.
Removal of per flow 'aldata' array. It contained a ptr for each ALPROTO. Instead now we have 2 ptrs in the flow: alparser and alstate.
Various cleanups and dead code removal from the app layer API.
Should safe 100+ bytes memory per flow on 64 bit.
Updated lots of unittests to reflect these changes.
Split stream reassembly in 2 parts: a part that sends ack'd data to the app
layer parsers as soon as it's available, and another part that queues up
data into larger chunks for raw inspection.
This patch generated via coccinelle is getting rid of logging
message after a SCMalloc failure. They were useless as SCMalloc
already displays a message.
Stateful detection for app layer detection keywords, except uricontent. Stores it's partial results in the flow structure. Other modifications:
- Generalize transaction tracking, logging and inspection.
- Adapt http and dcerpc to use the new transaction handling.
- Stream engine now always notifies app layer of a stream eof.
This commit fixes bug #124.
The stream engine memory handling needed updating as it didn't scale. Changes:
- pools can now be initialized to size 0, meaning unlimited
- stream engine uses a memcap setting. Sessions, segments and aldata is part
of this, app layer state isn't.
- memory is accounted using a global int that is spinlocked.
- a counter for sessions that have not been picked up because of memcap was
added.
- all reassembly errors are converted to debug msgs.
- improve locking of application layer handling, making sure that the flow cannot be freed/cleared when the detection engine is still working with it.
- add a check to the app layer detection to make sure that a match function will only inspect an app layer state if it's of the right type.
- Implement "closing" state in flow.
- Add protocol specific timeouts.
- Lots of stream tracking updates, fixing a lot of out of window issues.
- Stream reassembly fixes.
- Implement a new IDS runmode with 4 stream and detect threads.
- Added a BUG_ON macro that aborts the engine if the expression is true.
- Better balance the flow queue handler for traffic that doesn't have flow (like icmp currently).
- Simplify application level protocol in the Tcp Session.
- Add some debugging memory counters.
First version of a simple pattern based L7 proto detection engine. Currently just works by matching a single pattern in the initial data. Implemented HTTP, SSL, MSN, JABBER, SMTP and a few more.
Couple of pattern matcher cleanups.