New facility to allow a user to specify where to hook a rule
into the engine. This patch adds this for packets, adding two
hooks:
- `all`: to let a rule be evaluated by all rules
- `flow_start`: to have a rule be evaluated only for the first
packet in both directions
Implemented by adding a hook flags field in the packet.
Generic:
<app_proto>:request_started and <app_proto>:response_started
<app_proto>:request_complete and <app_proto>:response_complete
Per protocol, it uses the registered progress (state) values. E.g.
tls:client_hello_done
A rule ruleset could be:
pass tls:client_hello_done any any -> any any (tls.sni; content:"www.google.com"; sid:21; alert;)
drop tls:client_hello_done any any -> any any (sid:22;)
The pass rule is evaluated when the client hello is parsed, and if it
doesn't match the drop rule will be evaluated.
Registers each generic lists as "<alproto>:<progress state>:generic"
(e.g. "tls:client_hello_done:generic").
Ticket: #7485.
Flowbits can make a rule such as a packet rule be treated as a stateful
rule, without actually changing the rule type.
Add a flag to allow reporting such cases via engine analysis.
Task #7456
This commit removes the `_AL_` usage in detect keywords for improved
readability.
Some of the HTTP rule keywords already had counterparts without using
"_AL_". These rule keywords are the legacy content modifier keywords
that now have sticky buffer equivalents.
For these, "_AL_" was removed and a suffix was added to the #define:
src/detect-engine-register.h:151: DETECT_HTTP_COOKIE_CM
src/detect-engine-register.h:153: DETECT_HTTP_METHOD_CM
src/detect-engine-register.h:161: DETECT_HTTP_HEADER_CM
src/detect-engine-register.h:173: DETECT_HTTP_RAW_HEADER_CM
src/detect-engine-register.h:175: DETECT_HTTP_URI_CM
src/detect-engine-register.h:179: DETECT_HTTP_STAT_MSG_CM
src/detect-engine-register.h:181: DETECT_HTTP_STAT_CODE_CM
src/detect-engine-register.h:185: DETECT_HTTP_HOST_CM
Ticket: 5053
The names are now dynamically registered at runtime.
The AppProto alproto enum identifiers are still static for now.
This is the final step before app-layer plugins.
This was done following the fact that this setting was historically
named incorrectly. The purpose of the setting was always to define the
ports that will be prioritized and have rule groups associated w them on
priority. Rename all occurences of this to correctly reflect the purpose
of the setting.
Instead of a Host and IPPair table thresholding layer, use a dedicated
THash to store both. This allows hashing on host+sid+tracker or
ippair+sid+tracker, to create more unique hash keys.
This allows for fewer hash collisions.
The per rule tracking also uses this, so that the single big lock is no
longer a single point of contention.
Reimplement storage for flow thresholds to reuse as much logic as
possible from the host/ippair/rule thresholds.
Ticket: #426.
If a signature uses a condition that requires a real packet, filter
out pseudo packets as early as possible. To do this, the SignatureMask
logic is used.
This allows for the removal of checks for pseudo packets in individual
keywords `Match` functions, which will be done in a follow up commit.
Update analyzer to output the new flag.
Ticket: #7002.
So far, if a port was found to be single which was earlier a part of the
range, port + 1 was added to the list to honor the range that it was a
part of. But, this is incorrect in case the port is 65535 or if the port
was found to be of range when it was earlier a single port.
Bug 6896
If a single port happens before a range port, the port groups created
were incorrect. Fix it to use smarter range check.
For example, given,
80:80 - SGH1
100:120 - SGH2
Range created should be
80:80 - SGH1
100:120 - SGH2
Bug 6881
If a port point is single but later on also a part of a range, it ends
up only creating the port groups for single points and not the range.
Fix it by adding the port next to current single one to unique points
and marking it a range port.
Bug 6843
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.
Ticket #6795
To avoid getting multiple entries in the final port list and to also
make the next step more efficient by reducing the size of the items to
traverse over.
Ticket 6792
Bug 6414
Using the unique port points, create a list of small port ranges which
contain the DetectPort objects and the designated SGHs found by finding
the overlaps with the existing ports and copying the SGHs accordingly.
Ticket 6792
Bug 6414
After all the SGHs have been appropriately copied to the designated
ports, create an interval tree out of it for a faster lookup when later
a search for overlaps is made.
Ticket 6792
Bug 6414
In order to create the smallest possible port ranges, it is convenient
to first have a list of unique ports. Then, the work becomes simple. See
below:
Given, a port range P1 = [1, 8]; SGH1
and another, P2 = [3, 94]; SGH2
right now, the code will follow a logic of recursively cutting port
ranges until we create the small ranges. But, with the help of unique
port points, we get, unique_port_points = [1, 3, 8, 94]
So, now, in a later stage, we can create the ranges as
[1, 2], [3, 7], [8, 8], [9, 94] and copy the designated SGHs where they
belong. Note that the intervals are closed which means that the range
is inclusive of both the points.
The final result becomes:
1. [1, 2]; SGH1
2. [3, 7]; SGH1 + SGH2
3. [8, 8]; SGH1 + SGH2
4. [9, 94]; SGH2
There would be 3 unique rule groups made for the case above.
Group 1: [1, 2]
Group 2: [3, 7], [8, 8]
Group 3: [9, 94]
Ticket 6792
Bug 6414
In the commit 4a00ae607, the whitelisting check was updated in a quest
to make use of the conditional better but it made things worse as every
range would be whitelisted as long as it had any of the default
whitelisted port which is very common.
Present scenario
----------------
Currently, as a part of setting signature count per SGH, a max_idx is
passed which could be as high as the highest signature number (internal
ID).
Issue
-----
Not every SGH needs to evaluate all the signatures while setting
the signature count or while creating the match_array.
In a nonideal scenario, when say, there are 2 SGHs and one SGH has 2
signatures and the other one has 60k, given the current scheme of
evaluating max_idx, the max_idx will be set to 60k, and this shall
later be passed on to SigGroupHeadSetSigCnt or
SigGroupHeadBuildMatchArra which shall traverse over all the 60k sigs
for either SGHs.
Other info
----------
This is a very fast operation as the internal arithmetic is done
bitwise.
Patch
-----
The functions SigGroupHeadSetSigCnt and SigGroupHeadBuildMatchArray can
be optimized by storing the max signature id (internal) per SGH (which
also seemed to be the initial intention as per fn comments).
As a result of this, the sig_array is only walked up until the max sig
id of that respective SGH.
sig_array_size can easily be calculated with length and is only used at
one place for debugging purposes. Remove it from the DetectEngineCtx
struct to avoid making it unnecessarily heavy.
There is nothing Address specific going on in the preparations.
Stage 1: Preprocessing happens. Sigs classified as IP Only, Masks
applied, content specific limits applied, etc and sig array built.
Stage 2: Sigs grouped by IPOnly, ports and protocols.
Stage 3: Decoder Events SGH built.
Stage 4: File flags set, sig grouping done per prefilter, etc.
So far, the condition for checking if the whitelisted port was in the
port range of "a" said
a->port >= w->port && a->port2 <= w->port
But, if a->port <= a->port2, this condition could only be true when
a->port == w->port == a->port2. However, the motivation for this fn was
to be able to find if the whitelisted port for a carrier proto already
was in the range of the given protocol and calculate a score for the
port accordingly.
Fix the range check such that a->port <= w->port <= a->port2.
Shivani Bhardwaj
Victor Julien
Jason Ish
Juliana Fajardini
Jeff Lucovsky
Philippe Antoine