Allows matching on stickybuffers. Like dsize, it allows matching on
exact values, greater than and less than, and ranges.
For streaming buffers, such as HTTP bodies, the final size of the
body is only known at the end of the transaction.
As we can have multiple files per TX we use the multi inspect
buffer support.
By using this API file_data supports transforms.
Redo part of the flash decompression as a hard coded built-in sort
of transform.
Make sure content is applied to the transformed version of a buffer.
Support content with its modifiers, and also isdataat, pcre, bytetest
and bytejump.
Move previously global table into detect engine ctx. Now that we
can register buffers at rule loading time we need to take concurrency
into account.
Move DetectBufferType to detect.h and update DetectBufferCtx API calls
to include a detect engine ctx reference.
Introduce InspectionBuffer a structure for passing data between
prefilters, transforms and inspection engines.
At rule parsing time, we'll register new unique 'DetectBufferType's
for a 'parent' buffer (e.g. pure file_data) with its transformations.
Each unique combination of buffer with transformations gets it's
own buffer id.
Similarly, mpm registration and inspect engine registration will be
copied from the 'parent' (again, e.g. pure file_data) to the new id's.
The transforms are called from within the prefilter engines themselves.
Provide generic MPM matching and setup callbacks. Can be used by
keywords to avoid needless code duplication. Supports transformations.
Use unique name for profiling, to distinguish between pure buffers
and buffers with transformation.
Add new registration calls for mpm/prefilters and inspect engines.
Inspect engine api v2: Pass engine to itself. Add generic engine that
uses GetData callback and other registered settings.
The generic engine should be usable for every 'simple' case where
there is just a single non-streaming buffer. For example HTTP uri.
The v2 API assumes that registered MPM implements transformations.
Add util func to set new transform in rule and add util funcs for rule
parsing.
Remove DetectAppLayerInspectEngineRegister for TOCLIENT direction
because Modbus inspection engine is only performing in request (TOSERVER).
Detect Value keyword in read access rule. In read access, match on value
is not possible.
Update Modbus keyword documentation.
When loading an empty file, libyaml will fire a single scalar
event causing us to create a key that contains an empty string.
We're not interested in this, so skip an empty scalar value
when expecting a key.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2418
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Until now, the transaction space is assumed to be terse. Transactions
are handled sequentially so the difference between the lowest and highest
active tx id's is small. For this reason the logic of walking every id
between the 'minimum' and max id made sense. The space might look like:
[..........TTTT]
Here the looping starts at the first T and loops 4 times.
This assumption isn't a great fit though. A protocol like NFS has 2 types
of transactions. Long running file transfer transactions and short lived
request/reply pairs are causing the id space to be sparse. This leads to
a lot of unnecessary looping in various parts of the engine, but most
prominently: detection, tx house keeping and tx logging.
[.T..T...TTTT.T]
Here the looping starts at the first T and loops for every spot, even
those where no tx exists anymore.
Cases have been observed where the lowest tx id was 2 and the highest
was 50k. This lead to a lot of unnecessary looping.
This patch add an alternative approach. It allows a protocol to register
an iterator function, that simply returns the next transaction until
all transactions are returned. To do this it uses a bit of state the
caller must keep.
The registration is optional. If no iterator is registered the old
behaviour will be used.
The XDP CPU destination array/set, configured via xdp-cpu-redirect,
will always be fairly small. My different benchmarking showed that
the current modulo hashing into the CPU array can easily result in bad
distribution, expecially if the number of CPU is an even number.
This patch uses a proper hashing function on the input key. The key
used for hashing is inspired by the ippair hashing code in
src/tmqh-flow.c, and is based on the IP src + dst.
An important property is that the hashing is flow symmetric, meaning
that if the source and destintation gets swapped then the selected CPU
will remain the same. This is important for Suricate.
That hashing INITVAL (15485863 the 10^6th prime number) was fairly
arbitrary choosen, but experiments with kernel tree pktgen scripts
(pktgen_sample04_many_flows.sh +pktgen_sample05_flow_per_thread.sh)
showed this improved the distribution.
Signed-off-by: Jesper Dangaard Brouer <netoptimizer@brouer.com>
Adjusted function call API to take an initval. This allow the API
user to set the initial value, as a seed. This could also be used for
inputting the previous hash.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Victor Julien
Andreas Herz
Jason Ish
Philippe Antoine
Eric Leblond
David DIALLO
Pierre Chifflier
Jesper Dangaard Brouer
Jesper Dangaard Brouer