Capture methods that are non blocking will still not generate packets
that go through the system if there is no traffic. Some maintenance
tasks, like rule reloads rely on packets to complete.
This patch introduces a new thread flag, THV_CAPTURE_INJECT_PKT, that
instructs the capture thread to create a fake packet.
The capture implementations can call the TmThreadsCaptureInjectPacket
utility function either with the packet they already got from the pool
or without a packet. In this case the util func will get it's own
packet.
Implementations for pcap, AF_PACKET and PF_RING.
Split wait loop into three steps:
- first insert pseudo packets
- 2nd nudge all capture threads to break out of their loop
- third, wait for the detection thread contexts to be used
Interupt capture more than once if needed
Move packet injection into util func
Cppcheck 1.72 gives a warning on the following code pattern:
char blah[32] = "";
snprintf(blah, sizeof(blah), "something");
The warning is:
(error) Buffer is accessed out of bounds.
While this appears to be a FP, in most cases the initialization to ""
was unnecessary as the snprintf statement immediately follows the
variable declaration.
instead
mappings:
- vlan:
vlan-id: 1
tenant-id: 2
we'll now use:
mappings:
- vlan-id: 1
tenant-id: 2
For YAML it pretty much means the same thing.
Ticket: 1517
Move the tenant load and reload commands to be executed by the detect
loader thread(s).
Limitation: no yaml parsing in parallel. The Conf API is currently not
thread safe, so don't load the tenant config (yaml) in parallel.
To speed up startup with many tenants, tenant loading will be parallelized.
As no tempary threads should be used for these memory allocation heavy
tasks, this patch adds new type of 'command' thread that can be used to
load and reload tenants.
This patch hardcodes the number of loaders to 4. Future work will make it
dynamic.
The loader thread essentially sleeps constantly. When a tasks is sent to
it, it will wake up and execute it.
Register tenant handlers/selectors based on what the unix command
"register-tenant-handler" tells.
Check traffic id before adding it. No duplicated registrations for
a traffic id are allowed.
Make available to live mode and unix socket mode.
register-tenant:
Loads a new YAML, does basic validation.
Loads a new detection engine
Loads rules
Add new de_ctx to master store and stores tenant id in the de_ctx so
we can look it up by tenant id later.
unregister-tenant:
Gets the de_ctx, moves it to the freelist
Removes config
Introduce DetectEngineGetByTenantId, which gets a reference to the
detect engine by tenant id.
This commit do a find and replace of the following:
- DETECT_SM_LIST_HSBDMATCH by DETECT_SM_LIST_FILEDATA
sed -i 's/DETECT_SM_LIST_HSBDMATCH/DETECT_SM_LIST_FILEDATA/g' src/*
- HSBD by FILEDATA:
sed -i 's/HSBDMATCH/FILEDATA/g' src/*
Load the YAML into a prefix "detect-engine-reloads.N" where N is the
reload counter. This way we can load the updated config w/o overwriting
the current one.
Initalize detection engine by configuration prefix.
DetectEngineCtxInitWithPrefix(const char *prefix)
Takes the detection engine configuration from:
<prefix>.<config>
If prefix is NULL the regular config will be used.
Update sure that DetectLoadCompleteSigPath considers the prefix when
retrieving the configuration.
Rename the thread init function DetectEngineThreadCtxInitForLiveRuleSwap
to DetectEngineThreadCtxInitForReload and change it's logic to take the
new detection engine as argument and let it return the
DetectEngineThreadCtx or NULL on error.
The old approach used the thread init API format, but it wasn't used in
that way.
Add DetectEngineReference, which takes a reference to a detect engine,
and make DetectEngineThreadCtxInitForLiveRuleSwap use it. This way
reload will not depend on master staying the same. This allows master
to be updated in between w/o affecting the reload that is in progress.
The minimal detect engine has only the minimal memory use and setup
time. It's to be used for 'delayed' detect where the first detection
engine is essentially empty.
The threads setup are also minimal.
Add code to allow for unittests not following the complete api.
Update replace tests as they don't use the unittests runmode that
powers the workaround based on RunmodeIsUnittests().
Update detect engine management to make it easier to reload the detect
engine.
Core of the new approach is a 'master' ctx, that keeps a list of one or
more detect engines. The detect engines will not be passed to any thread
directly, but instead will only be accessed through the detect engine
thread contexts. As we can replace those atomically, replacing a detect
engine becomes easier.
Each thread keeps a reference to its detect context. When a detect engine
is replaced or removed, it's added to a free list. Once its reference
count reaches 0, it is freed.
Add the modbus.function and subfunction) keywords for public function match in rules (Modbus layer).
Matching based on code function, and if necessary, sub-function code
or based on category (assigned, unassigned, public, user or reserved)
and negation is permitted.
Add the modbus.access keyword for read/write Modbus function match in rules (Modbus layer).
Matching based on access type (read or write),
and/or function type (discretes, coils, input or holding)
and, if necessary, read or write address access,
and, if necessary, value to write.
For address and value matching, "<", ">" and "<>" is permitted.
Based on TLS source code and file size source code (address and value matching).
Signed-off-by: David DIALLO <diallo@et.esia.fr>
If a live reload signal was given before the engine was fully started
up (e.g. pcap file thread waiting for a disk to spin up), a segv could
occur.
This patch only enables live reloads after the threads have been
started up completely.
Buffers in per thread HTTP header, client body and server body storage
would be freed based on the usage indicator instead of the size
indicator.
As the usage indicator (e.g. hsbd_buffers_list_len) could be reset
while leaving the memory untouched for later reuse, the free function
would not iterate over all memory blocks.
Removed DrMemory suppressions as well.
Bug #980.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
Make sure we register the detect.alerts counter before packet runtime starts
even in delayed detect mode. The registration of new counters at packet
runtime is not supported by the counters api and might lead to crashes as there
is no proper locking to allow for this operation.
This changes how delayed detect works a bit. Now we call the ThreadInit
callback twice. The first call will only register the counter. The 2nd call
will do all the other setup. This way the counter is registered before the
counters api starts operating in the packet runtime.
Fixes the segv reported in ticket #1018.
Victor Julien
Justin Viiret
Jason Ish
Eric Leblond
Giuseppe Longo
Ken Steele
DIALLO David
Anoop Saldanha