Move the tenant load and reload commands to be executed by the detect
loader thread(s).
Limitation: no yaml parsing in parallel. The Conf API is currently not
thread safe, so don't load the tenant config (yaml) in parallel.
To speed up startup with many tenants, tenant loading will be parallelized.
As no tempary threads should be used for these memory allocation heavy
tasks, this patch adds new type of 'command' thread that can be used to
load and reload tenants.
This patch hardcodes the number of loaders to 4. Future work will make it
dynamic.
The loader thread essentially sleeps constantly. When a tasks is sent to
it, it will wake up and execute it.
Store the tenant id in the flow and use the stored id when setting
up pesudo packets.
For tunnel and defrag packets, get tenant from parent. This will only
pass tenant_id's set at capture time.
For defrag packets, the tenant selector based on vlan id will still
work as the vlan id(s) are stored in the defrag tracker before being
passed on.
Make threshold loading prefix aware, so it can be part of tenant
configuration.
If the setting is missing from the tenant, the global setting is tried
and if that too is missing, the global default is used.
Note: currently per host thresholds are tracked globally and NOT per
tenant.
Make reference loading prefix aware, so it can be part of tenant
configuration.
If the setting is missing from the tenant, the global setting is tried
and if that too is missing, the global default is used.
Make classification loading prefix aware, so it can be part of tenant
configuration.
If the setting is missing from the tenant, the global setting is tried
and if that too is missing, the global default is used.
Allow for a tenant to be reloaded. The command is the same as the
register-tenant command, so with a yaml and tenant-id as argument.
However this replaces an existing tenant.
Arguments:
- tenant id (int)
- name of handler (string)
- traffic id related to handler (int, optional)
Examples:
- register-tenant-handler 1 vlan 1111
- register-tenant-handler 8 pcap
Register tenant handlers/selectors based on what the unix command
"register-tenant-handler" tells.
Check traffic id before adding it. No duplicated registrations for
a traffic id are allowed.
Allow for an optional 'tenant id' argument to pcap-file. This will
allow us to force the pcap to be inspected by this tenant.
If ommited it's 0, which means it's disabled.
Make available to live mode and unix socket mode.
register-tenant:
Loads a new YAML, does basic validation.
Loads a new detection engine
Loads rules
Add new de_ctx to master store and stores tenant id in the de_ctx so
we can look it up by tenant id later.
unregister-tenant:
Gets the de_ctx, moves it to the freelist
Removes config
Introduce DetectEngineGetByTenantId, which gets a reference to the
detect engine by tenant id.
If a flow was 'pass'd, it means that no packet of it will flow be handled
by the detection engine. A side effect of this was that the per flow
inspect_id would never be moved forward. This in turn lead to a situation
where transactions wouldn't be freed.
This patch addresses this case by incrementing the inspect_id anyway for
the pass case.
Stream GAPs and stream reassembly depth are tracked per direction. In
many cases they will happen in one direction, but not in the other.
Example:
HTTP requests a generally smaller than responses. So on the response
side we may hit the depth limit, but not on the request side.
The asynchronious 'disruption' has a side effect in the transaction
engine. The 'progress' tracking would never mark such transactions
as complete, and thus some inspection and logging wouldn't happen
until the very last moment: when EOF's are passed around.
Especially in proxy environments with _very_ many transactions in a
single TCP connection, this could lead to serious resource issues. The
EOF handling would suddenly have to handle thousands or more
transactions. These transactions would have been stored for a long time.
This patch introduces the concept of disruption flags. Flags passed to
the tx progress logic that are and indication of disruptions in the
traffic or the traffic handling. The idea is that the progress is
marked as complete on disruption, even if a tx is not complete. This
allows the detection and logging engines to process the tx after which
it can be cleaned up.
The app layer state 'version' field is incremented with each update
to the state. It is used by the detection engine to see if the current
version of the state has already been inspected. Since app layer and
detect always run closely together there is no need for a big number
here. The detect code really only checks for equal/not-equal, so wrap
arounds are not an issue.
Victor Julien