Fixes memory leak when parsing threshold rules.
All parsed strings are less than 16 characters except
for the IP address which could be up to 48 characters.
Remove redefinition of MAX_SUBSTRINGS
Allow a plugin to register itself as a capture source. This isn't that
much different than how current sources register, it just happens
a little later on during startup.
One "slot" is reserved for capture plugins, but multiple plugins
implementing a capture can be loaded. The --capture-plugin command
line option must be used to tell Suricata which plugin
to use.
This is still very much a work in progress, but can load
PF_RING as a capture plugin.
A filetype plugin is a plugin that implements an eve filetype. Most
of the current filetypes could likely be implemented as such a plugin.
Such a plugin must implement Open, Close and Write, where Write
is provided the formatted JSON to be logged.
This commit also includes the plumbing for plugin loading. Example
plugin to come.
Plugins are loaded by the "plugin" section in the configuration
file:
plugins:
- /path/to/directory/plugins
- /path/to/plugin_file.so
This can also be done on the command line with:
--set plugins.0=/path/plugin_file.so
This commit logs http2 as an http event. The idea is to somewhat
normalize http/http2 so common info can be version agnostic.
This puts the http2 specific fields in an "http2" object inside
the "http" object.
HTTP2 headers/values that are in common with HTTP1 are logged
under the "http" object to be compatible with HTTP1 logging.
Goals:
- reduce locking
- take advantage of 'hot' caches
- better locality
Locking reduction
New flow spare pool. The global pool is implmented as a list of blocks,
where each block has a 100 spare flows. Worker threads fetch a block at
a time, storing the block in the local thread storage.
Flow Recycler now returns flows to the pool is blocks as well.
Flow Recycler fetches all flows to be processed in one step instead of
one at a time.
Cache 'hot'ness
Worker threads now check the timeout of flows they evaluate during lookup.
The worker will have to read the flow into cache anyway, so the added
overhead of checking the timeout value is minimal. When a flow is considered
timed out, one of 2 things happens:
- if the flow is 'owned' by the thread it is handled locally. Handling means
checking if the flow needs 'timeout' work.
- otherwise, the flow is added to a special 'evicted' list in the flow
bucket where it will be picked up by the flow manager.
Flow Manager timing
By default the flow manager now tries to do passes of the flow hash in
smaller steps, where the goal is to do full pass in 8 x the lowest timeout
value it has to enforce. So if the lowest timeout value is 30s, a full pass
will take 4 minutes. The goal here is to reduce locking overhead and not
get in the way of the workers.
In emergency mode each pass is full, and lower timeouts are used.
Timing of the flow manager is also no longer relying on pthread condition
variables, as these generally cause waking up much quicker than the desired
timout. Instead a simple (u)sleep loop is used.
Both changes reduce the number of hash passes a lot.
Emergency behavior
In emergency mode there a number of changes to the workers. In this scenario
the flow memcap is fully used up and it is unavoidable that some flows won't
be tracked.
1. flow spare pool fetches are reduced to once a second. This avoids locking
overhead, while the chance of success was very low.
2. getting an active flow directly from the hash skips flows that had very
recent activity to avoid the scenario where all flows get only into the
NEW state before getting reused. Rather allow some to have a chance of
completing.
3. TCP packets that are not SYN packets will not get a used flow, unless
stream.midstream is enabled. The goal here is again to avoid evicting
active flows unnecessarily.
Better Localily
Flow Manager injects flows into the worker threads now, instead of one or
two packets. Advantage of this is that the worker threads can get packets
from their local packet pools, avoiding constant overhead of packets returning
to 'foreign' pools.
Counters
A lot of flow counters have been added and some have been renamed.
Overall the worker threads increment 'flow.wrk.*' counters, while the flow
manager increments 'flow.mgr.*'.
Additionally, none of the counters are snapshots anymore, they all increment
over time. The flow.memuse and flow.spare counters are exceptions.
Misc
FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue.
Flow no longer has 'prev' pointers and used a unified 'next' pointer for
both hash and queue use.
Call Defrag and others only once per second. Flow Manager may wake
up (much) more often when flow engine is under resource pressure.
As this does not affect Defrag and others, it only unnecessarily
adds load.
Describe Changes
- Added ability to recursively read pcap directories
- src/suricata.c: addition of new command line parameter
--pcap-file-recursive
- src/source-pcap-file.c: parsing of the command line argument
- src/source-pcap-file-directory-helper.h: two thread vars tracking
directory depth and should recurse
- src/util-error.c / src/util-error.h:
Added new warning code "SC_WARN_PATH_READ_ERROR"
- Redmine ticket: https://redmine.openinfosecfoundation.org/issues/2363
Ticket: #2363
This commit provides changes to util-path.c and util-path.h
to support the recursive reading of directories. It adds
4 functions.
- SCIsRegularFile to provide OS independent file info.
- SCIsRegularDirectory to provide OS independent directory info.
- SCRealPath is an OS independent wrapper for realpath.
- PathJoin to manage path resolution logic.
This commit adds MAC address output to the EVE-JSON format. We follow the
remarks made in Redmine ticket #962: for packets, log MAC src/dst as a
scalar field in EVE; for flows, log MAC src/dst as lists in EVE. Field names
are different between flow and packet context to avoid type confusion
(src_mac vs. src_macs). Configuration approach and JSON representation is
taken from previous GitHub PR #2700.
This commit restricts the anomaly logger count. The restriction is
necessary due to state maintenance in the logger that doesn't scale
beyond a single logger.
Until that issue's solved, when multiple anomaly loggers are configured,
an error message will be emitted to highlight the restriction.
This commit modifies the JSON loggers with changes necessary to support
multi-threaded EVE output.
Each "thread-init" function sets up the per-thread log file context for
subsequent calls to the JSON output to buffer function.
This commit changes an internal-only function to remove a parameter
that's invariant in all use cases. This allows an JSON builder
optimization to be used.
Fixes https://redmine.openinfosecfoundation.org/issues/2689
Adds a new source file to handle this keyword.
And modifies documentation, Makefile, and registration accordingly.
url_decode decodes url-encoded data, ie replacing '+' with space
and '%HH' with its value.
When the flow engine enters emergency mode, 3 things happen:
1. a different set of (lower) timeout values are applied
2. the flow manager runs more often
3. worker threads go get a flow directly from the hash table
Testing showed that performance went down significantly due to concurrency
issues:
1. worker threads would fight each other over the hash access
2. flow manager would get in the way of workers
This patch changes the behavior in 2 ways:
1. it makes the flow manager slightly less aggressive. It will still
try to run ~3 times per second, but no longer 10 times.
This should be reducing the contention. At the same time flows
won't time out faster if they are checked many times per second.
2. The 'get a used flow' logic optimizes the use of atomics by only
doing an atomic operation once, and while doing so reserving
a slice of the hash per worker.
The worker will also give up much quicker, to avoid the overhead
of hash walking and taking and releasing locks.
These combined changes show much better 'under stress' behavior, esp
on multi-NUMA systems.
Flag the last flow timeout pseudo packet so that we can force
TX logging w/o setting both app-layer flags.
Case this fixes:
1. flow times out when only TS TCP data received, but non of it is ACK'd.
So there is no app-layer proto yet, or app state or Flow::alparser. So
EOF flags can't be set.
2. Flow timeout sees no reason to create pseudo packet in TC direction.
3. TS pseudo packet finds HTTP, creates HTTP state, flag EOF TS.
4. TX logging skips HTTP logging because:
- TC progress not reached
- EOF TC flag not set.
The solution has been to flag the very last packet for the flow as such
and use it has a master-EOF flag.
When the stream engine has data ready for the app-layer it will call
this API from a loop instead of just once. The loop is to ensure that
if we have a very lossy stream where between 'app_progress' and
'last_ack' there are multiple chunks of data and multiple gaps we
process all the chunks.
DCERPC parser so far provided support for single transactions only.
Extend that to support multiple transactions.
In order for multiple transactions to work, there is always a
transaction identifier for any protocol in its header that lets a
response match the request. In DCERPC, for TCP, that param is call_id in
the header which is a 32 bit field. For UDP, however since it uses
different version of RPC (4.x), this is defined by serial number field
defined in the header. This field however is not contiguous and needs to
be assembled by the provided serial_low and serial_hi fields.
Fixes issue 2845.
pcap_stats is based on 32bit counters and given a big enough throughput
will overflow them. This was reported by people using Myricom cards which
should only be a happenstance. The problem exists for all pcap-based
interfaces.
Let's use internal 64bit counters that drag along the pcap_stats and
handle pcap_stats wrap-around as we update the 64bit stats "often enough"
before the pcap_stats can wrap around twice.
Optional callback a parser can register for applying configuration
to the 'transaction'. Most parsers have a bidirectional tx. For those
parsers that have different types of transaction handling, this new
callback can be used to properly apply the config.
AppLayerTxData is a structure each tx should include that will contain
the common fields the engine needs for tracking logging, detection and
possibly other things.
AppLayerTxConfig will be used by the detection engine to configure
the transaction.
This way the AppLayerTxData is set up from the start. Any type of
processing (logging, detection) will lead to setting up the user
data later on anyway.
Remove other places where it was added.