A flow that hit an unrecoverable ssh parse error (invalid banner
or invalid record) was never visible in the eve ssh output: the
log condition only admitted a successful handshake (both
directions at kex or session), and the tx logger is one-shot, so
the successful-handshake log fired before a later failure could
be reported and the failure was then silent - the reason was
only visible through the app-layer events.
The log condition is now failure-only: the failure latches, so
the object is emitted exactly once - at the failure, carrying
the error field (invalid_banner / invalid_record) and the state
field (the state the failure occurred in - banner, kex, or
session if a later record fails after NewKeys) - and successful
flows are logged at the flow-end flush, where the engine logs
unconditionally. Logging on the success condition mid-flow would
consume the one-shot tx log and drop a later failure, so it is
not done. The state field is emitted only alongside the error.
The object may be error + state only, when no banner was
parsed. A unit test pins the condition for a failed, a
banner-only and a both-at-kex flow (the failure admits the log
on the failing delivery); the user guide and eve reference doc
commits that follow carry the field docs.
That flow-end flush is skipped for flows that end without a
further app-layer update, though: in IPS/firewall mode a drop
disables the app layer (StreamTcpCheckFlowDrops ->
StreamTcpDisableAppLayer), after which no app-layer update
reaches the tx loggers, and a bypassed flow gets no flow-end
packet either. A successful ssh record - e.g. one that only saw
the banners - was therefore never emitted for those flows. The
hook is the app-layer end: on the transition to app-layer EOF,
StreamTcpDisableAppLayer raises FLOW_APP_LAYER_FLUSH_PENDING,
and FlowUpdateState raises it on the transition to a bypassed
state; OutputTxLog runs the tx loggers once for the next packet,
also when that packet itself has no app-layer update and is not
marked established (the disabling packet returns from the
stream engine before that), and flow-worker clears the flag when
it frees the app layer of a bypassed flow. The flush is
deliberately not a flow-end flush: when the flush is the only
reason to run the loggers, eof is not forced, so each logger's
condition decides and no other parser starts logging records it
suppressed; a disabling packet that does carry an app-layer
update keeps the existing EOF behaviour. SSHTxLogCondition
admits an IPS drop or a bypass, so the pending record is emitted
there; an IDS drop does not end the app layer, so the condition
stays failure-only - a later failure still wins, and the success
record is emitted by the flow-end flush, which the flow timeout
path now also runs for IDS-dropped flows. The unrelated
schema-sort ordering of the new state keys is fixed up as well.
Ticket: #9046.
Add accessor functions to allow NDPI and our plugin examples to access
flow fields in an opaque manner.
Provides accessors for:
- is ipv4
- is ipv6
- ip protocol
- to server packet count
- to destination packet count
Ticket: #8632
Ticket: 8629
When we are in async-oneside mode, we see only one direction
of the traffic, and should not wait for the other direction
before cleaning up a transaction.
For an easier review process, this is a two-step change process,
in which pcap_cnt is first accessed by functions-to-be, implemented
as simple macros.
In the follow-up commit, the actual refactor is implemented with the new
function. The old macros are deleted.
Ticket: 7835
When FlowSwap() reverses the direction of a flow, the MAC address sets
stored in the flow also need to be swapped to maintain consistency with
the new direction. Previously, MAC addresses were not swapped along with
other flow properties like packet/byte counters.
Ticket #8172
The tracking for elephant flow is done per direction however, the flag
was set on the flow whenever either of the directions crossed the
rate-limit defined in the settings. Given that the tracking was already
split, it makes sense to split the detection tracking per direction as
well and allow user to have a better control via rule language.
Track per flow thread id for UDP and other non-TCP protocols. This
improves the timeout handling as the per thread timestamp is used in
offline mode.
Fixes: ada2bfe009 ("flow/worker: improve flow timeout time accuracy")
Fixes: ef396f7509 ("flow/manager: in offline mode, use owning threads time")
Bug #7687.
New facility to allow a user to specify where to hook a rule
into the engine. This patch adds this for packets, adding two
hooks:
- `all`: to let a rule be evaluated by all rules
- `flow_start`: to have a rule be evaluated only for the first
packet in both directions
Implemented by adding a hook flags field in the packet.
In order to track flow rate and thus determine a course of action or
categorize it as elephant flow, track a flow's byte count per direction
in a ring buffer for a given time interval.
The implementation is simple and keeps overwriting the buffer and
updating the final sum. The sum of all the elements in the ring buffer
at any point in time should reflect the number of bytes for the
respective flow in the last of a given configured interval.
e.g. if the definition says that the flows must be tracked by a rate of
100k bytes in 10 seconds, the ring buffer at any point in time should
carry the total number of bytes seen by the respective flow in the last
10 seconds.
So far, the implementation only supports reading the flow rate
definition from suricata.yaml and using it to track the flows.
This solution adds up a space complexity to the existing Flow struct.
However, the added space complexity should only take effect if the
feature is in use. Since this buffer extends the Flow struct, it does
not impact the usual business logic or complexity of the code.
This implementation is currently limited to defining the time interval
of flow rate in seconds only. However, the number of seconds defined are
directly proportional to the aforementioned added space complexity as
that's the size of the ring buffer.
Feature 5647
Instead of ip-only specific flags, reuse the FLOW_PKT_TOSERVER_FIRST and
FLOW_PKT_TOCLIENT_FIRST flags.
Fixes false positives on one sided streams that trigger a opposing flow
timeout packet at the flow's end. That pseudo packet would trigger a
match even though it shouldn't.
Ticket: #7521.
Adds user registerable callbacks for flow initialization, flow
update and flow finish.
Some plugins, such as other DPI libraries like nDPI need a way to hook
into these flow lifecycle events.
Ticket: #7319
Ticket: #7320
Store IPv6 decoder vars in a new Packet::l3 section in the packet.
Use inline functions instead of the often multi-layer macro's for
various IPv6 header getters.
Ticket: #6938.
Store IPv4 decoder vars in a new Packet::l3 section in the packet.
Use inline functions instead of the often multi-layer macro's for
various IPv4 header getters.
Ticket: #6938.
Replace it with inline functions. Adds inline functions to
wrap PKT_IS_IPV4/PKT_IS_IPV6.
This is in preparation of removing the macro's, and cleaning up the
header pointers.
Ticket: #5517.
Remove the global "run_mode" var as it was a duplicate of the runmode on
the "instance" struct. For direct access outside of suricata.c, use the
getter function.
Also expose a setter function for unit tests that need to change it.
We will register stats counters for all policies, even though for now
Suri only uses one possible configuration policy at a time. The idea is
that this could change in the near future, so we want to have this
ready.
Task #5816
Philippe Antoine
Victor Julien
Jason Ish
Juliana Fajardini
Lukas Sismis
Giuseppe Longo
Shivani Bhardwaj
Amir Boussejra
Jeff Lucovsky