A flow that hit an unrecoverable ssh parse error (invalid banner
or invalid record) was never visible in the eve ssh output: the
log condition only admitted a successful handshake (both
directions at kex or session), and the tx logger is one-shot, so
the successful-handshake log fired before a later failure could
be reported and the failure was then silent - the reason was
only visible through the app-layer events.
The log condition is now failure-only: the failure latches, so
the object is emitted exactly once - at the failure, carrying
the error field (invalid_banner / invalid_record) and the state
field (the state the failure occurred in - banner, kex, or
session if a later record fails after NewKeys) - and successful
flows are logged at the flow-end flush, where the engine logs
unconditionally. Logging on the success condition mid-flow would
consume the one-shot tx log and drop a later failure, so it is
not done. The state field is emitted only alongside the error.
The object may be error + state only, when no banner was
parsed. A unit test pins the condition for a failed, a
banner-only and a both-at-kex flow (the failure admits the log
on the failing delivery); the user guide and eve reference doc
commits that follow carry the field docs.
That flow-end flush is skipped for flows that end without a
further app-layer update, though: in IPS/firewall mode a drop
disables the app layer (StreamTcpCheckFlowDrops ->
StreamTcpDisableAppLayer), after which no app-layer update
reaches the tx loggers, and a bypassed flow gets no flow-end
packet either. A successful ssh record - e.g. one that only saw
the banners - was therefore never emitted for those flows. The
hook is the app-layer end: on the transition to app-layer EOF,
StreamTcpDisableAppLayer raises FLOW_APP_LAYER_FLUSH_PENDING,
and FlowUpdateState raises it on the transition to a bypassed
state; OutputTxLog runs the tx loggers once for the next packet,
also when that packet itself has no app-layer update and is not
marked established (the disabling packet returns from the
stream engine before that), and flow-worker clears the flag when
it frees the app layer of a bypassed flow. The flush is
deliberately not a flow-end flush: when the flush is the only
reason to run the loggers, eof is not forced, so each logger's
condition decides and no other parser starts logging records it
suppressed; a disabling packet that does carry an app-layer
update keeps the existing EOF behaviour. SSHTxLogCondition
admits an IPS drop or a bypass, so the pending record is emitted
there; an IDS drop does not end the app layer, so the condition
stays failure-only - a later failure still wins, and the success
record is emitted by the flow-end flush, which the flow timeout
path now also runs for IDS-dropped flows. The unrelated
schema-sort ordering of the new state keys is fixed up as well.
Ticket: #9046.
parse_row_description and parse_consolidated_data_row read a field_count
from the message body and use that to iterate a slice, without checking
whether the count actually fits the body.
A backend-controlled attack could lead to fewer fields logged, or
stalling of the parser to-client direction for the duration of the flow.
A malformed_response event is set for pgsql, if either of those
message's inner parsers has a field_count that can't be reconciled with
the body slice.
A new pgsql.response log field is added to indicate this, too:
pgsql.response.row_description_malformed, a boolean. This wasn't added
to the consolidated data row message, per its summarization nature.
The row_description unit test buffer declared a length one byte longer
than its three fields occupy. The old code silently discarded that
surplus byte; the new one reports it, so the buffer was corrected.
Ticket: #8885
The per-thread threshold decision cache (limit/both thresholds tracked
by_src/by_dst, IPv4) grew without bound: every distinct tracked address
allocated an item plus a hash-table bucket, while housekeeping reclaims
at most two entries per whole second.
Address this by:
Cap the cache at detect.thresholds.cache.max-entries entries per
thread (default 256; valid range 256-1048576, out-of-range values
fail startup); on overflow evict the entry with the earliest expiry
before inserting
Account cache memory in the new detect.thresholds.cache.memuse
counter: each entry (item plus bucket node) is charged on insert and
released on eviction, expiry and on thread teardown, and the fixed
per-thread charge (thread context, hash table struct and the eagerly
allocated bucket pointer array) is charged at cache init and released
symmetrically on teardown, so the gauge reflects the cache memory in
use and does not accumulate stale charges when threads are recreated,
e.g. unix-socket PCAP file jobs
Size the cache hash table to the entry cap (next power of two, minimum 256)
at thread init so lookups stay O(1) up to the cap.
Fail startup when the threshold table cannot be initialized, instead
of continuing with a NULL table.
Ticket: #8956.
Due to a lack of tests with varied ErrorResponse messages, it wasn't
noticed before that these were missing.
Partially covers
Ticket: #8903
Related to
Bug: #8886
CopyIn and CopyOut message parsers read the length field, but did not
use that information to parse the rest of the message, instead, parsing
as many columns as indicated in the columns field, without ever
checking against the informed length. This opened the door for
attackers to fabricate messages that were never sent by the server,
as well as to obfuscate following valid backend messages, in case the
columns value was bigger than the actual message (up to ~128KiB).
Those parsers also had a message length parsing check off-by-one, fixed
in this patch.
When a malformed CopyIn or CopyOut response is seen, it is logged as
previously, with a new field added to indicate whether it is malformed
or not.
Ticket: #8863
With this patch, the logged uuids will also have corresponding
stringified service names. The requests that will have an opnum will
have a corresponding stringified procedure name.
Feature 2727
Feature 8523
Extend the SCTP decoder to parse chunk headers after the 12-byte common
header. Each chunk is validated for minimum header size and length
consistency per RFC 4960 sec 3.2.
Add SCTPChunkHdr and SCTPVars structs to track per-packet chunk metadata
Add five new decoder events for protocol violations:
- SCTP_CHUNK_TOO_SMALL: insufficient data for a chunk header
- SCTP_CHUNK_LEN_INVALID: chunk length < 4 or exceeds packet
- SCTP_INIT_CHUNK_NOT_ALONE: INIT/INIT_ACK bundled (RFC 4960 sec 6.10)
- SCTP_INIT_WITH_NON_ZERO_VTAG: INIT with vtag != 0 (RFC 4960 sec 8.5.1)
- SCTP_DATA_WITH_ZERO_VTAG: DATA chunk with vtag == 0
Ticket #4251
This adds an LLMNR protocol logger that reuses existing DNS functions,
following the same approach as the mDNS logger:
- No grouped logging
- Rdata is logged in a field that is named after the rdata type
- Types are logged in lower case
- Flags are logged as an array
Ticket #8366
Support `alert` as a secondary action in app-layer firewall policies.
To implement this a Signature object is created per policy that uses
alert, and this is stored in a hash table. When the policy is applied
the signature is looked up and used in the PacketAlert.
Ticket: #8566.
Add a `firewall` stats counter aggregator for all firewall-related
stats.
De-overload "detect.alert_queue_overflow", by adding
"firewall.discarded_alerts" to account for discarded drops in
Firewall mode.
Add Debug statements for tracking corner cases where it can be
difficult to know where a drop is coming from.
Added counters:
- stats.firewall.blocked
- stats.firewall.accepted
- stats.firewall.rejected
- stats.firewall.drop_reason.default_app_policy
- stats.firewall.drop_reason.default_packet_policy
- stats.firewall.drop_reason.flow_drop
- stats.firewall.drop_reason.pre_flow_hook
- stats.firewall.drop_reason.pre_stream_hook
- stats.firewall.drop_reason.rules
- stats.firewall.discarded_alerts
Ticket #7699
When an alert is generated from firewall context, add an engine value of
"fw", otherwise "td" (for threat detect).
The engine field is only added when firewall mode is enabled.
Ticket: #8456
Store the NTP reference ID as raw network-order bytes so it can be
exposed as a sticky buffer and matched with payload keywords. The
reference ID is often a 4 character string, or an IP address and not
just an integer identifier.
Updates the log reference ID to be a string of colon separated hex
digits as this matches what tshark does.
Ticket: #8488
The tracking for elephant flow is done per direction however, the flag
was set on the flow whenever either of the directions crossed the
rate-limit defined in the settings. Given that the tracking was already
split, it makes sense to split the detection tracking per direction as
well and allow user to have a better control via rule language.
Victor Julien
William Ling
Philippe Antoine
Juliana Fajardini
Shivani Bhardwaj
Giuseppe Longo
Giuseppe Longo
Jeff Lucovsky
Jason Ish
Eric Leblond
Ofer Dagan