The error arm of a COMPOUND that is fully in hand only set a state event.
set_event marks the newest transaction, and there is none when the
record's first op is the one that fails: the record was then consumed,
nothing was logged, and nothing asked for the missing bytes. A v4 WRITE
whose count runs past the end of its record is that case - the parser
gives up at the data, so the size check in the WRITE handling, which
needs a parsed count, is never reached.
Reject what cannot be walked the way a partial record is rejected: scan
the bytes the parser was given, and log an oversized WRITE as a
write_request_too_large file transaction or another un-walkable record as
a malformed one, on a transaction of its own. What the parser skips is
not in that set: a defined op this parser does not implement keeps base's
event on the newest tx, and an RPCSEC_GSS privacy compound, whose result
is opaque by RFC 2203 on the call and the reply alike, is left unparsed
with no event at all. Either way the record is bounded by its own marker,
and a flow that carries un-parseable v4 traffic leaves a trace in EVE.
Ticket: #9265.
The record scanners unwrapped the (0, 2) combination and took every other
one as a bare compound. A control procedure or a privacy compound was
then scanned as if it were a COMPOUND: the envelope length was read as
the first tag, so the scan either stayed incomplete and buffered toward
the claim, or rejected a record that has nothing to reject.
Scan by the combination the call recorded: hand the credential pair to
gss_compound_scan_data, which leaves (0, 1) raw because rpc_gss_svc_none
wraps nothing, unwraps (0, 2), and reports a privacy (0, 3) compound as
opaque. There is nothing to scan in an encrypted compound, so the record
is skipped to its boundary and neither parsed nor scored: the request
path keeps the call in the request map for its reply, the reply path
drops it. Any other combination is one the full record path rejects as
well, so it is malformed, boundedly and without buffering. The v4
request path records the combination whatever it is, not only when it
unwraps.
Ticket: #9263.
The args of a WRITE request and of a READ reply were trusted to fit the
record that carries them. Where the claim fell below the extent the parse
covers, the file data ran past the record boundary into the next RPC,
which was then read as a record. A max-tx of 0 was accepted and evicted
every transaction on insert.
Compare the claim against that extent: for a WRITE the handle, offset,
count, stable flag and its data, for a READ the status, the optional
attributes, the count, the eof flag and its data. A record whose claim is
below the extent gets a transaction with the malformed data event and a
skip to the record boundary, as an oversized claim already does. Only the
v3 records arrive in pieces, so process_read_record now learns which
reply it is given and the check is v3's. Reject a max-tx of 0 at config
load.
Ticket: #9263.
The queue-limit work computed the record tail and the chunk promise from
the declared size, and added a count or a data length to it. A padded or
segmented record therefore left a promise armed past the end of its
record: a continuation could run on into the next RPC, an input that
carries only record slack grew the out-of-order queue and truncated a
file whose queued data fits, and the backstop walked the transactions of
a call that holds no file transaction at all.
Take the promise from the record itself, with record_tail(): what its
marker claims minus what the parser holds. Keep its two halves apart:
the stream side is the record tail, the queue side is the file data the
tracker still misses, so a continuation queues no more than its own
bytes. A pad a record carries is inside its claim, so the tail drains
it; it is neither file data nor fill. Release a promise whose
transaction is gone instead of consuming toward the claim, run the
backstop only when the call carries data and the state holds a file
transaction, and count a chunk only for input that carries file data. A
consume past the end of an input is now a debug assertion, with the
advance clamped to the input.
Ticket: #9263.
The chunk-continuation queue-limit rejection accounted its stream
consumption and skip from the count-based chunk promise
(chunk_left), but the parser accepts file_len < count and the
tracker only ever consumes file_len bytes. With count > file_len
the consume/skip ran past the record's actual tail into the
following RPC record and the stream desynced.
Consume/skip from the record's unbuffered tail plus the record's
XDR padding.
Ticket: #8791.
The chunk path (filetracker_update) enqueued per-segment data with no
queue limit. Check the limits on every enqueuing path, projected onto
actual queue growth: only OOO data at a new offset grows the queue,
and only a new OOO chunk adds a map entry. Overflow raises the queue
event on the affected tx and skips to the record boundary.
Replace the 2 MiB gap backstop with a queue-limit one (the chunk-count
cap and all-open-file scan come next), and tag the session gapped on
every gap (even one fully covered by a skip), since the backstop is
gated on that flag. Rejected and evicted tx now release their file
resources so the file logger can clean them up.
Ticket: #8791.
Replace the hardcoded 16 MiB record caps with configurable
max-{write,read}-size and observable events: an oversized WRITE
request or READ response is rejected, logged, and its payload skipped
to keep the stream in sync. 0 disables the check.
For v4 records, the issue is more complex due to the COMPOUND type:
A partial v4 COMPOUND record whose frag_len claim is huge must not be
buffered toward that claim (31-bit, attacker-controlled → OOM). The
full parser can't help: on a partial buffer it only returns Incomplete
("need more bytes"), with no information about what's coming. The scan
instead walks the buffered prefix's ops by wire extent and reads the
file op's header words alone, answering the gate's four-way question:
oversized claim detected (reject + skip to the record boundary before
any data buffers), data not yet present (buffer normally), structurall
malformed (bounded rejection), or no oversized op (fall through). It
exists because rejecting before buffering requires the claimed length,
which only a prefix walk can extract.
Ticket: #8791.
Bound the per-file-transaction OOO queue, mirroring SMB: add
max-{read,write}-queue-{size,cnt} (0 disables, default 64 MiB / 64),
raise a queue event on overflow, and skip the truncated record's
remainder from the stream.
Ticket: #8791.
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
NFS has a classic request response model where each request is mapped to
a corresponding response. Additionally, there's also file tracking and
handling done as a part of these transactions. Appropriate calls to
trigger raw stream inspection have been added on completion of each
request and response.
Task 7026
Bug 7004
Update APIs to store files in transactions instead of the per flow state.
Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.
Update all protocol implementations to support this.
Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.
Update the filestore keyword to consider a match if filestore output not
enabled.
After a file has been closed (CLOSE, COMMIT command or EOF/SYNC part of
READ/WRITE data block) mark it as such so that new file commands on that
file do not reuse the transaction.
When a file transfer is completed it will be flagged as such and not be
found anymore by the NFSState::get_file_tx_by_handle() method. This forces
a new transaction to be created.
Before, even if there were no outputs, all the arguments
were evaluated, which could turn expensive
All variables which are used only in certain build configurations
are now prefixed by underscore to avoid warnings
Add TX creation for NFS4 transactions. Start with the 'REMOVE' procedure.
Start on logging all procs. In NFS4 COMPOUND records there are multiple
procedures. One of them can be considered the 'main' procedure, with others
as supporting utility. This patch adds the first step in supporting to
track those in the TX for logging and inspection.
Victor Julien
Philippe Antoine
Shivani Bhardwaj
Jason Ish
Kristina Jefferson
Pierre Chifflier
Shivani Bhardwaj
Victor Julien
Philippe Antoine