DCERPC REQUEST/RESPONSE records can have their stub data fragmented
across multiple PDUs. The flags that convey the state of the current PDU
are as follows:
PFC_FIRST_FRAG: first fragment of (fragmented) data
PFC_LAST_FRAG: last fragment of (fragmented) data
PFC_FIRST_FRAG && PFC_LAST_FRAG: no fragmentation
Consider a REQUEST/RESPONSE complete only when the PFC_LAST_FRAG flag is
found to be set.
Ticket: 9240
The raw HTTP header buffer had to be registered one state past the
header state. The reason is that htp reports progress as soon as a block
starts: request_progress reaches HEADERS right after the request line
and TRAILER at the zero-length chunk, so a detection round at HEADERS
can run while the block is still arriving. For a byte buffer that shows
- negated content or a within: relative match would decide on a
truncated block. HEADERS + 1 avoided it by accident of numbering, BODY
being what htp reaches once the block is done.
Make completion an event instead of inferring it from numbering. htp's
REQUEST_HEADERS and RESPONSE_HEADERS hooks only ran in test builds,
which is why production had nothing to key on; they now run always and
are bound through the C API. Suricata marks per transaction that a block
ended, from those hooks and from the trailer hooks that already ran, and
the four HTTP/1 block buffers - http.header.raw, http.trailer,
http.trailer.raw and http.trailer_names - report no data until then.
Each registers at the state where its content completes, and no state
offset remains in the tree.
A block that ends because the stream closes needs no special case: htp's
closure handling advances progress past the block state and runs the
trailer hook, so the gate opens on its own and the partial content is
inspected as it is today. Suricata-Verify tests/http1-raw-header-
truncated covers both directions.
htp progress values, and the transaction lifecycle built on them, are
left alone: delaying those changes when firewall decisions become
definitive and fails the ruletype-firewall state hook tests.
Ticket: #8256.
The HEADERS block that closes a stream can be a trailer block, but
HTTP/2 kept it in the same header list as the opening headers, so
trailer content could be matched by the header keywords. HTTP/2 also had
no trailer state, so header fields that are only reachable through a
header state hook, and any rule hooked at the trailer state, matched
nothing: no rule could see a trailer at all.
The parser marks the frames instead of storing them elsewhere. HPACK
keeps one dynamic table per connection, updated by every header block in
either direction, so the fields have to reach that table; the size
update matcher walks the frames themselves, and continues to see
trailers, while every other HTTP/2 header keyword and getter now goes
through the header-section-only accessor. Three trailer accessors mirror
them over the marked frames.
http.trailer.raw has no wire format to reproduce for HTTP/2, so it is
built the way http.header.raw is.
HTTP/2 EVE output follows the buffers: a trailer field is no longer
logged as a request header, like HTTP/1. A dedicated log key for
trailers is a follow-up for both protocols.
Ticket: #8256.
An HTTP/2 stream puts its trailers in a HEADERS block after the body.
There was no state for it, so detection had nothing to hook: the block
was merged into the header buffers, where the header state had already
been declared complete. gRPC streams, which always end with such a
block, could not be matched at all.
Add HTTP2ProgTrailers between the data and the closed state, and mark
the frames of a trailer block. Marking keys on the header section having
completed in that direction, not on the progress value and not on body
data: RFC 9113 permits a trailer section behind a message that sent no
DATA frames at all. A second block that only follows an informational
response is not a trailer, so a to-client block completes the header
section only with a non-1xx :status, and a CONTINUATION takes the
classification of the block it continues, not the state the direction
reached since.
A state inserted into a shipped protocol defaults to drop:flow in
firewall mode, which would block every stream that has trailers the
moment this builds. So a parser can now describe its sub-states fully -
completion state and the built-in policy of each state - instead of
AppLayerParserGetSubStateCompletion guessing from hard coded numbers.
The trailer states register accept:hook, so behaviour only changes for
deployments that opt in; the generic drop:flow still applies to the
states the protocol has always had.
Ticket: #8256.
Trailer fields of a chunked message are parsed by the same code as
header fields and were appended to the header table, where a name
already used in the header section is combined into the existing value.
So a header keyword silently gained trailer data, and a chunked request
ending in "User-Agent: x" reported its user agent as "Mozilla, x".
Store them in a separate table per direction instead. Repeated names are
still combined and the field count limit still applies across both
sections, but the Content-Length bookkeeping is skipped: trailer fields
take no part in framing.
The C API gains htp_tx_request_trailers() and
htp_tx_response_trailers(); every existing getter keeps returning the
header section only.
A few HTTP unit tests assert the old guarantee that trailer content is
reachable through http_header, so they fail from here until the commit
that adds the trailer buffers converts them. Every commit in the series
still builds.
Ticket: #8256.
SCFTPFreeResponseLine gated the release of the code buffer on a null
test of the response pointer, so the code buffer had no check of its
own. Neither check could fire: parse_response_line sets both pointers
from Box::into_raw on a boxed slice, which never returns null, and C
code obtains an FTPResponseLine only from SCFTPParseResponseLine. The
comment on the code field claimed the pointer may be null; a line with
no status code gets an empty buffer with code_length 0 instead.
Remove both null checks, note why they are unnecessary, and correct the
field comment.
Ticket: 8803
parse_response_line allocates the code buffer with Box::into_raw on a
boxed slice, but SCFTPFreeResponseLine released it by rebuilding a
Vec<u8> with capacity code_length. Vec::from_raw_parts accepts any
global-allocator pointer whose size and alignment match, and both paths
compute the same layout for a byte slice, so the old free was sound. It
differed only from how the response buffer is released.
Release the code buffer through Box::from_raw on a slice pointer, so
both buffers are freed the same way they were allocated.
Ticket: 8803
In case a keyword should work in firewall mode, with firewall rules
only. The engine errors out if threat detection rules use the given keyword.
Part of
Ticket #9041
An empty SETTINGS frame (an ACK, or a SETTINGS without entries) opened
the settings array and closed it without items, producing a record the
eve schema rejects (settings has minItems 1). Skip the frame instead; a
tx whose only content is an empty SETTINGS frame is then not logged at
all, matching the empty headers handling.
Exposed by the flow-end flush of IDS-dropped flows: the dropped flow in
the drop-protocol-change test carries such a frame.
Victor Julien
Shivani Bhardwaj
Philippe Antoine
Jason Ish
Victor Julien
Jeff Lucovsky
Juliana Fajardini