Windows has no SIGHUP and no unix socket, so Suricata could not be told to
reopen its log files after external log rotation. When running as a service,
this can now be requested with the user-defined control code 129, for
example:
sc.exe control Suricata 129
The service control handler only records the request; the main loop then
handles it the same as SIGHUP.
Ticket: 7757
On Windows, fopen() does not open files with FILE_SHARE_DELETE, so a log file
can not be renamed or deleted while Suricata has it open. This prevents
external tools from rotating log files.
Add SCWin32Fopen(), an fopen() replacement for the "a" and "w" modes that
opens the file with CreateFile() and FILE_SHARE_DELETE, and use it for log
files, including suricata.log.
Ticket: 7757
Windows has no SIGUSR2 and no unix socket, so rules could not be reloaded
without restarting Suricata. When running as a service, a rule reload can now
be requested with the user-defined control code 128, for example:
sc.exe control Suricata 128
The service control handler only records the request; the main loop then
handles it the same as SIGUSR2.
Ticket: 7756
By default, a Windows service allows interactive users and service accounts
to send it user-defined control codes (128-255). When installing the Suricata
service, set a DACL that is the default DACL without this right for those
users, so only SYSTEM and Administrators can send them.
This prepares for using user-defined control codes to control Suricata at
runtime, such as reloading rules.
Ticket: 7756
So far the SSL parser returned an error if there were less than 4 bytes
available for the handshake header. This meant that a fragmented
handshake header could result in the SSL parser erroring out and thus
disabling any further inspection for the protocol.
The parser follows a multi-record parsing scheme which means that the
fragment of the header may exist in the next record. Use internal
buffering to assess and process the records just like the handshake
buffer.
Ticket: 9167
Ticket: 8684
The behavior of the rule
alert ip any any -> any any (pcre:"/.*mc|\H+$/"; sid:4;)
depended upon the presence of another rule such as
alert ip any any -> any any (content:"*mcH"; sid:5;)
With the sid 5 content rule, the code went through PrefilterPktStream
which updated raw_stream_progress, which ended up updating raw_progress_rel
which was used on later packets to know where to start.
Without the sid 5 content rule, there was no update of these progresses.
It led to quadratic complexity, as for each new packet, we ran
the pcre engine on the whole tcp stream from the beginning, with
the newly added payload data.
Fix is to process sid 4 pcre rule the same way whether there is
or there is not a sid 5 content rule.
So, in case we do not have a mpm store for a signature group head,
we register a dummy stream prefilter, whose goal will just be
to update raw_stream_progress
DCERPC REQUEST/RESPONSE records can have their stub data fragmented
across multiple PDUs. The flags that convey the state of the current PDU
are as follows:
PFC_FIRST_FRAG: first fragment of (fragmented) data
PFC_LAST_FRAG: last fragment of (fragmented) data
PFC_FIRST_FRAG && PFC_LAST_FRAG: no fragmentation
Consider a REQUEST/RESPONSE complete only when the PFC_LAST_FRAG flag is
found to be set.
Ticket: 9240
Serialize service arguments with Windows quoting and backslash escaping
for both installation and parameter updates. Reject truncated executable
paths and command lines before accessing the Service Control Manager.
Keep the existing service-option prefix matching unchanged. Limit the
fix to command-line construction, without changing the command-line
parser, service function interfaces or SCInstance layout.
Ticket: 9241
With rotate-interval set to minute, hour or day, rotations were not
aligned to the local time boundaries. The first rotation was computed
by SCGetSecondsUntil(), which overshoots the next hour by one minute
(hh:01:00) and the next day by one hour and one minute (01:01:00).
Every following rotation was re-armed with a fixed interval
(e.g. 86400 s) from the time of the rotation.
Local days are not always 86400 s long, so the day rotation also moved
by the DST shift on every DST transition and stayed there until the
next one. A file named after a date then held events of the next day.
Instead, this commit rotates logs when the local minute, hour or date
changes. The daily rotation now happens at local midnight.
On SIGHUP, the period is recorded before the file is reopened, so a boundary
(the next minute/hour/day) passing during the reopen leads to an extra
reopen instead of a missed rotation.
Ticket: #9139
In continuous pcap-file directory mode, a file that makes InitPcapFile()
fail (malformed content, not a pcap, unsupported link type, truncated
header) was skipped without folding its mtime into last_time_seen.
This caused the file(s) to be reinspected on every follow-up poll until
a newer valid file was processed.
Ticket: 9129
The passed poll-interval should determine how many seconds to wait
in-between polls. The previous code passed the value through localtime
function which understood the input integer as an epoch time and
converted it to time structure, of which only seconds were kept.
This kind of worked for sub-60 values but wrapped for values 60+ to
the usual seconds range 0-59. Additionally, setting poll-interval to
60 made it a busy loop with no sleep between polls.
With intervals of 60+ seconds now honored, a single sleep blocked the
shutdown (or a unix socket pcap-interrupt) until the interval elapsed,
and the engine was killed after the 60 second receive thread shutdown
timeout. The wait is now done in one second steps, checking the run
status in-between.
Ticket: 9129
Change the byte order of vlan TCI when extracting it
from vlan_hdr in xdp_filter.c. This fixes xdp bypass for vlan tagged
packets, as without the fix flows that are supposed to be bypassed by
the xdp filter are not bypassed.
Ticket: 8699
A rule whose relative pcre follows a keyword that moves the detection
pointer is refused at load time with "pcre with /R (relative) needs
preceding match in the same buffer":
alert tcp any any -> any any (msg:"x"; byte_jump:4,0; pcre:"/^x/R"; ...)
The search for the preceding match only asked for content and pcre
sigmatches, so a rule that anchors a relative pcre on a byte_jump had
nothing to find even though byte_jump leaves the detection pointer at
the jump target.
Add the keywords that update det_ctx->buffer_offset to the search:
byte_jump, byte_extract and byte_math. byte_test and isdataat only read
that pointer, so they stay rejected; a relative pcre after them alone
would silently match from the start of the buffer.
Ticket: #7987.
A packet that netmap delivers across several ring slots was handed to the
rest of the code as the first slot buffer together with the summed length
of all the slots. When those slot buffers were not next to each other in
memory this read past the first buffer, which could crash the process or
leak bytes from unrelated buffers into inspection, pcap output and IPS or
tap forwarding.
Multi-slot packets are now copied slot by slot into a per-thread buffer so
the caller always gets one addressable frame, and such packets are always
copied into the Packet instead of being referenced zero-copy. Single-slot
packets keep the zero-copy path. The walk over the fragment slots is bounded
by the slots we own and the total length is capped, so a broken or over-long
chain is dropped as one unit and the ring keeps moving forward.
Adds unit tests that build a fake ring and cover single-slot, adjacent,
non-adjacent, ring-wrap, empty-fragment, over-size and never-ending chains.
Ticket: #8777
NetmapReadPackets walked the slots of a single packet inline while also
driving the per ring iteration. Move the slot walk into
NetmapReadOnePacket so the ring loop only picks the next packet and the
slot handling can be changed on its own.
No functional change.
Ticket: #8777
Header functions in the Lua http library read the header tables and the raw header
buffer. Once trailer fields moved out of those, a script could no longer see them at
all, and nothing replaced the access: the only way to act on a trailer was a rule.
Add the trailer twins over the tables and blocks the http.trailer buffers use:
tx:request_trailer(), tx:request_trailers(), tx:request_trailers_raw() and the response
versions. Name lookup is case insensitive and reports a miss as an error, like the
header functions. Repeated fields need no merging here: the parser folds them while
storing, so a name has one entry.
HTTP/2 is out of reach of these functions: the Lua http library is driven by htp, so it
has no HTTP/2 transaction to read.
Ticket: #8256.
The raw HTTP header buffer had to be registered one state past the
header state. The reason is that htp reports progress as soon as a block
starts: request_progress reaches HEADERS right after the request line
and TRAILER at the zero-length chunk, so a detection round at HEADERS
can run while the block is still arriving. For a byte buffer that shows
- negated content or a within: relative match would decide on a
truncated block. HEADERS + 1 avoided it by accident of numbering, BODY
being what htp reaches once the block is done.
Make completion an event instead of inferring it from numbering. htp's
REQUEST_HEADERS and RESPONSE_HEADERS hooks only ran in test builds,
which is why production had nothing to key on; they now run always and
are bound through the C API. Suricata marks per transaction that a block
ended, from those hooks and from the trailer hooks that already ran, and
the four HTTP/1 block buffers - http.header.raw, http.trailer,
http.trailer.raw and http.trailer_names - report no data until then.
Each registers at the state where its content completes, and no state
offset remains in the tree.
A block that ends because the stream closes needs no special case: htp's
closure handling advances progress past the block state and runs the
trailer hook, so the gate opens on its own and the partial content is
inspected as it is today. Suricata-Verify tests/http1-raw-header-
truncated covers both directions.
htp progress values, and the transaction lifecycle built on them, are
left alone: delaying those changes when firewall decisions become
definitive and fails the ruletype-firewall state hook tests.
Ticket: #8256.
The HEADERS block that closes a stream can be a trailer block, but
HTTP/2 kept it in the same header list as the opening headers, so
trailer content could be matched by the header keywords. HTTP/2 also had
no trailer state, so header fields that are only reachable through a
header state hook, and any rule hooked at the trailer state, matched
nothing: no rule could see a trailer at all.
The parser marks the frames instead of storing them elsewhere. HPACK
keeps one dynamic table per connection, updated by every header block in
either direction, so the fields have to reach that table; the size
update matcher walks the frames themselves, and continues to see
trailers, while every other HTTP/2 header keyword and getter now goes
through the header-section-only accessor. Three trailer accessors mirror
them over the marked frames.
http.trailer.raw has no wire format to reproduce for HTTP/2, so it is
built the way http.header.raw is.
HTTP/2 EVE output follows the buffers: a trailer field is no longer
logged as a request header, like HTTP/1. A dedicated log key for
trailers is a follow-up for both protocols.
Ticket: #8256.
An HTTP/2 stream puts its trailers in a HEADERS block after the body.
There was no state for it, so detection had nothing to hook: the block
was merged into the header buffers, where the header state had already
been declared complete. gRPC streams, which always end with such a
block, could not be matched at all.
Add HTTP2ProgTrailers between the data and the closed state, and mark
the frames of a trailer block. Marking keys on the header section having
completed in that direction, not on the progress value and not on body
data: RFC 9113 permits a trailer section behind a message that sent no
DATA frames at all. A second block that only follows an informational
response is not a trailer, so a to-client block completes the header
section only with a non-1xx :status, and a CONTINUATION takes the
classification of the block it continues, not the state the direction
reached since.
A state inserted into a shipped protocol defaults to drop:flow in
firewall mode, which would block every stream that has trailers the
moment this builds. So a parser can now describe its sub-states fully -
completion state and the built-in policy of each state - instead of
AppLayerParserGetSubStateCompletion guessing from hard coded numbers.
The trailer states register accept:hook, so behaviour only changes for
deployments that opt in; the generic drop:flow still applies to the
states the protocol has always had.
Ticket: #8256.
And say on the header buffers that the trailer section is not part of
them. That is a change in behaviour for chunked traffic carrying
trailers: a rule that happened to match trailer content through
http.header, http.header.raw, http.header_names or a field keyword now
needs the matching trailer buffer.
Ticket: #8256.
Trailer fields were merged into the header fields by the parser, so
http.header, http.header.raw, http.header_names and every field keyword
could match trailer data, and whether they did was a matter of packet
timing: the header buffers are complete at the headers state, while a
trailer arrives after the body.
Add three buffers for the trailer section, registered at the state that
section completes in:
http.trailer trailer fields, normalized as "Name: value"
http.trailer_names trailer field names
http.trailer.raw the trailer section as seen on the wire
Like the other HTTP/1 buffers they become final one state later, so
inspection rounds while a trailer is still arriving get revisited rather
than retiring rules early. That also removes the need for the trailer
legs of the header prefilters, which existed only to revisit a buffer
that could still grow. With their last reader gone, the
request_has_trailers and response_has_trailers flags and the htp
callbacks setting them are dropped as well; the revisit trigger they
also provided is set by the trailer data callbacks.
http.header.raw keeps its engines at one state past the headers, unlike
its siblings: its data comes from the raw header data hook, which htp
finalizes as the parser leaves the header state, so a round at the
headers state can still see a short block. HTTP/2 keeps putting trailer
blocks in the same list as regular headers and is handled separately.
Ticket: #8256.
The trailer data hooks were pointed at the header data callbacks, so the
bytes of a chunked message's trailer section were appended to the same
buffer the header block goes to. http.header.raw therefore matched
trailer content, which rules written against the header section cannot
expect and cannot avoid.
Accumulate the trailer bytes per direction in a buffer of their own
instead. Nothing reads them yet.
Ticket: #8256.
Trailer fields of a chunked message are parsed by the same code as
header fields and were appended to the header table, where a name
already used in the header section is combined into the existing value.
So a header keyword silently gained trailer data, and a chunked request
ending in "User-Agent: x" reported its user agent as "Mozilla, x".
Store them in a separate table per direction instead. Repeated names are
still combined and the field count limit still applies across both
sections, but the Content-Length bookkeeping is skipped: trailer fields
take no part in framing.
The C API gains htp_tx_request_trailers() and
htp_tx_response_trailers(); every existing getter keeps returning the
header section only.
A few HTTP unit tests assert the old guarantee that trailer content is
reachable through http_header, so they fail from here until the commit
that adds the trailer buffers converts them. Every commit in the series
still builds.
Ticket: #8256.
A rule with an app-layer state hook required every buffer engine to sit
at exactly the hook state, which makes sensible combinations impossible
to load: a rule hooked at the trailer state cannot use the header
buffer, although that data is complete well before the trailer.
Relax the check to an upper bound on the buffer state. Engines stay at
the state their buffer registers: an engine runs as soon as the tx
reaches its own state, so the hook bounds which buffers a rule may use,
it does not delay evaluation. A rule hooked at the trailer state on a
header buffer is decided where the header data is complete, which is
also what the prefilter leg queueing its fast pattern relies on. Raising
an engine to the hook state instead moves matching off that leg, so the
buffer is never inspected.
Buffers completing after the hook stay rejected: their data falls
outside the state the hook opens, and for a firewall rule that is past
the point its policy lets the flow reach.
Protocol agnostic engines (app-layer-event, app-layer-state) register at
progress 0 in every protocol, so an upper bound would admit them under
any hook and the hook would stop gating them. They keep the exact state
requirement they had before.
Rules that loaded before are unaffected: for them the two states were
already equal.
A buffer that keeps growing after the hooked state gets the rule
revisited at those later states, so such a rule can alert more than once
per transaction. Documented with the hooks.
Ticket: #8256.
Victor Julien
Jason Ish
Shivani Bhardwaj
Philippe Antoine
Lukas Sismis
Philippe Antoine
Adam Kiripolsky
Victor Julien
Samaresh Kumar Singh