Windows has no SIGHUP and no unix socket, so Suricata could not be told to
reopen its log files after external log rotation. When running as a service,
this can now be requested with the user-defined control code 129, for
example:
sc.exe control Suricata 129
The service control handler only records the request; the main loop then
handles it the same as SIGHUP.
Ticket: 7757
On Windows, fopen() does not open files with FILE_SHARE_DELETE, so a log file
can not be renamed or deleted while Suricata has it open. This prevents
external tools from rotating log files.
Add SCWin32Fopen(), an fopen() replacement for the "a" and "w" modes that
opens the file with CreateFile() and FILE_SHARE_DELETE, and use it for log
files, including suricata.log.
Ticket: 7757
Windows has no SIGUSR2 and no unix socket, so rules could not be reloaded
without restarting Suricata. When running as a service, a rule reload can now
be requested with the user-defined control code 128, for example:
sc.exe control Suricata 128
The service control handler only records the request; the main loop then
handles it the same as SIGUSR2.
Ticket: 7756
By default, a Windows service allows interactive users and service accounts
to send it user-defined control codes (128-255). When installing the Suricata
service, set a DACL that is the default DACL without this right for those
users, so only SYSTEM and Administrators can send them.
This prepares for using user-defined control codes to control Suricata at
runtime, such as reloading rules.
Ticket: 7756
So far the SSL parser returned an error if there were less than 4 bytes
available for the handshake header. This meant that a fragmented
handshake header could result in the SSL parser erroring out and thus
disabling any further inspection for the protocol.
The parser follows a multi-record parsing scheme which means that the
fragment of the header may exist in the next record. Use internal
buffering to assess and process the records just like the handshake
buffer.
Ticket: 9167
Ticket: 8684
The behavior of the rule
alert ip any any -> any any (pcre:"/.*mc|\H+$/"; sid:4;)
depended upon the presence of another rule such as
alert ip any any -> any any (content:"*mcH"; sid:5;)
With the sid 5 content rule, the code went through PrefilterPktStream
which updated raw_stream_progress, which ended up updating raw_progress_rel
which was used on later packets to know where to start.
Without the sid 5 content rule, there was no update of these progresses.
It led to quadratic complexity, as for each new packet, we ran
the pcre engine on the whole tcp stream from the beginning, with
the newly added payload data.
Fix is to process sid 4 pcre rule the same way whether there is
or there is not a sid 5 content rule.
So, in case we do not have a mpm store for a signature group head,
we register a dummy stream prefilter, whose goal will just be
to update raw_stream_progress
DCERPC REQUEST/RESPONSE records can have their stub data fragmented
across multiple PDUs. The flags that convey the state of the current PDU
are as follows:
PFC_FIRST_FRAG: first fragment of (fragmented) data
PFC_LAST_FRAG: last fragment of (fragmented) data
PFC_FIRST_FRAG && PFC_LAST_FRAG: no fragmentation
Consider a REQUEST/RESPONSE complete only when the PFC_LAST_FRAG flag is
found to be set.
Ticket: 9240
Serialize service arguments with Windows quoting and backslash escaping
for both installation and parameter updates. Reject truncated executable
paths and command lines before accessing the Service Control Manager.
Keep the existing service-option prefix matching unchanged. Limit the
fix to command-line construction, without changing the command-line
parser, service function interfaces or SCInstance layout.
Ticket: 9241
With rotate-interval set to minute, hour or day, rotations were not
aligned to the local time boundaries. The first rotation was computed
by SCGetSecondsUntil(), which overshoots the next hour by one minute
(hh:01:00) and the next day by one hour and one minute (01:01:00).
Every following rotation was re-armed with a fixed interval
(e.g. 86400 s) from the time of the rotation.
Local days are not always 86400 s long, so the day rotation also moved
by the DST shift on every DST transition and stayed there until the
next one. A file named after a date then held events of the next day.
Instead, this commit rotates logs when the local minute, hour or date
changes. The daily rotation now happens at local midnight.
On SIGHUP, the period is recorded before the file is reopened, so a boundary
(the next minute/hour/day) passing during the reopen leads to an extra
reopen instead of a missed rotation.
Ticket: #9139
In continuous pcap-file directory mode, a file that makes InitPcapFile()
fail (malformed content, not a pcap, unsupported link type, truncated
header) was skipped without folding its mtime into last_time_seen.
This caused the file(s) to be reinspected on every follow-up poll until
a newer valid file was processed.
Ticket: 9129
The passed poll-interval should determine how many seconds to wait
in-between polls. The previous code passed the value through localtime
function which understood the input integer as an epoch time and
converted it to time structure, of which only seconds were kept.
This kind of worked for sub-60 values but wrapped for values 60+ to
the usual seconds range 0-59. Additionally, setting poll-interval to
60 made it a busy loop with no sleep between polls.
With intervals of 60+ seconds now honored, a single sleep blocked the
shutdown (or a unix socket pcap-interrupt) until the interval elapsed,
and the engine was killed after the 60 second receive thread shutdown
timeout. The wait is now done in one second steps, checking the run
status in-between.
Ticket: 9129
Change the byte order of vlan TCI when extracting it
from vlan_hdr in xdp_filter.c. This fixes xdp bypass for vlan tagged
packets, as without the fix flows that are supposed to be bypassed by
the xdp filter are not bypassed.
Ticket: 8699
A rule whose relative pcre follows a keyword that moves the detection
pointer is refused at load time with "pcre with /R (relative) needs
preceding match in the same buffer":
alert tcp any any -> any any (msg:"x"; byte_jump:4,0; pcre:"/^x/R"; ...)
The search for the preceding match only asked for content and pcre
sigmatches, so a rule that anchors a relative pcre on a byte_jump had
nothing to find even though byte_jump leaves the detection pointer at
the jump target.
Add the keywords that update det_ctx->buffer_offset to the search:
byte_jump, byte_extract and byte_math. byte_test and isdataat only read
that pointer, so they stay rejected; a relative pcre after them alone
would silently match from the start of the buffer.
Ticket: #7987.
A packet that netmap delivers across several ring slots was handed to the
rest of the code as the first slot buffer together with the summed length
of all the slots. When those slot buffers were not next to each other in
memory this read past the first buffer, which could crash the process or
leak bytes from unrelated buffers into inspection, pcap output and IPS or
tap forwarding.
Multi-slot packets are now copied slot by slot into a per-thread buffer so
the caller always gets one addressable frame, and such packets are always
copied into the Packet instead of being referenced zero-copy. Single-slot
packets keep the zero-copy path. The walk over the fragment slots is bounded
by the slots we own and the total length is capped, so a broken or over-long
chain is dropped as one unit and the ring keeps moving forward.
Adds unit tests that build a fake ring and cover single-slot, adjacent,
non-adjacent, ring-wrap, empty-fragment, over-size and never-ending chains.
Ticket: #8777
NetmapReadPackets walked the slots of a single packet inline while also
driving the per ring iteration. Move the slot walk into
NetmapReadOnePacket so the ring loop only picks the next packet and the
slot handling can be changed on its own.
No functional change.
Ticket: #8777
The design note gains the reason a rule which reads a variable is given no fast pattern, and
the http2 paragraphs describe a candidate group reaching its retirement at the trailer update.
The limitations entry about a denied body being delivered until the request closes is dropped:
measured against main on the trailer and body scenarios, the deciding packet is the one main
drops at.
The candidacy rule itself is in "firewall: fast pattern decides LTE candidacy".
Ticket: #9149.
A trailer HEADERS frame lands in the same header store as the request, so a rule
on http.host could see the authority of a trailer: a request to a denied host
that named an allowed one while closing itself was accepted, flow included. RFC
9113 8.1 forbids pseudo-header fields in trailers and RFC 9110 6.5.1 keeps
routing fields such as Host out of them, so the value is now taken from the
frames before the first DATA frame, or before the HEADERS frame that follows a
completed block.
The h2c upgrade path sends one HEADERS frame per field with no END_HEADERS, so
those frames are all header section and keep counting: cutting at the second
HEADERS frame rather than at a completed block breaks that stream, and
http2-keywords2 says so.
Ticket: #9149.
Several C loggers are registered through a cast to
EveJsonSimpleTxLogFunc, which has an incompatible signature. Clang's
function sanitizer reports this as a call through a pointer to an
incorrect function type.
Change these loggers to the generic callback signature and register them
without casts.
Ticket: 9226
An LTE rule of an app-ioctl protocol cannot carry a raw stream pattern: the packet scan of a
lower state is not a decision at this hook, so such a rule would be matched where it must not
be. Refuse it at parse time.
The window unit tests move to `flow:` predicates to keep that distinction. A `flowbits`
predicate no longer gates anything here: a rule which reads a variable gets no fast pattern
now, so it is a plain candidate in every update and the ordinary walk settles it. `flow:` is
refused by neither check and stays out of the variable list, so the packet test still proves a
packet accept at a lower state decides nothing on its own, and the policy test still proves a
default policy does not decide around a live rule. `DetectFwLtePendingLivePolicyTest01()` is
the one whose predicate has to do real work: with the liveness filter in `
DetectFwLteRuleStillLive()` forced to `return true` it is the only LTE window test that fails.
The semantics of a pending `<hook` rule live in the tx walk, so pin them
there: two `<hook` accept rules where only the second one's buffer completes,
in a later segment, plus a TD rule at the same hook. The first segment must
leave the flow undecided, the second must accept it through the pending rule,
and the TD rule must still have run once.
Ticket: #9149.
The candidate list of a "less than or equal" hook rule is decided by the fast
pattern at the rules hook, so the walk has to find the lowest rule the pattern
did not add from the window the rule is registered in: both are in id order, so
a single pass over them answers it. Test that pass, including the rules of other
groups that are in the list as well.
Ticket: #9149.
A "less than or equal" hook rule is only resolved once the transaction reaches
its hook state, so nothing decides it there: the rule ends up in the
non-prefilter list for every state from its hook upwards, which makes all of
them candidates. A ruleset with a few thousand of these rules pays that per
flow, in the thousands.
Set up the fast pattern for these rules too. What is needed to keep the walk
as it is:
- A rule the pattern does not add is not final at its hook, as the buffer may
still fill in, so the rules are registered for the states below the hook with
an upper bound. Above the bound a group is retired once its last rule there
has been inspected, which is legitimate only if no other rule of the group can
match anymore. A buffer of the hook decides that: SigValidateCheckBuffers()
requires the engines of an app hooked rule to sit at its hook, and a protocol
which declares sub-states can rewrite a buffer above the hook, an http2
trailer HEADERS frame updating the header buffers being the case in point. A
group which holds a rule with no fast pattern is inspected whole above the
bound, as it is without it: nothing can bring such a rule back.
- The rules the pattern did not add are missing from the list, while a list
with all rules in it reaches the lowest of them before any rule that takes a
higher id, and stops the walk there. Both the hook coverage and the walk take
that rule into account, so a rule that matches cannot decide the transaction
before a rule that is still in the running had its chance.
A pending rule whose hook the transaction has moved past keeps its hook covered,
which is what stops a default policy from deciding through it, and the
candidates above it are inspected as they would be without the bound. An accept
state which skips the firewall rules is consulted first, as it is for a rule in
the list.
The coverage count a pending rule borrows at walk start is kept rather than
returned when the walk steps past it: the rule is still in the running, and a
hook which reads as uncovered is exactly what lets a default policy decide. The
counts are per walk, so none of them outlives it.
The inspect engines of a LTE rule keep their order: the pass through engines
cover the states below the hook and have to run before the buffer engine at the
hook. They are not marked as an mpm engine either, as that would turn a miss
above the hook into a no match for the whole list, while the own end of the
buffer already knows when a rule is final.
While a rule is pending at the current state no default policy may decide the
transaction, the packet one included: this is not the packet to decide the flow.
A rule whose packet match reads a variable - a flowbit, xbit, flowint, flowvar, pktvar or a
lua test - is given no fast pattern at all. Its liveness cannot be settled before the rules
below it in the walk have run, since one of them may set the bit it reads, so it stays a
candidate in every update and is inspected where main would inspect it.
Removes DetectLteRevisit(), added earlier in this stack: the candidate list above the bound
already covers the buffer growth it was meant to catch, and keeping it would leave the whole
engine carrying an HTTP/2-only sub-state hook and its buffers for a decision nothing uses.
DetectTxCompleted() loses a signature argument it never read.
Ticket: #9149.
Tx engines run from a minimal tx progress upwards, with no way to stop at a
state: once such an engine fires, its rules stay candidates for every higher
state. The firewall "less than or equal" hook handling needs the opposite: a
rule that must be considered while the tx is still below its hook, but not at
the hook itself.
Add tx_max_progress to PrefilterEngine.ctx.app (skips the engine once the tx
reached it, 0 for no bound) and to PrefilterEngineList so the value survives
setup. Nothing sets it yet, so behavior is unchanged.
Ticket: #9149.
Victor Julien
Jason Ish
Shivani Bhardwaj
Philippe Antoine
Lukas Sismis
Philippe Antoine
Adam Kiripolsky
Victor Julien
Samaresh Kumar Singh