Windows has no SIGHUP and no unix socket, so Suricata could not be told to
reopen its log files after external log rotation. When running as a service,
this can now be requested with the user-defined control code 129, for
example:
sc.exe control Suricata 129
The service control handler only records the request; the main loop then
handles it the same as SIGHUP.
Ticket: 7757
On Windows, fopen() does not open files with FILE_SHARE_DELETE, so a log file
can not be renamed or deleted while Suricata has it open. This prevents
external tools from rotating log files.
Add SCWin32Fopen(), an fopen() replacement for the "a" and "w" modes that
opens the file with CreateFile() and FILE_SHARE_DELETE, and use it for log
files, including suricata.log.
Ticket: 7757
Windows has no SIGUSR2 and no unix socket, so rules could not be reloaded
without restarting Suricata. When running as a service, a rule reload can now
be requested with the user-defined control code 128, for example:
sc.exe control Suricata 128
The service control handler only records the request; the main loop then
handles it the same as SIGUSR2.
Ticket: 7756
By default, a Windows service allows interactive users and service accounts
to send it user-defined control codes (128-255). When installing the Suricata
service, set a DACL that is the default DACL without this right for those
users, so only SYSTEM and Administrators can send them.
This prepares for using user-defined control codes to control Suricata at
runtime, such as reloading rules.
Ticket: 7756
DetectEngineContentInspectionInternal checked smd for NULL only after
the recursion-limit return had read smd->type, and its NULL branch read
smd->type too. No caller passes NULL: engines without a keyword list
have no callback, empty sticky buffers fail to load, payload engines
exist only with payload keywords, and the recursion stops at the last
keyword.
Drop smd from the NULL check and assert it with DEBUG_VALIDATE_BUG_ON.
Document transform_result and both forms of absent, the transforms
that can signal an error, how must_error and error_or treat bodies and
files inspected in parts, and the rule-load constraints. Note the
behavior changes in upgrade.rst: absent's mode string in the analyzer,
absent: or_else rejected after a transform that can fail, gunzip and
zlib_deflate keeping their input on failure, pcrexform's empty capture,
and from_base64's bytes handling.
Issue: 7313
Issue: 7114
So far the SSL parser returned an error if there were less than 4 bytes
available for the handshake header. This meant that a fragmented
handshake header could result in the SSL parser erroring out and thus
disabling any further inspection for the protocol.
The parser follows a multi-record parsing scheme which means that the
fragment of the header may exist in the next record. Use internal
buffering to assess and process the records just like the handshake
buffer.
Ticket: 9167
Ticket: 8684
The behavior of the rule
alert ip any any -> any any (pcre:"/.*mc|\H+$/"; sid:4;)
depended upon the presence of another rule such as
alert ip any any -> any any (content:"*mcH"; sid:5;)
With the sid 5 content rule, the code went through PrefilterPktStream
which updated raw_stream_progress, which ended up updating raw_progress_rel
which was used on later packets to know where to start.
Without the sid 5 content rule, there was no update of these progresses.
It led to quadratic complexity, as for each new packet, we ran
the pcre engine on the whole tcp stream from the beginning, with
the newly added payload data.
Fix is to process sid 4 pcre rule the same way whether there is
or there is not a sid 5 content rule.
So, in case we do not have a mpm store for a signature group head,
we register a dummy stream prefilter, whose goal will just be
to update raw_stream_progress
DCERPC REQUEST/RESPONSE records can have their stub data fragmented
across multiple PDUs. The flags that convey the state of the current PDU
are as follows:
PFC_FIRST_FRAG: first fragment of (fragmented) data
PFC_LAST_FRAG: last fragment of (fragmented) data
PFC_FIRST_FRAG && PFC_LAST_FRAG: no fragmentation
Consider a REQUEST/RESPONSE complete only when the PFC_LAST_FRAG flag is
found to be set.
Ticket: 9240
Serialize service arguments with Windows quoting and backslash escaping
for both installation and parameter updates. Reject truncated executable
paths and command lines before accessing the Service Control Manager.
Keep the existing service-option prefix matching unchanged. Limit the
fix to command-line construction, without changing the command-line
parser, service function interfaces or SCInstance layout.
Ticket: 9241
With rotate-interval set to minute, hour or day, rotations were not
aligned to the local time boundaries. The first rotation was computed
by SCGetSecondsUntil(), which overshoots the next hour by one minute
(hh:01:00) and the next day by one hour and one minute (01:01:00).
Every following rotation was re-armed with a fixed interval
(e.g. 86400 s) from the time of the rotation.
Local days are not always 86400 s long, so the day rotation also moved
by the DST shift on every DST transition and stayed there until the
next one. A file named after a date then held events of the next day.
Instead, this commit rotates logs when the local minute, hour or date
changes. The daily rotation now happens at local midnight.
On SIGHUP, the period is recorded before the file is reopened, so a boundary
(the next minute/hour/day) passing during the reopen leads to an extra
reopen instead of a missed rotation.
Ticket: #9139
In continuous pcap-file directory mode, a file that makes InitPcapFile()
fail (malformed content, not a pcap, unsupported link type, truncated
header) was skipped without folding its mtime into last_time_seen.
This caused the file(s) to be reinspected on every follow-up poll until
a newer valid file was processed.
Ticket: 9129
The passed poll-interval should determine how many seconds to wait
in-between polls. The previous code passed the value through localtime
function which understood the input integer as an epoch time and
converted it to time structure, of which only seconds were kept.
This kind of worked for sub-60 values but wrapped for values 60+ to
the usual seconds range 0-59. Additionally, setting poll-interval to
60 made it a busy loop with no sleep between polls.
With intervals of 60+ seconds now honored, a single sleep blocked the
shutdown (or a unix socket pcap-interrupt) until the interval elapsed,
and the engine was killed after the 60 second receive thread shutdown
timeout. The wait is now done in one second steps, checking the run
status in-between.
Ticket: 9129
Change the byte order of vlan TCI when extracting it
from vlan_hdr in xdp_filter.c. This fixes xdp bypass for vlan tagged
packets, as without the fix flows that are supposed to be bypassed by
the xdp filter are not bypassed.
Ticket: 8699
A rule whose relative pcre follows a keyword that moves the detection
pointer is refused at load time with "pcre with /R (relative) needs
preceding match in the same buffer":
alert tcp any any -> any any (msg:"x"; byte_jump:4,0; pcre:"/^x/R"; ...)
The search for the preceding match only asked for content and pcre
sigmatches, so a rule that anchors a relative pcre on a byte_jump had
nothing to find even though byte_jump leaves the detection pointer at
the jump target.
Add the keywords that update det_ctx->buffer_offset to the search:
byte_jump, byte_extract and byte_math. byte_test and isdataat only read
that pointer, so they stay rejected; a relative pcre after them alone
would silently match from the start of the buffer.
Ticket: #7987.
A packet that netmap delivers across several ring slots was handed to the
rest of the code as the first slot buffer together with the summed length
of all the slots. When those slot buffers were not next to each other in
memory this read past the first buffer, which could crash the process or
leak bytes from unrelated buffers into inspection, pcap output and IPS or
tap forwarding.
Multi-slot packets are now copied slot by slot into a per-thread buffer so
the caller always gets one addressable frame, and such packets are always
copied into the Packet instead of being referenced zero-copy. Single-slot
packets keep the zero-copy path. The walk over the fragment slots is bounded
by the slots we own and the total length is capped, so a broken or over-long
chain is dropped as one unit and the ring keeps moving forward.
Adds unit tests that build a fake ring and cover single-slot, adjacent,
non-adjacent, ring-wrap, empty-fragment, over-size and never-ending chains.
Ticket: #8777
NetmapReadPackets walked the slots of a single packet inline while also
driving the per ring iteration. Move the slot walk into
NetmapReadOnePacket so the ring loop only picks the next packet and the
slot handling can be changed on its own.
No functional change.
Ticket: #8777
gunzip and zlib_deflate emptied the buffer when the input did not
decompress, and gave no signal when they could not get room for the
output, so transform_result could not be used with them.
Mark both SIGMATCH_TRANSFORM_CAN_FAIL, keep the input and flag the
error on either failure. When the input was in the buffer's own memory,
put the saved copy back.
Issue: 7313
transform_result stopped fast_pattern selection, and the validator
rejected fast_pattern on its buffer. Content after must_succeed only
matches when the transform succeeded, and the prefilter scans the same
transformed bytes, so "http.uri; from_base64; transform_result:
must_succeed; content:"evil";" lost its only fast pattern and ran on
every HTTP transaction.
Let content after must_succeed be the fast pattern, and accept
fast_pattern there.
Issue: 7313
Issue: 7114
RetrieveFPForSig stopped at absent and transform_result only while
marking candidate lists; the max_len pass and GetMpmForList walked
every instance of the chosen list to its end. Repeated multi-instance
buffers with no transforms, or with transforms such as to_lowercase,
share a list id, so in "http.request_header; content:"Host";
http.request_header; absent: or_else; content:"<longer>";" the or_else
content became the fast pattern.
Stop at absent and transform_result in all three passes.
Issue: 7313
Issue: 7114
file.data and http.request_body run the transforms on each window of a
large body or file. A strict from_base64 decode of a window cut off a
4-byte boundary fails, so must_error and error_or fired on valid base64:
a 200 KB body of unwrapped base64 raised must_error on both buffers.
Mark such windows DETECT_CI_FLAGS_PARTIAL (START and END not both set),
and have must_error and error_or ignore the error flag there.
must_succeed still keeps content from matching the window.
Issue: 7313
Issue: 7114
The file.name, file.magic and file.data engines built their content
inspection flags without buffer->flags, and DetectGetMultiData and the
dns.response getter overwrote buffer->flags after the transforms ran,
so a transform's error flag never reached content inspection.
OR buffer->flags into the flags passed on, and OR the single-buffer
bits into buffer->flags instead of assigning them.
Issue: 7313
Add transform_result, which matches on the error flag a transform sets
on its buffer: must_succeed does not match on an error, must_error
matches only on one, and error_or matches on an error, an absent
buffer, or the keywords after it. It shares DetectAbsentData with
absent, whose bool or_else becomes an enum DetectAbsentMode, and absent
rejects the transform_result arguments.
The validator requires a transform that can fail before
transform_result, rejects absent: or_else on such a buffer, rejects
must_error with other keywords and fast_pattern on the buffer, and says
when a transform placed after the keyword started a new buffer. The
engine analyzer reports a mode string for both keywords in place of the
or_else boolean.
Issue: 7313
Issue: 7114
pcrexform gave no signal when the regex did not match, when the capture
could not be extracted, or when there was no room for the capture. A
capture of nothing also left the input in place when the buffer had no
room yet, since InspectionBufferCopy then copies nothing.
Mark the transform SIGMATCH_TRANSFORM_CAN_FAIL and flag those failures
with SCInspectionBufferSetError. Set an empty capture's length to 0
directly, and leave an empty input alone, as from_base64 does.
Issue: 7313
Jeff Lucovsky
Jason Ish
Jeff Lucovsky
Shivani Bhardwaj
Philippe Antoine
Lukas Sismis
Philippe Antoine
Adam Kiripolsky
Victor Julien
Samaresh Kumar Singh