Cppcheck flagged this:
src/detect-engine-address.c:1035:48: warning: Either the condition 'ghn!=NULL' is redundant or there is possible null pointer dereference: gh. [nullPointerRedundantCheck]
int r = DetectAddressIsCompleteIPSpaceIPv4(gh->ipv4_head);
^
src/detect-engine-address.c:1297:17: note: Assuming that condition 'ghn!=NULL' is not redundant
if (ghn != NULL) {
^
src/detect-engine-address.c:1283:44: note: Calling function 'DetectAddressIsCompleteIPSpace', 1st argument 'ghn' value is 0
if (DetectAddressIsCompleteIPSpace(ghn)) {
^
src/detect-engine-address.c:1035:48: note: Null pointer dereference
int r = DetectAddressIsCompleteIPSpaceIPv4(gh->ipv4_head);
^
Cleanup code could only be reached with non-NULL pointers, so simplify checks.
Bug: #5291.
Remove useless allocation and free.
Found by cppcheck as a potential issue:
src/detect-engine-address-ipv6.c:385:12: warning: Either the condition 'tmp!=NULL' is redundant or there is possible null pointer dereference: tmp. [nullPointerRedundantCheck]
memset(tmp,0,sizeof(DetectAddress));
^
src/detect-engine-address-ipv6.c:525:13: note: Assuming that condition 'tmp!=NULL' is not redundant
if (tmp != NULL)
^
src/detect-engine-address-ipv6.c:385:12: note: Null pointer dereference
memset(tmp,0,sizeof(DetectAddress));
^
But code turned out not to do anything, so removed.
Bug: #5291.
Fix rules from the 'match' list getting added to the tx candidates list
unsorted. In some cases this could lead to the same sid getting inspected
twice leading to a DEBUG_VALIDATION_BUG_ON trigger.
Bug: #5144.
If a packet after the initialization would come with ACK flag set
but a ACK value of 0, the last_ack tracking could get confused. Fix
this by not checking for 0 but instead checking if the ACK flag
has been seen.
Bug: #4549.
The input data received in DATA and BDAT command modes can be huge and
could have important data, like a legit huge email. Therefore, exempt
these from the line buffering limits which were introduced to regulate
the size of lines that we buffer at any point in time.
As a part of this patch, anything that comes under DATA or BDAT is
processed early without buffering as and when it arrives. The ways of
processing remain the same as before.
Issue
-----
So far, with the SMTP parser, we would buffer data up until an LF char
was found indicating the end of one line. This would happen in case of
fragmented data where a line might come broken into multiple chunks.
This was problematic if there was a really long line without any LF
character. It would mean that we'd keep buffering data up until we
encounter one such LF char which may be many many bytes of data later.
Fix
---
Fix this issue by setting an upper limit of 4KB on the buffering of
lines. If the limit is reached then we save the data into current line
and process it as if it were a regular request/response up until 4KB
only. Any data after 4KB is discarded up until there is a new LF char in
the received input.
Cases
-----
1. Fragmentation
The limit is enforced for any cases where a line of >= 4KB comes as diff
fragments that are each/some < 4KB.
2. Single too long line
The limit is also enforced for any cases where a single line exceeds the
limit of buffer.
Reported by Victor Julien.
Ticket 5023
FTP control commands will be buffered forever until a new line is seen,
this can lead to memory exhaustion in Suricata.
To fix, set an upper bound, 4096 bytes on the size of the command that
is saved in the transaction. The input continues to be parsed to find
the end of the command so the parser can continue to move onto the next
command.
The result is that the command data in the transaction is truncated,
which also shows up in the ftp transaction logs.
This value is configurable with the max-line-length field in the ftp
app-layer.protocols section.
As FTP doesn't have events at this time, add a new fields to eve-log
that specificy if the request, or the response has been truncated.
Ticket #5024
Ticket: 5243
When switching from SMTP to TLS, and getting HTTP1 instead of
expected TLS, and HTTP1 requesting upgrade to HTTP2, we do not
overwrite the alproto_orig value so as not to have type confusion
in AppLayerParserStateProtoCleanup
Move raw detection logic out of main StreamReassembleRawDo() so that
it can be reused for other parts of the engine.
The caller now has to specify a right edge of the data.
Ticket: 4972
As is done in detect-lua-extensions.
We can have a flow with alproto unknown, no state, and therefore
cannot run AppLayerParserGetTx which could try to run a NULL
function
Since:
9551cd0535 ("threading: don't pass locked flow between threads")
`MoveToWorkQueue()` unconditionally unlocks the flow. This allows simpler
locking handling, including of tcp reuse flows.
The simpler logic also fixes a scenario where TCP reuse flows got "unlocked"
twice, once in `FlowGetFlowFromHash()` and once in `MoveToWorkQueue()`.
Bug: #5248.
Coverity: 1494354.
Some unittests used SCMalloc for allocating new Packet the unittests.
While this is valid, it leads to segmentation faults when we move to
dynamic allocation of the maximum alerts allowed to be triggered by a
single packet.
This massive patch uses PacketGetFromAlloc, which initializes a Packet
in such a way that any dynamic allocated structures within will also be
initialized.
Related to
Task #4207
If there is a space following a keyword that does not expect a value,
the rule fails to load due to improper value evaluation.
e.g. Space after "set" command
alert http any any -> any any (http.user_agent; dataset:set ,ua-seen,type string,save datasets.csv; sid:1;)
gives error
[ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - dataset action "" is not supported.
Fix this by handling values correctly for such cases.
A lot of time was spent in `SigMatchListSMBelongsTo` for the `mpm_sm`.
Optimize this by keeping the value at hand during Signature parsing and
detection engine setup.
The current code doesn't cover all rows when more than one flow manager is
used. It leaves a single row between ftd->max and ftd->min of the next
manager orphaned. As an example:
hash_size=1000
flowmgr_number=3
range=333
instance ftd->min ftd->max
0 0 333
1 334 666
2 667 1000
Rows not covered: 333, 666
With this check, on the first packet of a certificate presenting
a length of 16Mbytes, we only allocate up to 65Kb
When we get to the point where need more than 65Kb, we realloc
to the true size.
With this check, it makes it more expensive for an attacket to use
this allocation as a way to trigger ressource exhaustion...
Fix parsing of content like "|aa b cc|" which was parsed as "|aa bc|"
without error or warning. This will now fail out, requiring all hex
values to be 2 chars.
Ticket #5201
Ticket: 4972
Completes commit c3a220647
DETECT_CONFIG is added as DETECT_SM_LIST_POSTMATCH and not
as DETECT_SM_LIST_MATCH as other keywords handled in SignatureCreateMask
Only accept netmask in dotted quad notation if they can be turned
into a CIDR.
According to rfc 4632, CIDR (compat) netmasks are all that should be
used.
Bug: #5168.
Only accept netmask in dotted quad notation if they can be turned
into a CIDR.
According to rfc 4632, CIDR (compat) netmasks are all that should be
used.
Bug: #5168.
Ticket: 5132
In case of a duplicate range, we can return early, because
there is no new data to process.
More importantly, this commit adds a check about wether the file
got closed meanwhile, so that this just completed out of order
range, even if it brings new data, is now irrelevant.
This can happen for instance if there was a gap...
Separate the code paths between reusing a Packet stored host reference
and fetching a new reference from the host hash.
This addresses the issue where in some conditions use_cnt could get
desync'd.
Bug: #2802.
Introduce a new util function to free a Hosts iprep code. It also
handles the Host use_cnt decrement.
This change makes sure we also decrement the use_cnt when cleaning
up when shutting down the host table.
Move the BUG_ON check for use_cnt into the HostClearMemory() func
to check it in more cases.
Issue: 4550
This commit adjusts the per-thread stack size if a size has been
configured. If the setting has not been configured, the default
per-thread stack size provided by the runtime mechanisms are used.
Issue: 4550
This commit checks if there's a config setting for threading.stack-size
and assigns the value to a global variable for use during thread
creation.
On ARM 32bit with Musl `tv_usecs` is defined as `int64_t` which lead to
CreateIsoTimeString() printing all zeros on the usecs. Work around this
by first assigning to a `int64_t` and then updating the expected format
string to accept `int64_t`.
Bug: #5094.
Handle non-exact address ranges from string. This can come directly
from user input, so here it is accepted but the address is converted
to the address range start. A warning will be issued.
Debug validation checks are added to catch this.
This issue could lead to bad input from iprep (with cidr), defrag config
and htp server personalities to produce a bad radix tree.
Bug: #5084.
Bug: #5085.
Bug: #5086.
If the ipaddress was not the address range start, it was not masked to turn
it into that. So 1.2.3.4/24 was not stored as address 1.2.3.0 with netmask 24,
but as 1.2.3.4 with netmask 24. This was then propagated into the radix tree,
where it was used as an exact key in exact lookups, giving unexpected results.
This patch implements the netmask handling for IPv4 and IPv6, and adds a set
of tests for it.
Bug: #5081.
Bug: #5066.
A bug was reported about the IP-only rules not correctly matching. This was
traced to the rules in question not getting recorded into the IP-only radix
tree correctly.
Sequence:
- 100.117.241.0/25 inserted into the tree
- 100.117.241.0/26 inserted into the tree
Both are part of the same radix node, but recorded by their different netmasks
in the user data portion.
Then faulty insert happens:
- 100.117.241.64/26
For reference, these net blocks compute to:
- 100.117.241.0/25: 100.117.241.0 - 100.117.241.127
- 100.117.241.0/26: 100.117.241.0 - 100.117.241.63
- 100.117.241.64/26: 100.117.241.64 - 100.117.241.127
The IP-only engine first does a search to get to the user data it may need to
include. It does so for with `SCRadixFindKeyIPV4ExactMatch` for single IPs, or
using `SCRadixFindKeyIPV4Netblock` in case of a netblock. Any "match" from
either of these is considered an "exact match" by the IP-only setup code.
This exact match expectation turned out to be wrong and
`SCRadixFindKeyIPV4Netblock` behaved more like "best match" instead, which is
a non-exact match, but its the next best match if no exact match is found.
The way the look up for 100.117.241.64/26 went wrong, is that it returned
the user data for 100.117.241.0/26. This happens as follows:
- first it would do an exact find, which didn't give a result
- then it removed bits from the keystream until it found a matching node
and explore if any of the netmasks it contained matched. Here the first
step of the bug started:
it considered the netmask (with user data) a match that matched the
number of bits of the matching key, but not of the actual range netmask cidr
value.
So in this case the number of shared bits between `100.117.241.0/25` and
`100.117.241.64/26` was 25, so it assumed that the user data for the
netmask 25 was the match.
To summarize this step, there are 2 problems with this:
1. it returns a match on something that isn't an exact match
2. it considered the wrong netmask value
- the radix code then took the returned node, and did the netmask check
again. This time it did use its own netmask value, so this time
it did find the netmask 26 (+ user data). However because of the node that
was returned, this netmask (+user data) belongs to `100.117.241.0`, not to
`100.117.241.64`.
- the IP-only detection code was satisfied with what it assumed to be
"exact match" and just updated the user data to include the user data that
should have been associated with `100.117.241.64/26` to `100.117.241.0/26`.
This patch addresses the issue as follows:
It makes `SCRadixFindKeyIPV4Netblock` also return an exact match by propagating
the netmask in the search and in the evaluation of the stored netmasks.
It does away with the secondary netmask (+user data) evaluation.
`SCRadixFindKeyIPV4Netblock` is expected to handle this correctly.
The IP-only engine will fall back to the "not found" path, which does an explicit
"best match" lookup and then insert a new entry into the radix tree based on
the user data of the "best match".
Issue was present for IPv6 as well.
Bug: #5066.
Instead of building a table at init just calculate it on demand.
Callsites are all during init, so its not performance critical.
Add similar function for IPv6.
With these, the portion of code within the tags should be included
in the related code-snippets (for frame support documentation) w/o
errors, even if the code within changes. The tags can also work as
a reminder that the existing code is being shown elsewhere, so folks
know documentation might need updates, in case of major changes.
Ticket: 4948
This is not the perfect solution, but it prevents to trigger
the assert, and keep the assert.
A better solution would need to create transaction from
the reponse parsing, in case a later command was buffered and
not answered. But this would not be enough as NoNewTx prevents
the creation of a new transaction for RSET...
Ticket: 4530
As for HTTP2 and MQTT.
In FTP case, transactions are pipelined, not identified by an id.
So, there are less chances of DOS by quadratic complexity.
Parse extract-url-schemes from the mime config.
e.g. 'extract-urls-schemes: [http, https, ftp, mailto]'
Update MimeDecConfig struct to new url extraction fields.
Change app-layer-smtp.c & util-decode-mime.c to initialize new struct
fields for MimeDecConfig.
Sets the default value for extract-url-schemes if not found in the
config to 'extract-urls-schemes: [http]' for backwards compatibility.
Uses the schemes defined in the mime config value for
extract-urls-schemes to search for URLS starting with those scheme
names followed by "://".
Logs the URLS with the scheme + '://' at the start if the
log-url-scheme is set in the mime config, otherwise the old behaviour
is reverted to and the urls are logged with the schemes stripped.
Removed unused constant URL_STR now that URLS are being searched for
using extract-urls-schemes mime config values instead of just URL's
starting with 'http://'.
Added commented out new options for extract-urls-schemes and
log-url-scheme to suricata.yaml.in.
Update FindUrlStrings comments.
Remove old outdated comments/commented code from FindUrlStrings.
Update test case for mime which now needs schemes list to be set.
Add Test Cases for FindUrlStrings() method.
Feature: #2054
We want to check that a rule beginning with alert http
can be valid, that is if either HTTP1 or HTTP2 is enabled.
So, AppLayerProtoDetectGetProtoName will do a more complex
check for this ALPROTO_HTTP (any).
Suricata invokes the stream reassembly logic only for the current packet
direction if the packet contains a FIN flag. However, this does not
handle the case in which the packet ACKs data from the opposing direction.
This patch forces the invocation of the stream reassembly logic
on both direction when Suricata sees a FIN packet.
Completes commit e2370d6861
for all the fuzz targets processing pcaps
using a generic function.
FlowShutdown is not used because it uses the loop to destroy
mutexes, which we want to reuse for fuzzing
Indexing of Signature::init_data::smlists would fail for a rule that
used a frame w/o content, as the array would only be expanded when
adding a content. Adding a check to see if there list id is in bounds
is an implicit check for the "no content" case.
Bug #5011.
Pgsql was using bitwise operations to assign password output config to
its context flags, but mixing that with logic negation of the default
value, resulting in the expressions having a constant value as result.
Bug: #5007
When running with privilege dropping, the application log file
is opened before privileges are dropped resulting in Suricata
failing to re-open the file for file rotation.
If needed, chown the application to the run-as user/group after
opening.
Ticker #4523
Initialize the run-as user info after loading the config, but
before setting up logging (previously it was done while initializing
signal handlers). This will allow the log file to be given the
correct permissions if Suricata is configured to run as a non-root
user.
- add nom parsers for decoding most messages from StartupPhase and
SimpleQuery subprotocols
- add unittests
- tests/fuzz: add pgsql to confyaml
Feature: #4241
As is done for other targets,
so that all app-layer protocols are enabled,
even the ones disabled by default such as enip
And resets protocol detection every time we try
so that probing_parser_toserver_alproto_masks are fresh.
Implement a special sticky buffer to select frames for inspection.
This keyword takes an argument to specify the per protocol frame type:
alert <app proto name> ... frame:<specific frame name>
Or it can specify both in the keyword:
alert tcp ... frame:<app proto name>.<specific frame name>
The latter is useful in some cases like http, where "http" applies to
both HTTP and HTTP/2.
alert http ... frame:http1.request;
alert http1 ... frame:request;
Examples:
tls.pdu
smb.smb2.hdr
smb.smb3.data
Consider a rule like:
alert tcp ... flow:to_server; content:"|ff|SMB"; content:"some smb 1 issue";
this will scan all toserver TCP traffic, where it will only be limited by a port,
depending on how rules are grouped.
With this work we'll be able to do:
alert smb ... flow:to_server; frame:smb1.data; content:"some smb 1 issue";
This rule will only inspect the data portion of SMB1 frames. It will not affect
any other protocol, and it won't need special patterns to "search" for the
SMB1 frame in the raw stream.
The idea of stream frames is that the applayer parsers can tag PDUs and
other arbitrary frames in the stream while parsing. These frames can then
be inspected from the rule language. This will allow rules that are more
precise and less costly.
The frames are stored per direction in the `AppLayerParserState` and will only
be initialized when actual frames are in use. The per direction storage has a
fixed size static portion and dynamic support for a larger number. This is done
for effeciency.
When the Stream Buffer slides, frames are updated as they use offsets relative
to the stream. A negative offset is used for frames that started before the
current window.
Frames have events to inspect/log parser errors that don't fit the TX model.
Frame id starts at 1. So implementations can keep track of frame ids where 0
is not set.
Frames affect TCP window sliding. The frames keep a "left edge" which
signifies how much data to keep for frames that are still in progress.
Pruning of StreamBufferBlocks could remove blocks that fell entirely
after the target offset due to a logic error. This could lead to data
being evicted that was still meant to be processed in theapp-layer
parsers.
Bug: #4953.
This commit makes a small optimization when comparing IPv4 and IPv6
addresses by making the host order value invariant and calculating the
value once, before entering the loop.
Explicitly truncate file names to UINT16_MAX
Before, they got implicitly truncated, meaning a UINT16_MAX + 1
file name, went to 0 file name (because of modulo 65536)
This commit adds a signal handler for SIGSEGV when configured. The
signal handler emits a one line stack trace using SCLogError. The intent
is to provide diagnostic information in deployments where core files are
not possible.
The diagnostic message is from the offending thread and includes the
stack trace; each frame includes the symbol + offset.
Ticket: 4920
Completes commit c8dbe24fb6
which introduced AppProtoEquals to have a generic
check for http in signature can mean http1 or http2 in
traffic.
This commit missed this case, as I only looked for
git grep "alproto ==" and here we deal with
alproto_tc and alproto_ts, but not alproto by itself.
It appears that DNS servers will still process a DNS request even if the
z-bit is set, our parser will fail the transaction. So create the
transaction, but still set the event.
Ticket #4924
Set RSS hash function according to Intel ICE PMD available hash functions
Set hash functions according to the support by the ICE PMD, so that no warning
regarding RSS setting is issued.
Set RSS hash function according to Intel IXGBE PMD available hash functions.
During configuration, a warning appeared stating that RSS hash function
has been changed from one value to the other. This has meant that
the supported hash functions did not cover all required hash functions
by the configuration. This commit solves the warning.
Due to peculiar behavior of i40e PMD driver, the RSS is required to be set
via rte_flow rules or a hash filter as compared to other NICs where RSS is
configured through port configuration structure.
RTE_FLOW rules are created on 5-tuples (as opposed to 3-tuple configured
on the other NICs). Fragmented traffic have been tested with this setup
and it has been proven that fragmented packets of the same flow are
received on the same queue. At the same time, setting 3-tuple on rte_flow
rules have not yield in the expected results.
Notes from the experiments:
- Configuration of 5-tuple (as is in the commit):
fragmented and nonfragmented packets are received by the same workers
even when I applied seed to alter them via tcpreplay-edit (option --seed)
- Setting only ETH_RSS_FRAG_IPV4 and ETH_RSS_IPV4 (i.e. setting 3-tuple):
when setting ETH_RSS_IPV4, the PMD driver says that pctype is not
supported (generally this means that the "type" of traffic is not
a valid configuration for the i40e)
- Setting only ETH_RSS_FRAG_IPV4 and ETH_RSS_NONFRAG_IPV4_OTHER:
this doesn't work well, packets of the same flow are received on
the different workers (my explanation is that the fragmented packets are
matched with ETH_RSS_FRAG_IPV4 but the other UDP packets are not matched
with ETH_RSS_NONFRAG_IPV4_OTHER rte_flow rule (they would be matched with
ETH_RSS_NONFRAG_IPV4_UDP).
Register a new runmode - DPDK. This enables a new flag on Suricata start
(--dpdk).
With the flag given, DPDK runmode is enabled.
Runmode loads the configuration and then initializes EAL.
If successful, it configures the physical NICs according to the configuration
file. After that, worker threads are initialized and then are in continuous
receive loop.
Ticket: 4857
If a pattern such as GET is seen ine the beginning of the
file transferred over ftp-data, this flow will get recognized
as HTTP, and a HTTP state will be created during parsing.
Thus, we cannot override directly alproto's values
This solves the segfault, but not the logical bug that the flow
should be classified as FTP-DATA instead of HTTP
Instead of storing a name and description as a pointer in DetectBufferType
store them in fixed size arrays. This is in preparation of runtime registration
of buffer types, where a constant name/desc is not available.
In preparation of more dynamic logic in rule loading also doing
some registration, allow for buffers to be registered as fast_patterns
during rule parsing.
Leaves the register time registrations mostly as-is, but copies the
resulting list into the DetectEngineCtx and works with that onwards.
This list can then be extended.
Instead of a map that is constantly realloc'd, use 2 hash tables for
DetectBufferType entries: one by name (+transforms), the other by
id. Use these everywhere.
There is a hack to know the type of an integer
and do an explicit cast in the python script
generating the C file
Also extends some bounds check against negative values
Update the packet payload after decode, instead of during IPS send.
This means the updates happen in the capture thread, and the VLAN header
is available to logging as well.
Ticket: #4805.
Ticket: #4796.
V2 (for IDS and IPS) and V3 (for IDS) are widely supported. V2 was introduced
in 2008, so we can safely assume that all systems can run V2+.
314ec77f88 had the unintended side affect
of enabling the template parser and detection buffer if unittests were
enabled.
Fix this by using the new `Default` method for registering parsers.
However, the buffer still needs an explicit configuration check.
Also convert Notice debug messages to Debug to reduce output when in
unittest mode. If we feel stronly this should still be Notice in the
template, that is a conversion we can make in the generation script when
generating a new parser.
If a HTTP2 transaction gets freed before the end of the range
request, we need to have the files container which is in
the state, to transfer owernship of this file to the files
container.
Ticket: 4811
The SMTP transaction logger was not writing the log if the email
portion of the logger failed, such as in the case of STARTTLS
where this is no email decoded.
Ticket #4817
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.
This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
Rules profiling was returning invalid results when used with sample
rate. The problem was that the sample condition was run twice in the
packet flow. As a result, the second pass was not initializing the
variable storing the initial CPU ticks and the resulting performance
counters were reporting invalid values.
Bug: #4836.
Ticket: #4569
If a FIN+SYN packet is sent, the destination may keep the
connection alive instead of starting to close it.
In this case, a later SYN packet will be ignored by the
destination.
Previously, Suricata considered this a session reuse, and thus
used the sequence number of the last SYN packet, instead of
using the one of the live connection, leading to evasion.
This commit errors on FIN+SYN so that they do not get
processed as regular FIN packets.
Special handling for RST packets if they have an TCP MD5 or AO header option.
The options hash can't be validated. The end host might be able to validate
it, as it can have a key/password that was communicated out of band.
The sender could use this to move the TCP state to 'CLOSED', leading to
a desync of the TCP session.
This patch builds on top of
843d0b7a10 ("stream: support RST getting lost/ignored")
It flags the receiver as having received an RST and moves the TCP state
into the CLOSED state. It then reverts this if the sender continues to
send traffic. In this case it sets the following event:
stream-event:suspected_rst_inject;
Bug: #4710.
Ticket: #4562
As the data which triggered the opposing side
was the same protocol and not another one,
that means the protocol change failed.
Prevents a memory leak in later call of AppLayerParserParse
which would allocate a new state and leak the old one
Victor Julien
Shivani Bhardwaj
Jason Ish
Philippe Antoine
Juliana Fajardini
Arne Welzel
Modupe Falodun
Jeff Lucovsky
Victor Julien
Aaron Bungay
Angelo Mirabella
Lukas Sismis
Emmanuel Thompson
Steven Ottenhoff
Eric Leblond
Sam Muhammed
Benjamin Wilkins