Commit Graph

10509 Commits (56541c7c37d0de3f4c7eaafc19118ad837bbbc3d)

Author SHA1 Message Date
Victor Julien 253704961f pcap: free per thread resources
Bug: #4750.
(cherry picked from commit 3049151bc2)
3 years ago
Victor Julien f03a15688a pcap/file: minor code cleanup
(cherry picked from commit 6c1408c3c2)
3 years ago
Jason Ish 735f5aa9ca datasets: flag to disable "write" actions
Add a new configuration flag, "datasets.rules.allow-write" to control
if rules can contain "save" or "state" rules which allow write access
to the file system.

Ticket: #6123
3 years ago
Jason Ish aee1523b45 datasets: don't allow absolute or paths with directory traversal
For dataset filenames coming from rules, do not allow filenames that
are absolute or contain a directory traversal with "..". This prevents
datasets from escaping the define data-directory which may allow a bad
rule to overwrite any file that Suricata has permission to write to.

Add a new configuration option,
"datasets.rules.allow-absolute-filenames" to allow absolute filenames
in dataset rules. This will be a way to revert back to the pre 6.0.13
behavior where save/state rules could use any filename.

Ticket: #6118
3 years ago
Jason Ish b95bbcc66d lua: disable lua rules by default
To protect against possible supply chain attacks, disable Lua rules by
default. They can be enabled under the "security" section of
suricata.yaml.

Ticket: #6122
3 years ago
Shivani Bhardwaj 342e21a5ef smtp: handle long lines per direction
Issue:
Currently, while handling of long lines, if the line exceeded the limit,
we'd set a variable state->discard_till_lf which will be reset in the
later stages based on the data that arrives. However, because there was
one variable per state, this meant that a later stage in the other
direction could also modify it which is incorrect.

Fix:
Use separate variables for each direction.

Bug 6053
3 years ago
Shivani Bhardwaj 79a1b2edb5 smtp: handle following cmd if LF was found in long line
If a long line had LF post the limit, it should be considered complete
and not wait for the next line to complete it. However, currently, any
following lines were skipped which could sometimes also be important
commands for the entire transaction.

Fix this by setting a flag in case we're truncating a long line but
after having found the LF character.

Bug 5989
3 years ago
Shivani Bhardwaj f5db4bb7d5 smtp: add function docs 3 years ago
Shivani Bhardwaj 432842ad69 smtp: handle DATA mode in middle of input parsing
Before:
If the input was such that we'd enter DATA mode in the middle, the
entire data would be passed through SMTPGetLine fn and be processed with
line limits etc in place.

After:
Since we don't want any limits to be enforced on DATA, we pass it to
SMTPPreProcessCommands fn to take care of it differently from the
commands.

Bug 5981
3 years ago
Cole Dishington 96bb6ed034 decode-ipv6: Set IPv6 proto incase of ext header parsing error
Set the IPv6 packet proto before parsing the ext headers, similar to
decode-ipv4, incase of an ext header parsing error. Otherwise
rule decode-events are not triggered for packets encapsulated in IPv6.

Bug: #6086.
(cherry picked from commit 531d99f4cf)
3 years ago
Jeff Lucovsky e027080a4e detect/byte-math: Support multiplication operator
Issue: 6070

This commit adds support for the multiplication operator to byte-math.
The regex for parsing the keyword options was missing the `*` character.
3 years ago
Victor Julien b46d54178a counters: make tcp stats independent of flow, ssn
Counters depended on availability of flow and tcp session, meaning
that 2 memcaps could affect the counters.

Bug: #5017.
(cherry picked from commit 36f6e05155)
3 years ago
Victor Julien e275a1e28e stream: update no-flow checks
(cherry picked from commit 0360cb6542)
3 years ago
Shivani Bhardwaj 1b9e4fba06 ftp: don't decrement truncated line len
In case LF was found for a long line way outside of the limit, we should
not need to update the delimiter len and current line len because the
line is capped at 4k and the LF was not within these 4k bytes.
3 years ago
Shivani Bhardwaj 544ac300a9 ftp: separate truncated line markers
So far, we store one variable in state to hold whether we want to
discard a long line till LF irrespective of direction. This means that a
long command to the client followed by a regular command w LF can be
considered as one long line which is incorrect.

Bug 6055
3 years ago
Victor Julien ad041da715 windivert: fix compile warnings
(cherry picked from commit fd93f002a0)
3 years ago
Victor Julien 6767b1ce22 detect: remove flow drop unittest
Test broke after recent changes. Functionality is tested in
suricata-verify, so just remove the test.

(cherry picked from commit 8a535a0b89)
3 years ago
Victor Julien 5c2e6c4b83 detect: add check to validate drops
(cherry picked from commit 95bf7248e8)
3 years ago
Victor Julien 644a231e9a detect: fix stateful drops for rate_filter
(cherry picked from commit 418cc1fe94)
3 years ago
Victor Julien 043bbb9f51 flow/timeout: no pseudo packets for dropped flows
When a flow is in the drop flow state, don't use pseudo packets
when it is timing out. There should be no work left to do at this
point.

(cherry picked from commit 2a95154712)
3 years ago
Victor Julien 4b9cac426a stream: simplify drop handling
Remove logic to apply flow drop, as this is now handled in the
flow engine.

However, keep the logic that frees/cleans the session state.

(cherry picked from commit d91a1e8bc6)
3 years ago
Victor Julien 416cc8455f app-layer: don't update UDP applayer for dropped packets
(cherry picked from commit 77f49661fd)
3 years ago
Victor Julien 66aed4471d detect: update/document drop flow logic
Now that flow drop is applied to packets before other processing,
no drop has to be issued on a packet.

(cherry picked from commit 85ddba63f6)
3 years ago
Victor Julien 49051b637e flow: apply flow to packet on flow lookup
Issue drop to packet as early as possible.

(cherry picked from commit 71a033ac62)
3 years ago
Victor Julien 18a71913e3 respond/reject: fix IPv6 TCP resets
Fix length and next header field settings.

Bug: #6038.
(cherry picked from commit 235ee36211)
3 years ago
Victor Julien 8acf711667 respond/reject: minor code cleanups
(cherry picked from commit 1f0aed0775)
3 years ago
Victor Julien 2084354426 respond/reject: minor cleanups
(cherry picked from commit adf0bef7f0)
3 years ago
Shivani Bhardwaj e95e9281ca smtp: return on line completion
Problem:
If we receive a long line w/o LF, we cap it to 4k bytes and wait until a
line with LF comes in order to consider the previous line complete. Any
data post the 4k bytes is discarded. Currently, if a line with LF comes
in after a long line, we reset all the parameters used for processing it
like the line.len and line.delim_len but we still make the call to
SMTPProcessRequest fn without even the need to process anything. Since
such a line (with len and delim_len set to 0) should not reach mime
decoder, a debug assertion triggers there in this case.

Fix:
Make sure to return early as the line has to be skipped and not
processed at all.

Bug 6019

(cherry picked from commit c0067a5fff)
3 years ago
Jeff Lucovsky cb71800f7b pfring: Packet structure for ts fix
Issue: 5818

This commit addresses the issue with using the address of a packed
member of a structure. The pfring timeval is within a packed structure.

(cherry picked from commit 2d28c09ea1)
3 years ago
Jeff Lucovsky d782647af8 gen: Typo correction
This commit fixes various typos in the pf-ring source modules.

Issue: 5975
(cherry picked from commit e26e7b4f0a)
3 years ago
Jeff Lucovsky 32da57bce3 config/pf-ring: Change default cluster type: cluster_flow
This commit changes the default pf-ring cluster type to cluster-flow.
Round-robin clustering is not recommended for Suricata.

Issue: 5975
(cherry picked from commit 4f7a36ac2e)
3 years ago
Jeff Lucovsky 28026024c3 config/pf-ring: Recognize and set add'l cluster types
This commit extends the pf-ring config parser to recognize the
additional cluster types:
- cluster_inner_flow
- cluster_inner_flow_2_tuple
- cluster_inner_flow_4_tuple
- cluster_inner_flow_5_tuple

Issue: 5975
(cherry picked from commit b21a4ded6e)
3 years ago
Jeff Lucovsky a5752d138e pf-ring: Add add'l cluster types
This commit adds preprocessor values for additional pf-ring
cluster-types:
- CLUSTER_INNER_FLOW
- CLUSTER_INNER_FLOW_2_TUPLE
- CLUSTER_INNER_FLOW_4_TUPLE
- CLUSTER_INNER_FLOW_5_TUPLE

Issue: 5975
(cherry picked from commit 0ac3bee423)
3 years ago
Victor Julien 10d6c8796e stream: check debug check for multi-SYN/ACK in TFO
(cherry picked from commit 89c947129d)
3 years ago
Lukas Sismis a025070cc4 runmodes: introduce unknown engine runmode
To prevent unset values of engine runmode,
this commit introduces unknown runmode which
can detect when engine runmode is being used
uninitialized.

Ticket: #6033
3 years ago
Lukas Sismis 3c4bdf08de runmodes: earlier evaluation of IPS mode
Move evaluation of engine runmode to an earlier
phase so that dependend modules rely on properly
configured engine runmode.

Ticket: #5958
3 years ago
Lukas Sismis f08de8ea2e bpf: refactor the BPF code and postpone querying of the engine mode
BPF codebase queried engine mode earlier than it was determined from
the configuration file/command line. As a result it used the default (IDS)
mode where it could've been configured later on to the IPS mode.
This could lead into an undefined behavior as some Suricata modules behave
according to the engine mode.

PF-Ring, Netmap and AF-Packet all shared almost identical code for
determining the engine mode. It was put into one common function.
Omitted the usage of SCStrdup function in PF-Ring module as it is
uppercased during thread initialization phase.

Ticket: #5958
3 years ago
Jason Ish fe45258fbd detect: fix setting of flag for rule reload
As part of 6d8b50b748, the settings of
THV_CAPTURE_INJECT_PKT ended up in a location unreachable by capture
methods that did not have PktAcqBreakLoop.

Instead, always call TmThreadsCaptureBreakLoop which handles the logic
for how the read loop should be broken.

This fixes the case where read threads won't "break" for rule reloads
until packets are seen.

Ticket: #6021
(cherry picked from commit 1c6644ef4e)
3 years ago
Juliana Fajardini 75e57dde7c flow: make exc policy work w/ simulated flowmemcap
Exception policy wouldn't be applied if we were in the context of a
simulated flow memcap hit.

Bug #5998

(cherry picked from commit 1665f71a68)
3 years ago
Victor Julien 94b36cc4a3 src: fix extern max_pending_packets type 3 years ago
Philippe Antoine 79b89df331 detect: fix possible leak found by coverity
Conditions to create the leak are likely not reachable,
but this is still a bad pattern.

(cherry picked from commit e8060990d1)
3 years ago
Victor Julien 5f1651282d streaming: improve error handling
util-streaming-buffer.c:205:5: warning: Potential leak of memory pointed to by 'sbb2' [unix.Malloc]
    BUG_ON(sbb2->offset < sbb->len);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./suricata-common.h:290:27: note: expanded from macro 'BUG_ON'
        #define BUG_ON(x) assert(!(x))
                          ^~~~~~~~~~~~
/usr/include/assert.h:99:28: note: expanded from macro 'assert'
     ? __ASSERT_VOID_CAST (0)                                           \
                           ^
1 warning generated.
3 years ago
Victor Julien f8cec1e4f5 detect: improve prepare mpms routine
Based on hash table work in:
e624328deb ("detect: split mpm per alproto for file.data & others")

Instead of using a large stack array use a hash table for the intermediate
steps of the mpm build.
3 years ago
Victor Julien 36e07f874f streaming/sbb: propegate allocation errors
(cherry picked from commit d6b4c90225)
3 years ago
Victor Julien 0b8a46de39 scan-build: use simpler aligned alloc wrapper 3 years ago
Victor Julien cc0886d39d stream: fix minor scan-build warning
stream-tcp.c:134:14: warning: Value stored to 'presize' during its initialization is never read [deadcode.DeadStores]
    uint64_t presize = SC_ATOMIC_GET(st_memuse);
             ^~~~~~~   ~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.

(cherry picked from commit 0c15114450)
3 years ago
Victor Julien aaee63bab1 mime: address scan-build warnings
util-decode-mime.c:189:31: warning: Use of memory after it is freed [unix.Malloc]
            lastSibling->next = entity->child;
            ~~~~~~~~~~~~~~~~~ ^
util-decode-mime.c:827:24: warning: Potential leak of memory pointed to by 'val' [unix.Malloc]
        state->hname = NULL;
                       ^~~~
/usr/lib/llvm-16/lib/clang/16/include/stddef.h:89:24: note: expanded from macro 'NULL'
 #  define NULL ((void*)0)
                       ^
2 warnings generated.

Improve error handling and add assert to avoid these warnings.

Bug: #3147.
(cherry picked from commit 9224b3435b)
3 years ago
Victor Julien b802a6241f smtp/mime: no error logging in packet path
(cherry picked from commit 9d3c60bde3)
3 years ago
Victor Julien 07639c96f2 radix: add debug validation to assist scan-build
util-radix-tree.c:595:34: warning: Access to field 'stream' results in a dereference of a null pointer (loaded from field 'prefix') [core.NullDereference]
        if ((temp = (stream[i] ^ bottom_node->prefix->stream[i])) == 0) {
                                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~
util-radix-tree.c:717:30: warning: Access to field 'stream' results in a dereference of a null pointer (loaded from field 'prefix') [core.NullDereference]
        if (SC_RADIX_BITTEST(bottom_node->prefix->stream[differ_bit >> 3],
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~
./util-radix-tree.h:27:34: note: expanded from macro 'SC_RADIX_BITTEST'
 #define SC_RADIX_BITTEST(x, y) ((x) & (y))
                                 ^
2 warnings generated.

(cherry picked from commit fa5acc1743)
3 years ago
Victor Julien 88b305b539 detect/sigorder: assist scan-build
Bug: #3152.
(cherry picked from commit b625aa9748)
3 years ago
Victor Julien d2375ee7d2 detect/sigorder: remove unused struct fields
(cherry picked from commit 03e0a60f96)
3 years ago
Victor Julien 47b25329aa detect: fix scan-build warnings
detect-engine-address.c:1140:17: warning: Use of memory after it is freed [unix.Malloc]
            r = DetectAddressCmp(ag, ag2);
                ^~~~~~~~~~~~~~~~~~~~~~~~~
detect-engine-address.c:1169:17: warning: Use of memory after it is freed [unix.Malloc]
            r = DetectAddressCmp(ag, ag2);
                ^~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings generated.

detect-engine-port.c:1161:9: warning: Use of memory after it is freed [unix.Malloc]
        DetectPortPrint(ag2);
        ^~~~~~~~~~~~~~~~~~~~
1 warning generated.

Bug: #3150.
Bug: #3151.
(cherry picked from commit 000064de7d)
3 years ago
Victor Julien 37925fe2bd mpm/ac-bs: work around scan-build warnings
util-mpm-ac-bs.c:482:32: warning: Result of 'malloc' is converted to a pointer of type 'uint16_t[256]', which is incompatible with sizeof operand type 'uint16_t' [unix.MallocSizeof]
        ctx->state_table_u16 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
util-mpm-ac-bs.c:524:32: warning: Result of 'malloc' is converted to a pointer of type 'uint32_t[256]', which is incompatible with sizeof operand type 'uint32_t' [unix.MallocSizeof]
        ctx->state_table_u32 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
2 warnings generated.

Bug: #3148.
(cherry picked from commit c8694634af)
3 years ago
Victor Julien 763833c217 mpm/ac: work around scan-build warnings
util-mpm-ac.c:531:32: warning: Result of 'malloc' is converted to a pointer of type 'uint16_t[256]', which is incompatible with sizeof operand type 'uint16_t' [unix.MallocSizeof]
        ctx->state_table_u16 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
util-mpm-ac.c:575:32: warning: Result of 'malloc' is converted to a pointer of type 'uint32_t[256]', which is incompatible with sizeof operand type 'uint32_t' [unix.MallocSizeof]
        ctx->state_table_u32 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
2 warnings generated.

Bug: #3148.
(cherry picked from commit ee683a7074)
3 years ago
Victor Julien 77e60e3368 mpm/hs: fix scan-build warning
util-mpm-hs.c:340:20: warning: Potential leak of memory pointed to by 'p' [unix.Malloc]
        p->sids[0] = sid;
        ~~~~~~~~~~~^~~~~
1 warning generated.

Incorrect error handling could lead to a memory leak.

(cherry picked from commit ec84ba1a3c)
3 years ago
Victor Julien 333021e670 decode: suppress scan-build warning
(cherry picked from commit 59ca5cc655)
3 years ago
Victor Julien 16cadd7e59 suricata: work around scan-build warnings
suricata.c:691:17: warning: Value stored to 'bits' during its initialization is never read [deadcode.DeadStores]
    const char *bits = "<unknown>-bits";
                ^~~~   ~~~~~~~~~~~~~~~~
suricata.c:692:17: warning: Value stored to 'endian' during its initialization is never read [deadcode.DeadStores]
    const char *endian = "<unknown>-endian";
                ^~~~~~   ~~~~~~~~~~~~~~~~~~
2 warnings generated.

(cherry picked from commit c13a6fa73c)
3 years ago
Victor Julien 27ef9a96af scan-build: treat as debug validate
(cherry picked from commit bcf8187443)
3 years ago
Jason Ish 00a9edae3e detect-fast-pattern: remove unused var alertcnt
Set, but never read.

Caught by -Wunused-but-set-variable which is enabled in the newer
Fedora CI builders.
3 years ago
Justin Azoff a179f79eb6 detect/iponly: remove DetectEngineIPOnlyThreadCtx
This is unused.

Issue: 4578
(cherry picked from commit aacb7dc291)
3 years ago
Justin Azoff 03e767536a detect/iponly: Reduce the size of the SigNumArray bitsets
Instead of tracking ip only rules by the internal signum, track them by
a separate counter that starts at zero.  This results in dense
SigNumArrays instead of sparse ones and a much smaller max_idx.

Issue: 4578
(cherry picked from commit dfbc3da0eb)
3 years ago
Victor Julien f2cf58eec7 detect/content: add negated endswith test
(cherry picked from commit a42c225117)
3 years ago
Jeff Lucovsky 0a9b7d0528 detect/content: Negated endswith matches
Issue: 5541

This commit handles negated endswith matches.

(cherry picked from commit c083cbda33)
3 years ago
Jeff Lucovsky a7fbfd572c detect/byte_math: fix bug in byte_math detection
Issue: 5945

Avoid division by zero when the byte_math operation is division and the
rvalue is 0.

(cherry picked from commit 38c5e89e29)
3 years ago
Shivani Bhardwaj 92d72344fb smtp: enforce line limit even when LF is found
Before:
If LF character was found, so far, we won't enforce the line limit on
the line. We only enforced limits in case of LF character missing in a
long line.

After this patch:
Line limit is enforced on the line if it is bigger than 4096 Bytes
irrespective of whether LF was found or not.

Redmine Bug: 5819

(cherry picked from commit 5f52b199ff)
3 years ago
Shivani Bhardwaj 08fd66718a smtp: reset current line len at start 3 years ago
Shivani Bhardwaj 564c7793a5 util/mime: allow delim len 0 when line limit is hit
(cherry picked from commit fd4e0fbafe)
3 years ago
Shivani Bhardwaj 707dc705d9 smtp: move constant declaration to header
(cherry picked from commit c0bff5f921)
3 years ago
Victor Julien 80abc3121d pcap: improve pcap_breakloop support
When pcap_breakloop has been issued on a handle, the current pcap_dispatch
call may return -2 (PCAP_ERROR_BREAK), but it can also return the number
of processed packets if lower than the desired number. So add this condition
as a check.

(cherry picked from commit 9fe08f2374)
(cherry picked from commit 2ddd26446e)
3 years ago
Victor Julien 6d8b50b748 detect: only breakloop threads that are lagging
Sleep after all threads have been checked.

Bug: #5969.
(cherry picked from commit 8a968faa04)
3 years ago
Victor Julien 6658300c84 flow/worker: refresh detect thread during housekeeping
During housekeeping multiple flows are processed. If a rule reload happens
at that time, we need to use the new detect thread as soon as possible.

Bug: #5969.
(cherry picked from commit 5e4cf182ab)
3 years ago
Philippe Antoine 749277d46f http: complete multipart until request.body-limit
In the case we are truncating a multipart file because of reaching
request.body-limit, we used to not consume the whole buffer, but
keep expected_boundary_len bytes in case a new boundary begins
in these bytes.
Even if we cannot check the complete boundary, we can still check
the first bytes, as will be done in the rust version.

Ticket: #5952
(cherry picked from commit 578f328e06)
(cherry picked from commit caf9940fd1)
3 years ago
Victor Julien 2c600336ab stream: improve FIN checking
After recent next_seq changes, the FIN checks could be too strict
leading to stalling sessions in IPS mode.

This patch requires a FIN to be >= last ack and <= next_win to be
accepted.

(cherry picked from commit 39a6f411e9)
3 years ago
Shivani Bhardwaj 3442b6da9d util/base64: don't reset decoded bytes in RFC4648
Old behavior:
With RFC4648, the decoded bytes were reset to 0 in case an unusual
character was encountered in the encoded string. This worked out fine
for small test cases where there weren't many bytes to be decoded.

Problem:
If a big encoded string had a character outside of the base alphabet,
the processing would stop and the number of decoded bytes were set to 0.
However, even though the processing should stop at the invalid
character, the number of decoded bytes should correctly store the bytes
decoded up until the point an invalid characted was encountered.

New behavor:
For any base64 encoded string given to the base64 decoder in RFC4648
mode, we make sure that the number of decoded bytes correctly reflect
the number of bytes processed up until the string was valid. This makes
sure any further calculations/use of the decoded data is done correctly.

Redmine ticket: 5885

(cherry picked from commit 418ddba38e)
3 years ago
Philippe Antoine 3851504638 enip: optimized tx iterator
As for SMTP, having a linked list.

Ticket: #5927
(cherry picked from commit 4f7426fdcf)
3 years ago
Philippe Antoine ff2e02844b dnp3: optimized tx iterator
As for SMTP, having a linked list.

Ticket: #5927
(cherry picked from commit e15daf6a4b)
3 years ago
Philippe Antoine 8fd0862f94 smtp: optimized tx iterator
To be more efficient with larger number of transactions.
As was done for FTP.

Ticket: #5927
(cherry picked from commit f5f215dae7)
3 years ago
Eric Leblond ce12cb85d7 enip: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
3 years ago
Eric Leblond 4a3fdedc1f app-layer: add flag to skip detection on TX
Stamus team did discover a problem were a signature can shadow
other signatures.

For example, on a PCAP only containing Kerberos protocol and where the
following signature is matching:

alert krb5 $HOME_NET any -> any any (msg:"krb match"; krb5_cname; content:"marlo"; sid:3; rev:1;)

If we add the following signature to the list of signature

alert ssh $HOME_NET any -> any any (msg:"rr"; content:"rr"; flow:established,to_server; sid:4; rev:2;)

Then the Kerberos signature is not matching anymore.

To understand this case, we need some information:

- The krb5_cname is a to_client keyword
- The signal on ssh is to_server
- Kerberos has unidirectional transaction
- kerberos application state progress is a function always returning 1

As the two signatures are in opposite side, they end up in separate
sig group head.

Another fact is that, in the PCAP, the to_server side of the session
is sent first to the detection. It thus hit the sig group head of
the SSH signature. When Suricata runs detection in this direction
the Kerberos application layer send the transaction as it is existing
and because the alstate progress function just return 1 if the transaction
exists. So Suricata runs DetectRunTx() and stops when it sees that
sgh->tx_engines is NULL.

But the transaction is consumed by the engine as it has been evaluated
in one direction and the kerberos transaction are unidirectional so
there is no need to continue looking at it.

This results in no matching of the kerberos signature as the match
should occur in the evaluation of the other side but the transaction
with the data is already seen has been handled.

This problem was discovered on this Kerberos signature but all
the application layer with unidirectional transaction are impacted.

This patch introduces a flag that can be used by application layer
to signal that the TX should not be inspected. By using this flag
on the directional detect_flags_[ts|tc] the application layer can
prevent the TX to be consumed in the wrong direction.

Application layers with unidirectional TX will be updated
in separate commits to set the flag on the direction opposite
to the one they are.

Ticket: #5799
3 years ago
Eric Leblond 5c1995a78f detect: remove STREAM_FLUSH
It is unused in the code so can be removed.

Ticket: #5799
3 years ago
Eric Leblond e1b02a08ce app-layer-parser: give direction to progress func
The tx progress functions are expecting a direction and were given
a flow flags. As a result, they were not reporting correctly the
status if a DetectRunScratchPad flow_flags was containing some other
bits in the flag.

One case was when a signature was alterating the stream analysis
and triggering the addition of the STREAM_FLUSH flags.

The consequences are quite severe as the transactions are pilling
up waiting to be inspected causing sometimes a 10x performance hit
on pcap parsing. Also as the inspection was not done, Suricata is
missing a part of the alerts.

This was discovered when working on the following set of signatures:

alert ssh $HOME_NET any -> any any (msg:"pcre without content"; pcre:"/rabbit/"; sid:1; rev:1;)
alert smb $HOME_NET any -> any any (msg:"smb share content"; smb.share; content:"C"; sid:2; rev:1;)

When the first one is present the second is not triggering even
though the pcap file had no ssh inside. This is due to the fact
that the ssh signature was triggering the STREAM_FLUSH flag to
be set on the flowflags of the packet. But the application
layer will ask the smb state progress via

r = alp_ctx.ctxs[FlowGetProtoMapping(ipproto)][alproto].
        StateGetProgress(alstate, flags);

passing it the flow flags but the smb function is expecting
a direction so we end up in a unplanned case

pub unsafe extern "C" fn rs_smb_tx_get_alstate_progress(tx: *mut ffi::c_void,
                                                  direction: u8)
...
if direction == Direction::ToServer as u8 && tx.request_done {

This leads the signature to not be evaluated correctly.

Ticket: #5799
3 years ago
Juliana Fajardini e9da3378ca exception/policy: use pkt action if no flow support
Defrag memcap and flow memcap do not support flow action for the
exception policies, as there is no flow when the exception condition is
hit. In such cases, the exception policy must be considered for the
packet only, when that makes sense, or should be ignored, in case of
`bypass`.

Bug #5940

(cherry picked from commit d4333fb959)
4 years ago
Philippe Antoine c64261f374 detect: bump detect engine version for tenant reload
Because the engine version is used to free the old
variables and not the new ones.
As is done in DetectEngineReload.

Ticket: #5866
(cherry picked from commit d313b5d605)
4 years ago
Philippe Antoine 5dcf7f80df detect: bytemath do not left shift more than 64
As it is undefined behavior by C standard.
In this case, zeroes the value.

Ticket: #5900
(cherry picked from commit 473ca6dcf4)
4 years ago
Victor Julien 6ee3e79849 stream: accept and flag ack of ZWP data
Tcp Zero Window Probes try to send a single byte payload to "probe" if
the window has reopened. This single byte is, if accepted, not retransmitted.

(cherry picked from commit 30a716a4ab)
4 years ago
Victor Julien ad1efb8e53 stream: harden tcp reuse check against RST/FIN
(cherry picked from commit 64fb4066cf)
4 years ago
Victor Julien 516ddf4fec stream: improve SYN and SYN/ACK handling with ECN/CWR flags
(cherry picked from commit 0d1d288544)
4 years ago
Victor Julien 73ccd0de1d stream: fix TFO overlap detection with ECN/CWR flags
(cherry picked from commit 5fe2fba184)
4 years ago
Victor Julien 85fc446c55 stream: turn session flags into u32
Due to gaps/padding the size of the struct won't change.

(cherry picked from commit aa7d58b0c7)
4 years ago
Juliana Fajardini 7bfe59f936 flow/manager: apply clang formatting changes 4 years ago
Juliana Fajardini 2832b4a2ae flow/manager: fix coverity divide_by_zero warning
Updated all cases where flow_config.prealloc was used in a division.

*** CID 1524506:  Integer handling issues  (DIVIDE_BY_ZERO)
/src/flow-manager.c: 858 in FlowManager()
852                                "flow_spare_q status: %" PRIu32 "%% flows at the queue",
853                             spare_pool_len, flow_config.prealloc,
854                             spare_pool_len * 100 / flow_config.prealloc);
855
856                     /* only if we have pruned this "emergency_recovery" percentage
857                      * of flows, we will unset the emergency bit */
>>>     CID 1524506:  Integer handling issues  (DIVIDE_BY_ZERO)
>>>     In expression "spare_pool_len * 100U / flow_config.prealloc", division by expression "flow_config.prealloc" which may be zero has undefined behavior.
858                     if (spare_pool_len * 100 / flow_config.prealloc > flow_config.emergency_recovery) {
859                         emerg_over_cnt++;
860                     } else {
861                         emerg_over_cnt = 0;
862                     }

Related to
Bug #5919

(cherry picked from commit 754d2803dd)
4 years ago
Juliana Fajardini f2e863bfe8 flow/manager: fix prealloc unhandled division by 0
If flow.prealloc was set to zero in our yaml config, this led to
a floating point exception in the flow manager.

Bug: #5919.
(cherry picked from commit 5baa6c0024)
4 years ago
Shivani Bhardwaj 2e2483d27a util/mime: use uint8_t for cnt
cnt should not exceed B64_BLOCK which is 4, so, it is unnecessary to use
uint32_t for the variable.
4 years ago
Shivani Bhardwaj 082c7033ad util/mime: use uint32_t for consumed bytes
In a case of the line buffer being over 255 bytes, the consumed bytes
would reset to 0 as it was uint8_t. Fix this integer overflow by setting
the type to uint32_t.

Redmine ticket: 5883

(cherry picked from commit c089bbb7d7)
4 years ago
Jeff Lucovsky 6a66157e34 netmap/config: New API display during debug
This PR changes the log level associated with the message indicating
if the new Netmap API (v14+) is in use during module registration.

Issue: 5879
4 years ago
Bill Meeks 4835befec5 netmap: packet stall
- Fix packet processing stall under high load when using netmap in IPS mode.
- Detect and generate Fatal Error exit for rare case when hardware NIC exposes
unmatched RX/TX queue counts. This is rare, but would result in some traffic
bypassing Suricata since it assumes NIC queue counts are symmetrical.
- Fix instance of missing unlock call for netmap device list when exiting due
to an error condition.
- Clean up existing code comments and add additional ones to better document
the new netmap v14 API code.
4 years ago
Victor Julien 517132b6ad eve/drop: don't log drops unless packet is dropped
In pass/drop combinations where the pass rule took precendence over
the drop, a "drop" false positive could still be logged due to the
storing of the drop record in the packet drop alert store.

Bug: #5867.
(cherry picked from commit 09348564f0)
4 years ago
Victor Julien 7838fc8b82 detect/urilen: fix applying urilen as depth
If urilen induced depth was set, later DetectContentPropagateLimits()
would apply a wrong depth setting, leading to a false negative in
some cases.

Bug: #5929.
(cherry picked from commit ba7db2583b)
4 years ago
Victor Julien a48e0229ae stream: SYN queue support
Support case where there are multiple SYN retransmits, where
each has a new timestamp.

Before this patch, Suricata would only accept a SYN/ACK that
matches the last timestamp. However, observed behavior is that
the server may choose to only respond to the first. In IPS mode
this could lead to a connection timing out as Suricata drops
the SYN/ACK it considers wrong, and the server continues to
retransmit it.

This patch reuses the SYN/ACK queuing logic to keep a list
of SYN packets and their window, timestamp, wscale and sackok
settings. Then when the SYN/ACK arrives, it is first evaluated
against the normal session state. But if it fails due to a
timestamp mismatch, it will look for queued SYN's and see if
any of them match the timestamp. If one does, the ssn is updated
to use that SYN and the SYN/ACK is accepted.

Bug: #5856.
(cherry picked from commit 7bfee147ef)
4 years ago
Victor Julien 66ddbc3a8b stream: move state queue code into util func
(cherry picked from commit 288086af3f)
4 years ago
Victor Julien 7ccec8f143 debug: add bool string print helper macro
(cherry picked from commit 0ec136621d)
4 years ago
Victor Julien 1ed8066d4e stream: support SYN/ACK with TFO only ack'ing ISN
Not ack'ing the data.

(cherry picked from commit 7ef57cc7cb)
4 years ago
Victor Julien 7f4a800156 stream: add liberal timetamps option
Linux is slightly more permissive wrt timestamps than many
other OS'. To avoid many events/issues with linux hosts, add an
option to allow for this slightly more permissive behavior.

Ideally the host-os config would be used, but in practice this
setting is rarely set up correctly, if at all.

This option is enabled by default.

(cherry picked from commit 01b7ccc224)
4 years ago
Victor Julien fee655bae6 stream: D-SACK detection and logging
RFC 2883 specifies a special use of SACKs to indicate a host has
received a segment it considers a spurious retransmission.

(cherry picked from commit d79a926085)
4 years ago
Victor Julien dcefc00b17 stream: fix next_seq updates after temporary gap
On every accepted packet in established state, update next_seq if
packet seq+len is larger than existing next_seq. This allows it to
catch up after large gaps that are filled again a bit later.

Bug: #5877.
(cherry picked from commit 76225bf9ac)
4 years ago
Victor Julien 006911fd5a stream: improve first FIN next_seq update
For accepted FIN packets, always update next_seq. This helps track the
FIN sequence.

Bug: #5877.
(cherry picked from commit 551fb80150)
4 years ago
Victor Julien a8cff36467 stream: improve FIN next_seq handling
Update next_seq to SEQ + payload_len + 1, so retransmission checks
work better.

Bug: #5877.
(cherry picked from commit 80a012a787)
4 years ago
Victor Julien f1904398a7 stream: set next_seq before last_ack
next_seq sometimes depends on last_ack in cases of packet loss
catch up, so first update it.

Bug: #5877.
(cherry picked from commit 20df715e64)
4 years ago
Eric Leblond 984698f8c5 stream/tcp: update ack handling logic
Only update the ack value of a session for regular packets when
the ACK bit is set.

(cherry picked from commit 0d81173d6e)
4 years ago
Victor Julien 237e06a991 stream/tcp: fix wrong ACK trigger FIN1 to FIN2
An ACK that ACK'd older data while still being in-window could
lead to FIN_WAIT1 to FIN_WAIT2 state transition. Detect this
case and generally harden the check.

Bug: #5877.
(cherry picked from commit 67af94f2e0)
4 years ago
Victor Julien 2b18412509 stream: fix overlap detection
If a in-tree segment was partly before base_seq the overlap detection
miscalculated the data offset. This lead to memcmp comparing the wrong
data.

Bug: #5881.

(cherry picked from commit cbcd70537a)
4 years ago
Victor Julien d0b1a04eb0 stream: set event on bad timestamp on syn_sent state
(cherry picked from commit fc376f8145)
4 years ago
Victor Julien 68bd193868 stream/tcp: allow tcp session reuse on null sessions
When a "stream starter" packet finds an existing TCP flow, the flow will be
evaluated for reuse.

The following scenario wasn't handled well:

1. Suricata starts after a tool has just stopped using lots of connections
   (e.g. ab stress testing a webserver)
2. even though the client is closed already, the server is still doing
   connection cleanup sending many FINs and later RSTs
3. Suricata creates flows for these packets, but no TCP sessions
4. client resumes testing, creating flows that have the same 5 tuple as the
   flows created for the FIN/RST packets
5. Suricata refuses to "reuse" the flows as the condition "tcp flow w/o session"
   is not considered valid for session reuse
6. new TCP connection is not properly tracked and evaluated in parsing and
   detection

There may be other vectors into this, like a flow w/o session because of
memcap issues.

Bug: #5843.
(cherry picked from commit 93c9594dd8)
4 years ago
Victor Julien d13bb7f5a7 flowworker: don't keep unnecessary flow reference
Flow stream/detect/log flush packets, don't hold on to the flow
beyond the flow worker module.

(cherry picked from commit a2dc9a40e7)
4 years ago
Victor Julien c08521cd52 threshold: suppress debug notice message 4 years ago
Victor Julien 777d88bd21 decode/events: add strict mode for udp.hlen_invalid; remove rule 4 years ago
Shivani Bhardwaj eb26d1a64d decode/udp: fix payload_len calculation
Fix payload_len calculation post removal of the condition that returned
error code if the length to the decode fn did not match the length of
header from the UDP packet.

Bug 5379

(cherry picked from commit f941ceae2b)
4 years ago
Shivani Bhardwaj bbd53bcdf6 decode/events: add event type UDP_LEN_INVALID
(cherry picked from commit eebdfe9a3e)
4 years ago
Lukas Sismis f6fa16ce25 decode-udp: Allow shorter UDP packets than the remaining payload length
If the packet is shorter than IP payload length we no longer flag it as an
invalid UDP packet. UDP packet can be therefore shorter than IP payload.
Keyword "udp.hlen_invalid" became outdated as we no longer flag short UDP
packets as invalid. The keyword's evaluation remains the same.

Inspired by commit: d18e52ed93

Redmine ticket: #5793
4 years ago
Philippe Antoine fd358a5616 fuzz: checks pcap timestamp consistency
Inspired by commit e22b59b8c1

Master7 is now diverging with use of SCTime_t
4 years ago
Philippe Antoine 4b2a59a4d9 detect: config keyword transaction logic fix
When the keyword config:logging disable,type tx is used,
OutputTxLog checks a flag to skip the transaction without logging
it, but AppLayerParserTransactionsCleanup waits for the
transaction to be marked as logged to clean it.

So, OutputTxLog now marks the tx as logged, so that it can
get cleaned away.

Ticket: #5456
(cherry picked from commit 0c0fcc9411)
4 years ago
Juliana Fajardini d47e8fffad stream/tcp: fix typos, update copyright year
Bug #5765
4 years ago
Juliana Fajardini 700b0a37a1 exceptions: ignore policy if stream.midstream=true
Set the engine to ignore the stream.midstream-policy if stream.midstream
is enabled.

If we had both stream.midstream AND stream.midstream_policy enabled,
this could lead to midstream flows being dropped (or bypassed, or...)
instead of being accepted by the engine, as it was probably meant when
the user enabled midstream flows.

Bug #5765

(cherry picked from commit 8e9905e0d8)
4 years ago
Philippe Antoine 7986919df0 fuzz: specify protocol with fuzz target name
cf https://redmine.openinfosecfoundation.org/issues/4125

This allows fuzz_applayerparser_parse to fuzz one specific
app-layer protocol based on the binary name, as is done
with the environment variable FUZZ_APPLAYER
That is if we rename/copy to fuzz_applayerparser_parse_smb,
it will fuzz only SMB protocol
This way, we can easily produce different fuzz targets for
each protocol in oss-fuzz

(cherry picked from commit e9b76a0e66)
4 years ago
Jeff Lucovsky 2f8f45ff6a netmap: Fixup issues with v14+ backport
This commit reduces the changes associated with adding the v14 api to
6.0.x

During the preparation of this commit, issues in the original backport
were corrected
- Failure to release a lock under error conditions
- Typo in an CPP ifdef
- Incorrect target for goto statement in an error handling case.

Issue: 5744
4 years ago
Philippe Antoine 2c49a268e2 test: do not output non ascii character
The unit test for content |aa bz| transforms in place the string
str to replace the 2 characters aa by one character 0xaa
Then, when z is not recognized as a valid hexadeicmal character,
the whole modified string is printed out, inclusing the non-ascii
0xaa

Ticket: #5558
(cherry picked from commit b281199e9a)
4 years ago
Victor Julien 26fc02227b streaming/sbb: fix debug message
(cherry picked from commit 989e5c7ba2)
4 years ago
Victor Julien eb03cdea85 stream: remove bug on from packet path
(cherry picked from commit a00db15bd4)
4 years ago
Victor Julien 9664d97129 src: fix deprecated-non-prototype compile warnings
Tested on Fedora 37 with clang 15.

util-strlcatu.c:45:8: error: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Werror,-Wdeprecated-non-prototype]
size_t strlcat(dst, src, siz)
           ^
1 error generated.

(cherry picked from commit 41ca206fdf)
4 years ago
Victor Julien 0a17103ab8 src: fix unused-but-set-variable compile warnings
Tested on Fedora 37 with clang 15.

datasets.c:852:9: error: variable 'n' set but not used [-Werror,-Wunused-but-set-variable]
    int n = 0;
        ^
1 error generated.

(cherry picked from commit 1bc6976a06)
4 years ago
Victor Julien 9f6862cb50 src: fix strict-prototype warnings
Tested on Fedora 37 with clang 15.

app-layer.c:1055:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerSetupCounters()
                          ^
                           void
app-layer.c:1176:29: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerDeSetupCounters()
                            ^
                             void
2 errors generated.

(cherry picked from commit ebd8728219)
4 years ago
Philippe Antoine 5c0bb7a616 ftp: completely resets port_line
In the case port_line is first allocated and port_line_len is set,
Then a second request reaches memcap and frees port_line,
port_line_len should also be reset, because both will get used
by the response parsing.

Ticket: #5701
(cherry picked from commit 1660172a8b)
4 years ago
Philippe Antoine 462366d447 tls: fix off by one in supported versions extension
Ticket: #5663
(cherry picked from commit b5147189ae)
4 years ago
Victor Julien b24ae521a2 ssl: add debug validation checks for recent changes
Make sure the assumptions are correct.

(cherry picked from commit 8f0d820218)
4 years ago
Philippe Antoine 97f97dd72a tls: fix buffer overread
Recently introduced by commit 4bab6e24e5

Ticket: #5564
(cherry picked from commit 3fbcba8ab7)
4 years ago
Philippe Antoine 9305538309 ssl: fix compiler warning
implicit conversion loses integer precision: 'int' to 'uint16_t'
because C shifts << translates automatically to signed integers

(cherry picked from commit ae6abd8ca3)
4 years ago
Victor Julien f0d9cc5a5e tls: handle incomplete header sooner
Make sure to exit the parser early on incomplete header data.

Additionally, make sure to not create duplicated tls frames in this
case.

Add a debug validation check for the header parser parsing too much
data, which should never happen.

(cherry picked from commit 1701a6b14c)
4 years ago
Victor Julien 541945f4a8 ssl: add debug validation check for incomplete api
(cherry picked from commit 9053c49178)
4 years ago
Victor Julien 29740e1c0c tls: improve record checks
Improve unknown record handling. Inspired by Wireshark 'unknown record'
handling, we take a best effort approach for records with unknown content
types in TLS versions 1.0, 1.1 and 1.2.

Improve record length check and set 'invalid_record_length' event instead
of 'invalid_tls_header'.

(cherry picked from commit 69be41b241)
4 years ago
Victor Julien dfc332fe49 tls: improve versions extension logic
Skip over unusable versions like GREASE.

(cherry picked from commit c028800ae1)
4 years ago
Victor Julien 43a3a7ee39 tls: make version and size checks stricter
This way bad records won't buffer lots of stream data.

(cherry picked from commit 599791fa33)
4 years ago
Victor Julien 5ef12461cf tls: store cert data in heap buffer
Cert chain is a list of pointers into this buffer, so can't use a
stream slice approach.

(cherry picked from commit c73d812026)
4 years ago
Victor Julien 5e834aca0f tls: prepare for client cert parsing
(cherry picked from commit 10f5e6cb66)
4 years ago
Victor Julien 4427809994 tls: make cert handling more generic
In preparation for client cert handling.

(cherry picked from commit 14b2e04b58)
4 years ago
Victor Julien 0f987a99dd tls: avoid tls.invalid_handshake_message FP
Don't set TLS_DECODER_EVENT_INVALID_HANDSHAKE_MESSAGE event on encrypted
handshake messages.

(cherry picked from commit cf4c201acb)
4 years ago
Victor Julien d787558b0f tls: don't set 2 events for a single exception
Keep the more specific ones.

(cherry picked from commit e9d63f3355)
4 years ago
Victor Julien 75a129f124 tls: remove incomplete tests
These tests are incompatible with the incomplete API usage and should
have been pcap based tests in the first place.

(cherry picked from commit 214e466b29)
4 years ago
Victor Julien 2870809f4e tls: set event if record size exceeds limit
(cherry picked from commit 82e03bd8fc)
4 years ago
Victor Julien 5551388d90 tls: support server hello done message
(cherry picked from commit e857c864ca)
4 years ago
Victor Julien 7be7f32a3d tls: support handshake fragmentation
Implement TLS record defrag for handshake messages.

Ticket: #5183.
(cherry picked from commit 4bab6e24e5)
4 years ago