Commit Graph

10509 Commits (56541c7c37d0de3f4c7eaafc19118ad837bbbc3d)

Author SHA1 Message Date
Victor Julien d2375ee7d2 detect/sigorder: remove unused struct fields
(cherry picked from commit 03e0a60f96)
3 years ago
Victor Julien 47b25329aa detect: fix scan-build warnings
detect-engine-address.c:1140:17: warning: Use of memory after it is freed [unix.Malloc]
            r = DetectAddressCmp(ag, ag2);
                ^~~~~~~~~~~~~~~~~~~~~~~~~
detect-engine-address.c:1169:17: warning: Use of memory after it is freed [unix.Malloc]
            r = DetectAddressCmp(ag, ag2);
                ^~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings generated.

detect-engine-port.c:1161:9: warning: Use of memory after it is freed [unix.Malloc]
        DetectPortPrint(ag2);
        ^~~~~~~~~~~~~~~~~~~~
1 warning generated.

Bug: #3150.
Bug: #3151.
(cherry picked from commit 000064de7d)
3 years ago
Victor Julien 37925fe2bd mpm/ac-bs: work around scan-build warnings
util-mpm-ac-bs.c:482:32: warning: Result of 'malloc' is converted to a pointer of type 'uint16_t[256]', which is incompatible with sizeof operand type 'uint16_t' [unix.MallocSizeof]
        ctx->state_table_u16 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
util-mpm-ac-bs.c:524:32: warning: Result of 'malloc' is converted to a pointer of type 'uint32_t[256]', which is incompatible with sizeof operand type 'uint32_t' [unix.MallocSizeof]
        ctx->state_table_u32 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
2 warnings generated.

Bug: #3148.
(cherry picked from commit c8694634af)
3 years ago
Victor Julien 763833c217 mpm/ac: work around scan-build warnings
util-mpm-ac.c:531:32: warning: Result of 'malloc' is converted to a pointer of type 'uint16_t[256]', which is incompatible with sizeof operand type 'uint16_t' [unix.MallocSizeof]
        ctx->state_table_u16 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
util-mpm-ac.c:575:32: warning: Result of 'malloc' is converted to a pointer of type 'uint32_t[256]', which is incompatible with sizeof operand type 'uint32_t' [unix.MallocSizeof]
        ctx->state_table_u32 = SCMalloc(ctx->state_count *
                               ^~~~~~~~
./util-mem.h:35:18: note: expanded from macro 'SCMalloc'
 #define SCMalloc malloc
                 ^~~~~~
2 warnings generated.

Bug: #3148.
(cherry picked from commit ee683a7074)
3 years ago
Victor Julien 77e60e3368 mpm/hs: fix scan-build warning
util-mpm-hs.c:340:20: warning: Potential leak of memory pointed to by 'p' [unix.Malloc]
        p->sids[0] = sid;
        ~~~~~~~~~~~^~~~~
1 warning generated.

Incorrect error handling could lead to a memory leak.

(cherry picked from commit ec84ba1a3c)
3 years ago
Victor Julien 333021e670 decode: suppress scan-build warning
(cherry picked from commit 59ca5cc655)
3 years ago
Victor Julien 16cadd7e59 suricata: work around scan-build warnings
suricata.c:691:17: warning: Value stored to 'bits' during its initialization is never read [deadcode.DeadStores]
    const char *bits = "<unknown>-bits";
                ^~~~   ~~~~~~~~~~~~~~~~
suricata.c:692:17: warning: Value stored to 'endian' during its initialization is never read [deadcode.DeadStores]
    const char *endian = "<unknown>-endian";
                ^~~~~~   ~~~~~~~~~~~~~~~~~~
2 warnings generated.

(cherry picked from commit c13a6fa73c)
3 years ago
Victor Julien 27ef9a96af scan-build: treat as debug validate
(cherry picked from commit bcf8187443)
3 years ago
Jason Ish 00a9edae3e detect-fast-pattern: remove unused var alertcnt
Set, but never read.

Caught by -Wunused-but-set-variable which is enabled in the newer
Fedora CI builders.
3 years ago
Justin Azoff a179f79eb6 detect/iponly: remove DetectEngineIPOnlyThreadCtx
This is unused.

Issue: 4578
(cherry picked from commit aacb7dc291)
3 years ago
Justin Azoff 03e767536a detect/iponly: Reduce the size of the SigNumArray bitsets
Instead of tracking ip only rules by the internal signum, track them by
a separate counter that starts at zero.  This results in dense
SigNumArrays instead of sparse ones and a much smaller max_idx.

Issue: 4578
(cherry picked from commit dfbc3da0eb)
3 years ago
Victor Julien f2cf58eec7 detect/content: add negated endswith test
(cherry picked from commit a42c225117)
3 years ago
Jeff Lucovsky 0a9b7d0528 detect/content: Negated endswith matches
Issue: 5541

This commit handles negated endswith matches.

(cherry picked from commit c083cbda33)
3 years ago
Jeff Lucovsky a7fbfd572c detect/byte_math: fix bug in byte_math detection
Issue: 5945

Avoid division by zero when the byte_math operation is division and the
rvalue is 0.

(cherry picked from commit 38c5e89e29)
3 years ago
Shivani Bhardwaj 92d72344fb smtp: enforce line limit even when LF is found
Before:
If LF character was found, so far, we won't enforce the line limit on
the line. We only enforced limits in case of LF character missing in a
long line.

After this patch:
Line limit is enforced on the line if it is bigger than 4096 Bytes
irrespective of whether LF was found or not.

Redmine Bug: 5819

(cherry picked from commit 5f52b199ff)
3 years ago
Shivani Bhardwaj 08fd66718a smtp: reset current line len at start 3 years ago
Shivani Bhardwaj 564c7793a5 util/mime: allow delim len 0 when line limit is hit
(cherry picked from commit fd4e0fbafe)
3 years ago
Shivani Bhardwaj 707dc705d9 smtp: move constant declaration to header
(cherry picked from commit c0bff5f921)
3 years ago
Victor Julien 80abc3121d pcap: improve pcap_breakloop support
When pcap_breakloop has been issued on a handle, the current pcap_dispatch
call may return -2 (PCAP_ERROR_BREAK), but it can also return the number
of processed packets if lower than the desired number. So add this condition
as a check.

(cherry picked from commit 9fe08f2374)
(cherry picked from commit 2ddd26446e)
3 years ago
Victor Julien 6d8b50b748 detect: only breakloop threads that are lagging
Sleep after all threads have been checked.

Bug: #5969.
(cherry picked from commit 8a968faa04)
3 years ago
Victor Julien 6658300c84 flow/worker: refresh detect thread during housekeeping
During housekeeping multiple flows are processed. If a rule reload happens
at that time, we need to use the new detect thread as soon as possible.

Bug: #5969.
(cherry picked from commit 5e4cf182ab)
3 years ago
Philippe Antoine 749277d46f http: complete multipart until request.body-limit
In the case we are truncating a multipart file because of reaching
request.body-limit, we used to not consume the whole buffer, but
keep expected_boundary_len bytes in case a new boundary begins
in these bytes.
Even if we cannot check the complete boundary, we can still check
the first bytes, as will be done in the rust version.

Ticket: #5952
(cherry picked from commit 578f328e06)
(cherry picked from commit caf9940fd1)
3 years ago
Victor Julien 2c600336ab stream: improve FIN checking
After recent next_seq changes, the FIN checks could be too strict
leading to stalling sessions in IPS mode.

This patch requires a FIN to be >= last ack and <= next_win to be
accepted.

(cherry picked from commit 39a6f411e9)
3 years ago
Shivani Bhardwaj 3442b6da9d util/base64: don't reset decoded bytes in RFC4648
Old behavior:
With RFC4648, the decoded bytes were reset to 0 in case an unusual
character was encountered in the encoded string. This worked out fine
for small test cases where there weren't many bytes to be decoded.

Problem:
If a big encoded string had a character outside of the base alphabet,
the processing would stop and the number of decoded bytes were set to 0.
However, even though the processing should stop at the invalid
character, the number of decoded bytes should correctly store the bytes
decoded up until the point an invalid characted was encountered.

New behavor:
For any base64 encoded string given to the base64 decoder in RFC4648
mode, we make sure that the number of decoded bytes correctly reflect
the number of bytes processed up until the string was valid. This makes
sure any further calculations/use of the decoded data is done correctly.

Redmine ticket: 5885

(cherry picked from commit 418ddba38e)
3 years ago
Philippe Antoine 3851504638 enip: optimized tx iterator
As for SMTP, having a linked list.

Ticket: #5927
(cherry picked from commit 4f7426fdcf)
3 years ago
Philippe Antoine ff2e02844b dnp3: optimized tx iterator
As for SMTP, having a linked list.

Ticket: #5927
(cherry picked from commit e15daf6a4b)
3 years ago
Philippe Antoine 8fd0862f94 smtp: optimized tx iterator
To be more efficient with larger number of transactions.
As was done for FTP.

Ticket: #5927
(cherry picked from commit f5f215dae7)
3 years ago
Eric Leblond ce12cb85d7 enip: add TX orientation
Set no inspection in the opposite side of the transaction.

Ticket: #5799
3 years ago
Eric Leblond 4a3fdedc1f app-layer: add flag to skip detection on TX
Stamus team did discover a problem were a signature can shadow
other signatures.

For example, on a PCAP only containing Kerberos protocol and where the
following signature is matching:

alert krb5 $HOME_NET any -> any any (msg:"krb match"; krb5_cname; content:"marlo"; sid:3; rev:1;)

If we add the following signature to the list of signature

alert ssh $HOME_NET any -> any any (msg:"rr"; content:"rr"; flow:established,to_server; sid:4; rev:2;)

Then the Kerberos signature is not matching anymore.

To understand this case, we need some information:

- The krb5_cname is a to_client keyword
- The signal on ssh is to_server
- Kerberos has unidirectional transaction
- kerberos application state progress is a function always returning 1

As the two signatures are in opposite side, they end up in separate
sig group head.

Another fact is that, in the PCAP, the to_server side of the session
is sent first to the detection. It thus hit the sig group head of
the SSH signature. When Suricata runs detection in this direction
the Kerberos application layer send the transaction as it is existing
and because the alstate progress function just return 1 if the transaction
exists. So Suricata runs DetectRunTx() and stops when it sees that
sgh->tx_engines is NULL.

But the transaction is consumed by the engine as it has been evaluated
in one direction and the kerberos transaction are unidirectional so
there is no need to continue looking at it.

This results in no matching of the kerberos signature as the match
should occur in the evaluation of the other side but the transaction
with the data is already seen has been handled.

This problem was discovered on this Kerberos signature but all
the application layer with unidirectional transaction are impacted.

This patch introduces a flag that can be used by application layer
to signal that the TX should not be inspected. By using this flag
on the directional detect_flags_[ts|tc] the application layer can
prevent the TX to be consumed in the wrong direction.

Application layers with unidirectional TX will be updated
in separate commits to set the flag on the direction opposite
to the one they are.

Ticket: #5799
3 years ago
Eric Leblond 5c1995a78f detect: remove STREAM_FLUSH
It is unused in the code so can be removed.

Ticket: #5799
3 years ago
Eric Leblond e1b02a08ce app-layer-parser: give direction to progress func
The tx progress functions are expecting a direction and were given
a flow flags. As a result, they were not reporting correctly the
status if a DetectRunScratchPad flow_flags was containing some other
bits in the flag.

One case was when a signature was alterating the stream analysis
and triggering the addition of the STREAM_FLUSH flags.

The consequences are quite severe as the transactions are pilling
up waiting to be inspected causing sometimes a 10x performance hit
on pcap parsing. Also as the inspection was not done, Suricata is
missing a part of the alerts.

This was discovered when working on the following set of signatures:

alert ssh $HOME_NET any -> any any (msg:"pcre without content"; pcre:"/rabbit/"; sid:1; rev:1;)
alert smb $HOME_NET any -> any any (msg:"smb share content"; smb.share; content:"C"; sid:2; rev:1;)

When the first one is present the second is not triggering even
though the pcap file had no ssh inside. This is due to the fact
that the ssh signature was triggering the STREAM_FLUSH flag to
be set on the flowflags of the packet. But the application
layer will ask the smb state progress via

r = alp_ctx.ctxs[FlowGetProtoMapping(ipproto)][alproto].
        StateGetProgress(alstate, flags);

passing it the flow flags but the smb function is expecting
a direction so we end up in a unplanned case

pub unsafe extern "C" fn rs_smb_tx_get_alstate_progress(tx: *mut ffi::c_void,
                                                  direction: u8)
...
if direction == Direction::ToServer as u8 && tx.request_done {

This leads the signature to not be evaluated correctly.

Ticket: #5799
3 years ago
Juliana Fajardini e9da3378ca exception/policy: use pkt action if no flow support
Defrag memcap and flow memcap do not support flow action for the
exception policies, as there is no flow when the exception condition is
hit. In such cases, the exception policy must be considered for the
packet only, when that makes sense, or should be ignored, in case of
`bypass`.

Bug #5940

(cherry picked from commit d4333fb959)
4 years ago
Philippe Antoine c64261f374 detect: bump detect engine version for tenant reload
Because the engine version is used to free the old
variables and not the new ones.
As is done in DetectEngineReload.

Ticket: #5866
(cherry picked from commit d313b5d605)
4 years ago
Philippe Antoine 5dcf7f80df detect: bytemath do not left shift more than 64
As it is undefined behavior by C standard.
In this case, zeroes the value.

Ticket: #5900
(cherry picked from commit 473ca6dcf4)
4 years ago
Victor Julien 6ee3e79849 stream: accept and flag ack of ZWP data
Tcp Zero Window Probes try to send a single byte payload to "probe" if
the window has reopened. This single byte is, if accepted, not retransmitted.

(cherry picked from commit 30a716a4ab)
4 years ago
Victor Julien ad1efb8e53 stream: harden tcp reuse check against RST/FIN
(cherry picked from commit 64fb4066cf)
4 years ago
Victor Julien 516ddf4fec stream: improve SYN and SYN/ACK handling with ECN/CWR flags
(cherry picked from commit 0d1d288544)
4 years ago
Victor Julien 73ccd0de1d stream: fix TFO overlap detection with ECN/CWR flags
(cherry picked from commit 5fe2fba184)
4 years ago
Victor Julien 85fc446c55 stream: turn session flags into u32
Due to gaps/padding the size of the struct won't change.

(cherry picked from commit aa7d58b0c7)
4 years ago
Juliana Fajardini 7bfe59f936 flow/manager: apply clang formatting changes 4 years ago
Juliana Fajardini 2832b4a2ae flow/manager: fix coverity divide_by_zero warning
Updated all cases where flow_config.prealloc was used in a division.

*** CID 1524506:  Integer handling issues  (DIVIDE_BY_ZERO)
/src/flow-manager.c: 858 in FlowManager()
852                                "flow_spare_q status: %" PRIu32 "%% flows at the queue",
853                             spare_pool_len, flow_config.prealloc,
854                             spare_pool_len * 100 / flow_config.prealloc);
855
856                     /* only if we have pruned this "emergency_recovery" percentage
857                      * of flows, we will unset the emergency bit */
>>>     CID 1524506:  Integer handling issues  (DIVIDE_BY_ZERO)
>>>     In expression "spare_pool_len * 100U / flow_config.prealloc", division by expression "flow_config.prealloc" which may be zero has undefined behavior.
858                     if (spare_pool_len * 100 / flow_config.prealloc > flow_config.emergency_recovery) {
859                         emerg_over_cnt++;
860                     } else {
861                         emerg_over_cnt = 0;
862                     }

Related to
Bug #5919

(cherry picked from commit 754d2803dd)
4 years ago
Juliana Fajardini f2e863bfe8 flow/manager: fix prealloc unhandled division by 0
If flow.prealloc was set to zero in our yaml config, this led to
a floating point exception in the flow manager.

Bug: #5919.
(cherry picked from commit 5baa6c0024)
4 years ago
Shivani Bhardwaj 2e2483d27a util/mime: use uint8_t for cnt
cnt should not exceed B64_BLOCK which is 4, so, it is unnecessary to use
uint32_t for the variable.
4 years ago
Shivani Bhardwaj 082c7033ad util/mime: use uint32_t for consumed bytes
In a case of the line buffer being over 255 bytes, the consumed bytes
would reset to 0 as it was uint8_t. Fix this integer overflow by setting
the type to uint32_t.

Redmine ticket: 5883

(cherry picked from commit c089bbb7d7)
4 years ago
Jeff Lucovsky 6a66157e34 netmap/config: New API display during debug
This PR changes the log level associated with the message indicating
if the new Netmap API (v14+) is in use during module registration.

Issue: 5879
4 years ago
Bill Meeks 4835befec5 netmap: packet stall
- Fix packet processing stall under high load when using netmap in IPS mode.
- Detect and generate Fatal Error exit for rare case when hardware NIC exposes
unmatched RX/TX queue counts. This is rare, but would result in some traffic
bypassing Suricata since it assumes NIC queue counts are symmetrical.
- Fix instance of missing unlock call for netmap device list when exiting due
to an error condition.
- Clean up existing code comments and add additional ones to better document
the new netmap v14 API code.
4 years ago
Victor Julien 517132b6ad eve/drop: don't log drops unless packet is dropped
In pass/drop combinations where the pass rule took precendence over
the drop, a "drop" false positive could still be logged due to the
storing of the drop record in the packet drop alert store.

Bug: #5867.
(cherry picked from commit 09348564f0)
4 years ago
Victor Julien 7838fc8b82 detect/urilen: fix applying urilen as depth
If urilen induced depth was set, later DetectContentPropagateLimits()
would apply a wrong depth setting, leading to a false negative in
some cases.

Bug: #5929.
(cherry picked from commit ba7db2583b)
4 years ago
Victor Julien a48e0229ae stream: SYN queue support
Support case where there are multiple SYN retransmits, where
each has a new timestamp.

Before this patch, Suricata would only accept a SYN/ACK that
matches the last timestamp. However, observed behavior is that
the server may choose to only respond to the first. In IPS mode
this could lead to a connection timing out as Suricata drops
the SYN/ACK it considers wrong, and the server continues to
retransmit it.

This patch reuses the SYN/ACK queuing logic to keep a list
of SYN packets and their window, timestamp, wscale and sackok
settings. Then when the SYN/ACK arrives, it is first evaluated
against the normal session state. But if it fails due to a
timestamp mismatch, it will look for queued SYN's and see if
any of them match the timestamp. If one does, the ssn is updated
to use that SYN and the SYN/ACK is accepted.

Bug: #5856.
(cherry picked from commit 7bfee147ef)
4 years ago
Victor Julien 66ddbc3a8b stream: move state queue code into util func
(cherry picked from commit 288086af3f)
4 years ago
Victor Julien 7ccec8f143 debug: add bool string print helper macro
(cherry picked from commit 0ec136621d)
4 years ago
Victor Julien 1ed8066d4e stream: support SYN/ACK with TFO only ack'ing ISN
Not ack'ing the data.

(cherry picked from commit 7ef57cc7cb)
4 years ago
Victor Julien 7f4a800156 stream: add liberal timetamps option
Linux is slightly more permissive wrt timestamps than many
other OS'. To avoid many events/issues with linux hosts, add an
option to allow for this slightly more permissive behavior.

Ideally the host-os config would be used, but in practice this
setting is rarely set up correctly, if at all.

This option is enabled by default.

(cherry picked from commit 01b7ccc224)
4 years ago
Victor Julien fee655bae6 stream: D-SACK detection and logging
RFC 2883 specifies a special use of SACKs to indicate a host has
received a segment it considers a spurious retransmission.

(cherry picked from commit d79a926085)
4 years ago
Victor Julien dcefc00b17 stream: fix next_seq updates after temporary gap
On every accepted packet in established state, update next_seq if
packet seq+len is larger than existing next_seq. This allows it to
catch up after large gaps that are filled again a bit later.

Bug: #5877.
(cherry picked from commit 76225bf9ac)
4 years ago
Victor Julien 006911fd5a stream: improve first FIN next_seq update
For accepted FIN packets, always update next_seq. This helps track the
FIN sequence.

Bug: #5877.
(cherry picked from commit 551fb80150)
4 years ago
Victor Julien a8cff36467 stream: improve FIN next_seq handling
Update next_seq to SEQ + payload_len + 1, so retransmission checks
work better.

Bug: #5877.
(cherry picked from commit 80a012a787)
4 years ago
Victor Julien f1904398a7 stream: set next_seq before last_ack
next_seq sometimes depends on last_ack in cases of packet loss
catch up, so first update it.

Bug: #5877.
(cherry picked from commit 20df715e64)
4 years ago
Eric Leblond 984698f8c5 stream/tcp: update ack handling logic
Only update the ack value of a session for regular packets when
the ACK bit is set.

(cherry picked from commit 0d81173d6e)
4 years ago
Victor Julien 237e06a991 stream/tcp: fix wrong ACK trigger FIN1 to FIN2
An ACK that ACK'd older data while still being in-window could
lead to FIN_WAIT1 to FIN_WAIT2 state transition. Detect this
case and generally harden the check.

Bug: #5877.
(cherry picked from commit 67af94f2e0)
4 years ago
Victor Julien 2b18412509 stream: fix overlap detection
If a in-tree segment was partly before base_seq the overlap detection
miscalculated the data offset. This lead to memcmp comparing the wrong
data.

Bug: #5881.

(cherry picked from commit cbcd70537a)
4 years ago
Victor Julien d0b1a04eb0 stream: set event on bad timestamp on syn_sent state
(cherry picked from commit fc376f8145)
4 years ago
Victor Julien 68bd193868 stream/tcp: allow tcp session reuse on null sessions
When a "stream starter" packet finds an existing TCP flow, the flow will be
evaluated for reuse.

The following scenario wasn't handled well:

1. Suricata starts after a tool has just stopped using lots of connections
   (e.g. ab stress testing a webserver)
2. even though the client is closed already, the server is still doing
   connection cleanup sending many FINs and later RSTs
3. Suricata creates flows for these packets, but no TCP sessions
4. client resumes testing, creating flows that have the same 5 tuple as the
   flows created for the FIN/RST packets
5. Suricata refuses to "reuse" the flows as the condition "tcp flow w/o session"
   is not considered valid for session reuse
6. new TCP connection is not properly tracked and evaluated in parsing and
   detection

There may be other vectors into this, like a flow w/o session because of
memcap issues.

Bug: #5843.
(cherry picked from commit 93c9594dd8)
4 years ago
Victor Julien d13bb7f5a7 flowworker: don't keep unnecessary flow reference
Flow stream/detect/log flush packets, don't hold on to the flow
beyond the flow worker module.

(cherry picked from commit a2dc9a40e7)
4 years ago
Victor Julien c08521cd52 threshold: suppress debug notice message 4 years ago
Victor Julien 777d88bd21 decode/events: add strict mode for udp.hlen_invalid; remove rule 4 years ago
Shivani Bhardwaj eb26d1a64d decode/udp: fix payload_len calculation
Fix payload_len calculation post removal of the condition that returned
error code if the length to the decode fn did not match the length of
header from the UDP packet.

Bug 5379

(cherry picked from commit f941ceae2b)
4 years ago
Shivani Bhardwaj bbd53bcdf6 decode/events: add event type UDP_LEN_INVALID
(cherry picked from commit eebdfe9a3e)
4 years ago
Lukas Sismis f6fa16ce25 decode-udp: Allow shorter UDP packets than the remaining payload length
If the packet is shorter than IP payload length we no longer flag it as an
invalid UDP packet. UDP packet can be therefore shorter than IP payload.
Keyword "udp.hlen_invalid" became outdated as we no longer flag short UDP
packets as invalid. The keyword's evaluation remains the same.

Inspired by commit: d18e52ed93

Redmine ticket: #5793
4 years ago
Philippe Antoine fd358a5616 fuzz: checks pcap timestamp consistency
Inspired by commit e22b59b8c1

Master7 is now diverging with use of SCTime_t
4 years ago
Philippe Antoine 4b2a59a4d9 detect: config keyword transaction logic fix
When the keyword config:logging disable,type tx is used,
OutputTxLog checks a flag to skip the transaction without logging
it, but AppLayerParserTransactionsCleanup waits for the
transaction to be marked as logged to clean it.

So, OutputTxLog now marks the tx as logged, so that it can
get cleaned away.

Ticket: #5456
(cherry picked from commit 0c0fcc9411)
4 years ago
Juliana Fajardini d47e8fffad stream/tcp: fix typos, update copyright year
Bug #5765
4 years ago
Juliana Fajardini 700b0a37a1 exceptions: ignore policy if stream.midstream=true
Set the engine to ignore the stream.midstream-policy if stream.midstream
is enabled.

If we had both stream.midstream AND stream.midstream_policy enabled,
this could lead to midstream flows being dropped (or bypassed, or...)
instead of being accepted by the engine, as it was probably meant when
the user enabled midstream flows.

Bug #5765

(cherry picked from commit 8e9905e0d8)
4 years ago
Philippe Antoine 7986919df0 fuzz: specify protocol with fuzz target name
cf https://redmine.openinfosecfoundation.org/issues/4125

This allows fuzz_applayerparser_parse to fuzz one specific
app-layer protocol based on the binary name, as is done
with the environment variable FUZZ_APPLAYER
That is if we rename/copy to fuzz_applayerparser_parse_smb,
it will fuzz only SMB protocol
This way, we can easily produce different fuzz targets for
each protocol in oss-fuzz

(cherry picked from commit e9b76a0e66)
4 years ago
Jeff Lucovsky 2f8f45ff6a netmap: Fixup issues with v14+ backport
This commit reduces the changes associated with adding the v14 api to
6.0.x

During the preparation of this commit, issues in the original backport
were corrected
- Failure to release a lock under error conditions
- Typo in an CPP ifdef
- Incorrect target for goto statement in an error handling case.

Issue: 5744
4 years ago
Philippe Antoine 2c49a268e2 test: do not output non ascii character
The unit test for content |aa bz| transforms in place the string
str to replace the 2 characters aa by one character 0xaa
Then, when z is not recognized as a valid hexadeicmal character,
the whole modified string is printed out, inclusing the non-ascii
0xaa

Ticket: #5558
(cherry picked from commit b281199e9a)
4 years ago
Victor Julien 26fc02227b streaming/sbb: fix debug message
(cherry picked from commit 989e5c7ba2)
4 years ago
Victor Julien eb03cdea85 stream: remove bug on from packet path
(cherry picked from commit a00db15bd4)
4 years ago
Victor Julien 9664d97129 src: fix deprecated-non-prototype compile warnings
Tested on Fedora 37 with clang 15.

util-strlcatu.c:45:8: error: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Werror,-Wdeprecated-non-prototype]
size_t strlcat(dst, src, siz)
           ^
1 error generated.

(cherry picked from commit 41ca206fdf)
4 years ago
Victor Julien 0a17103ab8 src: fix unused-but-set-variable compile warnings
Tested on Fedora 37 with clang 15.

datasets.c:852:9: error: variable 'n' set but not used [-Werror,-Wunused-but-set-variable]
    int n = 0;
        ^
1 error generated.

(cherry picked from commit 1bc6976a06)
4 years ago
Victor Julien 9f6862cb50 src: fix strict-prototype warnings
Tested on Fedora 37 with clang 15.

app-layer.c:1055:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerSetupCounters()
                          ^
                           void
app-layer.c:1176:29: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerDeSetupCounters()
                            ^
                             void
2 errors generated.

(cherry picked from commit ebd8728219)
4 years ago
Philippe Antoine 5c0bb7a616 ftp: completely resets port_line
In the case port_line is first allocated and port_line_len is set,
Then a second request reaches memcap and frees port_line,
port_line_len should also be reset, because both will get used
by the response parsing.

Ticket: #5701
(cherry picked from commit 1660172a8b)
4 years ago
Philippe Antoine 462366d447 tls: fix off by one in supported versions extension
Ticket: #5663
(cherry picked from commit b5147189ae)
4 years ago
Victor Julien b24ae521a2 ssl: add debug validation checks for recent changes
Make sure the assumptions are correct.

(cherry picked from commit 8f0d820218)
4 years ago
Philippe Antoine 97f97dd72a tls: fix buffer overread
Recently introduced by commit 4bab6e24e5

Ticket: #5564
(cherry picked from commit 3fbcba8ab7)
4 years ago
Philippe Antoine 9305538309 ssl: fix compiler warning
implicit conversion loses integer precision: 'int' to 'uint16_t'
because C shifts << translates automatically to signed integers

(cherry picked from commit ae6abd8ca3)
4 years ago
Victor Julien f0d9cc5a5e tls: handle incomplete header sooner
Make sure to exit the parser early on incomplete header data.

Additionally, make sure to not create duplicated tls frames in this
case.

Add a debug validation check for the header parser parsing too much
data, which should never happen.

(cherry picked from commit 1701a6b14c)
4 years ago
Victor Julien 541945f4a8 ssl: add debug validation check for incomplete api
(cherry picked from commit 9053c49178)
4 years ago
Victor Julien 29740e1c0c tls: improve record checks
Improve unknown record handling. Inspired by Wireshark 'unknown record'
handling, we take a best effort approach for records with unknown content
types in TLS versions 1.0, 1.1 and 1.2.

Improve record length check and set 'invalid_record_length' event instead
of 'invalid_tls_header'.

(cherry picked from commit 69be41b241)
4 years ago
Victor Julien dfc332fe49 tls: improve versions extension logic
Skip over unusable versions like GREASE.

(cherry picked from commit c028800ae1)
4 years ago
Victor Julien 43a3a7ee39 tls: make version and size checks stricter
This way bad records won't buffer lots of stream data.

(cherry picked from commit 599791fa33)
4 years ago
Victor Julien 5ef12461cf tls: store cert data in heap buffer
Cert chain is a list of pointers into this buffer, so can't use a
stream slice approach.

(cherry picked from commit c73d812026)
4 years ago
Victor Julien 5e834aca0f tls: prepare for client cert parsing
(cherry picked from commit 10f5e6cb66)
4 years ago
Victor Julien 4427809994 tls: make cert handling more generic
In preparation for client cert handling.

(cherry picked from commit 14b2e04b58)
4 years ago
Victor Julien 0f987a99dd tls: avoid tls.invalid_handshake_message FP
Don't set TLS_DECODER_EVENT_INVALID_HANDSHAKE_MESSAGE event on encrypted
handshake messages.

(cherry picked from commit cf4c201acb)
4 years ago
Victor Julien d787558b0f tls: don't set 2 events for a single exception
Keep the more specific ones.

(cherry picked from commit e9d63f3355)
4 years ago
Victor Julien 75a129f124 tls: remove incomplete tests
These tests are incompatible with the incomplete API usage and should
have been pcap based tests in the first place.

(cherry picked from commit 214e466b29)
4 years ago
Victor Julien 2870809f4e tls: set event if record size exceeds limit
(cherry picked from commit 82e03bd8fc)
4 years ago
Victor Julien 5551388d90 tls: support server hello done message
(cherry picked from commit e857c864ca)
4 years ago
Victor Julien 7be7f32a3d tls: support handshake fragmentation
Implement TLS record defrag for handshake messages.

Ticket: #5183.
(cherry picked from commit 4bab6e24e5)
4 years ago
Victor Julien 0489987afd tls: remove certificate buffering code
TCP Buffering is now done in the app-layer using the incomplete API, on
the SSL/TLS record level. TLS level fragmentation will be implemented
separately.

(cherry picked from commit bcaf0f6f7d)
4 years ago
Victor Julien 3785e03426 tls: parse handshake protocol records in single pass
(cherry picked from commit 0839317ea7)
4 years ago
Victor Julien b3bbac5eae sslv2: use version from client hello
Remove streaming code that is now unused.

Incomplete handling makes this record parsing work on full data.

(cherry picked from commit 9f0ea5e70c)
4 years ago
Victor Julien 253cdb9f8a ssl: implement 'incomplete' handling for SSLv2
(cherry picked from commit c8d79fb81f)
4 years ago
Victor Julien ba70d7b91a tls: streaming mode for application records
To avoid overhead of stream buffering for records we don't do
much with anyway, pass through application records instead of
buffering the entire record in the stream engine.

(cherry picked from commit 6076a51511)
4 years ago
Victor Julien 2651110e0a tls: use incomplete API to get full TLS records
The TLS record header is parsed in streaming mode still, but once the
record size is known we tell the app-layer API to give us the full
record.

Ticket: #5481

(cherry picked from commit 129fcb5c72)
4 years ago
Jeff Lucovsky ad8147a8c1 detect/tls: Improve tls.fingerprint rule handling
Issue: 4581

This commit improves the runtime performance of rules with
tls.fingerprint by using the inspection logic from tls.cert_fingerprint.

(cherry picked from commit 63745a7879)
4 years ago
Jeff Lucovsky 2ca1b7f39a detect/uri: Remove unnecessary include
This commit removes an unnecessary #include for detect-uricontent.h

(cherry picked from commit 6bccd5aa30)
4 years ago
Victor Julien 4d5041f82f detect/pcre: don't use JIT if disabled
If we determined not to use JIT at start up, really don't use
JIT.

Bug: #5762.
4 years ago
Sascha Steinbiss 5ccdded6ca ebpf: update deprecated API calls
This fixes build errors when libbpf 1.0 is used. It removes previously
deprecated API functions that were still in use in Suricata's eBPF
code.
4 years ago
Victor Julien 44a2e59cb2 radix: fix ipv6 address parsing warning
The check meant to see if the ip address part of the ip/cidr combo
was more specific than needed wasn't fully implemented, leading to
warnings being issued on completely valid and correct input.

This patch implements the same logic as in IPv4. If the ip address
as specified is different from the ip after the mask has been applied,
a warning is displayed.

Bug: #5747.
(cherry picked from commit 991f9fde32)
4 years ago
Philippe Antoine fb6f1b184c mime: do not skip empty lines for quoted-printable
As these lines are in the file...

Ticket: #5725
(cherry picked from commit 0e054af1ac)
4 years ago
Victor Julien 453366ccb9 mime/base64: unify space handling
(cherry picked from commit 7bfb63210e)
4 years ago
Philippe Antoine 1be913d49c eve: add common options to loggers missing it
So that we get community id for RFB for instance

Ticket: #5723
4 years ago
Victor Julien 002cd8e301 flow: cleanup and clarify ancient debug messages 4 years ago
Victor Julien 64adb30b5c decode: enforce layer limit through tunnel layers
Bug: #5686.
Bug: #5688.
4 years ago
Shivani Bhardwaj 433266b076 util/base64: fix heap buffer overflow
While updating the destination pointer, we were also adding the padded
bytes which are not a part of the decoded bytes. This led to running out
of space on the destination buffer.
Fix it by only incrementing destination buffer ptr by the number of
actual bytes that were decoded.

Ticket 5623
Ticket 5694
4 years ago
Shivani Bhardwaj 51c4b48f57 util/base64: use padding var for calculations
(cherry picked from commit 58e5033a44)
4 years ago
Shivani Bhardwaj 02377e9c25 base64: add and clean tests
(cherry picked from commit 7005443b8b)
4 years ago
Shivani Bhardwaj 6b2c49292a base64: add new mode as per RFC 4648
As per RFC 4648,
Implementations MUST reject the encoded data if it contains characters
outside the base alphabet when interpreting base-encoded data, unless
the specification referring to this document explicitly states
otherwise.

Add a new mode BASE64_MODE_RFC4648, and handle input strictly as per the
specification.

Bug 5223

(cherry picked from commit dad52f133d)
4 years ago
Victor Julien 2f35376482 smtp/files: don't modify prev file on open failure
(cherry picked from commit 9f4dd4fc56)
4 years ago
Victor Julien 5db0ef4415 files: always initialize inspect_window and min_inspect_depth
This is to make sure the files buffers are properly managed even
when there are no rules or when there are no file.data rules.

Bug: #5703.
(cherry picked from commit e601ebdfd8)
4 years ago
Victor Julien b68b170271 streaming/buffer: set hard limit on buffer size
Don't allow the buffer to grow beyond 1GiB. Add a once per thread
warning if it does reach it.

Bug: #5703.
(cherry picked from commit df7d8d96c9)
4 years ago
Victor Julien 82c24bf738 detect: fixes to action handling; fix PASS
Fix PASS handling by setting and checking in the correct packet.

There are 3 types of packets:
1. tunnel packets (inner layer of encapsulation)
2. "root" packets (outmost layer of encapsulated packet)
3. normal packets (no encapsulation)

Tunnel packet have a pointer to their "root". The "root" is the packet
that is ultimately used by the capture method to issue a verdict:
DROP or ACCEPT (forward).

For tunnels:
DROP actions are always issued on the root packet.
The PASS action is issued on the packet currently in the detection
engine.

Non-tunnels:
DROP and PASS are both set in the current packet.

Bug: #5697.
4 years ago
Philippe Antoine b9be616406 fuzz: fixes a leak in applayerparse target
If a protocol change was requested, the target did not handle
it as Suricata, as the target is meant to handle only one
app-layer protocol.

(cherry picked from commit ca054f7264)
4 years ago
Jason Ish e9a0ac187c afpacket/netmap: warn about mixed ips, ids/tap deprecation
Suricata already logs if AF_PACKET or Netmap are running in a mixed IPS
and IDS/TAP mode.  As the behavior is undefined when these modes are
mixed, it is best to deprecate and to not allow this behavior. For now
warn that it will be unsupported and fail in Suricata 8.

Ticket: 5587
(cherry picked from commit 0c00f28ebc)
4 years ago
Philippe Antoine 0ca12493e3 mime: handles multiple sections for a parameter
Ticket: 4386

as per RFC2231.
For instance filename can be split between filename*0,
filename*1, etc...

(cherry picked from commit 784558df2e)
4 years ago
Philippe Antoine 55781229d7 mime: move FindMimeHeaderTokenRestrict to rust
Also fixes the case where the token name is present
in a value

(cherry picked from commit 8feb9c35ae)
4 years ago
Jeff Lucovsky 1ba259450b netmap: V14 API changes
This commit modifies the Netmap packet handling to use API version 14.

@bmeeks8 contributed many changes instrumental to this effort.

(cherry picked from commit ca7d097225)
4 years ago
Jeff Lucovsky 5d0fe15f69 util: Add sys/ioctl.h to common include
This commit adds another system include file based on autoconf to the
common Suricata include file for convenience.

(cherry picked from commit 3496e543af)
4 years ago
Victor Julien 778faaed96 detect: apply drops to tunnel root
`PacketDrop` applied a DROP to the current packet unconditionally,
while in tunnel/encapsulated cases the DROP should have been applied
to the root packet.

Fixes: 6742ecbc9e ("decode: make PacketDrop use action as parameter")

Bug: #5600.
4 years ago
Philippe Antoine bcd8d7d6c4 threshold: fix regex to accept by_both and by_rule
As is done in detect-threshold.c or in DETECT_RATE_REGEX
and is expected by switch (rule_type) which makes the same
for THRESHOLD_TYPE_THRESHOLD and THRESHOLD_TYPE_RATE

Ticket: #5327
(cherry picked from commit a2f857ed90)
4 years ago
Lukas Sismis afbd68ad22 transversal: remove suricata-ids.org references
(cherry picked from commit e101384e7b)
4 years ago
Eric Leblond 50f0b6acac flow: fix FlowSwapFileFlags function
Ticket: #5625
(cherry picked from commit 4e1ff9a7ff)
4 years ago
Victor Julien a5bc56cecb detect/tag: improve time handling on windows
Bug: #5584.
(cherry picked from commit 0977f40d1c)
4 years ago
Victor Julien 985b93e80f eve: log mac addresses in packet direction
(cherry picked from commit 90f3823cad)
4 years ago
Victor Julien 3570ec6f6e stream: fix reachable assertion
Fix `Flow::thread_id` not always getting properly set up, leading to
a reachable assertion.

Bug #4582.

(cherry picked from commit b06c0579f5)
4 years ago
Juliana Fajardini aafbf368ac stream/tcp: typo fix 4 years ago
Philippe Antoine 047661a5cb source: pcap timestamp microsecond consistency
That is it should be less than 1 000 000.
Have the same for fuzz targets where the bug came from.

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44177
(cherry picked from commit 8ecf7e403e)
4 years ago
Philippe Antoine c31d3592a4 flow/icmpv4: fix vlan.use-for-tracking
For ICMPv4 error messages the vlan ids were always considered,
even if the 'vlan.use-for-tracking' option was disabled.

Ticket: #5330
(cherry picked from commit ce2775d331)
4 years ago
Jason Ish 0ca09ecbcd detect: fix user after free in debug log
Found by gcc 12.2.1.
4 years ago
Victor Julien 7183e766f9 detect/flowbits: more permissive parsing by default
To get strict parsing, `--strict-rule-keywords=flowbits` command line
option can be used.

Bug: #5547.
4 years ago
Jason Ish 8174ba9e6f content: don't error out on incomplete hex
Before 6.0.6 if hex content was incomplete, Suricata didn't error out.
With 6.0.6 incomplete hex was detected and errored on which is a
breaking change in a release branch.  Instead, only emit a warning
unless strict content checking has been requested.

To enable strict behaviour on incomplete content hex in a rule,
"--strict-rule-keywords=content" can be used on the command line.

Issue: #5546
4 years ago
Jason Ish 726de4e70c detect: allow rule errors to display error, but not be fatal
Add a new setup return code, -4 which will print the error, but not
error out in test mode. This is to support allowing incomplete hex, as
there is no context as to which rule had the issue.

For context:
- -3: error ok, no message
- -2: error not ok, no message
- -4: error ok, message
4 years ago
Victor Julien f40ad90adf detect: fix duplicate detect state issue
For protocols with multi buffer inspection there could be multiple times
the same sid would be queued into the candidates queue. This triggered
a debug validation check.

W/o debug validation this would lead to duplicate work and possibly multiple
alerts where a single one would be appropriate.

Bug: 5419.
(cherry picked from commit 1bff888947)
4 years ago
Philippe Antoine dbadca1567 detect: transforms check for 0-sized buffer
So as to avoid undefined behavior with a 0-sized variable length
array

Ticket: #5521
(cherry picked from commit 61b73416e2)
4 years ago
Juliana Fajardini fff9d529b7 exceptions: add reject support to exception policy
This enables the usage of 'reject' as an exception policy. As for both
IPS and IDS modes the intended result of sending a reject packet is to
reject the related flow, this will effectively mean setting the reject
action to the packet that triggered the exception condition, and then
dropping the associated flow.

Task #5503

(cherry picked from commit bbd968c738)
4 years ago
Victor Julien 336cd57234 stream/ids: make sure we don't slide past last_ack
Bug: #5401.
(cherry picked from commit f04b7a1827)
4 years ago
Victor Julien a383f29aa2 decode/tcp: allow 4 byte TFO with 2 byte cookie
(cherry picked from commit f5bd55dac8)
4 years ago
Gleb Smirnoff ae9b4144be ipfw: use PF_DIVERT on modern FreeBSD
(cherry picked from commit 5dbbc52b06)
4 years ago
Victor Julien d483ace444 unittests: optimize RunmodeIsUnittests()
(cherry picked from commit 86e600dab8)
4 years ago
Victor Julien 0a6e2b9d1a flow/recycler: bring back pthread_cond_t sleep
Bug #4379.

(cherry picked from commit f271fb4575)
4 years ago
Victor Julien 33446b0f64 flow/manager: sleep handled by pthread_cond_t again
Use only in live mode to allow FM to respond quickly to time
increases in offline mode.

Bug #4379.

(cherry picked from commit e6ac2e4e8a)
4 years ago
Victor Julien bb8f743870 time: add timeradd implementation
timeradd isn't available on MinGW.

(cherry picked from commit 39141a8836)
4 years ago
Philippe Antoine 739eba94a5 fuzz: restrict flags passed to AppLayerProtoDetectGetProto
Manual adaptation of 05f9b3f and f4449d3
4 years ago
Juliana Fajardini 05378946dc stream: add exception policy for midstream flows
This allows to set a midstream-policy that can:
- fail closed (stream.midstream-policy=drop-flow)
- fail open (stream.midstream-policy=pass-flow)
- bypass stream (stream.midstream-policy=bypass)
- do nothing (default behavior)

Usage and behavior:

If stream.midstream-policy is set then if Suricata identifies a midstream flow
it will apply the corresponding action associated with the policy.

No setting means Suricata will not apply such policies, either inspecting the
flow (if stream.midstream=true) or ignoring it stream.midstream=false.

Task #5468

(cherry picked from commit aa5bb2c329)
4 years ago
Juliana Fajardini 430bb1cf74 exceptions: add callbacks for drop-flow policy
Make sure that when the policy is to drop the flow, we set no inspection
for payload and packet and disable applayer inspection as well.

Task #5468

(cherry picked from commit 242b8f7d65)
4 years ago
Juliana Fajardini 0a124f2d27 exceptions: error out when invalid policy is used
Before, if an invalid value was passed as exception policy, Suricata
would log a warning and set the exception policy to "ignore". This is a
very different result, than, say, dropping or bypassing a midstream flow.

Task #5504

(cherry picked from commit 58ef3cde7a)
4 years ago
Juliana Fajardini d21d4c757e detect/parse: test sig parsing for more actions
Our unittests were only covering sig parsing for alert actions. As in
environments without LibNet the reject action will not work, we must
ensure that our parser properly fails in such cases, instead of silently
accepting an unsupported action.

Added tests for the reject and drop action.

Task #5496

(cherry picked from commit c81b78fd1c)
4 years ago
Juliana Fajardini 15ec08881e detect/engine: fix typos in assorted files
(cherry picked from commit 5f4bcfe313)
4 years ago
Juliana Fajardini 8f3ec86e17 commandline: add alert-queue expand failure option
For testing purposes. Meant to simulate a reallocation failure when
dynamically growing the alert queue in DetectEngineThreadCtx, so we can
check that Suri's behavior doesn't break under such circumstances.

Task #5319

(cherry picked from commit 58928b249d)
4 years ago
Philippe Antoine 7dc493cf55 fuzz: disable enip detection based on source port
So as to avoid fuzzing detecting protocol polyglots with enip

(cherry picked from commit d1ebf320f7)
4 years ago
Juliana Fajardini 1710c5f13d decode: remove unused macros
With the recent changes, these macros weren't being used anymore.

Related to
Bug #5458

(cherry picked from commit e7727c3744)
4 years ago
Juliana Fajardini dcf990c7d0 stream/tcp: remove repeated header declaration
StreamTcpRegisterTests was being declared twice.

(cherry picked from commit d07a6c6174)
4 years ago
Juliana Fajardini 2b59f65d02 detect/alert: add unittests to check packet action
Add unittests to check that packet flags are correctly updated after
detection finds drop or reject rules that match.

Related to
Bug #5458

(cherry picked from commit f897761ecb)
4 years ago
Juliana Fajardini b3ae9f840a decode: validate if dropped packet has drop reason
Related to
Bug #5458

(cherry picked from commit abd595d695)
4 years ago
Juliana Fajardini 32f90371d9 detect/alert: ensure reject action is applied
Bug 5458 states that the reject action is no longer working. While SV
tests that use the reject action still pass, it indeed seems that a
regression has happened with commit aa93984, because while the
function that applies rule actions to the flow (RuleActionToFlow) does
check for the reject action, the newly added function PacketApply
SignatureActions only checks for ACTION_DROP or ACTION_PASS when
deciding to call RuleActionToFlow.

Bug #5458

(cherry picked from commit 1f54e8611a)
4 years ago
Juliana Fajardini 6742ecbc9e decode: make PacketDrop use action as parameter
A Packet may be dropped due to several different reasons. This change
adds action as a parameter, so we can update the packet action when we
drop it, instead of setting it to drop.

Related to
Bug #5458

(cherry picked from commit 1774ff18a6)
4 years ago
Juliana Fajardini c56871c9e6 source/pcap: fix infinite loop if interface goes down
When in live-pcap mode, if the sniffed interface went down and up again,
Suri would enter an infinite and keep running, while not registering new
events. This fixes that behavior by allowing Suri to retry to open the
pcap in case of a retry on an already activated capture
('PCAP_ERROR_ACTIVATED').

This change is based on Zhiyuan Liao's work.

Bug #3846

(cherry picked from commit 2544be4672)
4 years ago
Jufajardini Reichow 41967bf119 detect/engine: init alert queue counters on reload
alert_queue_overflow and alerts_suppressed were not being
reinitialized when there was a reload of Suricata rules, leading to
non-valid stats counters if that happened.

Bug #5457

(cherry picked from commit 93c2c9743d)
4 years ago
Philippe Antoine 5df3973cc9 output: fix integer warnings
Ticket: 4516

(cherry picked from commit e3180e3248)
4 years ago
Philippe Antoine e55eeb3bde protocol: forbids concurrent protocol upgrades
Ticket: 5243

When switching from SMTP to TLS, and getting HTTP1 instead of
expected TLS, and HTTP1 requesting upgrade to HTTP2, we do not
overwrite the alproto_orig value so as not to have type confusion
in AppLayerParserStateProtoCleanup

(cherry picked from commit cedffdf14c)
4 years ago
Philippe Antoine 2c7c7cf315 output: skip files logging for ICMP packets
Ticket: #5408
(cherry picked from commit f8bf581775)
4 years ago
Philippe Antoine a71a474302 output: use flow's proto for file loggers
As there can be an ICMP packet which gets related to a TCP flow.

Ticket: #5408
(cherry picked from commit 5781631f85)
4 years ago
Eric Leblond 789b1ab4df output/alert: don't call basic logging twice
Issue: 4106
(cherry picked from commit 264eddb81f)
4 years ago
Philippe Antoine 84885c11c0 ftp: optimized tx iterator
To be more efficient with larger number of transactions.

Ticket: #5314
(cherry picked from commit 5a31b3508d)
4 years ago
Philippe Antoine ac6cdd973c event: only sets APPLAYER_UNEXPECTED_PROTOCOL once
If f->alproto == ALPROTO_UNKNOWN, we do not know the new protocol
yet, so we do not set the event yet.

(cherry picked from commit e692530021)
4 years ago
Philippe Antoine 48206f373d events: reset logged event
Otherwise, if we log a first app_layer_event, then reset
with AppLayerDecoderEventsResetEvents for a new packet,
then get another event, it does not get logged

Ticket: #5391
(cherry picked from commit b0298dd046)
4 years ago
Victor Julien c08a929de1 detect: impose limits on pcrexform
Impose match and recursion limits on pcrexform keywords.

Based on: 585e5e0d3c ("detect: impose limits on pcrexform")

Bug: #5414.
4 years ago
Victor Julien ac93f568eb fuzz/mime: fix call conditions and args
The SMTP parser should not supply lines w/o EOL chars to the mime
parser unless its in the BODY parsing stage. Mimic this in the fuzz
target by testing the state for inputs that have no EOL.

Additionally, make sure the delim cnt reflects the missing EOL.

(cherry picked from commit 816bbeb7dc)
4 years ago
Victor Julien 3384ca9be2 mime: fix and cleanup tests
Line count check was failing after recent delim handling updates.

(cherry picked from commit 074cfb5c68)
4 years ago
Victor Julien 7256445c21 mime: fix corner case
Fix a corner case where a base64 sequence including a space was followed
by a newline in the input data.

(cherry picked from commit d81582c4a2)
4 years ago
Victor Julien d15e0d5baf mime: add base64 related debug messages
(cherry picked from commit 5805ed47f5)
4 years ago
Victor Julien 9c25d10970 mime: improved empty line handling
Make sure a new body is not set up on empty lines unless it is
a body that is not encoded as base64/quoted printable.

(cherry picked from commit 41c2c1ed5a)
4 years ago
Victor Julien 7061dec732 smtp: fix passing a wrong delim len around
(cherry picked from commit 6e2c066ce1)
4 years ago
Victor Julien 58bce05764 mime: properly pass full lines to non-decoded body
Use actual delim count and make sure we also pass on empty lines
(so delim(s) only).

(cherry picked from commit b82b8825e7)
4 years ago
Victor Julien 420760a7ac mime/base64: fix final data not getting processed
If the last data of the body was not a multple of 4 and not padded
to be a multiple of 4, it would not be processed.

(cherry picked from commit 0d6ab727c5)
4 years ago
Victor Julien 909a3fdf42 stream: fix GAP check
Gap check would consider a GAP when the current data was in fact
exactly not a gap, but next segment(s) were already available.

(cherry picked from commit 100d821a9f)
4 years ago
Victor Julien 5740d10e34 mime: remove unused 'linerem' logic
(cherry picked from commit 0871029d17)
4 years ago
Victor Julien 0a6c61a564 smtp/mime: fix parsing edge case
Correctly track "remaining" bytes after partial base64 decoding.

Add comment clarifications and debug validation checks.

(cherry picked from commit 5953a7d2eb)
4 years ago
Victor Julien b4fd4c048b smtp: skip preprocessing for mime headers
Mime parser doesn't expect partial lines, which preprocessing can
provide. Add a check to let mime headers be handled by regular line
parsing.

(cherry picked from commit a38f2f2a52)
4 years ago
Victor Julien bf5d9d542b detect: set drop reason for rule based drops
Call `PacketDrop` with drop reason for drops, keep old logic
in place for the rest.

(cherry picked from commit a89840929b)
4 years ago
Victor Julien a32b02fb65 detect/tests: improve detection entry
Lots of tests still use SigMatchSignatures as their main detection
entry function, which bypassed some logic. Make it match main logic
more closely.

(cherry picked from commit e36b9b89a1)
4 years ago
Victor Julien 5fcbc8bb50 tests: fix drop test; cleanup
SigTestDropFlow04 was incorrectly expecting an alert in the packet
following a "drop" packet. The first drop is applied to the flow, so
it should lead to the 2nd packet being dropped before inspection is
run.

Clean up the test as well.

(cherry picked from commit 3874d08015)
4 years ago
Victor Julien 660484404f exceptions: initial exception-policy implementation
Adds a framework for setting exception policies. These would be called
when the engine reaches some kind of exception condition, like hitting
a memcap or some traffic processing error.

The policy gives control over what should happen next: drop the packet,
drop the packet and flow, bypass, etc.

Implements the policy for:

    stream: If stream session or reassembly memcaps are hit call the
    memcap policy on the packet and flow.

    flow: Apply policy when memcap is reached and no flow could be
    freed up.

    defrag: Apply policy when no tracker could be picked up.

    app-layer: Apply ppolicy if a parser reaches an error state.

All options default to 'ignore', which means the default behavior
is unchanged.

Adds commandline options: add simulation options for exceptions. These
are only exposed if compiled with `--enable-debug`.

Ticket: #5214.
Ticket: #5215.
Ticket: #5216.
Ticket: #5218.
Ticket: #5194.
(cherry picked from commit 8580499ded)
4 years ago
Victor Julien 3a55873eb8 eve/drop: log drop reason
Ticket: #5202.
(cherry picked from commit 0035673208)
4 years ago
Victor Julien aa66765253 flow: log action applied to all packets
Log if action applied to whole flow is drop or pass.

(cherry picked from commit 9a09fe454b)
4 years ago
Victor Julien 38744135ef detect: enforce flow drops earlier
Enforcing flow drops is now done earlier in the detection engine and
moved out of the IP-only engine where it didn't belong.

(cherry picked from commit 802c1ffee3)
4 years ago
Victor Julien d8b5d3d34a clang-format: disable for option table 4 years ago
Lukas Sismis 4c4b1de168 bypass: fix memory leak - reassign of FlowBypassInfo
In some situations bypass callback is called on already bypassed
flow. This allocates FlowBypassInfo structure for the flow but
does not check if the flow already has one.

Issue: #5368
(cherry picked from commit 05797c45bb)
4 years ago