Commit Graph

11361 Commits (20df715e64ccab00c6f6778e1453b051aee7be17)

Author SHA1 Message Date
Victor Julien e621416d80 error: bootstrap errno like error code approach 4 years ago
Victor Julien cdd52dc8cd detect/mpm: output cleanups 4 years ago
Victor Julien cdac0425a8 device: special case handling of dpdk devices 4 years ago
Victor Julien d84bcdfd5c device: output tweak 4 years ago
Victor Julien c78ebd1352 dpdk: set RSS flags for igb driver
Tested on i340.
4 years ago
Victor Julien b08e3e7460 dpdk: ips output cleanups 4 years ago
Victor Julien 70b69a3171 output: minor code cleanups 4 years ago
Victor Julien 10c26bb628 unix-socket: don't print unix socket message twice 4 years ago
Jeff Lucovsky 82158f92c1 general: Correct typos 4 years ago
Victor Julien be895d67b6 runmodes: minor output cleanup 4 years ago
Victor Julien cef0a84bd2 detect: turn cleanup message into debug message 4 years ago
Victor Julien b76f6dfa42 device/mtu: clean up mtu message 4 years ago
Victor Julien 7ccfa9b9a6 device: cleanup stats print 4 years ago
Victor Julien fd4e522038 runmode: log test mode later 4 years ago
Victor Julien 181400bfdd pcap-file: output cleanups 4 years ago
Victor Julien ef881c942b af-packet: output cleanups 4 years ago
Victor Julien 393546af07 mpm/hs: turn cleanup messages in to debug 4 years ago
Victor Julien 892b7021b2 stream: debug message cleanup 4 years ago
Victor Julien d973e1021c threading: memset cpu_set_t to address cppcheck warning
src/tm-threads.c:734:5: error: Uninitialized variable: cs [uninitvar]
    CPU_ZERO(&cs);
4 years ago
Victor Julien 58c28d1778 dpdk: fix cppcheck warnings
src/runmode-dpdk.c:202:11: warning: Size of pointer 'argv' used instead of size of its data. This is likely to lead to a buffer overflow. You probably intend to write 'sizeof(*argv)'. [pointerSize]
    args->argv = SCCalloc(capacity, sizeof(args->argv));
          ^
src/runmode-dpdk.c:777:23: error: Shifting 32-bit value by 63 bits is undefined behaviour [shiftTooManyBits]
        if (bits & (1 << i))
                      ^
src/runmode-dpdk.c:776:23: note: Assuming that condition 'i<64' is not redundant
    for (int i = 0; i < 64; i++) {
                      ^
src/runmode-dpdk.c:777:23: note: Shift
        if (bits & (1 << i))
                      ^
4 years ago
Victor Julien 17cd41c887 frames: suppress cppcheck warning
src/app-layer-frames.c:471:5: warning: Identical condition 'stream_slice->input==NULL', second condition is always false [identicalConditionAfterEarlyExit]
    BUG_ON(stream_slice->input == NULL);
    ^
src/app-layer-frames.c:468:29: note: If condition 'stream_slice->input==NULL' is true, the function will return/exit
    if (stream_slice->input == NULL)
                            ^
src/app-layer-frames.c:471:5: note: Testing identical condition 'stream_slice->input==NULL'
    BUG_ON(stream_slice->input == NULL);
    ^
src/app-layer-frames.c:548:5: warning: Identical condition 'stream_slice->input==NULL', second condition is always false [identicalConditionAfterEarlyExit]
    BUG_ON(stream_slice->input == NULL);
    ^
src/app-layer-frames.c:545:29: note: If condition 'stream_slice->input==NULL' is true, the function will return/exit
    if (stream_slice->input == NULL)
                            ^
src/app-layer-frames.c:548:5: note: Testing identical condition 'stream_slice->input==NULL'
    BUG_ON(stream_slice->input == NULL);
    ^
4 years ago
Victor Julien 16c19bd2f6 capture: remove unnecessary mtu check 4 years ago
Victor Julien f45914cd99 classification: avoid duplicate errors 4 years ago
Victor Julien 7d46d4217f unix-socket: avoid duplicate errors 4 years ago
Victor Julien 4dfb584164 dnp3: no error logging in packet path 4 years ago
Haleema Khan 05e87bf32e detect-detection: Convert unittests to new FAIL/PASS API
Ticket: #4019
4 years ago
Haleema Khan 6094aeee9e detect-detection: Apply clang-format 4 years ago
Victor Julien eb69902719 dpdk: NUMA output adjustments
Reduce verbosity of the NUMA logs; switch them to start with the
iface.

Don't print the cpu number.
4 years ago
Haleema Khan 4a98c4bde6 detect/sigorder: Convert unittests to new FAIL/PASS API
Ticket: #4029
4 years ago
Lukas Sismis 563f8bb56b dpdk: decrease intensity of warnings related to NUMA placement
Ticket: #5617
4 years ago
Victor Julien 991f9fde32 radix: fix ipv6 address parsing warning
The check meant to see if the ip address part of the ip/cidr combo
was more specific than needed wasn't fully implemented, leading to
warnings being issued on completely valid and correct input.

This patch implements the same logic as in IPv4. If the ip address
as specified is different from the ip after the mask has been applied,
a warning is displayed.

Bug: #5747.
4 years ago
Bazzan Don b289030696 detect-engine-tag: convert unittests to new FAIL/PASS API
Ticket: #4030
4 years ago
Philippe Antoine a58ffe5b3e fuzz: do not use timestamps at the end of times
so as not to have integer overflows
4 years ago
Philippe Antoine 190c945e67 ftp: disables debug assertion
Ticket: #5443

Until we decide on the right way to fix this.
4 years ago
Philippe Antoine 4daee8bae1 quic: do not use stack for inspection buffer
Make the inspection buffer copy the content in one buffer it owns.

Ticket: #5707
4 years ago
Jason Ish 4a7567b3f0 template: rename template-rust to template
Remove the distinction between the C template protocol "template" and
the Rust template protocol "template-rust" and make the Rust parser
simply template now that we no longer have support to generate a C
protocol template.
4 years ago
Jason Ish 8683154115 templates: remove C app-layer templates 4 years ago
Jason Ish 05e16820de templates: clang format cleanups
Cleanup the trivial clang-formatting issues in templates.  Length of
protocol names may require clang-format after new protocol generation.
4 years ago
Jason Ish 50a787a9a3 app-layer-template-rust: remove C app-layer stub
Remove the app-layer-PROTO stub for Rust based parsers.  It is no longer
needed as Rust parsers now contain the registration function in Rust.

Ticket: 4939
4 years ago
Victor Julien 7bfb63210e mime/base64: unify space handling 4 years ago
Philippe Antoine 0e054af1ac mime: do not skip empty lines for quoted-printable
As these lines are in the file...

Ticket: #5725
4 years ago
Philippe Antoine 04fd2ae78e dns: do not oputput empty array for query 4 years ago
Richard McConnell 6e128f48a2 af-xdp: Add AF_XDP socket support
AF_XDP support is a recent technology introduced that aims at improving
capture performance. With this update, Suricata now provides a new
capture source 'af-xdp' that attaches an eBPF program to the network
interface card. Packets received in the NIC queue are forwarded to
a RX ring in user-space, bypassing the Linux network stack.

Note, there is a configuration option (force-xdp-mode) that forces the
packet through the normal Linux network stack.

libxdp and libbpf is required for this feature and is compile time
configured.

This capture source operates on single and multi-queue NIC's via
suricata.yaml. Here, various features can be enabled, disabled
or edited as required by the use case.

This feature currently only supports receiving packets via AF_XDP,
no TX support has been developed.

Ticket: https://redmine.openinfosecfoundation.org/issues/3306

Additional reading:
https://www.kernel.org/doc/html/latest/networking/af_xdp.html
4 years ago
Todd Mortimer 7d1a8cc335 file/swf: Use lzma-rs decompression instead of libhtp.
Use the lzma-rs crate for decompressing swf/lzma files instead of
the lzma decompressor in libhtp. This decouples suricata from libhtp
except for actual http parsing, and means libhtp no longer has to
export a lzma decompression interface.

Ticket: #5638
4 years ago
Victor Julien 9ed65907a7 fuzz/sigpcap: set pkt_src 4 years ago
Victor Julien f848e34bcc stream: stricter check inserting segments
In lossy streams, esp where TcpSession::lossy_be_liberal it is possible
to end up inserting a segment that is out of the expected sequence
number bounds.
4 years ago
Victor Julien 5b6193f4c4 flow: cleanup and clarify ancient debug messages 4 years ago
Victor Julien 03d049dadc decode: enforce layer limit through tunnel layers
Bug: #5686.
4 years ago
Shivani Bhardwaj f80c999db3 util/base64: fix heap buffer overflow
While updating the destination pointer, we were also adding the padded
bytes which are not a part of the decoded bytes. This led to running out
of space on the destination buffer.
Fix it by only incrementing destination buffer ptr by the number of
actual bytes that were decoded.

Ticket 5623
4 years ago
Victor Julien 9f4dd4fc56 smtp/files: don't modify prev file on open failure 4 years ago
Victor Julien e601ebdfd8 files: always initialize inspect_window and min_inspect_depth
This is to make sure the files buffers are properly managed even
when there are no rules or when there are no file.data rules.

Bug: #5703.
4 years ago
Victor Julien df7d8d96c9 streaming/buffer: set hard limit on buffer size
Don't allow the buffer to grow beyond 1GiB. Add a once per thread
warning if it does reach it.

Bug: #5703.
4 years ago
Victor Julien 2edfff7a0c src: unify how warnings specify ticket id's 4 years ago
Jason Ish 0c00f28ebc afpacket/netmap: warn about mixed ips, ids/tap deprecation
Suricata already logs if AF_PACKET or Netmap are running in a mixed IPS
and IDS/TAP mode.  As the behavior is undefined when these modes are
mixed, it is best to deprecate and to not allow this behavior. For now
warn that it will be unsupported and fail in Suricata 8.

Ticket: 5587
4 years ago
Philippe Antoine 3e4f58e375 detect: fix memory leak when parsing signature
Ticket: #5529
4 years ago
Victor Julien 1e653cc36d profiling: fix includes 4 years ago
Victor Julien 50b858aa49 ipfw: fix missing include 4 years ago
Victor Julien f3f2807202 netmap: fix missing include 4 years ago
Jason Ish dcd9dabc70 classification: continue processing on parse error
Instead of returning on the first line that fails to parse, log the
error and continue instead of returning.

The fail fast makes sense in test mode, but not in a normal run mode
where you don't want one bad line to abort processing the whole file.

This will still fail out in test mode.

Related issue: 4554
4 years ago
Philippe Antoine ad713246a9 src: remove double includes
Keep the unconditional include to be sure it works

git grep '#include "' src/*.c | sort | uniq -c | awk '$1 > 1'
4 years ago
Philippe Antoine 9af0dafbad src: fix some include orders
So as to be able to get include removal right
4 years ago
Philippe Antoine cc23923de1 src: remove obsolete comment
Should have been removed along by commit
82dba07579
4 years ago
Philippe Antoine 62352ad030 src: fix remaining cppclean warnings 4 years ago
Philippe Antoine 1f066cbbe8 unittest: fix unneeded includes as per cppclean
Especially because there is conditional inclusion from a header
4 years ago
Philippe Antoine 662f0ce503 util: fix includes for util-memcmp
u8_tolower is now in suricata-common.h

Fixes commit 19e94e93fa
4 years ago
Philippe Antoine e85f3916e3 src: fix integer warnings
and adds defrag debug validations
4 years ago
Philippe Antoine b5147189ae tls: fix off by one in supported versions extension
Ticket: #5663
4 years ago
Jeff Lucovsky a4239d433a detect/bsize: Validate bsize values after parsing
Issue: 2982

This commit moves bsize validation with respect to content matches to
the post-parse validation stage. This allows bsize to consider all
content-related values, including those that follow the bsize keyword.
4 years ago
Jeff Lucovsky 9d73777a46 bsize/general: Remove unnecessary includes
This commit removes unused/commented out #include lines.
4 years ago
Jeff Lucovsky 8b41754acd add to general: Typo fixup 4 years ago
Jeff Lucovsky 8df6701186 netmap: Fix include file issues 4 years ago
Jeff Lucovsky ecfdc24e08 detect/bsize: Semantic validation of bsize values
This commit adds validation of the bsize value(s) with the available
buffer size. Signatures are flagged if the bsize and buffer size are
incompatible.

Issue: 3682
4 years ago
Jeff Lucovsky 25c0a6ea7c tests/bsize: Test cases with preceding content
This commit adds test cases that validate behavior when "content"
immediately precedes "bsize".
4 years ago
Jeff Lucovsky c91b987732 general: Typo fixup 4 years ago
Jeff Lucovsky 115297c016 detect/content: Use SCCalloc instead of malloc/memset
This commit replaces a SCMalloc/memset with SCCalloc
4 years ago
Jeff Lucovsky 7eb5fb1826 detect/content: Consider distance in validation
Ticket: 2982

This commit validates that the content usage in a rule will not exceed
the dsize value.

Values of distance that cause the right edge to be exceeded are
considered an error and the signature will be rejected.
4 years ago
Shivani Bhardwaj 58e5033a44 util/base64: use padding var for calculations 4 years ago
Victor Julien ca6ce935b5 security: disable setrlimit with asan
Various parts of ASAN will require creation of threads. At least
LSAN reporting and ASAN stack traces require this. Detect if we
run with ASAN and bypass the noproc setting with a warning.

Bug: #5661.
4 years ago
Victor Julien 8f0d820218 ssl: add debug validation checks for recent changes
Make sure the assumptions are correct.
4 years ago
Philippe Antoine 3fbcba8ab7 tls: fix buffer overread
Recently introduced by commit 4bab6e24e5

Ticket: #5564
4 years ago
Philippe Antoine 4706083929 detect: avoids memory leak on ja3 signature parsing
If a later keyword enforces a protocol incompatible with ja3
4 years ago
Philippe Antoine c09ca7d171 clean: use SC_MD5_HEX_LEN instead of magic number 32 4 years ago
Philippe Antoine 7823653a30 clean: replace MD5_STRING_LENGTH with more used SC_MD5_HEX_LEN 4 years ago
Philippe Antoine 5a0d07b13f clean: remove useless * sizeof(char) 4 years ago
Philippe Antoine f75d18b077 quic: ja3 code deduplication
As quic transactions are unidirectional, the same function is
used to get ja3 from client or from server.
4 years ago
Philippe Antoine 7cb40a1dfc quic: allow ja3.hash keyword
Ticket: #5624
4 years ago
Daniel Young aeb0c0e71a detect/parse-port: fix whitespaces bug
Updated by Victor Julien to:
- accept but strip leading whitespaces
- update tests
- cleanup tests
4 years ago
Eric Leblond ea85e2ccf2 detect/filename: fix buffer description 4 years ago
Shivani Bhardwaj 82dba07579 src: remove unneeded header and refs 4 years ago
Haleema Khan df45fc9e97 detect-l3proto: Convert unittests to new FAIL/PASS API
Ticket: #4050
4 years ago
Alice Akaki d46ce170ba detect-icmp-seq: clean-up unittests
Task: #5597
4 years ago
Eric Leblond 4e1ff9a7ff flow: fix FlowSwapFileFlags function
Ticket: #5625
4 years ago
Haleema Khan 160c778a1f detect-filesize: Convert unittests to new FAIL/PASS API
Ticket: 4039
4 years ago
Haleema Khan f25d4cc546 detect-distance: Convert unittests to new FAIL/PASS API
Ticket : #4020
4 years ago
Alice Akaki f251b6af65 detect-icmp-id: convert unittests to FAIL/PASS APIs
Task: #4042
4 years ago
Haleema Khan 1fdd56a61d detect-itype: Convert unittests to new FAIL/PASS API
Bug: #5589
4 years ago
Simeon Miteff 6673846481 profiling: Make rule_perf.log JSON output newline-delimited
Adding a newline makes the output valid JSON when multiple sortings of
the rules profile object are written to the log.
4 years ago
Jason Ish 0d3cfbbe3f bittorrent-dht/eve: log as bittorrent_dht 4 years ago
Aaron Bungay 86037885a9 bittorrent-dht: add bittorrent-dht app layer
Parses and logs the bittorrent-dht protocol.

Note: Includes some compilation fixups after rebase by Jason Ish.

Feature: #3086
4 years ago
Jason Ish 27672c950c dnp3: fixups to work with unified json tx logger
Update DNP3 to work with a single TX logger, and just register one
logger instead of 2.

This primarily creates a TX per message instead of correlating replies
to requests, which fits the DNP3 model better, but we didn't really have
this concept nailed down when DNP3 was written.
4 years ago
Victor Julien 2f9ca8bb34 dpdk: set new running flag at thread startup 4 years ago
Victor Julien 91b7b63ed2 threads: count thread types after all initialized
To avoid double counting in case a thread took longer than
expected to start up.
4 years ago
Alice Akaki 6621b0ec93 detect-icmp-seq: convert unittests to FAIL/PASS APIs
Task: #4043
4 years ago
Haleema Khan 8d5c5f24a1 dns/eve: add 'HTTPS' type logging
Add a new DNS record type to represent HTTPS
Ticket: #4751
4 years ago
Haleema Khan b1972a5c61 detect-itype: remove unittests
Ticket: #5590
4 years ago
Gabriel Lima Luz 4b009eb907 detect-replace: Convert unittests to FAIL/PASS API
Ticket: 4054
4 years ago
Philippe Antoine a003640ecf security: prevents process creation
with setrlimit NPROC.

So that, if Suricata wants to execve or such to create a new process
the OS will forbid it so that RCE exploits are more painful to write.

Ticket: #5373
4 years ago
Jason Ish 2ab3646fad profiling: sort LoggerId's in same order as defined
Sort the LoggerId's in the order they are define in suricata-common.h.
4 years ago
Jason Ish 99cb8c666a loggers: all json tx loggers can share the same loggerid
This is to avoid the tx logging code that doesn't support LoggerId
values over 31 at this time. The simplest fix for now is to just have
all JSON (eve) loggers use the same ID.

DNP3 is left as-is for now as it needs some extra support in the parser.
4 years ago
Richard McConnell 13beba141c source: add THV_RUNNING flag to notify of running state
Each module (thread) updates its status to indicate running.
Main thread awaits for all threads to be in a running state
before continuing the initialisation process

Implements feature 5384
(https://redmine.openinfosecfoundation.org/issues/5384)
4 years ago
Eric Leblond 94664ef565 datasets: don't exit on invalid data 4 years ago
Eric Leblond 3bd48d9336 detect: doc link for ip.src and ip.dst 4 years ago
Eric Leblond 7518204ad4 datasets: introduce new IPv6 type
This patch also simplifies IPv6 parsing.

Feature: #5383
4 years ago
Eric Leblond 6fe9d510cf detect: add ip.dst keyword as sticky buffer
Feature: #5383
4 years ago
Eric Leblond 7e516aad94 detect: add ip.src keyword
It is a sticky buffer matching on src_ip.

Feature: #5383
4 years ago
Eric Leblond b2cdc6c899 datasets: introduce ipv4 type
This patch introduce the IPv4 type for dataset so Suricata commandmatch
on a set of IPv4 addresses. This is meant to complement iprep feature
for people that needs more flexibility such as settings the IP on
the packet path.

Feature: #5383
4 years ago
Eric Leblond a9c05c7d96 datasets: factorize serialised operations
Ticket: #5184
4 years ago
Eric Leblond 843dba0a28 datasets: add dataset-lookup command
Ticket: #5184
4 years ago
Eric Leblond 2f25e48897 datasets: add dataset-clear command
Ticket: #5184
4 years ago
Eric Leblond a480abcdd0 datasets: add dump via unix socket
This patch adds a dataset-dump command to the list of unix socket
commands. Implementation is not optimal as we are locking the
datasets when doing the dump. But if we consider that the current
alternative from an implementation point of view is to stop Suricata
then this is far better than current state.

Ticket: #5184
4 years ago
Victor Julien 1fafb83fed packet: turn tunnel lock into spinlock
Lock is only held to update/check ints, so spin lock will be more
efficient.

Place the member of Packet in a new "persistent" area to make it
clear this is not touched by the PacketReinit logic.

Ticket: #5592.
4 years ago
Victor Julien 57e70841c4 stream/tcp: remove obsolete and commented out tests 4 years ago
Victor Julien e72770c1b2 decode/vxlan/tests: don't memset new packet 4 years ago
Victor Julien edf93ae5b6 decode/mpls/tests: improve pkt handling; cleanups 4 years ago
Victor Julien 0f7fe2a4c3 app-layer/tests: don't memset new packet 4 years ago
Victor Julien 6dc53447f1 decode/geneve/tests: don't memset packet
Packet is already initialized.
4 years ago
Victor Julien b07c7ad14c threading: improve/add thread queues explanations 4 years ago
Victor Julien 951bcde0b2 eve/alert: remove tunnel locking
Tunnel lock is only used to sync verdict logic.
4 years ago
Victor Julien 0e7adc21a6 decode: alloc packets using calloc 4 years ago
Victor Julien 68a9da52ad packetpool: remove PKT_ALLOC flag
Use Packet::pool instead. If Packet::pool is non-NULL the packet is
owned by a pool. Otherwise it is allocated and should be freed after
use.
4 years ago
Victor Julien 3ed7b4473e runmodes: remove dead error check 4 years ago
Victor Julien 9d3c60bde3 smtp/mime: no error logging in packet path 4 years ago
Victor Julien 256f0f2c5c defrag: no error logging in packet path 4 years ago
Victor Julien 8f02a3e415 applayer: make sure to use correct ipproto 4 years ago
Victor Julien dba7103a96 eve/alert: use flow proto to avoid crash on ICMP 4 years ago
Alice Akaki 51c0714d71 detect-filemd5: convert unittests to FAIL/PASS APIs
Task: #4035
4 years ago
Alice Akaki fe31d51c30 detect-filesha256: convert unittests to FAIL/PASS APIs
Task: #4038
4 years ago
Haleema Khan 6988168114 src: Use WARN_UNUSED for ByteExtract* functions
Add WARN_UNUSED macro for ByteExtract* functions
Fix warning raised in code related to WARN_UNUSED for ByteExtract*

Ticket: #3658
4 years ago
Alice Akaki 76024f7571 detect-filename: convert unittests to FAIL/PASS APIs
Task: #4036
4 years ago
Alice Akaki 8614bff017 detect-filemagic: convert unittests to FAIL/PASS APIs
Task: #4034
4 years ago
Victor Julien 55cf11fdc2 flow/storage: use const for getter 4 years ago
Victor Julien 99fd69ee8c eve: mac logging code cleanup 4 years ago
Victor Julien 2d79a5206b eve/drop: log direction if we have a flow 4 years ago
Victor Julien 90f3823cad eve: log mac addresses in packet direction 4 years ago
Victor Julien f1068bbb08 dpdk: fix timestamp issues
Each thread had its own version of the `machine_start_time`, which
lead to slight time differences. This became apparent mostly in IPS,
where 2 threads each process a side of the flow.

This patch makes the `machine_start_time` global.
4 years ago
Victor Julien cd2a5ec84f packet: move action functions to packet files 4 years ago
Victor Julien 0977f40d1c detect/tag: improve time handling on windows
Bug: #5584.
4 years ago
Victor Julien 410acf226f detect: unittest guards for ut only function 4 years ago
Victor Julien 60285a629f detect: remove unused function 4 years ago
Victor Julien 5d6212183b eve/alert: minor cleanups 4 years ago
Victor Julien 6c200c7793 detect: issue drop to root packet in all cases
Update DROP action handling in tunnel packets. DROP/REJECT action is set
to outer (root) and inner packet.

Check action flags both against outer (root) and inner packet.

Remove PACKET_SET_ACTION macro. Replace with RESET for the one reset usecase.
The reason to remove is to make the logic easier to understand.

Reduce scope of RESET macros.

Rename PacketTestAction to PacketCheckAction except in unittests. Keep
PacketTestAction as a wrapper around PacketCheckAction. This makes it
easier to trace the action handling in the real code.

Fix rate_filter setting actions directly.

General code cleanups.

Bug: #5571.
4 years ago
Victor Julien 79fc8e74cb packetpool: debug message to assist drop checks 4 years ago
Victor Julien d1009e295c packetpool: remove debug validation check
Current packet might be the root or a child. Root would have set
drop action set, but Packet::pkt_src might be set in either.
4 years ago
Victor Julien fe5a8beb50 decode: minor code cleanup 4 years ago
Victor Julien ba3e0b3155 nfq: set drop reason on verdict error 4 years ago
Victor Julien a7333a3ea5 napatech: reduce size of Packet structure
Put napatech packet vars in the union that is meant for this type of
data.
4 years ago
Haleema Khan b31a286952 detect-fileext: convert unittests to FAIL/PASS APIs
Fixes Bug: #4033
4 years ago
Lukas Sismis 5365fdccf7 dpdk: fix mempool cache error message 4 years ago
Shivani Bhardwaj 7005443b8b base64: add and clean tests 4 years ago
Shivani Bhardwaj dad52f133d base64: add new mode as per RFC 4648
As per RFC 4648,
Implementations MUST reject the encoded data if it contains characters
outside the base alphabet when interpreting base-encoded data, unless
the specification referring to this document explicitly states
otherwise.

Add a new mode BASE64_MODE_RFC4648, and handle input strictly as per the
specification.

Bug 5223
4 years ago
Lukas Sismis e101384e7b transversal: remove suricata-ids.org references 4 years ago
Lukas Sismis aeb690317a dpdk: allow specifying RSS hash function flags in the config
Ticket: #5400
4 years ago
Eric Leblond e46a0bd46a eve: explicit default when setting port 4 years ago
Eric Leblond 00c419a6f8 eve: micro simplification 4 years ago
Eric Leblond a0065f4368 eve/alert: add direction field to log data way
Add a key in the event to specify if the data that did
trigger the alert are in to_client or to_server direction.
4 years ago
Eric Leblond f1300e68c9 eve/alert: add src and dest info to flow in alert
When looking at an alert event, it was impossible to determine which
side from src or dest IP in the alert was the client and wich side
was the server with regards to the underlying flow. This was a problem
when you try to known who belongs a metadata property such as a HTTP
hostname or a TLS JA3.

This patch updates the code to add src and dest IP in the flow
subobject as well as src and dst port. This way, we can now which
side is the client and which side is the server.

The result is looking like:

{
  "event_type": "alert",
  "src_ip": "22.47.184.196",
  "src_port": 81,
  "dest_ip": "192.168.1.47",
  "dest_port": 1063,
  "proto": "TCP",
  "tx_id": 0,
  "alert": {
    "signature_id": 2018959,
    "rev": 3,
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 22,
    "pkts_toclient": 35,
    "bytes_toserver": 1370,
    "bytes_toclient": 48852,
    "start": "2009-10-28T10:01:46.755232+0100",
    "src_ip": "192.168.1.47",
    "dest_ip": "22.47.184.196",
    "src_port": 1063,
    "dest_port": 81
  }
}
4 years ago
Eric Leblond bb93d67ddd unix-socket: add command to get flow stats
Add a command to extract the accounting data from a live
flow using the unix socket. It takes the flow_id as param
and return the volume of data seen on the flow as well as
its age.
4 years ago
Eric Leblond 19400a7d69 flow: add function to get flow using flow_id 4 years ago
Eric Leblond 06756314d6 flow: change flow id computation method
Previous method was truncating the flow hash value when building
the flow_id. It is interesting not to loose the flow hash value
as it can be used in other tools or to interact with a flow that
is still active.
4 years ago
Eric Leblond 06b6f85c1f json/flow: log if flow had gap in TCP 4 years ago
Eric Leblond f9faff5c4c flow: add function to say if there is gap 4 years ago
Eric Leblond e6768118da stream: flag TCP streams with gap 4 years ago
Eric Leblond 9cb06d4376 detect/smb: add smb.ntlmssp_domain keyword
Feature #5411.
4 years ago
Eric Leblond 69ef1bc194 detect/smb: add smb.ntlmssp_user keyword
Feature #5411.
4 years ago
Victor Julien a6e86fef04 detect: remove unused data struct 4 years ago
Victor Julien f837146321 flow/worker: process injected flows more gradually
Worker threads are responsible for final processing of timed out flows.
These are selected by the Flow Manager and inserted into a per thread
queue. The Flow Worker then checks this queue after each packet. Due to
the burstiness of this process, the packet threads would sometimes process
a lot of these flows in the context of a single packet, leading to spike
in latency which might cause packet loss.

This patch changes the behavior to only process at max 2 flows per packet.
This way added processing cost is amortized over many packets.
4 years ago
Victor Julien ce1bdcb474 flow: count max number of injected flows in workers 4 years ago
Victor Julien 235f369ab9 stream: reduce pool locking overhead
Add thread local cache to avoid locking overhead for ssns and segments.

A thread will return segments/ssns to a local cache first, and if that
is full, to a return queue where the actual return to the pool returns
a batch, to amortize locking overhead.

Adds segment and session pool/cache counters to see where how effective
the cache is.
4 years ago
Victor Julien 536d66e344 stream: minor test cleanup 4 years ago
Victor Julien 839a4f5ab4 files: only call loggers is there is work to do 4 years ago
Victor Julien fa6d41ca66 filestore: remove obsolete checks 4 years ago
Victor Julien 0b3053a376 filestore: fix empty file not opening, but trying to close 4 years ago
Victor Julien 23323a961f app-layer: reduce app cleanup and output-tx calls
Track packets that updated the app-layer, and for those run
the transaction housekeeping and output-tx logging loops.

Do the same of end of flow packets.

This skips needless iterations over the transaction stores.
4 years ago
Victor Julien 4bb7f827e0 output/tx: minor cleanups/optimizations 4 years ago
Victor Julien 39cf5b151a src: includes cleanup
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.

Update files to compile after this.

Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.

Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
4 years ago
Victor Julien 3e2295a963 detect: clean up detect-engine-state.h
Remove prototypes that are not about purely the data structures.
4 years ago
Victor Julien ad0a29cabc detect: remove wrapper func 4 years ago
Victor Julien 1dd0a2fed6 detect: move DetectTransaction to header its used in 4 years ago
Victor Julien 41265a859e detect/files: optimize file.data by skipping non-file txs
As well as 'file' txs not in our direction.

Implement the same logic for file.name and file.magic prefilter engines.
4 years ago
Victor Julien db0f9ddc69 files/tx: inspection, logging and loop optimizations
Introduce AppLayerTxData::file_tx as direction(s) indicator for transactions.
When set to 0, its not a file tx and it will not be considered for file
inspection, logging and housekeeping tasks.

Various tx loop optimizations in housekeeping and output.

Update the "file capable" app-layers to set the fields based on their
directional file support as well as on the traffic.
4 years ago
Victor Julien 3263202094 detect/tx: add AppLayerTxData to PrefilterTx
In preparation of some file inspection optimizations, for which we need the
tx data.

Update all users.
4 years ago
Victor Julien 602c39ed01 files: remove unused code 4 years ago
Victor Julien b1c22169f8 files: don't set NOSTORE in 'store all' case 4 years ago
Victor Julien d39a79b6f2 smtp: remove bad tests 4 years ago
Victor Julien 79499e4769 app-layer: move files into transactions
Update APIs to store files in transactions instead of the per flow state.

Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.

Update all protocol implementations to support this.

Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.

Update the filestore keyword to consider a match if filestore output not
enabled.
4 years ago
Victor Julien 01e64d80da app-layer: trunc parser per direction 4 years ago
Victor Julien ff9d1807f9 app-layer: parser flags to u16 4 years ago
Victor Julien a1d728bb65 app-layer: specify direction in tx cleanup
In preparation of per tx files storage.
4 years ago
Victor Julien c27df6304d app-layer: introduce common AppLayerStateData API
Add per state structure for storing flags and other variables.
4 years ago
Victor Julien 96b642c32d file: minor debug updates 4 years ago
Victor Julien 2218a3716e file: clean up file flags handling 4 years ago
Victor Julien 408b64558f files: debug log flags 4 years ago
Victor Julien bdbaaa3b24 lua: store id with tx ptr 4 years ago
Sascha Steinbiss 8438ee48aa decode-ipv4: adjust validation to RFC
RFC1108 only specifies a minimum field length of 3, not
a fixed length of 11.
4 years ago
Sascha Steinbiss fb790121bb decode-ipv4: implement extended security option
IP option 0x85 (extended security) is mentioned in the
documentation for the ipopts keyword but was not implemented.
4 years ago
Philippe Antoine ae6abd8ca3 ssl: fix compiler warning
implicit conversion loses integer precision: 'int' to 'uint16_t'
because C shifts << translates automatically to signed integers
4 years ago
Victor Julien b06c0579f5 stream: fix reachable assertion
Fix `Flow::thread_id` not always getting properly set up, leading to
a reachable assertion.

Bug #4582.
4 years ago
Juliana Fajardini 9d9bc04886 stream/tcp: typo fix 4 years ago
jason taylor 489af24132 detect: update ttl debug log messages
Signed-off-by: jason taylor <jtfas90@gmail.com>
4 years ago
Jeff Lucovsky ccd1063e43 detect/bytemath: convert parser to Rust
Issue: 5077

This commit
- Converts the PCRE based parser to Rust.
- Adds unit tests to the new Rust modules
- Removes the PCRE parser from detect-bytemath.c
- Adjusts the C source modules to refer to the Rust definitions
- Includes the multiply operator (missing from the C parser)
4 years ago
Philippe Antoine c6d8daecd3 log: fix coverity warning
CID 1515529

Checks ftell return value for negative/error
4 years ago
Victor Julien 1701a6b14c tls: handle incomplete header sooner
Make sure to exit the parser early on incomplete header data.

Additionally, make sure to not create duplicated tls frames in this
case.

Add a debug validation check for the header parser parsing too much
data, which should never happen.
4 years ago
Victor Julien 9053c49178 ssl: add debug validation check for incomplete api 4 years ago
Victor Julien 0ec136621d debug: add bool string print helper macro 4 years ago
Victor Julien 69be41b241 tls: improve record checks
Improve unknown record handling. Inspired by Wireshark 'unknown record'
handling, we take a best effort approach for records with unknown content
types in TLS versions 1.0, 1.1 and 1.2.

Improve record length check and set 'invalid_record_length' event instead
of 'invalid_tls_header'.
4 years ago
Victor Julien c028800ae1 tls: improve versions extension logic
Skip over unusable versions like GREASE.
4 years ago
Victor Julien 599791fa33 tls: make version and size checks stricter
This way bad records won't buffer lots of stream data.
4 years ago
Victor Julien c73d812026 tls: store cert data in heap buffer
Cert chain is a list of pointers into this buffer, so can't use a
stream slice approach.
4 years ago
Victor Julien 4a283d480d eve/tls: implement client cert logging
Enable client logging in extended mode.

Add "client", "client_certificate" and "client_chain", where the latter two
depend on "client".
4 years ago
Victor Julien e817a8f968 tls: parse client certificates
Parse client cerificates and store them in the state similar to how
this is done for server certificates.

Update "progress" handling to not consider the TLS handshake complete
if the server indicated a client cert was needed.
4 years ago
Victor Julien 10f5e6cb66 tls: prepare for client cert parsing 4 years ago
Victor Julien 6d4cc39c02 eve/tls: prepare for client cert logging
Code cleanups that work on per direction "connp" instead of hard coding
to the server side.
4 years ago
Victor Julien 14b2e04b58 tls: make cert handling more generic
In preparation for client cert handling.
4 years ago
Victor Julien cf4c201acb tls: avoid tls.invalid_handshake_message FP
Don't set TLS_DECODER_EVENT_INVALID_HANDSHAKE_MESSAGE event on encrypted
handshake messages.
4 years ago
Victor Julien e9d63f3355 tls: don't set 2 events for a single exception
Keep the more specific ones.
4 years ago
Victor Julien 214e466b29 tls: remove incomplete tests
These tests are incompatible with the incomplete API usage and should
have been pcap based tests in the first place.
4 years ago
Victor Julien 82e03bd8fc tls: set event if record size exceeds limit 4 years ago
Victor Julien e857c864ca tls: support server hello done message 4 years ago
Victor Julien 4bab6e24e5 tls: support handshake fragmentation
Implement TLS record defrag for handshake messages.

Ticket: #5183.
4 years ago
Victor Julien bcaf0f6f7d tls: remove certificate buffering code
TCP Buffering is now done in the app-layer using the incomplete API, on
the SSL/TLS record level. TLS level fragmentation will be implemented
separately.
4 years ago
Victor Julien 0839317ea7 tls: parse handshake protocol records in single pass 4 years ago
Victor Julien 9f0ea5e70c sslv2: use version from client hello
Remove streaming code that is now unused.

Incomplete handling makes this record parsing work on full data.
4 years ago
Victor Julien c8d79fb81f ssl: implement 'incomplete' handling for SSLv2 4 years ago
Victor Julien 6076a51511 tls: streaming mode for application records
To avoid overhead of stream buffering for records we don't do
much with anyway, pass through application records instead of
buffering the entire record in the stream engine.
4 years ago
Victor Julien 129fcb5c72 tls: use incomplete API to get full TLS records
The TLS record header is parsed in streaming mode still, but once the
record size is known we tell the app-layer API to give us the full
record.

Ticket: #5481
4 years ago
Gleb Smirnoff 7110ea75c4 ipfw: remove setting of SO_BROADCAST on the divert(4) socket
My review of the FreeBSD kernel code reveals that this setting
a) is ignored by the kernel b) is not required.  The sending
side of divert(4) never checks so->so_options, but always gives
IP_ALLOWBROADCAST to ip_output().
4 years ago
Andrei Shchapaniak ee5573c4ee dpdk/i40e: fix warning with number of queues for RSS configuration 4 years ago
Philippe Antoine 390cf9248f detect: adds flow.age keyword
Ticket: #5536
4 years ago
Philippe Antoine ce2775d331 flow/icmpv4: fix vlan.use-for-tracking
For ICMPv4 error messages the vlan ids were always considered,
even if the 'vlan.use-for-tracking' option was disabled.

Ticket: #5330
4 years ago
Sascha Steinbiss 148b53125b ebpf: update deprecated API calls
This fixes build errors when libbpf 1.0 is used. It removes previously
deprecated API functions that were still in use in Suricata's eBPF
code.
4 years ago
Jeff Lucovsky 63745a7879 detect/tls: Improve tls.fingerprint rule handling
Issue: 4581

This commit improves the runtime performance of rules with
tls.fingerprint by using the inspection logic from tls.cert_fingerprint.
4 years ago
Jeff Lucovsky 6bccd5aa30 detect/uri: Remove unnecessary include
This commit removes an unnecessary #include for detect-uricontent.h
4 years ago
Victor Julien d941703cd8 detect/build: minor code cleanup 4 years ago
Victor Julien 040404b093 detect/profiling: track bytes scanned by prefilter engines 4 years ago
Victor Julien 682e2a07fe detect/tls: add tls.cert_chain_len keyword 4 years ago
Victor Julien 224ba82569 eve/tls: warn on unsupported 'custom' options 4 years ago
Victor Julien dbf3d1e977 tls: make SSLSetEvent a macro to help debugging 4 years ago
Victor Julien 0e39c92fcf flow-manager: reduce locks at startup
Effectively busy looping on a mutex to wait for time to be ready.
4 years ago
Victor Julien 19e94e93fa common: move u8_tolower to common header 4 years ago
Victor Julien 18e63d4ede htp: remove user setup from request line callback
This used to be the first callback that was called, but its not anymore.

Codecov confirmed that this is no longer used and therefore not useful.
4 years ago