Commit Graph

1491 Commits (d16f9f3cb77497e9213be12dc5a6ca45039c471a)

Author SHA1 Message Date
Victor Julien f59c43b1c7 smb/ntlmssp: improve version check
Don't assume the ntlmssp version field is always present if the flag is
set. Instead keep track of the offsets of the data of the various blobs
and see if there is space for the version.

Inspired by how Wireshark does the parsing.

Bug: #7121.
2 years ago
Philippe Antoine 647e878f7c detect: helper function for multibuffer 2 years ago
Philippe Antoine 5bd17934df http2: do not expand duplicate headers
Ticket: 7104

As this can cause a big mamory allocation due to the quadratic
nature of the HPACK compression.
2 years ago
Philippe Antoine 37509e8e0e modbus: abort flow parsing on flood
Ticket: 6987

Let's not spend more resources for a flow which is trying to
make us do it...
2 years ago
Jeff Lucovsky 5b97f4040c detect/base64: Use Rust defined modes everywhere
Issue: 6487

To avoid ambiguity, a single definition for base 64 decoding modes will
be used. The Rust base64 transform contains the definitions for the
existing mode types: Strict, RFC2045, RFC4648
2 years ago
Jeff Lucovsky 1823681709 detect/transform: from_base64 option parsing
Issue: 6487

Implement from_base64 option parsing in Rust. The Rust module also
contains unit tests.
2 years ago
Jeff Lucovsky ab0cb960a1 detect/parser: Refactor utility routines
Refactor utility functions/definitions from the byte_math module into
the parser module. This includes parse_var and ResultValue

Issue: 6487
2 years ago
Philippe Antoine 5f35035928 filestore: do not try to store a file set to nostore
Ticket: 6390

This can happen with keyword filestore:both,flow
If one direction does not have a signature group with a filestore,
the file is set to nostore on opening, until a signature in
the other direction tries to set it to store.
Subsequent files will be stored in both directions as flow flags
are now set.
2 years ago
Philippe Antoine 5bb5b4f46f rust: remove unnecessary nested unsafe 2 years ago
Philippe Antoine 4ccbcc4684 sip: use right slice to take line from
We iterate over input, but we are now at start.
Avois quadratic complexity turning to OOM.

Ticket: 7093
2 years ago
Jason Ish 49ecf37126 rust/ike: prefix never read field names with _
New warning from rustc.

The other option is to allow dead code, however this is more explicit,
and when they are read, its obvious they should be renamed.
2 years ago
Jason Ish 29d7ff026a rust: simply matches with unwrap_or_default
New default clippy warning:
https://rust-lang.github.io/rust-clippy/master/index.html#manual_unwrap_or_default
2 years ago
Jason Ish ee2175cdb6 rust: fix clippy lint for legacy_numeric_constants
https://rust-lang.github.io/rust-clippy/master/index.html#legacy_numeric_constants
2 years ago
Jason Ish a1bb62c059 cargo: use default-features instead of default_features
"default_features" is being deprecated in Rust 2024.
2 years ago
Philippe Antoine 4fe3f04fa3 detect/enip: move keywords to rust
Ticket: 4863
2 years ago
Philippe Antoine ce1eea4ad6 detect/websocket: move keywords to rust
Ticket: 4863
2 years ago
Philippe Antoine 16952d67e7 detect/dhcp: move keywords to rust
Ticket: 4863
2 years ago
Philippe Antoine ae72376ebe detect/snmp: move keywords to rust
Ticket: 4863

On the way, convert unit test DetectSNMPCommunityTest to a SV test.

And also, make snmp.pdu_type use a generic uint32 for detection,
allowing operators, instead of just equality.
2 years ago
Philippe Antoine 4bbe7d92dc detect: helper to have pure rust keywords
detect: make number of keywords dynamic

Ticket: 4683
2 years ago
Philippe Antoine 08c511f1bf enip: remove unnecessary unsafe
As the function SCEnipRegisterParsers is already marked as unsafe
2 years ago
Victor Julien 37be66eef9 detect/iprep: update function naming
Bring in line with new Rust code naming for FFI functions.
2 years ago
Victor Julien 83976a4cd4 detect/iprep: implement isset and isnotset
Implement special "isset" and "isnotset" modes.

"isset" matches if an IP address is part of an iprep category with any
value.

It is internally implemented as ">=,0", which should always be true if
there is a value to evaluate, as valid reputation values are 0-127.

"isnotset" matches if an IP address is not part of an iprep category.

Internally it is implemented outside the uint support.

Ticket: #6857.
2 years ago
Victor Julien 539ab3a404 detect/iprep: update keyword parser for extendibility 2 years ago
Jason Ish f0dbfe863d misc: prefix functions with SC not Sc 2 years ago
Philippe Antoine d9d5170ec0 websocket: add data frame
Ticket: 7051
2 years ago
Juliana Fajardini bb45ac71ef dns: allow triggering raw stream reassembly
For TCP streams, app proto stream reassembly can start earlier, instead
of waiting and queueing up data before doing so.

Task #7018
Related to
Bug #7004
2 years ago
Philippe Antoine 82c03f72c3 enip: convert to rust
Ticket: 3958

- transactions are now bidirectional
- there is a logger
- gap support is improved with probing for resync
- frames support
- app-layer events
- enip_command keyword accepts now string enumeration as values.
- add enip.status keyword
- add keywords :
    enip.product_name, enip.protocol_version, enip.revision,
    enip.identity_status, enip.state, enip.serial, enip.product_code,
    enip.device_type, enip.vendor_id, enip.capabilities,
    enip.cip_attribute, enip.cip_class, enip.cip_instance,
    enip.cip_status, enip.cip_extendedstatus
2 years ago
Philippe Antoine 0d267e29a5 files: remove the need for state in callbacks
As files now belong to transactions
2 years ago
Philippe Antoine 5167ff6411 smtp/mime: look for urls in base64 message
Ticket: 5185

Previously, it was looked for message in plain text, and base64
encoding was only handled for attachments.

This commit also fixes the buffering got such base64 data streamed
into urls finding, by buffering a beginning non-empty line,
and by ensuring that we run extraction on the last line,
even if it had no EOL.
2 years ago
Juliana Fajardini 0946c213cd pgsql: trigger raw stream reassembly
Expose the raw stream earlier to the detection engine, as Pgsql can have
multiple messages per transaction and usually will have a message
complete within one TCP packet.

Bug #7000

Related to
Bug #7026
2 years ago
Juliana Fajardini 69e26de197 pgsql/logger: open json object from logger function
Before, the JsonBuilder object for the pgsql event was being created
from the C-side function that actually called the Rust logger.

This resulted that if another module - such as the Json Alert called the
PGSQL logger, we wouldn't have the `pgsql` key present in the log output
- only its inner fields.

Bug #6983
2 years ago
Philippe Antoine 7fb10676e7 dns: remove unneeded mut in logger 2 years ago
Philippe Antoine a10c1f1dde smtp: use rust for mime parsing
Ticket: #3487
2 years ago
Philippe Antoine 5f75b9a6e3 http: use rust for mime parsing
Ticket: #3487
2 years ago
Philippe Antoine 5555aa6788 mime: improved token parsing
Accepts escaped quote in escaped string
2 years ago
Jason Ish 936930778c rust/Makefile: cleanup "clean" targets
Remove maintainer-clean-local, this is not needed.

In distclean-local, remove "rust/dist" and "rust/vendor" as they are
created during "make dist".

In "clean-local", remove "rust/target" and "rust/gen" as they are
created during a normal "make".
2 years ago
Jason Ish 2e440169d6 lua: remove lua as a compile time feature
Its always built-in. However, can be disabled at runtime.
2 years ago
Jason Ish 1fd2c1a379 rust/lua: remove lua_int8 feature
Now that we're fixed to Lua 5.4, the integer size is always 8.
2 years ago
Jason Ish bc011f2205 lua: use rust crate to vendor (bundle) lua
Remove lua-dev(el) from all CI tests.
2 years ago
Shivani Bhardwaj 3a1c12414a tls: store list of subject alternative names
So far, the SANs were available as a part of IssuerDN via x509_parser
crate but SANs were not available to the SSLState* to be directly used
to setup and match against a sticky buffer.
Expose it to SSLStateConnp.

Feature 5234
2 years ago
Jason Ish 8560564657 rust: rename .cargo/config to .cargo/config.toml
Addresses this warning from the Rust compiler:

warning: `../rust/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
2 years ago
Philippe Antoine 3b5f1173ab Revert "rust: temporary: disable debug assertions"
This reverts commit 14ab9aa763.
2 years ago
Philippe Antoine 37a9003736 rust/probing: safety check for null input
Ticket: 7013

Done consistently for all protocols

This may change some protocols behaviors which failed early
if they found there was not enough data...
2 years ago
Philippe Antoine 5dc8dea869 rust: return empty slice without using from_raw_parts
As this triggers rustc 1.78
unsafe precondition(s) violated: slice::from_raw_parts requires
the pointer to be aligned and non-null,
and the total size of the slice not to exceed `isize::MAX`

Ticket: 7013
2 years ago
Philippe Antoine 806052d762 websocket: fix opcodes values for ping/pong
And also set close

Ticket: 7025
2 years ago
Philippe Antoine 8b103ae755 dns: set tx id for frames 2 years ago
Philippe Antoine 715bf048ee frames: rust API makes tx_id explicit
And set it right for SIP and websocket,
so that relevant tx app-layer metadata gets logged.

Ticket: 6973
2 years ago
Jeff Lucovsky cb56752bf7 config/ja3: Eliminate warnings when JA3 is disabled
This commit eliminates warnings when either ja3, ja4 or both are
disabled.
2 years ago
Jason Ish 14ab9aa763 rust: temporary: disable debug assertions 3 years ago
Philippe Antoine c53e9ac0dd sdp: fix logging medias
As introduced by bff790b6ac

Also handles errors in the caller

Ticket: 6994
3 years ago
Jason Ish df8568ee30 rust/dns: visibility cleanups
Remove pub from functions that don't require it.
3 years ago
Jason Ish 556cfe56bf rust/dns: ffi naming and visibility cleanups
- Remove no_mangle and pub from FFI functions that are only accessed
  with a function pointer.
- Rename all no_mangle FFI functions to our C naming scheme.
3 years ago
Giuseppe Longo 868493529b rust/sip: parse and log sdp
If SDP payload is found within a SIP message, it will be parsed and then
logged.

Ticket #6627
3 years ago
Giuseppe Longo bff790b6ac rust/sdp: implement logger
This implements a logger for the SDP protocol.
Given that SDP is encapsulated within other protocols (such as SIP),
enabling it separately is not necessary.

Ticket #6627
3 years ago
Giuseppe Longo 1ccfc35214 rust/sdp: implement protocol parser
This implements a parser for the SDP protocol.
Given that SDP is encapsulated within other protocols (such as SIP),
enabling it separately is not necessary.

Ticket #6627.
3 years ago
Philippe Antoine d8e0c72644 app-layer: remove unused C wrappers
Ticket: 4083
3 years ago
Philippe Antoine 03442c9071 http2: do not log duplicate headers
Ticket: 6900

And thus avoid DOS by logging a request using a compressed
header block repeated many times and having a long value...
3 years ago
Philippe Antoine 390f09692e http2: use a reference counter for headers
Ticket: 6892

As HTTP hpack header compression allows one single byte to
express a previously seen arbitrary-size header block (name+value)
we should avoid to copy the vectors data, but just point
to the same data, while reamining memory safe, even in the case
of later headers eviction from the dybnamic table.

Rust std solution is Rc, and the use of clone, so long as the
data is accessed by only one thread.
3 years ago
Philippe Antoine 0291d37009 websocket: configurable logging of payload in alerts 3 years ago
Philippe Antoine 44b6aa5e4b app-layer: websockets protocol support
Ticket: 2695
3 years ago
Philippe Antoine f83ec543e3 http2: add settings from newer RFCs
Including the one for websocket over HTTP/2
3 years ago
Juliana Fajardini ce1556cefd pgsql: check for eol when parsing response
It was brought to my attention by GLongo that Pgsql parser handled eof
diffrently for requests and responses, and apparently there isn't a good
reason for such a difference therefore, apply same logic used for
rs_pgsql_parse_request for checking for eof when parsing a response.
3 years ago
Sascha Steinbiss 120313f4da ja4: implement for TLS and QUIC
Ticket: OISF#6379
3 years ago
Sascha Steinbiss 9d0db71ebf ja3: make feature compile time configurable 3 years ago
Philippe Antoine 49caf005a4 detect/analyzer: create tojson function for generic integers
As will be needed for tcp.mss
3 years ago
Philippe Antoine 3643b6ed4b output: generic simple tx json logger
Ticket: 3827
3 years ago
Jason Ish 1657b6ff3c cargo: add description and license
Required for publishing to crates.io.
3 years ago
Jason Ish b9127e8b96 automake/rust: remove path.lib
Remove the path.lib parameter that is substituted into the output
Cargo.toml by autoconf. Instead, as part of the build, "cd" into the
source directory. We already set the Rust target directory to the
external build directory.

This makes the Cargo.toml more generic, and in a format suitable for
publishing to crates.io. It also makes it easier to pull in external
crates without needing to patch up their Cargo.toml, for example, it
might make pulling libhtp-rs easier.
3 years ago
Jason Ish 71f59e529c jsonbuilder: fix serialization of nan and inf
When outputting a float, check if its infinity, or not a number and
output a null instead.

Using a null was chosen as this is what serde_yaml, Firefox, Chrome,
Node, etc. do.

Ticket: #6921
3 years ago
Philippe Antoine ee50fe4c30 sip: convert transaction list to vecdeque
Ticket: 6891

So as to avoid quadratic complexity on tx cleanup with SIP/TCP
that can create many transactions in one go.
3 years ago
Philippe Antoine f7cde8f00e rust/smb: fix clippy nightly warning
error: unnecessary use of `to_vec`
    --> src/smb/smb.rs:1048:62
     |
1048 |         let (name, is_dcerpc) = match self.guid2name_map.get(&guid.to_vec()) {
     |                                                              ^^^^^^^^^^^^^^ help: replace it with: `guid`
     |
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_to_owned
     = note: `#[deny(clippy::unnecessary_to_owned)]` implied by `#[deny(warnings)]`

And also other uses of to_vec() on already Vec
3 years ago
Philippe Antoine 02f2fb8833 rust: fix clippy 1.77 warning
Ticket: 6883

error: field `0` is never read
  --> src/asn1/mod.rs:36:14
   |
36 |     BerError(Err<der_parser::error::BerError>),
   |     -------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |     |
   |     field in this variant
   |
3 years ago
Victor Julien a2c817243f rust: add MSRV as rust-version
Update github-actions to use it for the MSRV check.
3 years ago
Philippe Antoine c4b8fb7aca ssh: limit length for banner logs
Ticket: 6770
3 years ago
Philippe Antoine 271ed2008b ssh: avoid quadratic complexity from long banner
Ticket: 6799

When we find an overlong banner, we get into the state just
waiting for end of line, and we just want to skip the bytes
until then.
Returning AppLayerResult::incomplete made TCP engine retain
the bytes and grow the buffer that we parsed again and again...
3 years ago
Hadiqa Alamdar Bukhari 3aa313d0c5 dns: add dns.rcode keyword
dns.rcode matches the rcode header field in DNS messages
It's an unsigned integer
valid ranges = [0-15]
Does not support prefilter
Supports matches in both flow directions

Task #6621
3 years ago
Victor Julien 1d3a156179 rust: update parser dependencies
Time locked to 0.3.20 to guarantee MSRV of 1.63.
Update snmp-parser to 0.10.0.
Update asn1-rs to 0.6.1.
Update kerberos-parser to 0.8.0.
Update x509-parser 0.16.0.
Update der-parser to 9.0.0.
Remove specific use of der-parser 6.

Ticket: #6817.
Ticket: #6818.
3 years ago
Hadiqa Alamdar Bukhari 4b81851097 dns: add dns.rrtype keyword
It matches the rrtype field in DNS
It's an unsigned integer match
valid ranges = [0-65535]
Does not support prefilter
Supports flow in both directions
Feature #6666
3 years ago
Giuseppe Longo fe77def816 rust/sip: register pattern matching
This permits to detect the SIP protocol using pattern matching instead of
probing parser.

Since it is no longer used, the respective probing functions have been removed.
3 years ago
Giuseppe Longo 9c9b1a4230 rust/sip: add direction to transaction
This patch permits to set a direction when a new transaction is created in order
to avoid 'signature shadowing' as reported by Eric Leblond in commit
5aaf50760f
3 years ago
Giuseppe Longo c9d309219e rust/sip: register parser for tcp
This patch lets the parser to work over tcp protocol, taking care of handling
data before calling the request/response parsers.

Ticket #3351.
3 years ago
Giuseppe Longo 69f841c998 sip/parser: enforce valid chars for sip version
The `is_version_char` function incorrectly allowed characters that are not
part of the valid SIP version "SIP/2.0".

For instance, 'HTTP/1.1' was mistakenly accepted as a valid SIP version,
although it's not.

This commit fixes the issue by updating the condition to strictly
check for the correct version string.
3 years ago
Giuseppe Longo 7e993d5081 sip/parser: accept valid chars
Accepts valid characters as defined in RFC3261.
3 years ago
Giuseppe Longo 8ff80cb84d rust/sip: rustfmt sip module 3 years ago
Jeff Lucovsky f9a20dafc6 mqtt: Improve frame parsing w/mult. PDUs
This commit improves the mqtt parsing of frames to handle multiple PDUs.

Issue: 6592
3 years ago
Philippe Antoine c99d93c257 app-layer/template: use a max number of txs
Ticket: 6773
3 years ago
Philippe Antoine 68b0052018 rust: fix clippy ptr_arg warnings
error: writing `&Vec` instead of `&[_]` involves a new object where a slice will do
   --> src/dns/log.rs:371:29
    |
371 | pub fn dns_print_addr(addr: &Vec<u8>) -> std::string::String {
    |                             ^^^^^^^^ help: change this to: `&[u8]`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
3 years ago
Philippe Antoine 80abc22f64 http2: limit number of concurrent transactions
Ticket: 6481

Instead of just setting the old transactions to a drop state so
that they get later cleaned up by Suricata, fail creating new ones.

This is because one call to app-layer parsing can create many
transactions, and quadratic complexity could happen in one
single app-layer parsing because of find_or_create_tx
3 years ago
Philippe Antoine 86de7cffa7 pgsql: parse only PDU when type is unknown
A next PDU may already be in the slice to parse.
Do not skip its parsing, ie do not use rest, but take just
the length of the pdu
3 years ago
Philippe Antoine f52c033e56 pgsql: parse auth message within its bound
If the next PDU is already in the slice next, do not use it and
restrict ourselves to the length of this PDU.
Avoids overconsumption of memory by quadratic complexity, when
having many small PDUS in one big chunk being parsed

Ticket: #6411
3 years ago
Philippe Antoine aff54f29f8 http2: handle reassembly for continuation frames
Ticket: 5926

HTTP2 continuation frames are defined in RFC 9113.
They allow header blocks to be split over multiple HTTP2 frames.
For Suricata to process correctly these header blocks, it
must do the reassembly of the payload of these HTTP2 frames.
Otherwise, we get incomplete decoding for headers names and/or
values while decoding a single frame.

Design is to add a field to the HTTP2 state, as the RFC states that
these continuation frames form a discrete unit :
> Field blocks MUST be transmitted as a contiguous sequence of frames,
> with no interleaved frames of any other type or from any other stream.
So, we do not have to duplicate this reassembly field per stream id.

Another design choice is to wait for the reassembly to be complete
before doing any decoding, to avoid quadratic complexity on partially
decoding of the data.
3 years ago
Philippe Antoine f6e1a20215 detect: dns.opcode as first-class integer
Ticket: 5446

That means it can accept ranges
3 years ago
Philippe Antoine d05f3ac791 detect: integer keywords now accept bitmasks
Ticket: 6648

Like &0x40=0x40 to test for a specific bit set
3 years ago
Philippe Antoine 370ac05419 detect/integer: rust derive for enumerations
Ticket: 6647

Allows keywords using integers to use strings in signature
parsing based on a rust enumeration with a derive.
3 years ago
Philippe Antoine 06c5dd3133 detect: integer keywords now accept negated ranges
Ticket: 6646
3 years ago
Philippe Antoine 3b65a2bb61 detect: integer keywords now support hexadecimal
So that we can write enip.revision: 0x203

Ticket: 6645
3 years ago
Philippe Antoine 38db51b878 rust: make cargo clippy clean
Fixing single_match and manual_find intertwined with SCLogDebug
3 years ago
Philippe Antoine 89936b6530 mqtt: fix logic when setting event
Especially sets transactions to complete when we get a response
without having seen the request, so that the transactions
end up getting cleaned (instead of living/leaking in the state).

Also try to set the event on the relevant transaction, instead
of creating a new transaction just for the purpose of having
the event.

Ticket: #6299
3 years ago
jason taylor 3cb7112aa5 detect: update smb.version keyword
Signed-off-by: jason taylor <jtfas90@gmail.com>
3 years ago
jason taylor bfc0790d87 rust: fix rustfmt warnings for smb detect
Signed-off-by: jason taylor <jtfas90@gmail.com>
3 years ago