Commit Graph

272 Commits (4e1945415273bf58876c5da43c506af71ff5b6bc)

Author SHA1 Message Date
Victor Julien e6f49e5a05 app/frames: implement name to id API for frames 3 years ago
Philippe Antoine e1c0725e05 doc: fix typo lenght/length 3 years ago
Victor Julien c073d5cfbf app-layer: use StreamSlice as input to parsers
Remove input, input_len and flags in favor of stream slice.
3 years ago
Victor Julien 6466296b32 app-layer: add StreamSlice to pass data to parsers
Since object to contain relevant pointer, length, offset, flags to make
it easy to pass these to the parsers.
3 years ago
Pierre Chifflier d67f8f9196 rust/smb: convert parser to nom7 functions (SMB1) 3 years ago
Pierre Chifflier 895a54cea4 rust/smb: convert parser to nom7 functions (DCERPC records) 3 years ago
Pierre Chifflier 8d77ce1ffc rust/smb: convert parser to nom7 functions (SMB2) 3 years ago
Pierre Chifflier 5cadb878ff rust/smb: convert parser to nom7 functions (SMB3) 3 years ago
Pierre Chifflier 4c97dfa851 rust/smb: convert parser to nom7 functions (NTLM/SSP records) 3 years ago
Pierre Chifflier 3da816eb23 rust/smb: convert parser to nom7 functions (NBSS records) 3 years ago
Jason Ish 7732efbec2 app-layer: include decoder events in app-layer tx data
As most parsers use an events structure we can include it in the
tx_data structure to reduce some boilerplate/housekeeping code
in app-layer parsers.
3 years ago
Jason Ish 9c67c634c1 app-layer: include DetectEngineState in AppLayerTxData
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.

This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
3 years ago
Jason Ish fcfc9876ce smb: use generic tx iterator 3 years ago
Shivani Bhardwaj 89cb337930 smb: use Direction enum 3 years ago
Shivani Bhardwaj 0c6e9ac931 files: use Direction enum 3 years ago
Philippe Antoine f37240a3e2 smb: midstream probing checks for netbios message type
If it is available

Bug: #4620.
3 years ago
Modupe Falodun 74c39500c3 rust: fix inherent to string
Bug: OISF#4618
3 years ago
Sam Muhammed 922a453da5 rust(lint): use is_null() instead of ptr::null_mut()
Bug: #4594
3 years ago
Sam Muhammed da0a976e23 rust(lint): use let for binding single value
`match` is better used with binding to multiple variables,
for binding to a single value, `let` statement is recommended.

Bug: #4616
3 years ago
Shivani Bhardwaj 42da0fb5c5 smb: fix broken stream depth setting
The stream depth setting was broken since it was moved to Rust because
of a missing parser for memory values in configuration.
Use get_memval fn from conf.rs to calculate and fetch the correct
values.
4 years ago
Shivani Bhardwaj 1f48714e75 smb: remove futile default port setting 4 years ago
Victor Julien c9cee7af49 smb: add debug validation on file counts 4 years ago
Victor Julien 114d3ba730 smb: count files in tx 4 years ago
Philippe Antoine 9b8be5a650 smb: get file name in case of chained commands 4 years ago
Philippe Antoine 3e5f59e2cb smb: fix parsing of file deletion over SMB1 4 years ago
Philippe Antoine fde753d9d2 smb: recognizes file deletion over SMB2
using set_info_level == SMB2_FILE_DISPOSITION_INFO
4 years ago
Jason Ish b9f10ba22f smb: use derived get_event_info/get_event_info_by_id 4 years ago
Jason Ish 5bf5de3350 rust(lint): don't use unwrap_or for function calls
Calling a function in unwrap_or causes that function to always
be called even when not needed. Instead use unwrap_or_else with
a closure which will only be called when needed.
4 years ago
Jason Ish 602bb05e75 rust(lint): fix redundant closures
This lint checks for a closure where a function can be directly
supplied.  Runtime performance is unchanged, but this makes
less work for the compiler.
4 years ago
Jason Ish 69cf5c9eea rust(lint): remove needless borrows
These are needless borrows (references) as the item is already
a reference.
4 years ago
Jason Ish 363b5f99c3 rust: functions that reference raw pointers are unsafe
Based on the Rust clippy lint that recommends that any public
function that dereferences a raw pointer, mark all FFI functions
that reference raw pointers with build_slice and cast_pointer
as unsafe.

This commits starts by removing the unsafe wrapper inside
the build_slice and cast_pointer macros then marks all
functions that use these macros as unsafe.

Then fix all not_unsafe_ptr_arg_deref warnings from clippy.

Fixes clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#not_unsafe_ptr_arg_deref
4 years ago
Jason Ish 53413f2d7a rust: remove all usage of transmute
All cases of our transmute can be replaced with more idiomatic
solutions and do no require the power of transmute.

When returning an object to C for life-time management, use
Box::into_raw to convert the boxed object to pointer and use
Box::from_raw to convert back.

For cases where we're just returning a pointer to Rust managed
data, use a cast.
4 years ago
Victor Julien aa9d8658ef smb: minor formatting fixup 4 years ago
Victor Julien 094208823b smb: minor code cleanup 4 years ago
Shivani Bhardwaj e5c948df87 smb: Add rust registration function
Get rid of the C glue code and move registration completely to Rust.
4 years ago
Shivani Bhardwaj 27af4bb002 smb: add missing code from rust impl of fns 4 years ago
Shivani Bhardwaj 6420df84b7 smb: Change fn sign as per rust registration requirement
Registering parsers in Rust requires signatures to be a certain way and
compatible with C. Change signatures of all the functions.
4 years ago
Shivani Bhardwaj 4d6b6b5dfe smb: add constants 4 years ago
Philippe Antoine ef5755338f rust: SCLogDebug is real nop when built as release
Before, even if there were no outputs, all the arguments
were evaluated, which could turn expensive

All variables which are used only in certain build configurations
are now prefixed by underscore to avoid warnings
4 years ago
Victor Julien 3587033d9e files: construct with default, free on drop
Update protocols.
4 years ago
Jeff Lucovsky 02dccb1529 smb: Rework constructs to use Self/Default
This commit modifies the constructors to use Self and/or
Default::default() when feasible
4 years ago
Philippe Antoine 8307010255 smb: relax probing parser to handle first NBSS message
cf dcerpc-udp S-V test :
First message is Message Type: Session request (0x81)
Second message is SMB
4 years ago
Philippe Antoine 1b6e81cd72 smb: probing parser for start and midstream
The probing parser is more strict at the start of the stream
4 years ago
Philippe Antoine 9dc5258a21 smb: split probing function for code style
Introduces rs_smb_probe_tcp_midstream
4 years ago
Philippe Antoine 2d14606224 smb: andx support
Add AndX support for SMB1. Finishes #3475.

[Updated by Victor Julien to split functions]
4 years ago
Philippe Antoine aee8e60149 rust: better panic message for missing file config 4 years ago
Philippe Antoine 8ac363c34d rust: fix warning about unused values in smb tests 4 years ago
Juliana Fajardini e33bbee9b7 nbss: add parser tests
Add tests to parse_nbss_record and parse_nbss_record_partial
4 years ago
Juliana Fajardini 5226ba1c15 Rust: generic files definition
Issue: Optimization 3825
- filecontainer: add Files structure, to replace/unify SMBFiles,
NFSFiles and HTTP2Files
- smb/files: delete SMBFiles implementation
- smb/smb: replace SMBFiles with Files
- nfs/nfs: delete NFSFiles implementation, replace its former
 declarations with Files' ones
- http2/http2: replace HTTP2Files with Files
- http2/mod: Delete reference to file files.rs
- http2/files: Delete
4 years ago
Victor Julien efc9a7a398 app-layer: remove callback for completion status
Since the completion status was a constant for all parsers, remove the
callback logic and instead register the values themselves. This should
avoid a lot of unnecessary callback calls.

Update all parsers to take advantage of this.
4 years ago
Philippe Antoine 6181459086 smb: do not rely on one valid NBSS byte for probing
Need to have the SMB header so as to validate
4 years ago
Philippe Antoine aaa69fe3c5 smb: resistance against padding evasions
Scenario is use of dummy padding in write AndX request
or other similar commands using a data offset.

Parsing skips now these dummy bytes, and generates one event
4 years ago
Philippe Antoine caa7946888 smb: adds file overlap event against evasions
Evasion scenario is
- a first dummy write of one byte at offset 0 is done
- the second full write of EICAR at offset 0 is then done
and does not trigger detection

The last write had the final value, and as we cannot "cancel"
the previous write, we set an event which is then transformed into
an app-layer decoder alert
4 years ago
Philippe Antoine 547d6c2d78 applayer: pass parameter to StateAlloc
This parameter is NULL or the pointer to the previous state
for the previous protocol in the case of a protocol change,
for instance from HTTP1 to HTTP2

This way, the new protocol can use the old protocol context.
For instance, HTTP2 mimicks the HTTP1 request, to have a HTTP2
transaction with both request and response
5 years ago
Jason Ish ac93ab281d rust/logging: allow log macros to be used by plugins
Fix plugin macros so they can be used by external Rust crates
such as plugins.
5 years ago
Shivani Bhardwaj 2529eb35e1 smb: use uuid crate 5 years ago
Victor Julien 77a95eddd9 smb: support AppLayerTxData 5 years ago
Victor Julien 65e9a7c31c smb: fix 'dangling' files in lossy sessions
In case of lossy connections the SMB state would properly clean up
transactions, including file transactions. However for files the
state was never set to 'truncated', leading to files to stay 'active'.

This would lead these files staying in the SMB's state. In long running
sessions with lots of files this would lead to performance and memory
use issues.

This patch cleans truncates the file that was being transmitted when
a file transaction is being closed.
5 years ago
Victor Julien 25f2efe977 smb: check post-gap timeouts once a second at most 5 years ago
Victor Julien 8aa380600d smb: update ts only if it changed 5 years ago
Shivani Bhardwaj a7535099b4 smb/eve: convert to jsonbuilder
Closes redmine ticket 3712.
5 years ago
Shivani Bhardwaj 80adf7d1cf smb: Import constants from DCERPC
Remove DCERPC constants to avoid duplicate name errors. Import the
required constants from DCERPC implementation.
5 years ago
Shivani Bhardwaj 6457754fd6 dcerpc: Replace C function calls with Rust
All the dead code in C after the Rust implementation is hereby removed.
Invalid/migrated tests have also been deleted.
All the function calls in C have been replaced with appropriate calls to
Rust functions. Same has been done for smb/detect.rs as a part of this
migration.
5 years ago
Victor Julien 2f5834cdfa rust: merge parser.rs into applayer.rs
Both were serving the same purpose.
5 years ago
Victor Julien 5cd9cfb5a0 smb: update return type of GAP handling 5 years ago
Victor Julien 4bf87d30e4 smb: convert to return AppLayerResult
Support returning 'incomplete' and remove the buffering
code from the parser.
5 years ago
Victor Julien 3bcf948a75 app-layer: change return codes
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.

Introduce two macros:

APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)

Update all parsers to use this.
5 years ago
Victor Julien a729d266c3 smb: fix rustc 1.42 warnings 5 years ago
Pierre Chifflier 442500678b rust: use the streaming version of combinators to fix incomplete reads 5 years ago
Pierre Chifflier f3ddd7127c rust: Add types annotation when required
Unfortunately, the transition to nom 5 (and functions instead of macros)
has side-effects, one of them being requiring lots of types annotations
when using a parsing, for ex in a match instruction.
5 years ago
Pierre Chifflier d1bf34a427 rust/smb: add custom error handling 5 years ago
Pierre Chifflier 1ab8c5763c rust: add SecBlobError custom error type for the Kerberos parser 5 years ago
Pierre Chifflier 5b809f77f1 rust: upgrade all parsers to nom 5 5 years ago
Victor Julien d41aeccea4 smb: handle file transactions post-GAP
After a GAP all normal transactions are closed. File transactions
are left open as they can handle GAPs in principle. However, the
GAP might have contained the closing of a file and therefore it
may remain active until the end of the flow.

This patch introduces a time based heuristic for these transactions.
After the GAP all file transactions are stamped with the current
timestamp. If 60 seconds later a file has seen no update, its marked
as closed.

This is meant to fix resource starvation issues observed in long
running SMB sessions where packet loss was causing GAPs.
5 years ago
Jason Ish e68dfa46a6 rust: cargo fix for Rust 2018 edition 5 years ago
Jason Ish 5ee8323028 rust: remove unnecessary parentheses (Rust 1.40 fixup)
Rust 1.40 in strict mode will now fail the build on the
presence of unnecessary parentheses.

warning: unnecessary parentheses around type
  --> src/smb/smb2_ioctl.rs:41:12
   |
41 |         -> (&mut SMBTransaction)
   |            ^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
   |
   = note: `#[warn(unused_parens)]` on by default
5 years ago
Victor Julien f5b33a070a smb1: allow empty trans records 5 years ago
Victor Julien 40fe29de96 smb1: fix 'event' txs not getting closed
If the only reason we created a request side TX was to set an event,
we would not close it.

This patch always looks up the TX from the response side.
5 years ago
Victor Julien 129cd28058 smb/dcerpc: close request tx sooner 5 years ago
Victor Julien 44ac3e30dd smb: post-GAP handling update
Close all prior transactions in the direction of the GAP, except the
file xfers. Those use their own logic.
5 years ago
Victor Julien a7ee2ffbde smb: winreg is a DCERPC facility 5 years ago
Jason Ish 42e5065ab8 rust: update to Rust 2018 with cargo fix
Migrate to Rust 2018 edition.

Credit to Danny Browning for first demontrating this:
https://github.com/OISF/suricata/pull/3604/commits
6 years ago
Shivani Bhardwaj 8bebea5d4c rust: Get rid of unneeded macros, fix warnings 6 years ago
Victor Julien 579cc9f02b const: constify decoder, app-layer, detect funcs 6 years ago
Jeff Lucovsky 9ccc28baeb rust/smb: Implement get event by id 6 years ago
Victor Julien 3f6624bf16 rust: remove libc crate dependency
Use std::os::raw instead.
6 years ago
Jeff Lucovsky d568e7fadd eve/logging: 2991 Optimize logging by TX
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
6 years ago
Victor Julien b1d4931842 rust: fix warnings about wrong type of comments
"rustdoc does not generate documentation for macro expansions"
6 years ago
Victor Julien bf1bd407dd rust: fix libc deprecation warnings for int types 6 years ago
Victor Julien 422e4892cc proto-detect: improve midstream support
When Suricata picks up a flow it assumes the first packet is
toserver. In a perfect world without packet loss and where all
sessions neatly start after Suricata itself started, this would be
true. However, in reality we have to account for packet loss and
Suricata starting to get packets for flows already active be for
Suricata is (re)started.

The protocol records on the wire would often be able to tell us more
though. For example in SMB1 and SMB2 records there is a flag that
indicates whether the record is a request or a response. This patch
is enabling the procotol detection engine to utilize this information
to 'reverse' the flow.

There are three ways in which this is supported in this patch:

1. patterns for detection are registered per direction. If the proto
   was not recognized in the traffic direction, and midstream is
   enabled, the pattern set for the opposing direction is also
   evaluated. If that matches, the flow is considered to be in the
   wrong direction and is reversed.

2. probing parsers now have a way to feed back their understanding
   of the flow direction. They are now passed the direction as
   Suricata sees the traffic when calling the probing parsers. The
   parser can then see if its own observation matches that, and
   pass back it's own view to the caller.

3. a new pattern + probing parser set up: probing parsers can now
   be registered with a pattern, so that when the pattern matches
   the probing parser is called as well. The probing parser can
   then provide the protocol detection engine with the direction
   of the traffic.

The process of reversing takes a multi step approach as well:

a. reverse the current packets direction
b. reverse most of the flows direction sensitive flags
c. tag the flow as 'reversed'. This is because the 5 tuple is
   *not* reversed, since it is immutable after the flows creation.

Most of the currently registered parsers benefit already:

- HTTP/SMTP/FTP/TLS patterns are registered per direction already
  so they will benefit from the pattern midstream logic in (1)
  above.

- the Rust based SMB parser uses a mix of pattern + probing parser
  as described in (3) above.

- the NFS detection is purely done by probing parser and is updated
  to consider the direction in that parser.

Other protocols, such as DNS, are still to do.

Ticket: #2572
6 years ago
Victor Julien f7a41412d6 smb1: fix NT create andx records filename parsing
Use file name parsing routines that take unicode into account
and consider padding bytes as well.
6 years ago
Wesley van der Ree cc50908f8d smb: fix NT create filename parsing
parse_smb_create_andx_request_record skipped 1 byte too much before
the filename.

Fixes: #2894
6 years ago
Victor Julien 25112ee7e3 rust/smb: fix and optimize record search
Get rid of struct with just a slice reference as well.
6 years ago
Pierre Chifflier f22695130b rust: nom4 requires to add complete!() when using many! combinators 6 years ago
Pierre Chifflier 8c0cde36c6 rust: fix warnings for unused variables (add _) 6 years ago
Pierre Chifflier 13b7399790 rust: upgrade all parsers to nom4 6 years ago
Pierre Chifflier 2f08b3eabd rust/nom4: error_code is superseded by error_position 6 years ago
Victor Julien 8b570c0293 smb: improve request/response mapping
Only use ssn_id and msg_id for mapping a response to a request.

By not using the tree_id it can always be included in the tx.hdr which
means it can be logged properly in case of IOCTL and DCERPC.
6 years ago
Victor Julien 0e40231189 app-layer: improve transaction cleanup handling
The app layers with a custom iterator would skip a tx if during
the ..Cleanup() pass a transaction was removed.

Address this by storing the current index instead of the next
index. Also pass in the next "min_tx_id" to be incremented from
the last TX. Update loops to do this increment.

Also make sure that the min_id is properly updated if the last
TX is removed when out of order.

Finally add a SMB unittest to test this.

Reported by: Ilya Bakhtin
6 years ago
Victor Julien c62273f4fd rust/smb: silence noisy debug messages 6 years ago
Victor Julien 4d5024255f smb/dcerpc: remove now unused ssn2maxsize_map 7 years ago
Victor Julien 4d044483cf smb/dcerpc: clean up and unify DCERPC probe logic 7 years ago
Victor Julien ac4e888597 smb2/dcerpc: probe if response data is dcerpc
If we missed the tree connect we can't know for sure if we're
reading from a (DCERPC) PIPE or not. In this case probe the data
to see if it looks like DCERPC.

If the detection succeeds, use a special 'suricata::dcerpc' service
in the TX.

Simplify handling of DCERPC records that cross records

Update logging for the response only TXs.
7 years ago
Victor Julien 9dd7c38113 smb2: skip rest of READ response if status is not success 7 years ago
Victor Julien edd0c2246c smb1: add SMB1_COMMAND_QUERY_INFO_DISK command mapping 7 years ago
Victor Julien 177966970a smb: probing parser improvement 7 years ago
Victor Julien 2b581cd6db smb: log trans2 that enable delete on close 7 years ago
Victor Julien eefac0ef95 smb1: add support for trans2 set_path_info rename 7 years ago
Victor Julien 1b86d4e1a2 smb: improve dcerpc logic
Detect whether a pipe is a dcerpc channel based on the name of the
pipe.
7 years ago
Victor Julien 7c8a078a2c smb1: improve NT Create response record parsing 7 years ago
Victor Julien 2e6014b15c rust/smb: search for record on midstream start
Calls with both START and MIDSTREAM mean the record might be cut and the
start of it could be missing. For this case, enable the same logic as is
used when catching up after a GAP. Search for the start of the record
instead of assuming it sits exactly at the start of the input data.
7 years ago
Victor Julien f40fc0293b smb: minor optimizations 7 years ago
Victor Julien b1e2783788 auth/krb5: move kerberos5 wrapper to rust root
Make it available outside of just the SMB parser.
7 years ago
Victor Julien 4d58aaae90 smb: clean up partial read/write record handling 7 years ago
Victor Julien aa8d64c2b8 smb: improve skip handling
When skipping records the skip tracker could underflow if the record
parsing had more data than expected.

Enforce the calculation by moving it into a method and make the actual
fields private.
7 years ago
Victor Julien eac7a92200 smb2: improve read/write record parsing
parse_smb2_response_read()/parse_smb2_response_write() can be called on
incomplete data, so they didn't use the read/write length field to grab
the data field. Instead it just used rest(). However in some cases
SMB2 records have trailing data, which would be included in the
READ/WRITE data.

This patch addresses this by using the length field if enough data is
available.
7 years ago
Victor Julien ea1e13cb00 smb: suppress notice messages 7 years ago
Pierre Chifflier 576b8ef722 SMB: simplify code 7 years ago
Pierre Chifflier cf5de0c58e SMB: use String::from_utf8_lossy in logging functions 7 years ago
Pierre Chifflier b5529e4ffb SMB: use kerberos-parser to extract Real and PrincipalName 7 years ago
Victor Julien 0dfb3f0e7f smb1: extract rename info from TRANS2
Exclude TRANS2 from generic TX lookup bypass.
7 years ago
Victor Julien 8eeda113c8 smb1: add parsing for RENAME command 7 years ago
Victor Julien 7b61f2c589 smb2: log renames 7 years ago
Victor Julien 15978d4e85 smb: if filename is missing, use '<unknown>' 7 years ago
Victor Julien 71742ed52b smb: share can't be <share_root> 7 years ago
Victor Julien bc193242ad smb1: add OPEN_ANDX command name for logging 7 years ago
Victor Julien 32b19fac99 smb2: don't log/track each READ/WRITE/etc 7 years ago
Victor Julien fb986abe81 smb: log file FID/GUID as fuid 7 years ago
Victor Julien 816bd022a6 smb1: improve non nt-status handling
Support SRV error, with a couple of codes.
Rename statux field to status_code.
7 years ago
Victor Julien 0519807639 smb1: ignore tree_id in session setup 7 years ago
Victor Julien 286c054472 smb: improve nbss/smb record detection 7 years ago
Victor Julien 7ab071a58d rust/smb: implement minimal record parsing in probing 7 years ago
Victor Julien 283be3cade smb2: break out ioctl handling 7 years ago
Victor Julien bf08285602 smb2: parse async records 7 years ago
Victor Julien 5c26020714 smb2: add ioctl transactions to log the funcs 7 years ago
Victor Julien 75265ec376 smb2: map ioctl funcs to names
List is based on Wireshark's list.
7 years ago
Victor Julien 7cd66516f0 smb: use formal MS names for disposition 7 years ago
Victor Julien f7ed749d4f smb: disable debug output 7 years ago
Victor Julien eed492547c smb1: extract server guid from negotiate 7 years ago
Victor Julien 6d56edc3de smb2: log client and server guid from negotiate 7 years ago
Victor Julien c56f5e11ca smb2: log share type 7 years ago
Victor Julien d75ebdb981 smb: log create empty filename as '<share_root>' like Bro does 7 years ago
Victor Julien fcbeab70a4 smb1: log create 'service' fields 7 years ago
Victor Julien 90e2abaac4 smb1: use generic string parsing for trans 7 years ago
Victor Julien 76917a8732 smb1: generic smb string parse func 7 years ago
Victor Julien 668c747aee smb1: more exact tree connect record parsing 7 years ago
Victor Julien 0ed00cf104 smb: move common parsing funcs into own file 7 years ago
Victor Julien 1c701dc50e smb: make string parsing functions public 7 years ago
Victor Julien 1d4aac1d4d smb1: set event on empty/malformed dialect 7 years ago
Victor Julien c91242e71c smb: rename file to filename in output 7 years ago