Commit Graph

99 Commits (3a6ecab0984745f0cefe3ca3c6dfee1eec7470d8)

Author SHA1 Message Date
Philippe Antoine 999582b149 app-layer: standardize includes with iwyu
Ticket: 945
4 weeks ago
Philippe Antoine 15b1bf4865 rust: bindgen SCHTPFileCloseHandleRange
Ticket: 7762
11 months ago
Philippe Antoine dc7874b2ac rust: bindgen more http range functions used in http2
Ticket: 7667
1 year ago
Victor Julien 73c1c489f7 unittests: don't leak memory in checks
Result of bstr_util_strdup_to_c needs to be freed
1 year ago
Philippe Antoine 1f084f8cb7 rust: move HttpRangeFreeBlock out of SuricataContext
Ticket: 7667
1 year ago
Philippe Antoine c52121586a rust: move AppLayerEvents funcs out of RustContext
Ticket: 7667

just use it directly, with bindgen, and cfg not test
1 year ago
Philippe Antoine 573dd2367e rust: bindgen SCFileFlowFlagsToFlags
Ticket: 7667
1 year ago
Jason Ish 717e06e351 rust/http2: replace rs_ naming with SC 2 years ago
Philippe Antoine 5f84114958 http: minor cleanups for htp-file.c
In preparation of libhtp rust
2 years ago
Philippe Antoine 18dc86f50a http: constify some function arguments
In preparation of libhtp rust
2 years ago
Philippe Antoine 9a8e377a7a http: include htp-libhtp.h from header
Instead of C files requiring it.
In preparation of libhtp rust, to minimize the final commit.
2 years ago
Philippe Antoine 23050d70ed http: aliases for htp opaque htp_tx_t
In preparation of libhtp rust
2 years ago
Philippe Antoine 0d4efe0c0f app-layer: fix -Wshorten-64-to-32 warnings
Ticket: #6186

Warnings about downcast from 64 to 32 bits
2 years ago
Philippe Antoine a262e203f9 src: remove some unused parameters 2 years ago
Philippe Antoine 3a7a4cd581 http: code simplification
removing function unused parameter tx_id in HTPFileOpen
And using directly tx instead of its id in HTPFileOpenWithRange
3 years ago
Philippe Antoine 8f63a8f3bf http1: remove transactions from their list
instead of keeping a NULL pointer in an array

Ticket: #5921
3 years ago
Victor Julien 389f166d78 file: remove FILE_USE_DETECT flag
All implementations were converted to use the logic, so the flag itself
can be removed.
3 years ago
Jeff Lucovsky f2e2576ff4 file/htp: Add logic for file access
Set file inspection sizes and marker for use with detect logic when
opening files by name or as part of a range.

Issue: 4145
3 years ago
Philippe Antoine f2a18e91c4 rust: define AppLayerEventType only in rust
And detect.h does no longer depend on app-layer-events.h
3 years ago
Victor Julien d0f1507c83 htp: simplify streaming buffer config
Use a single static config instead of the per profile config.
4 years ago
Victor Julien e3e55406a7 files: update API and callers to take stream config
This is to allow not storing the stream buffer config in each file.
4 years ago
Victor Julien 39cf5b151a src: includes cleanup
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.

Update files to compile after this.

Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.

Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
4 years ago
Victor Julien 79499e4769 app-layer: move files into transactions
Update APIs to store files in transactions instead of the per flow state.

Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.

Update all protocol implementations to support this.

Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.

Update the filestore keyword to consider a match if filestore output not
enabled.
4 years ago
Victor Julien e250ef6402 debug: remove empty header 4 years ago
Philippe Antoine 02f2602dde src: rework includes as per cppclean 4 years ago
Victor Julien 9537d119b9 http: fix reassembled range file accounting 5 years ago
Victor Julien 6d30f4442c http2: fix file accounting for ranged files
Increment files_opened for tx that 'gets' reassembled ranged file
5 years ago
Philippe Antoine bfcd6cb46a range: validity check when end is bigger than size
Ticket: 5132

Down the line, HttpRangeOpenFileAux assumes the range has a
valid value when doing buflen = end - start + 1;
5 years ago
Philippe Antoine 334b1382e0 http: : fix int warnings
Explicitly truncate file names to UINT16_MAX

Before, they got implicitly truncated, meaning a UINT16_MAX + 1
file name, went to 0 file name (because of modulo 65536)
5 years ago
Jason Ish 7732efbec2 app-layer: include decoder events in app-layer tx data
As most parsers use an events structure we can include it in the
tx_data structure to reduce some boilerplate/housekeeping code
in app-layer parsers.
5 years ago
Philippe Antoine c023116857 range: prevents memory leak of file from HTTP2
If a HTTP2 transaction gets freed before the end of the range
request, we need to have the files container which is in
the state, to transfer owernship of this file to the files
container.

Ticket: 4811
5 years ago
Philippe Antoine bba70607e8 range: checks that end is after start
Otherwise, we end up allocating too much memory
5 years ago
Philippe Antoine 65a6f61004 http: delete obsolete range log
Commit d776d72711
has been transfering ownership of file container

So, we cannot log it
5 years ago
Philippe Antoine 98f84d5a9b http2: follow range requests
Move the content-range parsing code to rust
5 years ago
Philippe Antoine 3b9a4f9f19 range: no final zero in key
for future compatibility with rust
5 years ago
Philippe Antoine 7819f3262f http: more consistent return values for HTPFileOpenWithRange 5 years ago
Philippe Antoine bb7cc6f5e3 http: increase tracked files for ranges as well 5 years ago
Philippe Antoine 3ed38d2d5d http: avoid one lock for range append data
Better structure design to ensure that one flow maximum
is owning and appending into the file, adding fileOwning field.

Adds also a gap field in a range buffer, so that we can
feed the gap on closing, when we are protected from concurrency
by a lock, (lock which got removed in the append path)

Fixes memcap when encountering a duplicate while inserting
in red and black tree

Adds many comments
5 years ago
Victor Julien 7a797631e0 http/range: cleanup and simplification
Simplify locking by using the THashData lock instead of a separate
range lock.

Avoid size_t in function arguments.

Clean up file handling functions.

Implement handling of alloc errors.

Rename yaml entry to byterange

Unify public api naming
5 years ago
Philippe Antoine e82416a415 http/range: reassemble files from different flows with range
adds a container, ie a thread safe hash table whose
key is the filename

keep a tree of unordered ranges, up to a memcap limit

adds HTPFileOpenWithRange to handle like HTPFileOpen
if there is a range : open 2 files, one for the whole reassembled,
and one only for the current range
5 years ago
Victor Julien d74c18ee28 http: support per-tx file accounting 5 years ago
Jeff Lucovsky 1eeb96696b general: Cleanup bool usage 6 years ago
Philippe Antoine 707f027231 protos: renaming ALPROTO_HTTP* constants
Having now ALPROTO_HTTP1, ALPROTO_HTTP2 and ALPROTO_HTTP

Run with 3 sed commands
git grep ALPROTO_HTTP | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP/ALPROTO_HTTP1/g'
git grep ALPROTO_HTTP12 | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP12/ALPROTO_HTTP2/g'
git grep ALPROTO_HTTP1_ANY | cut -d: -f1 | uniq |
 xargs sed -i -e 's/ALPROTO_HTTP1_ANY/ALPROTO_HTTP/g'

and then running clang-format
6 years ago
Victor Julien b6b317cae6 http: enable and fix content range tests 6 years ago
Victor Julien 4b7599af90 http/file: modernize unittests
Part of ticket #2975.
7 years ago
Jason Ish ebcc4db84a file extraction: always prune files after detect
If a keyword like filemd5 was being used without a filestore,
or a file output enabled, it would be pruned before detection
had a chance to match.

Consolidate file pruning to the end of the flow worker so files
are available for detection even when a file output is not
enabled.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2490
7 years ago
Philippe Antoine 2d217e6661 http: fixes overflow in range parsing 7 years ago
Jeff Lucovsky d568e7fadd eve/logging: 2991 Optimize logging by TX
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
7 years ago
Philippe Antoine a3e38861b9 http: adds unit tests for HTPParseContentRange 7 years ago
Philippe Antoine 90ab0b0ec2 http: fix overflow in HTPParseContentRange 7 years ago