Commit Graph

11237 Commits (33f6a1629092dd8120eb591d2644efec0e2e4ba9)

Author SHA1 Message Date
Victor Julien d57510a10f files: remove unused Rust binding for file pruning 4 years ago
Victor Julien f19b40a7f6 streaming: set size and max regions defaults 4 years ago
Victor Julien 229b82721d htp: remove duplicate prototypes 4 years ago
Victor Julien f788d31f22 htp/body: minor optimization 4 years ago
Victor Julien b62a513c47 flow: inline commonly used getters 4 years ago
Victor Julien 76a256a8b1 streaming: remove inefficient buffer grow logic 4 years ago
Victor Julien 467234659d streaming: remove unused slide logic 4 years ago
Victor Julien 0b36bde9be streaming: remove unused config member 4 years ago
Victor Julien 2cfbefb6c6 streaming: make minimum region gap size configurable 4 years ago
Victor Julien 8e9dac99f3 stream/reassembly: make max-regions configurable; set default 4 years ago
Victor Julien f896f03b7b streaming: add max regions config option 4 years ago
Victor Julien 42d3cd2061 stream/list: hack around GAP handling in tests 4 years ago
Victor Julien d2001ef94b stream: improve gap tests 4 years ago
Victor Julien 96dfd65b96 eve: log max regions 4 years ago
Victor Julien 1bb6f44ff0 stream: flow timeout improvement
Check continuous data for app-layer and post gap data as well for
stream inspection.
4 years ago
Victor Julien 1dac2467c5 streaming: implement memory regions
In TCP, large gaps in the data could lead to an extremely poor utilization
of the streaming buffer memory. This was caused by the implementation using
a single continues memory allocation from the "stream offset" to the
current data. If a 100 byte segment was inserted for ISN + 20MiB, we would
allocate 20MiB, even if only 100 bytes were actually used.

This patch addresses the issue by implementing a list of memory regions.
The StreamingBuffer structure holds a static "main" region, which can be
extended in the form of a simple list of regions.

    [ main region ] [ gap ] [ aux region ]
    [ sbb ]                 [ sbb ]

On insert, find the correct region and see if the new data fits. If it
doesn't, see if we can expand the current region, or than we need to add
a new region. If expanding the current region means we overlap or get
too close to the next region, we merge them.

On sliding, we free any regions that slide out of window and consolidate
auxilary regions into main where needed.

Bug: #4580.
4 years ago
Victor Julien 61e47ad6f5 stream: reduce streaming buffer internals use 4 years ago
Philippe Antoine e22b59b8c1 fuzz: check libpcap timestamp consistency
That is microseconds should be positive
4 years ago
Philippe Antoine 1c436fe0ca fuzz: fix use of uninitialized value
packet timestamp is not set when function returns error.
also use C positive modulo for microseconds
4 years ago
Philippe Antoine 1660172a8b ftp: completely resets port_line
In the case port_line is first allocated and port_line_len is set,
Then a second request reaches memcap and frees port_line,
port_line_len should also be reset, because both will get used
by the response parsing.

Ticket: #5701
4 years ago
Victor Julien b79c14f710 flow-worker: prune frames and stream for pseudo packets as well 4 years ago
Victor Julien af211cbfad debug: add pkt_src to detect/stream entry log 4 years ago
Victor Julien ef2ed35311 stream: add debug statements 4 years ago
Victor Julien db28d85d01 stream: consider any uninspected data 4 years ago
Victor Julien 1a35801f96 stream: minor constification 4 years ago
Victor Julien 989e5c7ba2 streaming/sbb: fix debug message 4 years ago
Victor Julien a00db15bd4 stream: remove bug on from packet path 4 years ago
Victor Julien ee6c39b985 output: improve debug format by adding function name 4 years ago
Victor Julien 41ca206fdf src: fix deprecated-non-prototype compile warnings
Tested on Fedora 37 with clang 15.

util-strlcatu.c:45:8: error: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Werror,-Wdeprecated-non-prototype]
size_t strlcat(dst, src, siz)
           ^
1 error generated.
4 years ago
Victor Julien 1bc6976a06 src: fix unused-but-set-variable compile warnings
Tested on Fedora 37 with clang 15.

datasets.c:852:9: error: variable 'n' set but not used [-Werror,-Wunused-but-set-variable]
    int n = 0;
        ^
1 error generated.
4 years ago
Victor Julien ebd8728219 src: fix strict-prototype warnings
Tested on Fedora 37 with clang 15.

app-layer.c:1055:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerSetupCounters()
                          ^
                           void
app-layer.c:1176:29: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
void AppLayerDeSetupCounters()
                            ^
                             void
2 errors generated.
4 years ago
Philippe Antoine 8c3107472b detect/frames: fix crash when parsing bad rule
Indexing of Signature::init_data::smlists would fail for a rule that
used a frame and fast_pattern w/o content, as the array would only be
expanded when adding a content.
Adding a check to see if there list id is in bounds
is an implicit check for the "no content" case.

Ticket: #5530
4 years ago
Jason Ish 531c4336c9 logging: use ISO-like timestamp in file log
Also update suricata.yaml to show this default, as updating the
"default-log-format" does not affect the file output.
4 years ago
Jason Ish bac8b8d26d logging: add new date format: YYYY-MM-DD HH:MM:SS
Adds a new logging format character, %z that uses a more standard time
format of "YYYY-MM-DD HH:MM:SS".

Ticket #5764
4 years ago
Philippe Antoine 9f21e2bcde src: remove unneeded double colon
git grep ';;' src/ | grep -v for | cut -d: -f1 | uniq
| while read f; do sed -i -e 's/;;/;/' $f; done
4 years ago
Philippe Antoine 0dfaf2802f test: remove one duplicate line 4 years ago
Philippe Antoine 27d2bce1a1 flow: complete stats for app_layer protocol counters
In the case of a gap, or in the case of a flow where one side
is recognized, but the other is not before the end, we still
increase the counters to have consistency between

jq 'select(.event_type=="flow" and .app_proto=="ftp") | .app_proto'  log/eve.json | wc -l
jq 'select(.event_type=="stats") | .stats."app_layer".flow.ftp' log/eve.json

Ticket: #5769
4 years ago
Philippe Antoine fff4475d91 stream: fix memory leak when failing to insert segment
As introduced by commit f848e34bcc

DoInsertSegment can now return `-EINVAL` and there was no
generic handling of error values to return the tcp segment
to its pool.

Ticket: #5777
4 years ago
Jeff Lucovsky e15c9451f2 log/stats: Ensure output exists for the stats thread
Issue: 5198

This commit ensures that a log output destination exists for the stats
thread.
4 years ago
Jeff Lucovsky 018ea2625f log: Use hash table for slot maintenance
Issue: 5198

This commit modifies the threaded logging support to use the hash table
for handling thread/slot mappings. As a result, it's no longer necessary
to provide the thread id when ensuring the log output exists.
4 years ago
Jeff Lucovsky f06aabc32a log: Add hash table for thread/slot mappings
Issue: 5198

This commit adds a hash table to manage thread id to slot mappings. This
ensures that each thread will have its own slot (file output device)
4 years ago
Jeff Lucovsky dfcb429524 detect/cert: Use client side certs
Issue: 5516

This commit modifies the detect logic to choose the certificate based on
the flow direction -- to server or to client.
4 years ago
Victor Julien 23eeba3e35 app/protos: reformat string to proto 4 years ago
Victor Julien 4f6d388740 app-layer: check name vs proto mapping in registration 4 years ago
Victor Julien 0ff88c0315 netmap/runmode: minor code cleanup 4 years ago
Victor Julien 0e5af9e8a7 ioctl: output cleanups 4 years ago
Victor Julien 07008f0da2 netmap: output cleanups 4 years ago
Victor Julien 597601b294 dpdk: don't "fix" broken ips setting 4 years ago
Victor Julien 4e945351a5 dpdk: output cleanups 4 years ago
Victor Julien 7196257f68 nfq: minor error message improvement 4 years ago