Commit Graph

117 Commits (7afc2e3aed27beab3002923dd9c8d249b3a7e3e9)

Author SHA1 Message Date
Victor Julien d0f1507c83 htp: simplify streaming buffer config
Use a single static config instead of the per profile config.
4 years ago
Victor Julien 229b82721d htp: remove duplicate prototypes 4 years ago
Victor Julien 39cf5b151a src: includes cleanup
Work towards making `suricata-common.h` only introduce system headers
and other things that are independent of complex internal Suricata
data structures.

Update files to compile after this.

Remove special DPDK handling for strlcpy and strlcat, as this caused
many compilation failures w/o including DPDK headers for all files.

Remove packet macros from decode.h and move them into their own file,
turn them into functions and rename them to match our function naming
policy.
4 years ago
Victor Julien 79499e4769 app-layer: move files into transactions
Update APIs to store files in transactions instead of the per flow state.

Goal is to avoid the overhead of matching up files and transactions in
cases where there are many of both.

Update all protocol implementations to support this.

Update file logging logic to account for having files in transactions. Instead
of it acting separately on file containers, it is now tied into the
transaction logging.

Update the filestore keyword to consider a match if filestore output not
enabled.
4 years ago
Victor Julien c27df6304d app-layer: introduce common AppLayerStateData API
Add per state structure for storing flags and other variables.
4 years ago
Philippe Antoine 11f849c3ee protocol-change: sets event in case of failure
Protocol change can fail if one protocol change is already
occuring.

Ticket: #5509
4 years ago
Philippe Antoine 02f2602dde src: rework includes as per cppclean 4 years ago
Victor Julien b06bd1a1fe htp: rearrange tx user data for more efficiency 5 years ago
Victor Julien 52ad906d31 htp: implement basic request/response frames 5 years ago
Philippe Antoine 334b1382e0 http: : fix int warnings
Explicitly truncate file names to UINT16_MAX

Before, they got implicitly truncated, meaning a UINT16_MAX + 1
file name, went to 0 file name (because of modulo 65536)
5 years ago
Jason Ish 7732efbec2 app-layer: include decoder events in app-layer tx data
As most parsers use an events structure we can include it in the
tx_data structure to reduce some boilerplate/housekeeping code
in app-layer parsers.
5 years ago
Jason Ish 9c67c634c1 app-layer: include DetectEngineState in AppLayerTxData
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.

This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
5 years ago
Philippe Antoine e82416a415 http/range: reassemble files from different flows with range
adds a container, ie a thread safe hash table whose
key is the filename

keep a tree of unordered ranges, up to a memcap limit

adds HTPFileOpenWithRange to handle like HTPFileOpen
if there is a range : open 2 files, one for the whole reassembled,
and one only for the current range
5 years ago
Philippe Antoine a04b5566a6 http: makes decompression time limit configurable 6 years ago
Philippe Antoine 9b5c923327 http: disables lzma by default for HTTP 6 years ago
Philippe Antoine 7011bddf84 http2: mimic HTTP1 request from upgrade 6 years ago
Victor Julien db3b637ada htp: minor UNITTESTS guarding cleanup 6 years ago
Victor Julien 910922cdc4 htp: support AppLayerTxData 6 years ago
Philippe Antoine 053c728871 http: adds debug check against too many warnings 6 years ago
Victor Julien 3ae1854d2f htp: fix globals use for flags 7 years ago
Philippe Antoine af4f816204 http: sets compression bomb limit 7 years ago
Philippe Antoine 9cbf9ef7a4 HTTP new parser warning for Ambiguous C-L 7 years ago
Giuseppe Longo de904db830 app-layer-htp: use stream depth with filestore
This permits to use stream-depth value set for file-store.

Currently if a file is being stored and hits a limit,
such as request or response body, it will be truncated
although file-store.stream-depth is enabled but the file should be
closed and not truncated.

Two unit tests have been added to verify that:
- a file is stored correctly
- chunk's length computation doesn’t cause an underflow
7 years ago
Jason Ish 61a6eaf330 htp/lzma: set limit from configuration
Also use a default defined in Suricata, not libhtp.
7 years ago
Victor Julien c9c23d5cda htp: set lzma memlimit from config 7 years ago
Jeff Lucovsky 07c05f7dd8 logging/anomaly: TX based logging addition 7 years ago
Philippe Antoine b5f3e03209 New app layer event for invalid http request line
Handles logs from libhtp even if case of error
7 years ago
Philippe Antoine 8a339e73d3 http: adds an event for double encoded uri 7 years ago
Philippe Antoine 3e12066819 http: adds events for each libhtp log
Fixes #997
7 years ago
Philippe Antoine b6b7778e2d http: adds event for header repetition 7 years ago
Victor Julien 9132e4032a files: open files with track id only 7 years ago
Philippe Antoine a1c6e091ac http: new event for auth unrecognized
activates libhtp auth parsing
Fixes #984
7 years ago
Victor Julien e02b74dee7 http: implement min size stream logic
Update HTTP parser to set the min inspect depth per transaction. This
allows for signatures to have their fast_pattern in the HTTP body,
while still being able to inspect the raw stream reliably with it.

The inspect depth is set per transaction as it:
- depends on the per personality config for min inspect size
- is set to the size of the actual body if it is smaller

After the initial inspection is done, it is set to 0 which disables
the feature for the rest of the transaction.

This removes the rescanning flush logic in commit
7e004f52c6 and provides an alternative
fix for bug #2522. The old approach caused too much rescanning of
HTTP body data leading to a performance degradation.

Bug #2522
8 years ago
Victor Julien 269313a53e stream/segments: change packing to reduce size
Change the way fields are ordered to reduce TcpSegment structure
with 8 bytes.
8 years ago
Victor Julien d0cded2523 http: set events for too many layers of compression
libhtp would already issue warnings, but these were not mapped
to events yet.
8 years ago
Victor Julien 7548944b49 app-layer: remove unused HasTxDetectState call
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.

Update app-layer parsers to remove the unused call and update the
modified call.
9 years ago
Giuseppe Longo d0f92e2a56 app-layer-htp: add swf decompression settings
This adds some settings needed to do swf file decompression
under libhtp section in suricata.yaml
9 years ago
Victor Julien e86be22737 htp: remove unused field from tx state 9 years ago
Victor Julien c63b1ce2c6 htp: remove used body operation field 9 years ago
Victor Julien 9ca71beb03 htp: remove usused file flags 9 years ago
Victor Julien daeba48f77 htp: remove usused flags 9 years ago
Victor Julien e64941144e htp: allow HTTP pickup of response data
Now that libhtp can pick up sessions that start with a response
we can enable support for it as well.
9 years ago
Victor Julien 00b0a41b55 http: move from MpmIDs to DetectFlags API 9 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
9 years ago
Victor Julien a0fad6bb7f mpm: run engines as few times as possible
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.

For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.

This was obviously inefficent, so this patch changes the logic.

The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.

Implemented for HTTP, TLS and SSH.
9 years ago
Victor Julien 44022743f2 http: track if request/response have trailers 10 years ago
Victor Julien d6f051cdf9 http: removed unused flags 10 years ago
Victor Julien feafc838db http: make htpstate cfg ptr const 10 years ago
Victor Julien 24a2f51569 http: move body settings into per dir struct 10 years ago
Victor Julien 6fb808fc1a http: add per direction config for body parsing
The HTPCfgDir structure is meant to contain config for per direction
body parsing parameters.

This patch stores the streaming API config.
10 years ago