Commit Graph

107 Commits (7732efbec2a35db5c9a9b73f8c373d99813f3a14)

Author SHA1 Message Date
Jason Ish 7732efbec2 app-layer: include decoder events in app-layer tx data
As most parsers use an events structure we can include it in the
tx_data structure to reduce some boilerplate/housekeeping code
in app-layer parsers.
5 years ago
Jason Ish 9c67c634c1 app-layer: include DetectEngineState in AppLayerTxData
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.

This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
5 years ago
Philippe Antoine e82416a415 http/range: reassemble files from different flows with range
adds a container, ie a thread safe hash table whose
key is the filename

keep a tree of unordered ranges, up to a memcap limit

adds HTPFileOpenWithRange to handle like HTPFileOpen
if there is a range : open 2 files, one for the whole reassembled,
and one only for the current range
5 years ago
Philippe Antoine a04b5566a6 http: makes decompression time limit configurable 6 years ago
Philippe Antoine 9b5c923327 http: disables lzma by default for HTTP 6 years ago
Philippe Antoine 7011bddf84 http2: mimic HTTP1 request from upgrade 6 years ago
Victor Julien db3b637ada htp: minor UNITTESTS guarding cleanup 6 years ago
Victor Julien 910922cdc4 htp: support AppLayerTxData 6 years ago
Philippe Antoine 053c728871 http: adds debug check against too many warnings 7 years ago
Victor Julien 3ae1854d2f htp: fix globals use for flags 7 years ago
Philippe Antoine af4f816204 http: sets compression bomb limit 7 years ago
Philippe Antoine 9cbf9ef7a4 HTTP new parser warning for Ambiguous C-L 7 years ago
Giuseppe Longo de904db830 app-layer-htp: use stream depth with filestore
This permits to use stream-depth value set for file-store.

Currently if a file is being stored and hits a limit,
such as request or response body, it will be truncated
although file-store.stream-depth is enabled but the file should be
closed and not truncated.

Two unit tests have been added to verify that:
- a file is stored correctly
- chunk's length computation doesn’t cause an underflow
7 years ago
Jason Ish 61a6eaf330 htp/lzma: set limit from configuration
Also use a default defined in Suricata, not libhtp.
7 years ago
Victor Julien c9c23d5cda htp: set lzma memlimit from config 7 years ago
Jeff Lucovsky 07c05f7dd8 logging/anomaly: TX based logging addition 7 years ago
Philippe Antoine b5f3e03209 New app layer event for invalid http request line
Handles logs from libhtp even if case of error
7 years ago
Philippe Antoine 8a339e73d3 http: adds an event for double encoded uri 8 years ago
Philippe Antoine 3e12066819 http: adds events for each libhtp log
Fixes #997
8 years ago
Philippe Antoine b6b7778e2d http: adds event for header repetition 8 years ago
Victor Julien 9132e4032a files: open files with track id only 8 years ago
Philippe Antoine a1c6e091ac http: new event for auth unrecognized
activates libhtp auth parsing
Fixes #984
8 years ago
Victor Julien e02b74dee7 http: implement min size stream logic
Update HTTP parser to set the min inspect depth per transaction. This
allows for signatures to have their fast_pattern in the HTTP body,
while still being able to inspect the raw stream reliably with it.

The inspect depth is set per transaction as it:
- depends on the per personality config for min inspect size
- is set to the size of the actual body if it is smaller

After the initial inspection is done, it is set to 0 which disables
the feature for the rest of the transaction.

This removes the rescanning flush logic in commit
7e004f52c6 and provides an alternative
fix for bug #2522. The old approach caused too much rescanning of
HTTP body data leading to a performance degradation.

Bug #2522
8 years ago
Victor Julien 269313a53e stream/segments: change packing to reduce size
Change the way fields are ordered to reduce TcpSegment structure
with 8 bytes.
8 years ago
Victor Julien d0cded2523 http: set events for too many layers of compression
libhtp would already issue warnings, but these were not mapped
to events yet.
8 years ago
Victor Julien 7548944b49 app-layer: remove unused HasTxDetectState call
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.

Update app-layer parsers to remove the unused call and update the
modified call.
9 years ago
Giuseppe Longo d0f92e2a56 app-layer-htp: add swf decompression settings
This adds some settings needed to do swf file decompression
under libhtp section in suricata.yaml
9 years ago
Victor Julien e86be22737 htp: remove unused field from tx state 9 years ago
Victor Julien c63b1ce2c6 htp: remove used body operation field 9 years ago
Victor Julien 9ca71beb03 htp: remove usused file flags 9 years ago
Victor Julien daeba48f77 htp: remove usused flags 9 years ago
Victor Julien e64941144e htp: allow HTTP pickup of response data
Now that libhtp can pick up sessions that start with a response
we can enable support for it as well.
9 years ago
Victor Julien 00b0a41b55 http: move from MpmIDs to DetectFlags API 9 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
10 years ago
Victor Julien a0fad6bb7f mpm: run engines as few times as possible
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.

For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.

This was obviously inefficent, so this patch changes the logic.

The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.

Implemented for HTTP, TLS and SSH.
10 years ago
Victor Julien 44022743f2 http: track if request/response have trailers 10 years ago
Victor Julien d6f051cdf9 http: removed unused flags 10 years ago
Victor Julien feafc838db http: make htpstate cfg ptr const 11 years ago
Victor Julien 24a2f51569 http: move body settings into per dir struct 11 years ago
Victor Julien 6fb808fc1a http: add per direction config for body parsing
The HTPCfgDir structure is meant to contain config for per direction
body parsing parameters.

This patch stores the streaming API config.
11 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
11 years ago
Mats Klepsland c52fe9a5fa http: register logger functions 11 years ago
Giuseppe Longo a459376d2e app-layer-htp: add http_body_inline setting 12 years ago
Victor Julien f536099a67 app-layer: de_state optimization
Add API to bypass expensive TX list walks. This API call is optional.

Implement it for HTTP and DNS.
12 years ago
Victor Julien 52195a4192 http: add event for leading spaces on request line
Libhtp will issue a warning in this case, so we can match on this.
This patch adds event, rule and unittest.
12 years ago
Victor Julien 6279da0fbd http: support per TX destate storage 12 years ago
Victor Julien e78e33a428 http: add event for suspicious method delimeter
Add event and rule for suspicious delim(s) between method and uri.

Add unittests as well.
12 years ago
Victor Julien 5ad7198dc0 http: add libhtp uri warning event
Add event for libhtp warning added 0.5.17 for URI's with suspicious
delimeters.
12 years ago
Victor Julien e90714afd6 http: remove unused and broken 'content-len' logic
The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.

This patch simply removes the offending code.

Reported-by: The Yahoo pentest team
12 years ago
Victor Julien bac6c3ab02 streaming logger: support Http Body logging
Add an argument to the registration to indicate which iterator
needs to be used: Stream or HttpBody

Add HttpBody Iterator, calling the logger(s) for each Http body chunk.
12 years ago