Commit Graph

286 Commits (2dc6b6ee14b0886783674fecaaa50e62328f9727)

Author SHA1 Message Date
Victor Julien 7bc3c3ac6e app-layer: pass STREAM_* flags to parser
Pass the STREAM_* flags to the app-layer parser functions so that
the parser can know more about how it is called.
7 years ago
Victor Julien 26b61bad90 htp: cleanup and fix test 7 years ago
Victor Julien 08af5ddd88 http: add tests for header folding
To test for https://github.com/OISF/libhtp/issues/159
7 years ago
Victor Julien 953dceece2 http: clean up unittest 7 years ago
Jason Ish c411519605 app-layer: remove has events callback - not used 8 years ago
Victor Julien 7548944b49 app-layer: remove unused HasTxDetectState call
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.

Update app-layer parsers to remove the unused call and update the
modified call.
8 years ago
Giuseppe Longo d0f92e2a56 app-layer-htp: add swf decompression settings
This adds some settings needed to do swf file decompression
under libhtp section in suricata.yaml
8 years ago
Victor Julien 6f339abdf0 htp: minor debug addition 8 years ago
Victor Julien c63b1ce2c6 htp: remove used body operation field 8 years ago
Victor Julien 07cbbfb0d1 htp: code cleanups 8 years ago
Victor Julien daeba48f77 htp: remove usused flags 8 years ago
Victor Julien e64941144e htp: allow HTTP pickup of response data
Now that libhtp can pick up sessions that start with a response
we can enable support for it as well.
8 years ago
Victor Julien 49927024c6 http: add tests for malformed response lines 8 years ago
Martin Natano 18f64e0d21 app-layer-htp, stream-tcp: prevent modulo bias in RandomGetWrap()
RAND_MAX is not guaranteed to be a divisor of ULONG_MAX, so take the
necessary precautions to get unbiased random numbers. Although the
bias might be negligible, it's not advisable to rely on it.
8 years ago
Victor Julien 1df00749df detect: fix multiple files per tx inspect
Fix the inspection of multiple files in a single TX, where new files
may be added to the TX after inspection started.

Assign the hard coded id DE_STATE_FLAG_FILE_INSPECT to the file
inspect engine.

Make sure that sigs that do file inspection and don't match on the
current file always store a detailed state. This state will include
the DE_STATE_FLAG_FILE_INSPECT flag.

When the app-layer indicates a new file is available, for each sig
that has the DE_STATE_FLAG_FILE_INSPECT flag set, reset part of the
state so that the sig is evaluated again.
8 years ago
Victor Julien 00b0a41b55 http: move from MpmIDs to DetectFlags API 8 years ago
Victor Julien 56b1df1b4d http: clean up & improve unittests 8 years ago
Victor Julien bca0cd71ae app-layer: use logger bits to avoid looping
Avoid looping in transaction output.

Update app-layer API to store the bits in one step
and retrieve the bits in a single step as well.

Update users of the API.
8 years ago
Giuseppe Longo 7addc24566 htp: destroy atomic vars
Atomic vars declared in app-layer-htp-mem.c were not
destroyed when suricata is shutdown.
8 years ago
Victor Julien 7fb58e6783 random: fix random logic with getrandom
The older random functions returned random values in the range of
0 - RAND_MAX. This is what the http randomize code was expecting.

Newer methods, based on getrandom (or probably Windows too), return
a much large range of values, including negative values and >RAND_MAX.

This patch adds a wrapper to turn the returned value into the expected
range before using it in the http code.

The same is true for the stream engine.
8 years ago
Victor Julien 5c01b40931 tests: update tests for app-layer changes 8 years ago
Victor Julien 6f42ae91c7 app-layer: protocol change API
Add API calls to upgrade to TLS or to request a protocol change
without a specific protocol expectation.

If the HTTP CONNECT session includes a port on the url, use that to
look up the probing parser during protocol detection. Solves a
missed detection of a SSLv2 session that upgrades to TLSv1. SSLv2
relies on the probing parser which is limited to certain ports.

In case of STARTTLS in SMTP and FTP, the port is hardcoded to 443.

A new event APPLAYER_UNEXPECTED_PROTOCOL is set if there was a
mismatch.
8 years ago
Mats Klepsland b6c2b7052b app-layer-htp: add HTTP CONNECT support 8 years ago
Victor Julien ab1200fbd7 compiler: more strict compiler warnings
Set flags by default:

    -Wmissing-prototypes
    -Wmissing-declarations
    -Wstrict-prototypes
    -Wwrite-strings
    -Wcast-align
    -Wbad-function-cast
    -Wformat-security
    -Wno-format-nonliteral
    -Wmissing-format-attribute
    -funsigned-char

Fix minor compiler warnings for these new flags on gcc and clang.
8 years ago
Victor Julien a0fad6bb7f mpm: run engines as few times as possible
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.

For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.

This was obviously inefficent, so this patch changes the logic.

The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.

Implemented for HTTP, TLS and SSH.
8 years ago
Victor Julien 2d223b69cd stream: set 'trigger raw' per direction 8 years ago
Victor Julien c3b4dd5a7d qa: add --no-random commandline option 8 years ago
Victor Julien 535d9e3529 random: convert stream and htp to new call 8 years ago
Victor Julien ec964ebf84 mingw: don't use uint type as mingw doesn't have it 8 years ago
Victor Julien 579d6d3ff7 http: allow lower/mixed case in proto detect
In HTTP detection registered patterns were upper case only. Since the
detection is based on both sides this would still work for sessions
where one of the talkers misbehaved. If both sides misbehave this
would fail however, so this patch introduces case insensive matching.
9 years ago
Victor Julien 44022743f2 http: track if request/response have trailers 9 years ago
Giuseppe Longo 675fa56497 app-layer: add ThreadVars to AppLayerParserParse
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
9 years ago
Victor Julien 6530c3d0d8 unittests: replace SCMutex* calls by FLOWLOCK_* 9 years ago
Victor Julien 5ec885e451 http: set of response body decompress limit
This is a per personality setting.
9 years ago
Victor Julien b3bf7a5729 output: introduce config and perf output levels
Goal is to reduce info output
9 years ago
Victor Julien feafc838db http: make htpstate cfg ptr const 9 years ago
Victor Julien 24a2f51569 http: move body settings into per dir struct 9 years ago
Victor Julien 6fb808fc1a http: add per direction config for body parsing
The HTPCfgDir structure is meant to contain config for per direction
body parsing parameters.

This patch stores the streaming API config.
9 years ago
Victor Julien 46e55f1e34 http body handling: use streaming buffer API
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.

In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.

Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
9 years ago
Mats Klepsland c52fe9a5fa http: register logger functions 9 years ago
Mats Klepsland c4b918b6c4 tx: do not store ProgressCompletionStatus per ipproto
Change AppLayerParserRegisterGetStateProgressCompletionStatus to
only store one ProgressCompletionStatus callback function for each
alproto, instead of storing one for each ipproto.

This enables us to use AppLayerParserGetStateProgressCompletionStatus
in functions where we do not know the ipproto used.
9 years ago
Victor Julien 83e0529b2b http: flag destate about new files
The stateful detection engine needs some assistance when inspecting
transactions with multiple files. This patch flags the detect state
(if any) about the availability of new files in http. For http it
should only apply to multipart bodies although the flag is set for
all files.
9 years ago
Victor Julien 077ac81688 QA: direct access from commandline to AppLayer API
This patch introduces a new set of commandline options meant for
assisting in fuzz testing the app layer implementations.

Per protocol, 2 commandline options are added:

--afl-http-request=<filename>
--afl-http=<filename>

In the former case, the contents of the file are passed directly to
the HTTP parser as request data.

In the latter case, the data is devided between request and responses.
First 64 bytes are request, then next 64 are response, next 64 are
request, etc, etc.
9 years ago
Jason Ish 796dd5223b tests: no longer necessary to provide successful return code
1 pass, 0 is fail.
9 years ago
Victor Julien 444c4b546e htp: optimize multipart parsing
Store boundary in a single stack buffer instead of 2 heap allocated
buffers.
10 years ago
Victor Julien 665cafbe3b htp: fix boundry error handling (CID 1153934) 10 years ago
Victor Julien bfc4be230c htp: fix dead code warning (CID 1153933) 10 years ago
Victor Julien 83f400dd60 http: remove unused param from HtpBodyAppendChunk 10 years ago
Giuseppe Longo 7a29aa116e http: close file when http body limit is reached
In some conditions, if stream.reassembly.depth is greater than
request/response-body-limit size, the logging output is wrong
if filestore keyword is used with http.

For example, we get:
{... "app_proto":"http","fileinfo":{"filename":"\/file.pdf","state":"CLOSED","stored":false,"size":1049292,"tx_id":0}}

"state":"CLOSED","stored":false should be "state":"TRUNCATED","stored":true.

This happens because the file state and file flags,
which is the information that determine a correct output,
are not set properly since a file is logged before and then closed (HTPFileClose).

The logic of this patch is to close a file when we are above
the limits, such that the proper state and flags can be set
and the file will be logged correctly.
10 years ago
Victor Julien b25bd2e180 http: fix NULL deref on certain out of memory conditions 10 years ago