Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.
This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
adds a container, ie a thread safe hash table whose
key is the filename
keep a tree of unordered ranges, up to a memcap limit
adds HTPFileOpenWithRange to handle like HTPFileOpen
if there is a range : open 2 files, one for the whole reassembled,
and one only for the current range
This permits to use stream-depth value set for file-store.
Currently if a file is being stored and hits a limit,
such as request or response body, it will be truncated
although file-store.stream-depth is enabled but the file should be
closed and not truncated.
Two unit tests have been added to verify that:
- a file is stored correctly
- chunk's length computation doesn’t cause an underflow
Update HTTP parser to set the min inspect depth per transaction. This
allows for signatures to have their fast_pattern in the HTTP body,
while still being able to inspect the raw stream reliably with it.
The inspect depth is set per transaction as it:
- depends on the per personality config for min inspect size
- is set to the size of the actual body if it is smaller
After the initial inspection is done, it is set to 0 which disables
the feature for the rest of the transaction.
This removes the rescanning flush logic in commit
7e004f52c6 and provides an alternative
fix for bug #2522. The old approach caused too much rescanning of
HTTP body data leading to a performance degradation.
Bug #2522
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Set flags by default:
-Wmissing-prototypes
-Wmissing-declarations
-Wstrict-prototypes
-Wwrite-strings
-Wcast-align
-Wbad-function-cast
-Wformat-security
-Wno-format-nonliteral
-Wmissing-format-attribute
-funsigned-char
Fix minor compiler warnings for these new flags on gcc and clang.
In various scenarios buffers would be checked my MPM more than
once. This was because the buffers would be inspected for a
certain progress value or higher.
For example, for each packet in a file upload, the engine would
not just rerun the 'http client body' MPM on the new data, it
would also rerun the method, uri, headers, cookie, etc MPMs.
This was obviously inefficent, so this patch changes the logic.
The patch only runs the MPM engines when the progress is exactly
the intended progress. If the progress is beyond the desired
value, it is run once. A tracker is added to the app layer API,
where the completed MPMs are tracked.
Implemented for HTTP, TLS and SSH.
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.
In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.
Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.
This patch simply removes the offending code.
Reported-by: The Yahoo pentest team
Add an argument to the registration to indicate which iterator
needs to be used: Stream or HttpBody
Add HttpBody Iterator, calling the logger(s) for each Http body chunk.