In HTTP detection registered patterns were upper case only. Since the
detection is based on both sides this would still work for sessions
where one of the talkers misbehaved. If both sides misbehave this
would fail however, so this patch introduces case insensive matching.
To be able to add a transaction counter we will need a ThreadVars
in the AppLayerParserParse function.
This function is massively used in unittests
and this result in an long commit.
Convert HTTP body handling to use the Streaming Buffer API. This means
the HtpBodyChunks no longer maintain their own data segments, but
instead add their data to the StreamingBuffer instance in the HtpBody
structure.
In case the HtpBodyChunk needs to access it's data it can do so still
through the Streaming Buffer API.
Updates & simplifies the various users of the reassembled bodies:
multipart parsing and the detection engine.
Change AppLayerParserRegisterGetStateProgressCompletionStatus to
only store one ProgressCompletionStatus callback function for each
alproto, instead of storing one for each ipproto.
This enables us to use AppLayerParserGetStateProgressCompletionStatus
in functions where we do not know the ipproto used.
The stateful detection engine needs some assistance when inspecting
transactions with multiple files. This patch flags the detect state
(if any) about the availability of new files in http. For http it
should only apply to multipart bodies although the flag is set for
all files.
This patch introduces a new set of commandline options meant for
assisting in fuzz testing the app layer implementations.
Per protocol, 2 commandline options are added:
--afl-http-request=<filename>
--afl-http=<filename>
In the former case, the contents of the file are passed directly to
the HTTP parser as request data.
In the latter case, the data is devided between request and responses.
First 64 bytes are request, then next 64 are response, next 64 are
request, etc, etc.
In some conditions, if stream.reassembly.depth is greater than
request/response-body-limit size, the logging output is wrong
if filestore keyword is used with http.
For example, we get:
{... "app_proto":"http","fileinfo":{"filename":"\/file.pdf","state":"CLOSED","stored":false,"size":1049292,"tx_id":0}}
"state":"CLOSED","stored":false should be "state":"TRUNCATED","stored":true.
This happens because the file state and file flags,
which is the information that determine a correct output,
are not set properly since a file is logged before and then closed (HTPFileClose).
The logic of this patch is to close a file when we are above
the limits, such that the proper state and flags can be set
and the file will be logged correctly.
Optimize HTTP multipart body parsing. Big records that were not files
could slow down Suricata. The reason was that the body tracker was not
moved forward. This lead to growing body buffers, which were expensive
wrt memory and inspection.
This patch add logic to move the tracker forward in this case.
In case the body wasn't inspected the body_inspected variable wouldn't
get updated leading to the body not getting pruned at all.
This patch adds support for this case.
It's not uncommon to see an header like:
X-Forwarded-For: 1.2.3.4:56789
This patch recognizes this case and ignores the port. It also supports
this for IPv6 if the address has the following notation:
X-Forwarded-For: [12::34]:1234
This patch also adds unittests.
On receiving TCP end of stream packets (e.g. RST, but also sometimes FIN
packets), in some cases the AppLayer parser would not be notified. This
could happen in IDS mode, but would especially be an issue in IPS mode.
This patch changes the logic of the AppLayer API to handle this. When no
new data is available, and the stream ends, the AppLayer API now gets
called with a NULL/0 input, but with the EOF flag set.
This allows the AppLayer parser to call it's final routines still in the
context of a real packet.
Added more WebDAV functions. A complete list of what http
methods libhtp can handle can be found at:
https://github.com/OISF/libhtp/blob/0.5.x/htp/htp_core.h#L260.
So now the methods array reflects these available functions.
The comments have also been changed to reflect the desired style.
1) Reworked pattern registration for http methods and versions.
Instead of being a manual and verbose action of adding one
and one http method with N-amount if prefix spacings and
the same for HTTP versions (eg. HTTP/1.1) i moved it all
to be loop based actions reading values from char arrays.
In the future all that is needed is to add new methods
to the arrays and they will be added as a pattern.
2) Modified pattern registration after feedback.
Changed variable used in snprintf for http method registration
Should have been size of dest buffer at not another var (catsize)
that i had created. Also removed this variable.
Fixed a typo in the comment for registering http versions.
TO_CIENT -> TO_CLIENT.
The HTTP tracking code would parse the content lenght and store it
in the TX user data. It didn't take the possibility or errors into
account though, leading to a possible negative int being cases to
unsigned int. Luckily, the result was unused.
This patch simply removes the offending code.
Reported-by: The Yahoo pentest team
Victor Julien
Giuseppe Longo
Mats Klepsland
Jason Ish
Giuseppe Longo
maxtors
Ken Steele
sxhlinux