The input data received in DATA and BDAT command modes can be huge and
could have important data, like a legit huge email. Therefore, exempt
these from the line buffering limits which were introduced to regulate
the size of lines that we buffer at any point in time.
As a part of this patch, anything that comes under DATA or BDAT is
processed early without buffering as and when it arrives. The ways of
processing remain the same as before.
Issue
-----
So far, with the SMTP parser, we would buffer data up until an LF char
was found indicating the end of one line. This would happen in case of
fragmented data where a line might come broken into multiple chunks.
This was problematic if there was a really long line without any LF
character. It would mean that we'd keep buffering data up until we
encounter one such LF char which may be many many bytes of data later.
Fix
---
Fix this issue by setting an upper limit of 4KB on the buffering of
lines. If the limit is reached then we save the data into current line
and process it as if it were a regular request/response up until 4KB
only. Any data after 4KB is discarded up until there is a new LF char in
the received input.
Cases
-----
1. Fragmentation
The limit is enforced for any cases where a line of >= 4KB comes as diff
fragments that are each/some < 4KB.
2. Single too long line
The limit is also enforced for any cases where a single line exceeds the
limit of buffer.
Reported by Victor Julien.
Ticket 5023
Ticket: 4948
This is not the perfect solution, but it prevents to trigger
the assert, and keep the assert.
A better solution would need to create transaction from
the reponse parsing, in case a later command was buffered and
not answered. But this would not be enough as NoNewTx prevents
the creation of a new transaction for RSET...
Parse extract-url-schemes from the mime config.
e.g. 'extract-urls-schemes: [http, https, ftp, mailto]'
Update MimeDecConfig struct to new url extraction fields.
Change app-layer-smtp.c & util-decode-mime.c to initialize new struct
fields for MimeDecConfig.
Sets the default value for extract-url-schemes if not found in the
config to 'extract-urls-schemes: [http]' for backwards compatibility.
Uses the schemes defined in the mime config value for
extract-urls-schemes to search for URLS starting with those scheme
names followed by "://".
Logs the URLS with the scheme + '://' at the start if the
log-url-scheme is set in the mime config, otherwise the old behaviour
is reverted to and the urls are logged with the schemes stripped.
Removed unused constant URL_STR now that URLS are being searched for
using extract-urls-schemes mime config values instead of just URL's
starting with 'http://'.
Added commented out new options for extract-urls-schemes and
log-url-scheme to suricata.yaml.in.
Update FindUrlStrings comments.
Remove old outdated comments/commented code from FindUrlStrings.
Update test case for mime which now needs schemes list to be set.
Add Test Cases for FindUrlStrings() method.
Feature: #2054
Every transaction has an existing mandatory field, tx_data. As
DetectEngineState is also mandatory, include it in tx_data.
This allows us to remove the boilerplate every app-layer has
for managing detect engine state.
Since the completion status was a constant for all parsers, remove the
callback logic and instead register the values themselves. This should
avoid a lot of unnecessary callback calls.
Update all parsers to take advantage of this.
This parameter is NULL or the pointer to the previous state
for the previous protocol in the case of a protocol change,
for instance from HTTP1 to HTTP2
This way, the new protocol can use the old protocol context.
For instance, HTTP2 mimicks the HTTP1 request, to have a HTTP2
transaction with both request and response
This commit improves how the parser handles the `RSET` command.
Termination of the transaction occurs when the `RSET` ack is seen (reply
code 250).
Bug: #3677
This patch simplifies the return codes app-layer parsers use,
in preparation of a patch set for overhauling the return type.
Introduce two macros:
APP_LAYER_OK (value 0)
APP_LAYER_ERROR (value -1)
Update all parsers to use this.
Now that we call the file prune loop very regularly, we can move the
SMTP specific inspection pruning logic into this loop. Helps with
cases there we don't (often) update a files inspection trackers.
Implement min_inspect_depth for SMTP so that file_data and
regular stream matches don't go out of sync on the stream start.
Added toserver bytes tracking.
Bug #3190.
If a keyword like filemd5 was being used without a filestore,
or a file output enabled, it would be pruned before detection
had a chance to match.
Consolidate file pruning to the end of the flow worker so files
are available for detection even when a file output is not
enabled.
Redmine issue:
https://redmine.openinfosecfoundation.org/issues/2490
This changeset makes changes to the TX logging path. Since the txn
is passed to the TX logger, the TX can be used directly instead of
through the TX id.
Add a raw-extraction option for smtp. When enabled, this feature will
store the raw e-mail inside a file, including headers, e-mail content,
attachments (base64 encoded). This content is stored in a normal File *,
allowing for normal file detection.
It'd also allow for all-emails extraction if a rule has
detect-filename:"rawmsg" matcher (and filestore).
Note that this feature is in contrast with decode-mime.
This feature is disabled by default, and will be disabled automatically
if decode-mime is enabled.
Due to missing error handling, a bad mime message could put the
mime parser in an error state, without the SMTP layer taking this
into account. So the SMTP layer would continue to pass data to the
mime parser, even though it was in an error state.
When the parser would be fed a very long line while in this state,
it would try to set an error flag in the state. However, due to
the error state, this setting of the flag would dereference a null
pointer.
This patch fixes this issue by updating the mime parser to check
the state it is in when receiving new input. It will refuse to
process futher data while in the error state. It will also return
a new error code to indicate to the SMTP layer that the parser
was in an error state.
Also remove the now useless 'state' argument from the SetTxDetectState
calls. For those app-layer parsers that use a state == tx approach,
the state pointer is passed as tx.
Update app-layer parsers to remove the unused call and update the
modified call.
Victor Julien
Shivani Bhardwaj
Philippe Antoine
Aaron Bungay
Jason Ish
Victor Julien
Jeff Lucovsky
Maurizio Abba