As SMTP file_data detection uses the file API, the file's inspect
tracker should be considered when pruning files.
This patch sets the FILE_USE_DETECT flag on files tracked by smtp.
It also adds logic to move inspected tracker ahead if detection
doesn't do it, like when no rules are matching or detection engine
is disabled.
This patch fixes the following leak:
Direct leak of 9982880 byte(s) in 2902 object(s) allocated from:
#0 0x4c253b in malloc ??:?
#1 0x10c39ac in MimeDecInitParser /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/util-decode-mime.c:2379
#2 0x6a0f91 in SMTPProcessRequest /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1085
#3 0x697658 in SMTPParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1185
#4 0x68fa7a in SMTPParseClientRecord /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-smtp.c:1208
#5 0x6561c5 in AppLayerParserParse /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer-parser.c:908
#6 0x53dc2e in AppLayerHandleTCPData /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/app-layer.c:444
#7 0xf8e0af in DoReassemble /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:2635
#8 0xf8c3f8 in StreamTcpReassembleAppLayer /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3028
#9 0xf94267 in StreamTcpReassembleHandleSegmentUpdateACK /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3404
#10 0xf9643d in StreamTcpReassembleHandleSegment /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp-reassemble.c:3432
#11 0xf578b4 in HandleEstablishedPacketToClient /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2245
#12 0xeea3c7 in StreamTcpPacketStateEstablished /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:2489
#13 0xec1d38 in StreamTcpPacket /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:4568
#14 0xeb0e16 in StreamTcp /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/stream-tcp.c:5064
#15 0xff52a4 in TmThreadsSlotVarRun /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:130
#16 0xffdad1 in TmThreadsSlotVar /home/victor/qa/buildbot/donkey/z600fuzz/Private/src/tm-threads.c:474
#17 0x7f7cd678d181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 (discriminator 2)
We come to this case when a SMTP session contains at least 2 mails
and then the ending of the first is not correctly detected. In that
case, switching to a new tx seems a good solution. This way we still
have partial logging.
If SMTP session is weird then we may reach a state where a field
like MAIL FROM is seen as duplicated.
Valgrind output is:
30 bytes in 1 blocks are definitely lost in loss record 96 of 399
at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
by 0x4A5803: SMTPParseCommandWithParam (app-layer-smtp.c:996)
by 0x4A4DCE: SMTPParseCommandMAILFROM (app-layer-smtp.c:1016)
by 0x4A3F55: SMTPProcessRequest (app-layer-smtp.c:1127)
by 0x4A1F8C: SMTPParse (app-layer-smtp.c:1191)
by 0x493AD7: SMTPParseClientRecord (app-layer-smtp.c:1214)
by 0x4878A6: AppLayerParserParse (app-layer-parser.c:908)
by 0x42384E: AppLayerHandleTCPData (app-layer.c:444)
by 0x8D7EAD: DoReassemble (stream-tcp-reassemble.c:2635)
by 0x8D795F: StreamTcpReassembleAppLayer (stream-tcp-reassemble.c:3028)
by 0x8D8BE0: StreamTcpReassembleHandleSegmentUpdateACK (stream-tcp-reassemble.c:3404)
by 0x8D8F6E: StreamTcpReassembleHandleSegment (stream-tcp-reassemble.c:3432)
Previous code was wrongly supposing the lines are ending with CRLF.
But this is not the case so we must propagate the length of end of
line delimiter to be able to compute the md5 of the mail body.
Instead, intruduce StreamTcpDisableAppLayer to disable app layer
tracking and reassembly. StreamTcpAppLayerIsDisabled can be used
to check it.
Replace all uses of FlowSetSessionNoApplayerInspectionFlag and
the FLOW_NO_APPLAYER_INSPECTION.
If a boundary was longer than 254 bytes a stack overflow would result
in mime decoding.
Ticket #1449
Reported-by: Kostya Kortchinsky of the Google Security Team
cc1: warnings being treated as errors
app-layer-smtp.c: In function ‘SMTPParseCommandBDAT’:
app-layer-smtp.c:908: warning: dereferencing type-punned pointer will break strict-aliasing rules
Fix issue where SMTPStateGetTxCnt would return the actual active tx'.
The 'GetCnt' API call is not named correctly. It should be 'GetMaxId',
as this is actually the expected behavior.
A tx is considered complete after the data command completed. However,
this would lead to RSET and QUIT commands setting up a new tx.
This patch simply adds a check that refuses to setup a new tx when these
commands are encountered after the data portion is complete.
Turn all buffers into uint8_t (from char) and no longer use the
string functions like strncpy/strncasecmp on them.
Store url and field names as lowercase, and also search/compare
them as lowercase. This allows us to use SCMemcmp.
For AppLayerThreadCtx, AppLayerParserState, AppLayerParserThreadCtx
and AppLayerProtoDetectThreadCtx, use opaque pointers instead of
void pointers.
AppLayerParserState is declared in flow.h as it's part of the Flow
structure.
AppLayerThreadCtx is declared in decode.h, as it's part of the
DecodeThreadVars structure.
app-layer.[ch], app-layer-detect-proto.[ch] and app-layer-parser.[ch].
Things addressed in this commit:
- Brings out a proper separation between protocol detection phase and the
parser phase.
- The dns app layer now is registered such that we don't use "dnstcp" and
"dnsudp" in the rules. A user who previously wrote a rule like this -
"alert dnstcp....." or
"alert dnsudp....."
would now have to use,
alert dns (ipproto:tcp;) or
alert udp (app-layer-protocol:dns;) or
alert ip (ipproto:udp; app-layer-protocol:dns;)
The same rules extend to other another such protocol, dcerpc.
- The app layer parser api now takes in the ipproto while registering
callbacks.
- The app inspection/detection engine also takes an ipproto.
- All app layer parser functions now take direction as STREAM_TOSERVER or
STREAM_TOCLIENT, as opposed to 0 or 1, which was taken by some of the
functions.
- FlowInitialize() and FlowRecycle() now resets proto to 0. This is
needed by unittests, which would try to clean the flow, and that would
call the api, AppLayerParserCleanupParserState(), which would try to
clean the app state, but the app layer now needs an ipproto to figure
out which api to internally call to clean the state, and if the ipproto
is 0, it would return without trying to clean the state.
- A lot of unittests are now updated where if they are using a flow and
they need to use the app layer, we would set a flow ipproto.
- The "app-layer" section in the yaml conf has also been updated as well.
Victor Julien
Eric Leblond
Maurizio Abba
Giuseppe Longo
Tom DeCanio
David Abarbanel
Anoop Saldanha