Remove local copies from each MPM file and use include file instead.
Might be better to also add util-memcpy.c rather than inlining it each time,
to get smaller code, since only seems to be used at initialization.
This is required because SCMemcmpLowercase() expects it first argument
to be already lowercase for the comparison. This is done by using
memcpy_tolower() for NO_CASE patterns.
This addresses code review comments from Victor.
The case_state in MPMs was just to track when a pid could have no-case and
case-sensitive matches for the same PID. Now that can't happen after fixing
bug 1110, so remove the code and storage for case_state.
This fixes bug 1110. When assigning PIDs, use the NO_CASE flag when comparing
for duplicates. The state of the flag must be the same, but also use the same
type of comparisons when checking for duplicates.
Previously, "foo":CS would match with "foo":CI when it should not.
and "foo":CI would not match "FoO":CI when it should. Both of those
cases are fixed with this change.
This then allows simplifying the use of pid in MPMs because now if they
pids match, then so do the flags, so checking the flags is not required.
In the stream engine, Init() can fail if the memcap is reached. In this
case the segment was not freed by PoolGet:
==8600== Thread 1:
==8600== 70,480 bytes in 1,762 blocks are definitely lost in loss record 611 of 612
==8600== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8600== by 0x914CC8: TcpSegmentPoolAlloc (stream-tcp-reassemble.c:166)
==8600== by 0xA0D315: PoolGet (util-pool.c:297)
==8600== by 0x9302CD: StreamTcpGetSegment (stream-tcp-reassemble.c:3768)
==8600== by 0x921FE8: StreamTcpReassembleHandleSegmentHandleData (stream-tcp-reassemble.c:1873)
==8600== by 0x92EEDA: StreamTcpReassembleHandleSegment (stream-tcp-reassemble.c:3584)
==8600== by 0x8D3BB1: HandleEstablishedPacketToServer (stream-tcp.c:1969)
==8600== by 0x8D7F98: StreamTcpPacketStateEstablished (stream-tcp.c:2323)
==8600== by 0x8F13B8: StreamTcpPacket (stream-tcp.c:4243)
==8600== by 0x8F2537: StreamTcp (stream-tcp.c:4485)
==8600== by 0x95DFBB: TmThreadsSlotVarRun (tm-threads.c:559)
==8600== by 0x8BE60D: TmThreadsSlotProcessPkt (tm-threads.h:142)
tcp.segment_memcap_drop | PcapFile | 1762
This patch fixes PoolGet to both Cleanup and Free the Alloc'd data in
case Init fails.
==15745== 3 bytes in 1 blocks are definitely lost in loss record 5 of 615
==15745== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745== by 0x71858C1: strdup (strdup.c:42)
==15745== by 0xA20814: SCProtoNameInit (util-proto-name.c:75)
==15745== by 0x952D1B: PostConfLoadedSetup (suricata.c:1983)
==15745== by 0x9537CD: main (suricata.c:2112)
Also, clean up and add a check to make sure it's initialized only once.
The full path of the script names is stored in a buffer that wasn't
freed at exit.
==24195== 41 bytes in 1 blocks are definitely lost in loss record 300 of 613
==24195== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24195== by 0x565D06: DetectLoadCompleteSigPath (detect.c:251)
==24195== by 0x7CABE8: DetectLuajitParse (detect-luajit.c:595)
==24195== by 0x7CD2AE: DetectLuajitSetup (detect-luajit.c:827)
==24195== by 0x7DC273: SigParseOptions (detect-parse.c:547)
==24195== by 0x7DDC75: SigParse (detect-parse.c:856)
==24195== by 0x7E1C2B: SigInitHelper (detect-parse.c:1336)
==24195== by 0x7E2968: SigInit (detect-parse.c:1559)
==24195== by 0x7E37B1: DetectEngineAppendSig (detect-parse.c:1831)
==24195== by 0x566D17: DetectLoadSigFile (detect.c:335)
==24195== by 0x567636: SigLoadSignatures (detect.c:423)
==24195== by 0x951A97: LoadSignatures (suricata.c:1816)
This patch frees the buffer.
For packets that were freed, not recycled, profiling memory wasn't
freed:
==15745== 13,312 bytes in 8 blocks are definitely lost in loss record 611 of 615
==15745== at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745== by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745== by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745== by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745== by 0x8413BF: FlowForceReassemblyForHash (flow-timeout.c:588)
==15745== by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745== by 0x9540F6: main (suricata.c:2296)
==15745==
==15745== 14,976 bytes in 9 blocks are definitely lost in loss record 612 of 615
==15745== at 0x4C2C494: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15745== by 0xA190D5: SCProfilePacketStart (util-profiling.c:963)
==15745== by 0x4E4345: PacketGetFromAlloc (decode.c:134)
==15745== by 0x83FE75: FlowForceReassemblyPseudoPacketGet (flow-timeout.c:276)
==15745== by 0x841508: FlowForceReassemblyForHash (flow-timeout.c:620)
==15745== by 0x841897: FlowForceReassembly (flow-timeout.c:716)
==15745== by 0x9540F6: main (suricata.c:2296)
This patch addresses that.
If lock profiling was compiled in, but disabled in the config a
serious memory leak condition was triggered.
Valgrind output:
==11169== 9,091,248 bytes in 189,401 blocks are definitely lost in loss record 564 of 564
==11169== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==11169== by 0xABC44C: LockRecordAdd (util-profiling-locks.c:112)
==11169== by 0xABC950: SCProfilingAddPacketLocks (util-profiling-locks.c:141)
==11169== by 0xA04CD5: TmThreadsSlotVarRun (tm-threads.c:562)
==11169== by 0x958793: TmThreadsSlotProcessPkt (tm-threads.h:142)
==11169== by 0x9599C3: PcapFileCallbackLoop (source-pcap-file.c:172)
==11169== by 0x56FC130: ??? (in /usr/lib/x86_64-linux-gnu/libpcap.so.1.4.0)
==11169== by 0x959D24: ReceivePcapFileLoop (source-pcap-file.c:210)
==11169== by 0xA05B9E: TmThreadsSlotPktAcqLoop (tm-threads.c:703)
==11169== by 0x6155F6D: start_thread (pthread_create.c:311)
==11169== by 0x6E399CC: clone (clone.S:113)
Some of the packets counters were using a 32bit integer. Given the
bandwidth that is often seen, this is not a good idea. This patch
switches to 64bit counter.
Packets counter is incremented in AFPDumpCounters and it was
also incremented during packet reading. The result was a value
that is twice the expected result.
Spotted-by: Victor Julien <victor@inliniac.net>
Until now a PoolInit failure for the segment pools would result in
an abort() through BUG_ON(). This patch adds a proper error message,
then exits.
Bug #1108.
When TcpSegmentPoolInit fails (e.g. because of a too low memcap),
it would free the segment. However, the segment memory is managed
by the Pool API, which would also free the same memory location.
This patch fixes that.
Also, memset the structure before any checks are done, as the segment
memory is passed to TcpSegmentPoolCleanup in case of error as well.
Bug #1108
Fixes:
** CID 1075221: Dereference after null check (FORWARD_NULL)
/src/suricata.c: 1344 in ParseCommandLine()
The reason it gave this warning is that in other paths using optarg
there was a check, so the checker assumed optarg can be NULL.
This patch fixes:
** CID 1187544: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 268 in DecodeICMPV6()
** CID 1187545: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 270 in DecodeICMPV6()
** CID 1187546: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 272 in DecodeICMPV6()
** CID 1187547: Missing break in switch (MISSING_BREAK)
/src/decode-icmpv6.c: 274 in DecodeICMPV6()
It duplicates the logic instead of adding 'fall through' statements
as the debug statements were wrong and confusing. For ND_REDIRECT
all 5 ND_* types would have been printed.
By assuming that HTPCallbackRequestLine would always be run first,
an memory leak was introduced. It would not check if user data already
existed in the tx, causing it to overwrite the user data pointer is
it already existed.
Bug #1092.
In really short Suricata runtimes the capture counters would not
be updated. This patch does a force update at the end of the
capture loops in pcap and af-packet.
The probing parser registration function
AppLayerProtoDetectPPParseConfPorts was a void, meaning it would
give no feedback to the registering protocol implementation. If a
config was missing, it would just give up.
This patch changes it to return a bool. 0 if no config was found,
1 if a config was found.
This allows the caller to setup a default case.
The probing parser detection ports yaml settings of the TCP part
of the DNS parser accidentally used udp as protocol string, causing
the wrong part of the YAML to be evaluated.
If the protocol is disabled, app-layer-event would print a cryptic
error message. This patch makes sure we inform the user the protocol
is in fact disabled.
This patch introduces a new counter "decoder.vlan_qinq". It counts
packets that have more than two stacked vlan layers.
Packets with 2 vlan layers will both increment "decoder.vlan" and
"decoder.vlan_qinq".
A node isn't known to be a sequence node until the YAML is parsed.
If a node sequence node was set on the command line, promote
it to a sequence node when it is discovered by YAML to be
a sequence node.
Fixes comment #18 in issue 921.
When creating a pseudo packet with the reassembled IP packet, the
parent's vlan id or id's are also needed. The defrag packet is run
through decode and the flow engine, where the vlan id is necessary
for connecting the packet to the correct flow.
Some old distribution don't ship recent enough linux header. This
result in TP_STATUS_VLAN_VALID being undefined. This patch defines
the constant and use it as it is used in backward compatible method
in the code: the flag is not set by kernel and a test on vci value
will be made.
This should fix https://redmine.openinfosecfoundation.org/issues/1106
Flow-timeout code injects pseudo packets into the decoders, leading
to various issues. For a full explanation, see:
https://redmine.openinfosecfoundation.org/issues/1107
This patch works around the issues with a hack. It adds a check to
each of the decoder entry points to bail out as soon as a pseudo
packet from the flow timeout is encountered.
Ticket #1107.
FlowReference stores the flow in the destination pointer and increases
the flow reference counter (use_cnt). This should only be called once
per destination pointer. The reference counter is decremented when
FlowDereference is called. Multiple FlowReference calls would lead to
multiple use_cnt bumps, while there would be only one FlowRereference.
This lead to a use_cnt that would never become 0, meaning the flow
would stay in the hash for the entire lifetime of the process.
The fix here is to check if the destination pointer is already set to
the flow. If so, we don't increase the reference counter.
As this is really a bug, this condition will lead to a BUG_ON if the
DEBUG_VALIDATION checking is enabled.
Ken Steele
Victor Julien
Eric Leblond
Jason Ish