You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
suricata/src/detect-engine-payload.c

1248 lines
34 KiB
C

New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
/* Copyright (C) 2007-2013 Open Information Security Foundation
*
* You can copy, redistribute or modify this Program under the terms of
* the GNU General Public License version 2 as published by the Free
* Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* version 2 along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
* 02110-1301, USA.
*/
16 years ago
/**
* \file
*
* \author Victor Julien <victor@inliniac.net>
*
* Performs payload matching functions
16 years ago
*/
#include "suricata-common.h"
#include "suricata.h"
#include "decode.h"
#include "detect.h"
#include "detect-engine.h"
#include "detect-parse.h"
#include "detect-engine-content-inspection.h"
#include "stream.h"
#include "util-debug.h"
#include "util-print.h"
#include "util-unittest.h"
#include "util-unittest-helper.h"
#include "util-validate.h"
#include "util-mpm-ac.h"
uint32_t PacketPatternSearchWithStreamCtx(DetectEngineThreadCtx *det_ctx,
Packet *p)
{
SCEnter();
uint32_t ret = 0;
const MpmCtx *mpm_ctx = NULL;
if (p->flowflags & FLOW_PKT_TOSERVER) {
DEBUG_VALIDATE_BUG_ON(det_ctx->sgh->mpm_stream_ctx_ts == NULL);
mpm_ctx = det_ctx->sgh->mpm_stream_ctx_ts;
} else {
DEBUG_VALIDATE_BUG_ON(det_ctx->sgh->mpm_stream_ctx_tc == NULL);
mpm_ctx = det_ctx->sgh->mpm_stream_ctx_tc;
}
if (unlikely(mpm_ctx == NULL)) {
SCReturnInt(0);
}
ret = mpm_table[mpm_ctx->mpm_type].
Search(mpm_ctx, &det_ctx->mtc, &det_ctx->pmq,
p->payload, p->payload_len);
SCReturnInt(ret);
}
/** \brief Pattern match -- searches for only one pattern per signature.
*
* \param det_ctx detection engine thread ctx
* \param p packet
* \param smsg stream msg (reassembled stream data)
* \param flags stream flags
*
* \retval ret number of matches
*/
uint32_t StreamPatternSearch(DetectEngineThreadCtx *det_ctx, Packet *p,
StreamMsg *smsg, uint8_t flags)
{
SCEnter();
uint32_t ret = 0;
uint8_t cnt = 0;
//PrintRawDataFp(stdout, smsg->data.data, smsg->data.data_len);
uint32_t r;
if (flags & STREAM_TOSERVER) {
for ( ; smsg != NULL; smsg = smsg->next) {
r = mpm_table[det_ctx->sgh->mpm_stream_ctx_ts->mpm_type].
Search(det_ctx->sgh->mpm_stream_ctx_ts, &det_ctx->mtcs,
&det_ctx->pmq, smsg->data, smsg->data_len);
if (r > 0) {
ret += r;
}
cnt++;
}
} else if (flags & STREAM_TOCLIENT) {
for ( ; smsg != NULL; smsg = smsg->next) {
r = mpm_table[det_ctx->sgh->mpm_stream_ctx_tc->mpm_type].
Search(det_ctx->sgh->mpm_stream_ctx_tc, &det_ctx->mtcs,
&det_ctx->pmq, smsg->data, smsg->data_len);
if (r > 0) {
ret += r;
}
cnt++;
}
}
SCReturnInt(ret);
}
/** \brief Pattern match -- searches for only one pattern per signature.
*
* \param det_ctx detection engine thread ctx
* \param p packet to inspect
*
* \retval ret number of matches
*/
uint32_t PacketPatternSearch(DetectEngineThreadCtx *det_ctx, Packet *p)
{
SCEnter();
uint32_t ret;
const MpmCtx *mpm_ctx = NULL;
if (p->proto == IPPROTO_TCP) {
if (p->flowflags & FLOW_PKT_TOSERVER) {
mpm_ctx = det_ctx->sgh->mpm_proto_tcp_ctx_ts;
} else if (p->flowflags & FLOW_PKT_TOCLIENT) {
mpm_ctx = det_ctx->sgh->mpm_proto_tcp_ctx_tc;
}
} else if (p->proto == IPPROTO_UDP) {
if (p->flowflags & FLOW_PKT_TOSERVER) {
mpm_ctx = det_ctx->sgh->mpm_proto_udp_ctx_ts;
} else if (p->flowflags & FLOW_PKT_TOCLIENT) {
mpm_ctx = det_ctx->sgh->mpm_proto_udp_ctx_tc;
}
} else {
mpm_ctx = det_ctx->sgh->mpm_proto_other_ctx;
}
if (unlikely(mpm_ctx == NULL))
SCReturnInt(0);
#ifdef __SC_CUDA_SUPPORT__
if (p->cuda_pkt_vars.cuda_mpm_enabled && p->pkt_src == PKT_SRC_WIRE) {
ret = SCACCudaPacketResultsProcessing(p, mpm_ctx, &det_ctx->pmq);
} else {
ret = mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
&det_ctx->mtc,
&det_ctx->pmq,
p->payload,
p->payload_len);
}
#else
ret = mpm_table[mpm_ctx->mpm_type].Search(mpm_ctx,
&det_ctx->mtc,
&det_ctx->pmq,
p->payload,
p->payload_len);
#endif
SCReturnInt(ret);
}
/**
* \brief Do the content inspection & validation for a signature
*
* \param de_ctx Detection engine context
* \param det_ctx Detection engine thread context
* \param s Signature to inspect
* \param f flow (for pcre flowvar storage)
* \param p Packet
*
* \retval 0 no match
* \retval 1 match
*/
int DetectEngineInspectPacketPayload(DetectEngineCtx *de_ctx,
DetectEngineThreadCtx *det_ctx, Signature *s, Flow *f, Packet *p)
{
SCEnter();
int r = 0;
if (s->sm_lists[DETECT_SM_LIST_PMATCH] == NULL) {
SCReturnInt(0);
}
det_ctx->buffer_offset = 0;
det_ctx->discontinue_matching = 0;
det_ctx->inspection_recursion_counter = 0;
det_ctx->replist = NULL;
//det_ctx->flags |= DETECT_ENGINE_THREAD_CTX_INSPECTING_PACKET;
r = DetectEngineContentInspection(de_ctx, det_ctx, s, s->sm_lists[DETECT_SM_LIST_PMATCH],
f, p->payload, p->payload_len, 0,
DETECT_ENGINE_CONTENT_INSPECTION_MODE_PAYLOAD, p);
//r = DoInspectPacketPayload(de_ctx, det_ctx, s, s->sm_lists[DETECT_SM_LIST_PMATCH], p, f, p->payload, p->payload_len);
//det_ctx->flags &= ~DETECT_ENGINE_THREAD_CTX_INSPECTING_PACKET;
if (r == 1) {
SCReturnInt(1);
}
SCReturnInt(0);
}
/**
* \brief Do the content inspection & validation for a signature for a stream chunk
*
* \param de_ctx Detection engine context
* \param det_ctx Detection engine thread context
* \param s Signature to inspect
* \param f flow (for pcre flowvar storage)
* \param payload ptr to the payload to inspect
* \param payload_len length of the payload
*
* \retval 0 no match
* \retval 1 match
*
* \todo we might also pass the packet to this function for the pktvar
* storage. Only, would that be right? We're not inspecting data
* from the current packet here.
*/
int DetectEngineInspectStreamPayload(DetectEngineCtx *de_ctx,
DetectEngineThreadCtx *det_ctx, Signature *s, Flow *f,
uint8_t *payload, uint32_t payload_len)
{
SCEnter();
int r = 0;
if (s->sm_lists[DETECT_SM_LIST_PMATCH] == NULL) {
SCReturnInt(0);
}
det_ctx->buffer_offset = 0;
det_ctx->discontinue_matching = 0;
det_ctx->inspection_recursion_counter = 0;
//det_ctx->flags |= DETECT_ENGINE_THREAD_CTX_INSPECTING_STREAM;
r = DetectEngineContentInspection(de_ctx, det_ctx, s, s->sm_lists[DETECT_SM_LIST_PMATCH],
f, payload, payload_len, 0,
DETECT_ENGINE_CONTENT_INSPECTION_MODE_STREAM, NULL);
//r = DoInspectPacketPayload(de_ctx, det_ctx, s, s->sm_lists[DETECT_SM_LIST_PMATCH], NULL, f, payload, payload_len);
//det_ctx->flags &= ~DETECT_ENGINE_THREAD_CTX_INSPECTING_STREAM;
if (r == 1) {
SCReturnInt(1);
}
SCReturnInt(0);
}
#ifdef UNITTESTS
/** \test Not the first but the second occurence of "abc" should be used
* for the 2nd match */
static int PayloadTestSig01 (void)
{
uint8_t *buf = (uint8_t *)
"abcabcd";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (content:\"abc\"; content:\"d\"; distance:0; within:1; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/** \test Nocase matching */
static int PayloadTestSig02 (void)
{
uint8_t *buf = (uint8_t *)
"abcaBcd";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (content:\"abc\"; nocase; content:\"d\"; distance:0; within:1; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
16 years ago
/** \test Negative distance matching */
static int PayloadTestSig03 (void)
{
16 years ago
uint8_t *buf = (uint8_t *)
"abcaBcd";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (content:\"aBc\"; nocase; content:\"abca\"; distance:-10; within:4; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
16 years ago
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test multiple relative matches.
*/
static int PayloadTestSig04(void)
{
uint8_t *buf = (uint8_t *)"now this is is big big string now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"this\"; content:\"is\"; within:6; content:\"big\"; within:8; "
"content:\"string\"; within:8; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test multiple relative matches.
*/
static int PayloadTestSig05(void)
{
uint8_t *buf = (uint8_t *)"now this is is is big big big string now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"this\"; content:\"is\"; within:9; content:\"big\"; within:12; "
"content:\"string\"; within:8; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test multiple relative matches.
*/
static int PayloadTestSig06(void)
{
uint8_t *buf = (uint8_t *)"this this now is is big string now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"now\"; content:\"this\"; content:\"is\"; within:12; content:\"big\"; within:8; "
"content:\"string\"; within:8; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test multiple relative matches.
*/
static int PayloadTestSig07(void)
{
uint8_t *buf = (uint8_t *)" thus thus is a big";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"thus\"; offset:8; content:\"is\"; within:6; content:\"big\"; within:8; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test multiple relative matches with negative matches
* and show the need for det_ctx->discontinue_matching.
*/
static int PayloadTestSig08(void)
{
uint8_t *buf = (uint8_t *)"we need to fix this and yes fix this now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"fix\"; content:\"this\"; within:6; content:!\"and\"; distance:0; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) != 1) {
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test pcre recursive matching.
*/
static int PayloadTestSig09(void)
{
uint8_t *buf = (uint8_t *)"this is a super duper nova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"pcre:/super/; content:\"nova\"; within:7; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test invalid sig.
*/
static int PayloadTestSig10(void)
{
uint8_t *buf = (uint8_t *)"this is a super duper nova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert udp any any -> any any (msg:\"crash\"; "
"byte_test:4,>,2,0,relative; sid:11;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 1) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test invalid sig.
*/
static int PayloadTestSig11(void)
{
uint8_t *buf = (uint8_t *)"this is a super duper nova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert udp any any -> any any (msg:\"crash\"; "
"byte_jump:1,0,relative; sid:11;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 1) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test invalid sig.
*/
static int PayloadTestSig12(void)
{
uint8_t *buf = (uint8_t *)"this is a super duper nova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert udp any any -> any any (msg:\"crash\"; "
"isdataat:10,relative; sid:11;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 1) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Used to check the working of recursion_limit counter.
*/
static int PayloadTestSig13(void)
{
uint8_t *buf = (uint8_t *)"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
uint16_t mpm_type = DEFAULT_MPM;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"aa\"; content:\"aa\"; distance:0; content:\"aa\"; distance:0; "
"byte_test:1,>,200,0,relative; sid:1;)";
struct timeval tv_start, tv_end, tv_diff;
gettimeofday(&tv_start, NULL);
do {
DecodeThreadVars dtv;
ThreadVars th_v;
DetectEngineThreadCtx *det_ctx = NULL;
memset(&dtv, 0, sizeof(DecodeThreadVars));
memset(&th_v, 0, sizeof(th_v));
DetectEngineCtx *de_ctx = DetectEngineCtxInit();
if (de_ctx == NULL) {
printf("de_ctx == NULL: ");
goto end;
}
de_ctx->inspection_recursion_limit = 3000;
de_ctx->flags |= DE_QUIET;
de_ctx->mpm_matcher = mpm_type;
de_ctx->sig_list = SigInit(de_ctx, sig);
if (de_ctx->sig_list == NULL) {
printf("signature == NULL: ");
goto end;
}
SigGroupBuild(de_ctx);
DetectEngineThreadCtxInit(&th_v, (void *)de_ctx, (void *)&det_ctx);
SigMatchSignatures(&th_v, de_ctx, det_ctx, p);
if (PacketAlertCheck(p, de_ctx->sig_list->id) != 1) {
goto end;
}
result = 1;
end:
SigGroupCleanup(de_ctx);
SigCleanSignatures(de_ctx);
if (det_ctx != NULL)
DetectEngineThreadCtxDeinit(&th_v, (void *)det_ctx);
if (de_ctx != NULL)
DetectEngineCtxFree(de_ctx);
} while (0);
gettimeofday(&tv_end, NULL);
tv_diff.tv_sec = tv_end.tv_sec - tv_start.tv_sec;
tv_diff.tv_usec = tv_end.tv_usec - tv_start.tv_usec;
printf("%ld.%06ld\n", (long int)tv_diff.tv_sec, (long int)tv_diff.tv_usec);
result = 1;
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test normal & negated matching, both absolute and relative
*/
static int PayloadTestSig14(void)
{
uint8_t *buf = (uint8_t *)"User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.6 GTB5";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (content:\"User-Agent|3A| Mozilla/5.0 |28|Macintosh|3B| \"; content:\"Firefox/3.\"; distance:0; content:!\"Firefox/3.6.12\"; distance:-10; content:!\"Mozilla/5.0 |28|Macintosh|3B| U|3B| Intel Mac OS X 10.5|3B| en-US|3B| rv|3A|1.9.1b4|29| Gecko/20090423 Firefox/3.6 GTB5\"; sid:1; rev:1;)";
//char sig[] = "alert tcp any any -> any any (content:\"User-Agent: Mozilla/5.0 (Macintosh; \"; content:\"Firefox/3.\"; distance:0; content:!\"Firefox/3.6.12\"; distance:-10; content:!\"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.6 GTB5\"; sid:1; rev:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 1) {
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig15(void)
{
uint8_t *buf = (uint8_t *)"this is a super duper nova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"nova\"; isdataat:18,relative; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig16(void)
{
uint8_t *buf = (uint8_t *)"this is a super duper nova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"nova\"; isdataat:!20,relative; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig17(void)
{
uint8_t buf[] = { 0xEB, 0x29, 0x25, 0x38, 0x78, 0x25, 0x38, 0x78, 0x25 };
uint16_t buflen = 9;
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"%\"; depth:4; offset:0; "
"content:\"%\"; within:2; distance:1; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig18(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x35, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,dec,relative; "
"content:\"|0C 0D 0E 0F|\"; distance:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig19(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x35, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,hex,relative; "
"content:\"|0C 0D 0E 0F|\"; distance:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig20(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x35, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,dec,relative; "
"content:\"|06 35 07 08|\"; offset:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig21(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x36, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,dec,relative; "
"content:\"|03 04 05 06|\"; depth:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig22(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x36, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,dec,relative; "
"content:\"|09 0A 0B 0C|\"; within:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig23(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x32, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x33, 0x0B, 0x0C, 0x0D,
0x32, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,dec,relative; "
"byte_extract:1,3,two,string,dec,relative; "
"byte_test:1,=,one,two,string,dec,relative; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig24(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x32, /* the last byte is 2 */
0x07, 0x08, 0x33, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|01 02 03 04|\"; "
"byte_extract:1,2,one,string,dec,relative; "
"byte_jump:1,one,string,dec,relative; "
"content:\"|0D 0E 0F|\"; distance:0; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/*
* \test Test negative byte extract.
*/
static int PayloadTestSig25(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x35, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|35 07 08 09|\"; "
"byte_extract:1,-4,one,string,dec,relative; "
"content:\"|0C 0D 0E 0F|\"; distance:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/*
* \test Test negative byte extract.
*/
static int PayloadTestSig26(void)
{
uint8_t buf[] = {
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x35, /* the last byte is 2 */
0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D,
0x0E, 0x0F,
};
uint16_t buflen = sizeof(buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"content:\"|35 07 08 09|\"; "
"byte_extract:1,-3000,one,string,dec,relative; "
"content:\"|0C 0D 0E 0F|\"; distance:one; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) != 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/*
* \test Test packet/stream sigs
*/
static int PayloadTestSig27(void)
{
uint8_t buf[] = "dummypayload";
uint16_t buflen = sizeof(buf) - 1;
int result = 0;
Packet *p = UTHBuildPacket(buf, buflen, IPPROTO_TCP);
char sig[] = "alert tcp any any -> any any (content:\"dummy\"; "
"depth:5; sid:1;)";
p->flags |= PKT_STREAM_ADD;
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) != 1)
goto end;
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/*
* \test Test packet/stream sigs
*/
static int PayloadTestSig28(void)
{
uint8_t buf[] = "dummypayload";
uint16_t buflen = sizeof(buf) - 1;
int result = 0;
Packet *p = UTHBuildPacket(buf, buflen, IPPROTO_TCP);
char sig[] = "alert tcp any any -> any any (content:\"payload\"; "
"offset:4; depth:12; sid:1;)";
p->flags |= PKT_STREAM_ADD;
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) != 1)
goto end;
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test pcre recursive matching - bug #529
*/
static int PayloadTestSig29(void)
{
uint8_t *buf = (uint8_t *)"this is a super dupernova in super nova now";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"dummy\"; "
"pcre:/^.{4}/; content:\"nova\"; within:4; sid:1;)";
New Multi-pattern matcher, ac-tile, optimized for Tile architecture. Aho-Corasick mpm optimized for Tilera Tile-Gx architecture. Based on the util-mpm-ac.c code base. The primary optimizations are: 1) Matching function used Tilera specific instructions. 2) Alphabet compression to reduce delta table size to increase cache utilization and performance. The basic observation is that not all 256 ASCII characters are used by the set of multiple patterns in a group for which a DFA is created. The first reason is that Suricata's pattern matching is case-insensitive, so all uppercase characters are converted to lowercase, leaving a hole of 26 characters in the alphabet. Previously, this hole was simply left in the middle of the alphabet and thus in the generated Next State (delta) tables. A new, smaller, alphabet is created using a translation table of 256 bytes per mpm group. Previously, there was one global translation table for converting upper case to lowercase. Additional, unused characters are found by creating a histogram of all the characters in all the patterns. Then all the characters with zero counts are mapped to one character (0) in the new alphabet. Since These characters appear in no pattern, they can all be mapped to a single character and still result in the same matches being found. Zero was chosen for the value in the new alphabet since this "character" is more likely to appear in the input. The unused character always results in the next state being state zero, but that fact is not currently used by the code, since special casing takes additional instructions. The characters that do appear in some pattern are mapped to consecutive characters in the new alphabet, starting at 1. This results in a dense packing of next state values in the delta tables and additionally can allow for a smaller number of columns in that table, thus using less memory and better packing into the cache. The size of the new alphabet is the number of used characters plus 1 for the unused catch-all character. The alphabet size is rounded up to the next larger power-of-2 so that multiplication by the alphabet size can be done with a shift. It might be possible to use a multiply instruction, so that the exact alphabet size could be used, which would further reduce the size of the delta tables, increase cache density and not require the specialized search functions. The multiply would likely add 1 cycle to the inner search loop. Since the multiply by alphabet-size is cleverly merged with a mask instruction (in the SINDEX macro), specialized versions of the SCACSearch function are generated for alphabet sizes 256, 128, 64, 32 and 16. This is done by including the file util-mpm-ac-small.c multiple times with a redefined SINDEX macro. A function pointer is then stored in the mpm context for the search function. For alpha bit sizes of 8 or smaller, the number of states usually small, so the DFA is already very small, so there is little difference using the 16 state search function. The SCACSearch function is also specialized by the size of the value stored in the next state (delta) tables, either 16-bits or 32-bits. This removes a conditional inside the Search function. That conditional is only called once, but doesn't hurt to remove it. 16-bits are used for up to 32K states, with the sign bit set for states with matches. Future optimization: The state-has-match values is only needed per state, not per next state, so checking the next-state sign bit could be replaced with reading a different value, at the cost of an additional load, but increasing the 16-bit next state span to 64K. Since the order of the characters in the new alphabet doesn't matter, the new alphabet could be sorted by the frequency of the characters in the expected input stream for that multi-pattern matcher. This would group more frequent characters into the same cache lines, thus increasing the probability of reusing a cache-line. All the next state values for each state live in their own set of cache-lines. With power-of-two sizes alphabets, these don't overlap. So either 32 or 16 character's next states are loaded in each cache line load. If the alphabet size is not an exact power-of-2, then the last cache-line is not completely full and up to 31*2 bytes of that line could be wasted per state. The next state table could be transposed, so that all the next states for a specific character are stored sequentially, this could be better if some characters, for example the unused character, are much more frequent.
12 years ago
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 1) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig30(void)
{
uint8_t *buf = (uint8_t *)
"xyonexxxxxxtwojunkonetwo";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (content:\"one\"; pcre:\"/^two/R\"; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
static int PayloadTestSig31(void)
{
uint8_t *buf = (uint8_t *)
"xyonexxxxxxtwojunkonetwo";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket( buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (content:\"one\"; pcre:\"/(fiv|^two)/R\"; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0) {
result = 0;
goto end;
}
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test byte_jump.
*/
static int PayloadTestSig32(void)
{
uint8_t *buf = (uint8_t *)"dummy2xxcardmessage";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket(buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"crash\"; "
"content:\"message\"; byte_jump:2,-14,string,dec,relative; content:\"card\"; within:4; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0)
goto end;
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test byte_test.
*/
static int PayloadTestSig33(void)
{
uint8_t *buf = (uint8_t *)"dummy2xxcardmessage";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket(buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"crash\"; "
"content:\"message\"; byte_test:1,=,2,-14,string,dec,relative; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0)
goto end;
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
/**
* \test Test byte_extract.
*/
static int PayloadTestSig34(void)
{
uint8_t *buf = (uint8_t *)"dummy2xxcardmessage";
uint16_t buflen = strlen((char *)buf);
Packet *p = UTHBuildPacket(buf, buflen, IPPROTO_TCP);
int result = 0;
char sig[] = "alert tcp any any -> any any (msg:\"crash\"; "
"content:\"message\"; byte_extract:1,-14,boom,string,dec,relative; sid:1;)";
if (UTHPacketMatchSigMpm(p, sig, DEFAULT_MPM) == 0)
goto end;
result = 1;
end:
if (p != NULL)
UTHFreePacket(p);
return result;
}
#endif /* UNITTESTS */
void PayloadRegisterTests(void)
{
#ifdef UNITTESTS
UtRegisterTest("PayloadTestSig01", PayloadTestSig01, 1);
UtRegisterTest("PayloadTestSig02", PayloadTestSig02, 1);
16 years ago
UtRegisterTest("PayloadTestSig03", PayloadTestSig03, 1);
UtRegisterTest("PayloadTestSig04", PayloadTestSig04, 1);
UtRegisterTest("PayloadTestSig05", PayloadTestSig05, 1);
UtRegisterTest("PayloadTestSig06", PayloadTestSig06, 1);
UtRegisterTest("PayloadTestSig07", PayloadTestSig07, 1);
UtRegisterTest("PayloadTestSig08", PayloadTestSig08, 1);
UtRegisterTest("PayloadTestSig09", PayloadTestSig09, 1);
UtRegisterTest("PayloadTestSig10", PayloadTestSig10, 1);
UtRegisterTest("PayloadTestSig11", PayloadTestSig11, 1);
UtRegisterTest("PayloadTestSig12", PayloadTestSig12, 1);
UtRegisterTest("PayloadTestSig13", PayloadTestSig13, 1);
UtRegisterTest("PayloadTestSig14", PayloadTestSig14, 1);
UtRegisterTest("PayloadTestSig15", PayloadTestSig15, 1);
UtRegisterTest("PayloadTestSig16", PayloadTestSig16, 1);
UtRegisterTest("PayloadTestSig17", PayloadTestSig17, 1);
UtRegisterTest("PayloadTestSig18", PayloadTestSig18, 1);
UtRegisterTest("PayloadTestSig19", PayloadTestSig19, 1);
UtRegisterTest("PayloadTestSig20", PayloadTestSig20, 1);
UtRegisterTest("PayloadTestSig21", PayloadTestSig21, 1);
UtRegisterTest("PayloadTestSig22", PayloadTestSig22, 1);
UtRegisterTest("PayloadTestSig23", PayloadTestSig23, 1);
UtRegisterTest("PayloadTestSig24", PayloadTestSig24, 1);
UtRegisterTest("PayloadTestSig25", PayloadTestSig25, 1);
UtRegisterTest("PayloadTestSig26", PayloadTestSig26, 1);
UtRegisterTest("PayloadTestSig27", PayloadTestSig27, 1);
UtRegisterTest("PayloadTestSig28", PayloadTestSig28, 1);
UtRegisterTest("PayloadTestSig29", PayloadTestSig29, 1);
UtRegisterTest("PayloadTestSig30", PayloadTestSig30, 1);
UtRegisterTest("PayloadTestSig31", PayloadTestSig31, 1);
UtRegisterTest("PayloadTestSig32", PayloadTestSig32, 1);
UtRegisterTest("PayloadTestSig33", PayloadTestSig33, 1);
UtRegisterTest("PayloadTestSig34", PayloadTestSig34, 1);
#endif /* UNITTESTS */
return;
}