suricata/ChangeLog

2.0beta1 -- 2013-07-18

- Luajit flow vars and flow ints support (#593)
- DNS parser, logger and keyword support (#792), funded by Emerging Threats
- deflate support for HTTP response bodies (#470, #775)
- update to libhtp 0.5 (#775)
- improved gzip support for HTTP response bodies (#470, #775)
- redesigned transaction handling, improving both accuracy and performance (#753)
- redesigned CUDA support (#729)
- Be sure to always apply verdict to NFQ packet (#769)
- stream engine: SACK allocs should adhere to memcap (#794)
- stream: deal with multiple different SYN/ACK's better (#796)
- stream: Randomize stream chunk size for raw stream inspection (#804)
- Introduce per stream thread ssn pool (#519)
- "pass" IP-only rules should bypass detection engine after matching (#718)
- Generate error if bpf is used in IPS mode (#777)
- Add support for batch verdicts in NFQ, thanks to Florian Westphal
- Update Doxygen config, thanks to Phil Schroeder
- Improve libnss detection, thanks to Christian Kreibich
- Fix a FP on rules looking for port 0 and fragments (#847), thanks to Rmkml
- OS X unix socket build fixed (#830)
- bytetest, bytejump and byteextract negative offset failure (#827)
- Fix fast.log formatting issues (#771), thanks to Rmkml
- Invalidate negative depth (#774), thanks to Rmkml
- Fixed accuracy issues with relative pcre matching (#791)
- Fix deadlock in flowvar capture code (#802)
- Improved accuracy of file_data keyword (#817)
- Fix af-packet ips mode rule processing bug (#819), thanks to Laszlo Madarassy
- stream: fix injecting pseudo packet too soon leading to FP (#883), thanks to Francis Trudeau

1.4.4 -- 2013-07-18

- Bug #834: Unix socket - showing as compiled when it is not desired to do so
- Bug #835: Unix Socket not working as expected
- Bug #841: configure --enable-unix-socket does not err out if libs/pkgs are not present
- Bug #846: FP on IP frag and sig use udp port 0, thanks to Rmkml
- Bug #864: backport packet action macro's
- Bug #876: htp tunnel fix
- Bug #877: Flowbit check with content doesn't match consistently, thanks to Francis Trudeau

1.4.3 -- 2013-06-20

- Fix missed detection in bytetest, bytejump and byteextract for negative offset (#828)
- Fix IPS mode being unable to drop tunneled packets (#826)
- Fix OS X Unix Socket build (#829)

1.4.2 -- 2013-05-29

- No longer force nocase to be used on http_host
- Invalidate rule if uppercase content is used for http_host w/o nocase
- Warn user if bpf is used in af-packet IPS mode
- Better test for available libjansson version
- Fixed accuracy issues with relative pcre matching (#784)
- Improved accuracy of file_data keyword (#788)
- Invalidate negative depth (#770)
- Fix http host parsing for IPv6 addresses (#761)
- Fix fast.log formatting issues (#773)
- Fixed deadlock in flowvar set code for http buffers (#801)
- Various signature ordering improvements
- Minor stream engine fix

1.4.1 -- 2013-03-08

- GeoIP keyword, allowing matching on Maxmind's database, contributed by Ignacio Sanchez (#559)
- Introduce http_host and http_raw_host keywords (#733, #743)
- Add python module for interacting with unix socket (#767)
- Add new unix socket commands: fetching config, counters, basic runtime info (#764, #765)
- Big Napatech support update by Matt Keeler
- Configurable sensor id in unified2 output, contributed by Jake Gionet (#667)
- FreeBSD IPFW fixes by Nikolay Denev
- Add "default" interface setting to capture configuration in yaml (#679)
- Make sure "snaplen" can be set by the user (#680)
- Improve HTTP URI query string normalization (#739)
- Improved error reporting in MD5 loading (#693)
- Improve reference.config parser error reporting (#737)
- Improve build info output to include all configure options (#738)
- Segfault in TLS parsing reported by Charles Smutz (#725)
- Fix crash in teredo decoding, reported by Rmkml (#736)
- fixed UDPv4 packets without checksum being detected as invalid (#760)
- fixed DCE/SMB parsers getting confused in some fragmented cases (#764)
- parsing ipv6 address/subnet parsing in thresholding was fixed by Jamie Strandboge (#697)
- FN: IP-only rule ip_proto not matching for some protocols (#689)
- Fix build failure with other libhtp installs (#688)
- Fix malformed yaml loading leading to a crash (#694)
- Various Mac OS X fixes (#700, #701, #703)
- Fix for autotools on Mac OS X by Jason Ish (#704)
- Fix AF_PACKET under high load not updating stats (#706)

1.3.6 -- 2013-03-07

- fix decoder event rules not checked in all cases (#671)
- checksum detection for icmpv6 was fixed (#673)
- crash in HTTP server body inspection code fixed (#675)
- fixed a icmpv6 payload bug (#676)
- IP-only rule ip_proto not matching for some protocols was addressed (#690)
- fixed malformed yaml crashing suricata (#702)
- parsing ipv6 address/subnet parsing in thresholding was fixed by Jamie Strandboge (#717)
- crash in tls parser was fixed (#759)
- fixed UDPv4 packets without checksum being detected as invalid (#762)
- fixed DCE/SMB parsers getting confused in some fragmented cases (#763)

1.4 2012-12-13

- Decoder event matching fixed (#672)
- Unified2 would overwrite files if file rotation happened within a second of file creation, leading to loss of events/alerts (#665)
- Add more events to IPv6 extension header anomolies (#678)
- Fix ICMPv6 payload and checksum calculation (#677, #674)
- Clean up flow timeout handling (#656)
- Fix a shutdown bug when using AF_PACKET under high load (#653)
- Fix TCP sessions being cleaned up to early (#652)

1.3.5 2012-12-06

- Flow engine memory leak fixed by Ludovico Cavedon (#651)
- Unified2 would overwrite files if file rotation happened within a second of file creation, leading to loss of events/alerts (#664)
- Flow manager mutex used unintialized, fixed by Ludovico Cavedon (#654)
- Windows building in CYGWIN fixed (#630)

1.4rc1 2012-11-29

- Interactive unix socket mode (#571, #552)
- IP Reputation: loading and matching (#647)
- Improved --list-keywords commandline option gives detailed info for supported keyword, including doc link (#435)
- Rule analyzer improvement wrt ipv4/ipv6, invalid rules (#494)
- User-Agent added to file log and filestore meta files (#629)
- Endace DAG supports live stats and at exit drop stats (#638)
- Add support for libhtp event "request port doesn't match tcp port" (#650)
- Rules with negated addresses will not be considered IP-only (#599)
- Rule reloads complete much faster in low traffic conditions (#526)
- Suricata -h now displays all available options (#419)
- Luajit configure time detection was improved (#636)
- Flow manager mutex used w/o initialization (#628)
- Cygwin work around for windows shell mangling interface string (#372)
- Fix a Prelude output crash with alerts generated by rules w/o classtype or msg (#648)
- CLANG compiler build fixes (#649)
- Several fixes found by code analyzers

1.4beta3 2012-11-14

- support for Napatech cards was greatly improved by Matt Keeler from Npulse (#430, #619)
- support for pkt_data keyword was added
- user and group to run as can now be set in the config file
- make HTTP request and response body inspection sizes configurable per HTTP server config (#560)
- PCAP/AF_PACKET/PF_RING packet stats are now printed in stats.log (#561, #625)
- add contrib directory to the dist (#567)
- performance improvements to signatures with dsize option
- improved rule analyzer: print fast_pattern along with the rule (#558)
- fixes to stream engine reducing the number of events generated (#604)
- add stream event to match on overlaps with different data in stream reassembly (#603)
- stream.inline option new defaults to "auto", meaning enabled in IPS mode, disabled in IDS mode (#592)
- HTTP handling in OOM condition was greatly improved (#557)
- filemagic keyword performance was improved (#585)
- fixes and improvements to daemon mode (#624)
- fix drop rules not working correctly when thresholded (#613)
- fixed a possible FP when a regular and "chopped" fast_pattern were the same (#581)
- fix a false possitive condition in http_header (#607)
- fix inaccuracy in byte_jump keyword when using "from_beginning" option (#627)
- fixes to rule profiling (#576)
- cleanups and misc fixes (#379, #395)
- updated bundled libhtp to 0.2.11
- build system improvements and cleanups
- fix to SSL record parsing

1.3.4 -- 2012-11-14

- fix crash in flow and host engines in cases of low memory or low memcap settings (#617)
- improve http handling in low memory conditions (#620)
- fix inaccuracy in byte_jump keyword when using "from_beginning" option (#626)
- fix building on OpenBSD 5.2
- update default config's defrag settings to reflect all available options
- fixes to make check
- fix to SSL record parsing

1.3.3 -- 2012-11-01

- fix drop rules not working correctly when thresholded (#615)
- fix a false possitive condition in http_header (#606)
- fix extracted file corruption (#601)
- fix a false possitive condition with the pcre keyword and relative matching (#588)
- fix PF_RING set cluster problem on dma interfaces (#598)
- improve http handling in low memory conditions (#586, #587)
- fix FreeBSD inline mode crash (#612)
- suppress pcre jit warning (#579)

1.4beta2 -- 2012-10-04

- New keyword: "luajit" to inspect packet, payload and all HTTP buffers with a Lua script (#346)
- Added ability to control per server HTTP parser settings in much more detail (#503)
- Rewrite of IP Defrag engine to improve performance and fix locking logic (#512, #540)
- Big performance improvement in inspecting decoder, stream and app layer events (#555)
- Pool performance improvements (#541)
- Improved performance of signatures with simple pattern setups (#577)
- Bundled docs are installed upon make install (#527)
- Support for a number of global vs rule thresholds [3] was added (#425)
- Improved rule profiling performance
- If not explicit fast_pattern is set, pick HTTP patterns over stream patterns. HTTP method, stat code and stat msg are excluded.
- Fix compilation on architectures other than x86 and x86_64 (#572)
- Fix FP with anchored pcre combined with relative matching (#529)
- Fix engine hanging instead of exitting if the pcap device doesn't exist (#533)
- Work around for potential FP, will get properly fixed in next release (#574)
- Improve ERF handling. Thanks to Jason Ish
- Always set cluster_id in PF_RING
- IPFW: fix broken broadcast handling
- AF_PACKET kernel offset issue, IPS fix and cleanup
- Fix stream engine sometimes resending the same data to app layer
- Fix multiple issues in HTTP multipart parsing
- Fixed a lockup at shutdown with NFQ (#537)

1.3.2 -- 2012-10-03

- Fixed a possible FP when a regular and "chopped" fast_pattern were the same (#562)
- Fixed a FN condition with the flow:no_stream option (#575)
- Fix building of perf profiling code on i386 platform. By Simon Moon (#534)
- Fix multiple issues in HTTP multipart parsing
- Fix stream engine sometimes resending the same data to app layer
- Always set cluster_id in PF_RING
- Defrag: silence some potentially noisy errors/warnings
- IPFW: fix broken broadcast handling
- AF_PACKET kernel offset issue

1.4beta1 -- 2012-09-06

- Custom HTTP logging contributed by Ignacio Sanchez (#530)
- TLS certificate logging and fingerprint computation and keyword (#443)
- TLS certificate store to disk feature (#444)
- Decoding of IPv4-in-IPv6, IPv6-in-IPv6 and Teredo tunnels (#462, #514, #480)
- AF_PACKET IPS support (#516)
- Rules can be set to inspect only IPv4 or IPv6 (#494)
- filesize keyword for matching on sizes of files in HTTP (#489)
- Delayed detect initialization. Starts processing packets right away and loads detection engine in the background (#522)
- NFQ fail open support (#507)
- Highly experimental lua scripting support for detection
- Live reloads now supports HTTP rule updates better (#522)
- AF_PACKET performance improvements (#197, #415)
- Make defrag more configurable (#517, #528)
- Improve pool performance (#518)
- Improve file inspection keywords by adding a separate API (#531)
- Example threshold.config file provided (#302)
- Fix building of perf profiling code on i386 platform. By Simon Moon (#534)
- Various spelling corrections by Simon Moon (#533)

1.3.1 -- 2012-08-21

- AF_PACKET performance improvements
- Defrag engine performance improvements
- HTTP: add per server options to enable/disable double decoding of URI (#464, #504)
- Stream engine packet handling for packets with non-standard flag combinations (#508)
- Improved stream engine handling of packet loss (#523)
- Stream engine checksum alerting fixed
- Various rule analyzer fixes (#495, #496, #497)
- (Rule) profiling fixed and improved (#460, #466)
- Enforce limit on max-pending-packets (#510)
- fast_pattern on negated content improved
- TLS rule keyword parsing issues
- Windows build fixes (#502)
- Host OS parsing issues fixed (#499)
- Reject signatures where content length is bigger than "depth" setting (#505)
- Removed unused "prune-flows" option
- Set main thread and live reload thread names (#498)

1.3 -- 2012-07-06

- make live rule reloads optional and disabled by default
- fix a shutdown bug
- fix several memory leaks (#492)
- warn user if global and rule thresholding conflict (#455)
- set thread names on FreeBSD (Nikolay Denev)
- Fix PF_RING building on Ubuntu 12.04
- rule analyzer updates
- file inspection improvements when dealing with limits (#493)

1.3rc1 -- 2012-06-29

- experimental live rule reload by sending a USR2 signal (#279)
- AF_PACKET BPF support (#449)
- AF_PACKET live packet loss counters (#441)
- Rule analyzer (#349)
- add pcap workers runmode for use with libpcap wrappers that support load balancing, such as  Napatech's or Myricom's
- negated filemd5 matching, allowing for md5 whitelisting
- signatures with depth and/or offset are now checked against packets in addition to the stream (#404)
- http_cookie keyword now also inspects "Set-Cookie" header (#479)
- filemd5 keyword no longer depends on log-file output module (#447)
- http_raw_header keyword inspects original header line terminators (#475)
- deal with double encoded URI (#464)
- improved SMB/SMB2/DCERPC robustness
- ICMPv6 parsing fixes
- improve HTTP body inspection
- stream.inline accuracy issues fixed (#339)
- general stability fixes (#482, #486)
- missing unittests added (#471)
- "threshold.conf not found" error made more clear (#446)
- IPS mode segment logging for Unified2 improved

1.3beta2 -- 2012-06-08

- experimental support for matching on large lists of known file MD5 checksums
- Improved performance for file_data, http_server_body and http_client_body keywords
- Improvements to HTTP handling: multipart parsing, gzip decompression
- Byte_extract can support negative offsets now (#445)
- Support for PF_RING 5.4 added. Many thanks to Chris Wakelin (#459)
- HOME_NET and EXTERNAL_NET and the other vars are now checked for common errors (#454)
- Improved error reporting when using too long address strings (#451)
- MD5 calculation improvements for daemon mode and other cases (#449)
- File inspection scripts: Added Syslog action for logging to local syslog. Thanks to Martin Holste.
- Rule parser is made more strict.
- Unified2 output overhaul, logging individual segments in more cases.
- detection_filter keyword accuracy problem was fixed (#453)
- Don't inspect cookie header with http header (#461)
- Crash with a rule with two byte_extract keywords (#456)
- SSL parser fixes. Thanks to Chris Wakelin for testing the patches! (#476)
- Accuracy issues in HTTP inspection fixed. Thanks to Rmkml (#452)
- Improve escaping of some characters in logs (#418)
- Checksum calculation bugs fixed
- IPv6 parsing issues fixed. Thanks to Michel Saborde.
- Endace DAG issues fixed. Thanks to Jason Ish from Endace.
- Various OpenBSD related fixes.
- Fixes for bugs found by Coverity source code analyzer.

1.3beta1 -- 2012-04-04

- TLS/SSL handshake parser, tls.subjectdn and tls.issuerdn keywords (#296, contributed by Pierre Chifflier)
- Napatech capture card support (contributed by Randy Caldejon -- nPulse)
- Scripts for looking up files / file md5's at Virus Total and others (contributed by Martin Holste)
- Test mode: -T option to test the config (#271)
- Ringbuffer and zero copy support for AF_PACKET
- Commandline options to list supported app layer protocols and keywords (#344, #414)
- File extraction for HTTP POST request that do not use multipart bodies
- On the fly md5 checksum calculation of extracted files
- Line based file log, in json format
- Basic support for including other yaml files into the main yaml
- New multi pattern engine: ac-bs
- Profiling improvements, added lock profiling code
- Improved HTTP CONNECT support in libhtp (#427, Brian Rectanus -- Qualys)
- Unified yaml naming convention, including fallback support (by Nikolay Denev)
- Improved Endace DAG support (#431, Jason Ish -- Endace)
- New default runmode: "autofp" (#433)
- Major rewrite of flow engine, improving scalability.
- Improved http_stat_msg and http_stat_code keywords (#394)
- Improved scalability for Tag and Threshold subsystems
- Made the rule keyword parser much stricter in detecting syntax errors
- Split "file" output into "file-store" and "file-log" outputs
- Much improved file extraction
- CUDA build fixes (#421)
- Various FP's reported by Rmkml (#403, #405, #411)
- IPv6 decoding and detection issues (reported by Michel Sarborde)
- PCAP logging crash (#422)
- Fixed many (potential) issues with the help of the Coverity source code analyzer
- Fixed several (potential) issues with the help of the cppcheck and clang/scan-build source code analyzers

1.2.1 -- 2012-01-20

- fix malformed unified2 records when writing alerts trigger by stream inspection (#402)
- only force a pseudo packet inspection cycle for TCP streams in a state >= established

1.2 -- 2012-01-19

- improved Windows/CYGWIN path handling (#387)
- fixed some issues with passing an interface or ip address with -i
- make live worker runmode threads adhere to the 'detect' cpu affinity settings

1.2rc1 -- 2012-01-11

- app-layer-events keyword: similar to the decoder-events and stream-events, this will allow matching on HTTP and SMTP events
- auto detection of checksum offloading per interface (#311)
- urilen options to match on raw or normalized URI (#341)
- flow keyword option "only_stream" and "no_stream"
- unixsock output options for all outputs except unified2 (PoC python script in the qa/ dir) (#250)
- in IPS mode, reject rules now also drop (#399)
- http_header now also inspects response headers (#389)
- "worker" runmodes for NFQ and IPFW
- performance improvement for "ac" pattern matcher
- allow empty/non-initialized flowints to be incremented
- PCRE-JIT is now enabled by default if available (#356)
- many file inspection and extraction improvements
- flowbits and flowints are now modified in a post-match action list
- general performance increasements
- fixed parsing really high sid numbers >2 Billion (#393)
- fixed ICMPv6 not matching in IP-only sigs (#363)

1.2beta1 -- 2011-12-19

- File name, type inspection and extraction for HTTP
- filename, fileext, filemagic and filestore keywords added
- "file" output for storing extracted files to disk
- file_data keyword support, inspecting normalized, dechunked, decompressed HTTP response body (feature #241
- new keyword http_server_body, pcre regex /S option
- Option to enable/disable core dumping from the suricata.yaml (enabled by default)
- Human readable size limit settings in suricata.yaml
- PF_RING bpf support (required PF_RING >= 5.1) (feature #334)
- tos keyword support (feature #364)
- IPFW IPS mode does now support multiple divert sockets
- New IPS running modes, Linux and FreeBSD do now support "worker" and "autofp"
- Improved alert accuracy in autofp and single runmodes
- major performance optimizations for the ac-gfbs pattern matcher implementation
- unified2 output fixes
- PF_RING supports privilege dropping now (bug #367)
- Improved detection of duplicate signatures

1.1.1 -- 2011-12-07

- Fix for a error in the smtp parser that could crash Suricata.
- Fix for AF_PACKET not compiling on modern linux systems like Fedora 16.

1.1 -- 2011-11-10

- CUDA build fixed
- minor pcap, AF_PACKET and PF_RING fixes (#368)
- bpf handling fix
- Windows CYGWIN build
- more cleanups

1.1rc1 -- 2011-11-03

- extended HTTP request logging for use with (among other things) http_agent for Sguil (#38)
- AF_PACKET report drop stats on shutdown (#325)
- new counters in stats.log for flow and stream engines (#348)
- SMTP parsing code support for BDAT command (#347)
- HTTP URI normalization no longer converts to lowercase (#362)
- AF_PACKET works with privileges dropping now (#361)
- Prelude output for state matches (#264, #355)
- update of the pattern matching code that should improve accuracy
- rule parser was made more strict (#295, #312)
- multiple event suppressions for the same SID was fixed (#366)
- several accuracy fixes
- removal of the unified1 output plugins (#353)

1.1beta3 -- 2011-10-25

- af-packet support for high speed packet capture
- "replace" keyword support (#303)
- new "workers" runmode for multi-dev and/or clustered PF_RING, AF_PACKET, pcap
- added "stream-event" keyword to match on TCP session anomalies
- support for suppress keyword was added (#274)
- byte_extract keyword support was added
- improved handling of timed out TCP sessions in the detection engine
- unified2 payload logging if detection was in the HTTP state (#264)
- improved accuracy of the HTTP transaction logging
- support for larger (64 bit) Flow/Stream memcaps (#332)
- major speed improvements for PCRE, including support for PCRE JIT
- support setting flowbits in ip-only rules (#292)
- performance increases on SSE3+ CPU's
- overhaul of the packet acquisition subsystem
- packet based performance profiling subsystem was added
- TCP SACK support was added to the stream engine
- updated included libhtp to 0.2.6 which fixes several issues

1.1beta2 -- 2011-04-13

- New keyword support: http_raw_uri (including /I for pcre), ssl_state, ssl_version (#258, #259, #260, #262).
- Inline mode for the stream engine (#230, #248).
- New keyword support: nfq_set_mark
- Included an example decoder-events.rules file
- api for adding and selecting runmodes was added
- pcap logging / recording output was added
- basic SCTP protocol parsing was added
- more fine grained CPU affinity setting support was added
- stream engine inspects stream in larger chunks
- fast_pattern support for http_method content modifier (#255)
- negation support for isdataat keyword (#257)
- configurable interval for stats.log updates (#247)
- new pf_ring runmode was added that scales better
- pcap live mode now handles the monitor interface going up and down
- several QA additions to "make check"
- NFQ (linux inline) mode was improved
- Alerts classification fix (#275)
- compiles and runs on big-endian systems (#63)
- unified2 output works around barnyard2 issues with DLT_RAW + IPv6

1.1beta1 -- 2010-12-21

- New keyword support: http_raw_header, http_stat_msg, http_stat_code.
- A new default pattern matcher, Aho-Corasick based, that uses much less memory.
- reference.config support as supplied by ET/ETpro and VRT.
- Much improved fast_pattern support, including for http_uri, http_client_body, http_header, http_raw_header.
- Improved parsers, especially the DCERPC parser.
- Much improved performance & accuracy.

1.0.5 -- 2011-07-25

- Fix stream reassembly bug #300. Thanks to Rmkml for the report.
- Fix several (potential) issues fixed after a source code scan with Coverity generously contributed by RedHat.

1.0.4 -- 2011-06-24

- LibHTP updated to 0.2.6
- Large number of (potential) issues fixed after a source code scan with Coverity generously contributed by RedHat.
- Large number of (potential) issues fixed after source code scans with the Clang static analizer.

1.0.3 -- 2011-04-13

- Fix broken checksum calculation for TCP/UDP in some cases
- Fix errors in the byte_test, byte_jump, http_method and http_header keywords
- Fix a ASN1 parsing issue
- Improve LibHTP memory handling
- Fix a defrag issue
- Fix several stream engine issues