From 7806ae8f57155b94c95065b2002c3e7b21eecdd4 Mon Sep 17 00:00:00 2001 From: Andreas Herz Date: Wed, 13 Jul 2016 22:51:05 +0200 Subject: [PATCH] docs: sync up to recent redmine --- doc/userguide/configuration/log-rotation.rst | 8 - doc/userguide/configuration/suricata-yaml.rst | 38 ++++- doc/userguide/dropping-privileges.rst | 21 --- doc/userguide/file-extraction/md5.rst | 7 - doc/userguide/output/custom-http-logging.rst | 2 - doc/userguide/performance/hyperscan.rst | 138 ++++++++++++++++++ doc/userguide/performance/packet_capture.rst | 80 ++++++++++ doc/userguide/rules/flow-keywords.rst | 15 +- doc/userguide/rules/header-keywords.rst | 7 +- doc/userguide/rules/http-keywords.rst | 36 ++--- doc/userguide/rules/rule-lua-scripting.rst | 4 - 11 files changed, 273 insertions(+), 83 deletions(-) create mode 100644 doc/userguide/performance/hyperscan.rst create mode 100644 doc/userguide/performance/packet_capture.rst diff --git a/doc/userguide/configuration/log-rotation.rst b/doc/userguide/configuration/log-rotation.rst index 63ee24e5b3..4bf507189e 100644 --- a/doc/userguide/configuration/log-rotation.rst +++ b/doc/userguide/configuration/log-rotation.rst @@ -21,11 +21,3 @@ Example logrotate file: endscript } -newsyslog based log rotation (e.g. on OpenBSD) /etc/newsyslog.conf: - -:: - - /var/log/suricata/eve.json root:wheel 640 1 * 24 B /var/run/suricata.pid SIGHUP - -The above rotates every 24h; the 'B' prevents a rotation logmessage in -eve.json. Fieldseperator is a TAB. diff --git a/doc/userguide/configuration/suricata-yaml.rst b/doc/userguide/configuration/suricata-yaml.rst index f318e1c5ed..7a574d1745 100644 --- a/doc/userguide/configuration/suricata-yaml.rst +++ b/doc/userguide/configuration/suricata-yaml.rst @@ -807,8 +807,7 @@ application layer, detection, and outputs. # The packet acquisition module reads packets from the network. # The decode module decodes the packets and the stream application -layer performs its job here. The stream application layer has three -tasks: +application layer has three tasks: :: @@ -926,6 +925,41 @@ the settings in 'default' will count. By default Suricata creates one prio: default: "medium" +Relevant cpu-affinity settings for IDS/IPS modes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +IDS mode +~~~~~~~~ + +Runmode AutoFp:: + + management-cpu-set - used for management (example - flow.managers, flow.recyclers) + recive-cpu-set - used for receive and decode + detect-cpu-set - used for streamtcp,detect,output(logging),reject + +Rumode Workers:: + + management-cpu-set - used for management (example - flow.managers, flow.recyclers) + detect-cpu-set - used for receive,streamtcp,decode,detect,output(logging),respond/reject + + +IPS mode +~~~~~~~~ + +Runmode AutoFp:: + + management-cpu-set - used for management (example - flow.managers, flow.recyclers) + recive-cpu-set - used for receive and decode + detect-cpu-set - used for streamtcp,detect,output(logging) + verdict-cpu-set - used for verdict and respond/reject + +Runmode Workers:: + + management-cpu-set - used for management (example - flow.managers, flow.recyclers) + detect-cpu-set - used for receive,streamtcp,decode,detect,output(logging),respond/reject, verdict + + + IP Defrag --------- diff --git a/doc/userguide/dropping-privileges.rst b/doc/userguide/dropping-privileges.rst index bdcd930d63..ade11c1b49 100644 --- a/doc/userguide/dropping-privileges.rst +++ b/doc/userguide/dropping-privileges.rst @@ -48,24 +48,3 @@ suricata can still write to its log files which are usually located in chown -R root:suri /var/log/suricata chmod -R 775 /var/log/suricata -If you rely on the pid files be aware that as of version 1.4.x -Suricata writes the pid file before it switches uids. This means that -the pid file will be owned by root and will not be readable by the -alternative uid. If this is an issue for you then a work around is to -set the sgid bit on the directory that the pid file will be created -in. This will cause all files in this directory to inherit the gid -from the parent directory rather than the creating process. - -:: - - chmod g+s ~sensors/sensor1/run - - ls -ld ~sensors/sensor1/run - drwxr-sr-x 2 sensors sensors 4096 Aug 9 09:20 ~sensors/sensor1/run - -and so after running suricata we get - -:: - - ls -ld ~sensors/sensor1/run/suricata.pid - -rw-r----- 1 root sensors 6 Aug 9 09:20 ~sensors/sensor1/run/suricata.pid diff --git a/doc/userguide/file-extraction/md5.rst b/doc/userguide/file-extraction/md5.rst index 419a3970b0..3cc94cad18 100644 --- a/doc/userguide/file-extraction/md5.rst +++ b/doc/userguide/file-extraction/md5.rst @@ -74,13 +74,6 @@ You have to compile/install suri like this in order to enable MD5s: make sudo make install -For Fedora 64-bit users, the configure line will look like - - -:: - - - ./configure --with-libnss-libraries=/usr/lib64 --with-libnss-includes=/usr/include/nss3 --with-libnspr-libraries=/usr/lib64 --with-libnspr-includes=/usr/include/nspr4 Output of configure: diff --git a/doc/userguide/output/custom-http-logging.rst b/doc/userguide/output/custom-http-logging.rst index 9ddb5ba80a..f7d21adac5 100644 --- a/doc/userguide/output/custom-http-logging.rst +++ b/doc/userguide/output/custom-http-logging.rst @@ -28,8 +28,6 @@ And in your http.log file you would get the following, for example: 08/28/12-22:14:30.693856 - Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:11.0) Gecko/20100101 Firefox/11.0 HTTP/1.1 GET us.cnn.com /video/data/3.0/video/showbiz/2012/08/28/conan-reports-from-rnc-convention.teamcoco/index.xml 200 15789 192.168.1.91:45108 -> 157.166.255.18:80 -It should be noted that each line represents the http request and response all in one, depending on how you specified your custom format. HTTP responses do not get their own line and if you wish to see that information, you will need to modify the custom format to include it using %{header_field}o for example. - The list of supported format strings is the following: * %h - Host HTTP Header (remote host name). ie: google.com diff --git a/doc/userguide/performance/hyperscan.rst b/doc/userguide/performance/hyperscan.rst new file mode 100644 index 0000000000..64c23a7cc3 --- /dev/null +++ b/doc/userguide/performance/hyperscan.rst @@ -0,0 +1,138 @@ +Hyperscan guide for Ubuntu +========================== + +Introduction +============ + +"Hyperscan is a high-performance multiple regex matching library." https://01.org/hyperscan + +In Suricata it can be used to perform multi pattern matching (mpm). Support was implemented by Justin Viiret and Jim Xu from Intel: https://github.com/inliniac/suricata/pull/1965, https://redmine.openinfosecfoundation.org/issues/1704 + + +Installation +============ + +To use Suricata with Hyperscan support, install dependencies: + + +:: + + + apt-get install cmake ragel + +libboost headers +---------------- + +Hyperscan needs the libboost headers from 1.58+. + +On Ubuntu 15.10 or 16.04+, simply do: + + +:: + + + apt-get install libboost-dev + + +Trusty +~~~~~~ + +Trusty has 1.57, so it's too old. We can grab a newer libboost version, but we *don't* install it system wide. It's only the headers we care about during compilation of Hyperscan. + + +:: + + + sudo apt-get python-dev libbz2-dev + wget http://downloads.sourceforge.net/project/boost/boost/1.60.0/boost_1_60_0.tar.gz + tar xvzf boost_1_60_0.tar.gz + cd boost_1_60_0 + ./bootstrap.sh --prefix=~/tmp/boost-1.60 + ./b2 install + +Hyperscan +--------- + +We'll install version 4.2.0. + + +:: + + + git clone https://github.com/01org/hyperscan + cd hyperscan + mkdir build + cd build + cmake -DBUILD_STATIC_AND_SHARED=1 ../ + +If you have your own libboost headers, use this cmake line instead: + +:: + + + cmake -DBUILD_STATIC_AND_SHARED=1 -DBOOST_ROOT=~/tmp/boost-1.60 ../ + +Finally, make and make install: + +:: + + + make + sudo make install + +Compilation can take a long time, but it should in the end look something like this: + + +:: + + + Install the project... + -- Install configuration: "RELWITHDEBINFO" + -- Installing: /usr/local/lib/pkgconfig/libhs.pc + -- Up-to-date: /usr/local/include/hs/hs.h + -- Up-to-date: /usr/local/include/hs/hs_common.h + -- Up-to-date: /usr/local/include/hs/hs_compile.h + -- Up-to-date: /usr/local/include/hs/hs_runtime.h + -- Installing: /usr/local/lib/libhs_runtime.a + -- Installing: /usr/local/lib/libhs_runtime.so.4.2.0 + -- Installing: /usr/local/lib/libhs_runtime.so.4.2 + -- Installing: /usr/local/lib/libhs_runtime.so + -- Installing: /usr/local/lib/libhs.a + -- Installing: /usr/local/lib/libhs.so.4.2.0 + -- Installing: /usr/local/lib/libhs.so.4.2 + -- Installing: /usr/local/lib/libhs.so + +Note that you may have to add /usr/local/lib to your ld search path + + +:: + + + echo "/usr/local/lib" | sudo tee --append /etc/ld.so.conf.d/usrlocal.conf + sudo ldconfig + +Suricata +-------- + +Compilation +~~~~~~~~~~~ + +Suricata's installation is now quite standard. + +It's possible to pass --with-libhs-includes=/usr/local/include/hs/ --with-libhs-libraries=/usr/local/lib/, although by default this shouldn't be necessary. Suricata should pick up Hyperscan's pkg-config file automagically. + +When Suricata's compilation succeeded, you should have: + +:: + + + suricata --build-info|grep Hyperscan + Hyperscan support: yes + + +Using Hyperscan +~~~~~~~~~~~~~~~ + +To use the hyperscan support edit your suricata.yaml and change the mpm-algo value to 'hs'. + +Alternatively, use this commandline option: --set mpm-algo=hs diff --git a/doc/userguide/performance/packet_capture.rst b/doc/userguide/performance/packet_capture.rst new file mode 100644 index 0000000000..148c0b416f --- /dev/null +++ b/doc/userguide/performance/packet_capture.rst @@ -0,0 +1,80 @@ +Packet Capture +============== + +Load balancing +-------------- + +To get the best performance, Suricata will need to run in 'workers' mode. This effectively means that there are multiple threads, each running a full packet pipeline and each receiving packets from the capture method. This means that we rely on the capture method to distribute the packets over the various threads. One critical aspect of this is that Suricata needs to get both sides of a flow in the same thread, in the correct order. + +The AF_PACKET and PF_RING capture methods both have options to select the 'cluster-type'. These default to 'cluster_flow' which instructs the capture method to hash by flow (5 tuple). This hash is symmetric. Netmap does not have a cluster_flow mode built-in. It can be added separately by using the "'lb' tool":https://github.com/luigirizzo/netmap/tree/master/apps/lb + +> **WARNING** Recent AF_PACKET changes have "broken":https://redmine.openinfosecfoundation.org/issues/1777 this symmetry. Work is under way to "address this":https://redmine.openinfosecfoundation.org/issues/1777#note-7, but for now stay on kernel <=4.2. + +On multi-queue NICs, which is almost any modern NIC, RSS settings need to be considered. + + +RSS +--- + +Receive Side Scaling is a technique used by network cards to distribute incoming traffic over various queues on the NIC. This is meant to improve performance but it is important to realize that it was designed for normal traffic, not for the IDS packet capture scenario. RSS using a hash algorithm to distribute the incoming traffic over the various queues. This hash is normally *not* symmetrical. This means that when receiving both sides of a flow, each side may end up in a different queue. Sadly, when deploying Suricata, this is the common scenario when using span ports or taps. + +The problem here is that by having both sides of the traffic in different queues, the order of processing of packets becomes unpredictable. Timing differences on the NIC, the driver, the kernel and in Suricata will lead to a high chance of packets coming in at a different order than on the wire. This is specifically about a mismatch between the two traffic directions. For example, Suricata tracks the TCP 3-way handshake. Due to this timing issue, the SYN/ACK may only be received by Suricata long after the client to server side has already started sending data. Suricata would see this traffic as invalid. + +None of the supported capture methods like AF_PACKET, PF_RING or NETMAP can fix this problem for us. It would require buffering and packet reordering which is expensive. + +To see how many queues are configured: + +:: + + + $ ethtool -l ens2f1 + Channel parameters for ens2f1: + Pre-set maximums: + RX: 0 + TX: 0 + Other: 1 + Combined: 64 + Current hardware settings: + RX: 0 + TX: 0 + Other: 1 + Combined: 8 + +Some NIC's allow you to set it into a symmetric mode. The Intel X(L)710 card can do this in theory, but the drivers aren't capable of enabling this yet (work is underway to try to address this). Another way to address is by setting a special "Random Secret Key" that will make the RSS symmetrical. See http://www.ndsl.kaist.edu/~kyoungsoo/papers/TR-symRSS.pdf (PDF). + +In most scenario's however, the optimal solution is to reduce the number of RSS queues to 1: + +Example: + +:: + + + # Intel X710 with i40e driver: + ethtool -L $DEV combined 1 + +Some drivers do not support setting the number of queues through ethtool. In some cases there is a module load time option. Read the driver docs for the specifics. + + +Offloading +---------- + +Network cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled. + +LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking. + +Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others. + + + +Recommendations +--------------- + +Read your drivers documentation! E.g. for i40e the ethtool change of RSS queues may lead to kernel panics if done wrong. + +Generic: set RSS queues to 1 or make sure RSS hashing is symmetric. Disable NIC offloading. + +AF_PACKET: 1 RSS queue and stay on kernel <=4.2 until futher notice. Exception: if RSS is symmetric cluster-type 'cluster_qm' can be used to bind Suricata to the RSS queues. Disable NIC offloading except the rx/tx csum. + +PF_RING: 1 RSS queue and use cluster-type 'cluster_flow'. Disable NIC offloading except the rx/tx csum. + +NETMAP: 1 RSS queue. There is no flow based load balancing built-in, but the 'lb' tool can be helpful. Another option is to use the 'autofp' runmode. Exception: if RSS is symmetric, load balancing is based on the RSS hash and multiple RSS queues can be used. Disable all NIC offloading. \ No newline at end of file diff --git a/doc/userguide/rules/flow-keywords.rst b/doc/userguide/rules/flow-keywords.rst index ba057fd99a..75824db03c 100644 --- a/doc/userguide/rules/flow-keywords.rst +++ b/doc/userguide/rules/flow-keywords.rst @@ -29,7 +29,7 @@ Flowbits have different actions. These are: flowbits: unset, name Can be used to unset the condition in the flow. flowbits: isnotset, name Can be used in the rule to make sure it generates an alert when it matches and the condition is not set in the flow. - flowbits: noalert Does not generate an alert for this rule. + flowbits: noalert No alert will be generated by this rule. Example: @@ -52,13 +52,10 @@ different functions. Flow ~~~~ -The flow keyword can be used to match on direction of the flow, so -to/from client or to/from server. The to_client is equal to -from_server, same goes for from_client and to_server - there's no -difference, both names are supported due to historical reasons. It can -also match if the flow is established or not. The flow keyword can -also be use to say the signature has to match on stream only -(only_stream) or on packet only (no_stream). +The flow keyword can be used to match on direction of the flow, so to/from +client or to/from server. It can also match if the flow is established or not. +The flow keyword can also be use to say the signature has to match on stream +only (only_stream) or on packet only (no_stream). So with the flow keyword you can match on: @@ -73,7 +70,7 @@ So with the flow keyword you can match on: *to_server* and _from_client_. This comes from the original Snort language and we support it for compatibility reasons. -These options from the different columns can be combined. You can use +These options can be combined. You can use up to three options. For example: :: diff --git a/doc/userguide/rules/header-keywords.rst b/doc/userguide/rules/header-keywords.rst index a4bc69d305..dc1c589f9a 100644 --- a/doc/userguide/rules/header-keywords.rst +++ b/doc/userguide/rules/header-keywords.rst @@ -88,12 +88,9 @@ Example of ip_proto in a rule: .. image:: header-keywords/ip_proto.png -An example of the text based version of the protocol, say IPv6 whose -code is 41 would be:: +The named variante of that example would be:: - ip_proto:IPv6 - -No quotes are needed around the name. + ip_proto:PIM Id -- diff --git a/doc/userguide/rules/http-keywords.rst b/doc/userguide/rules/http-keywords.rst index 9584014a2f..594ecd4376 100644 --- a/doc/userguide/rules/http-keywords.rst +++ b/doc/userguide/rules/http-keywords.rst @@ -3,13 +3,11 @@ HTTP Keywords ============= -There are additional content modifiers that can provide -protocol-specific capabilities at the application layer (if you are -unfamiliar with content modifiers, please visit the page -:doc:`payload-keywords` These keywords make sure the signature checks -only specific parts of the network traffic. For instance, to check -specifically on the request URI, cookies, or the HTTP request or -response body, etc. +There are additional content modifiers that can provide protocol-specific +capabilities at the application layer. More information can be found at +:doc:`payload-keywords` These keywords make sure the signature checks only +specific parts of the network traffic. For instance, to check specifically on +the request URI, cookies, or the HTTP request or response body, etc. Use ``http_method`` to match on the HTTP request method, ``http_uri`` or ``http_raw_uri`` to match on the request URI, ``http_stat_code`` to @@ -159,9 +157,8 @@ modifiers, like ``depth``, ``distance``, ``offset``, ``nocase`` and **Note**: the header buffer is *normalized*. Any trailing whitespace and tab characters are removed. See: - http://lists.openinfosecfoundation.org/pipermail/oisf-users/2011-October/000935.html. Match - on the ``http_raw_header`` buffer if you require the raw - (non-normalized) headers. + http://lists.openinfosecfoundation.org/pipermail/oisf-users/2011-October/000935.html. + To avoid that, use the ``http_raw_header`` keyword. Example of a header in a HTTP request: @@ -279,9 +276,7 @@ http_server_body With the ``http_server_body`` content modifier, it is possible to match specifically and only on the HTTP response body. The keyword can be used in combination with all previously mentioned content modifiers -like ``distance``, ``offset``, ``nocase``, ``within``, etc. If the -response body is *gzip* encoded, it is first uncompressed for -inspection. +like ``distance``, ``offset``, ``nocase``, ``within``, etc. Note: how much of the response/server body is inspected is controlled in your :ref:`libhtp configuration section @@ -320,20 +315,13 @@ The ``file_data`` keyword affects all following content matches, until the ``pkt_data`` keyword is encountered or it reaches the end of the rule. This makes it a useful shortcut for applying many content matches to the HTTP response body, eliminating the need to modify each -content match individually. If the response body is *gzip* encoded, it -is first uncompressed for inspection. +content match individually. Note: how much of the response/server body is inspected is controlled in your :ref:`libhtp configuration section ` via the ``response-body-limit`` setting. -**NOTE:** In 2.0.x, ``file_data`` is only supported for HTTP server -bodies (specified as flow direction **to_client**). Starting with -2.1-dev and above, ``file_data`` is handled more intelligently -per-protocol, supported with **to_server** for SMTP and **to_client** -for HTTP. - urilen ------ @@ -360,10 +348,8 @@ Example of ``urilen`` in a signature: .. image:: http-keywords/urilen1.png -You can specify whether the inspected URI buffer is either the -normalized or the raw buffer by appending ``norm`` or ``raw``:: - - urilen:<5,raw; +You can also append ``norm`` or ``raw`` to define what sort of buffer you want +to use (normalized or raw buffer). pcre ---- diff --git a/doc/userguide/rules/rule-lua-scripting.rst b/doc/userguide/rules/rule-lua-scripting.rst index 91f3de1af8..726f1ab322 100644 --- a/doc/userguide/rules/rule-lua-scripting.rst +++ b/doc/userguide/rules/rule-lua-scripting.rst @@ -1,10 +1,6 @@ Lua Scripting ============= -In order to enable Lua scripting, please reference this page before -continuing `Installation from GIT with luajit -`_. - Syntax: ::