From 60265e023a4c146503d49f6b56c00e9cdb92a027 Mon Sep 17 00:00:00 2001 From: Eric Leblond Date: Fri, 5 Jan 2018 19:31:43 +0100 Subject: [PATCH] doc: update xdp documentation Also remove configuration info from yaml as they are now in the documentation. --- .../{xdp.rst => ebpf-xdp.rst} | 206 +++++++++++++----- doc/userguide/capture-hardware/index.rst | 1 + suricata.yaml.in | 23 +- 3 files changed, 158 insertions(+), 72 deletions(-) rename doc/userguide/capture-hardware/{xdp.rst => ebpf-xdp.rst} (51%) diff --git a/doc/userguide/capture-hardware/xdp.rst b/doc/userguide/capture-hardware/ebpf-xdp.rst similarity index 51% rename from doc/userguide/capture-hardware/xdp.rst rename to doc/userguide/capture-hardware/ebpf-xdp.rst index 13e1a62c2a..e5ad750062 100644 --- a/doc/userguide/capture-hardware/xdp.rst +++ b/doc/userguide/capture-hardware/ebpf-xdp.rst @@ -1,9 +1,27 @@ +eBPF and XDP +============ -Set up -====== +Introduction +------------ + +eBPF stands for extended BPF. This is an extended version of Berkeley Packet Filter available in recent +Linux kernel versions. + +It provides more advanced features with eBPF programs developed in C and able to use structured data shared +between kernel and userspace. + +eBPF is used for three things in Suricata: + +- eBPF filter: any BPF like filter can be developed. An example of filter accepting only packet for some VLANs is provided. +- eBPF load balancing: provide programmable load balancing. A simple ippair load balancing is provided. +- XDP programs: suricata can load XDP programs. A bypass program is provided. + +Bypass can be implemented in eBPF and XDP. The advantage of XDP is that the packets are dropped at the earliest stage +possible. So performance is better. But bypassed packets don't reach the network so you can't use this on regular +traffic but only on duplicated/sniffed traffic. XDP ---- +~~~ XDP provides another Linux native way of optimising Suricata's performance on sniffing high speed networks. @@ -16,14 +34,22 @@ XDP provides another Linux native way of optimising Suricata's performance on sn Requirements ------------ -- Newer kernel that supports XDP (4.13.10 in our case) -- RSS symmetric hashing on the NIC ( Intel 82599ES 10-Gigabit/x520/x540 in our case) -- In tree kernel drivers NIC drivers +You will need a kernel that supports XDP and, for real performance improvement, a network +card that support XDP in the driver. + +Suricata XDP code has been tested with 4.13.10 but 4.15 or later is necessary to have all +features like the CPU redirect map. + +If yu are using an Intel netword card, you will need to stay with in tree kernel NIC drivers. +The out of tree drivers do not contain the XDP support. + +Having a network card with support for RSS symmetric hashing is a good point or you will have to +use the XDP CPU redirect map feature. Prerequisites ------------- -This guide has been confirmed on Debian/Ubutnu LTS Linux. +This guide has been confirmed on Debian/Ubuntu "LTS" Linux. Disable irqbalance ~~~~~~~~~~~~~~~~~~ @@ -41,18 +67,20 @@ Install kernel 4.13.+ and reboot. Clang ~~~~~ -Make sure you have clang installed on the system. -:: +Make sure you have clang (>=3.9) installed on the system :: - apt-get install clang + sudo apt-get install clang + +The version 3.9 is recommended as it seems some more recent versions of clang +generate invalid eBPF code that fail to load. BPF ~~~ -Suricata use libbpf to interact with eBPF and XDP. This library is available -in the Linux tree. Before 4.16, a patched libbpf library is also needed:: +Suricata uses libbpf to interact with eBPF and XDP. This library is available +in the Linux tree. Before Linux 4.16, a patched libbpf library is also needed:: - git clone -b libbpf-xdp https://github.com/regit/linux.git + git clone -b libbpf-v14 https://github.com/regit/linux.git If you have a recent enough kernel, you can skip this part. @@ -61,21 +89,22 @@ Now, you can build and install the library :: cd linux/tools/lib/bpf/ make && sudo make install - sudo mkdir -p /usr/local/include/bpf/ - sudo cp *bpf.h /usr/local/include/bpf/ + sudo make install_headers sudo ldconfig Compile and install Suricata ---------------------------- -:: +If ever you don't have the source :: git clone https://github.com/OISF/suricata.git cd suricata && git clone https://github.com/OISF/libhtp.git -b 0.5.x ./autogen.sh +Then you need to add the ebpf flags to configure :: + CC=clang ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ \ --enable-ebpf --enable-ebpf-build @@ -84,6 +113,102 @@ Compile and install Suricata sudo ldconfig sudo mkdir /etc/suricata/ebpf/ +Setup bypass +------------ + +If you plan to use eBPF or XDP for a kernel/hardware level bypass, you need to do +the following: + +First, enable `bypass` in the `stream` section :: + + stream: + bypass: true + +If you want, you can also bypass encrypted flow by setting `no-reassemble` to `yes` +in the app-layer tls section :: + + app-layer: + protocols: + tls: + enabled: yes + detection-ports: + dp: 443 + + # Completely stop processing TLS/SSL session after the handshake + # completed. If bypass is enabled this will also trigger flow + # bypass. If disabled (the default), TLS/SSL session is still + # tracked for Heartbleed and other anomalies. + no-reassemble: yes + + +Setup eBPF filter +----------------- + +Copy the resulting ebpf fiter as needed :: + + cp src/ebpf/vlan_filter.bpf /etc/suricata/ebpf/ + +Then setup the `ebpf-filter-file` variable in af-packet section :: + + - interface: eth3 + threads: 16 + cluster-id: 97 + cluster-type: cluster_flow # choose any type suitable + defrag: yes + # eBPF file containing a 'loadbalancer' function that will be inserted into the + # kernel and used as load balancing function + ebpf-filter-file: /etc/suricata/ebpf/vlan_filter.bpf + use-mmap: yes + ring-size: 200000 + +You can then run suricata normally :: + + /usr/bin/suricata --pidfile /var/run/suricata.pid --af-packet=eth3 -vvv + +You can also use eBPF bypass. To do that load the `bypass_filter.bpf` file and +update af-packet configuration to set bypass to yes :: + + - interface: eth3 + threads: 16 + cluster-id: 97 + cluster-type: cluster_qm # symmetric hashing is a must! + defrag: yes + # eBPF file containing a 'loadbalancer' function that will be inserted into the + # kernel and used as load balancing function + #ebpf-lb-file: /etc/suricata/ebpf/lb.bpf + # eBPF file containing a 'filter' function that will be inserted into the + # kernel and used as packet filter function + # eBPF file containing a 'xdp' function that will be inserted into the + # kernel and used as XDP packet filter function + ebpf-filter-file: /etc/suricata/ebpf/bypass_filter.bpf + bypass: yes + use-mmap: yes + ring-size: 200000 + + +Setup eBPF load balancing +------------------------- + +Copy the resulting ebpf fiter as needed :: + + cp src/ebpf/lb.bpf /etc/suricata/ + +We will use ``cluster_ebpf`` in the interface section of af-packet :: + + - interface: eth3 + threads: 16 + cluster-id: 97 + cluster-type: cluster_ebpf + defrag: yes + # eBPF file containing a 'loadbalancer' function that will be inserted into the + # kernel and used as load balancing function + ebpf-lb-file: /etc/suricata/ebpf/lb.bpf + use-mmap: yes + ring-size: 200000 + +Setup XDP +--------- + Copy the resulting xdp fiter as needed:: cp src/ebpf/xdp_filter.bpf /etc/suricata/ebpf/ @@ -96,7 +221,7 @@ also use the ``/etc/suricata/ebpf/xdp_filter.bpf`` (in our example TCP offloadin - interface: eth3 threads: 16 cluster-id: 97 - cluster-type: cluster_qm # symmetric hashing is a must! + cluster-type: cluster_qm # symmetric hashing is a must! defrag: yes # eBPF file containing a 'loadbalancer' function that will be inserted into the # kernel and used as load balancing function @@ -116,43 +241,18 @@ also use the ``/etc/suricata/ebpf/xdp_filter.bpf`` (in our example TCP offloadin use-mmap: yes ring-size: 200000 -Also enable "bypass" in the "stream" section :: - - stream: - bypass: true - -If you with you can also bypass encrypted flow by setting no-reassemble to yes -in the app-layer tls section :: - - app-layer: - protocols: - tls: - enabled: yes - detection-ports: - dp: 443 - - # Completely stop processing TLS/SSL session after the handshake - # completed. If bypass is enabled this will also trigger flow - # bypass. If disabled (the default), TLS/SSL session is still - # tracked for Heartbleed and other anomalies. - no-reassemble: yes - Setup symmetric hashing on the NIC ----------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Intel network card don't support symmetric hashing but it is possible to emulate +it by using a specific hashing function. Follow these instructions closely for desired result:: ifconfig eth3 down -Use in tree kernel drivers, XDP support is not available in out of source Intel drivers. - -*NOTE:* In this case the kernel and sources used is 4.13.10 - -Reload the module with updated parameters :: - - cd /lib/modules/4.13.10-amd64/kernel/drivers/net/ethernet/intel/ixgbe - rmmod ixgbe && insmod ixgbe.ko MQ=1,1 RSS=0,0 InterruptThrottleRate=12500,12500 LRO=0,0 vxlan_rx=0,0 +Use in tree kernel drivers: XDP support is not available in Intel drivers available on Intel website. Enable symmetric hashing:: @@ -166,22 +266,22 @@ Enable symmetric hashing:: ethtool -x eth3 ethtool -n eth3 -In the above set up you are free to use any recent ``set_irq_affinity`` script. It is available in any Intel x520/710 NIC sources driver download. +In the above setup you are free to use any recent ``set_irq_affinity`` script. It is available in any Intel x520/710 NIC sources driver download. **NOTE:** We use a special low entropy key for the symmetric hashing. `More info about the research for symmetric hashing set up `_ -Disable an NIC offloading -------------------------- +Disable any NIC offloading +~~~~~~~~~~~~~~~~~~~~~~~~~~ -Run the following to disable offloading :: +Run the following command to disable offloading :: for i in rx tx tso ufo gso gro lro tx nocache copy sg txvlan rxvlan; do /sbin/ethtool -K eth3 $i off 2>&1 > /dev/null; done Balance as much as you can --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ Try to use the network's card balancing as much as possible :: @@ -190,7 +290,7 @@ Try to use the network's card balancing as much as possible :: done Start Suricata with XDP ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ You can now start Suricata with XDP bypass activated :: diff --git a/doc/userguide/capture-hardware/index.rst b/doc/userguide/capture-hardware/index.rst index a078541e36..309e664fd8 100644 --- a/doc/userguide/capture-hardware/index.rst +++ b/doc/userguide/capture-hardware/index.rst @@ -6,3 +6,4 @@ Using Capture Hardware endace-dag napatech myricom + ebpf-xdp diff --git a/suricata.yaml.in b/suricata.yaml.in index 96334717ca..5d2498a33c 100644 --- a/suricata.yaml.in +++ b/suricata.yaml.in @@ -620,31 +620,14 @@ af-packet: # Requires at least Linux 3.14. # * cluster_rollover: kernel rotates between sockets filling each socket before moving # to the next. Requires at least Linux 3.10. - # * cluster_ebpf: kernel rotates between sockets following an extended BPF function loaded - # from a file. User needs to use ebpf-lb-file option to set the path to the file - # to load. Requires at least Linux 4.2. + # * cluster_ebpf: eBPF file load balancing. See doc/userguide/capture/ebpf-xdt.rst for + # more info. # Recommended modes are cluster_flow on most boxes and cluster_cpu or cluster_qm on system # with capture card using RSS (require cpu affinity tuning and system irq tuning) cluster-type: cluster_flow # In some fragmentation case, the hash can not be computed. If "defrag" is set # to yes, the kernel will do the needed defragmentation before sending the packets. defrag: yes - # eBPF file containing a 'loadbalancer' function that will be inserted into the - # kernel and used as load balancing function - #ebpf-lb-file: @e_sysconfdir@/ebpf/lb.bpf - # eBPF file containing a 'filter' function that will be inserted into the - # kernel and used as packet filter function - #ebpf-filter-file: @e_sysconfdir@/ebpf/filter.bpf - # eBPF file containing a 'xdp' function that will be inserted into the - # kernel and used as XDP packet filter function - #ebpf-filter-file: @e_sysconfdir@filter.bpf - # Xdp mode, "soft" for skb based version, "driver" for network card based - # and "hw" for card supporting eBPF. - #xdp-mode: driver - #xdp-filter-file: @e_sysconfdir@xdp_filter.bpf - # if the ebpf filter implements a bypass function, you can set 'bypass' to - # yes and benefit from these feature - #bypass: yes # After Linux kernel 3.10 it is possible to activate the rollover option: if a socket is # full then kernel will send the packet on the next socket with room available. This option # can minimize packet drop and increase the treated bandwidth on single intensive flow. @@ -697,6 +680,8 @@ af-packet: # will not be copied. #copy-mode: ips #copy-iface: eth1 + # For eBPF and XDP setup including bypass, filter and load balancing, please + # see doc/userguide/capture/ebpf-xdt.rst for more info. # Put default values here. These will be used for an interface that is not # in the list above.