@ -1,9 +1,27 @@
eBPF and XDP
============
Set up
======
Introduction
------------
eBPF stands for extended BPF. This is an extended version of Berkeley Packet Filter available in recent
Linux kernel versions.
It provides more advanced features with eBPF programs developed in C and able to use structured data shared
between kernel and userspace.
eBPF is used for three things in Suricata:
- eBPF filter: any BPF like filter can be developed. An example of filter accepting only packet for some VLANs is provided.
- eBPF load balancing: provide programmable load balancing. A simple ippair load balancing is provided.
- XDP programs: suricata can load XDP programs. A bypass program is provided.
Bypass can be implemented in eBPF and XDP. The advantage of XDP is that the packets are dropped at the earliest stage
possible. So performance is better. But bypassed packets don't reach the network so you can't use this on regular
traffic but only on duplicated/sniffed traffic.
XDP
---
~~~
XDP provides another Linux native way of optimising Suricata's performance on sniffing high speed networks.
@ -16,14 +34,22 @@ XDP provides another Linux native way of optimising Suricata's performance on sn
Requirements
------------
- Newer kernel that supports XDP (4.13.10 in our case)
- RSS symmetric hashing on the NIC ( Intel 82599ES 10-Gigabit/x520/x540 in our case)
- In tree kernel drivers NIC drivers
You will need a kernel that supports XDP and, for real performance improvement, a network
card that support XDP in the driver.
Suricata XDP code has been tested with 4.13.10 but 4.15 or later is necessary to have all
features like the CPU redirect map.
If yu are using an Intel netword card, you will need to stay with in tree kernel NIC drivers.
The out of tree drivers do not contain the XDP support.
Having a network card with support for RSS symmetric hashing is a good point or you will have to
use the XDP CPU redirect map feature.
Prerequisites
-------------
This guide has been confirmed on Debian/Ubutnu LTS Linux.
This guide has been confirmed on Debian/Ubunt u " LTS" Linux.
Disable irqbalance
~~~~~~~~~~~~~~~~~~
@ -41,18 +67,20 @@ Install kernel 4.13.+ and reboot.
Clang
~~~~~
Make sure you have clang installed on the system.
::
Make sure you have clang (>=3.9) installed on the system ::
apt-get install clang
sudo apt-get install clang
The version 3.9 is recommended as it seems some more recent versions of clang
generate invalid eBPF code that fail to load.
BPF
~~~
Suricata use libbpf to interact with eBPF and XDP. This library is available
in the Linux tree. Before 4.16, a patched libbpf library is also needed::
Suricata uses libbpf to interact with eBPF and XDP. This library is available
in the Linux tree. Before Linux 4.16, a patched libbpf library is also needed::
git clone -b libbpf-xdp https://github.com/regit/linux.git
git clone -b libbpf-v14 https://github.com/regit/linux.git
If you have a recent enough kernel, you can skip this part.
@ -61,21 +89,22 @@ Now, you can build and install the library ::
cd linux/tools/lib/bpf/
make && sudo make install
sudo mkdir -p /usr/local/include/bpf/
sudo cp *bpf.h /usr/local/include/bpf/
sudo make install_headers
sudo ldconfig
Compile and install Suricata
----------------------------
::
If ever you don't have the source ::
git clone https://github.com/OISF/suricata.git
cd suricata && git clone https://github.com/OISF/libhtp.git -b 0.5.x
./autogen.sh
Then you need to add the ebpf flags to configure ::
CC=clang ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ \
--enable-ebpf --enable-ebpf-build
@ -84,6 +113,102 @@ Compile and install Suricata
sudo ldconfig
sudo mkdir /etc/suricata/ebpf/
Setup bypass
------------
If you plan to use eBPF or XDP for a kernel/hardware level bypass, you need to do
the following:
First, enable `bypass` `stream`
stream:
bypass: true
If you want, you can also bypass encrypted flow by setting `no-reassemble` `yes`
in the app-layer tls section ::
app-layer:
protocols:
tls:
enabled: yes
detection-ports:
dp: 443
# Completely stop processing TLS/SSL session after the handshake
# completed. If bypass is enabled this will also trigger flow
# bypass. If disabled (the default), TLS/SSL session is still
# tracked for Heartbleed and other anomalies.
no-reassemble: yes
Setup eBPF filter
-----------------
Copy the resulting ebpf fiter as needed ::
cp src/ebpf/vlan_filter.bpf /etc/suricata/ebpf/
Then setup the `ebpf-filter-file`
- interface: eth3
threads: 16
cluster-id: 97
cluster-type: cluster_flow # choose any type suitable
defrag: yes
# eBPF file containing a 'loadbalancer' function that will be inserted into the
# kernel and used as load balancing function
ebpf-filter-file: /etc/suricata/ebpf/vlan_filter.bpf
use-mmap: yes
ring-size: 200000
You can then run suricata normally ::
/usr/bin/suricata --pidfile /var/run/suricata.pid --af-packet=eth3 -vvv
You can also use eBPF bypass. To do that load the `bypass_filter.bpf`
update af-packet configuration to set bypass to yes ::
- interface: eth3
threads: 16
cluster-id: 97
cluster-type: cluster_qm # symmetric hashing is a must!
defrag: yes
# eBPF file containing a 'loadbalancer' function that will be inserted into the
# kernel and used as load balancing function
#ebpf-lb-file: /etc/suricata/ebpf/lb.bpf
# eBPF file containing a 'filter' function that will be inserted into the
# kernel and used as packet filter function
# eBPF file containing a 'xdp' function that will be inserted into the
# kernel and used as XDP packet filter function
ebpf-filter-file: /etc/suricata/ebpf/bypass_filter.bpf
bypass: yes
use-mmap: yes
ring-size: 200000
Setup eBPF load balancing
-------------------------
Copy the resulting ebpf fiter as needed ::
cp src/ebpf/lb.bpf /etc/suricata/
We will use `` cluster_ebpf `` in the interface section of af-packet ::
- interface: eth3
threads: 16
cluster-id: 97
cluster-type: cluster_ebpf
defrag: yes
# eBPF file containing a 'loadbalancer' function that will be inserted into the
# kernel and used as load balancing function
ebpf-lb-file: /etc/suricata/ebpf/lb.bpf
use-mmap: yes
ring-size: 200000
Setup XDP
---------
Copy the resulting xdp fiter as needed::
cp src/ebpf/xdp_filter.bpf /etc/suricata/ebpf/
@ -96,7 +221,7 @@ also use the ``/etc/suricata/ebpf/xdp_filter.bpf`` (in our example TCP offloadin
- interface: eth3
threads: 16
cluster-id: 97
cluster-type: cluster_qm # symmetric hashing is a must!
cluster-type: cluster_qm # symmetric hashing is a must!
defrag: yes
# eBPF file containing a 'loadbalancer' function that will be inserted into the
# kernel and used as load balancing function
@ -116,43 +241,18 @@ also use the ``/etc/suricata/ebpf/xdp_filter.bpf`` (in our example TCP offloadin
use-mmap: yes
ring-size: 200000
Also enable "bypass" in the "stream" section ::
stream:
bypass: true
If you with you can also bypass encrypted flow by setting no-reassemble to yes
in the app-layer tls section ::
app-layer:
protocols:
tls:
enabled: yes
detection-ports:
dp: 443
# Completely stop processing TLS/SSL session after the handshake
# completed. If bypass is enabled this will also trigger flow
# bypass. If disabled (the default), TLS/SSL session is still
# tracked for Heartbleed and other anomalies.
no-reassemble: yes
Setup symmetric hashing on the NIC
----------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Intel network card don't support symmetric hashing but it is possible to emulate
it by using a specific hashing function.
Follow these instructions closely for desired result::
ifconfig eth3 down
Use in tree kernel drivers, XDP support is not available in out of source Intel drivers.
*NOTE:* In this case the kernel and sources used is 4.13.10
Reload the module with updated parameters ::
cd /lib/modules/4.13.10-amd64/kernel/drivers/net/ethernet/intel/ixgbe
rmmod ixgbe && insmod ixgbe.ko MQ=1,1 RSS=0,0 InterruptThrottleRate=12500,12500 LRO=0,0 vxlan_rx=0,0
Use in tree kernel drivers: XDP support is not available in Intel drivers available on Intel website.
Enable symmetric hashing::
@ -166,22 +266,22 @@ Enable symmetric hashing::
ethtool -x eth3
ethtool -n eth3
In the above set up you are free to use any recent `` set_irq_affinity `` script. It is available in any Intel x520/710 NIC sources driver download.
In the above setup you are free to use any recent `` set_irq_affinity `` script. It is available in any Intel x520/710 NIC sources driver download.
**NOTE:**
We use a special low entropy key for the symmetric hashing. `More info about the research for symmetric hashing set up <http://www.ndsl.kaist.edu/~kyoungsoo/papers/TR-symRSS.pdf> `_
Disable an NIC offloading
-------------------------
Disable any NIC offloading
~~~~~~~~~~~~~~~~~~~~~~~~~~
Run the following to disable offloading ::
Run the following command to disable offloading ::
for i in rx tx tso ufo gso gro lro tx nocache copy sg txvlan rxvlan; do
/sbin/ethtool -K eth3 $i off 2>&1 > /dev/null;
done
Balance as much as you can
--------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~
Try to use the network's card balancing as much as possible ::
@ -190,7 +290,7 @@ Try to use the network's card balancing as much as possible ::
done
Start Suricata with XDP
-----------------------
~~~~~~~~~~~~~~~~~~~~~~~
You can now start Suricata with XDP bypass activated ::
Eric Leblond
8 years ago