summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* ARM: net: bpf: improve 64-bit ALU implementationRussell King2018-07-131-5/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improbe the 64-bit ALU implementation from: movw r8, #65532 movt r8, #65535 movw r9, #65535 movt r9, #65535 ldr r7, [fp, #-44] adds r7, r7, r8 str r7, [fp, #-44] ldr r7, [fp, #-40] adc r7, r7, r9 str r7, [fp, #-40] to: movw r8, #65532 movt r8, #65535 movw r9, #65535 movt r9, #65535 ldrd r6, [fp, #-44] adds r6, r6, r8 adc r7, r7, r9 strd r6, [fp, #-44] Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* ARM: net: bpf: improve 64-bit store implementationRussell King2018-07-131-26/+26
| | | | | | | | | | | | | | | | | | | | | | | Improve the 64-bit store implementation from: ldr r6, [fp, #-8] str r8, [r6] ldr r6, [fp, #-8] mov r7, #4 add r7, r6, r7 str r9, [r7] to: ldr r6, [fp, #-8] str r8, [r6] str r9, [r6, #4] We leave the store as two separate STR instructions rather than using STRD as the store may not be aligned, and STR can handle misalignment. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* ARM: net: bpf: improve 64-bit sign-extended immediate loadRussell King2018-07-131-4/+3
| | | | | | | | | | | | | | | | | | Improve the 64-bit sign-extended immediate from: mov r6, #1 str r6, [fp, #-52] ; 0xffffffcc mov r6, #0 str r6, [fp, #-48] ; 0xffffffd0 to: mov r6, #1 mov r7, #0 strd r6, [fp, #-52] ; 0xffffffcc Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* ARM: net: bpf: improve 64-bit load immediate implementationRussell King2018-07-131-12/+20
| | | | | | | | | | | | | | | | | | | | | Rather than writing each 32-bit half of the 64-bit immediate value separately when the register is on the stack: movw r6, #45056 ; 0xb000 movt r6, #60979 ; 0xee33 str r6, [fp, #-44] ; 0xffffffd4 mov r6, #0 str r6, [fp, #-40] ; 0xffffffd8 arrange to use the double-word store when available instead: movw r6, #45056 ; 0xb000 movt r6, #60979 ; 0xee33 mov r7, #0 strd r6, [fp, #-44] ; 0xffffffd4 Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* Merge branch 'bpf-arm-jit-improvements'Daniel Borkmann2018-07-122-481/+543
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Russell King says: ==================== This series improves the ARM BPF JIT compiler by: - enumerating the stack layout rather than using constants that happen to be multiples of four - rejig the BPF "register" accesses to use negative numbers instead of positive, which could be confused with register numbers in the bpf2a32 array. - since we maintain the ARM FP register as a pointer to the top of our scratch space (or, with frame pointers enabled, a valid ARM frame pointer register), we can access our scratch space using FP, which is constant across all BPF programs, including tail-called programs. - use immediate forms of ARM instructions where possible, rather than first loading the immediate into an ARM register. - use load-with-shift instruction rather than seperate shift instruction followed by load - avoid reloading index and array in the tail-call code - use double-word load/store instructions where available Version 2: - Fix ARMv5 test pointed out by Olof - Fix build error found by 0-day (adding an additional patch) ==================== Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: use double-word load/stores where availableRussell King2018-07-122-12/+45
| | | | | | | | | | | | | | | | Use double-word load and stores where support for this instruction is supported by the CPU architecture. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: always use odd/even register pairRussell King2018-07-121-14/+14
| | | | | | | | | | | | | | | | Always use an odd/even register pair for our 64-bit registers, so that we're able to use the double-word load/store instructions in the future. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: avoid reloading 'array'Russell King2018-07-121-5/+4
| | | | | | | | | | | | | | | | Rearranging the order of the initial tail call code a little allows is to avoid reloading the 'array' pointer. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: avoid reloading 'index'Russell King2018-07-121-2/+3
| | | | | | | | | | | | | | | | | | | | Avoid reloading 'index' after we have validated it - it remains in tmp2[1] up to the point that we begin the code to index the pointer array, so with a little rearrangement of the registers, we can use the already loaded value. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: use ldr instructions with shifted rm registerRussell King2018-07-122-2/+5
| | | | | | | | | | | | | | | | | | Rather than pre-shifting the rm register for the ldr in the tail call, shift it in the load instruction. This eliminates one unnecessary instruction. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: use immediate forms of instructions where possibleRussell King2018-07-122-9/+13
| | | | | | | | | | | | | | | | | | | | Rather than moving constants to a register and then using them in a subsequent instruction, use them directly in the desired instruction cutting out the "middle" register. This removes two instructions from the tail call code path. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: imm12 constant conversionRussell King2018-07-121-1/+49
| | | | | | | | | | | | | | | | Provide a version of the imm8m() function that the compiler can optimise when used with a constant expression. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: access eBPF scratch space using ARM FP registerRussell King2018-07-121-19/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | Access the eBPF scratch space using the frame pointer rather than our stack pointer, as the offsets from the ARM frame pointer are constant across all eBPF programs. Since we no longer reference the scratch space registers from the stack pointer, this simplifies emit_push_r64() as it no longer needs to know how many words are pushed onto the stack. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: 64-bit accessor functions for BPF registersRussell King2018-07-121-113/+122
| | | | | | | | | | | | | | | | Provide a couple of 64-bit register accessors, and use them where appropriate Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: provide accessor functions for BPF registersRussell King2018-07-121-201/+128
| | | | | | | | | | | | | | | | | | | | Many of the code paths need to have knowledge about whether a register is stacked or in a CPU register. Move this decision making to a pair of helper functions instead of having it scattered throughout the code. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: remove is_on_stack() and sstk/dstkRussell King2018-07-121-176/+160
| | | | | | | | | | | | | | | | | | | | | | | | The decision about whether a BPF register is on the stack or in a CPU register is detected at the top BPF insn processing level, and then percolated throughout the remainder of the code. Since we now use negative register values to represent stacked registers, we can detect where a BPF register is stored without restoring to carrying this additional metadata through all code paths. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: use negative numbers for stacked registersRussell King2018-07-121-98/+102
| | | | | | | | | | | | | | Use negative numbers for eBPF registers that live on the stack. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: provide load/store ops with negative immediatesRussell King2018-07-122-22/+41
| | | | | | | | | | | | | | | | Provide a set of load/store opcode generators that work with negative immediates as well as positive ones. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * ARM: net: bpf: enumerate the JIT scratch stack layoutRussell King2018-07-121-17/+42
|/ | | | | | | | Enumerate the contents of the JIT scratch stack layout used for storing some of the JITs 64-bit registers, tail call counter and AX register. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* Merge branch 'bpf-helper-man-install'Daniel Borkmann2018-07-124-17/+93
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Quentin Monnet says: ==================== The three patches in this series are related to the documentation for eBPF helpers. The first patch brings minor formatting edits to the documentation in include/uapi/linux/bpf.h, and the second one updates the related header file under tools/. The third patch adds a Makefile under tools/bpf for generating the documentation (man pages) about eBPF helpers. The targets defined in this file can also be called from the bpftool directory (please refer to relevant commit logs for details). ==================== Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpf: build and install man page for eBPF helpers from bpftool/Quentin Monnet2018-07-122-5/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Provide a new Makefile.helpers in tools/bpf, in order to build and install the man page for eBPF helpers. This Makefile is also included in the one used to build bpftool documentation, so that it can be called either on its own (cd tools/bpf && make -f Makefile.helpers) or from bpftool directory (cd tools/bpf/bpftool && make doc, or cd tools/bpf/bpftool/Documentation && make helpers). Makefile.helpers is not added directly to bpftool to avoid changing its Makefile too much (helpers are not 100% directly related with bpftool). But the possibility to build the page from bpftool directory makes us able to package the helpers man page with bpftool, and to install it along with bpftool documentation, so that the doc for helpers becomes easily available to developers through the "man" program. Cc: linux-man@vger.kernel.org Suggested-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpf: synchronise BPF UAPI header with toolsQuentin Monnet2018-07-121-8/+24
| | | | | | | | | | | | | | | | Update with latest changes from include/uapi/linux/bpf.h header. Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * bpf: fix documentation for eBPF helpersQuentin Monnet2018-07-121-4/+2
|/ | | | | | | | | | Minor formatting edits for eBPF helpers documentation, including blank lines removal, fix of item list for return values in bpf_fib_lookup(), and missing prefix on bpf_skb_load_bytes_relative(). Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* Merge branch 'bpf-bpftool-improved-prog-load'Daniel Borkmann2018-07-1118-108/+906
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Jakub Kicinski says: ==================== This series starts with two minor clean ups to test_offload.py selftest script. The next 11 patches extend the abilities of bpftool prog load beyond the simple cgroup use cases. Three new parameters are added: - type - allows specifying program type, independent of how code sections are named; - map - allows reusing existing maps, instead of creating a new map on every program load; - dev - offload/binding to a device. A number of changes to libbpf is required to accomplish the task. The section - program type logic mapping is exposed. We should probably aim to use the libbpf program section naming everywhere. For reuse of maps we need to allow users to set FD for bpf map object in libbpf. Examples Load program my_xdp.o and pin it as /sys/fs/bpf/my_xdp, for xdp program type: $ bpftool prog load my_xdp.o /sys/fs/bpf/my_xdp \ type xdp As above but for offload: $ bpftool prog load my_xdp.o /sys/fs/bpf/my_xdp \ type xdp \ dev netdevsim0 Load program my_maps.o, but for the first map reuse map id 17, and for the map called "other_map" reuse pinned map /sys/fs/bpf/map0: $ bpftool prog load my_maps.o /sys/fs/bpf/prog \ map idx 0 id 17 \ map name other_map pinned /sys/fs/bpf/map0 v3: - fix return codes in patch 5; - rename libbpf_prog_type_by_string() -> libbpf_prog_type_by_name(); - fold file path into xattr in patch 8; - add patch 10; - use dup3() in patch 12; - depend on fd value in patch 12; - close old fd in patch 12. v2: - add compat for reallocarray(). ==================== Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpftool: allow reuse of maps with bpftool prog loadJakub Kicinski2018-07-115-23/+219
| | | | | | | | | | | | | | | | | | | | | | | | | | Add map parameter to prog load which will allow reuse of existing maps instead of creating new ones. We need feature detection and compat code for reallocarray, since it's not available in many libc versions. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Acked-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: libbpf: allow map reuseJakub Kicinski2018-07-112-0/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | More advanced applications may want to only replace programs without destroying associated maps. Allow libbpf users to achieve that. Instead of always creating all of the maps at load time, expose to users an API to reconstruct the map object from already existing map. The map parameters are read from the kernel and replace the parameters of the ELF map. libbpf does not restrict the map replacement, i.e. the reused map does not have to be compatible with the ELF map definition. We relay on the verifier for checking the compatibility between maps and programs. The ELF map definition is completely overwritten by the information read from the kernel, to make sure libbpf's view of map object corresponds to the actual map. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Acked-by: Andrey Ignatov <rdna@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpf: make use of reallocarrayJakub Kicinski2018-07-1110-9/+334
| | | | | | | | | | | | | | | | | | | | | | | | | | reallocarray() is a safer variant of realloc which checks for multiplication overflow in case of array allocation. Since it's not available in Glibc < 2.26 import kernel's overflow.h and add a static inline implementation when needed. Use feature detection to probe for existence of reallocarray. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Reviewed-by: Jiong Wang <jiong.wang@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: libbpf: move library error code into a separate fileJakub Kicinski2018-07-113-49/+75
| | | | | | | | | | | | | | | | | | | | | | | | | | libbpf_strerror() depends on XSI-compliant (POSIX) version of strerror_r(), which prevents us from using GNU-extensions in libbpf.c, like reallocarray() or dup3(). Move error printing code into a separate file to allow it to continue using POSIX strerror_r(). No functional changes. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpftool: reimplement bpf_prog_load() for prog loadJakub Kicinski2018-07-111-13/+48
| | | | | | | | | | | | | | | | | | | | | | | | bpf_prog_load() is a very useful helper but it doesn't give us full flexibility of modifying the BPF objects before loading. Open code bpf_prog_load() in bpftool so we can add extra logic in following commits. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Acked-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: libbpf: add extended attributes version of bpf_object__open()Jakub Kicinski2018-07-112-6/+26
| | | | | | | | | | | | | | | | | | | | | | | | Similarly to bpf_prog_load() users of bpf_object__open() may need to specify the expected program type. Program type is needed at open to avoid the kernel version check for program types which don't require it. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Acked-by: Andrey Ignatov <rdna@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: libbpf: recognize offload neutral mapsJakub Kicinski2018-07-112-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | Add helper to libbpf for recognizing maps which should not have ifindex set when program is loaded. These maps only contain host metadata and therefore are not marked for offload, e.g. the perf event map. Use this helper in bpf_prog_load_xattr(). Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpftool: allow users to specify program type for prog loadJakub Kicinski2018-07-113-5/+60
| | | | | | | | | | | | | | | | | | | | | | Sometimes program section names don't match with libbpf's expectation. In particular XDP's default section names differ between libbpf and iproute2. Allow users to pass program type on command line. Name the types like the libbpf expected section names. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: libbpf: expose the prog type guessing from section name logicJakub Kicinski2018-07-112-18/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | libbpf can guess program type based on ELF section names. As libbpf becomes more popular its association between section name strings and types becomes more of a standard. Allow libbpf users to use the same logic for matching strings to types, e.g. when the string originates from command line. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Acked-by: Andrey Ignatov <rdna@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpftool: add support for loading programs for offloadJakub Kicinski2018-07-113-6/+58
| | | | | | | | | | | | | | | | | | Extend the bpftool prog load command to also accept "dev" parameter, which will allow us to load programs onto devices. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * tools: bpftool: refactor argument parsing for prog loadJakub Kicinski2018-07-112-4/+22
| | | | | | | | | | | | | | | | | | | | Add a new macro for printing more informative message than straight usage() when parameters are missing, and use it for prog do_load(). Save the object and pin path argument to variables for clarity. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * selftests/bpf: add Error: prefix in check_extack helperJakub Kicinski2018-07-111-4/+4
| | | | | | | | | | | | | | | | | | Currently the test only checks errors, not warnings, so save typing and prefix the extack messages with "Error:" inside the check helper. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * selftests/bpf: remove duplicated word from test offloadsJakub Kicinski2018-07-111-1/+1
|/ | | | | | | | Trivial removal of duplicated "mode" in error message. Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com> Reviewed-by: Quentin Monnet <quentin.monnet@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* Merge branch 'bpf-nfp-mul-div-support'Daniel Borkmann2018-07-077-37/+483
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Jiong Wang says: ==================== NFP supports u16 and u32 multiplication. Multiplication is done 8-bits per step, therefore we need 2 steps for u16 and 4 steps for u32. We also need one start instruction to initialize the sequence and one or two instructions to fetch the result depending on either you need the high halve of u32 multiplication. For ALU64, if either operand is beyond u32's value range, we reject it. One thing to note, if the source operand is BPF_K, then we need to check "imm" field directly, and we'd reject it if it is negative. Because for ALU64, "imm" (with s32 type) is expected to be sign extended to s64 which NFP mul doesn't support. For ALU32, it is fine for "imm" be negative though, because the result is 32-bits and here is no difference on the low halve of result for signed/unsigned mul, so we will get correct result. NFP doesn't have integer divide instruction, this patch set uses reciprocal algorithm (the basic one, reciprocal_div) to emulate it. For each u32 divide, we would need 11 instructions to finish the operation. 7 (for multiplication) + 4 (various ALUs) = 11 Given NFP only supports multiplication no bigger than u32, we'd require divisor and dividend no bigger than that as well. Also eBPF doesn't support signed divide and has enforced this on C language level by failing compilation. However LLVM assembler hasn't enforced this, so it is possible for negative constant to leak in as a BPF_K operand through assembly code, we reject such cases as well. Meanwhile reciprocal_div.h only implemented the basic version of: "Division by Invariant Integers Using Multiplication" - Torbjörn Granlund and Peter L. Montgomery This patch set further implements the optimized version (Figure 4.2 in the paper) inside existing reciprocal_div.h. When the divider is even and the calculated reciprocal magic number doesn't fit u32, we could reduce the required ALU instructions from 4 to 2 or 1 for some cases. The advanced version requires more complex calculation to get the reciprocal multiplier and other control variables, but then could reduce the required emulation operations. It makes sense to use it for JIT divide code generation (for example eBPF JIT backends) for which we are willing to trade performance of JITed code with that of host. v2: - add warning in l == 32 code path. (Song Liu/Jakub) - jit separate insn sequence for l == 32. (Jakub/Edwin) - should use unrestricted operand for mul. - once place should use "1U << exp" instead of "1 << exp". ==================== Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * nfp: bpf: migrate to advanced reciprocal divide in reciprocal_div.hJiong Wang2018-07-071-11/+53
| | | | | | | | | | | | | | | | | | | | | | As we are doing JIT, we would want to use the advanced version of the reciprocal divide (reciprocal_value_adv) to trade performance with host. We could reduce the required ALU instructions from 4 to 2 or 1. Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * nfp: bpf: support u32 divide using reciprocal_div.hJiong Wang2018-07-073-1/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | NFP doesn't have integer divide instruction, this patch use reciprocal algorithm (the basic one, reciprocal_div) to emulate it. For each u32 divide, we would need 11 instructions to finish the operation. 7 (for multiplication) + 4 (various ALUs) = 11 Given NFP only supports multiplication no bigger than u32, we'd require divisor and dividend no bigger than that as well. Also eBPF doesn't support signed divide and has enforced this on C language level by failing compilation. However LLVM assembler hasn't enforced this, so it is possible for negative constant to leak in as a BPF_K operand through assembly code, we reject such cases as well. Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Acked-by: Song Liu <songliubraving@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * nfp: bpf: support u16 and u32 multiplicationsJiong Wang2018-07-074-11/+217
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | NFP supports u16 and u32 multiplication. Multiplication is done 8-bits per step, therefore we need 2 steps for u16 and 4 steps for u32. We also need one start instruction to initialize the sequence and one or two instructions to fetch the result depending on either you need the high halve of u32 multiplication. For ALU64, if either operand is beyond u32's value range, we reject it. One thing to note, if the source operand is BPF_K, then we need to check "imm" field directly, and we'd reject it if it is negative. Because for ALU64, "imm" (with s32 type) is expected to be sign extended to s64 which NFP mul doesn't support. For ALU32, it is fine for "imm" be negative though, because the result is 32-bits and here is no difference on the low halve of result for signed/unsigned mul, so we will get correct result. Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Acked-by: Song Liu <songliubraving@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * nfp: bpf: copy range info for all operands of all ALU operationsJiong Wang2018-07-073-23/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | NFP verifier hook is coping range information of the shift amount for indirect shift operation so optimized shift sequences could be generated. We want to use range info to do more things. For example, to decide whether multiplication and divide are supported on the given range. This patch simply let NFP verifier hook to copy range info for all operands of all ALU operands. Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Acked-by: Song Liu <songliubraving@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * nfp: bpf: rename umin/umax to umin_src/umax_srcJiong Wang2018-07-074-14/+14
| | | | | | | | | | | | | | | | | | | | | | | | The two fields are a copy of umin and umax info of bpf_insn->src_reg generated by verifier. Rename to make their meaning clear. Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Acked-by: Song Liu <songliubraving@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
| * lib: reciprocal_div: implement the improved algorithm on the paper mentionedJiong Wang2018-07-072-0/+109
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new added "reciprocal_value_adv" implements the advanced version of the algorithm described in Figure 4.2 of the paper except when "divisor > (1U << 31)" whose ceil(log2(d)) result will be 32 which then requires u128 divide on host. The exception case could be easily handled before calling "reciprocal_value_adv". The advanced version requires more complex calculation to get the reciprocal multiplier and other control variables, but then could reduce the required emulation operations. It makes no sense to use this advanced version for host divide emulation, those extra complexities for calculating multiplier etc could completely waive our saving on emulation operations. However, it makes sense to use it for JIT divide code generation (for example eBPF JIT backends) for which we are willing to trade performance of JITed code with that of host. As shown by the following pseudo code, the required emulation operations could go down from 6 (the basic version) to 3 or 4. To use the result of "reciprocal_value_adv", suppose we want to calculate n/d, the C-style pseudo code will be the following, it could be easily changed to real code generation for other JIT targets. struct reciprocal_value_adv rvalue; u8 pre_shift, exp; // handle exception case. if (d >= (1U << 31)) { result = n >= d; return; } rvalue = reciprocal_value_adv(d, 32) exp = rvalue.exp; if (rvalue.is_wide_m && !(d & 1)) { // floor(log2(d & (2^32 -d))) pre_shift = fls(d & -d) - 1; rvalue = reciprocal_value_adv(d >> pre_shift, 32 - pre_shift); } else { pre_shift = 0; } // code generation starts. if (imm == 1U << exp) { result = n >> exp; } else if (rvalue.is_wide_m) { // pre_shift must be zero when reached here. t = (n * rvalue.m) >> 32; result = n - t; result >>= 1; result += t; result >>= rvalue.sh - 1; } else { if (pre_shift) result = n >> pre_shift; result = ((u64)result * rvalue.m) >> 32; result >>= rvalue.sh; } Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* bpftool: add bash completion for cgroup tree commandRoman Gushchin2018-07-071-1/+5
| | | | | | | | | | | | | This commit adds a bash completion to the bpftool cgroup tree command. Signed-off-by: Roman Gushchin <guro@fb.com> Cc: Jakub Kicinski <jakub.kicinski@netronome.com> Cc: Quentin Monnet <quentin.monnet@netronome.com> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Alexei Starovoitov <ast@kernel.org> Acked-by: Jakub Kicinski <jakub.kicinski@netronome.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* bpftool: document cgroup tree commandRoman Gushchin2018-07-071-1/+11
| | | | | | | | | | | Describe cgroup tree command in the corresponding bpftool man page. Signed-off-by: Roman Gushchin <guro@fb.com> Acked-by: Jakub Kicinski <jakub.kicinski@netronome.com> Cc: Quentin Monnet <quentin.monnet@netronome.com> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* bpftool: introduce cgroup tree commandRoman Gushchin2018-07-071-5/+165
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit introduces a new bpftool command: cgroup tree. The idea is to iterate over the whole cgroup tree and print all attached programs. I was debugging a bpf/systemd issue, and found, that there is no simple way to listen all bpf programs attached to cgroups. I did master something in bash, but after some time got tired of it, and decided, that adding a dedicated bpftool command could be a better idea. So, here it is: $ sudo ./bpftool cgroup tree CgroupPath ID AttachType AttachFlags Name /sys/fs/cgroup/system.slice/systemd-machined.service 18 ingress 17 egress /sys/fs/cgroup/system.slice/systemd-logind.service 20 ingress 19 egress /sys/fs/cgroup/system.slice/systemd-udevd.service 16 ingress 15 egress /sys/fs/cgroup/system.slice/systemd-journald.service 14 ingress 13 egress Signed-off-by: Roman Gushchin <guro@fb.com> Acked-by: Jakub Kicinski <jakub.kicinski@netronome.com> Cc: Quentin Monnet <quentin.monnet@netronome.com> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
* Merge branch 'IP-listification-follow-ups'David S. Miller2018-07-064-24/+133
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | Edward Cree says: ==================== IP listification follow-ups While working on IPv6 list processing, I found another bug in the IPv4 version. So this patch series has that fix, and the IPv6 version with both fixes incorporated. ==================== Signed-off-by: David S. Miller <davem@davemloft.net>
| * net: ipv6: listify ipv6_rcv() and ip6_rcv_finish()Edward Cree2018-07-063-16/+118
| | | | | | | | | | | | | | Essentially the same as the ipv4 equivalents. Signed-off-by: Edward Cree <ecree@solarflare.com> Signed-off-by: David S. Miller <davem@davemloft.net>
| * net: ipv4: fix list processing on L3 slave devicesEdward Cree2018-07-061-8/+15
|/ | | | | | | | | | | | | If we have an L3 master device, l3mdev_ip_rcv() will steal the skb, but we were returning NET_RX_SUCCESS from ip_rcv_finish_core() which meant that ip_list_rcv_finish() would keep it on the list. Instead let's move the l3mdev_ip_rcv() call into the caller, so that our response to a steal can be different in the single packet path (return NET_RX_SUCCESS) and the list path (forget this packet and continue). Fixes: 5fa12739a53d ("net: ipv4: listify ip_rcv_finish") Signed-off-by: Edward Cree <ecree@solarflare.com> Signed-off-by: David S. Miller <davem@davemloft.net>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-30 15:27:11 +0100