summaryrefslogtreecommitdiffstats
path: root/Documentation/target/tcmu-design.txt
diff options
context:
space:
mode:
authorMauro Carvalho Chehab <mchehab+samsung@kernel.org>2019-06-12 19:52:59 +0200
committerJonathan Corbet <corbet@lwn.net>2019-06-14 22:31:36 +0200
commit4ca9bc225e46eb7bc040dd948be7cb68975d80d3 (patch)
tree9da0a501a84850b63ff00a4c5ba5a578a1b17fd5 /Documentation/target/tcmu-design.txt
parentdocs: riscv: convert docs to ReST and rename to *.rst (diff)
downloadlinux-4ca9bc225e46eb7bc040dd948be7cb68975d80d3.tar.xz
linux-4ca9bc225e46eb7bc040dd948be7cb68975d80d3.zip
docs: target: convert docs to ReST and rename to *.rst
Convert the TCM docs to ReST format and add them to the bookset. This has a mix of userspace-faced and Kernelspace faced docs. Still, it sounds a better candidate to be added at the kernel API set of docs. The conversion is actually: - add blank lines and identation in order to identify paragraphs; - fix tables markups; - add some lists markups; - mark literal blocks; - adjust title markups. At its new index.rst, let's add a :orphan: while this is not linked to the main index.rst file, in order to avoid build warnings. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Diffstat (limited to 'Documentation/target/tcmu-design.txt')
-rw-r--r--Documentation/target/tcmu-design.txt381
1 files changed, 0 insertions, 381 deletions
diff --git a/Documentation/target/tcmu-design.txt b/Documentation/target/tcmu-design.txt
deleted file mode 100644
index 4cebc1ebf99a..000000000000
--- a/Documentation/target/tcmu-design.txt
+++ /dev/null
@@ -1,381 +0,0 @@
-Contents:
-
-1) TCM Userspace Design
- a) Background
- b) Benefits
- c) Design constraints
- d) Implementation overview
- i. Mailbox
- ii. Command ring
- iii. Data Area
- e) Device discovery
- f) Device events
- g) Other contingencies
-2) Writing a user pass-through handler
- a) Discovering and configuring TCMU uio devices
- b) Waiting for events on the device(s)
- c) Managing the command ring
-3) A final note
-
-
-TCM Userspace Design
---------------------
-
-TCM is another name for LIO, an in-kernel iSCSI target (server).
-Existing TCM targets run in the kernel. TCMU (TCM in Userspace)
-allows userspace programs to be written which act as iSCSI targets.
-This document describes the design.
-
-The existing kernel provides modules for different SCSI transport
-protocols. TCM also modularizes the data storage. There are existing
-modules for file, block device, RAM or using another SCSI device as
-storage. These are called "backstores" or "storage engines". These
-built-in modules are implemented entirely as kernel code.
-
-Background:
-
-In addition to modularizing the transport protocol used for carrying
-SCSI commands ("fabrics"), the Linux kernel target, LIO, also modularizes
-the actual data storage as well. These are referred to as "backstores"
-or "storage engines". The target comes with backstores that allow a
-file, a block device, RAM, or another SCSI device to be used for the
-local storage needed for the exported SCSI LUN. Like the rest of LIO,
-these are implemented entirely as kernel code.
-
-These backstores cover the most common use cases, but not all. One new
-use case that other non-kernel target solutions, such as tgt, are able
-to support is using Gluster's GLFS or Ceph's RBD as a backstore. The
-target then serves as a translator, allowing initiators to store data
-in these non-traditional networked storage systems, while still only
-using standard protocols themselves.
-
-If the target is a userspace process, supporting these is easy. tgt,
-for example, needs only a small adapter module for each, because the
-modules just use the available userspace libraries for RBD and GLFS.
-
-Adding support for these backstores in LIO is considerably more
-difficult, because LIO is entirely kernel code. Instead of undertaking
-the significant work to port the GLFS or RBD APIs and protocols to the
-kernel, another approach is to create a userspace pass-through
-backstore for LIO, "TCMU".
-
-
-Benefits:
-
-In addition to allowing relatively easy support for RBD and GLFS, TCMU
-will also allow easier development of new backstores. TCMU combines
-with the LIO loopback fabric to become something similar to FUSE
-(Filesystem in Userspace), but at the SCSI layer instead of the
-filesystem layer. A SUSE, if you will.
-
-The disadvantage is there are more distinct components to configure, and
-potentially to malfunction. This is unavoidable, but hopefully not
-fatal if we're careful to keep things as simple as possible.
-
-Design constraints:
-
-- Good performance: high throughput, low latency
-- Cleanly handle if userspace:
- 1) never attaches
- 2) hangs
- 3) dies
- 4) misbehaves
-- Allow future flexibility in user & kernel implementations
-- Be reasonably memory-efficient
-- Simple to configure & run
-- Simple to write a userspace backend
-
-
-Implementation overview:
-
-The core of the TCMU interface is a memory region that is shared
-between kernel and userspace. Within this region is: a control area
-(mailbox); a lockless producer/consumer circular buffer for commands
-to be passed up, and status returned; and an in/out data buffer area.
-
-TCMU uses the pre-existing UIO subsystem. UIO allows device driver
-development in userspace, and this is conceptually very close to the
-TCMU use case, except instead of a physical device, TCMU implements a
-memory-mapped layout designed for SCSI commands. Using UIO also
-benefits TCMU by handling device introspection (e.g. a way for
-userspace to determine how large the shared region is) and signaling
-mechanisms in both directions.
-
-There are no embedded pointers in the memory region. Everything is
-expressed as an offset from the region's starting address. This allows
-the ring to still work if the user process dies and is restarted with
-the region mapped at a different virtual address.
-
-See target_core_user.h for the struct definitions.
-
-The Mailbox:
-
-The mailbox is always at the start of the shared memory region, and
-contains a version, details about the starting offset and size of the
-command ring, and head and tail pointers to be used by the kernel and
-userspace (respectively) to put commands on the ring, and indicate
-when the commands are completed.
-
-version - 1 (userspace should abort if otherwise)
-flags:
-- TCMU_MAILBOX_FLAG_CAP_OOOC: indicates out-of-order completion is
- supported. See "The Command Ring" for details.
-cmdr_off - The offset of the start of the command ring from the start
-of the memory region, to account for the mailbox size.
-cmdr_size - The size of the command ring. This does *not* need to be a
-power of two.
-cmd_head - Modified by the kernel to indicate when a command has been
-placed on the ring.
-cmd_tail - Modified by userspace to indicate when it has completed
-processing of a command.
-
-The Command Ring:
-
-Commands are placed on the ring by the kernel incrementing
-mailbox.cmd_head by the size of the command, modulo cmdr_size, and
-then signaling userspace via uio_event_notify(). Once the command is
-completed, userspace updates mailbox.cmd_tail in the same way and
-signals the kernel via a 4-byte write(). When cmd_head equals
-cmd_tail, the ring is empty -- no commands are currently waiting to be
-processed by userspace.
-
-TCMU commands are 8-byte aligned. They start with a common header
-containing "len_op", a 32-bit value that stores the length, as well as
-the opcode in the lowest unused bits. It also contains cmd_id and
-flags fields for setting by the kernel (kflags) and userspace
-(uflags).
-
-Currently only two opcodes are defined, TCMU_OP_CMD and TCMU_OP_PAD.
-
-When the opcode is CMD, the entry in the command ring is a struct
-tcmu_cmd_entry. Userspace finds the SCSI CDB (Command Data Block) via
-tcmu_cmd_entry.req.cdb_off. This is an offset from the start of the
-overall shared memory region, not the entry. The data in/out buffers
-are accessible via tht req.iov[] array. iov_cnt contains the number of
-entries in iov[] needed to describe either the Data-In or Data-Out
-buffers. For bidirectional commands, iov_cnt specifies how many iovec
-entries cover the Data-Out area, and iov_bidi_cnt specifies how many
-iovec entries immediately after that in iov[] cover the Data-In
-area. Just like other fields, iov.iov_base is an offset from the start
-of the region.
-
-When completing a command, userspace sets rsp.scsi_status, and
-rsp.sense_buffer if necessary. Userspace then increments
-mailbox.cmd_tail by entry.hdr.length (mod cmdr_size) and signals the
-kernel via the UIO method, a 4-byte write to the file descriptor.
-
-If TCMU_MAILBOX_FLAG_CAP_OOOC is set for mailbox->flags, kernel is
-capable of handling out-of-order completions. In this case, userspace can
-handle command in different order other than original. Since kernel would
-still process the commands in the same order it appeared in the command
-ring, userspace need to update the cmd->id when completing the
-command(a.k.a steal the original command's entry).
-
-When the opcode is PAD, userspace only updates cmd_tail as above --
-it's a no-op. (The kernel inserts PAD entries to ensure each CMD entry
-is contiguous within the command ring.)
-
-More opcodes may be added in the future. If userspace encounters an
-opcode it does not handle, it must set UNKNOWN_OP bit (bit 0) in
-hdr.uflags, update cmd_tail, and proceed with processing additional
-commands, if any.
-
-The Data Area:
-
-This is shared-memory space after the command ring. The organization
-of this area is not defined in the TCMU interface, and userspace
-should access only the parts referenced by pending iovs.
-
-
-Device Discovery:
-
-Other devices may be using UIO besides TCMU. Unrelated user processes
-may also be handling different sets of TCMU devices. TCMU userspace
-processes must find their devices by scanning sysfs
-class/uio/uio*/name. For TCMU devices, these names will be of the
-format:
-
-tcm-user/<hba_num>/<device_name>/<subtype>/<path>
-
-where "tcm-user" is common for all TCMU-backed UIO devices. <hba_num>
-and <device_name> allow userspace to find the device's path in the
-kernel target's configfs tree. Assuming the usual mount point, it is
-found at:
-
-/sys/kernel/config/target/core/user_<hba_num>/<device_name>
-
-This location contains attributes such as "hw_block_size", that
-userspace needs to know for correct operation.
-
-<subtype> will be a userspace-process-unique string to identify the
-TCMU device as expecting to be backed by a certain handler, and <path>
-will be an additional handler-specific string for the user process to
-configure the device, if needed. The name cannot contain ':', due to
-LIO limitations.
-
-For all devices so discovered, the user handler opens /dev/uioX and
-calls mmap():
-
-mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)
-
-where size must be equal to the value read from
-/sys/class/uio/uioX/maps/map0/size.
-
-
-Device Events:
-
-If a new device is added or removed, a notification will be broadcast
-over netlink, using a generic netlink family name of "TCM-USER" and a
-multicast group named "config". This will include the UIO name as
-described in the previous section, as well as the UIO minor
-number. This should allow userspace to identify both the UIO device and
-the LIO device, so that after determining the device is supported
-(based on subtype) it can take the appropriate action.
-
-
-Other contingencies:
-
-Userspace handler process never attaches:
-
-- TCMU will post commands, and then abort them after a timeout period
- (30 seconds.)
-
-Userspace handler process is killed:
-
-- It is still possible to restart and re-connect to TCMU
- devices. Command ring is preserved. However, after the timeout period,
- the kernel will abort pending tasks.
-
-Userspace handler process hangs:
-
-- The kernel will abort pending tasks after a timeout period.
-
-Userspace handler process is malicious:
-
-- The process can trivially break the handling of devices it controls,
- but should not be able to access kernel memory outside its shared
- memory areas.
-
-
-Writing a user pass-through handler (with example code)
--------------------------------------------------------
-
-A user process handing a TCMU device must support the following:
-
-a) Discovering and configuring TCMU uio devices
-b) Waiting for events on the device(s)
-c) Managing the command ring: Parsing operations and commands,
- performing work as needed, setting response fields (scsi_status and
- possibly sense_buffer), updating cmd_tail, and notifying the kernel
- that work has been finished
-
-First, consider instead writing a plugin for tcmu-runner. tcmu-runner
-implements all of this, and provides a higher-level API for plugin
-authors.
-
-TCMU is designed so that multiple unrelated processes can manage TCMU
-devices separately. All handlers should make sure to only open their
-devices, based opon a known subtype string.
-
-a) Discovering and configuring TCMU UIO devices:
-
-(error checking omitted for brevity)
-
-int fd, dev_fd;
-char buf[256];
-unsigned long long map_len;
-void *map;
-
-fd = open("/sys/class/uio/uio0/name", O_RDONLY);
-ret = read(fd, buf, sizeof(buf));
-close(fd);
-buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
-
-/* we only want uio devices whose name is a format we expect */
-if (strncmp(buf, "tcm-user", 8))
- exit(-1);
-
-/* Further checking for subtype also needed here */
-
-fd = open(/sys/class/uio/%s/maps/map0/size, O_RDONLY);
-ret = read(fd, buf, sizeof(buf));
-close(fd);
-str_buf[ret-1] = '\0'; /* null-terminate and chop off the \n */
-
-map_len = strtoull(buf, NULL, 0);
-
-dev_fd = open("/dev/uio0", O_RDWR);
-map = mmap(NULL, map_len, PROT_READ|PROT_WRITE, MAP_SHARED, dev_fd, 0);
-
-
-b) Waiting for events on the device(s)
-
-while (1) {
- char buf[4];
-
- int ret = read(dev_fd, buf, 4); /* will block */
-
- handle_device_events(dev_fd, map);
-}
-
-
-c) Managing the command ring
-
-#include <linux/target_core_user.h>
-
-int handle_device_events(int fd, void *map)
-{
- struct tcmu_mailbox *mb = map;
- struct tcmu_cmd_entry *ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
- int did_some_work = 0;
-
- /* Process events from cmd ring until we catch up with cmd_head */
- while (ent != (void *)mb + mb->cmdr_off + mb->cmd_head) {
-
- if (tcmu_hdr_get_op(ent->hdr.len_op) == TCMU_OP_CMD) {
- uint8_t *cdb = (void *)mb + ent->req.cdb_off;
- bool success = true;
-
- /* Handle command here. */
- printf("SCSI opcode: 0x%x\n", cdb[0]);
-
- /* Set response fields */
- if (success)
- ent->rsp.scsi_status = SCSI_NO_SENSE;
- else {
- /* Also fill in rsp->sense_buffer here */
- ent->rsp.scsi_status = SCSI_CHECK_CONDITION;
- }
- }
- else if (tcmu_hdr_get_op(ent->hdr.len_op) != TCMU_OP_PAD) {
- /* Tell the kernel we didn't handle unknown opcodes */
- ent->hdr.uflags |= TCMU_UFLAG_UNKNOWN_OP;
- }
- else {
- /* Do nothing for PAD entries except update cmd_tail */
- }
-
- /* update cmd_tail */
- mb->cmd_tail = (mb->cmd_tail + tcmu_hdr_get_len(&ent->hdr)) % mb->cmdr_size;
- ent = (void *) mb + mb->cmdr_off + mb->cmd_tail;
- did_some_work = 1;
- }
-
- /* Notify the kernel that work has been finished */
- if (did_some_work) {
- uint32_t buf = 0;
-
- write(fd, &buf, 4);
- }
-
- return 0;
-}
-
-
-A final note
-------------
-
-Please be careful to return codes as defined by the SCSI
-specifications. These are different than some values defined in the
-scsi/scsi.h include file. For example, CHECK CONDITION's status code
-is 2, not 1.