diff options
| author | Ming Lei <ming.lei@redhat.com> | 2021-12-03 14:15:32 +0100 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2021-12-03 22:51:29 +0100 |
| commit | 704b914f15fb7daaf517e3acc4bed472b50ca19e (patch) | |
| tree | 5fffc1140fef4ddceeaae58e44fbb65333c33861 /block/blk.h | |
| parent | blk-mq: remove hctx_lock and hctx_unlock (diff) | |
| download | linux-704b914f15fb7daaf517e3acc4bed472b50ca19e.tar.xz linux-704b914f15fb7daaf517e3acc4bed472b50ca19e.zip | |
blk-mq: move srcu from blk_mq_hw_ctx to request_queue
In case of BLK_MQ_F_BLOCKING, per-hctx srcu is used to protect dispatch
critical area. However, this srcu instance stays at the end of hctx, and
it often takes standalone cacheline, often cold.
Inside srcu_read_lock() and srcu_read_unlock(), WRITE is always done on
the indirect percpu variable which is allocated from heap instead of
being embedded, srcu->srcu_idx is read only in srcu_read_lock(). It
doesn't matter if srcu structure stays in hctx or request queue.
So switch to per-request-queue srcu for protecting dispatch, and this
way simplifies quiesce a lot, not mention quiesce is always done on the
request queue wide.
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/20211203131534.3668411-3-ming.lei@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'block/blk.h')
| -rw-r--r-- | block/blk.h | 10 |
1 files changed, 9 insertions, 1 deletions
diff --git a/block/blk.h b/block/blk.h index 24d8b333bb03..7ccb7c7d86b3 100644 --- a/block/blk.h +++ b/block/blk.h @@ -27,6 +27,7 @@ struct blk_flush_queue { }; extern struct kmem_cache *blk_requestq_cachep; +extern struct kmem_cache *blk_requestq_srcu_cachep; extern struct kobj_type blk_queue_ktype; extern struct ida blk_queue_ida; @@ -424,7 +425,14 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); -struct request_queue *blk_alloc_queue(int node_id); +static inline struct kmem_cache *blk_get_queue_kmem_cache(bool srcu) +{ + if (srcu) + return blk_requestq_srcu_cachep; + return blk_requestq_cachep; +} +struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu); + int disk_scan_partitions(struct gendisk *disk, fmode_t mode); int disk_alloc_events(struct gendisk *disk); |