diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2020-11-22 16:35:46 +0100 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2020-12-08 01:12:49 +0100 |
| commit | 661d4f55a79483aee4970a76e3bd9d4cdc74ac79 (patch) | |
| tree | 2d239a9e27ebd6e016d4c80cca528883ad4bcc5d /lib/sbitmap.c | |
| parent | sbitmap: optimise sbitmap_deferred_clear() (diff) | |
| download | linux-661d4f55a79483aee4970a76e3bd9d4cdc74ac79.tar.xz linux-661d4f55a79483aee4970a76e3bd9d4cdc74ac79.zip | |
sbitmap: remove swap_lock
map->swap_lock protects map->cleared from concurrent modification,
however sbitmap_deferred_clear() is already atomically drains it, so
it's guaranteed to not loose bits on concurrent
sbitmap_deferred_clear().
A one threaded tag heavy test on top of nullbk showed ~1.5% t-put
increase, and 3% -> 1% cycle reduction of sbitmap_get() according to perf.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'lib/sbitmap.c')
| -rw-r--r-- | lib/sbitmap.c | 14 |
1 files changed, 3 insertions, 11 deletions
diff --git a/lib/sbitmap.c b/lib/sbitmap.c index c1c8a4e69325..4fd877048ba8 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -15,13 +15,9 @@ static inline bool sbitmap_deferred_clear(struct sbitmap_word *map) { unsigned long mask, val; - bool ret = false; - unsigned long flags; - spin_lock_irqsave(&map->swap_lock, flags); - - if (!map->cleared) - goto out_unlock; + if (!READ_ONCE(map->cleared)) + return false; /* * First get a stable cleared mask, setting the old mask to 0. @@ -35,10 +31,7 @@ static inline bool sbitmap_deferred_clear(struct sbitmap_word *map) val = map->word; } while (cmpxchg(&map->word, val, val & ~mask) != val); - ret = true; -out_unlock: - spin_unlock_irqrestore(&map->swap_lock, flags); - return ret; + return true; } int sbitmap_init_node(struct sbitmap *sb, unsigned int depth, int shift, @@ -80,7 +73,6 @@ int sbitmap_init_node(struct sbitmap *sb, unsigned int depth, int shift, for (i = 0; i < sb->map_nr; i++) { sb->map[i].depth = min(depth, bits_per_word); depth -= sb->map[i].depth; - spin_lock_init(&sb->map[i].swap_lock); } return 0; } |