summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorLong Li <leo.lilong@huawei.com>2024-04-16 08:16:28 +0200
committerAndrew Morton <akpm@linux-foundation.org>2024-05-06 02:53:38 +0200
commitba591801a3df861b3b327f6122b9de4ef213aae6 (patch)
tree1bb2326efbe66f1eaca201e42a3f1849157de644
parentmm/ksm: remove page_mapcount() usage in stable_tree_search() (diff)
downloadlinux-ba591801a3df861b3b327f6122b9de4ef213aae6.tar.xz
linux-ba591801a3df861b3b327f6122b9de4ef213aae6.zip
xarray: inline xas_descend to improve performance
The commit 63b1898fffcd ("XArray: Disallow sibling entries of nodes") modified the xas_descend function in such a way that it was no longer being compiled as an inline function, because it increased the size of xas_descend(), and the compiler no longer optimizes it as inline. This had a negative impact on performance, xas_descend is called frequently to traverse downwards in the xarray tree, making it a hot function. Inlining xas_descend has been shown to significantly improve performance by approximately 4.95% in the iozone write test. Machine: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz #iozone i 0 -i 1 -s 64g -r 16m -f /test/tmptest Before this patch: kB reclen write rewrite read reread 67108864 16384 2230080 3637689 6315197 5496027 After this patch: kB reclen write rewrite read reread 67108864 16384 2340360 3666175 6272401 5460782 Percentage change: 4.95% 0.78% -0.68% -0.64% This patch introduces inlining to the xas_descend function. While this change increases the size of lib/xarray.o, the performance gains in critical workloads make this an acceptable trade-off. Size comparison before and after patch: .text .data .bss file 0x3502 0 0 lib/xarray.o.before 0x3602 0 0 lib/xarray.o.after Link: https://lkml.kernel.org/r/20240416061628.3768901-1-leo.lilong@huawei.com Signed-off-by: Long Li <leo.lilong@huawei.com> Cc: Hou Tao <houtao1@huawei.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: yangerkun <yangerkun@huawei.com> Cc: Zhang Yi <yi.zhang@huawei.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
-rw-r--r--lib/xarray.c3
1 files changed, 2 insertions, 1 deletions
diff --git a/lib/xarray.c b/lib/xarray.c
index da79128ad754..1c87d871cacf 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -200,7 +200,8 @@ static void *xas_start(struct xa_state *xas)
return entry;
}
-static void *xas_descend(struct xa_state *xas, struct xa_node *node)
+static __always_inline void *xas_descend(struct xa_state *xas,
+ struct xa_node *node)
{
unsigned int offset = get_offset(xas->xa_index, node);
void *entry = xa_entry(xas->xa, node, offset);