diff options
author | Paul E. McKenney <paulmck@linux.vnet.ibm.com> | 2015-07-12 01:24:45 +0200 |
---|---|---|
committer | Paul E. McKenney <paulmck@linux.vnet.ibm.com> | 2015-07-17 23:59:06 +0200 |
commit | cdacbe1f91264687af956e810278030f2ab5a3d0 (patch) | |
tree | 3a4e912e5c319329bc2a1e68b99ee041f0777150 /kernel/rcu/tree.c | |
parent | rcu: Rename RCU_GP_DONE_FQS to RCU_GP_DOING_FQS (diff) | |
download | linux-cdacbe1f91264687af956e810278030f2ab5a3d0.tar.xz linux-cdacbe1f91264687af956e810278030f2ab5a3d0.zip |
rcu: Add fastpath bypassing funnel locking
In the common case, there will be only one expedited grace period in
the system at a given time, in which case it is not helpful to use
funnel locking. This commit therefore adds a fastpath that bypasses
funnel locking when the root ->exp_funnel_mutex is not held.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Diffstat (limited to 'kernel/rcu/tree.c')
-rw-r--r-- | kernel/rcu/tree.c | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f66f6e7730bc..3af0dee2d045 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3356,6 +3356,22 @@ static struct rcu_node *exp_funnel_lock(struct rcu_state *rsp, unsigned long s) struct rcu_node *rnp1 = NULL; /* + * First try directly acquiring the root lock in order to reduce + * latency in the common case where expedited grace periods are + * rare. We check mutex_is_locked() to avoid pathological levels of + * memory contention on ->exp_funnel_mutex in the heavy-load case. + */ + rnp0 = rcu_get_root(rsp); + if (!mutex_is_locked(&rnp0->exp_funnel_mutex)) { + if (mutex_trylock(&rnp0->exp_funnel_mutex)) { + if (sync_exp_work_done(rsp, rnp0, NULL, + &rsp->expedited_workdone0, s)) + return NULL; + return rnp0; + } + } + + /* * Each pass through the following loop works its way * up the rcu_node tree, returning if others have done the * work or otherwise falls through holding the root rnp's |