d/numam-spdk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

scheduler_dynamic: scale up core load when moving thread

Browse Source 
Before this patch the idle time of a core was increased
by the amount of busy time of thread that was moved out.
No assumption was made as to how the remaining threads,
would behave during next scheduling period.

This approach is fine, as over multiple scheduling periods
we'd arrive at a point where threads could do no more work
or all cores would be busy.

Yet this requires multiple scheduling periods to sort out
the threads.
Later in the series core_load will be used to determine,
when to start moving threads out of the core. So changing
this assumption will allow for faster responses to thread load,
at cost of sometimes spreading threads too much briefly.

With this patch, we are assuming that threads remaining
on the core will do proportionally the same amount of work
during next scheduling period.

See an example illustrating the change:

Before moving Thread1
Thread1	Busy 80		Idle 20		Load 80%
Thread2	Busy 60		Idle 40		Load 60%
Core	Busy 140	Idle 60		Load 70%

After moving Thread1 out (original code)
Core	Busy 140-80=60	Idle 60+80=140	Load 30%

After moving Thread1 out (this patch)
Core	Busy 140-80=60	Idle 60-20=40	Load 60%

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1f347983449b2fde476dab360c4df689965ca3ea
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8279
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
This commit is contained in:
Tomasz Zawadzki 2021-06-09 06:19:07 -04:00 committed by Ben Walker
parent 97c5373fc7
commit d2ed0f45e7
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
lib/event/scheduler_dynamic.c
View File

@ -109,6 +109,7 @@ _move_thread(struct spdk_lw_thread *lw_thread, uint32_t dst_core)
struct core_stats *dst = &g_cores[dst_core];
struct core_stats *src = &g_cores[lw_thread->lcore];
uint64_t busy_tsc = lw_thread->current_stats.busy_tsc;
uint64_t idle_tsc = lw_thread->current_stats.idle_tsc;
if (src == dst) {
/* Don't modify stats if thread is already on that core. */
@ -119,8 +120,10 @@ _move_thread(struct spdk_lw_thread *lw_thread, uint32_t dst_core)
dst->idle -= spdk_min(dst->idle, busy_tsc);
dst->thread_count++;
/* Decrease busy/idle from core as if thread was not present on it.
* Core load will reflect the sum of all other threads on it. */
src->busy -= spdk_min(src->busy, busy_tsc);
src->idle += spdk_min(UINT64_MAX - src->busy, busy_tsc);
src->idle -= spdk_min(src->idle, idle_tsc);
assert(src->thread_count > 0);
src->thread_count--;