Issue: spdk_top tracked pollers by the poller name string and the
thread_id they are running on. This shows incorrect stats when
multiple pollers exist on the same thread with the same name.
Solution: Added a unique poller id for each poller on a thread and
to allow spdk_top to track pollers by thread_id and poller_id.
Signed-off-by: Michael Piszczek <mpiszczek@ddn.com>
Change-Id: I1879e2afc9a929d1df9e8e35510f0092c5443bdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5868
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
There could be an outstanding for_each_reactor operation
in progress when the application shuts down. If we don't
wait for that operation to complete, we'll get a memory
leak for its context.
So when stopping the reactors, before setting the
state to EXITING, do one final for_each_reactor to
ensure any outstanding for_each_reactor operations are
flushed. Once this final for_each_reactor operation
is complete, we can set the state to EXITING safely.
Also use a global mutex and flag to ensure no more
for_each_reactor operations can start while the final
for_each_reactor is in progress. If a new operation
is requested, we'll just ignore since the reactors are
shutting down anyways.
Fixes issue #2206.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6d86eb3b0c7855e98481ab4164af82e76c009618
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9928
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
For debugging purposes, take a name for identifying fds added to a group.
Signed-off-by: John Levon <john.levon@nutanix.com>
Change-Id: If1654e56e19f7fa964446ef1b9e71debf74979d1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9731
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Currently, there is no way to prevent spdk_app_start() from calling
app_setup_signal_handlers() and setting SPDK's signal handlers.
We'd like to use our own set of signal handlers, therefore this
patch adds a flag to the spdk_app_opts struct that disables this
behaviour.
Signed-off-by: Andreas Economides <andreas.economides@nutanix.com>
Change-Id: I61d7cd66527d819fd5f687d5cc8a03be4fe10a6a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9380
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch moves schedueler and governor related API from
the internal event.h to public scheduler.h.
With this it is possible to create subsystem responsible
for handling the schedulers.
Three schedulers and a governor were moved to scheduler modules
from event framework.
This will allow next patch to add JSON RPC configuration
to the whole subsystem.
Along with easier addition of other schedulers.
Removed debug logs from gscheduler, as they serve little purpose.
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I98ca1ea4fb281beb71941656444267842a8875b7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6995
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Make sure to cancel the scheduling during:
1) gather_metrics error (already present)
2) setting the scheduler to NULL (new)
3) shutdown of the application (new)
In all of the above scheduler cannot proceed to
the balance() function.
Prior to this patch lack of setting
g_scheduling_in_progress to false would block
_start_subsystem_fini() from proceeding.
Resuling in application never shutting down.
thread_infos are allocated during gather_metrics,
and freed on threads_reschedule.
The added cleanup is used during 1) and 2).
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5ab77c85aaa9f94bd4e1a1660b55ab2f2c47bbdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9326
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
There is no need to track if governor was enabled during
initialization of the scheduler dynamic.
Instead _spdk_governor_get() can be used to determine that.
While here set the default g_governor to NULL.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ibdfde3d2714dd0b6d629f19f0d246b4fc3c9d214
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9325
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch cleans up the header file, structures and
parameters of governor API. While documenting the
functionality.
- made governor name const
- renamed _spdk_governor_list_add() to _spdk_governor_register()
This is preparation to making this API public.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ie394109c839dead0e7ade946f95be8105b00e674
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8843
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This patch cleans up the header file, structures and
parameters of scheduler API. While documenting the
functionality.
- made scheduler name const
- removed typedefs for schedueler callbacks
- balance() now accepts uint32_t for array size instead of an int
- removed unused _spdk_lw_thread_set_core()
- renamed _spdk_scheduler_period_set() to _spdk_scheduler_set_period()
- renamed _spdk_scheduler_period_get() to _spdk_scheduler_get_period()
- renamed _spdk_scheduler_list_add() to _spdk_scheduler_register()
This is preparation to making this API public.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia7b6b6a5eafb052ac275db6c04113a8ad442383f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8842
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Registering multiple governors would fail due to them having
the same name. Only saved by the fact that right now,
there is only one governor registered in this fashion.
Fix it by adding name of the governor structure passed
to the function name.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic7a206da2c8f5dc1e72e41629bccf989c030f182
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8792
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
By default g_scheduler is now NULL. It can be set
either by event framework or RPC.
To keep RPC consistent, the 'static' scheduler was kept.
All access to the g_scheduler is done via _spdk_scheduler_get().
Access to its members is done to balance, get name for RPC and
through _spdk_scheduler_set(). All of them happen on scheduling
reactor and don't pose race condition when changing scheduler.
There is no need to delay init/deinit of scheduler via g_new_scheduler.
This variable was removed and all that happens in _spdk_scheduler_set().
To unset and deinitialize current scheduler,
_spdk_scheduler_set(NULL) has to be called.
This results in moving scheduler deinitalization to that
call too.
Every spdk_scheduler callback is now mandatory.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1fed766989de9bcaeb7e82b490c1275f3572d77d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8811
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
By default g_governor is now NULL. It can be set
either by event framework or schedulers directly.
Dynamic_scheduler and gscheduler specifically want
to use the dpdk_governor, so their initialization
now sets it explicitly.
To unset and deinitialize current governor,
_spdk_governor_set(NULL) has to be called.
This results in moving governor deinitalization to that
call too.
The "default" governor has been removed.
Every spdk_governor callback is now mandatory.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ibf76bd28bfbb159416026996fa217bb3325a3d31
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8810
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
There is no explicit need for the spdk governors initialization
to occur on per core basis.
This implementation detail for dpdk_governor is now hidden
in the init/deinit calls. There is no recourse when failing
deinit for a certain core, so ignore the return code.
Changed return type for deinit in governor and scheduler to void.
While here modified the callbacks for scheduler to no
longer require passing currently selected governor as an argument.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I7f0b7a09aa7f5d12ae47fca25186faeedac31a95
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8791
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@gmail.com>
spdk_reactor_set_interrupt_mode() writes/reads from fds created during
reactor_interrupt_init(). Since spdk_fd_group_create() depends
on eventfd, this will not work for systems that do not have it.
reactor_interrupt_init() handled lack of support for eventfd correctly,
while spdk_reactor_set_interrupt_mode() did not check for it.
Reported-by: Nick Connolly <nick.connolly@mayadata.io>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5181d436636c55cca3a06b1947e944502a9204ce
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9131
Reviewed-by: Nick Connolly <nick.connolly@mayadata.io>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
In rte_power all that enabling/disabling turbo does is allows
for additional entry in frequency array for particular core.
Instead of exposing this API through spdk governor,
just make sure that dpdk_governor enables turbo by default.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I994b326a57c01889bccea26635753c56637259d2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8789
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Those calls went unused, in favor or much more useful
up/down/min/max variants.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I432896196a1a6edfc6799c8658df49567f73d457
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8788
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The spdk_governor_capabilities added lots of capabilities
which went unused, suposedly to mark which callbacks
a governor had implemented.
This made little sense, since capabilities are per core and
not implmenting this APIs made little sense.
With this patch spdk_governor_capabilities is brought in
line with rte_power_core_capabilities.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I85296fce2999cb41957162b63ee13d86a0be919f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8787
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Remove _spdk_scheduler_disable() to avoid confusion as there is
no spdk_scheduler_enable function. Since spdk_scheduler_disable
sets scheduler period to 0, use spdk_scheduler_period_set(0) instead.
Signed-off-by: Krzysztof Karas <krzysztof.karas@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4f1390a635f80e8b92775aa4be2e37f5b95467f8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7448
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
The size of the core_info->threads will always be equal
to reactor thread_count, there is no need to count it
separately.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Icfa84606bd29d7766738eb2053362a20d78c23af
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8733
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Replaced multiple functions calls to _reactors_scheduler_update_core_mode(),
with a for loop.
Since changing reactor to interrupt mode is rare operation, most of the
time we ended up with unnecessarily long callstack.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1c7858653be9e2256943c1da5a27001be41682b6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8714
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
There is only one g_scheduling_reactor (main core), the is_scheduling
flag for it is used to block starting new gather_metrics before
previous one is finished.
Meanwhile is_scheduling flag on other reactors was used to block
destroying lw_threads while scheduling happens. It was only needed
because scheduler interacted with the same lw_thread pointers as
each reactor. Previous patch removed this dependency, instead
spdk_thread ids is used. If an spdk_thread is destroyed,
while scheduling _threads_reschedule_thread() handles it.
It is no longer required to block destruction of lw_threads
based on this flag.
Instead of using the main core reactor flag, a g_scheduling_in_progress
is introduced.
Removed _spdk_get_scheduling_reactor() and instead shared the value
of g_scheduling_in_progress between reactor.c and app.c.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ica57326a552477add522174cc3e96b3bab918350
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8732
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Removing dependency on schedulers to directly modify
lw_thread field structures will help making schedulers
truly plugable.
Instead of using lw_thread, new structure is created
that holds copy of stats and refer to the thread by
spdk_thread id.
As an added benefit of not changing lw_thread directly,
we won't run into issue of balancing function changing it
while other reactor removes and frees it.
In the future an API will be added for scheduler to call
in order to move the thread directly. Rather than for
event framework to rely on modified core_info/thread_info
structure.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I8f85bb8dc080fd13b78b07ee9ef8e8be7051659b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8411
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Limit of 50% to mark thread as active or idle
didn't allow for multiple active threads to be placed
on single core.
Lowering the limit to 20% will allow that and force
more threads to be actively balanced.
Removing the limit was considered, but that would
cause too much thread moves when a thread with load
in single digits increased briefly. Either
due to actually doing any operation or placement
of other threads on the same core.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5f8c9ff15461feb71a2d82853cfe048e412ba921
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8289
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
In this case, user could specify the core number like:
-m [0,1,10] besides the core mask like -m 0xF
Change-Id: I48621c5a84e5436deae07101591d0ef85b1e129e
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8746
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
For cases where cpumask for a thread was not set,
all bits were turned on for whole length of cpuset structure.
This resulted in JSON RPC reponses with way too long cpumask
for what is useful.
Now the response is limited to the applications core mask,
as that makes sense so long as number of cores cannot change.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib5cf271d3b219ba679f1abe498516796693a87dd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8288
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
The round-robin logic is no longer necessary to spread
the threads around the cores. Starting from core other
than first is even counter-productive to bunching up
threads.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5fcee2bacc2d0b4af26336caf381ed954814d731
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8085
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Before this patch _find_optimal_core() returned
1) any core that could fit the thread
2) if current core was over the limit, the least busy core
3) current core if no better candidate was found
Combined with _get_next_target_core() round-robining
the first core to consider, resulted in threads being
unnecessarily spread over the cores.
This patch only places threads on lower lcore id,
or when current core is over limit then any core that can fit it.
Next patch will remove round-robin logic to always start with
lowest lcore id.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I54e373d3ca02a5633607d22978305baa1142f8bd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8112
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Before this patch the idle time of a core was increased
by the amount of busy time of thread that was moved out.
No assumption was made as to how the remaining threads,
would behave during next scheduling period.
This approach is fine, as over multiple scheduling periods
we'd arrive at a point where threads could do no more work
or all cores would be busy.
Yet this requires multiple scheduling periods to sort out
the threads.
Later in the series core_load will be used to determine,
when to start moving threads out of the core. So changing
this assumption will allow for faster responses to thread load,
at cost of sometimes spreading threads too much briefly.
With this patch, we are assuming that threads remaining
on the core will do proportionally the same amount of work
during next scheduling period.
See an example illustrating the change:
Before moving Thread1
Thread1 Busy 80 Idle 20 Load 80%
Thread2 Busy 60 Idle 40 Load 60%
Core Busy 140 Idle 60 Load 70%
After moving Thread1 out (original code)
Core Busy 140-80=60 Idle 60+80=140 Load 30%
After moving Thread1 out (this patch)
Core Busy 140-80=60 Idle 60-20=40 Load 60%
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1f347983449b2fde476dab360c4df689965ca3ea
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8279
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
In cases when all cores are already doing too much work
to fit a thread, active threads should still be balanced
over all cores.
When current core is overloaded, place the thread
on another that is less busy.
The core limit is set to 95% to catch only ones that are
fully busy.
Decreasing that value would make spreading out the threads
move aggressive.
Changed thread load in one of the unit tests to reflect the
95% limit.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3b3bc5f7fbd22725441fa811d61446950000cc46
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8113
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Previously core load was only considered for main lcore.
Other cores were used based on cpumask only.
Once an active thread was placed on core it remained there
until idle. If _get_next_target_core() looped around,
the core might receive another active thread.
This patch makes the core load matter for placement of any thread.
As of this patch if no core can fit a thread it will remain there.
Later in the series least busy core will be used to balance
threads when every core is already busy.
Modified the functional test that depended on always selecting
consecutive core, even if 'current' one fit the bill.
Later in the series the round robin logic for core selection
is removed all together.
Fixed typo in test while here.
Note: _can_core_fit_thread() intentionally does not check
core->interrupt_mode and uses tsc. That flag is only updated
at the end of balancing right now. Meanwhile tsc is updated
one first thread moved to the core, so it is no longer
considered in interrupt mode.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I95f58c94e3f5ae8a468723d1dd6e53b0e417dcc3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8069
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Idle threads are always moved to main core, there are no
other considations. Doing it as separate first pass,
allows to have the core stats be up to date for second
pass for active threads.
Core load stats will be used later in the series to determine
optimal target core for an active thread.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6a9bc11b86e954e461f7badebf3a6e4d1718f63c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8067
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This will be needed when doing multiple passes over
all threads. See next patch.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I4e9c749d69314fc268cbcb9334862392100b651e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8066
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When picking a path to go down with a thread,
conditions unnecessarily piled up.
Instead do it either of two ways:
- move idle threads to main core
- find best core for active threads and move them there
There is no need to worry about cpumask of the thread,
since _find_optimal_core() will always return a core
within the cpumask.
If the found core is the same one as the current,
_move_thread() won't perform any action.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I0f4782766c15c86b5db0c970cfc9547058845b2a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8065
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Refactors logic for finding the optimal core for a thread
to single function.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ifc2b09acb6f698640ce9602fec4f567eb32b79fa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6732
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Refactor all thread moves and core stats updates to single function.
At this time in series only tsc of main core was modified and
only idle tsc of main core was used. Main core would be either
the destination core or the source core. In both cases, the idle
time for main core had to be updated.
This patch generalizes this logic to always move the execution
time from source core to destination core.
As a byproduct cores besides main core have the stats updated,
which will be useful later in the series. Once core load will
be the deciding factor for choosing a core.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I57564e8b2632f919869d74e8f10b01fb3dda3be9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6658
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Move the definition of structure spdk_io_channel into
lib/thread/thread_internal.h, so we don't have to update SO_VER for
other libraries in future when we need to change the internal details on
the structure.
Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: I3d2ca7a8737972e0b33ce92e464da42c48f89dec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8189
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
At this time only main lcore frequency is changed,
depending on its load either up or down.
Exception is when at least a single busy thread is present
on non-g_main_lcore. Then the main lcore frequency is set
to the maximum possible.
This patch moves when that is determined, from 'moving'
logic to one that sets reactors to interrupt mode.
If at least one thread is present on non-g_main_lcore,
it has to be busy. Otherwise it would be placed on main lcore.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2900598afe53fb609e1f06a60d5245f74511e1c3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8050
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
This field was only used to keep track of number of threads
that will be present on a core after scheduler moves.
It was used only internally within scheduler_dynamic.
Event framework has no need to keep such field in core_info.
Instead added field in cores_stats internal to scheduler_dynamic.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3ce74d4a25eac81e58da8705a1c4553730fc1e57
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8049
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Added core_stats structure that will hold stats modified
during balancing.
Further patches will modify the values in this structure,
to for example judge how much execution time a core
has left.
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib8e611e36642c4543b5cb43bc2695c613d38f0fc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6657
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
This patch expands spdk_scheduler_core_info with two new
fields that will contain core stats only from last scheduling
period.
This will make sure that schedulers do not have to keep track
and calculate this value on their own.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I3aa7dfa6a60c1d14d95a0e684e84c2e83f0a4496
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8048
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
(a5ad0f80) lib/event: update reactor tsc_last going poll mode
Patch above updated the tsc_last at the very end of changing
interrupt mode of the reactor.
The flow for turning from interrupt mode to poll mode is
first to send an event to the target lcore, then to iterate
over all reactors updating notify_cpuset on each.
Previous patch updated the tsc_last after notify_cpuset was
updated, meanwhile the threads could already been put on it.
This patch moves it immidietly to the point of changing
the in_interrupt state.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6aea252016f4706369b8b597b765593bc6edca3b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8111
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Renamed core_busy_tsc and core_idle_tsc to better
describe that they contain particular core stats for
its whole lifetime.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I6f16b2b0a162aad8fbaf18f549fc50a2372b920b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8047
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
There is no need to keep new_lcore field.
lcore value is enough to determine the target core.
Meanwhile _threads_reschedule() can see if the target
core matches the one from core_info.
Removed _spdk_lw_thread_set_core() since it did not
serve much purpose.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I82c7cfebf1107b4a55b2af9b891052084a788907
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8046
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
lw_thread->lcore was set during gather_metrics,
rather than just after the thread reschedule.
This patch just moves it to the right place.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I0477830902f68102e4e4f0ffc9359bd004a8ad42
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7961
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
So far the schedulers had to calculate the diff of
current_stats - last_stats on their own to get tsc
from last scheduling period.
Renamed the current_stats to total_stats, but kept the meaning
as stats describing tsc for lifetime of a thread.
Instead change the meaning of the last_stats to describe
the tsc of only last scheduling period and change its name
to current_stats.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I1a165ff7c1afe659b432c3127a351a96878d1f3d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7843
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Rather than to rely on schedulers to access and modify
last_stats values over multiple scheduling periods, move that
operation to event framework.
Providing this to the schedulers in generic manner is better
than enforcing that each scheduler has to keep track of this
data on their own.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Icaf3b4af80d86fafaddf328fd230db9743d21ab5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7971
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
_reactor_schedule_thread() zeroes out the lw_thread on move.
To properly calculate thread stats since the move,
save them right after reschedule.
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I44cc3b5907adda35b3117c2dd7268dc813d59853
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
