Commit Graph

23 Commits

Author SHA1 Message Date
Konstantin Belousov
523d94dbea x86: switch vdso TSC timecounter to RDTSCP on AMD Zen CPUs
Reported by:	many
Tested by:	gallatin, mikael, rhurlin
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2021-01-21 14:55:31 +02:00
Konstantin Belousov
cd240c9cf1 x86 vdso gettc: Add RDTSCP support
Detect and use RDTSCP if available, instead of fence+RDTSC.  For AMD Zens+,
use LFENCE+RDTSC instead of RDTSCP (or MFENCE;RDTSC previously).

Reviewed by:	gallatin, markj
Tested by:	pho
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D27986
2021-01-10 04:42:34 +02:00
Konstantin Belousov
5bf4bafd13 x86 vdso gettc: eliminate duplicated code in ifunc selectors.
Create array of rdtsc selectors and provide helper that calculate the
index into the selectors array.

Reviewed by:	gallatin, markj
MFC after:      1 week
Sponsored by:   The FreeBSD Foundation
Differential Revision:  https://reviews.freebsd.org/D27986
2021-01-10 04:42:34 +02:00
Konstantin Belousov
de898cb960 x86 vdso gettc: reorganize ifunctions.
Instead of providing ifuncs for each kind of fence, define ifuncs
that combine fence and invocation of RDTSC.  This refactoring makes
introduction of RDTSCP use possible.

Reviewed by:	gallatin, markj
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D27986
2021-01-10 04:42:34 +02:00
Gordon Bergling
f6d234d870 libc: Fix most issues reported by mandoc
- varios "new sentence, new line" warnings
- varios "sections out of conventional order" warnings
- varios "unusual Xr order" warnings
- varios "missing section argument" warnings
- varios "no blank before trailing delimiter" warnings
- varios "normalizing date format" warnings

MFC after:	1 month
2020-12-19 14:54:28 +00:00
Konstantin Belousov
7c5a46a1bc Remove resolver_qual from DEFINE_IFUNC/DEFINE_UIFUNC macros.
In all practical situations, the resolver visibility is static.

Requested by:	markj
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
Approved by:	so (emaste)
Differential revision:	https://reviews.freebsd.org/D20281
2019-05-16 22:20:54 +00:00
Konstantin Belousov
03d89f7a04 pkru(3) man page.
Reviewed by:	alc, markj, jilles
With more input from:	pho
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
Differential revision:	https://reviews.freebsd.org/D19211
2019-02-20 10:03:03 +00:00
Konstantin Belousov
a2d95495ee Add usermode helpers for for Intel userspace protection keys feature.
Reviewed by:	markj
Tested by:	pho
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
Differential revision:	https://reviews.freebsd.org/D18893
2019-02-20 09:56:23 +00:00
Konstantin Belousov
071bca67ee Unify i386 and amd64 getcontextx.c, and use ifuncs while there.
In particular, use ifuncs for __getcontextx_size(), also calculate the
size of the extended save area in resolver.  Same for __fillcontextx2().

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2019-02-14 14:02:33 +00:00
Konstantin Belousov
b5d72efb7a x86 __vdso_gettc(): use machine/cpufunc.h function for CPUID.
Based on the discussion with:	jkim
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2019-02-14 13:59:00 +00:00
Konstantin Belousov
fd90aeaabd Add comment noting that the strange spelling of GenuineIntel is for reason.
Requested by:	rpokala
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2019-02-07 03:58:29 +00:00
Konstantin Belousov
5925a98339 Use ifunc to select the barrier instruction for RDTSC.
This optimizes out runtime switch and removes yet another cpuid from
libc.

Note that this is the first use of ifunc in i386 libc, so
ifunc-capable toolchain is required for building runnable userspace on
i386, same as on amd64.

Discussed with:	emaste
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2019-02-07 02:56:10 +00:00
Konstantin Belousov
b1cbddaf5d Fix a regression in r321608.
On i386 with CPUID but without SSE2, set lfence_works to LMB_NONE
instead of looping.

Reported and tested by:	Andre Albsmeier <andre@fbsd.e4m.org>
Sponsored by:	The FreeBSD Foundation
MFC after:	3 days
2017-08-13 14:42:23 +00:00
Konstantin Belousov
d00ed7642f Simplify flow control.
Also add explicit comment why libc cannot simply rely on open(2)
failure in capability mode.

Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2017-07-28 12:22:32 +00:00
Konstantin Belousov
9a3083660d Use MFENCE to serialize RDTSC on non-Intel CPUs.
Kernel already used the stronger barrier instruction for AMDs, correct
the userspace fast gettimeofday() implementation as well.

Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
Differential revision:	https://reviews.freebsd.org/D11728
2017-07-27 08:37:07 +00:00
Konstantin Belousov
4f969e10d1 Fix indent.
Sponsored by:	The FreeBSD Foundation
MFC after:	3 days
2017-07-25 09:48:33 +00:00
Mariusz Zaborski
b62c1f2b18 Remove unneeded variable initialization from r314319.
Pointed out by:	kib
2017-02-26 22:15:39 +00:00
Mariusz Zaborski
ba1b663656 Don't try to open devices in the gettc() function which will always
fail in the Capability mode. Instead silently fallback to the syscall
method, which is done for example in the gettimeofday(2) function.

Reviewed by:	kib
2017-02-26 22:07:26 +00:00
Enji Cooper
0195177b26 Only conditionally add in hyperv support if we're building amd64
This unbreaks the build because the assembly is written for x64.

MFC after:	3 weeks
X-MFC with:	r312418
Pointyhat to:	ngie
Reported by:	Jenkins (i386 job)
Sponsored by:	Dell EMC Isilon
2017-01-19 18:07:24 +00:00
Enji Cooper
ab9bd41a5a Conditionalize hyperv support in gettimeofday(2) based on MK_HYPERV
The effect at runtime is negligible as the hyperv timer isn't available
except when hyperv is loaded.

This is a prerequisite for conditionalizing the header build/install out
of the build

MFC after:	3 weeks
Reviewed by:	sephe
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D9242
2017-01-19 17:03:45 +00:00
Konstantin Belousov
4f64c5b3cf __vdso_gettc(): be extra careful with /dev/hpet mappings, never unmap
the mapping which might be accessed by other threads.

If a pointer to the /dev/hpet register page mapping was stored into
the hpet_dev_map, other threads might access the page at any time.
Never unmap it, instead, keep track of mappings for all hpet units in
smal array.  Store pointer to the newly mapped registers page using
CAS, to detect parallel mappings.

It appeared relatively easy to demonstrate the problem by arranging
two threads which perform gettimeofday(2) concurently, first time in
the process address space, when HPET is used for timecounter.

PR:	215715
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2017-01-04 16:10:52 +00:00
Sepherosa Ziehau
fff5be0be3 hyperv: Implement userspace gettimeofday(2) with Hyper-V reference TSC
This 6 times gettimeofday performance, as measured by
tools/tools/syscall_timing

Reviewed by:	kib
MFC after:	1 week
Sponsored by:	Microsoft
Differential Revision:	https://reviews.freebsd.org/D8789
2016-12-19 07:40:45 +00:00
Konstantin Belousov
1680854946 Implement userspace gettimeofday(2) with HPET timecounter.
Right now, userspace (fast) gettimeofday(2) on x86 only works for
RDTSC.  For older machines, like Core2, where RDTSC is not C2/C3
invariant, and which fall to HPET hardware, this means that the call
has both the penalty of the syscall and of the uncached hw behind the
QPI or PCIe connection to the sought bridge.  Nothing can me done
against the access latency, but the syscall overhead can be removed.
System already provides mappable /dev/hpetX devices, which gives
straight access to the HPET registers page.

Add yet another algorithm to the x86 'vdso' timehands. Libc is updated
to handle both RDTSC and HPET.  For HPET, the index of the hpet device
to mmap is passed from kernel to userspace, index might be changed and
libc invalidates its mapping as needed.

Remove cpu_fill_vdso_timehands() KPI, instead require that
timecounters which can be used from userspace, to provide
tc_fill_vdso_timehands{,32}() methods.  Merge i386 and amd64
libc/<arch>/sys/__vdso_gettc.c into one source file in the new
libc/x86/sys location.  __vdso_gettc() internal interface is changed
to move timecounter algorithm detection into the MD code.

Measurements show that RDTSC even with the syscall overhead is faster
than userspace HPET access.  But still, userspace HPET is three-four
times faster than syscall HPET on several Core2 and SandyBridge
machines.

Tested by:	Howard Su <howard0su@gmail.com>
Sponsored by:	The FreeBSD Foundation
MFC after:	1 month
Differential revision:	https://reviews.freebsd.org/D7473
2016-08-17 09:52:09 +00:00