Commit Graph

69 Commits

Author SHA1 Message Date
Konstantin Belousov
0e2af3b59a jemalloc: Define SWAP_RESERVE_* if not provided by the system headers
instead ifdef-ing out the return statement.

Reported by:	Mark Millard <marklmi@yahoo.com>
Fixes:	87384c51e0
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
2022-09-17 01:35:35 +03:00
Konstantin Belousov
0ae364adcd jemalloc: use auxv ELF_BSDF_VMNOOVERCOMMIT instead of sysctl("vm.overcommit")
Reviewed by:	brooks, imp (previous version)
Discussed with:	markj
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
Differential revision:	https://reviews.freebsd.org/D36540
2022-09-16 23:23:52 +03:00
Konstantin Belousov
87384c51e0 jemalloc: use symbolic definitions for bits in vm_overcommit
Reviewed by:	brooks, imp (previous version)
Discussed with:	markj
Sponsored by:	The FreeBSD Foundation
MFC after:	2 weeks
Differential revision:	https://reviews.freebsd.org/D36540
2022-09-16 23:23:19 +03:00
Piotr Kubaj
884ba43116 powerpc: enable initial-exec TLS
Summary:
Use initial-exec, like other architectures.

While here, switch MACHINE_ARCH in lib/libc/Makefile to LIBC_ARCH and consistently use powerpc.

Subscribers: imp, #contributor_reviews_base

Differential Revision: https://reviews.freebsd.org/D34315
Reviewed by:	luporl
MFC after:	2 weeks
2022-02-22 19:34:28 +01:00
Marcin Wojtas
b62ae61446 Fix abort in jemalloc extent coalescing.
Fix error in extent_try_coalesce_impl(), which could cause abort
to happen when trying to coalesce extents backwards. The error could
happen because of how extent_before_get() function works. This function
gets address of previous extent, by subtracting page size from current
extent address. If current extent is located at PAGE_SIZE offset, this
address resolved to 0x0000. An assertion in rtree_leaf_elm_lookup
then caused the running program to abort.

This problem was discovered when trying to build world on 32-bit
machines with ASLR and PIE enabled. The problem was encountered
on armv7 and i386 machines, but most likely other 32-bit
architectures are affected as well.

While this patch fixes one problem with buildworld on 32-bit platforms
with ASLR, the build still fails, however it happens much later
and due to lack of memory.

The change is aligned with accepted fix in the upstream Jemalloc
repository (https://github.com/jemalloc/jemalloc/pull/1973).
As it doesn't apply on top of Jemalloc tree, its updated version
was eventually merged: https://github.com/jemalloc/jemalloc/pull/2003

PR: 249937
Submitted by: Dawid Gorecki <dgr@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D27025
2020-12-18 10:09:21 +00:00
Dimitry Andric
7d4374f65f Turn MALLOC_PRODUCTION into a regular src.conf(5) option
For historical reasons, defining MALLOC_PRODUCTION in /etc/make.conf has
been used to turn off potentially expensive debug checks and statistics
gathering in the implementation of malloc(3).

It seems more consistent to turn this into a regular src.conf(5) option,
e.g. WITH_MALLOC_PRODUCTION / WITHOUT_MALLOC_PRODUCTION. This can then
be toggled similar to any other source build option, and turned on or
off by default for e.g. stable branches.

Reviewed by:	imp, #manpages
MFC after:	1 week
Differential Revision: https://reviews.freebsd.org/D26337
2020-09-05 23:30:17 +00:00
Brooks Davis
ea69bf7f5d Set LG_VADDR to 48 on RISC-V.
The Sv48 PTE format is the largest currently defined address space for
RISC-V. It makes no sense to define a larger size and doing so (at
least for 64-bits) forces rtrees down a slow path.

Reviewed by:	vangyzen, jhb, mhorne
Obtained from:	CheriBSD
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24658
2020-05-04 17:16:30 +00:00
Eric van Gyzen
c5ad81420c Update jemalloc to version 5.2.1
Revert r354606 to restore r354605.

Apply one line from jemalloc commit d01b425e5d1e1 in hash_x86_128()
to fix the build with gcc, which only allows a fallthrough attribute
to appear before a case or default label.

Submitted by:	jasone in r354605
Discussed with:	jasone
Reviewed by:	bdrewery
MFC after:	never, due to gcc 4.2.1
Relnotes:	yes
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D24522
2020-04-23 23:57:43 +00:00
Edward Tomasz Napierala
568f01f6c3 Make jemalloc(3) default to retain:true on 64-bit platforms,
like it already does on Linux and OSX.  This results in significantly
fewer calls to mmap(2).  This should result in a small reduction
in system CPU time and improved superpage usage.

Reviewed by:	markj
Tested by:	markj
MFC after:	2 weeks
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D23874
2020-03-31 13:48:06 +00:00
Ryan Libby
79457a8ad1 jemalloc: pick from upstream: Fix GCC-9.1 warning with macro GET_ARG_NUMERIC
Pick 2d6d099fed05b1509e81e54458516528bfbbf38d from upstream jemalloc:

    Fix GCC-9.1 warning with macro GET_ARG_NUMERIC

    GCC-9.1 reports following error when trying to compile file
    src/malloc_io.c and with CFLAGS='-Werror' :

    src/malloc_io.c: In function ‘malloc_vsnprintf’:
    src/malloc_io.c:369:2: error: case label value exceeds maximum value for type [-Werror]
      369 |  case '?' | 0x80:      \
          |  ^~~~
    src/malloc_io.c:581:5: note: in expansion of macro ‘GET_ARG_NUMERIC’
      581 |     GET_ARG_NUMERIC(val, 'p');
          |     ^~~~~~~~~~~~~~~
    ...
    <snip>
    cc1: all warnings being treated as errors
    make: *** [Makefile:388: src/malloc_io.sym.o] Error 1

    The warning is reported as by default the type 'char' is 'signed char'
    and or-ing 0x80 will turn the case label char negative which will be
    beyond the printable ascii range (0 - 127).

    The patch fixes this by explicitly casting the 'len' variable as
    unsigned char' inside the 'switch' statement so that value of
    expression " '?' | 0x80 " falls within the legal values of the
    variable 'len'.

Discussed with:	jasone (maintainer)
Sponsored by:	Dell EMC Isilon
2019-12-21 02:44:38 +00:00
Jason Evans
f2cb29075f Revert r354605: Update jemalloc to version 5.2.1.
Compilation fails for non-llvm-based platforms.
2019-11-11 05:06:49 +00:00
Jason Evans
e1c167d019 Update jemalloc to version 5.2.1. 2019-11-11 03:27:14 +00:00
Konstantin Belousov
0edc114ac0 realloc(x, 0) should not return NULL.
See http://www.open-std.org/jtc1/sc22/wg14/www/docs/summary.htm#dr_400.
Upstream jemalloc issue is opened by emaste at
https://github.com/jemalloc/jemalloc/issues/1629.

Reviewed by:	emaste
PR:	240456
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
DIfferential revision:	https://reviews.freebsd.org/D21632
2019-09-17 18:36:29 +00:00
Edward Tomasz Napierala
d5aef6d6ca Pick 57553c3b1a5592dc4c03f3c6831d9b794e523865 from upstream:
Avoid touching all pages in extent_recycle for debug build.

    We may have a large number of pages with *zero set (since they are populated on
    demand).  Only check the first page to avoid paging in all of them.

This makes it easy to compare performance with and without 'retain:true'.

Discussed with:	jasone
Obtained from:	Qi Wang <interwq at gwu dot edu>
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2018-11-14 13:06:48 +00:00
Edward Tomasz Napierala
086165ecb5 Pick 50b473c8839f5408df179bdf6f2b3fd2cf5c3b2f from upstream:
Set commit properly for FreeBSD w/ overcommit.

    When overcommit is enabled, commit needs to be set when doing mmap().  The
    regression was introduced in f80c97e.

This fixes 'retain:true'.

Discussed with:	jasone
Obtained from:	Qi Wang <interwq at gwu dot edu>
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2018-11-06 12:05:46 +00:00
Edward Tomasz Napierala
0c885e274a Pick f80c97e477d1b3fe7778c65d9439d673738b4131 from upstream:
Rework the way jemalloc uses mmap(2) on FreeBSD.

    This makes it directly use MAP_EXCL and MAP_ALIGNED() instead
    of weird workarounds involving mapping at random places and then
    unmapping parts of them.

Discussed with:	jasone
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2018-10-23 14:11:35 +00:00
Edward Tomasz Napierala
bff19560ee Pick 676cdd66792ccb629a978837ea2a066d5db342cc from upstream:
Disable runtime detection of lazy purging support on FreeBSD.

    The check doesn't seem to serve any purpose here, and this shaves
    off three syscalls on binary startup.

Discussed by:	jasone
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2018-10-23 13:54:54 +00:00
Jason Evans
0ef50b4ec8 Update jemalloc to version 5.1.0. 2018-05-11 00:32:31 +00:00
Marius Strobl
c2a2eeff5c Account for the fact that jemalloc 5.0.0 dropped STATIC_PAGE_SHIFT
in favor for using LG_PAGE directly and, thus, for the fact that
host and target don't necessarily use pages of the same sizes.

Approved by:	jasone
2018-01-31 21:56:23 +00:00
Michal Meloun
d65a63e3aa Relax too restrictive assert.
The problem has been reported to upstream and similar change will
be included in next jemalloc release.

Submitted by:	David Goldblatt <davidtgoldblatt@gmail.com>
MFC after: 	2 weeks
2017-12-10 09:17:21 +00:00
Michal Meloun
e7dad90fe5 Revert r326740. I committed wrong diff. 2017-12-10 09:15:37 +00:00
Michal Meloun
3ee2fe0199 Relax too restrictive assert.
The problem has been reported to upstream and similar change will
be included in next jemalloc release.

Submitted by:	David Goldblatt <davidtgoldblatt@gmail.com>
MFC after: 	2 weeks
2017-12-10 08:55:48 +00:00
Dimitry Andric
cc718496a9 After jemalloc was updated to version 5.0.0 in r319971, i386 executables
linked with AddressSanitizer (even those linked on earlier versions of
FreeBSD, or with external versions of clang) started failing with errors
similar to:

  ==14688==AddressSanitizer CHECK failed:
  /usr/src/contrib/compiler-rt/lib/asan/asan_poisoning.cc:36
  "((AddrIsAlignedByGranularity(addr))) != (0)" (0x0, 0x0)

This is because AddressSanitizer expects all the TLS data in the program
to be aligned to at least 8 bytes.

Before the jemalloc 5.0.0 update, all the TLS data in the i386 version
of libc.so added up to 80 bytes (a multiple of 8), but 5.0.0 made this
grow to 2404 bytes (not a multiple of 8).  This is due to added caching
data in jemalloc's internal struct tsd_s.

To fix AddressSanitizer, ensure this struct is aligned to at least 16
bytes, which can be done unconditionally for all architectures.  (An
earlier version of the fix aligned the struct to 8 bytes, but only for
ILP32 architectures.  This was deemed unnecessarily complicated.)

PR:		221337
X-MFC-With:	r319971
2017-10-23 21:31:04 +00:00
Ryan Libby
20100e740e Pick 'Remove external linkage for spin_adaptive' from upstream jemalloc
Apply the changes from upstream jemalloc 048c6679.  This is actually not
quite a cherry pick due to makefile difference and because FreeBSD does
not carry the msvc project files which were also modified in that
commit.

Approved by:	jasone (maintainer), markj (mentor)
Sponsored by:	Dell EMC Isilon
2017-08-09 22:58:42 +00:00
Ruslan Bukin
ca20f8ec29 o Replace __riscv__ with __riscv
o Replace __riscv64 with (__riscv && __riscv_xlen == 64)

This is required to support new GCC 7.1 compiler.
This is compatible with current GCC 6.1 compiler.

RISC-V is extensible ISA and the idea here is to have built-in define
per each extension, so together with __riscv we will have some subset
of these as well (depending on -march string passed to compiler):

__riscv_compressed
__riscv_atomic
__riscv_mul
__riscv_div
__riscv_muldiv
__riscv_fdiv
__riscv_fsqrt
__riscv_float_abi_soft
__riscv_float_abi_single
__riscv_float_abi_double
__riscv_cmodel_medlow
__riscv_cmodel_medany
__riscv_cmodel_pic
__riscv_xlen

Reviewed by:	ngie
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D11901
2017-08-07 14:09:57 +00:00
Jason Evans
8b2f5aaf29 Update jemalloc to 5.0.1. 2017-07-03 23:27:57 +00:00
Jason Evans
1a78ec8941 Default the abort_conf malloc option to false.
This avoids troublesome backward compatibility issues.
2017-06-30 17:45:51 +00:00
Jason Evans
b7eaed250f Update jemalloc to 5.0.0. 2017-06-15 07:15:05 +00:00
Jason Evans
8244f2aa7c Update jemalloc to 4.5.0. 2017-03-02 01:14:48 +00:00
Jason Evans
7fa7f12ff8 Update jemalloc to 4.4.0. 2016-12-04 21:13:26 +00:00
Jason Evans
bde951447f Update jemalloc to 4.3.1. 2016-11-09 18:42:30 +00:00
Jason Evans
62b2691e48 Update jemalloc to 4.2.1. 2016-06-09 06:10:20 +00:00
Jason Evans
fbb1d85eb3 Work around invalid gcc warning (explicit cast apparently lost). 2016-05-13 21:18:10 +00:00
Jason Evans
1f0a49e863 Update jemalloc to 4.2.0. 2016-05-13 04:03:20 +00:00
Andrew Turner
4b6f23dfaf Stop using sbrk(2) with malloc. This helps reduce the number of places
within the tree where we call this legacy interface.

Reviewed by:	jasone
Obtained from:	brooks
Sponsored by:	ABT Systems Ltd
2016-04-28 12:24:58 +00:00
Jason Evans
cbc3697dda Restore support for decay time of -1 (no decay). 2016-03-03 01:43:36 +00:00
Jason Evans
cb2ababcfd Add a cast to prevent a compiler warning. 2016-03-03 01:30:28 +00:00
Ruslan Bukin
ac8f9de31a Restore configuration for RISC-V. 2016-03-01 17:43:06 +00:00
Jason Evans
df0d881d94 Update jemalloc to 4.1.0.
Add missing Symbol.map entry for __aligned_alloc.

Add weak-->strong symbol binding for
{malloc_stats_print,mallctl,mallctlnametomib,mallctlbymib} -->
{__malloc_stats_print,__mallctl,__mallctlnametomib,__mallctlbymib}.  These
bindings complete the set necessary to allow applications to replace all
malloc-related symbols.
2016-02-29 19:10:32 +00:00
Ruslan Bukin
47f9f6eb3a Add configuration for RISC-V ISA.
Reviewed by:	emaste
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D5020
2016-01-22 16:37:26 +00:00
Jason Evans
ba4f5cc0a6 Update jemalloc to version 4.0.4. 2015-10-24 23:18:05 +00:00
Jason Evans
536b3538e0 Update jemalloc to 4.0.2. 2015-09-22 03:02:18 +00:00
Jason Evans
c13244b92e Fix minor malloc regressions.
- Use _Bool rather than bool to resolve missing type errors in malloc_np.h.
- Fix malloc manual page #include documentation.
- Add *allocm manual pages to obsolete files.

Submitted by:	jbeich
2015-08-19 00:06:46 +00:00
Jason Evans
337776f858 Define CPU_SPINWAIT as cpu_spinwait().
Submitted by:	cem
2015-08-18 20:42:08 +00:00
Jason Evans
d8e39d2d4f Re-add LG_SIZEOF_PTR definition for __aarch64__.
This definition was erroneously removed during the 4.0.0 import.
2015-08-18 09:09:27 +00:00
Jason Evans
a4cee496bc Fix build failure due to missing CPU_SPINWAIT definition. 2015-08-18 08:10:46 +00:00
Jason Evans
8d13fecf5f Add jemalloc support for 8 KiB, 16 KiB, and 64 KiB pages. 2015-08-18 06:28:37 +00:00
Jason Evans
d0e79aa362 Update jemalloc to version 4.0.0. 2015-08-18 00:21:25 +00:00
Ed Maste
552f44af8b Add aarch64 (arm64) #define for jemalloc
Sponsored by:	The FreeBSD Foundation
2015-03-12 08:52:00 +00:00
Konstantin Belousov
8495e8b1e9 Fix known issues which blow up the process after dlopen("libthr.so")
(or loading a dso linked to libthr.so into process which was not
linked against threading library).

- Remove libthr interposers of the libc functions, including
  __error(). Instead, functions calls are indirected through the
  interposing table, similar to how pthread stubs in libc are already
  done.  Libc by default points either to syscall trampolines or to
  existing libc implementations.  On libthr load, libthr rewrites the
  pointers to the cancellable implementations already in libthr.  The
  interposition table is separate from pthreads stubs indirection
  table to not pull pthreads stubs into static binaries.

- Postpone the malloc(3) internal mutexes initialization until libthr
  is loaded.  This avoids recursion between calloc(3) and static
  pthread_mutex_t initialization.

- Reinstall signal handlers with wrapper on libthr load.  The
  _rtld_is_dlopened(3) is used to avoid useless calls to sigaction(2)
  when libthr is statically referenced from the main binary.

In the process, fix openat(2), swapcontext(2) and setcontext(2)
interposing.  The libc symbols were exported at different versions
than libthr interposers.  Export both libc and libthr versions from
libc now, with default set to the higher version from libthr.

Remove unused and disconnected swapcontext(3) userspace implementation
from libc/gen.

No objections from:	deischen
Tested by:	pho, antoine (exp-run) (previous versions)
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2015-01-03 18:38:46 +00:00