d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

MFV r357608: Limit memory usage in xz(1) instead of in tuklib.

Browse Source 
Apply upstream 353970510895f6a80adfe60cf71b70a95adfa8bc to limit memory
usage on 32-bit binary to 4020 MiB.

Submitted by:	Lasse Collin <lasse.collin at tukaani.org>
Reviewed by:	kib, bcr
Differential Revision:	https://reviews.freebsd.org/D23474
This commit is contained in:
Xin LI 2020-02-06 07:47:28 +00:00
commit f99e4a2d11
Notes: svn2git 2020-12-20 02:59:44 +00:00 
svn path=/head/; revision=357609
3 changed files with 51 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
contrib/xz/src/common/tuklib_physmem.c
View File

@ -45,7 +45,6 @@
# include <sys/systemcfg.h>
#elif defined(TUKLIB_PHYSMEM_SYSCONF)
# include <limits.h>
# include <unistd.h>
#elif defined(TUKLIB_PHYSMEM_SYSCTL)
@ -146,16 +145,13 @@ tuklib_physmem(void)
#elif defined(TUKLIB_PHYSMEM_SYSCONF)
const long pagesize = sysconf(_SC_PAGESIZE);
const long pages = sysconf(_SC_PHYS_PAGES);
if (pagesize != -1 && pages != -1) {
if (pagesize != -1 && pages != -1)
// According to docs, pagesize * pages can overflow.
// Simple case is 32-bit box with 4 GiB or more RAM,
// which may report exactly 4 GiB of RAM, and "long"
// being 32-bit will overflow. Casting to uint64_t
// hopefully avoids overflows in the near future.
ret = (uint64_t)pagesize * (uint64_t)pages;
if (ret > SIZE_T_MAX)
ret = SIZE_T_MAX;
}
#elif defined(TUKLIB_PHYSMEM_SYSCTL)
int name[2] = {

32
contrib/xz/src/xz/hardware.c
View File

@ -68,9 +68,39 @@ hardware_memlimit_set(uint64_t new_memlimit,
new_memlimit = (uint32_t)new_memlimit * total_ram / 100;
}
if (set_compress)
if (set_compress) {
memlimit_compress = new_memlimit;
#if SIZE_MAX == UINT32_MAX
// FIXME?
//
// When running a 32-bit xz on a system with a lot of RAM and
// using a percentage-based memory limit, the result can be
// bigger than the 32-bit address space. Limiting the limit
// below SIZE_MAX for compression (not decompression) makes
// xz lower the compression settings (or number of threads)
// to a level that *might* work. In practice it has worked
// when using a 64-bit kernel that gives full 4 GiB address
// space to 32-bit programs. In other situations this might
// still be too high, like 32-bit kernels that may give much
// less than 4 GiB to a single application.
//
// So this is an ugly hack but I will keep it here while
// it does more good than bad.
//
// Use a value less than SIZE_MAX so that there's some room
// for the xz program and so on. Don't use 4000 MiB because
// it could look like someone mixed up base-2 and base-10.
const uint64_t limit_max = UINT64_C(4020) << 20;
// UINT64_MAX is a special case for the string "max" so
// that has to be handled specially.
if (memlimit_compress != UINT64_MAX
&& memlimit_compress > limit_max)
memlimit_compress = limit_max;
#endif
}
if (set_decompress)
memlimit_decompress = new_memlimit;

19
contrib/xz/src/xz/xz.1
View File

@ -1005,6 +1005,25 @@ instead of
until the details have been decided.
.RE
.IP ""
For 32-bit
.BR xz
there is a special case: if the
.I limit
would be over
.BR "4020\ MiB" ,
the
.I limit
is set to
.BR "4020\ MiB" .
(The values
.B 0
and
.B max
aren't affected by this.
A similar feature doesn't exist for decompression.)
This can be helpful when a 32-bit executable has access
to 4\ GiB address space while hopefully doing no harm in other situations.
.IP ""
See also the section
.BR "Memory usage" .
.TP