1. align to 128 bytes to avoid possible waste from the preceeding section
2. sort entries by alignment SORT_BY_ALIGNMENT, plugging the holes (most
entries are one byte in size, but they got interleaved with bigger ones)
Interestingly I was looking for a feature of the sort earlier and failed
to find it. It turns out the script was already utilizing sorting in other
places, so shame on me.
Thanks for Travis Geiselbrecht for pointing me at the feature.
This aligns the section itself compared to other sections, does not change
internal alignment of fields stored inside. This may or may not come later.
The motivation is partially combating adverse effects of the adjacent cache
line prefetcher. Without the annotation part of read_mostly section was on
the line of fire.
correctly for the data contained on each memory page.
There are several components to this change:
* Add a variable to indicate the start of the R/W portion of the
initial memory.
* Stop detecting NX bit support for each AP. Instead, use the value
from the BSP and, if supported, activate the feature on the other
APs just before loading the correct page table. (Functionally, we
already assume that the BSP and all APs had the same support or
lack of support for the NX bit.)
* Set the RW and NX bits correctly for the kernel text, data, and
BSS (subject to some caveats below).
* Ensure DDB can write to memory when necessary (such as to set a
breakpoint).
* Ensure GDB can write to memory when necessary (such as to set a
breakpoint). For this purpose, add new MD functions gdb_begin_write()
and gdb_end_write() which the GDB support code can call before and
after writing to memory.
This change is not comprehensive:
* It doesn't do anything to protect modules.
* It doesn't do anything for kernel memory allocated after the kernel
starts running.
* In order to avoid excessive memory inefficiency, it may let multiple
types of data share a 2M page, and assigns the most permissions
needed for data on that page.
Reviewed by: jhb, kib
Discussed with: emaste
MFC after: 2 weeks
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D14282
While __read_mostly groups variables together, their placement is not
specified. In particular 2 frequently used variables can end up in
different lines.
This annotation is only expected to be used for variables read all the time,
e.g. on each syscall entry.
MFC after: 1 week
We should never end up executing the inter-function padding, so we
are better off faulting than silently carrying on to whatever function
happens to be next.
Note that LLD will soon do this by default (although it currently pads
with zeros).
Reviewed by: dim, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10047
The intended use is to annotate frequently used globals which either rarely
change (and thus can be grouped in the same cacheline) or are an atomic counter
(which means it may benefit from being the only variable in the cacheline).
Linker script support is provided only for amd64. Architectures without it risk
having other variables put in, i.e. as if they were not annotated. This is
harmless from correctness point of view.
Reviewed by: bde (previous version)
MFC after: 1 month
MAXPAGESIZE is not well defined by the GNU ld documentation.
Different linkers, and different versions of the same linker, use
different MAXPAGESIZE values. Current versions of GNU gold and LLVM's
lld use 4K. When set to 4K the kernel panics at boot due to an issue
with x86bios.
Here we want the kernel physaddr to be the amd64 superpage size, so use
that value (2MB) explicitly. With this change GNU gold and LLVM lld can
link a working amd64 kernel.
PR: 214718 (x86bios)
Differential Revision: https://reviews.freebsd.org/D8610
The linker script CONSTRUCTORS keyword is only meaningful "when linking
object file formats which do not support arbitrary sections, such as
ECOFF and XCOFF"[1] and is ignored for other object file formats.
LLVM's lld does not yet accept (and ignore) CONSTRUCTORS, so just remove
CONSTRUCTORS from the linker scripts as it has no effect.
[1] https://sourceware.org/binutils/docs/ld/Output-Section-Keywords.html
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7343
The current linker script generates program headers with VMA == LMA:
Entry point 0xffffffff802e7000
There are 6 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0xffffffff80200040 0xffffffff80200040
0x0000000000000150 0x0000000000000150 R E 8
INTERP 0x0000000000000190 0xffffffff80200190 0xffffffff80200190
0x000000000000000d 0x000000000000000d R 1
[Requesting program interpreter: /red/herring]
LOAD 0x0000000000000000 0xffffffff80200000 0xffffffff80200000
0x00000000010559b0 0x00000000010559b0 R E 200000
LOAD 0x0000000001056000 0xffffffff81456000 0xffffffff81456000
0x0000000000132638 0x000000000052ecf8 RW 200000
DYNAMIC 0x0000000001056000 0xffffffff81456000 0xffffffff81456000
0x00000000000000d0 0x00000000000000d0 RW 8
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RWE 8
This is fine for the FreeBSD loader, because it completely ignores p_paddr
and instead uses p_vaddr with a hardcoded offset. Other loaders however
acknowledge p_paddr (like the Xen ELF loader), in which case they will try
to load the kernel at the wrong place. Fix this by adding an AT keyword to
the first section specifying the physical address, other sections will
follow suit, so it ends up looking like:
Entry point 0xffffffff802e7000
There are 6 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0xffffffff80200040 0x0000000000200040
0x0000000000000150 0x0000000000000150 R E 8
INTERP 0x0000000000000190 0xffffffff80200190 0x0000000000200190
0x000000000000000d 0x000000000000000d R 1
[Requesting program interpreter: /red/herring]
LOAD 0x0000000000000000 0xffffffff80200000 0x0000000000200000
0x00000000010559b0 0x00000000010559b0 R E 200000
LOAD 0x0000000001056000 0xffffffff81456000 0x0000000001456000
0x0000000000132638 0x000000000052ecf8 RW 200000
DYNAMIC 0x0000000001056000 0xffffffff81456000 0x0000000001456000
0x00000000000000d0 0x00000000000000d0 RW 8
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RWE 8
Tested on bare metal using the native FreeBSD loader and grub2 from TRUEOS.
Sponsored by: Citrix Systems R&D
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D2783
instead of 0x100000. As a side effect, an amd64 kernel now loads at
physical address 0x200000 instead of 0x100000. This is probably for the
best because it avoids the use of a 2MB page mapping for the first 1MB of
the kernel that also spans the fixed MTRRs. However, getmemsize() still
thinks that the kernel loads at 0x100000, and so the physical memory between
0x100000 and 0x200000 is lost. Fix this problem by replacing the hard-wired
constant in getmemsize() by a symbol "kernphys" that is defined by the
linker script.
In collaboration with: kib
ELF_MAXPAGESIZE for amd64 from 0x00100000 to 0x00200000. This caused
the kernel to be incorrectly linked, using the existing linker script,
resulting in a virtual address of 0xffffffff80000000 for the LOAD
program header.
The boot loader will load such a kernel at a real address of 0x00000000,
which either causes protection faults in btx, crashes the machine, or
(in case of a VMware guest) even makes it power down. :)
Fix this by partially synchronizing the amd64 linker script with
binutils own updated version, in particular replacing a hardcoded
value of 0x00100000 by CONSTANT(MAXPAGESIZE).
bootblocks in order to boot the kernel after this! Also note that this
change breaks BSDI BSD/OS compatibility.
Also increased default NKPT to 17 so that FreeBSD can boot on machines
with >=2GB of RAM. Booting on machines with exactly 4GB requires other
patches, not included.
the in-kernel linker to access the _DYNAMIC data for doing loadable elf
modules. The alpha kernel is already done this way, I've borrowed some of
the hacks from there.
This is primarily aimed at the 3-stage boot process which is intended to
be able to do pre-loading of kernel modules.
Note that the entry point isn't 0xf0100000 any more, it'll be a little
further on - but this value is stored in the headers. I don't think this
will be a problem, but I'm sure somebody will tell me if it is. :-)
I'm not sure if btxboot is going to like this, it doesn't do proper ELF
header checking and assumes that there are exactly two program header
entries and that they are both PT_LOAD entries - a bad assumption.
