freebsd-dev/sys/kern
Mark Johnston 2c10be9e06 arm64: Handle translation faults for thread structures
The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translation fault while accessing curthread, and data_abort() accesses
the thread again before pmap_fault() can translate the address and
return.

Normally this isn't a problem because we have a hack to ensure that
slabs used by the thread zone are always accessed via the direct map,
where promotions and demotions are rare.  However, this hack doesn't
work properly with UMA_MD_SMALL_ALLOC disabled, as is the case with
KASAN configured (since our KASAN implementation does not shadow the
direct map and so tries to force the use of the kernel map wherever
possible).

Fix the problem by modifying data_abort() to handle translation faults
in the kernel map without dereferencing "td", i.e., curthread, and
without enabling interrupts.  pmap_klookup() has special handling for
translation faults which makes it safe to call in this context.  Then,
revert the aforementioned hack.

Reviewed by:	kevans, alc, kib, andrew
MFC after:	1 month
Sponsored by:	Juniper Networks, Inc.
Sponsored by:	Klara, Inc.
Differential Revision:	https://reviews.freebsd.org/D37231
2022-11-02 13:46:25 -04:00
..
bus_if.m subr_bus: restore bus_null_rescan() 2022-06-23 16:07:00 -03:00
clock_if.m
cpufreq_if.m
device_if.m
firmw.S Use a template assembly file for firmware object files. 2020-12-17 20:31:17 +00:00
genassym.sh genassym.sh: Fix two minor issues found by shellcheck 2021-07-28 13:49:16 -06:00
genoffset.c
genoffset.sh genoffset.sh: Use 10 X's instead of 5 for pick mkdtemp implementations 2021-09-07 10:08:51 -06:00
imgact_aout.c Retire sv_transtrap 2022-05-20 14:54:03 +03:00
imgact_binmisc.c
imgact_elf32.c
imgact_elf64.c
imgact_elf.c Add AT_USRSTACK{BASE, LIM} AT vectors, and ELF_BSDF_VMNOOVERCOMMIT flag 2022-09-16 23:23:26 +03:00
imgact_shell.c
init_main.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
init_sysent.c sysent: regen after 52a1d90c8b, posix_fadvise in capmode 2022-04-14 15:17:36 -04:00
kern_acct.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
kern_alq.c vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd) 2022-03-24 10:20:51 +00:00
kern_boottrace.c boottrace: a simple boot and shutdown-time tracing facility 2022-02-21 20:15:45 -04:00
kern_clock.c time(3): Fix spelling. 2022-10-23 18:42:11 +02:00
kern_clocksource.c eventtimer: Simplify KTR traces 2022-07-11 15:58:43 -04:00
kern_condvar.c Use sleepq_signal(SLEEPQ_DROP) in cv_signal(). 2021-07-09 20:57:58 -04:00
kern_conf.c Retire clone_drain_lock 2022-08-20 09:44:05 +00:00
kern_cons.c Adjust function definitions in kern_cons.c to avoid clang 15 warnings 2022-07-26 19:59:56 +02:00
kern_context.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
kern_cpu.c cpufreq: Remove unused devclass argument to DRIVER_MODULE. 2022-05-06 15:46:58 -07:00
kern_cpuset.c powerpc: cpuset: add local functions for copyin/copyout 2022-10-03 12:03:09 -03:00
kern_ctf.c vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd) 2022-03-24 10:20:51 +00:00
kern_descrip.c file: Move code to share fdtol structs into kern_descrip.c 2022-08-04 09:39:25 -04:00
kern_devctl.c devd: move all devd notification logic to a separate file. 2022-08-10 18:56:01 +00:00
kern_dtrace.c Adjust function definitions in kern_dtrace.c to avoid clang 15 warnings 2022-07-26 19:59:57 +02:00
kern_dump.c kerneldump: remove physical from dump routines 2022-05-13 10:43:19 -03:00
kern_environment.c kern: Support duplicate variables in early kenv 2022-10-17 23:02:20 -07:00
kern_et.c
kern_event.c kqueue: retire knlist_init_rw_reader() 2022-08-20 21:17:39 -08:00
kern_exec.c vfs: always retain path buffer after lookup 2022-09-17 09:10:38 +00:00
kern_exit.c jail: add process linkage 2022-09-05 11:54:47 +00:00
kern_fail.c
kern_ffclock.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
kern_fork.c jail: add process linkage 2022-09-05 11:54:47 +00:00
kern_hhook.c
kern_idle.c
kern_intr.c kern_intr: Check for NULL event in intr_destroy() 2022-10-15 15:51:44 -03:00
kern_jail.c Import the WireGuard driver from zx2c4.com. 2022-10-28 13:36:12 -07:00
kern_kcov.c kcov: use __func__ in KASSERT instead of old function name 2022-03-07 10:47:27 -05:00
kern_khelp.c
kern_kthread.c umtx: Split umtx.h on two counterparts. 2021-07-29 12:41:29 +03:00
kern_ktr.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
kern_ktrace.c ktrace: change AST handler to require AST flag set 2022-08-02 21:11:10 +03:00
kern_linker.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
kern_lock.c lockmgr: fix lock profiling of face adaptive spinning 2021-09-18 10:16:58 +00:00
kern_lockf.c lockf: elide vnode interlock in the common case in lf_purgelocks 2022-09-14 23:04:22 +00:00
kern_lockstat.c
kern_loginclass.c
kern_malloc.c kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers. 2022-09-22 15:09:19 -07:00
kern_mbuf.c debugnet: Fix parameter order in the calls to m_get() 2022-09-21 06:55:20 -04:00
kern_mib.c jail: Consistently handle the pr_allow bitmask 2020-12-26 20:25:02 -08:00
kern_module.c modules: increase MAXMODNAME and provide backward compat 2021-12-09 18:09:53 +00:00
kern_mtxpool.c
kern_mutex.c callout(9): Allow spin locks use with callout_init_mtx(). 2021-09-02 21:16:46 -04:00
kern_ntptime.c kern_ntptime.c: Remove ntp_init() 2022-02-07 14:16:16 -07:00
kern_osd.c osd: Fix racy assertions 2021-09-09 10:11:02 -04:00
kern_physio.c SPDX: Not BSD-4-Clause 2022-09-16 21:49:16 -06:00
kern_pmc.c
kern_poll.c Adjust function definition in kern_poll.c to avoid clang 15 warning 2022-08-14 21:27:34 +02:00
kern_priv.c jail: Consistently handle the pr_allow bitmask 2020-12-26 20:25:02 -08:00
kern_proc.c AST: rework 2022-08-02 21:11:09 +03:00
kern_procctl.c Remove struct proc p_singlethr member 2022-08-20 20:34:30 +03:00
kern_prot.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
kern_racct.c AST: rework 2022-08-02 21:11:09 +03:00
kern_rangelock.c
kern_rctl.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
kern_resource.c Adjust function definitions in kern_resource.c to avoid clang 15 warnings 2022-07-26 19:59:57 +02:00
kern_rmlock.c rms: add rms_assert_rlock_ok 2022-08-23 19:15:48 +00:00
kern_rwlock.c lockprof: pass lock type as an argument instead of reading the spin flag 2021-05-23 17:55:27 +00:00
kern_sdt.c
kern_sema.c
kern_sendfile.c Add getsock 2022-09-10 19:47:47 +00:00
kern_sharedpage.c Rework how shared page related data is stored 2022-07-18 16:27:32 +02:00
kern_shutdown.c Add kern.reboot_wait_time sysctl 2022-07-18 17:23:25 -07:00
kern_sig.c signal: use proc_iterate to save on work 2022-09-05 11:54:47 +00:00
kern_switch.c Fix the build with SCHED_STATS after d3f96f6610 2022-09-26 20:20:46 -06:00
kern_sx.c lockprof: pass lock type as an argument instead of reading the spin flag 2021-05-23 17:55:27 +00:00
kern_synch.c AST: rework 2022-08-02 21:11:09 +03:00
kern_syscalls.c sys/kern: Use C99 fixed-width integer types. 2021-12-28 09:41:08 -08:00
kern_sysctl.c show_sysctl_all: reduce copying, please coverity 2022-10-01 12:20:04 -05:00
kern_tc.c set_cputicker: use a bool 2022-06-23 15:15:11 -03:00
kern_thr.c thread_create(): call cpu_copy_thread() after td_pflags is zeroed 2022-08-08 19:44:17 +03:00
kern_thread.c arm64: Handle translation faults for thread structures 2022-11-02 13:46:25 -04:00
kern_time.c time: Make realitexpire() local to kern_time.c 2022-07-13 09:57:28 -04:00
kern_timeout.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
kern_tslog.c tslog: Add CTLFLAG_SKIP to sysctls 2022-03-20 11:31:16 -07:00
kern_ubsan.c ubsan: Fix a typo in an error message 2021-09-25 11:47:24 +02:00
kern_umtx.c time(3): Align fast clock times to avoid firing multiple timers. 2022-10-03 17:53:17 +02:00
kern_uuid.c
kern_vnodedumper.c kerneldump: remove physical argument from d_dumper 2022-05-13 10:42:48 -03:00
kern_xxx.c
ksched.c ksched: correct return code for invalid priority 2022-10-17 15:12:13 -03:00
link_elf_obj.c vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd) 2022-03-24 10:20:51 +00:00
link_elf.c vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd) 2022-03-24 10:20:51 +00:00
linker_if.m kernel linker: do not read debug symbol tables for non-debug symbols 2021-12-08 23:32:29 +02:00
Make.tags.inc Bring the tags and links entries for amd64 up to date. 2021-09-27 20:04:51 -07:00
Makefile syscalls.master: switch to CAPENABLED flags 2021-09-01 21:58:16 +01:00
md4c.c sys/kern: Use C99 fixed-width integer types. 2021-12-28 09:41:08 -08:00
md5c.c sys/kern: Use C99 fixed-width integer types. 2021-12-28 09:41:08 -08:00
msi_if.m Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights 2021-08-08 10:42:24 -04:00
p1003_1b.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
pic_if.m
posix4_mib.c
sched_4bsd.c sched_4bsd: Fix a racy thread state modification 2022-09-23 20:09:06 -04:00
sched_ule.c AST: rework 2022-08-02 21:11:09 +03:00
serdev_if.m
stack_protector.c
subr_acl_nfs4.c
subr_acl_posix1e.c
subr_asan.c atomic: Intercept atomic_(load|store)_bool for kernel sanitizers 2022-10-29 11:10:58 -04:00
subr_atomic64.c sys/kern: Use C99 fixed-width integer types. 2021-12-28 09:41:08 -08:00
subr_autoconf.c Adjust function definition in subr_autoconf.c to avoid clang 15 warnings 2022-07-26 19:59:57 +02:00
subr_blist.c blist: Correct the node count computed in blist_create() 2021-07-13 17:47:27 -04:00
subr_boot.c
subr_bufring.c
subr_bus_dma.c bus_dma: Deduplicate locking helper functions. 2022-01-05 13:50:40 -08:00
subr_bus.c Fix a panic on boot introduced by 555a861d68 2022-11-01 13:44:39 -04:00
subr_busdma_bounce.c Add sched_ithread_prio to set the base priority of an interrupt thread. 2022-07-14 13:13:10 -07:00
subr_busdma_bufalloc.c kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers. 2022-09-22 15:09:19 -07:00
subr_capability.c Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights 2021-08-08 10:42:24 -04:00
subr_clock.c
subr_clockcalib.c clockcalib: Fix an overflow bug 2022-01-20 08:23:38 -05:00
subr_compressor.c Fix gzip compressed core dumps on big endian architectures 2022-02-10 09:34:37 -06:00
subr_counter.c
subr_coverage.c kcov: Integrate with KMSAN 2021-09-14 14:29:27 -04:00
subr_csan.c atomic: Intercept atomic_(load|store)_bool for kernel sanitizers 2022-10-29 11:10:58 -04:00
subr_devmap.c Remove pre-armv6 support from devmap 2022-10-05 09:56:17 +01:00
subr_devstat.c devstat: Remove DTrace io probes lacking a BIO reference 2022-08-29 13:22:36 -04:00
subr_disk.c
subr_dummy_vdso_tc.c
subr_early.c
subr_epoch.c stack_zero is not needed before stack_save 2022-03-25 20:10:38 -05:00
subr_eventhandler.c
subr_fattime.c
subr_filter.c kern: Fix two typos in source code comments 2022-04-02 14:15:27 +02:00
subr_firmware.c subr_firmware.c::unloadentry(): remote write-only variable 2021-10-21 21:40:46 +03:00
subr_gtaskqueue.c
subr_hash.c
subr_hints.c Optimize res_find(). 2021-08-08 21:54:49 -04:00
subr_intr.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
subr_kdb.c mac: kdb/ddb framework hooks 2022-07-18 22:06:13 +00:00
subr_kobj.c kobj: plug set-but-not-used vars 2021-12-14 14:52:25 +00:00
subr_lock.c locks: whack a failed experiment in form of restrict_starvation 2022-09-16 17:29:37 +00:00
subr_log.c Document kern.log_wakeups_per_second. 2021-08-04 11:50:34 -07:00
subr_mchain.c
subr_module.c sysctl debug.dump_modinfo should recognize font module 2021-01-08 09:24:49 +02:00
subr_msan.c atomic: Intercept atomic_(load|store)_bool for kernel sanitizers 2022-10-29 11:10:58 -04:00
subr_msgbuf.c msgbuf: Allow microsecond granularity timestamps 2022-05-07 09:32:22 -06:00
subr_param.c time(3): Optimize tvtohz() function. 2022-10-23 10:04:50 +02:00
subr_pcpu.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
subr_pctrie.c
subr_physmem.c physmem: Add physmem_excluded to query if a region is excluded 2022-10-25 09:32:49 -06:00
subr_pidctrl.c
subr_power.c
subr_prf.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
subr_prng.c
subr_prof.c AST: rework 2022-08-02 21:11:09 +03:00
subr_rangeset.c Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights 2021-08-08 10:42:24 -04:00
subr_rman.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
subr_rtc.c
subr_sbuf.c sbuf(9): Microoptimize sbuf_put_byte() 2021-10-05 14:47:38 -04:00
subr_scanf.c
subr_sfbuf.c
subr_sglist.c sglist: Add sglist_append_single_mbuf(). 2021-05-25 16:59:18 -07:00
subr_sleepqueue.c sleepq_set_timeout_sbt(): correct comment to not talk about ticks 2022-08-20 20:33:17 +03:00
subr_smp.c AST: rework 2022-08-02 21:11:09 +03:00
subr_smr.c
subr_stack.c
subr_stats.c rb_tree: reduce duplication in balancing code 2022-09-07 23:46:19 -05:00
subr_syscall.c proc: load/store p_cowgen using atomic primitives 2022-02-13 13:07:08 +00:00
subr_taskqueue.c LinuxKPI: Implement kthread_worker related functions 2022-05-17 15:10:20 +03:00
subr_terminal.c Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights 2021-08-08 10:42:24 -04:00
subr_trap.c kern/subr_trap.c: cleanup no longer needed headers 2022-08-02 21:11:10 +03:00
subr_turnstile.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
subr_uio.c
subr_unit.c unr: remove UNR64_LOCKED 2022-10-08 10:41:21 +00:00
subr_vmem.c vmem: disable debug.vmem_check by default 2021-09-02 18:28:45 +00:00
subr_witness.c ddb: annotate some commands with DB_CMD_MEMSAFE 2022-07-18 22:06:09 +00:00
sys_capability.c fd: rename fget*_locked to fget*_noref 2022-02-22 18:53:43 +00:00
sys_eventfd.c struct kinfo_file changes needed for lsof to work using only usermode APIs` 2022-06-18 12:34:25 +03:00
sys_generic.c AST: rework 2022-08-02 21:11:09 +03:00
sys_getrandom.c
sys_pipe.c SPDX: Not BSD-4-Clause 2022-09-14 21:29:31 -06:00
sys_procdesc.c vfs: remove thread argument from VOP_STAT 2021-10-11 13:22:32 +00:00
sys_process.c AST: rework 2022-08-02 21:11:09 +03:00
sys_socket.c soo_stat: Ensure error is always initialized. 2022-08-26 11:17:55 -07:00
syscalls.c Regen 2021-12-09 02:49:10 +02:00
syscalls.master Allow posix_fadvise in capability mode 2022-04-14 15:11:21 -04:00
systrace_args.c Regen 2021-12-09 02:49:10 +02:00
sysv_ipc.c freebsd32: rename old SysV IPC types 2021-11-17 20:12:23 +00:00
sysv_msg.c Adjust function definitions in sysv_msg.c to avoid clang 15 warnings 2022-07-26 19:59:57 +02:00
sysv_sem.c sysvsem: Fix a typo 2022-05-14 14:07:20 +03:00
sysv_shm.c Make SHMMAXPGS an unsigned long 2022-03-21 10:27:35 +00:00
tty_compat.c
tty_info.c tty: Default to printing kernel stack traceback only on INVARIANT kernels 2022-07-02 08:02:12 -06:00
tty_inq.c tty: Remove an incorrect assertion from ttyinq_line_iterate() 2022-04-12 17:30:04 -04:00
tty_outq.c Minor style cleanup 2021-04-18 11:14:17 -06:00
tty_pts.c sysent: Get rid of bogus sys/sysent.h include. 2022-05-28 20:52:17 +03:00
tty_tty.c Retire clone_drain_lock 2022-08-20 09:44:05 +00:00
tty_ttydisc.c
tty.c fd: rename fget*_locked to fget*_noref 2022-02-22 18:53:43 +00:00
uipc_accf.c Consistently use the SOLISTENING() macro 2021-06-14 17:32:27 -04:00
uipc_debug.c domains: use queue(9) SLIST for linked list of domains 2022-08-29 19:15:01 -07:00
uipc_domain.c protocols: make socket buffers ioctl handler changeable 2022-09-28 10:20:09 +00:00
uipc_ktls.c tcp: remove INP_TIMEWAIT flag 2022-10-06 19:24:37 -07:00
uipc_mbuf2.c mbuf_tags: use explicitly sized type for 'type' parameter 2021-12-29 09:23:52 +01:00
uipc_mbuf.c mbufs: isolate max_linkhdr and max_protohdr handling in the mbuf code 2022-08-29 19:14:25 -07:00
uipc_mbufhash.c
uipc_mqueue.c vfs: always retain path buffer after lookup 2022-09-17 09:10:38 +00:00
uipc_sem.c freebsd32: semid_t -> int32_t 2021-11-17 20:12:26 +00:00
uipc_shm.c Convert runtime param checks to KASSERTs for fo_fspacectl 2022-07-23 15:16:23 -04:00
uipc_sockbuf.c sockbufs: add sbreserve_locked_limit() with custom maxsockbuf limit. 2022-09-28 10:20:09 +00:00
uipc_socket.c Revert "listen(): improve POSIX compliance" 2022-10-12 04:33:00 +02:00
uipc_syscalls.c Fixup build after recent getsock changes 2022-09-10 20:40:43 +00:00
uipc_usrreq.c unix/dgram: don't leak file descriptors when socket write failed 2022-09-30 13:43:08 -07:00
vfs_acl.c vfs: introduce V_PCATCH to stop abusing PCATCH 2022-09-17 15:41:37 +00:00
vfs_aio.c vfs: introduce V_PCATCH to stop abusing PCATCH 2022-09-17 15:41:37 +00:00
vfs_bio.c Adjust function definition in vfs_bio.c to avoid clang 15 warnings 2022-07-26 19:59:57 +02:00
vfs_cache.c cache: add cache_assert_no_entries 2022-10-24 15:37:43 +00:00
vfs_cluster.c cluster_write(): do not access buffer after it is released 2021-09-02 21:36:33 +03:00
vfs_default.c vfs: NDFREE(&nd, NDF_ONLY_PNBUF) -> NDFREE_PNBUF(&nd) 2022-03-24 10:20:51 +00:00
vfs_export.c
vfs_extattr.c vfs: introduce V_PCATCH to stop abusing PCATCH 2022-09-17 15:41:37 +00:00
vfs_hash.c vfs_hash_rehash(): require the vnode to be exclusively locked 2021-08-27 18:39:45 +03:00
vfs_init.c Fix O(n^2) behavior in sysctl 2022-09-26 18:03:34 -06:00
vfs_lookup.c vfs_lookup(): Minor performance optimizations 2022-10-26 19:33:33 -05:00
vfs_mount.c vfs_domount(): ensure that v_mountedhere and VIRF_MOUNTPOINT are set under the vnode lock 2022-10-29 14:29:55 +03:00
vfs_mountroot.c vfs: clean up parse_mount_dev_present 2022-09-17 12:42:46 +00:00
vfs_subr.c vfs_busy(): fix wording in comment 2022-10-26 19:33:30 -05:00
vfs_syscalls.c vfs: introduce V_PCATCH to stop abusing PCATCH 2022-09-17 15:41:37 +00:00
vfs_vnops.c vn_bmap_seekhole: check that passed offset is non-negative 2022-10-19 20:24:07 +03:00
vnode_if.src vnode(9): Fix a typo in a source code comment 2022-08-07 16:08:43 +02:00