d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
0d69690e8f
freebsd-nq/sys/powerpc/aim/mmu_oea.c

2530 lines
65 KiB
C
Raw Normal View History

/* -> /*- for license, minor formatting changes
2005-01-07 02:29:27 +00:00
/*-
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Copyright (c) 2001 The NetBSD Foundation, Inc.
* All rights reserved.
*
* This code is derived from software contributed to The NetBSD Foundation
* by Matt Thomas <matt@3am-software.com> of Allegro Networks, Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by the NetBSD
* Foundation, Inc. and its contributors.
* 4. Neither the name of The NetBSD Foundation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
* ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
* TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
* BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
/* -> /*- for license, minor formatting changes
2005-01-07 02:29:27 +00:00
/*-
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
* Copyright (C) 1995, 1996 Wolfgang Solfrank.
* Copyright (C) 1995, 1996 TooLs GmbH.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by TooLs GmbH.
* 4. The name of TooLs GmbH may not be used to endorse or promote products
* derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY TOOLS GMBH ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
* IN NO EVENT SHALL TOOLS GMBH BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
* OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
* OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
Fix comment breakage.
2001-06-27 12:20:48 +00:00
* $NetBSD: pmap.c,v 1.28 2000/03/26 20:42:36 kleink Exp $
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
/* -> /*- for license, minor formatting changes
2005-01-07 02:29:27 +00:00
/*-
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
* Copyright (C) 2001 Benno Rice.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY Benno Rice ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
* IN NO EVENT SHALL TOOLS GMBH BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
* OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
* OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
* ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
Use __FBSDID rather than rcsid[].
2003-04-03 21:36:33 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Manages physical address maps.
*
* Since the information managed by this module is also stored by the
* logical address mapping module, this module may throw away valid virtual
* to physical mappings at almost any time. However, invalidations of
* mappings must be done as requested.
*
* In order to cope with hardware architectures which make virtual to
* physical map invalidates expensive, this module may delay invalidate
* reduced protection operations until such time as they are actually
* necessary. This module is given full information as to which processors
* are currently using which maps, and to when physical maps must be made
* correct.
*/
Deal with 'options KSTACK_PAGES' being a global option.
2003-07-31 01:31:32 +00:00
#include "opt_kstack_pages.h"
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
#include <sys/param.h>
Make PMAP_SHPGPERPROC tunable. One shouldn't need to recompile a kernel for this, since it is easy to run into with large systems with lots of shared mmap space. Obtained from: yahoo
2001-07-27 01:08:59 +00:00
#include <sys/kernel.h>
Add the powerpc support. Note that there is a dirty hack for calling openpic_write(), but nwhitehorn approved it. Discussed with: nwhitehorn
2011-05-09 16:16:15 +00:00
#include <sys/queue.h>
#include <sys/cpuset.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <sys/ktr.h>
Fix includes based on recent changes to lock.h, mutex.h and ktr.h.
2001-10-19 22:45:46 +00:00
#include <sys/lock.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <sys/msgbuf.h>
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
#include <sys/mutex.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <sys/proc.h>
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
#include <sys/rwlock.h>
Add the powerpc support. Note that there is a dirty hack for calling openpic_write(), but nwhitehorn approved it. Discussed with: nwhitehorn
2011-05-09 16:16:15 +00:00
#include <sys/sched.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <sys/sysctl.h>
#include <sys/systm.h>
#include <sys/vmmeter.h>
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <dev/ofw/openfirm.h>
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
#include <vm/vm.h>
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
#include <vm/vm_param.h>
#include <vm/vm_kern.h>
#include <vm/vm_page.h>
#include <vm/vm_map.h>
#include <vm/vm_object.h>
#include <vm/vm_extern.h>
#include <vm/vm_pageout.h>
#include <vm/vm_pager.h>
Remove references to vm_zone.h and switch over to the new uma API.
2002-03-21 01:11:31 +00:00
#include <vm/uma.h>
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Cleaned up param.h: - culled long-dead #define's - segment register defs moved to sr.h - NPMAPS moved to pmap.h - KERNBASE moved to vmparam.h - removed include of <machine/cpu.h> and fixed src files that relied on this. Modifying segment register code no longer causes gcc rebuilds :-)
2004-02-11 07:27:34 +00:00
#include <machine/cpu.h>
Factor out platform dependent things unrelated to device drivers into a new platform module. These are probed in early boot, and have the responsibility of determining the layout of physical memory, determining the CPU timebase frequency, and handling the zoo of SMP mechanisms found on PowerPC. Reviewed by: marcel, raj Book-E parts by: raj
2009-05-14 00:34:26 +00:00
#include <machine/platform.h>
Add missing include file.
2001-09-20 15:32:56 +00:00
#include <machine/bat.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <machine/frame.h>
#include <machine/md_var.h>
#include <machine/psl.h>
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
#include <machine/pte.h>
MFp4: SMP support
2008-04-27 22:33:43 +00:00
#include <machine/smp.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#include <machine/sr.h>
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
#include <machine/mmuvar.h>
Allow this to work on embedded systems without Open Firmware by making lack of a /chosen non-fatal, and manually removing memory in use by the kernel from the physical memory map. Submitted by: rpaulo
2011-12-16 23:46:05 +00:00
#include <machine/trap_aim.h>
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
#include "mmu_if.h"
#define MOEA_DEBUG
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#define TODO panic("%s: not implemented", __func__);
#define VSID_MAKE(sr, hash) ((sr) | (((hash) & 0xfffff) << 4))
#define VSID_TO_SR(vsid) ((vsid) & 0xf)
#define VSID_TO_HASH(vsid) (((vsid) >> 4) & 0xfffff)
struct ofw_map {
vm_offset_t om_va;
vm_size_t om_len;
vm_offset_t om_pa;
u_int om_mode;
};
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Map of physical memory regions.
*/
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
static struct mem_region *regions;
static struct mem_region *pregions;
MFppc64: Kernel sources for 64-bit PowerPC, along with build-system changes to keep 32-bit kernels compiling (build system changes for 64-bit kernels are coming later). Existing 32-bit PowerPC kernel configurations must be updated after this change to specify their architecture.
2010-07-13 05:32:19 +00:00
static u_int phys_avail_count;
static int regions_sz, pregions_sz;
Remove the statically allocated array that holds OpenFirmware memory mappings during pmap_bootstrap. Instead, temporarily help ourselves to some memory from phys_avail since we won't need it post-boostrap.
2002-07-18 12:43:08 +00:00
static struct ofw_map *translations;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
/*
* Lock for the pteg and pvo tables.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
struct mtx moea_table_mutex;
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
struct mtx moea_vsid_mutex;
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
/* tlbie instruction synchronization */
static struct mtx tlbie_mtx;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* PTEG data.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static struct pteg *moea_pteg_table;
u_int moea_pteg_count;
u_int moea_pteg_mask;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* PVO data.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
struct pvo_head *moea_pvo_table; /* pvo entries by pteg index */
struct pvo_head moea_pvo_kunmanaged =
LIST_HEAD_INITIALIZER(moea_pvo_kunmanaged); /* list of unmanaged pages */
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Rework the known rwlock to benefit about staying on their own cache line in order to avoid manual frobbing but using struct rwlock_padalign. Reviewed by: alc, jimharris
2012-11-03 23:03:14 +00:00
static struct rwlock_padalign pvh_global_lock;
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
uma_zone_t moea_upvo_zone; /* zone for pvo entries for unmanaged pages */
uma_zone_t moea_mpvo_zone; /* zone for pvo entries for managed pages */
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
#define BPVO_POOL_SIZE 32768
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static struct pvo_entry *moea_bpvo_pool;
static int moea_bpvo_pool_index = 0;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#define VSID_NBPW (sizeof(u_int32_t) * 8)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static u_int moea_vsid_bitmap[NPMAPS / VSID_NBPW];
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static boolean_t moea_initialized = FALSE;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Statistics.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
u_int moea_pte_valid = 0;
u_int moea_pte_overflow = 0;
u_int moea_pte_replacements = 0;
u_int moea_pvo_entries = 0;
u_int moea_pvo_enter_calls = 0;
u_int moea_pvo_remove_calls = 0;
u_int moea_pte_spills = 0;
SYSCTL_INT(_machdep, OID_AUTO, moea_pte_valid, CTLFLAG_RD, &moea_pte_valid,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
0, "");
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
SYSCTL_INT(_machdep, OID_AUTO, moea_pte_overflow, CTLFLAG_RD,
&moea_pte_overflow, 0, "");
SYSCTL_INT(_machdep, OID_AUTO, moea_pte_replacements, CTLFLAG_RD,
&moea_pte_replacements, 0, "");
SYSCTL_INT(_machdep, OID_AUTO, moea_pvo_entries, CTLFLAG_RD, &moea_pvo_entries,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
0, "");
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
SYSCTL_INT(_machdep, OID_AUTO, moea_pvo_enter_calls, CTLFLAG_RD,
&moea_pvo_enter_calls, 0, "");
SYSCTL_INT(_machdep, OID_AUTO, moea_pvo_remove_calls, CTLFLAG_RD,
&moea_pvo_remove_calls, 0, "");
SYSCTL_INT(_machdep, OID_AUTO, moea_pte_spills, CTLFLAG_RD,
&moea_pte_spills, 0, "");
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
* Allocate physical memory for use in moea_bootstrap.
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static vm_offset_t moea_bootstrap_alloc(vm_size_t, u_int);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* PTE calls.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static int moea_pte_insert(u_int, struct pte *);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* PVO calls.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static int moea_pvo_enter(pmap_t, uma_zone_t, struct pvo_head *,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
vm_offset_t, vm_offset_t, u_int, int);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static void moea_pvo_remove(struct pvo_entry *, int);
static struct pvo_entry *moea_pvo_find_va(pmap_t, vm_offset_t, int *);
static struct pte *moea_pvo_to_pte(const struct pvo_entry *, int);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Utility routines.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
static void moea_enter_locked(pmap_t, vm_offset_t, vm_page_t,
vm_prot_t, boolean_t);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static void moea_syncicache(vm_offset_t, vm_size_t);
static boolean_t moea_query_bit(vm_page_t, int);
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
static u_int moea_clear_bit(vm_page_t, int);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static void moea_kremove(mmu_t, vm_offset_t);
int moea_pte_spill(vm_offset_t);
/*
* Kernel MMU interface
*/
void moea_change_wiring(mmu_t, pmap_t, vm_offset_t, boolean_t);
void moea_clear_modify(mmu_t, vm_page_t);
void moea_clear_reference(mmu_t, vm_page_t);
void moea_copy_page(mmu_t, vm_page_t, vm_page_t);
void moea_enter(mmu_t, pmap_t, vm_offset_t, vm_page_t, vm_prot_t, boolean_t);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
void moea_enter_object(mmu_t, pmap_t, vm_offset_t, vm_offset_t, vm_page_t,
vm_prot_t);
Remove mpte optimization from pmap_enter_quick(). There is a race with the current locking scheme and removing it should have no measurable performance impact. This fixes page faults leading to panics in pmap_enter_quick_locked() on amd64/i386. Reviewed by: alc,jhb,peter,ps
2006-06-15 01:01:06 +00:00
void moea_enter_quick(mmu_t, pmap_t, vm_offset_t, vm_page_t, vm_prot_t);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
vm_paddr_t moea_extract(mmu_t, pmap_t, vm_offset_t);
vm_page_t moea_extract_and_hold(mmu_t, pmap_t, vm_offset_t, vm_prot_t);
void moea_init(mmu_t);
boolean_t moea_is_modified(mmu_t, vm_page_t);
Implement pmap_is_prefaultable(). Reviewed by: nwhitehorn
2010-11-01 02:22:48 +00:00
boolean_t moea_is_prefaultable(mmu_t, pmap_t, vm_offset_t);
Resurrect pmap_is_referenced() and use it in mincore(). Essentially, pmap_ts_referenced() is not always appropriate for checking whether or not pages have been referenced because it clears any reference bits that it encounters. For example, in mincore(), clearing the reference bits has two negative consequences. First, it throws off the activity count calculations performed by the page daemon. Specifically, a page on which mincore() has called pmap_ts_referenced() looks less active to the page daemon than it should. Consequently, the page could be deactivated prematurely by the page daemon. Arguably, this problem could be fixed by having mincore() duplicate the activity count calculation on the page. However, there is a second problem for which that is not a solution. In order to clear a reference on a 4KB page, it may be necessary to demote a 2/4MB page mapping. Thus, a mincore() by one process can have the side effect of demoting a superpage mapping within another process!
2010-04-24 17:32:52 +00:00
boolean_t moea_is_referenced(mmu_t, vm_page_t);
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
int moea_ts_referenced(mmu_t, vm_page_t);
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
vm_offset_t moea_map(mmu_t, vm_offset_t *, vm_paddr_t, vm_paddr_t, int);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
boolean_t moea_page_exists_quick(mmu_t, pmap_t, vm_page_t);
Prevent the leakage of wired pages in the following circumstances: First, a file is mmap(2)ed and then mlock(2)ed. Later, it is truncated. Under "normal" circumstances, i.e., when the file is not mlock(2)ed, the pages beyond the EOF are unmapped and freed. However, when the file is mlock(2)ed, the pages beyond the EOF are unmapped but not freed because they have a non-zero wire count. This can be a mistake. Specifically, it is a mistake if the sole reason why the pages are wired is because of wired, managed mappings. Previously, unmapping the pages destroys these wired, managed mappings, but does not reduce the pages' wire count. Consequently, when the file is unmapped, the pages are not unwired because the wired mapping has been destroyed. Moreover, when the vm object is finally destroyed, the pages are leaked because they are still wired. The fix is to reduce the pages' wired count by the number of wired, managed mappings destroyed. To do this, I introduce a new pmap function pmap_page_wired_mappings() that returns the number of managed mappings to the given physical page that are wired, and I use this function in vm_object_page_remove(). Reviewed by: tegge MFC after: 6 weeks
2007-11-17 22:52:29 +00:00
int moea_page_wired_mappings(mmu_t, vm_page_t);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
void moea_pinit(mmu_t, pmap_t);
void moea_pinit0(mmu_t, pmap_t);
void moea_protect(mmu_t, pmap_t, vm_offset_t, vm_offset_t, vm_prot_t);
void moea_qenter(mmu_t, vm_offset_t, vm_page_t *, int);
void moea_qremove(mmu_t, vm_offset_t, int);
void moea_release(mmu_t, pmap_t);
void moea_remove(mmu_t, pmap_t, vm_offset_t, vm_offset_t);
void moea_remove_all(mmu_t, vm_page_t);
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
void moea_remove_write(mmu_t, vm_page_t);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
void moea_zero_page(mmu_t, vm_page_t);
void moea_zero_page_area(mmu_t, vm_page_t, int, int);
void moea_zero_page_idle(mmu_t, vm_page_t);
void moea_activate(mmu_t, struct thread *);
void moea_deactivate(mmu_t, struct thread *);
Add support for 64-bit PowerPC CPUs operating in the 64-bit bridge mode provided, for example, on the PowerPC 970 (G5), as well as on related CPUs like the POWER3 and POWER4. This also adds support for various built-in hardware found on Apple G5 hardware (e.g. the IBM CPC925 northbridge). Reviewed by: grehan
2009-04-04 00:22:44 +00:00
void moea_cpu_bootstrap(mmu_t, int);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
void moea_bootstrap(mmu_t, vm_offset_t, vm_offset_t);
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
void *moea_mapdev(mmu_t, vm_paddr_t, vm_size_t);
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
void *moea_mapdev_attr(mmu_t, vm_offset_t, vm_size_t, vm_memattr_t);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
void moea_unmapdev(mmu_t, vm_offset_t, vm_size_t);
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
vm_paddr_t moea_kextract(mmu_t, vm_offset_t);
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
void moea_kenter_attr(mmu_t, vm_offset_t, vm_offset_t, vm_memattr_t);
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
void moea_kenter(mmu_t, vm_offset_t, vm_paddr_t);
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
void moea_page_set_memattr(mmu_t mmu, vm_page_t m, vm_memattr_t ma);
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
boolean_t moea_dev_direct_mapped(mmu_t, vm_paddr_t, vm_size_t);
o Introduce vm_sync_icache() for making the I-cache coherent with the memory or D-cache, depending on the semantics of the platform. vm_sync_icache() is basically a wrapper around pmap_sync_icache(), that translates the vm_map_t argumument to pmap_t. o Introduce pmap_sync_icache() to all PMAP implementation. For powerpc it replaces the pmap_page_executable() function, added to solve the I-cache problem in uiomove_fromphys(). o In proc_rwmem() call vm_sync_icache() when writing to a page that has execute permissions. This assures that when breakpoints are written, the I-cache will be coherent and the process will actually hit the breakpoint. o This also fixes the Book-E PMAP implementation that was missing necessary locking while trying to deal with the I-cache coherency in pmap_enter() (read: mmu_booke_enter_locked). The key property of this change is that the I-cache is made coherent *after* writes have been done. Doing it in the PMAP layer when adding or changing a mapping means that the I-cache is made coherent *before* any writes happen. The difference is key when the I-cache prefetches.
2009-10-21 18:38:02 +00:00
static void moea_sync_icache(mmu_t, pmap_t, vm_offset_t, vm_size_t);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static mmu_method_t moea_methods[] = {
MMUMETHOD(mmu_change_wiring, moea_change_wiring),
MMUMETHOD(mmu_clear_modify, moea_clear_modify),
MMUMETHOD(mmu_clear_reference, moea_clear_reference),
MMUMETHOD(mmu_copy_page, moea_copy_page),
MMUMETHOD(mmu_enter, moea_enter),
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
MMUMETHOD(mmu_enter_object, moea_enter_object),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
MMUMETHOD(mmu_enter_quick, moea_enter_quick),
MMUMETHOD(mmu_extract, moea_extract),
MMUMETHOD(mmu_extract_and_hold, moea_extract_and_hold),
MMUMETHOD(mmu_init, moea_init),
MMUMETHOD(mmu_is_modified, moea_is_modified),
Implement pmap_is_prefaultable(). Reviewed by: nwhitehorn
2010-11-01 02:22:48 +00:00
MMUMETHOD(mmu_is_prefaultable, moea_is_prefaultable),
Resurrect pmap_is_referenced() and use it in mincore(). Essentially, pmap_ts_referenced() is not always appropriate for checking whether or not pages have been referenced because it clears any reference bits that it encounters. For example, in mincore(), clearing the reference bits has two negative consequences. First, it throws off the activity count calculations performed by the page daemon. Specifically, a page on which mincore() has called pmap_ts_referenced() looks less active to the page daemon than it should. Consequently, the page could be deactivated prematurely by the page daemon. Arguably, this problem could be fixed by having mincore() duplicate the activity count calculation on the page. However, there is a second problem for which that is not a solution. In order to clear a reference on a 4KB page, it may be necessary to demote a 2/4MB page mapping. Thus, a mincore() by one process can have the side effect of demoting a superpage mapping within another process!
2010-04-24 17:32:52 +00:00
MMUMETHOD(mmu_is_referenced, moea_is_referenced),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
MMUMETHOD(mmu_ts_referenced, moea_ts_referenced),
MMUMETHOD(mmu_map, moea_map),
MMUMETHOD(mmu_page_exists_quick,moea_page_exists_quick),
Prevent the leakage of wired pages in the following circumstances: First, a file is mmap(2)ed and then mlock(2)ed. Later, it is truncated. Under "normal" circumstances, i.e., when the file is not mlock(2)ed, the pages beyond the EOF are unmapped and freed. However, when the file is mlock(2)ed, the pages beyond the EOF are unmapped but not freed because they have a non-zero wire count. This can be a mistake. Specifically, it is a mistake if the sole reason why the pages are wired is because of wired, managed mappings. Previously, unmapping the pages destroys these wired, managed mappings, but does not reduce the pages' wire count. Consequently, when the file is unmapped, the pages are not unwired because the wired mapping has been destroyed. Moreover, when the vm object is finally destroyed, the pages are leaked because they are still wired. The fix is to reduce the pages' wired count by the number of wired, managed mappings destroyed. To do this, I introduce a new pmap function pmap_page_wired_mappings() that returns the number of managed mappings to the given physical page that are wired, and I use this function in vm_object_page_remove(). Reviewed by: tegge MFC after: 6 weeks
2007-11-17 22:52:29 +00:00
MMUMETHOD(mmu_page_wired_mappings,moea_page_wired_mappings),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
MMUMETHOD(mmu_pinit, moea_pinit),
MMUMETHOD(mmu_pinit0, moea_pinit0),
MMUMETHOD(mmu_protect, moea_protect),
MMUMETHOD(mmu_qenter, moea_qenter),
MMUMETHOD(mmu_qremove, moea_qremove),
MMUMETHOD(mmu_release, moea_release),
MMUMETHOD(mmu_remove, moea_remove),
MMUMETHOD(mmu_remove_all, moea_remove_all),
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
MMUMETHOD(mmu_remove_write, moea_remove_write),
o Introduce vm_sync_icache() for making the I-cache coherent with the memory or D-cache, depending on the semantics of the platform. vm_sync_icache() is basically a wrapper around pmap_sync_icache(), that translates the vm_map_t argumument to pmap_t. o Introduce pmap_sync_icache() to all PMAP implementation. For powerpc it replaces the pmap_page_executable() function, added to solve the I-cache problem in uiomove_fromphys(). o In proc_rwmem() call vm_sync_icache() when writing to a page that has execute permissions. This assures that when breakpoints are written, the I-cache will be coherent and the process will actually hit the breakpoint. o This also fixes the Book-E PMAP implementation that was missing necessary locking while trying to deal with the I-cache coherency in pmap_enter() (read: mmu_booke_enter_locked). The key property of this change is that the I-cache is made coherent *after* writes have been done. Doing it in the PMAP layer when adding or changing a mapping means that the I-cache is made coherent *before* any writes happen. The difference is key when the I-cache prefetches.
2009-10-21 18:38:02 +00:00
MMUMETHOD(mmu_sync_icache, moea_sync_icache),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
MMUMETHOD(mmu_zero_page, moea_zero_page),
MMUMETHOD(mmu_zero_page_area, moea_zero_page_area),
MMUMETHOD(mmu_zero_page_idle, moea_zero_page_idle),
MMUMETHOD(mmu_activate, moea_activate),
MMUMETHOD(mmu_deactivate, moea_deactivate),
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
MMUMETHOD(mmu_page_set_memattr, moea_page_set_memattr),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
/* Internal interfaces */
MMUMETHOD(mmu_bootstrap, moea_bootstrap),
Add support for 64-bit PowerPC CPUs operating in the 64-bit bridge mode provided, for example, on the PowerPC 970 (G5), as well as on related CPUs like the POWER3 and POWER4. This also adds support for various built-in hardware found on Apple G5 hardware (e.g. the IBM CPC925 northbridge). Reviewed by: grehan
2009-04-04 00:22:44 +00:00
MMUMETHOD(mmu_cpu_bootstrap, moea_cpu_bootstrap),
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
MMUMETHOD(mmu_mapdev_attr, moea_mapdev_attr),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
MMUMETHOD(mmu_mapdev, moea_mapdev),
MMUMETHOD(mmu_unmapdev, moea_unmapdev),
MMUMETHOD(mmu_kextract, moea_kextract),
MMUMETHOD(mmu_kenter, moea_kenter),
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
MMUMETHOD(mmu_kenter_attr, moea_kenter_attr),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
MMUMETHOD(mmu_dev_direct_mapped,moea_dev_direct_mapped),
{ 0, 0 }
};
Introduce inheritance into the PowerPC MMU kobj interface. include/mmuvar.h - Change the MMU_DEF macro to also create the class definition as well as define the DATA_SET. Add a macro, MMU_DEF_INHERIT, which has an extra parameter specifying the MMU class to inherit methods from. Update the comments at the start of the header file to describe the new macros. booke/pmap.c aim/mmu_oea.c aim/mmu_oea64.c - Collapse mmu_def_t declaration into updated MMU_DEF macro The MMU_DEF_INHERIT macro will be used in the PS3 MMU implementation to allow it to inherit the stock powerpc64 MMU methods. Reviewed by: nwhitehorn
2010-09-15 00:17:52 +00:00
MMU_DEF(oea_mmu, MMU_TYPE_OEA, moea_methods, 0);
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
static __inline uint32_t
moea_calc_wimg(vm_offset_t pa, vm_memattr_t ma)
{
uint32_t pte_lo;
int i;
if (ma != VM_MEMATTR_DEFAULT) {
switch (ma) {
case VM_MEMATTR_UNCACHEABLE:
return (PTE_I | PTE_G);
case VM_MEMATTR_WRITE_COMBINING:
case VM_MEMATTR_WRITE_BACK:
case VM_MEMATTR_PREFETCHABLE:
return (PTE_I);
case VM_MEMATTR_WRITE_THROUGH:
return (PTE_W | PTE_M);
}
}
/*
* Assume the page is cache inhibited and access is guarded unless
* it's in our available memory array.
*/
pte_lo = PTE_I | PTE_G;
for (i = 0; i < pregions_sz; i++) {
if ((pa >= pregions[i].mr_start) &&
(pa < (pregions[i].mr_start + pregions[i].mr_size))) {
pte_lo = PTE_M;
break;
}
}
return pte_lo;
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
static void
tlbie(vm_offset_t va)
{
mtx_lock_spin(&tlbie_mtx);
Follow exactly the steps in architecture manual for correctly invalidating TLB entries instead of trying to cut corners.
2010-10-04 16:07:48 +00:00
__asm __volatile("ptesync");
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
__asm __volatile("tlbie %0" :: "r"(va));
Follow exactly the steps in architecture manual for correctly invalidating TLB entries instead of trying to cut corners.
2010-10-04 16:07:48 +00:00
__asm __volatile("eieio; tlbsync; ptesync");
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
mtx_unlock_spin(&tlbie_mtx);
}
static void
tlbia(void)
{
vm_offset_t va;
for (va = 0; va < 0x00040000; va += 0x00001000) {
__asm __volatile("tlbie %0" :: "r"(va));
powerpc_sync();
}
__asm __volatile("tlbsync");
powerpc_sync();
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline int
va_to_sr(u_int *sr, vm_offset_t va)
{
return (sr[(uintptr_t)va >> ADDR_SR_SHFT]);
}
static __inline u_int
va_to_pteg(u_int sr, vm_offset_t addr)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
u_int hash;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
hash = (sr & SR_VSID_MASK) ^ (((u_int)addr & ADDR_PIDX) >>
ADDR_PIDX_SHFT);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
return (hash & moea_pteg_mask);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline struct pvo_head *
vm_page_to_pvoh(vm_page_t m)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (&m->md.mdpg_pvoh);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_attr_clear(vm_page_t m, int ptebit)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_assert(&pvh_global_lock, RA_WLOCKED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
m->md.mdpg_attrs &= ~ptebit;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
static __inline int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_attr_fetch(vm_page_t m)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (m->md.mdpg_attrs);
}
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_attr_save(vm_page_t m, int ptebit)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_assert(&pvh_global_lock, RA_WLOCKED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
m->md.mdpg_attrs |= ptebit;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
static __inline int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_compare(const struct pte *pt, const struct pte *pvo_pt)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pt->pte_hi == pvo_pt->pte_hi)
return (1);
return (0);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
static __inline int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_match(struct pte *pt, u_int sr, vm_offset_t va, int which)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (pt->pte_hi & ~PTE_VALID) ==
(((sr & SR_VSID_MASK) << PTE_VSID_SHFT) |
((va >> ADDR_API_SHFT) & PTE_API) | which);
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_create(struct pte *pt, u_int sr, vm_offset_t va, u_int pte_lo)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Construct a PTE. Default to IMB initially. Valid bit only gets
* set when the real pte is set in memory.
*
* Note: Don't set the valid bit for correct operation of tlb update.
*/
pt->pte_hi = ((sr & SR_VSID_MASK) << PTE_VSID_SHFT) |
(((va & ADDR_PIDX) >> ADDR_API_SHFT) & PTE_API);
pt->pte_lo = pte_lo;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_synch(struct pte *pt, struct pte *pvo_pt)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo_pt->pte_lo |= pt->pte_lo & (PTE_REF | PTE_CHG);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_clear(struct pte *pt, vm_offset_t va, int ptebit)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* As shown in Section 7.6.3.2.3
*/
pt->pte_lo &= ~ptebit;
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
tlbie(va);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_set(struct pte *pt, struct pte *pvo_pt)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo_pt->pte_hi |= PTE_VALID;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Update the PTE as defined in section 7.6.3.1.
* Note that the REF/CHG bits are from pvo_pt and thus should havce
* been saved so this routine can restore them (if desired).
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pt->pte_lo = pvo_pt->pte_lo;
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pt->pte_hi = pvo_pt->pte_hi;
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_valid++;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_unset(struct pte *pt, struct pte *pvo_pt, vm_offset_t va)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo_pt->pte_hi &= ~PTE_VALID;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Force the reg & chg bits back into the PTEs.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Invalidate the pte.
*/
pt->pte_hi &= ~PTE_VALID;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
tlbie(va);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Save the reg & chg bits.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_synch(pt, pvo_pt);
moea_pte_valid--;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_change(struct pte *pt, struct pte *pvo_pt, vm_offset_t va)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Invalidate the PTE
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_unset(pt, pvo_pt, va);
moea_pte_set(pt, pvo_pt);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Quick sort callout for comparing memory regions.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static int om_cmp(const void *a, const void *b);
static int
om_cmp(const void *a, const void *b)
{
const struct ofw_map *mapa;
const struct ofw_map *mapb;
mapa = a;
mapb = b;
if (mapa->om_pa < mapb->om_pa)
return (-1);
else if (mapa->om_pa > mapb->om_pa)
return (1);
else
return (0);
}
MFp4: SMP support
2008-04-27 22:33:43 +00:00
void
Add support for 64-bit PowerPC CPUs operating in the 64-bit bridge mode provided, for example, on the PowerPC 970 (G5), as well as on related CPUs like the POWER3 and POWER4. This also adds support for various built-in hardware found on Apple G5 hardware (e.g. the IBM CPC925 northbridge). Reviewed by: grehan
2009-04-04 00:22:44 +00:00
moea_cpu_bootstrap(mmu_t mmup, int ap)
MFp4: SMP support
2008-04-27 22:33:43 +00:00
{
u_int sdr;
int i;
if (ap) {
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
MFp4: SMP support
2008-04-27 22:33:43 +00:00
__asm __volatile("mtdbatu 0,%0" :: "r"(battable[0].batu));
__asm __volatile("mtdbatl 0,%0" :: "r"(battable[0].batl));
isync();
__asm __volatile("mtibatu 0,%0" :: "r"(battable[0].batu));
__asm __volatile("mtibatl 0,%0" :: "r"(battable[0].batl));
isync();
}
The first argment of mtdbatu or mtibatu is part of the encoding. It needs to be constant, so eliminate the loop and "hand-unroll".
2008-04-28 03:04:41 +00:00
__asm __volatile("mtdbatu 1,%0" :: "r"(battable[8].batu));
__asm __volatile("mtdbatl 1,%0" :: "r"(battable[8].batl));
isync();
MFp4: SMP support
2008-04-27 22:33:43 +00:00
The first argment of mtdbatu or mtibatu is part of the encoding. It needs to be constant, so eliminate the loop and "hand-unroll".
2008-04-28 03:04:41 +00:00
__asm __volatile("mtibatu 1,%0" :: "r"(0));
__asm __volatile("mtdbatu 2,%0" :: "r"(0));
__asm __volatile("mtibatu 2,%0" :: "r"(0));
__asm __volatile("mtdbatu 3,%0" :: "r"(0));
__asm __volatile("mtibatu 3,%0" :: "r"(0));
MFp4: SMP support
2008-04-27 22:33:43 +00:00
isync();
for (i = 0; i < 16; i++)
Remove use of a separate ofw_pmap on 32-bit CPUs. Many Open Firmware mappings need to end up in the kernel anyway since the kernel begins executing in OF context. Separating them adds needless complexity, especially since the powerpc64 and mmu_oea64 code gave up on it a long time ago. As a side effect, the PPC ofw_machdep code is no longer AIM-specific, so move it to powerpc/ofw.
2010-11-12 05:12:38 +00:00
mtsrin(i << ADDR_SR_SHFT, kernel_pmap->pm_sr[i]);
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
MFp4: SMP support
2008-04-27 22:33:43 +00:00
sdr = (u_int)moea_pteg_table | (moea_pteg_mask >> 10);
__asm __volatile("mtsdr1 %0" :: "r"(sdr));
isync();
Invalidate the TLB in pmap_cpu_bootstrap(), so that it also happens on the APs.
2008-05-23 19:16:24 +00:00
tlbia();
MFp4: SMP support
2008-04-27 22:33:43 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_bootstrap(mmu_t mmup, vm_offset_t kernelstart, vm_offset_t kernelend)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
ihandle_t mmui;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
phandle_t chosen, mmu;
int sz;
int i, j;
Allow user to undersize memory with hw.physmem loader variable. Obtained from: i386/machdep.c:getmemsize()
2005-03-07 01:46:06 +00:00
vm_size_t size, physsz, hwphyssz;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
vm_offset_t pa, va, off;
Implement a facility for dynamic per-cpu variables. - Modules and kernel code alike may use DPCPU_DEFINE(), DPCPU_GET(), DPCPU_SET(), etc. akin to the statically defined PCPU_*. Requires only one extra instruction more than PCPU_* and is virtually the same as __thread for builtin and much faster for shared objects. DPCPU variables can be initialized when defined. - Modules are supported by relocating the module's per-cpu linker set over space reserved in the kernel. Modules may fail to load if there is insufficient space available. - Track space available for modules with a one-off extent allocator. Free may block for memory to allocate space for an extent. Reviewed by: jhb, rwatson, kan, sam, grehan, marius, marcel, stas
2009-06-23 22:42:39 +00:00
void *dpcpu;
Temporarily disable instruction relocation while setting up the kernel's IBAT entry in early boot in order to prevent possible faults from races between the instruction cache and the MMU. PR: powerpc/148003 MFC after: 3 days
2010-06-20 16:56:48 +00:00
register_t msr;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
/*
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
* Set up BAT0 to map the lowest 256 MB area
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
*/
battable[0x0].batl = BATL(0x00000000, BAT_M, BAT_PP_RW);
battable[0x0].batu = BATU(0x00000000, BAT_BL_256M, BAT_Vs);
/*
* Map PCI memory space.
*/
battable[0x8].batl = BATL(0x80000000, BAT_I|BAT_G, BAT_PP_RW);
battable[0x8].batu = BATU(0x80000000, BAT_BL_256M, BAT_Vs);
battable[0x9].batl = BATL(0x90000000, BAT_I|BAT_G, BAT_PP_RW);
battable[0x9].batu = BATU(0x90000000, BAT_BL_256M, BAT_Vs);
battable[0xa].batl = BATL(0xa0000000, BAT_I|BAT_G, BAT_PP_RW);
battable[0xa].batu = BATU(0xa0000000, BAT_BL_256M, BAT_Vs);
battable[0xb].batl = BATL(0xb0000000, BAT_I|BAT_G, BAT_PP_RW);
battable[0xb].batu = BATU(0xb0000000, BAT_BL_256M, BAT_Vs);
/*
* Map obio devices.
*/
battable[0xf].batl = BATL(0xf0000000, BAT_I|BAT_G, BAT_PP_RW);
battable[0xf].batu = BATU(0xf0000000, BAT_BL_256M, BAT_Vs);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Use an IBAT and a DBAT to map the bottom segment of memory
Temporarily disable instruction relocation while setting up the kernel's IBAT entry in early boot in order to prevent possible faults from races between the instruction cache and the MMU. PR: powerpc/148003 MFC after: 3 days
2010-06-20 16:56:48 +00:00
* where we are. Turn off instruction relocation temporarily
* to prevent faults while reprogramming the IBAT.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Temporarily disable instruction relocation while setting up the kernel's IBAT entry in early boot in order to prevent possible faults from races between the instruction cache and the MMU. PR: powerpc/148003 MFC after: 3 days
2010-06-20 16:56:48 +00:00
msr = mfmsr();
mtmsr(msr & ~PSL_IR);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
__asm (".balign 32; \n"
Fix boot-time hang/panic on G3 systems when modifying IBAT0 in pmap_bootstrap by using the sync;isync big hammer to make sure all prior operations have completed. Reported by: Nathan Whitehorn <nathan at uchicago edu> MFC after: 2 days
2005-09-10 21:03:10 +00:00
"mtibatu 0,%0; mtibatl 0,%1; isync; \n"
G4 requires isync after 256Mb ibat/dbat update, G3 requires isync after each bat update. Otherwise, pmap_bootstrap causes an ISI exception. A fall-out of loader BAT removal.
2004-07-08 12:47:36 +00:00
"mtdbatu 0,%0; mtdbatl 0,%1; isync"
MFp4: SMP support
2008-04-27 22:33:43 +00:00
:: "r"(battable[0].batu), "r"(battable[0].batl));
Temporarily disable instruction relocation while setting up the kernel's IBAT entry in early boot in order to prevent possible faults from races between the instruction cache and the MMU. PR: powerpc/148003 MFC after: 3 days
2010-06-20 16:56:48 +00:00
mtmsr(msr);
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
/* map pci space */
MFp4: SMP support
2008-04-27 22:33:43 +00:00
__asm __volatile("mtdbatu 1,%0" :: "r"(battable[8].batu));
__asm __volatile("mtdbatl 1,%0" :: "r"(battable[8].batl));
isync();
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
Add support for 64-bit PowerPC CPUs operating in the 64-bit bridge mode provided, for example, on the PowerPC 970 (G5), as well as on related CPUs like the POWER3 and POWER4. This also adds support for various built-in hardware found on Apple G5 hardware (e.g. the IBM CPC925 northbridge). Reviewed by: grehan
2009-04-04 00:22:44 +00:00
/* set global direct map flag */
hw_direct_map = 1;
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
mem_regions(&pregions, &pregions_sz, &regions, &regions_sz);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
CTR0(KTR_PMAP, "moea_bootstrap: physical memory");
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
for (i = 0; i < pregions_sz; i++) {
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
vm_offset_t pa;
vm_offset_t end;
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
CTR3(KTR_PMAP, "physregion: %#x - %#x (%#x)",
pregions[i].mr_start,
pregions[i].mr_start + pregions[i].mr_size,
pregions[i].mr_size);
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
/*
* Install entries into the BAT table to allow all
* of physmem to be convered by on-demand BAT entries.
* The loop will sometimes set the same battable element
* twice, but that's fine since they won't be used for
* a while yet.
*/
pa = pregions[i].mr_start & 0xf0000000;
end = pregions[i].mr_start + pregions[i].mr_size;
do {
u_int n = pa >> ADDR_SR_SHFT;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
battable[n].batl = BATL(pa, BAT_M, BAT_PP_RW);
battable[n].batu = BATU(pa, BAT_BL_256M, BAT_Vs);
pa += SEGMENT_LENGTH;
} while (pa < end);
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
}
if (sizeof(phys_avail)/sizeof(phys_avail[0]) < regions_sz)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_bootstrap: phys_avail too small");
Remove some dead code: unnecessary isyncs and memory sorting, which are handled in mtmsr() and mem_regions(), respectively.
2011-06-02 14:15:44 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
phys_avail_count = 0;
Calculate physmem.
2002-03-07 10:09:24 +00:00
physsz = 0;
Replaced previous hw.physmem extraction with des's mods to getenv_ulong() - much simpler. Pointed out by: des
2005-03-07 07:31:20 +00:00
hwphyssz = 0;
TUNABLE_ULONG_FETCH("hw.physmem", (u_long *) &hwphyssz);
Get the correct memory regions from OpenFirmware. We were getting the "available" ranges, not the "physical" ranges. Clean up some of the bootstrap code in the process. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-05-27 11:18:12 +00:00
for (i = 0, j = 0; i < regions_sz; i++, j += 2) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
CTR3(KTR_PMAP, "region: %#x - %#x (%#x)", regions[i].mr_start,
regions[i].mr_start + regions[i].mr_size,
regions[i].mr_size);
Allow user to undersize memory with hw.physmem loader variable. Obtained from: i386/machdep.c:getmemsize()
2005-03-07 01:46:06 +00:00
if (hwphyssz != 0 &&
(physsz + regions[i].mr_size) >= hwphyssz) {
if (physsz < hwphyssz) {
phys_avail[j] = regions[i].mr_start;
phys_avail[j + 1] = regions[i].mr_start +
hwphyssz - physsz;
physsz = hwphyssz;
phys_avail_count++;
}
break;
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
phys_avail[j] = regions[i].mr_start;
phys_avail[j + 1] = regions[i].mr_start + regions[i].mr_size;
phys_avail_count++;
Calculate physmem.
2002-03-07 10:09:24 +00:00
physsz += regions[i].mr_size;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Allow this to work on embedded systems without Open Firmware by making lack of a /chosen non-fatal, and manually removing memory in use by the kernel from the physical memory map. Submitted by: rpaulo
2011-12-16 23:46:05 +00:00
/* Check for overlap with the kernel and exception vectors */
for (j = 0; j < 2*phys_avail_count; j+=2) {
if (phys_avail[j] < EXC_LAST)
phys_avail[j] += EXC_LAST;
if (kernelstart >= phys_avail[j] &&
kernelstart < phys_avail[j+1]) {
if (kernelend < phys_avail[j+1]) {
phys_avail[2*phys_avail_count] =
(kernelend & ~PAGE_MASK) + PAGE_SIZE;
phys_avail[2*phys_avail_count + 1] =
phys_avail[j+1];
phys_avail_count++;
}
phys_avail[j+1] = kernelstart & ~PAGE_MASK;
}
if (kernelend >= phys_avail[j] &&
kernelend < phys_avail[j+1]) {
if (kernelstart > phys_avail[j]) {
phys_avail[2*phys_avail_count] = phys_avail[j];
phys_avail[2*phys_avail_count + 1] =
kernelstart & ~PAGE_MASK;
phys_avail_count++;
}
phys_avail[j] = (kernelend & ~PAGE_MASK) + PAGE_SIZE;
}
}
Calculate physmem.
2002-03-07 10:09:24 +00:00
physmem = btoc(physsz);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Allocate PTEG table.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#ifdef PTEGCOUNT
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pteg_count = PTEGCOUNT;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
#else
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pteg_count = 0x1000;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
while (moea_pteg_count < physmem)
moea_pteg_count <<= 1;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pteg_count >>= 1;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#endif /* PTEGCOUNT */
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
size = moea_pteg_count * sizeof(struct pteg);
CTR2(KTR_PMAP, "moea_bootstrap: %d PTEGs, %d bytes", moea_pteg_count,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
size);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pteg_table = (struct pteg *)moea_bootstrap_alloc(size, size);
CTR1(KTR_PMAP, "moea_bootstrap: PTEG table at %p", moea_pteg_table);
bzero((void *)moea_pteg_table, moea_pteg_count * sizeof(struct pteg));
moea_pteg_mask = moea_pteg_count - 1;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Correct a comment.
2002-04-16 12:15:17 +00:00
* Allocate pv/overflow lists.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
size = sizeof(struct pvo_head) * moea_pteg_count;
moea_pvo_table = (struct pvo_head *)moea_bootstrap_alloc(size,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
PAGE_SIZE);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
CTR1(KTR_PMAP, "moea_bootstrap: PVO table at %p", moea_pvo_table);
for (i = 0; i < moea_pteg_count; i++)
LIST_INIT(&moea_pvo_table[i]);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
/*
* Initialize the lock that synchronizes access to the pteg and pvo
* tables.
*/
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_init(&moea_table_mutex, "pmap table", NULL, MTX_DEF |
MTX_RECURSE);
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
mtx_init(&moea_vsid_mutex, "VSID table", NULL, MTX_DEF);
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
mtx_init(&tlbie_mtx, "tlbie", NULL, MTX_SPIN);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Initialise the unmanaged pvo pool.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_bpvo_pool = (struct pvo_entry *)moea_bootstrap_alloc(
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
BPVO_POOL_SIZE*sizeof(struct pvo_entry), 0);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_bpvo_pool_index = 0;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Make sure kernel vsid is allocated as well as VSID 0.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_vsid_bitmap[(KERNEL_VSIDBITS & (NPMAPS - 1)) / VSID_NBPW]
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
|= 1 << (KERNEL_VSIDBITS % VSID_NBPW);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_vsid_bitmap[0] |= 1;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
Remove use of a separate ofw_pmap on 32-bit CPUs. Many Open Firmware mappings need to end up in the kernel anyway since the kernel begins executing in OF context. Separating them adds needless complexity, especially since the powerpc64 and mmu_oea64 code gave up on it a long time ago. As a side effect, the PPC ofw_machdep code is no longer AIM-specific, so move it to powerpc/ofw.
2010-11-12 05:12:38 +00:00
* Initialize the kernel pmap (which is statically allocated).
*/
PMAP_LOCK_INIT(kernel_pmap);
for (i = 0; i < 16; i++)
kernel_pmap->pm_sr[i] = EMPTY_SEGMENT + i;
Add the powerpc support. Note that there is a dirty hack for calling openpic_write(), but nwhitehorn approved it. Discussed with: nwhitehorn
2011-05-09 16:16:15 +00:00
CPU_FILL(&kernel_pmap->pm_active);
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
RB_INIT(&kernel_pmap->pmap_pvo);
Remove use of a separate ofw_pmap on 32-bit CPUs. Many Open Firmware mappings need to end up in the kernel anyway since the kernel begins executing in OF context. Separating them adds needless complexity, especially since the powerpc64 and mmu_oea64 code gave up on it a long time ago. As a side effect, the PPC ofw_machdep code is no longer AIM-specific, so move it to powerpc/ofw.
2010-11-12 05:12:38 +00:00
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
/*
* Initialize the global pv list lock.
*/
rw_init(&pvh_global_lock, "pmap pv global");
Remove use of a separate ofw_pmap on 32-bit CPUs. Many Open Firmware mappings need to end up in the kernel anyway since the kernel begins executing in OF context. Separating them adds needless complexity, especially since the powerpc64 and mmu_oea64 code gave up on it a long time ago. As a side effect, the PPC ofw_machdep code is no longer AIM-specific, so move it to powerpc/ofw.
2010-11-12 05:12:38 +00:00
/*
* Set up the Open Firmware mappings
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
*/
Allow this to work on embedded systems without Open Firmware by making lack of a /chosen non-fatal, and manually removing memory in use by the kernel from the physical memory map. Submitted by: rpaulo
2011-12-16 23:46:05 +00:00
chosen = OF_finddevice("/chosen");
if (chosen != -1 && OF_getprop(chosen, "mmu", &mmui, 4) != -1 &&
(mmu = OF_instance_to_package(mmui)) != -1 &&
(sz = OF_getproplen(mmu, "translations")) != -1) {
translations = NULL;
for (i = 0; phys_avail[i] != 0; i += 2) {
if (phys_avail[i + 1] >= sz) {
translations = (struct ofw_map *)phys_avail[i];
break;
}
}
if (translations == NULL)
panic("moea_bootstrap: no space to copy translations");
bzero(translations, sz);
if (OF_getprop(mmu, "translations", translations, sz) == -1)
panic("moea_bootstrap: can't get ofw translations");
CTR0(KTR_PMAP, "moea_bootstrap: translations");
sz /= sizeof(*translations);
qsort(translations, sz, sizeof (*translations), om_cmp);
for (i = 0; i < sz; i++) {
CTR3(KTR_PMAP, "translation: pa=%#x va=%#x len=%#x",
translations[i].om_pa, translations[i].om_va,
translations[i].om_len);
/*
* If the mapping is 1:1, let the RAM and device
* on-demand BAT tables take care of the translation.
*/
if (translations[i].om_va == translations[i].om_pa)
continue;
/* Enter the pages */
for (off = 0; off < translations[i].om_len;
off += PAGE_SIZE)
moea_kenter(mmup, translations[i].om_va + off,
translations[i].om_pa + off);
Modify loop test when cycling through phys_avail array. It's possible for an OpenFirmware implementation to have a single memory region (hello PearPC).
2004-07-01 08:01:49 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
/*
* Calculate the last available physical address.
*/
for (i = 0; phys_avail[i + 2] != 0; i += 2)
;
Maxmem = powerpc_btop(phys_avail[i + 1]);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Add support for 64-bit PowerPC CPUs operating in the 64-bit bridge mode provided, for example, on the PowerPC 970 (G5), as well as on related CPUs like the POWER3 and POWER4. This also adds support for various built-in hardware found on Apple G5 hardware (e.g. the IBM CPC925 northbridge). Reviewed by: grehan
2009-04-04 00:22:44 +00:00
moea_cpu_bootstrap(mmup,0);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pmap_bootstrapped++;
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
/*
* Set the start and end of kva.
*/
virtual_avail = VM_MIN_KERNEL_ADDRESS;
Reduce KVA pressure on OEA64 systems running in bridge mode by mapping UMA segments at their physical addresses instead of into KVA. This emulates the direct mapping behavior of OEA32 in an ad-hoc way. To make this work properly required sharing the entire kernel PMAP with Open Firmware, so ofw_pmap is transformed into a stub on 64-bit CPUs. Also implement some more tweaks to get more mileage out of our limited amount of KVA, principally by extending KVA into segment 16 until the beginning of the first OFW mapping. Reported by: linimon
2010-02-20 16:23:29 +00:00
virtual_end = VM_MAX_SAFE_KERNEL_ADDRESS;
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
/*
* Allocate a kernel stack with a guard page for thread0 and map it
* into the kernel page map.
*/
pa = moea_bootstrap_alloc(KSTACK_PAGES * PAGE_SIZE, PAGE_SIZE);
va = virtual_avail + KSTACK_GUARD_PAGES * PAGE_SIZE;
virtual_avail = va + KSTACK_PAGES * PAGE_SIZE;
CTR2(KTR_PMAP, "moea_bootstrap: kstack0 at %#x (%#x)", pa, va);
thread0.td_kstack = va;
thread0.td_kstack_pages = KSTACK_PAGES;
for (i = 0; i < KSTACK_PAGES; i++) {
Remove extraneous semicolons, no functional changes. Submitted by: Marc Balmer <marc@msys.ch> MFC after: 1 week
2010-01-07 21:01:37 +00:00
moea_kenter(mmup, va, pa);
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
pa += PAGE_SIZE;
va += PAGE_SIZE;
}
/*
* Allocate virtual address space for the message buffer.
*/
Make MSGBUF_SIZE kernel option a loader tunable kern.msgbufsize. Submitted by: perryh pluto.rain.com (previous version) Reviewed by: jhb Approved by: kib (mentor) Tested by: universe
2011-01-21 10:26:26 +00:00
pa = msgbuf_phys = moea_bootstrap_alloc(msgbufsize, PAGE_SIZE);
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
msgbufp = (struct msgbuf *)virtual_avail;
va = virtual_avail;
Make MSGBUF_SIZE kernel option a loader tunable kern.msgbufsize. Submitted by: perryh pluto.rain.com (previous version) Reviewed by: jhb Approved by: kib (mentor) Tested by: universe
2011-01-21 10:26:26 +00:00
virtual_avail += round_page(msgbufsize);
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
while (va < virtual_avail) {
Remove extraneous semicolons, no functional changes. Submitted by: Marc Balmer <marc@msys.ch> MFC after: 1 week
2010-01-07 21:01:37 +00:00
moea_kenter(mmup, va, pa);
Allocate a stack (with optional guard pages) for thread0 and switch to it before calling mi_startup().
2008-04-16 23:28:12 +00:00
pa += PAGE_SIZE;
va += PAGE_SIZE;
}
Implement a facility for dynamic per-cpu variables. - Modules and kernel code alike may use DPCPU_DEFINE(), DPCPU_GET(), DPCPU_SET(), etc. akin to the statically defined PCPU_*. Requires only one extra instruction more than PCPU_* and is virtually the same as __thread for builtin and much faster for shared objects. DPCPU variables can be initialized when defined. - Modules are supported by relocating the module's per-cpu linker set over space reserved in the kernel. Modules may fail to load if there is insufficient space available. - Track space available for modules with a one-off extent allocator. Free may block for memory to allocate space for an extent. Reviewed by: jhb, rwatson, kan, sam, grehan, marius, marcel, stas
2009-06-23 22:42:39 +00:00
/*
* Allocate virtual address space for the dynamic percpu area.
*/
pa = moea_bootstrap_alloc(DPCPU_SIZE, PAGE_SIZE);
dpcpu = (void *)virtual_avail;
va = virtual_avail;
virtual_avail += DPCPU_SIZE;
while (va < virtual_avail) {
Remove extraneous semicolons, no functional changes. Submitted by: Marc Balmer <marc@msys.ch> MFC after: 1 week
2010-01-07 21:01:37 +00:00
moea_kenter(mmup, va, pa);
Implement a facility for dynamic per-cpu variables. - Modules and kernel code alike may use DPCPU_DEFINE(), DPCPU_GET(), DPCPU_SET(), etc. akin to the statically defined PCPU_*. Requires only one extra instruction more than PCPU_* and is virtually the same as __thread for builtin and much faster for shared objects. DPCPU variables can be initialized when defined. - Modules are supported by relocating the module's per-cpu linker set over space reserved in the kernel. Modules may fail to load if there is insufficient space available. - Track space available for modules with a one-off extent allocator. Free may block for memory to allocate space for an extent. Reviewed by: jhb, rwatson, kan, sam, grehan, marius, marcel, stas
2009-06-23 22:42:39 +00:00
pa += PAGE_SIZE;
va += PAGE_SIZE;
}
dpcpu_init(dpcpu, 0);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
/*
* Activate a user pmap. The pmap must be activated before it's address
* space can be accessed in any way.
*/
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_activate(mmu_t mmu, struct thread *td)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
pmap_t pm, pmr;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
* Load all the data we need up front to encourage the compiler to
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* not issue any loads while we have interrupts disabled below.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pm = &td->td_proc->p_vmspace->vm_pmap;
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pmr = pm->pmap_phys;
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
Remove pc_other_cpus and pc_cpumask usage from powerpc support. Tested and reviewed by: andreast
2011-06-16 07:27:13 +00:00
CPU_SET(PCPU_GET(cpuid), &pm->pm_active);
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
PCPU_SET(curpmap, pmr);
- Modify pmap_activate so it only marks the pmap as active. - Add a pmap_deactivate function.
2002-02-28 11:55:44 +00:00
}
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_deactivate(mmu_t mmu, struct thread *td)
- Modify pmap_activate so it only marks the pmap as active. - Add a pmap_deactivate function.
2002-02-28 11:55:44 +00:00
{
pmap_t pm;
pm = &td->td_proc->p_vmspace->vm_pmap;
Remove pc_other_cpus and pc_cpumask usage from powerpc support. Tested and reviewed by: andreast
2011-06-16 07:27:13 +00:00
CPU_CLR(PCPU_GET(cpuid), &pm->pm_active);
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
PCPU_SET(curpmap, NULL);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_change_wiring(mmu_t mmu, pmap_t pm, vm_offset_t va, boolean_t wired)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Implement the following functions: - pmap_addr_hint - pmap_change_wiring - pmap_extract - pmap_is_modified
2002-05-10 14:21:48 +00:00
struct pvo_entry *pvo;
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK(pm);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pvo = moea_pvo_find_va(pm, va & ~ADDR_POFF, NULL);
Implement the following functions: - pmap_addr_hint - pmap_change_wiring - pmap_extract - pmap_is_modified
2002-05-10 14:21:48 +00:00
if (pvo != NULL) {
if (wired) {
if ((pvo->pvo_vaddr & PVO_WIRED) == 0)
pm->pm_stats.wired_count++;
pvo->pvo_vaddr |= PVO_WIRED;
} else {
if ((pvo->pvo_vaddr & PVO_WIRED) != 0)
pm->pm_stats.wired_count--;
pvo->pvo_vaddr &= ~PVO_WIRED;
}
}
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(pm);
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_copy_page(mmu_t mmu, vm_page_t msrc, vm_page_t mdst)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Implement pmap_copy and pmap_copy_page.
2002-05-28 07:38:55 +00:00
vm_offset_t dst;
vm_offset_t src;
dst = VM_PAGE_TO_PHYS(mdst);
src = VM_PAGE_TO_PHYS(msrc);
We don't need kcopy() in any of the remaining places it is used, so remove it. MFC after: 2 weeks
2012-04-11 22:23:50 +00:00
bcopy((void *)src, (void *)dst, PAGE_SIZE);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Zero a page of physical memory by temporarily mapping it into the tlb.
*/
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_zero_page(mmu_t mmu, vm_page_t m)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Pass vm_page_t instead of physical addresses to pmap_zero_page[_area]() and pmap_copy_page(). This gets rid of a couple more physical addresses in upper layers, with the eventual aim of supporting PAE and dealing with the physical addressing mostly within pmap. (We will need either 64 bit physical addresses or page indexes, possibly both depending on the circumstances. Leaving this to pmap itself gives more flexibilitly.) Reviewed by: jake Tested on: i386, ia64 and (I believe) sparc64. (my alpha was hosed)
2002-04-15 16:00:03 +00:00
vm_offset_t pa = VM_PAGE_TO_PHYS(m);
Simplify the pmap_zero_page family of functions by making use of the fact that we have a 1:1 mapping by virtue of the BATs. Eliminate the now unused moea_rkva_alloc(), moea_pa_map() and moea_pa_unmap() functions. Pointed out by: grehan.
2008-04-17 00:37:40 +00:00
void *va = (void *)pa;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
bzero(va, PAGE_SIZE);
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_zero_page_area(mmu_t mmu, vm_page_t m, int off, int size)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Add an implementation for pmap_zero_page_area.
2002-07-09 13:44:24 +00:00
vm_offset_t pa = VM_PAGE_TO_PHYS(m);
Simplify the pmap_zero_page family of functions by making use of the fact that we have a 1:1 mapping by virtue of the BATs. Eliminate the now unused moea_rkva_alloc(), moea_pa_map() and moea_pa_unmap() functions. Pointed out by: grehan.
2008-04-17 00:37:40 +00:00
void *va = (void *)(pa + off);
Add an implementation for pmap_zero_page_area.
2002-07-09 13:44:24 +00:00
Simplify the pmap_zero_page family of functions by making use of the fact that we have a 1:1 mapping by virtue of the BATs. Eliminate the now unused moea_rkva_alloc(), moea_pa_map() and moea_pa_unmap() functions. Pointed out by: grehan.
2008-04-17 00:37:40 +00:00
bzero(va, size);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Add a special page zero entry point intended to be called via the single threaded VM pagezero kthread outside of Giant. For some platforms, this is really easy since it can just use the direct mapped region. For others, IPI sending is involved or there are other issues, so grab Giant when needed. We still have preemption issues to deal with, but Alan Cox has an interesting suggestion on how to minimize the problem on x86. Use Luigi's hack for preserving the (lack of) priority. Turn the idle zeroing back on since it can now actually do something useful outside of Giant in many cases.
2002-07-08 04:24:26 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_zero_page_idle(mmu_t mmu, vm_page_t m)
Add a special page zero entry point intended to be called via the single threaded VM pagezero kthread outside of Giant. For some platforms, this is really easy since it can just use the direct mapped region. For others, IPI sending is involved or there are other issues, so grab Giant when needed. We still have preemption issues to deal with, but Alan Cox has an interesting suggestion on how to minimize the problem on x86. Use Luigi's hack for preserving the (lack of) priority. Turn the idle zeroing back on since it can now actually do something useful outside of Giant in many cases.
2002-07-08 04:24:26 +00:00
{
Simplify the pmap_zero_page family of functions by making use of the fact that we have a 1:1 mapping by virtue of the BATs. Eliminate the now unused moea_rkva_alloc(), moea_pa_map() and moea_pa_unmap() functions. Pointed out by: grehan.
2008-04-17 00:37:40 +00:00
vm_offset_t pa = VM_PAGE_TO_PHYS(m);
void *va = (void *)pa;
Add a special page zero entry point intended to be called via the single threaded VM pagezero kthread outside of Giant. For some platforms, this is really easy since it can just use the direct mapped region. For others, IPI sending is involved or there are other issues, so grab Giant when needed. We still have preemption issues to deal with, but Alan Cox has an interesting suggestion on how to minimize the problem on x86. Use Luigi's hack for preserving the (lack of) priority. Turn the idle zeroing back on since it can now actually do something useful outside of Giant in many cases.
2002-07-08 04:24:26 +00:00
Simplify the pmap_zero_page family of functions by making use of the fact that we have a 1:1 mapping by virtue of the BATs. Eliminate the now unused moea_rkva_alloc(), moea_pa_map() and moea_pa_unmap() functions. Pointed out by: grehan.
2008-04-17 00:37:40 +00:00
bzero(va, PAGE_SIZE);
Add a special page zero entry point intended to be called via the single threaded VM pagezero kthread outside of Giant. For some platforms, this is really easy since it can just use the direct mapped region. For others, IPI sending is involved or there are other issues, so grab Giant when needed. We still have preemption issues to deal with, but Alan Cox has an interesting suggestion on how to minimize the problem on x86. Use Luigi's hack for preserving the (lack of) priority. Turn the idle zeroing back on since it can now actually do something useful outside of Giant in many cases.
2002-07-08 04:24:26 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Map the given physical page at the specified virtual address in the
* target pmap with the protection requested. If specified the page
* will be wired down.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_enter(mmu_t mmu, pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
boolean_t wired)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_LOCK(pmap);
Correct a typo in the previous revision.
2006-06-06 02:02:10 +00:00
moea_enter_locked(pmap, va, m, prot, wired);
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_UNLOCK(pmap);
}
/*
* Map the given physical page at the specified virtual address in the
* target pmap with the protection requested. If specified the page
* will be wired down.
*
* The page queues and pmap must be locked.
*/
static void
moea_enter_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot,
boolean_t wired)
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pvo_head *pvo_head;
Remove references to vm_zone.h and switch over to the new uma API.
2002-03-21 01:11:31 +00:00
uma_zone_t zone;
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
vm_page_t pg;
Reduce the frequency that the PowerPC/AIM pmaps invalidate instruction caches, by invalidating kernel icaches only when needed and not flushing user caches for shared pages. Suggested by: kib MFC after: 2 weeks
2012-04-06 16:03:38 +00:00
u_int pte_lo, pvo_flags;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
int error;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (!moea_initialized) {
pvo_head = &moea_pvo_kunmanaged;
zone = moea_upvo_zone;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo_flags = 0;
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
pg = NULL;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
} else {
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
pvo_head = vm_page_to_pvoh(m);
pg = m;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
zone = moea_mpvo_zone;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo_flags = PVO_MANAGED;
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
}
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
if (pmap_bootstrapped)
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_assert(&pvh_global_lock, RA_WLOCKED);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & (VPO_UNMANAGED | VPO_BUSY)) != 0 ||
VM_OBJECT_LOCKED(m->object),
On entry to pmap_enter(), assert that the page is busy. While I'm here, make the style of assertion used by pmap_enter() consistent across all architectures. On entry to pmap_remove_write(), assert that the page is neither unmanaged nor fictitious, since we cannot remove write access to either kind of page. With the push down of the page queues lock, pmap_remove_write() cannot condition its behavior on the state of the PG_WRITEABLE flag if the page is busy. Assert that the object containing the page is locked. This allows us to know that the page will neither become busy nor will PG_WRITEABLE be set on it while pmap_remove_write() is running. Correct a long-standing bug in vm_page_cowsetup(). We cannot possibly do copy-on-write-based zero-copy transmit on unmanaged or fictitious pages, so don't even try. Previously, the call to pmap_remove_write() would have failed silently.
2010-05-16 23:45:10 +00:00
("moea_enter_locked: page %p is not busy", m));
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
/* XXX change the pvo head for fake pages */
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
if ((m->oflags & VPO_UNMANAGED) != 0) {
Change the PVO zone for fictitious pages to the unmanaged PVO zone, to match the unmanaged flag set in the PVO attributes. Without doing this, pmap_remove() could try to remove fictitious pages (like those created by mmap of physical memory) from the wrong UMA zone, causing a panic. Reported by: Justin Hibbits MFC after: 1 week
2009-03-11 03:19:19 +00:00
pvo_flags &= ~PVO_MANAGED;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pvo_head = &moea_pvo_kunmanaged;
Change the PVO zone for fictitious pages to the unmanaged PVO zone, to match the unmanaged flag set in the PVO attributes. Without doing this, pmap_remove() could try to remove fictitious pages (like those created by mmap of physical memory) from the wrong UMA zone, causing a panic. Reported by: Justin Hibbits MFC after: 1 week
2009-03-11 03:19:19 +00:00
zone = moea_upvo_zone;
}
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
Fix pmap_page_set_memattr() behavior in the presence of fictitious pages by just caching the mode for later use by pmap_enter(), following amd64. While here, correct some mismerges from mmu_oea64 -> mmu_oea and clean up some dead code found while fixing the fictitious page behavior.
2010-10-01 18:59:30 +00:00
pte_lo = moea_calc_wimg(VM_PAGE_TO_PHYS(m), pmap_page_get_memattr(m));
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Make pmap_enter() responsible for setting PG_WRITEABLE instead of its caller. (As a beneficial side-effect, a high-contention acquisition of the page queues lock in vm_fault() is eliminated.)
2006-11-12 21:48:34 +00:00
if (prot & VM_PROT_WRITE) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pte_lo |= PTE_BW;
Don't set PG_WRITEABLE in pmap_enter() unless the page is managed.
2010-06-05 06:56:06 +00:00
if (pmap_bootstrapped &&
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
(m->oflags & VPO_UNMANAGED) == 0)
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
vm_page_aflag_set(m, PGA_WRITEABLE);
Make pmap_enter() responsible for setting PG_WRITEABLE instead of its caller. (As a beneficial side-effect, a high-contention acquisition of the page queues lock in vm_fault() is eliminated.)
2006-11-12 21:48:34 +00:00
} else
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pte_lo |= PTE_BR;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
if (prot & VM_PROT_EXECUTE)
pvo_flags |= PVO_EXECUTABLE;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (wired)
pvo_flags |= PVO_WIRED;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
error = moea_pvo_enter(pmap, zone, pvo_head, va, VM_PAGE_TO_PHYS(m),
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
pte_lo, pvo_flags);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
/*
Reduce the frequency that the PowerPC/AIM pmaps invalidate instruction caches, by invalidating kernel icaches only when needed and not flushing user caches for shared pages. Suggested by: kib MFC after: 2 weeks
2012-04-06 16:03:38 +00:00
* Flush the real page from the instruction cache. This has be done
* for all user mappings to prevent information leakage via the
Fix error in r233949. Synchronizing icaches on uncacheable pages turns out not to be a good idea, and of course the PV entry list for a page is never empty after the page has been mapped.
2012-04-11 20:28:05 +00:00
* instruction cache. moea_pvo_enter() returns ENOENT for the first
* mapping for a page.
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
*/
Fix error in r233949. Synchronizing icaches on uncacheable pages turns out not to be a good idea, and of course the PV entry list for a page is never empty after the page has been mapped.
2012-04-11 20:28:05 +00:00
if (pmap != kernel_pmap && error == ENOENT &&
(pte_lo & (PTE_I | PTE_G)) == 0)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_syncicache(VM_PAGE_TO_PHYS(m), PAGE_SIZE);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
}
/*
* Maps a sequence of resident pages belonging to the same object.
* The sequence begins with the given page m_start. This page is
* mapped at the given virtual address start. Each subsequent page is
* mapped at a virtual address that is offset from start by the same
* amount as the page is offset from m_start within the object. The
* last page in the sequence is the page with the largest offset from
* m_start that can be mapped at a virtual address less than the given
* virtual address end. Not every virtual page between start and end
* is mapped; only those for which a resident page exists with the
* corresponding offset from m_start are mapped.
*/
void
moea_enter_object(mmu_t mmu, pmap_t pm, vm_offset_t start, vm_offset_t end,
vm_page_t m_start, vm_prot_t prot)
{
vm_page_t m;
vm_pindex_t diff, psize;
psize = atop(end - start);
m = m_start;
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_LOCK(pm);
while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) {
moea_enter_locked(pm, start + ptoa(diff), m, prot &
(VM_PROT_READ | VM_PROT_EXECUTE), FALSE);
m = TAILQ_NEXT(m, listq);
}
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_UNLOCK(pm);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Remove mpte optimization from pmap_enter_quick(). There is a race with the current locking scheme and removing it should have no measurable performance impact. This fixes page faults leading to panics in pmap_enter_quick_locked() on amd64/i386. Reviewed by: alc,jhb,peter,ps
2006-06-15 01:01:06 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_enter_quick(mmu_t mmu, pmap_t pm, vm_offset_t va, vm_page_t m,
Remove mpte optimization from pmap_enter_quick(). There is a race with the current locking scheme and removing it should have no measurable performance impact. This fixes page faults leading to panics in pmap_enter_quick_locked() on amd64/i386. Reviewed by: alc,jhb,peter,ps
2006-06-15 01:01:06 +00:00
vm_prot_t prot)
- Export pmap_enter_quick() to the MI VM. This will permit the implementation of a largely MI pmap_object_init_pt() for vnode-backed objects. pmap_enter_quick() is implemented via pmap_enter() on sparc64 and powerpc. - Correct a mismatch between pmap_object_init_pt()'s prototype and its various implementations. (I plan to keep pmap_object_init_pt() as the MD hook for device-backed objects on i386 and amd64.) - Correct an error in ia64's pmap_enter_quick() and adjust its interface to match the other versions. Discussed with: marcel
2003-06-29 21:20:04 +00:00
{
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_LOCK(pm);
moea_enter_locked(pm, va, m, prot & (VM_PROT_READ | VM_PROT_EXECUTE),
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
FALSE);
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Introduce the function pmap_enter_object(). It maps a sequence of resident pages from the same object. Use it in vm_map_pmap_enter() to reduce the locking overhead of premapping objects. Reviewed by: tegge@
2006-06-05 20:35:27 +00:00
PMAP_UNLOCK(pm);
- Export pmap_enter_quick() to the MI VM. This will permit the implementation of a largely MI pmap_object_init_pt() for vnode-backed objects. pmap_enter_quick() is implemented via pmap_enter() on sparc64 and powerpc. - Correct a mismatch between pmap_object_init_pt()'s prototype and its various implementations. (I plan to keep pmap_object_init_pt() as the MD hook for device-backed objects on i386 and amd64.) - Correct an error in ia64's pmap_enter_quick() and adjust its interface to match the other versions. Discussed with: marcel
2003-06-29 21:20:04 +00:00
}
Correct pmap_extract()'s return type. It should be vm_paddr_t, not vm_offset_t.
2004-07-05 23:08:27 +00:00
vm_paddr_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_extract(mmu_t mmu, pmap_t pm, vm_offset_t va)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Implement the following functions: - pmap_addr_hint - pmap_change_wiring - pmap_extract - pmap_is_modified
2002-05-10 14:21:48 +00:00
struct pvo_entry *pvo;
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
vm_paddr_t pa;
Implement the following functions: - pmap_addr_hint - pmap_change_wiring - pmap_extract - pmap_is_modified
2002-05-10 14:21:48 +00:00
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK(pm);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pvo = moea_pvo_find_va(pm, va & ~ADDR_POFF, NULL);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
if (pvo == NULL)
pa = 0;
else
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pa = (pvo->pvo_pte.pte.pte_lo & PTE_RPGN) | (va & ADDR_POFF);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(pm);
return (pa);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Soften assert in pmap_remove_all. Introduct pmap_extract_and_hold. Stolen from: sparc64
2003-09-22 11:59:05 +00:00
/*
* Atomically extract and hold the physical page with the given
* pmap and virtual address pair if that mapping permits the given
* protection.
*/
vm_page_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_extract_and_hold(mmu_t mmu, pmap_t pmap, vm_offset_t va, vm_prot_t prot)
Soften assert in pmap_remove_all. Introduct pmap_extract_and_hold. Stolen from: sparc64
2003-09-22 11:59:05 +00:00
{
Implement the protection check required by the pmap_extract_and_hold() specification. Reviewed and tested by: grehan@
2004-07-26 18:10:10 +00:00
struct pvo_entry *pvo;
Soften assert in pmap_remove_all. Introduct pmap_extract_and_hold. Stolen from: sparc64
2003-09-22 11:59:05 +00:00
vm_page_t m;
On Alan's advice, rather than do a wholesale conversion on a single architecture from page queue lock to a hashed array of page locks (based on a patch by Jeff Roberson), I've implemented page lock support in the MI code and have only moved vm_page's hold_count out from under page queue mutex to page lock. This changes pmap_extract_and_hold on all pmaps. Supported by: Bitgravity Inc. Discussed with: alc, jeffr, and kib
2010-04-30 00:46:43 +00:00
vm_paddr_t pa;
Soften assert in pmap_remove_all. Introduct pmap_extract_and_hold. Stolen from: sparc64
2003-09-22 11:59:05 +00:00
m = NULL;
On Alan's advice, rather than do a wholesale conversion on a single architecture from page queue lock to a hashed array of page locks (based on a patch by Jeff Roberson), I've implemented page lock support in the MI code and have only moved vm_page's hold_count out from under page queue mutex to page lock. This changes pmap_extract_and_hold on all pmaps. Supported by: Bitgravity Inc. Discussed with: alc, jeffr, and kib
2010-04-30 00:46:43 +00:00
pa = 0;
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK(pmap);
On Alan's advice, rather than do a wholesale conversion on a single architecture from page queue lock to a hashed array of page locks (based on a patch by Jeff Roberson), I've implemented page lock support in the MI code and have only moved vm_page's hold_count out from under page queue mutex to page lock. This changes pmap_extract_and_hold on all pmaps. Supported by: Bitgravity Inc. Discussed with: alc, jeffr, and kib
2010-04-30 00:46:43 +00:00
retry:
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pvo = moea_pvo_find_va(pmap, va & ~ADDR_POFF, NULL);
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo != NULL && (pvo->pvo_pte.pte.pte_hi & PTE_VALID) &&
((pvo->pvo_pte.pte.pte_lo & PTE_PP) == PTE_RW ||
Implement the protection check required by the pmap_extract_and_hold() specification. Reviewed and tested by: grehan@
2004-07-26 18:10:10 +00:00
(prot & VM_PROT_WRITE) == 0)) {
On Alan's advice, rather than do a wholesale conversion on a single architecture from page queue lock to a hashed array of page locks (based on a patch by Jeff Roberson), I've implemented page lock support in the MI code and have only moved vm_page's hold_count out from under page queue mutex to page lock. This changes pmap_extract_and_hold on all pmaps. Supported by: Bitgravity Inc. Discussed with: alc, jeffr, and kib
2010-04-30 00:46:43 +00:00
if (vm_page_pa_tryrelock(pmap, pvo->pvo_pte.pte.pte_lo & PTE_RPGN, &pa))
goto retry;
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
m = PHYS_TO_VM_PAGE(pvo->pvo_pte.pte.pte_lo & PTE_RPGN);
Soften assert in pmap_remove_all. Introduct pmap_extract_and_hold. Stolen from: sparc64
2003-09-22 11:59:05 +00:00
vm_page_hold(m);
}
On Alan's advice, rather than do a wholesale conversion on a single architecture from page queue lock to a hashed array of page locks (based on a patch by Jeff Roberson), I've implemented page lock support in the MI code and have only moved vm_page's hold_count out from under page queue mutex to page lock. This changes pmap_extract_and_hold on all pmaps. Supported by: Bitgravity Inc. Discussed with: alc, jeffr, and kib
2010-04-30 00:46:43 +00:00
PA_UNLOCK_COND(pa);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(pmap);
Soften assert in pmap_remove_all. Introduct pmap_extract_and_hold. Stolen from: sparc64
2003-09-22 11:59:05 +00:00
return (m);
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_init(mmu_t mmu)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_upvo_zone = uma_zcreate("UPVO entry", sizeof (struct pvo_entry),
Remove pmap_pvo_allocf zone alloc function. It was a way of using the direct-mapping of physmem to force PTE data structures to be physically addressable so the interrupt-time real-mode DSI trap handler could perform PTE spills. However, the memory may have been > 256Mb, which would have caused a BAT spill and double-interrupt. The new trap code no longer handles PTE spills, so the requirement that these pages be direct-mapped no longer applies. The irony is UMA_MD_SMALL_ALLOC will return direct mappings for these structs :-)
2004-02-04 13:16:21 +00:00
NULL, NULL, NULL, NULL, UMA_ALIGN_PTR,
UMA_ZONE_VM | UMA_ZONE_NOFREE);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_mpvo_zone = uma_zcreate("MPVO entry", sizeof(struct pvo_entry),
Remove pmap_pvo_allocf zone alloc function. It was a way of using the direct-mapping of physmem to force PTE data structures to be physically addressable so the interrupt-time real-mode DSI trap handler could perform PTE spills. However, the memory may have been > 256Mb, which would have caused a BAT spill and double-interrupt. The new trap code no longer handles PTE spills, so the requirement that these pages be direct-mapped no longer applies. The irony is UMA_MD_SMALL_ALLOC will return direct mappings for these structs :-)
2004-02-04 13:16:21 +00:00
NULL, NULL, NULL, NULL, UMA_ALIGN_PTR,
UMA_ZONE_VM | UMA_ZONE_NOFREE);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_initialized = TRUE;
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
}
Resurrect pmap_is_referenced() and use it in mincore(). Essentially, pmap_ts_referenced() is not always appropriate for checking whether or not pages have been referenced because it clears any reference bits that it encounters. For example, in mincore(), clearing the reference bits has two negative consequences. First, it throws off the activity count calculations performed by the page daemon. Specifically, a page on which mincore() has called pmap_ts_referenced() looks less active to the page daemon than it should. Consequently, the page could be deactivated prematurely by the page daemon. Arguably, this problem could be fixed by having mincore() duplicate the activity count calculation on the page. However, there is a second problem for which that is not a solution. In order to clear a reference on a 4KB page, it may be necessary to demote a 2/4MB page mapping. Thus, a mincore() by one process can have the side effect of demoting a superpage mapping within another process!
2010-04-24 17:32:52 +00:00
boolean_t
moea_is_referenced(mmu_t mmu, vm_page_t m)
{
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
boolean_t rv;
Resurrect pmap_is_referenced() and use it in mincore(). Essentially, pmap_ts_referenced() is not always appropriate for checking whether or not pages have been referenced because it clears any reference bits that it encounters. For example, in mincore(), clearing the reference bits has two negative consequences. First, it throws off the activity count calculations performed by the page daemon. Specifically, a page on which mincore() has called pmap_ts_referenced() looks less active to the page daemon than it should. Consequently, the page could be deactivated prematurely by the page daemon. Arguably, this problem could be fixed by having mincore() duplicate the activity count calculation on the page. However, there is a second problem for which that is not a solution. In order to clear a reference on a 4KB page, it may be necessary to demote a 2/4MB page mapping. Thus, a mincore() by one process can have the side effect of demoting a superpage mapping within another process!
2010-04-24 17:32:52 +00:00
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
Push down page queues lock acquisition in pmap_enter_object() and pmap_is_referenced(). Eliminate the corresponding page queues lock acquisitions from vm_map_pmap_enter() and mincore(), respectively. In mincore(), this allows some additional cases to complete without ever acquiring the page queues lock. Assert that the page is managed in pmap_is_referenced(). On powerpc/aim, push down the page queues lock acquisition from moea*_is_modified() and moea*_is_referenced() into moea*_query_bit(). Again, this will allow some additional cases to complete without ever acquiring the page queues lock. Reorder a few statements in vm_page_dontneed() so that a race can't lead to an old reference persisting. This scenario is described in detail by a comment. Correct a spelling error in vm_page_dontneed(). Assert that the object is locked in vm_page_clear_dirty(), and restrict the page queues lock assertion to just those cases in which the page is currently writeable. Add object locking to vnode_pager_generic_putpages(). This was the one and only place where vm_page_clear_dirty() was being called without the object being locked. Eliminate an unnecessary vm_page_lock() around vnode_pager_setsize()'s call to vm_page_clear_dirty(). Change vnode_pager_generic_putpages() to the modern-style of function definition. Also, change the name of one of the parameters to follow virtual memory system naming conventions. Reviewed by: kib
2010-05-26 18:00:44 +00:00
("moea_is_referenced: page %p is not managed", m));
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wlock(&pvh_global_lock);
rv = moea_query_bit(m, PTE_REF);
rw_wunlock(&pvh_global_lock);
return (rv);
Resurrect pmap_is_referenced() and use it in mincore(). Essentially, pmap_ts_referenced() is not always appropriate for checking whether or not pages have been referenced because it clears any reference bits that it encounters. For example, in mincore(), clearing the reference bits has two negative consequences. First, it throws off the activity count calculations performed by the page daemon. Specifically, a page on which mincore() has called pmap_ts_referenced() looks less active to the page daemon than it should. Consequently, the page could be deactivated prematurely by the page daemon. Arguably, this problem could be fixed by having mincore() duplicate the activity count calculation on the page. However, there is a second problem for which that is not a solution. In order to clear a reference on a 4KB page, it may be necessary to demote a 2/4MB page mapping. Thus, a mincore() by one process can have the side effect of demoting a superpage mapping within another process!
2010-04-24 17:32:52 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
boolean_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_is_modified(mmu_t mmu, vm_page_t m)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
boolean_t rv;
Implement the following functions: - pmap_addr_hint - pmap_change_wiring - pmap_extract - pmap_is_modified
2002-05-10 14:21:48 +00:00
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
("moea_is_modified: page %p is not managed", m));
Implement the following functions: - pmap_addr_hint - pmap_change_wiring - pmap_extract - pmap_is_modified
2002-05-10 14:21:48 +00:00
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
/*
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
* If the page is not VPO_BUSY, then PGA_WRITEABLE cannot be
* concurrently set while the object is locked. Thus, if PGA_WRITEABLE
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
* is clear, no PTEs can have PTE_CHG set.
*/
VM_OBJECT_LOCK_ASSERT(m->object, MA_OWNED);
if ((m->oflags & VPO_BUSY) == 0 &&
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
(m->aflags & PGA_WRITEABLE) == 0)
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
return (FALSE);
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wlock(&pvh_global_lock);
rv = moea_query_bit(m, PTE_CHG);
rw_wunlock(&pvh_global_lock);
return (rv);
Migrate pmap_prefault() into the machine-independent virtual memory layer. A small helper function pmap_is_prefaultable() is added. This function encapsulate the few lines of pmap_prefault() that actually vary from machine to machine. Note: pmap_is_prefaultable() and pmap_mincore() have much in common. Going forward, it's worth considering their merger.
2003-10-03 22:46:53 +00:00
}
Implement pmap_is_prefaultable(). Reviewed by: nwhitehorn
2010-11-01 02:22:48 +00:00
boolean_t
moea_is_prefaultable(mmu_t mmu, pmap_t pmap, vm_offset_t va)
{
struct pvo_entry *pvo;
boolean_t rv;
PMAP_LOCK(pmap);
pvo = moea_pvo_find_va(pmap, va & ~ADDR_POFF, NULL);
rv = pvo == NULL || (pvo->pvo_pte.pte.pte_hi & PTE_VALID) == 0;
PMAP_UNLOCK(pmap);
return (rv);
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_clear_reference(mmu_t mmu, vm_page_t m)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
("moea_clear_reference: page %p is not managed", m));
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wlock(&pvh_global_lock);
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
moea_clear_bit(m, PTE_REF);
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wunlock(&pvh_global_lock);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
}
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_clear_modify(mmu_t mmu, vm_page_t m)
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
{
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
("moea_clear_modify: page %p is not managed", m));
VM_OBJECT_LOCK_ASSERT(m->object, MA_OWNED);
KASSERT((m->oflags & VPO_BUSY) == 0,
("moea_clear_modify: page %p is busy", m));
/*
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
* If the page is not PGA_WRITEABLE, then no PTEs can have PTE_CHG
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
* set. If the object containing the page is locked and the page is
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
* not VPO_BUSY, then PGA_WRITEABLE cannot be concurrently set.
Roughly half of a typical pmap_mincore() implementation is machine- independent code. Move this code into mincore(), and eliminate the page queues lock from pmap_mincore(). Push down the page queues lock into pmap_clear_modify(), pmap_clear_reference(), and pmap_is_modified(). Assert that these functions are never passed an unmanaged page. Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m: Contrary to what the comment says, pmap_mincore() is not simply an optimization. Without a complete pmap_mincore() implementation, mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED because only the pmap can provide this information. Eliminate the page queues lock from vfs_setdirty_locked_object(), vm_pageout_clean(), vm_object_page_collect_flush(), and vm_object_page_clean(). Generally speaking, these are all accesses to the page's dirty field, which are synchronized by the containing vm object's lock. Reduce the scope of the page queues lock in vm_object_madvise() and vm_page_dontneed(). Reviewed by: kib (an earlier version)
2010-05-24 14:26:57 +00:00
*/
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
if ((m->aflags & PGA_WRITEABLE) == 0)
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
return;
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wlock(&pvh_global_lock);
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
moea_clear_bit(m, PTE_CHG);
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wunlock(&pvh_global_lock);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
/*
* Clear the write and modified bits in each of the given page's mappings.
*/
void
moea_remove_write(mmu_t mmu, vm_page_t m)
{
struct pvo_entry *pvo;
struct pte *pt;
pmap_t pmap;
u_int lo;
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
On entry to pmap_enter(), assert that the page is busy. While I'm here, make the style of assertion used by pmap_enter() consistent across all architectures. On entry to pmap_remove_write(), assert that the page is neither unmanaged nor fictitious, since we cannot remove write access to either kind of page. With the push down of the page queues lock, pmap_remove_write() cannot condition its behavior on the state of the PG_WRITEABLE flag if the page is busy. Assert that the object containing the page is locked. This allows us to know that the page will neither become busy nor will PG_WRITEABLE be set on it while pmap_remove_write() is running. Correct a long-standing bug in vm_page_cowsetup(). We cannot possibly do copy-on-write-based zero-copy transmit on unmanaged or fictitious pages, so don't even try. Previously, the call to pmap_remove_write() would have failed silently.
2010-05-16 23:45:10 +00:00
("moea_remove_write: page %p is not managed", m));
/*
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
* If the page is not VPO_BUSY, then PGA_WRITEABLE cannot be set by
* another thread while the object is locked. Thus, if PGA_WRITEABLE
On entry to pmap_enter(), assert that the page is busy. While I'm here, make the style of assertion used by pmap_enter() consistent across all architectures. On entry to pmap_remove_write(), assert that the page is neither unmanaged nor fictitious, since we cannot remove write access to either kind of page. With the push down of the page queues lock, pmap_remove_write() cannot condition its behavior on the state of the PG_WRITEABLE flag if the page is busy. Assert that the object containing the page is locked. This allows us to know that the page will neither become busy nor will PG_WRITEABLE be set on it while pmap_remove_write() is running. Correct a long-standing bug in vm_page_cowsetup(). We cannot possibly do copy-on-write-based zero-copy transmit on unmanaged or fictitious pages, so don't even try. Previously, the call to pmap_remove_write() would have failed silently.
2010-05-16 23:45:10 +00:00
* is clear, no page table entries need updating.
*/
VM_OBJECT_LOCK_ASSERT(m->object, MA_OWNED);
if ((m->oflags & VPO_BUSY) == 0 &&
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
(m->aflags & PGA_WRITEABLE) == 0)
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
return;
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
lo = moea_attr_fetch(m);
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
LIST_FOREACH(pvo, vm_page_to_pvoh(m), pvo_vlink) {
pmap = pvo->pvo_pmap;
PMAP_LOCK(pmap);
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if ((pvo->pvo_pte.pte.pte_lo & PTE_PP) != PTE_BR) {
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
pt = moea_pvo_to_pte(pvo, -1);
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pvo->pvo_pte.pte.pte_lo &= ~PTE_PP;
pvo->pvo_pte.pte.pte_lo |= PTE_BR;
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
if (pt != NULL) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_synch(pt, &pvo->pvo_pte.pte);
lo |= pvo->pvo_pte.pte.pte_lo;
pvo->pvo_pte.pte.pte_lo &= ~PTE_CHG;
moea_pte_change(pt, &pvo->pvo_pte.pte,
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
pvo->pvo_vaddr);
mtx_unlock(&moea_table_mutex);
}
}
PMAP_UNLOCK(pmap);
}
if ((lo & PTE_CHG) != 0) {
moea_attr_clear(m, PTE_CHG);
vm_page_dirty(m);
}
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
vm_page_aflag_clear(m, PGA_WRITEABLE);
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Complete the transition from pmap_page_protect() to pmap_remove_write(). Originally, I had adopted sparc64's name, pmap_clear_write(), for the function that is now pmap_remove_write(). However, this function is more like pmap_remove_all() than like pmap_clear_modify() or pmap_clear_reference(), hence, the name change. The higher-level rationale behind this change is described in src/sys/amd64/amd64/pmap.c revision 1.567. The short version is that I'm trying to clean up and fix our support for execute access. Reviewed by: marcel@ (ia64)
2006-08-01 19:06:06 +00:00
}
Fix a horribly suboptimal algorithm in the vm_daemon. In order to determine what to page out, the vm_daemon checks reference bits on all pages belonging to all processes. Unfortunately, the algorithm used reacted badly with shared pages; each shared page would be checked once per process sharing it; this caused an O(N^2) growth of tlb invalidations. The algorithm has been changed so that each page will be checked only 16 times. Prior to this change, a fork/sleepbomb of 1300 processes could cause the vm_daemon to take over 60 seconds to complete, effectively freezing the system for that time period. With this change in place, the vm_daemon completes in less than a second. Any system with hundreds of processes sharing pages should benefit from this change. Note that the vm_daemon is only run when the system is under extreme memory pressure. It is likely that many people with loaded systems saw no symptoms of this problem until they reached the point where swapping began. Special thanks go to dillon, peter, and Chuck Cranor, who helped me get up to speed with vm internals. PR: 33542, 20393 Reviewed by: dillon MFC after: 1 week
2002-02-27 18:03:02 +00:00
/*
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
* moea_ts_referenced:
Fix a horribly suboptimal algorithm in the vm_daemon. In order to determine what to page out, the vm_daemon checks reference bits on all pages belonging to all processes. Unfortunately, the algorithm used reacted badly with shared pages; each shared page would be checked once per process sharing it; this caused an O(N^2) growth of tlb invalidations. The algorithm has been changed so that each page will be checked only 16 times. Prior to this change, a fork/sleepbomb of 1300 processes could cause the vm_daemon to take over 60 seconds to complete, effectively freezing the system for that time period. With this change in place, the vm_daemon completes in less than a second. Any system with hundreds of processes sharing pages should benefit from this change. Note that the vm_daemon is only run when the system is under extreme memory pressure. It is likely that many people with loaded systems saw no symptoms of this problem until they reached the point where swapping began. Special thanks go to dillon, peter, and Chuck Cranor, who helped me get up to speed with vm internals. PR: 33542, 20393 Reviewed by: dillon MFC after: 1 week
2002-02-27 18:03:02 +00:00
*
* Return a count of reference bits for a page, clearing those bits.
* It is not necessary for every reference bit to be cleared, but it
* is necessary that 0 only be returned when there are truly no
* reference bits set.
*
* XXX: The exact number of bits to check and clear is a matter that
* should be tested and standardized at some point in the future for
* optimal aging of shared pages.
*/
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_ts_referenced(mmu_t mmu, vm_page_t m)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
int count;
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
("moea_ts_referenced: page %p is not managed", m));
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_wlock(&pvh_global_lock);
count = moea_clear_bit(m, PTE_REF);
rw_wunlock(&pvh_global_lock);
return (count);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
/*
* Modify the WIMG settings of all mappings for a page.
*/
void
moea_page_set_memattr(mmu_t mmu, vm_page_t m, vm_memattr_t ma)
{
struct pvo_entry *pvo;
Fix pmap_page_set_memattr() behavior in the presence of fictitious pages by just caching the mode for later use by pmap_enter(), following amd64. While here, correct some mismerges from mmu_oea64 -> mmu_oea and clean up some dead code found while fixing the fictitious page behavior.
2010-10-01 18:59:30 +00:00
struct pvo_head *pvo_head;
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
struct pte *pt;
pmap_t pmap;
u_int lo;
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
if ((m->oflags & VPO_UNMANAGED) != 0) {
Fix pmap_page_set_memattr() behavior in the presence of fictitious pages by just caching the mode for later use by pmap_enter(), following amd64. While here, correct some mismerges from mmu_oea64 -> mmu_oea and clean up some dead code found while fixing the fictitious page behavior.
2010-10-01 18:59:30 +00:00
m->md.mdpg_cache_attrs = ma;
return;
}
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Fix pmap_page_set_memattr() behavior in the presence of fictitious pages by just caching the mode for later use by pmap_enter(), following amd64. While here, correct some mismerges from mmu_oea64 -> mmu_oea and clean up some dead code found while fixing the fictitious page behavior.
2010-10-01 18:59:30 +00:00
pvo_head = vm_page_to_pvoh(m);
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
lo = moea_calc_wimg(VM_PAGE_TO_PHYS(m), ma);
Fix pmap_page_set_memattr() behavior in the presence of fictitious pages by just caching the mode for later use by pmap_enter(), following amd64. While here, correct some mismerges from mmu_oea64 -> mmu_oea and clean up some dead code found while fixing the fictitious page behavior.
2010-10-01 18:59:30 +00:00
LIST_FOREACH(pvo, pvo_head, pvo_vlink) {
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
pmap = pvo->pvo_pmap;
PMAP_LOCK(pmap);
pt = moea_pvo_to_pte(pvo, -1);
pvo->pvo_pte.pte.pte_lo &= ~PTE_WIMG;
pvo->pvo_pte.pte.pte_lo |= lo;
if (pt != NULL) {
moea_pte_change(pt, &pvo->pvo_pte.pte,
pvo->pvo_vaddr);
if (pvo->pvo_pmap == kernel_pmap)
isync();
}
mtx_unlock(&moea_table_mutex);
PMAP_UNLOCK(pmap);
}
m->md.mdpg_cache_attrs = ma;
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Map a wired page into kernel virtual address space.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
void
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
moea_kenter(mmu_t mmu, vm_offset_t va, vm_paddr_t pa)
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
{
moea_kenter_attr(mmu, va, pa, VM_MEMATTR_DEFAULT);
}
void
moea_kenter_attr(mmu_t mmu, vm_offset_t va, vm_offset_t pa, vm_memattr_t ma)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
u_int pte_lo;
int error;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
#if 0
if (va < VM_MIN_KERNEL_ADDRESS)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_kenter: attempt to enter non-kernel address %#x",
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
va);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
#endif
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
pte_lo = moea_calc_wimg(pa, ma);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Lock the kernel pmap in pmap_kenter(). Tested by: gallatin@
2004-09-13 20:36:01 +00:00
PMAP_LOCK(kernel_pmap);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
error = moea_pvo_enter(kernel_pmap, moea_upvo_zone,
&moea_pvo_kunmanaged, va, pa, pte_lo, PVO_WIRED);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (error != 0 && error != ENOENT)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_kenter: failed to enter va %#x pa %#x: %d", va,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pa, error);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Lock the kernel pmap in pmap_kenter(). Tested by: gallatin@
2004-09-13 20:36:01 +00:00
PMAP_UNLOCK(kernel_pmap);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
/*
* Extract the physical page address associated with the given kernel virtual
* address.
*/
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
vm_paddr_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_kextract(mmu_t mmu, vm_offset_t va)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
struct pvo_entry *pvo;
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
vm_paddr_t pa;
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
When UMA_MD_SMALL_ALLOC is defined, pmap_kextract will be called for direct-mapped addresses. Assume that any address less than KVA is one of these and return it. Also assert that an address is KVA does have a valid mapping - callers of pmap_kextract don't check the return value, since they assume that they have a valid virtual address.
2004-01-29 00:45:41 +00:00
/*
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
* Allow direct mappings on 32-bit OEA
When UMA_MD_SMALL_ALLOC is defined, pmap_kextract will be called for direct-mapped addresses. Assume that any address less than KVA is one of these and return it. Also assert that an address is KVA does have a valid mapping - callers of pmap_kextract don't check the return value, since they assume that they have a valid virtual address.
2004-01-29 00:45:41 +00:00
*/
if (va < VM_MIN_KERNEL_ADDRESS) {
return (va);
}
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK(kernel_pmap);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pvo = moea_pvo_find_va(kernel_pmap, va & ~ADDR_POFF, NULL);
KASSERT(pvo != NULL, ("moea_kextract: no addr found"));
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pa = (pvo->pvo_pte.pte.pte_lo & PTE_RPGN) | (va & ADDR_POFF);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(kernel_pmap);
return (pa);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
/*
* Remove a wired page from kernel virtual address space.
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_kremove(mmu_t mmu, vm_offset_t va)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_remove(mmu, kernel_pmap, va, va + PAGE_SIZE);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Map a range of physical addresses into kernel virtual address space.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* The value passed in *virt is a suggested virtual address for the mapping.
* Architectures which can support a direct-mapped physical to virtual region
* can return the appropriate address within that region, leaving '*virt'
* unchanged. We cannot and therefore do not; *virt is updated with the
* first usable address after the mapped region.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
vm_offset_t
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
moea_map(mmu_t mmu, vm_offset_t *virt, vm_paddr_t pa_start,
vm_paddr_t pa_end, int prot)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
vm_offset_t sva, va;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
sva = *virt;
va = sva;
for (; pa_start < pa_end; pa_start += PAGE_SIZE, va += PAGE_SIZE)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_kenter(mmu, va, pa_start);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
*virt = va;
return (sva);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Fix a horribly suboptimal algorithm in the vm_daemon. In order to determine what to page out, the vm_daemon checks reference bits on all pages belonging to all processes. Unfortunately, the algorithm used reacted badly with shared pages; each shared page would be checked once per process sharing it; this caused an O(N^2) growth of tlb invalidations. The algorithm has been changed so that each page will be checked only 16 times. Prior to this change, a fork/sleepbomb of 1300 processes could cause the vm_daemon to take over 60 seconds to complete, effectively freezing the system for that time period. With this change in place, the vm_daemon completes in less than a second. Any system with hundreds of processes sharing pages should benefit from this change. Note that the vm_daemon is only run when the system is under extreme memory pressure. It is likely that many people with loaded systems saw no symptoms of this problem until they reached the point where swapping began. Special thanks go to dillon, peter, and Chuck Cranor, who helped me get up to speed with vm internals. PR: 33542, 20393 Reviewed by: dillon MFC after: 1 week
2002-02-27 18:03:02 +00:00
/*
* Returns true if the pmap's pv is one of the first
* 16 pvs linked to from this page. This count may
* be changed upwards or downwards in the future; it
* is only necessary that true be returned for a small
* subset of pmaps for proper page aging.
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
boolean_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_page_exists_quick(mmu_t mmu, pmap_t pmap, vm_page_t m)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
int loops;
struct pvo_entry *pvo;
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
boolean_t rv;
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
("moea_page_exists_quick: page %p is not managed", m));
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
loops = 0;
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
rv = FALSE;
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
LIST_FOREACH(pvo, vm_page_to_pvoh(m), pvo_vlink) {
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
if (pvo->pvo_pmap == pmap) {
rv = TRUE;
break;
}
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
if (++loops >= 16)
break;
}
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
return (rv);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Prevent the leakage of wired pages in the following circumstances: First, a file is mmap(2)ed and then mlock(2)ed. Later, it is truncated. Under "normal" circumstances, i.e., when the file is not mlock(2)ed, the pages beyond the EOF are unmapped and freed. However, when the file is mlock(2)ed, the pages beyond the EOF are unmapped but not freed because they have a non-zero wire count. This can be a mistake. Specifically, it is a mistake if the sole reason why the pages are wired is because of wired, managed mappings. Previously, unmapping the pages destroys these wired, managed mappings, but does not reduce the pages' wire count. Consequently, when the file is unmapped, the pages are not unwired because the wired mapping has been destroyed. Moreover, when the vm object is finally destroyed, the pages are leaked because they are still wired. The fix is to reduce the pages' wired count by the number of wired, managed mappings destroyed. To do this, I introduce a new pmap function pmap_page_wired_mappings() that returns the number of managed mappings to the given physical page that are wired, and I use this function in vm_object_page_remove(). Reviewed by: tegge MFC after: 6 weeks
2007-11-17 22:52:29 +00:00
/*
* Return the number of managed mappings to the given physical page
* that are wired.
*/
int
moea_page_wired_mappings(mmu_t mmu, vm_page_t m)
{
struct pvo_entry *pvo;
int count;
count = 0;
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
if ((m->oflags & VPO_UNMANAGED) != 0)
Prevent the leakage of wired pages in the following circumstances: First, a file is mmap(2)ed and then mlock(2)ed. Later, it is truncated. Under "normal" circumstances, i.e., when the file is not mlock(2)ed, the pages beyond the EOF are unmapped and freed. However, when the file is mlock(2)ed, the pages beyond the EOF are unmapped but not freed because they have a non-zero wire count. This can be a mistake. Specifically, it is a mistake if the sole reason why the pages are wired is because of wired, managed mappings. Previously, unmapping the pages destroys these wired, managed mappings, but does not reduce the pages' wire count. Consequently, when the file is unmapped, the pages are not unwired because the wired mapping has been destroyed. Moreover, when the vm object is finally destroyed, the pages are leaked because they are still wired. The fix is to reduce the pages' wired count by the number of wired, managed mappings destroyed. To do this, I introduce a new pmap function pmap_page_wired_mappings() that returns the number of managed mappings to the given physical page that are wired, and I use this function in vm_object_page_remove(). Reviewed by: tegge MFC after: 6 weeks
2007-11-17 22:52:29 +00:00
return (count);
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Prevent the leakage of wired pages in the following circumstances: First, a file is mmap(2)ed and then mlock(2)ed. Later, it is truncated. Under "normal" circumstances, i.e., when the file is not mlock(2)ed, the pages beyond the EOF are unmapped and freed. However, when the file is mlock(2)ed, the pages beyond the EOF are unmapped but not freed because they have a non-zero wire count. This can be a mistake. Specifically, it is a mistake if the sole reason why the pages are wired is because of wired, managed mappings. Previously, unmapping the pages destroys these wired, managed mappings, but does not reduce the pages' wire count. Consequently, when the file is unmapped, the pages are not unwired because the wired mapping has been destroyed. Moreover, when the vm object is finally destroyed, the pages are leaked because they are still wired. The fix is to reduce the pages' wired count by the number of wired, managed mappings destroyed. To do this, I introduce a new pmap function pmap_page_wired_mappings() that returns the number of managed mappings to the given physical page that are wired, and I use this function in vm_object_page_remove(). Reviewed by: tegge MFC after: 6 weeks
2007-11-17 22:52:29 +00:00
LIST_FOREACH(pvo, vm_page_to_pvoh(m), pvo_vlink)
if ((pvo->pvo_vaddr & PVO_WIRED) != 0)
count++;
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Prevent the leakage of wired pages in the following circumstances: First, a file is mmap(2)ed and then mlock(2)ed. Later, it is truncated. Under "normal" circumstances, i.e., when the file is not mlock(2)ed, the pages beyond the EOF are unmapped and freed. However, when the file is mlock(2)ed, the pages beyond the EOF are unmapped but not freed because they have a non-zero wire count. This can be a mistake. Specifically, it is a mistake if the sole reason why the pages are wired is because of wired, managed mappings. Previously, unmapping the pages destroys these wired, managed mappings, but does not reduce the pages' wire count. Consequently, when the file is unmapped, the pages are not unwired because the wired mapping has been destroyed. Moreover, when the vm object is finally destroyed, the pages are leaked because they are still wired. The fix is to reduce the pages' wired count by the number of wired, managed mappings destroyed. To do this, I introduce a new pmap function pmap_page_wired_mappings() that returns the number of managed mappings to the given physical page that are wired, and I use this function in vm_object_page_remove(). Reviewed by: tegge MFC after: 6 weeks
2007-11-17 22:52:29 +00:00
return (count);
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
static u_int moea_vsidcontext;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pinit(mmu_t mmu, pmap_t pmap)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
int i, mask;
u_int entropy;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
KASSERT((int)pmap < VM_MIN_KERNEL_ADDRESS, ("moea_pinit: virt pmap"));
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK_INIT(pmap);
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
RB_INIT(&pmap->pmap_pvo);
Increase kernel VA from 256Mb to 512Mb by shifting the segment used for user copyinout down to 12, and keeping segments 13/14 for kernel VA. It would be nice to have more available, but segments lower than this are reserved for either memory or 1:1 mapped device i/o, and seg 15 is OpenFirmware ROM. Also, the effort to keep OpenFirmware available for callbacks limits the use of VA-mapped segments. Fortunately UMA_MD_SMALL_ALLOC takes away a lot of VM pressure. Obtained from: NetBSD
2004-03-02 06:49:21 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
entropy = 0;
__asm __volatile("mftb %0" : "=r"(entropy));
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if ((pmap->pmap_phys = (pmap_t)moea_kextract(mmu, (vm_offset_t)pmap))
== NULL) {
pmap->pmap_phys = pmap;
}
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
mtx_lock(&moea_vsid_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Allocate some segment registers for this pmap.
*/
for (i = 0; i < NPMAPS; i += VSID_NBPW) {
u_int hash, n;
/*
* Create a new value by mutiplying by a prime and adding in
* entropy from the timebase register. This is to make the
* VSID more random so that the PT hash function collides
* less often. (Note that the prime casues gcc to do shifts
* instead of a multiply.)
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_vsidcontext = (moea_vsidcontext * 0x1105) + entropy;
hash = moea_vsidcontext & (NPMAPS - 1);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (hash == 0) /* 0 is special, avoid it */
continue;
n = hash >> 5;
mask = 1 << (hash & (VSID_NBPW - 1));
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
hash = (moea_vsidcontext & 0xfffff);
if (moea_vsid_bitmap[n] & mask) { /* collision? */
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/* anything free in this bucket? */
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (moea_vsid_bitmap[n] == 0xffffffff) {
entropy = (moea_vsidcontext >> 20);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
continue;
}
Fix a typo in the original import of this code from NetBSD that caused the wrong element of the VSID bitmap array to be examined after a collision, leading to reallocation of in-use VSIDs under some circumstances, with attendant memory corruption. Also add an assert to check for this kind of problem in the future. MFC after: 4 days
2010-09-08 16:58:06 +00:00
i = ffs(~moea_vsid_bitmap[n]) - 1;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
mask = 1 << i;
hash &= 0xfffff & ~(VSID_NBPW - 1);
hash |= i;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Add an extra invariant here which was useful on 64-bit CPUs.
2011-11-17 15:48:12 +00:00
KASSERT(!(moea_vsid_bitmap[n] & mask),
("Allocating in-use VSID group %#x\n", hash));
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_vsid_bitmap[n] |= mask;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
for (i = 0; i < 16; i++)
pmap->pm_sr[i] = VSID_MAKE(i, hash);
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
mtx_unlock(&moea_vsid_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
mtx_unlock(&moea_vsid_mutex);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pinit: out of segments");
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Initialize the pmap associated with process 0.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pinit0(mmu_t mmu, pmap_t pm)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pinit(mmu, pm);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
bzero(&pm->pm_stats, sizeof(pm->pm_stats));
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
/*
* Set the physical protection on the specified range of this map as requested.
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_protect(mmu_t mmu, pmap_t pm, vm_offset_t sva, vm_offset_t eva,
vm_prot_t prot)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
struct pvo_entry *pvo, *tpvo, key;
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
struct pte *pt;
KASSERT(pm == &curproc->p_vmspace->vm_pmap || pm == kernel_pmap,
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
("moea_protect: non current pmap"));
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
if ((prot & VM_PROT_READ) == VM_PROT_NONE) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_remove(mmu, pm, sva, eva);
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
return;
}
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK(pm);
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
key.pvo_vaddr = sva;
for (pvo = RB_NFIND(pvo_tree, &pm->pmap_pvo, &key);
pvo != NULL && PVO_VADDR(pvo) < eva; pvo = tpvo) {
tpvo = RB_NEXT(pvo_tree, &pm->pmap_pvo, pvo);
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
if ((prot & VM_PROT_EXECUTE) == 0)
pvo->pvo_vaddr &= ~PVO_EXECUTABLE;
/*
* Grab the PTE pointer before we diddle with the cached PTE
* copy.
*/
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
pt = moea_pvo_to_pte(pvo, -1);
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
/*
* Change the protection of the page.
*/
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pvo->pvo_pte.pte.pte_lo &= ~PTE_PP;
pvo->pvo_pte.pte.pte_lo |= PTE_BR;
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
/*
* If the PVO is in the page table, update that pte as well.
*/
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
if (pt != NULL) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_change(pt, &pvo->pvo_pte.pte, pvo->pvo_vaddr);
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_unlock(&moea_table_mutex);
}
Implement the following functions: - pmap_kextract - pmap_object_init_pt - pmap_protect - pmap_remove_pages I'm pretty sure pmap_remove_pages is at least somewhat bogus.
2002-04-16 12:13:10 +00:00
}
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(pm);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
/*
* Map a list of wired pages into kernel virtual address space. This is
* intended for temporary mappings which do not need page modification or
* references recorded. Existing mappings in the region are overwritten.
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_qenter(mmu_t mmu, vm_offset_t sva, vm_page_t *m, int count)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
vm_offset_t va;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
va = sva;
while (count-- > 0) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_kenter(mmu, va, VM_PAGE_TO_PHYS(*m));
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
va += PAGE_SIZE;
m++;
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
/*
* Remove page mappings from kernel virtual address space. Intended for
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
* temporary mappings entered by moea_qenter.
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_qremove(mmu_t mmu, vm_offset_t sva, int count)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
vm_offset_t va;
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
va = sva;
while (count-- > 0) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_kremove(mmu, va);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
va += PAGE_SIZE;
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_release(mmu_t mmu, pmap_t pmap)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
int idx, mask;
/*
* Free segment register's VSID
*/
if (pmap->pm_sr[0] == 0)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_release");
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
mtx_lock(&moea_vsid_mutex);
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
idx = VSID_TO_HASH(pmap->pm_sr[0]) & (NPMAPS-1);
mask = 1 << (idx % VSID_NBPW);
idx /= VSID_NBPW;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_vsid_bitmap[idx] &= ~mask;
Fix the same race condition on 32-bit AIM CPUs that was fixed for 64-bit ones in r211967 involving VSID allocation.
2010-09-06 23:07:58 +00:00
mtx_unlock(&moea_vsid_mutex);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK_DESTROY(pmap);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
/*
* Remove the given range of addresses from the specified map.
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_remove(mmu_t mmu, pmap_t pm, vm_offset_t sva, vm_offset_t eva)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
struct pvo_entry *pvo, *tpvo, key;
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_LOCK(pm);
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
key.pvo_vaddr = sva;
for (pvo = RB_NFIND(pvo_tree, &pm->pmap_pvo, &key);
pvo != NULL && PVO_VADDR(pvo) < eva; pvo = tpvo) {
tpvo = RB_NEXT(pvo_tree, &pm->pmap_pvo, pvo);
moea_pvo_remove(pvo, -1);
Implement the following functions: - pmap_remove - pmap_kremove - pmap_qremove
2002-02-28 02:54:16 +00:00
}
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(pm);
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
/*
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
* Remove physical page from all pmaps in which it resides. moea_pvo_remove()
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
* will reflect changes in pte's back to the vm_page.
*/
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_remove_all(mmu_t mmu, vm_page_t m)
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
{
struct pvo_head *pvo_head;
struct pvo_entry *pvo, *next_pvo;
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
pmap_t pmap;
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wlock(&pvh_global_lock);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
pvo_head = vm_page_to_pvoh(m);
for (pvo = LIST_FIRST(pvo_head); pvo != NULL; pvo = next_pvo) {
next_pvo = LIST_NEXT(pvo, pvo_vlink);
In pmap_page_protect, clear the vm page's PG_WRITEABLE flag if downgrading to read-only. Found by triggering the KASSERT in vm_pageout_flush().
2004-08-05 12:44:12 +00:00
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
pmap = pvo->pvo_pmap;
PMAP_LOCK(pmap);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_remove(pvo, -1);
Add pmap locking to many of the functions. Many thanks to Andrew Gallatin for resolving a powerpc-specific initialization problem in my original patch. Tested by: gallatin@
2004-08-26 04:15:36 +00:00
PMAP_UNLOCK(pmap);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
}
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
if ((m->aflags & PGA_WRITEABLE) && moea_query_bit(m, PTE_CHG)) {
Correct a harmless typo introduced when copying code from mmu_oea64. Submitted by: alc MFC after: 8.1-RELEASE
2010-06-05 18:24:41 +00:00
moea_attr_clear(m, PTE_CHG);
Fix a bug where pages being removed from memory entirely no longer have PVOs, and so the modified state of the page can no longer be communicated to the VM layer, causing pages not to be flushed to swap when needed, in turn causing memory corruption. Also make several correctness adjustments to I-Cache synchronization and TLB invalidation for 64-bit Book-S CPUs. Obtained from: projects/ppc64 Discussed with: grehan MFC after: 2 weeks
2010-02-18 15:00:43 +00:00
vm_page_dirty(m);
}
Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic flags field. Updates to the atomic flags are performed using the atomic ops on the containing word, do not require any vm lock to be held, and are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9) functions are provided to modify afalgs. Document the changes to flags field to only require the page lock. Introduce vm_page_reference(9) function to provide a stable KPI and KBI for filesystems like tmpfs and zfs which need to mark a page as referenced. Reviewed by: alc, attilio Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64) Approved by: re (bz)
2011-09-06 10:30:11 +00:00
vm_page_aflag_clear(m, PGA_WRITEABLE);
Replace all uses of the vm page queues lock by a r/w lock that is private to this pmap. Tested by: andreast, jhibbits
2012-07-06 02:18:49 +00:00
rw_wunlock(&pvh_global_lock);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* Allocate a physical page of memory directly from the phys_avail map.
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
* Can only be called from moea_bootstrap before avail start and end are
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* calculated.
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static vm_offset_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_bootstrap_alloc(vm_size_t size, u_int align)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
vm_offset_t s, e;
int i, j;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
size = round_page(size);
for (i = 0; phys_avail[i + 1] != 0; i += 2) {
if (align != 0)
s = (phys_avail[i] + align - 1) & ~(align - 1);
else
s = phys_avail[i];
e = s + size;
if (s < phys_avail[i] || e > phys_avail[i + 1])
continue;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (s == phys_avail[i]) {
phys_avail[i] += size;
} else if (e == phys_avail[i + 1]) {
phys_avail[i + 1] -= size;
} else {
for (j = phys_avail_count * 2; j > i; j -= 2) {
phys_avail[j] = phys_avail[j - 2];
phys_avail[j + 1] = phys_avail[j - 1];
}
phys_avail[i + 3] = phys_avail[i + 1];
phys_avail[i + 1] = s;
phys_avail[i + 2] = e;
phys_avail_count++;
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (s);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_bootstrap_alloc: could not allocate memory");
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_syncicache(vm_offset_t pa, vm_size_t len)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
__syncicache((void *)pa, len);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_enter(pmap_t pm, uma_zone_t zone, struct pvo_head *pvo_head,
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
vm_offset_t va, vm_offset_t pa, u_int pte_lo, int flags)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pvo_entry *pvo;
u_int sr;
int first;
u_int ptegidx;
int i;
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
int bootstrap;
Make PMAP_SHPGPERPROC tunable. One shouldn't need to recompile a kernel for this, since it is easy to run into with large systems with lots of shared mmap space. Obtained from: yahoo
2001-07-27 01:08:59 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_enter_calls++;
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
first = 0;
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
bootstrap = 0;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Compute the PTE Group index.
*/
va &= ~ADDR_POFF;
sr = va_to_sr(pm->pm_sr, va);
ptegidx = va_to_pteg(sr, va);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Remove any existing mapping for this page. Reuse the pvo entry if
* there is a mapping.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_lock(&moea_table_mutex);
LIST_FOREACH(pvo, &moea_pvo_table[ptegidx], pvo_olink) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pvo->pvo_pmap == pm && PVO_VADDR(pvo) == va) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if ((pvo->pvo_pte.pte.pte_lo & PTE_RPGN) == pa &&
(pvo->pvo_pte.pte.pte_lo & PTE_PP) ==
Improve our detection of an attempted duplicate entry. We may be trying to change the page protection bits.
2002-05-10 06:27:08 +00:00
(pte_lo & PTE_PP)) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_unlock(&moea_table_mutex);
Changes and fixes in preparation for UMA: - Bootstrap pvo entries are now allocated by stealing pages. - Just return if we're pmap_enter'ing a mapping that's already there. Don't remove it and re-enter it.
2002-03-17 23:58:12 +00:00
return (0);
Improve our detection of an attempted duplicate entry. We may be trying to change the page protection bits.
2002-05-10 06:27:08 +00:00
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_remove(pvo, -1);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
break;
}
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* If we aren't overwriting a mapping, try to allocate.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (moea_initialized) {
Remove references to vm_zone.h and switch over to the new uma API.
2002-03-21 01:11:31 +00:00
pvo = uma_zalloc(zone, M_NOWAIT);
Changes and fixes in preparation for UMA: - Bootstrap pvo entries are now allocated by stealing pages. - Just return if we're pmap_enter'ing a mapping that's already there. Don't remove it and re-enter it.
2002-03-17 23:58:12 +00:00
} else {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (moea_bpvo_pool_index >= BPVO_POOL_SIZE) {
panic("moea_enter: bpvo pool exhausted, %d, %d, %d",
moea_bpvo_pool_index, BPVO_POOL_SIZE,
- Initialise battable to cover I/O spaces. - Statically size the bpvo entries to avoid conflicts between bpvo allocation and the vm allocator. - Shift pmap_init2 code into pmap_init. - Add UMA_ZONE_VM flag to uma_zcreate. Submitted by: Peter Grehan <peterg@ptree32.com.au>
2002-06-29 09:43:59 +00:00
BPVO_POOL_SIZE * sizeof(struct pvo_entry));
Changes and fixes in preparation for UMA: - Bootstrap pvo entries are now allocated by stealing pages. - Just return if we're pmap_enter'ing a mapping that's already there. Don't remove it and re-enter it.
2002-03-17 23:58:12 +00:00
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pvo = &moea_bpvo_pool[moea_bpvo_pool_index];
moea_bpvo_pool_index++;
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
bootstrap = 1;
Changes and fixes in preparation for UMA: - Bootstrap pvo entries are now allocated by stealing pages. - Just return if we're pmap_enter'ing a mapping that's already there. Don't remove it and re-enter it.
2002-03-17 23:58:12 +00:00
}
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pvo == NULL) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (ENOMEM);
}
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_entries++;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo->pvo_vaddr = va;
pvo->pvo_pmap = pm;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
LIST_INSERT_HEAD(&moea_pvo_table[ptegidx], pvo, pvo_olink);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo->pvo_vaddr &= ~ADDR_POFF;
if (flags & VM_PROT_EXECUTE)
pvo->pvo_vaddr |= PVO_EXECUTABLE;
if (flags & PVO_WIRED)
pvo->pvo_vaddr |= PVO_WIRED;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (pvo_head != &moea_pvo_kunmanaged)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo->pvo_vaddr |= PVO_MANAGED;
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
if (bootstrap)
pvo->pvo_vaddr |= PVO_BOOTSTRAP;
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_create(&pvo->pvo_pte.pte, sr, va, pa | pte_lo);
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
Keep track of PVO entries in each pmap, which allows much faster pmap_remove() for large sparse requests. This can prevent pmap_remove() operations on 64-bit process destruction or swapout that would take several hundred times the lifetime of the universe to complete. This behavior is largely indistinguishable from a hang.
2011-12-11 17:19:48 +00:00
/*
* Add to pmap list
*/
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
RB_INSERT(pvo_tree, &pm->pmap_pvo, pvo);
Keep track of PVO entries in each pmap, which allows much faster pmap_remove() for large sparse requests. This can prevent pmap_remove() operations on 64-bit process destruction or swapout that would take several hundred times the lifetime of the universe to complete. This behavior is largely indistinguishable from a hang.
2011-12-11 17:19:48 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Remember if the list was empty and therefore will be the first
* item.
*/
1. Better track the executable status of mappings. 2. Set a pcpu variable to the real address of the active pmap (used when exiting from traps. Obtained from: NetBSD (1)
2002-05-09 14:09:19 +00:00
if (LIST_FIRST(pvo_head) == NULL)
first = 1;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
LIST_INSERT_HEAD(pvo_head, pvo, pvo_vlink);
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo->pvo_pte.pte.pte_lo & PVO_WIRED)
Eliminate unnecessary indirection.
2004-08-28 20:27:12 +00:00
pm->pm_stats.wired_count++;
pm->pm_stats.resident_count++;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* We hope this succeeds but it isn't required.
*/
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
i = moea_pte_insert(ptegidx, &pvo->pvo_pte.pte);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (i >= 0) {
PVO_PTEGIDX_SET(pvo, i);
} else {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pvo_enter: overflow");
moea_pte_overflow++;
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_unlock(&moea_table_mutex);
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (first ? ENOENT : 0);
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_remove(struct pvo_entry *pvo, int pteidx)
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pte *pt;
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* If there is an active pte entry, we need to deactivate it (and
* save the ref & cfg bits).
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pt = moea_pvo_to_pte(pvo, pteidx);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pt != NULL) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_unset(pt, &pvo->pvo_pte.pte, pvo->pvo_vaddr);
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
PVO_PTEGIDX_CLR(pvo);
} else {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_overflow--;
Add PVO_FAKE flag to pvo entries for PG_FICTITIOUS mappings, to avoid trying to reverse-map a device physical address to the vm_page array and walking into non-existent vm weeds. found by: Xorg server exiting
2005-02-25 02:42:15 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Update our statistics.
*/
pvo->pvo_pmap->pm_stats.resident_count--;
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo->pvo_pte.pte.pte_lo & PVO_WIRED)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo->pvo_pmap->pm_stats.wired_count--;
/*
* Save the REF/CHG bits into their cache if the page is managed.
*/
- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag to VPO_UNMANAGED (and also making the flag protected by the vm object lock, instead of vm page queue lock). - Mark the fake pages with both PG_FICTITIOUS (as it is now) and VPO_UNMANAGED. As a consequence, pmap code now can use use just VPO_UNMANAGED to decide whether the page is unmanaged. Reviewed by: alc Tested by: pho (x86, previous version), marius (sparc64), marcel (arm, ia64, powerpc), ray (mips) Sponsored by: The FreeBSD Foundation Approved by: re (bz)
2011-08-09 21:01:36 +00:00
if ((pvo->pvo_vaddr & PVO_MANAGED) == PVO_MANAGED) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct vm_page *pg;
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pg = PHYS_TO_VM_PAGE(pvo->pvo_pte.pte.pte_lo & PTE_RPGN);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pg != NULL) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_attr_save(pg, pvo->pvo_pte.pte.pte_lo &
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
(PTE_REF | PTE_CHG));
Update PowerPC MD code to compile and do initial bootstrap based on recent changes (KSE and VM requiring physmem to be setup). Reviewed by: benno, jhb, julian
2001-09-20 00:47:17 +00:00
}
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
Keep track of PVO entries in each pmap, which allows much faster pmap_remove() for large sparse requests. This can prevent pmap_remove() operations on 64-bit process destruction or swapout that would take several hundred times the lifetime of the universe to complete. This behavior is largely indistinguishable from a hang.
2011-12-11 17:19:48 +00:00
* Remove this PVO from the PV and pmap lists.
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
*/
LIST_REMOVE(pvo, pvo_vlink);
Replace the list of PVOs owned by each PMAP with an RB tree. This simplifies range operations like pmap_remove() and pmap_protect() as well as allowing simple operations like pmap_extract() not to involve any global state. This substantially reduces lock coverages for the global table lock and improves concurrency.
2012-05-20 14:33:28 +00:00
RB_REMOVE(pvo_tree, &pvo->pvo_pmap->pmap_pvo, pvo);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Remove this from the overflow list and return it to the pool
* if we aren't going to reuse it.
*/
LIST_REMOVE(pvo, pvo_olink);
Changes and fixes in preparation for UMA: - Bootstrap pvo entries are now allocated by stealing pages. - Just return if we're pmap_enter'ing a mapping that's already there. Don't remove it and re-enter it.
2002-03-17 23:58:12 +00:00
if (!(pvo->pvo_vaddr & PVO_BOOTSTRAP))
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
uma_zfree(pvo->pvo_vaddr & PVO_MANAGED ? moea_mpvo_zone :
moea_upvo_zone, pvo);
moea_pvo_entries--;
moea_pvo_remove_calls++;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static __inline int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_pte_index(const struct pvo_entry *pvo, int ptegidx)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
int pteidx;
/*
* We can find the actual pte entry without searching by grabbing
* the PTEG index from 3 unused bits in pte_lo[11:9] and by
* noticing the HID bit.
*/
pteidx = ptegidx * 8 + PVO_PTEGIDX_GET(pvo);
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo->pvo_pte.pte.pte_hi & PTE_HID)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pteidx ^= moea_pteg_mask * 8;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (pteidx);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static struct pvo_entry *
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_find_va(pmap_t pm, vm_offset_t va, int *pteidx_p)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pvo_entry *pvo;
int ptegidx;
u_int sr;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
va &= ~ADDR_POFF;
sr = va_to_sr(pm->pm_sr, va);
ptegidx = va_to_pteg(sr, va);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_lock(&moea_table_mutex);
LIST_FOREACH(pvo, &moea_pvo_table[ptegidx], pvo_olink) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pvo->pvo_pmap == pm && PVO_VADDR(pvo) == va) {
if (pteidx_p)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
*pteidx_p = moea_pvo_pte_index(pvo, ptegidx);
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
break;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_unlock(&moea_table_mutex);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
return (pvo);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static struct pte *
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pvo_to_pte(const struct pvo_entry *pvo, int pteidx)
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pte *pt;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* If we haven't been supplied the ptegidx, calculate it.
*/
if (pteidx == -1) {
int ptegidx;
u_int sr;
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
sr = va_to_sr(pvo->pvo_pmap->pm_sr, pvo->pvo_vaddr);
ptegidx = va_to_pteg(sr, pvo->pvo_vaddr);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pteidx = moea_pvo_pte_index(pvo, ptegidx);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pt = &moea_pteg_table[pteidx >> 3].pt[pteidx & 7];
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_lock(&moea_table_mutex);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if ((pvo->pvo_pte.pte.pte_hi & PTE_VALID) && !PVO_PTEGIDX_ISSET(pvo)) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pvo_to_pte: pvo %p has valid pte in pvo but no "
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
"valid pte index", pvo);
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if ((pvo->pvo_pte.pte.pte_hi & PTE_VALID) == 0 && PVO_PTEGIDX_ISSET(pvo)) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pvo_to_pte: pvo %p has valid pte index in pvo "
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
"pvo but no valid pte", pvo);
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if ((pt->pte_hi ^ (pvo->pvo_pte.pte.pte_hi & ~PTE_VALID)) == PTE_VALID) {
if ((pvo->pvo_pte.pte.pte_hi & PTE_VALID) == 0) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pvo_to_pte: pvo %p has valid pte in "
"moea_pteg_table %p but invalid in pvo", pvo, pt);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (((pt->pte_lo ^ pvo->pvo_pte.pte.pte_lo) & ~(PTE_CHG|PTE_REF))
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
!= 0) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pvo_to_pte: pvo %p pte does not match "
"pte %p in moea_pteg_table", pvo, pt);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (pt);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo->pvo_pte.pte.pte_hi & PTE_VALID) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pvo_to_pte: pvo %p has invalid pte %p in "
"moea_pteg_table but valid in pvo", pvo, pt);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (NULL);
Bring in NetBSD code used in the PowerPC port. Reviewed by: obrien, dfr Obtained from: NetBSD
2001-06-10 02:39:37 +00:00
}
Fix comment breakage.
2001-06-27 12:20:48 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* XXX: THIS STUFF SHOULD BE IN pte.c?
Fix comment breakage.
2001-06-27 12:20:48 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_spill(vm_offset_t addr)
Fix comment breakage.
2001-06-27 12:20:48 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pvo_entry *source_pvo, *victim_pvo;
struct pvo_entry *pvo;
int ptegidx, i, j;
u_int sr;
struct pteg *pteg;
struct pte *pt;
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_spills++;
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Use mtsrin() instead of inline asm.
2002-04-16 12:07:41 +00:00
sr = mfsrin(addr);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
ptegidx = va_to_pteg(sr, addr);
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Have to substitute some entry. Use the primary hash for this.
* Use low bits of timebase as random generator.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pteg = &moea_pteg_table[ptegidx];
mtx_lock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
__asm __volatile("mftb %0" : "=r"(i));
i &= 7;
pt = &pteg->pt[i];
source_pvo = NULL;
victim_pvo = NULL;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
LIST_FOREACH(pvo, &moea_pvo_table[ptegidx], pvo_olink) {
Fix comment breakage.
2001-06-27 12:20:48 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* We need to find a pvo entry for this address.
Fix comment breakage.
2001-06-27 12:20:48 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (source_pvo == NULL &&
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_match(&pvo->pvo_pte.pte, sr, addr,
pvo->pvo_pte.pte.pte_hi & PTE_HID)) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Now found an entry to be spilled into the pteg.
* The PTE is now valid, so we know it's active.
*/
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
j = moea_pte_insert(ptegidx, &pvo->pvo_pte.pte);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (j >= 0) {
PVO_PTEGIDX_SET(pvo, j);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_overflow--;
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (1);
}
source_pvo = pvo;
if (victim_pvo != NULL)
break;
}
Fix comment breakage.
2001-06-27 12:20:48 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* We also need the pvo entry of the victim we are replacing
* so save the R & C bits of the PTE.
Fix comment breakage.
2001-06-27 12:20:48 +00:00
*/
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if ((pt->pte_hi & PTE_HID) == 0 && victim_pvo == NULL &&
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_compare(pt, &pvo->pvo_pte.pte)) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
victim_pvo = pvo;
if (source_pvo != NULL)
break;
}
}
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
if (source_pvo == NULL) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (0);
- Introduce a lock for synchronizing access to the pvo and pteg tables. - In pmap_enter(), only the acquisition and release of the page queues lock needs to check the bootstrap flag. Tested by: gallatin@
2004-08-30 21:39:22 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (victim_pvo == NULL) {
if ((pt->pte_hi & PTE_HID) == 0)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pte_spill: victim p-pte (%p) has no pvo"
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
"entry", pt);
Fix comment breakage.
2001-06-27 12:20:48 +00:00
/*
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
* If this is a secondary PTE, we need to search it's primary
* pvo bucket for the matching PVO.
Fix comment breakage.
2001-06-27 12:20:48 +00:00
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
LIST_FOREACH(pvo, &moea_pvo_table[ptegidx ^ moea_pteg_mask],
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
pvo_olink) {
/*
* We also need the pvo entry of the victim we are
* replacing so save the R & C bits of the PTE.
*/
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (moea_pte_compare(pt, &pvo->pvo_pte.pte)) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
victim_pvo = pvo;
break;
}
}
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (victim_pvo == NULL)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pte_spill: victim s-pte (%p) has no pvo"
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
"entry", pt);
}
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* We are invalidating the TLB entry for the EA we are replacing even
* though it's valid. If we don't, we lose any ref/chg bit changes
* contained in the TLB entry.
*/
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
source_pvo->pvo_pte.pte.pte_hi &= ~PTE_HID;
Fix comment breakage.
2001-06-27 12:20:48 +00:00
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_unset(pt, &victim_pvo->pvo_pte.pte, victim_pvo->pvo_vaddr);
moea_pte_set(pt, &source_pvo->pvo_pte.pte);
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
PVO_PTEGIDX_CLR(victim_pvo);
PVO_PTEGIDX_SET(source_pvo, i);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_replacements++;
Fix comment breakage.
2001-06-27 12:20:48 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (1);
}
static int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_insert(u_int ptegidx, struct pte *pvo_pt)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
struct pte *pt;
int i;
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_assert(&moea_table_mutex, MA_OWNED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* First try primary hash.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
for (pt = moea_pteg_table[ptegidx].pt, i = 0; i < 8; i++, pt++) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if ((pt->pte_hi & PTE_VALID) == 0) {
pvo_pt->pte_hi &= ~PTE_HID;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_set(pt, pvo_pt);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (i);
}
Fix comment breakage.
2001-06-27 12:20:48 +00:00
}
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Now try secondary hash.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
ptegidx ^= moea_pteg_mask;
Remove bogus increment of re-hashed PTEG index. This snuck in with r1.12 of pmap.c, and is potentially the cause of hangs reported on machines with a small amount of memory. On machines with sufficient RAM, and without a lot of processes running, this situation would probably never occur. Testing is still incomplete, but it is obviously wrong so remove the offending code now. The issue of what to do when both the primary and secondary hash overflow is still open. Reported by: Dan Kresja at windriver dot com, via alc
2006-12-20 01:10:21 +00:00
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
for (pt = moea_pteg_table[ptegidx].pt, i = 0; i < 8; i++, pt++) {
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if ((pt->pte_hi & PTE_VALID) == 0) {
pvo_pt->pte_hi |= PTE_HID;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_set(pt, pvo_pt);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (i);
}
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_pte_insert: overflow");
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (-1);
Fix comment breakage.
2001-06-27 12:20:48 +00:00
}
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
static boolean_t
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_query_bit(vm_page_t m, int ptebit)
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
{
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pvo_entry *pvo;
struct pte *pt;
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_assert(&pvh_global_lock, RA_WLOCKED);
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (moea_attr_fetch(m) & ptebit)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (TRUE);
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
LIST_FOREACH(pvo, vm_page_to_pvoh(m), pvo_vlink) {
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* See if we saved the bit off. If so, cache it and return
* success.
*/
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo->pvo_pte.pte.pte_lo & ptebit) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_attr_save(m, ptebit);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (TRUE);
}
}
/*
* No luck, now go through the hard part of looking at the PTEs
* themselves. Sync so that any pending REF/CHG bits are flushed to
* the PTEs.
*/
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
LIST_FOREACH(pvo, vm_page_to_pvoh(m), pvo_vlink) {
/*
* See if this pvo has a valid PTE. if so, fetch the
* REF/CHG bits from the valid PTE. If the appropriate
* ptebit is set, cache it and return success.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pt = moea_pvo_to_pte(pvo, -1);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pt != NULL) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_synch(pt, &pvo->pvo_pte.pte);
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_unlock(&moea_table_mutex);
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
if (pvo->pvo_pte.pte.pte_lo & ptebit) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_attr_save(m, ptebit);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
return (TRUE);
}
}
}
pmap_query_bit() should return false if the bit is not set. Reviewed by: grehan
2003-12-09 14:47:33 +00:00
return (FALSE);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
static u_int
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
moea_clear_bit(vm_page_t m, int ptebit)
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
{
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
u_int count;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
struct pvo_entry *pvo;
struct pte *pt;
Reduce the scope of the page queues lock and the number of PG_REFERENCED changes in vm_pageout_object_deactivate_pages(). Simplify this function's inner loop using TAILQ_FOREACH(), and shorten some of its overly long lines. Update a stale comment. Assert that PG_REFERENCED may be cleared only if the object containing the page is locked. Add a comment documenting this. Assert that a caller to vm_page_requeue() holds the page queues lock, and assert that the page is on a page queue. Push down the page queues lock into pmap_ts_referenced() and pmap_page_exists_quick(). (As of now, there are no longer any pmap functions that expect to be called with the page queues lock held.) Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever be passed an unmanaged page. Assert this rather than returning "0" and "FALSE" respectively. ARM: Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH(). Push down the page queues lock inside of pmap_clearbit(), simplifying pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write(). Additionally, this allows for avoiding the acquisition of the page queues lock in some cases. PowerPC/AIM: moea*_page_exits_quick() and moea*_page_wired_mappings() will never be called before pmap initialization is complete. Therefore, the check for moea_initialized can be eliminated. Push down the page queues lock inside of moea*_clear_bit(), simplifying moea*_clear_modify() and moea*_clear_reference(). The last parameter to moea*_clear_bit() is never used. Eliminate it. PowerPC/BookE: Simplify mmu_booke_page_exists_quick()'s control flow. Reviewed by: kib@
2010-06-10 16:56:35 +00:00
Avoid recursion on the pvh global lock in the aim oea pmap. Correct the return type of the pmap_ts_referenced() implementations. Reported by: jhibbits [1] Tested by: andreast
2012-07-10 22:10:21 +00:00
rw_assert(&pvh_global_lock, RA_WLOCKED);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Clear the cached value.
*/
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_attr_clear(m, ptebit);
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* Sync so that any pending REF/CHG bits are flushed to the PTEs (so
* we can reset the right ones). note that since the pvo entries and
* list heads are accessed via BAT0 and are never placed in the page
* table, we don't have to worry about further accesses setting the
* REF/CHG bits.
*/
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
powerpc_sync();
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
/*
* For each pvo entry, clear the pvo's ptebit. If this pvo has a
* valid pte clear the ptebit from the valid pte.
*/
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
count = 0;
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
LIST_FOREACH(pvo, vm_page_to_pvoh(m), pvo_vlink) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
pt = moea_pvo_to_pte(pvo, -1);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
if (pt != NULL) {
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
moea_pte_synch(pt, &pvo->pvo_pte.pte);
if (pvo->pvo_pte.pte.pte_lo & ptebit) {
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
count++;
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_pte_clear(pt, PVO_VADDR(pvo), ptebit);
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
}
Synchronize accesses to the PTEG table. Add many lock assertions. Tested by: grehan@
2006-06-25 19:07:01 +00:00
mtx_unlock(&moea_table_mutex);
Complete rework of the PowerPC pmap and a number of other bits in the early boot sequence. The new pmap.c is based on NetBSD's newer pmap.c (for the mpc6xx processors) which is 70% faster than the older code that the original pmap.c was based on. It has also been based on the framework established by jake's initial sparc64 pmap.c. There is no change to how far the kernel gets (it makes it to the mountroot prompt in psim) but the new pmap code is a lot cleaner. Obtained from: NetBSD (pmap code)
2002-02-14 01:39:11 +00:00
}
In preparation for PowerPC G5 support, allow PVO objects to contain page table entries for both the 32-bit and 64-bit AIM MMUs.
2008-09-23 03:02:57 +00:00
pvo->pvo_pte.pte.pte_lo &= ~ptebit;
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
}
- add pmap_pagedaemon_waken variable - remove dead code and fix warnings in pmap_zero_page/zero_page_area - implement pmap_clear_reference pmap_ts_referenced pmap_page_exists_quick pmap_remove_all - align pmap_qenter/qremove closer with i386 code - fix vm_page locking in pmap_new_thread (from benno) - add new parameter to pmap_clear_bit to return original pte value Approved by: benno
2003-02-01 02:56:48 +00:00
return (count);
Implement pmap_mapdev.
2001-10-14 08:38:16 +00:00
}
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
/*
* Return true if the physical range is encompassed by the battable[idx]
*/
static int
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_bat_mapped(int idx, vm_offset_t pa, vm_size_t size)
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
{
u_int prot;
u_int32_t start;
u_int32_t end;
u_int32_t bat_ble;
/*
* Return immediately if not a valid mapping
*/
Add some missing parentheses so that moea_bat_mapped() actually works. Submitted by: alc MFC after: 3 days
2010-10-31 15:07:09 +00:00
if (!(battable[idx].batu & BAT_Vs))
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
return (EINVAL);
/*
* The BAT entry must be cache-inhibited, guarded, and r/w
* so it can function as an i/o page
*/
prot = battable[idx].batl & (BAT_I|BAT_G|BAT_PP_RW);
if (prot != (BAT_I|BAT_G|BAT_PP_RW))
return (EPERM);
/*
* The address should be within the BAT range. Assume that the
* start address in the BAT has the correct alignment (thus
* not requiring masking)
*/
start = battable[idx].batl & BAT_PBS;
bat_ble = (battable[idx].batu & ~(BAT_EBS)) | 0x03;
end = start | (bat_ble << 15) | 0x7fff;
if ((pa < start) || ((pa + size) > end))
return (ERANGE);
return (0);
}
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
boolean_t
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
moea_dev_direct_mapped(mmu_t mmu, vm_paddr_t pa, vm_size_t size)
Add /dev/mem and /dev/kmem to powerpc. Approved by: grehan (mentor)
2004-08-16 13:07:40 +00:00
{
int i;
/*
* This currently does not work for entries that
* overlap 256M BAT segments.
*/
for(i = 0; i < 16; i++)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (moea_bat_mapped(i, pa, size) == 0)
Add /dev/mem and /dev/kmem to powerpc. Approved by: grehan (mentor)
2004-08-16 13:07:40 +00:00
return (0);
return (EFAULT);
}
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
/*
* Map a set of physical memory pages into the kernel virtual
* address space. Return a pointer to where it is mapped. This
* routine is intended to be used for mapping device memory,
* NOT real memory.
*/
void *
Fix physical address type to vm_paddr_t.
2012-05-24 21:13:24 +00:00
moea_mapdev(mmu_t mmu, vm_paddr_t pa, vm_size_t size)
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
{
return (moea_mapdev_attr(mmu, pa, size, VM_MEMATTR_DEFAULT));
}
void *
moea_mapdev_attr(mmu_t mmu, vm_offset_t pa, vm_size_t size, vm_memattr_t ma)
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
{
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
vm_offset_t va, tmpva, ppa, offset;
int i;
ppa = trunc_page(pa);
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
offset = pa & PAGE_MASK;
size = roundup(offset + size, PAGE_SIZE);
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
/*
* If the physical address lies within a valid BAT table entry,
* return the 1:1 mapping. This currently doesn't work
* for regions that overlap 256M BAT segments.
*/
for (i = 0; i < 16; i++) {
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
if (moea_bat_mapped(i, pa, size) == 0)
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
return ((void *) pa);
}
Use kmem_alloc_nofault() rather than kmem_alloc_pageable() in pmap_mapdev(). See revision 1.140 of kern/sys_pipe.c for a detailed rationale. Submitted by: tegge
2003-08-02 19:26:09 +00:00
va = kmem_alloc_nofault(kernel_map, size);
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
if (!va)
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
panic("moea_mapdev: Couldn't alloc kernel virtual memory");
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
for (tmpva = va; size > 0;) {
Add support for memory attributes (pmap_mapdev_attr() and friends) on PowerPC/AIM. This is currently stubbed out on Book-E, since I have no idea how to implement it there.
2010-09-30 18:14:12 +00:00
moea_kenter_attr(mmu, tmpva, ppa, ma);
o When not making a translation cache-inhibit and guarded (PTE_I|PTE_G) make it memory-coherency enforced (PTE_M). This is required for SMP to work. o Serialize tlbie operations and implement the tlbie operation in a function called tlbie(). Hardware can end up in a live-lock if between the tlbsync and subsequent sync on one processor another processor executes a tlbie or tlbsync. o Eliminate the following defines: TLBIE, TLBSYNC, SYNC and EIEIO Use either inline assembly statements or inline functions defined in <machine/cpufunc.h>
2008-09-16 19:16:33 +00:00
tlbie(tmpva);
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
size -= PAGE_SIZE;
tmpva += PAGE_SIZE;
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
ppa += PAGE_SIZE;
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
}
return ((void *)(va + offset));
}
void
Name change from pmap_* to moea_* to fit into the new order of mmu implementation. This code handles the 32-bit 'OEA' MMU found on G2/G3/G4 PPC cores.
2005-11-08 06:49:45 +00:00
moea_unmapdev(mmu_t mmu, vm_offset_t va, vm_size_t size)
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
{
vm_offset_t base, offset;
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
/*
* If this is outside kernel virtual space, then it's a
* battable entry and doesn't require unmapping
*/
Reduce KVA pressure on OEA64 systems running in bridge mode by mapping UMA segments at their physical addresses instead of into KVA. This emulates the direct mapping behavior of OEA32 in an ad-hoc way. To make this work properly required sharing the entire kernel PMAP with Open Firmware, so ofw_pmap is transformed into a stub on 64-bit CPUs. Also implement some more tweaks to get more mileage out of our limited amount of KVA, principally by extending KVA into segment 16 until the beginning of the first OFW mapping. Reported by: linimon
2010-02-20 16:23:29 +00:00
if ((va >= VM_MIN_KERNEL_ADDRESS) && (va <= virtual_end)) {
- use BAT registers to map device space and physical memory - remove test in pmap_activate that prevented vmspace sharing (v/rfork) - always sync icache in pmap_enter until problems are sorted - fix incorrect use of regions in pmap_kenter - bring in pmap_release from NetBSD - fix overwrite of bootstrap flag in pmap_pvo_enter Approved by: benno
2002-09-19 04:36:20 +00:00
base = trunc_page(va);
offset = va & PAGE_MASK;
size = roundup(offset + size, PAGE_SIZE);
kmem_free(kernel_map, base, size);
}
Add pmap_mapdev and pmap_unmapdev.
2002-06-29 09:45:59 +00:00
}
o Introduce vm_sync_icache() for making the I-cache coherent with the memory or D-cache, depending on the semantics of the platform. vm_sync_icache() is basically a wrapper around pmap_sync_icache(), that translates the vm_map_t argumument to pmap_t. o Introduce pmap_sync_icache() to all PMAP implementation. For powerpc it replaces the pmap_page_executable() function, added to solve the I-cache problem in uiomove_fromphys(). o In proc_rwmem() call vm_sync_icache() when writing to a page that has execute permissions. This assures that when breakpoints are written, the I-cache will be coherent and the process will actually hit the breakpoint. o This also fixes the Book-E PMAP implementation that was missing necessary locking while trying to deal with the I-cache coherency in pmap_enter() (read: mmu_booke_enter_locked). The key property of this change is that the I-cache is made coherent *after* writes have been done. Doing it in the PMAP layer when adding or changing a mapping means that the I-cache is made coherent *before* any writes happen. The difference is key when the I-cache prefetches.
2009-10-21 18:38:02 +00:00
static void
moea_sync_icache(mmu_t mmu, pmap_t pm, vm_offset_t va, vm_size_t sz)
{
struct pvo_entry *pvo;
vm_offset_t lim;
vm_paddr_t pa;
vm_size_t len;
PMAP_LOCK(pm);
while (sz > 0) {
lim = round_page(va);
len = MIN(lim - va, sz);
pvo = moea_pvo_find_va(pm, va & ~ADDR_POFF, NULL);
if (pvo != NULL) {
pa = (pvo->pvo_pte.pte.pte_lo & PTE_RPGN) |
(va & ADDR_POFF);
moea_syncicache(pa, len);
}
va += len;
sz -= len;
}
PMAP_UNLOCK(pm);
}
Reference in New Issue Copy Permalink