d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add missed barrier for pm_gen/pm_active interaction.

Browse Source 
When we issue shootdown IPIs, we first assign zero to pm_gens to
indicate the need to flush on the next context switch in case our IPI
misses the context, next we read pm_active. On context switch we set
our bit in pm_active, then we read pm_gen. It is crucial that both
threads see the memory in the program order, otherwise invalidation
thread might read pm_active bit as zero and the context switching
thread might read pm_gen as zero.

IA32 allows CPU for both reads to see zero. We must use the barriers
between write and read. The pm_active bit set is already locked, so
only the invalidation functions need it.

I never saw it in real life, or at least I do not have a good
reproduction case. I found this during code inspection when hunting
for the Xen TLB issue reported by cperciva.

Reviewed by:	alc, markj
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
Differential revision:	https://reviews.freebsd.org/D15506
This commit is contained in:
Konstantin Belousov 2018-05-21 18:41:16 +00:00
parent 733efc21c4
commit 0a4b04a616
1 changed files with 16 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
sys/amd64/amd64/pmap.c
View File

@ -1721,6 +1721,18 @@ pmap_invalidate_page(pmap_t pmap, vm_offset_t va)
if (cpuid != i)
pmap->pm_pcids[i].pm_gen = 0;
}
/*
* The fence is between stores to pm_gen and the read of
* the pm_active mask. We need to ensure that it is
* impossible for us to miss the bit update in pm_active
* and simultaneously observe a non-zero pm_gen in
* pmap_activate_sw(), otherwise TLB update is missed.
* Without the fence, IA32 allows such an outcome.
* Note that pm_active is updated by a locked operation,
* which provides the reciprocal fence.
*/
atomic_thread_fence_seq_cst();
}
mask = &pmap->pm_active;
}
@ -1792,6 +1804,8 @@ pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva)
if (cpuid != i)
pmap->pm_pcids[i].pm_gen = 0;
}
/* See comment int pmap_invalidate_page(). */
atomic_thread_fence_seq_cst();
}
mask = &pmap->pm_active;
}
@ -1863,6 +1877,8 @@ pmap_invalidate_all(pmap_t pmap)
if (cpuid != i)
pmap->pm_pcids[i].pm_gen = 0;
}
/* See comment int pmap_invalidate_page(). */
atomic_thread_fence_seq_cst();
}
mask = &pmap->pm_active;
}