Commit Graph

52 Commits

Author SHA1 Message Date
Attilio Rao
3a4730256a Add an unified macro to deny ability from the compiler to reorder
instruction loads/stores at its will.
The macro __compiler_membar() is currently supported for both gcc and
clang, but kernel compilation will fail otherwise.

Reviewed by:	bde, kib
Discussed with:	dim, theraven
MFC after:	2 weeks
2012-10-09 14:32:30 +00:00
Konstantin Belousov
fa9f322df9 Use plain store for atomic_store_rel on x86, instead of implicitly
locked xchg instruction.  IA32 memory model guarantees that store has
release semantic, since stores cannot pass loads or stores.

Reviewed by:	  bde, jhb
Tested by:	  pho
MFC after:	  2 weeks
2012-06-02 18:10:16 +00:00
Konstantin Belousov
7222d2fbee Inform a compiler which asm statements in the x86 implementation of
atomics change eflags.

Reviewed by:	jhb
MFC after:	2 weeks
2010-12-18 16:41:11 +00:00
Poul-Henning Kamp
065b12a703 Rename an argument from "exp" to "expect" since the former makes FlexeLint
uneasy, in case anybody think it might be exp(3) in libm.

This also makes it consistent with other archs.
2010-05-20 06:18:03 +00:00
Attilio Rao
8448afced8 atomic_cmpset_barr_* was added in order to cope with compilers willing to
specify their own version of atomic_cmpset_* which could have been
different than the membar version.

Right now, however, FreeBSD is bound mostly to GCC-like compilers and
it is desired to add new support and compat shim mostly when there is
a real necessity, in order to avoid too much compatibility bloats.

In this optic, bring back atomic_cmpset_{acq, rel}_* to be the same as
atomic_cmpset_* and unwind the atomic_cmpset_barr_* introduction.

Requested by:	jhb
Reviewed by:	jhb
Tested by:	Giovanni Trematerra <giovanni dot trematerra at
		gmail dot com>
2009-10-09 15:51:40 +00:00
Attilio Rao
d9492a4483 - All the functions in atomic.h needs to be in "physical" form (like
not defined through macros or similar) in order to be later compiled in
  the kernel and offer this way the support for modules (and
  compatibility among the UP case and SMP case).
  Fix this for the newly introduced atomic_cmpset_barr_* cases by defining
  and specifying a template.  Note that the new DEFINE_CMPSET_GEN()
  template save more typing on amd64 than the current code. [1]
- Fix the style for memory barriers on amd64.

[1] Reported by:	Paul B. Mahol <onemda at gmail dot com>
2009-10-06 23:48:28 +00:00
Attilio Rao
86d2e48c22 Per their definition, atomic instructions used in conjuction with
memory barriers should also ensure that the compiler doesn't reorder paths
where they are used.  GCC, however, does that aggressively, even in
presence of volatile operands.  The most reliable way GCC offers for avoid
instructions reordering is clobbering "memory" even if that is
theoretically an heavy-weight operation, flushing the content of all
the registers and forcing reload of them (We could rely, however, on
gcc DTRT by just understanding the purpose as this is a well-known
pattern for many modern operating-systems).

Not all our memory barriers, right now, clobber memory for GCC-like
compilers. The most notable cases are IA32 and amd64 where the memory
barrier are treacted the same as normal atomic instructions.
Fix this by offering the possibility to implement atomic instructions
with memory barriers separately from the normal version and implement
the GCC-like specific one using memory clobbering.
Thanks to Chris Lattner (@apple) for his discussion on llvm specifics.

Reported by:	jhb
Reviewed by:	jhb
Tested by:	rdivacky, Giovanni Trematerra
		<giovanni dot trematerra at gmail dot com>
2009-10-06 13:45:49 +00:00
Kip Macy
db7f0b974f - bump __FreeBSD version to reflect added buf_ring, memory barriers,
and ifnet functions

- add memory barriers to <machine/atomic.h>
- update drivers to only conditionally define their own

- add lockless producer / consumer ring buffer
- remove ring buffer implementation from cxgb and update its callers

- add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to
  allow drivers to efficiently manage multiple hardware queues
  (i.e. not serialize all packets through one ifq)
- expose if_qflush to allow drivers to flush any driver managed queues

This work was supported by Bitgravity Inc. and Chelsio Inc.
2008-11-22 05:55:56 +00:00
Pawel Jakub Dawidek
6eb4157ffc Implement atomic_fetchadd_long() for all architectures and document it.
Reviewed by:	attilio, jhb, jeff, kris (as a part of the uidinfo_waitfree.patch)
2008-03-16 21:20:50 +00:00
Bruce Evans
f28e1c8f99 Fixed some style bugs (mainly assorted errors in comments, and inconsistent
spelling of `result').
2006-12-29 15:29:49 +00:00
Bruce Evans
6c296ffa81 Fixed some style bugs (whitespace only). 2006-12-29 14:28:23 +00:00
Bruce Evans
7e4277e591 Try harder to garbage-collect the "LOCORE" (really asm) version of
MPLOCKED.  The cleaning in rev.1.25 was supposed to have been undone
by rev.1.26, but 1.26 could never have actually affected asm files
since atomic.h is full of C declarations so including it in asm files
would just give syntax errors.  The asm MPLOCKED is even less needed
than when misplaced definitions of it were first removed, and is now
unused in any asm file in the src tree except in anachronismns in
sys/i386/i386/support.s.
2006-12-29 13:36:26 +00:00
Bruce Evans
276c702d8d Removed gratuitous cosmetic differences with the i386 version. This
mainly involves removing all __CC_SUPPORTS___INLINE__ ifdefs.  These
ifdefs are even less needed for amd64 than for i386, but the i386
atomic.h never had them.  The ifdefs here were just an optimization
of obsolescent compatibility cruft (__inline) for a null set of
compilers.  I think null sets of compilers should only be supported
in cases where this is more than an optimization, doesn't require
extensive ifdefs, and only involves not-so-obsolescent compatibility
cruft (plain inline here).
2006-12-28 08:15:14 +00:00
Bruce Evans
26ab2d1d23 Avoid an instruction in atomic_cmpset_{int_long)() in most cases.
These functions are used a lot for mutexes, so this reduces the text
size of an average kernel by about 0.75%.  This wasn't intended to
be a significant optimization, but it somehow increased the maximum
number of packets per second that can be transmitted by my bge hardware
from 320000 to 460000 (this benchmark is CPU-bound and remarkably
sensitive to changes in the text section).

Details: we would prefer to leave the result of the cmpxchg in %al,
but cannot tell gcc that it is there, so we have to convert it to an
integer register.  We converted  to %al, then to %[re]ax, but the
latter step is usually wasted since gcc usually only wants the condition
code and can recover it from %al just as easily as from %[re]ax.  Let
gcc promote %al in the few cases where this is needed.

Nearby style fixes;
- let gcc manage the load of `res', and don't abuse `res' for a copy of `exp'
- don't echo `res's name in comments
- consistently spell the condition code as 'e' after comparison for equality
- don't hard-code %al anywhere except in constraints
- for the version that doesn't use cmpxchg, there is no requirement to use
  %al anywhere, so don't hard-code it in the constraints either.

Style non-fix:
- for the versions that use cmpxchg, keep using "a" (was %[re]ax, now %al)
  for the main output operand, although this is not required.  The input
  and output operands that use the "a" constraint are now decoupled, and
  this makes things clearer except for the reason that the output register
  is hard-coded.  It is now just a hack to tell gcc that the input "a" has
  been clobbered without increasing the number of operands.
2006-12-27 20:26:00 +00:00
John Baldwin
3c2bc2bf26 Add a new atomic_fetchadd() primitive that atomically adds a value to a
variable and returns the previous value of the variable.

Tested on:	i386, alpha, sparc64, arm (cognet)
Reviewed by:	arch@
Submitted by:	cognet (arm)
MFC after:	1 week
2005-09-27 17:39:11 +00:00
John Baldwin
80d52f16da Stop using the '+' constraint modifier with inline assembly. The '+'
constraint is actually only allowed for register operands.  Instead, use
separate input and output memory constraints.

Education from:	alc
Reviewed by:	alc
Tested on:	i386, alpha
MFC after:	1 week
2005-09-15 19:31:22 +00:00
John Baldwin
5d2f4de5da Add aliases for atomic operations on 64-bit integers just like other
64-bit platforms.

MFC after:	1 week
2005-08-18 14:36:47 +00:00
Peter Wemm
9e76f9ad3f Like on i386, bypass lock prefix for atomic ops on !SMP kernels. 2005-07-21 22:35:02 +00:00
John Baldwin
122eceef61 Convert the atomic_ptr() operations over to operating on uintptr_t
variables rather than void * variables.  This makes it easier and simpler
to get asm constraints and volatile keywords correct.

MFC after:	3 days
Tested on:	i386, alpha, sparc64
Compiled on:	ia64, powerpc, amd64
Kernel toolchain busted on:	arm
2005-07-15 18:17:59 +00:00
John Baldwin
48281036d7 Some cleanups and tweaks to some of the atomic.h files in preparation for
further changes and fixes in the future:
- Use aliases via macros rather than duplicated inlines wherever possible.
- Move all the aliases to the bottom of these files and the inline
  functions to the top.
- Add various comments.
- On alpha, drop atomic_{load_acq,store_rel}_{8,char,16,short}().
- On i386 and amd64, don't duplicate the extern declarations for functions
  in the two non-inline cases (KLD_MODULE and compiler doesn't do inlines),
  instead, consolidate those two cases.
- Some whitespace fixes.

Approved by:	re (scottl)
2005-07-09 12:38:53 +00:00
Joerg Wunsch
a5f50ef9e4 netchild's mega-patch to isolate compiler dependencies into a central
place.

This moves the dependency on GCC's and other compiler's features into
the central sys/cdefs.h file, while the individual source files can
then refer to #ifdef __COMPILER_FEATURE_FOO where they by now used to
refer to #if __GNUC__ > 3.1415 && __BARC__ <= 42.

By now, GCC and ICC (the Intel compiler) have been actively tested on
IA32 platforms by netchild.  Extension to other compilers is supposed
to be possible, of course.

Submitted by:	netchild
Reviewed by:	various developers on arch@, some time ago
2005-03-02 21:33:29 +00:00
Peter Wemm
cda078658e Cosmetic and/or trivial sync up with i386.
Approved by:  re (rwatson)
2003-11-21 03:02:00 +00:00
Peter Wemm
0d2a298904 Initial landing of SMP support for FreeBSD/amd64.
- This is heavily derived from John Baldwin's apic/pci cleanup on i386.
- I have completely rewritten or drastically cleaned up some other parts.
  (in particular, bootstrap)
- This is still a WIP.  It seems that there are some highly bogus bioses
  on nVidia nForce3-150 boards.  I can't stress how broken these boards
  are.  I have a workaround in mind, but right now the Asus SK8N is broken.
  The Gigabyte K8NPro (nVidia based) is also mind-numbingly hosed.
- Most of my testing has been with SCHED_ULE.  SCHED_4BSD works.
- the apic and acpi components are 'standard'.
- If you have an nVidia nForce3-150 board, you are stuck with 'device
  atpic' in addition, because they somehow managed to forget to connect the
  8254 timer to the apic, even though its in the same silicon!  ARGH!
  This directly violates the ACPI spec.
2003-11-17 08:58:16 +00:00
Peter Wemm
afa8862328 Commit MD parts of a loosely functional AMD64 port. This is based on
a heavily stripped down FreeBSD/i386 (brutally stripped down actually) to
attempt to get a stable base to start from.  There is a lot missing still.
Worth noting:
- The kernel runs at 1GB in order to cheat with the pmap code.  pmap uses
  a variation of the PAE code in order to avoid having to worry about 4
  levels of page tables yet.
- It boots in 64 bit "long mode" with a tiny trampoline embedded in the
  i386 loader.  This simplifies locore.s greatly.
- There are still quite a few fragments of i386-specific code that have
  not been translated yet, and some that I cheated and wrote dumb C
  versions of (bcopy etc).
- It has both int 0x80 for syscalls (but using registers for argument
  passing, as is native on the amd64 ABI), and the 'syscall' instruction
  for syscalls.  int 0x80 preserves all registers, 'syscall' does not.
- I have tried to minimize looking at the NetBSD code, except in a couple
  of places (eg: to find which register they use to replace the trashed
  %rcx register in the syscall instruction).  As a result, there is not a
  lot of similarity.  I did look at NetBSD a few times while debugging to
  get some ideas about what I might have done wrong in my first attempt.
2003-05-01 01:05:25 +00:00
Jim Pirzyk
77e8341280 Add a knob to turn on and off the CMPXCHG instruction on > i386 IA32 systems.
This is most beneficial for vmware client os installs.

Reviewed by: jmallet, iedowse, tlambert2@mindspring.com
MFC After: never, -STABLE does not currently use this instruction
2002-10-14 19:33:12 +00:00
Mark Murray
4c5aee92a7 Beautify. This has the side effect of improving portability and
making lint work cleaner.

Inspired to do this by:	jhb
2002-07-18 15:56:46 +00:00
Mark Murray
8306a37bbb Clean up the syntax WRT semicolons at the end of function-like-macros, and protect GCCisms from non-GNU compilers and lint. 2002-07-17 16:19:37 +00:00
Bosko Milekic
71acb2477f Make MPLOCKED work again in asm files and stringify it explicitly
where necessary.

Reviewed by: jake
2002-02-28 06:17:05 +00:00
Bruce Evans
d2f22d707a Garbage-collect the "LOCORE" version of MPLOCKED. 2002-02-11 03:41:59 +00:00
John Baldwin
e4e991e117 Allow the ATOMIC_ASM() macro to pass in the constraints on the V parameter
since the char versions need to use either ax, bx, cx, or dx.

Submitted by:	Peter Jeremy (mostly)
Recommended by:	bde
2001-12-18 08:51:34 +00:00
John Baldwin
d900909439 Use newer constraints for atomic_cmpset().
Requested by:	bde
2001-11-12 18:53:45 +00:00
John Baldwin
20a2016a81 Use newer constraints for inline assembly for an operand that is both an
input and an output by using the '+' modifier rather than listing the
operand in both the input and output sections.

Reviwed by:	bde
2001-11-12 16:57:33 +00:00
John Baldwin
2a89a48fbd Allow atomic ops to be somewhat safely used in userland. We always use
lock prefixes in the userland case so that the binaries will work on both
SMP and UP systems.
2001-10-08 20:58:24 +00:00
Mark Murray
d888fc4e73 RIP <machine/lock.h>.
Some things needed bits of <i386/include/lock.h> - cy.c now has its
own (only) copy of the COM_(UN)LOCK() macros, and IMASK_(UN)LOCK()
has been moved to <i386/include/apic.h> (AKA <machine/apic.h>).
Reviewed by:	jhb
2001-02-11 10:44:09 +00:00
John Baldwin
324fffaed1 - Sort of lie and say that %eax is an output only and not an input for the
non-386 atomic_load_acq().  %eax is an input since its value is used in
  the cmpxchg instruction, but we don't care what value it is, so setting
  it to a specific value is just wasteful.  Thus, it is being used without
  being initialized as the warning stated, but it is ok for it to be used
  because its value isn't important.  Thus, we are only sort of lying when
  we say it is an output only operand.
- Add "cc" to the clobber list for atomic_load_acq() since the cmpxchgl
  changes ZF.
2001-01-17 02:15:11 +00:00
John Baldwin
8a6b1c8f54 - Fix atomic_load_* and atomic_store_* to generate functions for atomic.c
that modules can call.
- Remove the old gcc <= 2.8 versions of the atomic ops.
- Resort the order of some things in the file so that there is only
  one #ifdef for KLD_MODULE, and so that all WANT_FUNCTIONS stuff is
  moved to the bottom of the file.
- Remove ATOMIC_ACQ_REL() and just use explicit macros instead.
2001-01-16 00:18:36 +00:00
John Baldwin
9d979d8912 Fix the atomic_load_acq() and atomic_store_rel() functions to properly
implement memory fences for the 486+.  The 386 still uses versions w/o
memory fences as all operations on the 386 are not program ordered.
The 386 versions are not MP safe.
2001-01-14 09:55:21 +00:00
John Baldwin
55d79ad0db The x86 atomic operations are already locked, so they do not need an
additional locked instruction to guarantee a write barrier for the acquire
variants.

Approved by:	dfr
Pointy hat to:	jhb
2000-10-28 00:28:15 +00:00
John Baldwin
ee8f2f372c - Add atomic_cmpset_{acq_,rel_,}_long
- Add in atomic operations for 8-bit, 16-bit, and 32-bit integers
2000-10-25 21:56:16 +00:00
John Baldwin
ccbdd9ee59 - Expand the set of atomic operations to optionally include memory barriers
in most of the atomic operations.  Now for these operations, you can
  use the normal atomic operation, you can use the operation with a read
  barrier, or you can use the operation with a write barrier.  The function
  names follow the same semantics used in the ia64 instruction set.  An
  atomic operation with a read barrier has the extra suffix 'acq', due to
  it having "acquire" semantics.  An atomic operation with a write barrier
  has the extra suffix 'rel'.  These suffixes are inserted between the
  name of the operation to perform and the typename.  For example, the
  atomic_add_int() function now has 3 variants:
  - atomic_add_int() - this is the same as the previous function
  - atomic_add_acq_int() - this function combines the add operation with a
    read memory barrier
  - atomic_add_rel_int() - this function combines the add operation with a
    write memory barrier
- Add 'ptr' to the list of types that we can perform atomic operations
  on.  This allows one to do atomic operations on uintptr_t's.  This is
  useful in the mutex code, for example, because the actual mutex lock is
  a pointer.
- Add two new operations for doing loads and stores with memory barriers.
  The new load operations use a read barrier before the load, and the
  new store operations use a write barrier after the load.  For example,
  atomic_load_acq_int() will atomically load an integer as well as
  enforcing a read barrier.
2000-10-20 07:00:48 +00:00
John Baldwin
b4645202b5 Add atomic_readandclear_int and atomic_readandclear_long. 2000-10-05 22:19:50 +00:00
Poul-Henning Kamp
819e370c3b Introduce atomic_cmpset_int() and atomic_cmpset_long() from SMPng a
few hours earlier than the rest.

The next DEVFS commit needs these functions.

Alpha versions by: dfr
i386 versions by: jakeb

Approved by:    SMPng
2000-09-06 11:21:14 +00:00
David E. O'Brien
25f0103084 When using _asm{} in GCC, one must specify the operand's size if one
specifies the instruction's operation size.  GCC will default to 32-bit
operands reguardless of the prototype (ie, formal parameters' type)
of an inline function.
2000-05-10 01:15:55 +00:00
Peter Wemm
693612eb0b Use the rev 1.1.2.1 code from RELENG_3 for atomic operations rather
than the non-atomic C macros.
1999-10-04 16:24:08 +00:00
Peter Wemm
b87104737a Typo: s/__GNUC_MINOR_/__GNUC_MINOR__/
(__GNUC_MINOR__ on egcs in -current is "91" and is going to be "95" soon)
1999-10-04 16:18:04 +00:00
Eivind Eklund
181d21375b Allow compilation with older versions of GCC, in order to make it possible
to bootstrap and work with -current from older versions of FreeBSD.
1999-10-03 21:15:25 +00:00
Peter Wemm
c3aac50f28 $Id$ -> $FreeBSD$ 1999-08-28 01:08:13 +00:00
Alan Cox
08c40841d8 Create callable (non-inline) versions of the atomic_OP_TYPE functions
that are linked into the kernel.  The KLD compilation options are
changed to call these functions, rather than in-lining the
atomic operations.

This approach makes atomic operations from KLDs significantly
faster on UP systems (though somewhat slower on SMP systems).

PR:		i386/13111
Submitted by:	peter.jeremy@alcatel.com.au
1999-08-18 04:08:31 +00:00
Alan Cox
03e3bc8e62 atomic.h:
Change "void *" to "volatile TYPE *", improving type safety
	and eliminating some warnings (e.g., mp_machdep.c rev 1.106).

cpufunc.h:
	Eliminate setbits.  As defined, it's not precisely correct;
	and it's redundant.  (Use atomic_set_int instead.)

ipl_funcs.c:
	Use atomic_set_int instead of setbits.

systm.h:
	Include atomic.h.

Reviewed by:	bde
1999-07-23 23:45:50 +00:00
Alan Cox
47b8bc92e8 Commit the correct patch, i.e., the one that actually corresponds
to the rev 1.2 log entry.
1999-07-13 06:35:25 +00:00