Commit Graph

219 Commits

Author SHA1 Message Date
Mark Johnston
d6c8335623 Add get_cpu() and put_cpu().
MFC after:	1 week
2017-05-21 00:06:36 +00:00
Mark Johnston
c12488bbe0 Add on_each_cpu() and wbinvd_on_all_cpus().
Reviewed by:	hselasky
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D10550
2017-05-01 16:32:28 +00:00
Hans Petter Selasky
a1be2ead3a Use __typeof() instead of typeof() in some RCU related macros in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-19 13:04:34 +00:00
Hans Petter Selasky
f3de9af633 Fix problem regarding priority inversion when using the concurrency
kit, CK, in the LinuxKPI.

When threads are pinned to a CPU core or when there is only one CPU,
it can happen that a higher priority thread can call the CK
synchronize function while a lower priority thread holds the read
lock. Because the CK's synchronize is a simple wait loop this can lead
to a deadlock situation. To solve this problem use the recently
introduced CK's wait callback function.

When detecting a CK blocking condition figure out the lowest priority
among the blockers and update the calling thread's priority and
yield. If another CPU core is holding the read lock, pin the thread to
the blocked CPU core and update the priority. The calling threads
priority and CPU bindings are restored before return.

If a thread holding a CK read lock is detected to be sleeping, pause()
will be used instead of yield().

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-19 13:03:29 +00:00
Gleb Smirnoff
9ed01c32e0 All these files need sys/vmmeter.h, but now they got it implicitly
included via sys/pcpu.h.
2017-04-17 17:07:00 +00:00
Hans Petter Selasky
76fe8c9330 Fix compilation of LinuxKPI for PowerPC.
Found by:		emaste @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-09 14:31:41 +00:00
Hans Petter Selasky
22cbd6ef2e Create the LinuxKPI current task structure on the fly if it doesn't
exist when the current macro is used.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-07 14:43:28 +00:00
Hans Petter Selasky
99e690772a The __stringify() macro in the LinuxKPI should expand any macros
before stringifying.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-07 12:27:49 +00:00
Hans Petter Selasky
c9dd0b48c9 Cleanup the bitmap_xxx() functions in the LinuxKPI:
- Move all bitmap related functions from bitops.h to bitmap.h, similar
  to what Linux does.

- Apply some minor code cleanup and simplifications to optimize the
  generated code when using static inline functions.

- Implement the following list of bitmap functions which are needed by
  drm-next and ibcore:
  - bitmap_find_next_zero_area_off()
  - bitmap_find_next_zero_area()
  - bitmap_or()
  - bitmap_and()
  - bitmap_xor()

- Add missing include directives to the qlnxe driver
  (davidcs@ has been notified)

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 13:30:31 +00:00
Hans Petter Selasky
480e2fd3d5 Define VM_READ, VM_WRITE and VM_EXEC in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 10:52:52 +00:00
Hans Petter Selasky
fc51649e31 Implement need_resched() in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 10:29:54 +00:00
Hans Petter Selasky
8402f058ad Fix implementation of task_pid_group_leader() in the LinuxKPI.
In FreeBSD thread IDs and procedure IDs have distinct number
spaces. When asking for the group leader task ID in the LinuxKPI,
return the procedure ID and let this resolve to the first task in the
procedure having a valid LinuxKPI task structure pointer.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 10:26:03 +00:00
Hans Petter Selasky
1ea4c85781 Implement proper support for memory map operations in the LinuxKPI,
like open, close and fault using the character device pager.

Some notes about the implementation:

1) Linux drivers set the vm_ops and vm_private_data fields during a
mmap() call to indicate that the driver wants to use the LinuxKPI VM
operations. Else these operations are not used.

2) The vm_private_data pointer is associated with a VM area structure
and inserted into an internal LinuxKPI list. If the vm_private_data
pointer already exists, the existing VM area structure is used instead
of the allocated one which gets freed.

3) The LinuxKPI's vm_private_data pointer is used as the callback
handle for the FreeBSD VM object. The VM subsystem in FreeBSD has a
similar list to identify equal handles and will only call the
character device pager's close function once.

4) All LinuxKPI VM operations are serialized through the mmap_sem
sempaphore, which is per procedure, which prevents simultaneous access
to the shared VM area structure when receiving page faults.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 09:34:54 +00:00
Hans Petter Selasky
0bd68b774d Implement down_write_killable() in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-05 12:10:02 +00:00
Hans Petter Selasky
0a475c59a9 Implement vmalloc_32() in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-27 17:18:04 +00:00
Hans Petter Selasky
8186b52744 Implement a series of physical page management related functions in
the LinuxKPI for accessing user-space memory in the kernel.

Add functions to hold and wire physical page(s) based on a given range
of user-space virtual addresses.

Add functions to get and put a reference on, wire, hold, mark
accessed, copy and dirty a physical page.

Add new VM related structures and defines as a preparation step for
advancing the memory map capabilities of the LinuxKPI.

Add function to figure out if a virtual address was allocated using
malloc().

Add function to convert a virtual kernel address into its physical
page pointer.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-27 17:04:11 +00:00
Hans Petter Selasky
8f7eee5a63 Use ppsratecheck() for ratelimiting in the LinuxKPI.
Suggested by:		cem @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 16:23:55 +00:00
Hans Petter Selasky
d2f312e0e7 Add proper error checking for the string to number conversion
functions in the LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 16:01:51 +00:00
Hans Petter Selasky
8293738e64 Function macros are preferred in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 13:28:16 +00:00
Hans Petter Selasky
e9db3df254 Add support for ratelimited printouts in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 10:48:10 +00:00
Mark Johnston
e5fe3ae2b4 Extend cmpxchg() to support 8- and 16-bit values, and add xchg().
These are needed to support updated revisions of the DRM code.

Reviewed by:	hselasky (previous version)
MFC after:	2 weeks
2017-03-22 17:33:57 +00:00
Hans Petter Selasky
82d0140707 Add full VNET support to the inet_get_local_port_range() function in
the LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-22 15:46:31 +00:00
Hans Petter Selasky
303bd80ad5 Add support for more IPv4 and IPv6 related macros in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-22 15:44:00 +00:00
Hans Petter Selasky
a0699ebf77 Implement get_pid_task(), pid_task() and some other PID helper
functions in the LinuxKPI. Add a usage atomic to the task_struct
structure to facilitate refcounting the task structure when returned
from get_pid_task(). The get_task_struct() and put_task_struct()
function is used to manage atomic refcounting. After this change the
task_struct should only be freed through put_task_struct().

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-17 15:40:24 +00:00
Hans Petter Selasky
05d4f501dc Implement minimalistic memory mapping structure, struct mm_struct, and
some associated helper functions in the LinuxKPI. Let the existing
linux_alloc_current() function allocate and initialize the new
structure and let linux_free_current() drop the refcount on the memory
mapping structure. When the mm_struct's refcount reaches zero, the
structure is freed.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-17 10:30:06 +00:00
Hans Petter Selasky
0a2f4606bb Add comment describing the use of pagefault_disable() and
pagefault_enable() in the LinuxKPI.

Suggested by:  		rpokala@
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-17 08:02:46 +00:00
Hans Petter Selasky
810ea5b270 The LinuxKPI pagefault disable and enable functions can only be used
pairwise to support the FreeBSD way of pushing and popping the page
fault flags. Ensure this by requiring every occurrence of pagefault
disable function call to have a corresponding pagefault enable call.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 16:53:22 +00:00
Hans Petter Selasky
0e05589b39 Implement more userspace memory access functions in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 16:49:27 +00:00
Hans Petter Selasky
b8a8ed7c96 Define some more LinuxKPI task related macros.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 12:33:34 +00:00
Hans Petter Selasky
8300fb13dd Add helper function similar to ip_dev_find() to the LinuxKPI to lookup
a network device by its IPv6 address in the given VNET.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 10:02:45 +00:00
Hans Petter Selasky
404027276b Add basic support for VIMAGE to the LinuxKPI and ibcore.
Support is implemented by mapping Linux's "struct net" into FreeBSD's
"struct vnet". Currently only vnet0 is supported by ibcore.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 09:59:35 +00:00
Hans Petter Selasky
f73286645f Fix implementation of the DECLARE_WORK() macro in the LinuxKPI to fully
initialize the declared work structure and not only the function callback
pointer.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-09 18:37:17 +00:00
Hans Petter Selasky
9760ac0a3e Implement support for mutexes with deadlock avoidance in the LinuxKPI.
When locking a mutex and deadlock is detected the first mutex lock
call that sees the deadlock will return -EDEADLK .

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-09 18:33:40 +00:00
Hans Petter Selasky
4f688d19fb Cleanup the LinuxKPI mutex wrappers.
Add support for using mutexes during KDB and shutdown. This is also
required for doing mode-switching during panic for drm-next.

Add new mutex functions mutex_init_witness() and mutex_destroy()
allowing LinuxKPI mutexes to be tracked by witness.

Declare mutex_is_locked() and mutex_is_owned() like inline functions
to get cleaner warnings. These functions are used inside WARN_ON()
statements which might look a bit odd if these functions get fully
expanded.

Give mutexes better debug names through the mutex_name() macro when
WITNESS_ALL is defined. The mutex_name() macro can prefix parts of the
filename and line number before the mutex name.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-09 17:01:00 +00:00
Hans Petter Selasky
14c5024db8 Cleanup the LinuxKPI slab implementation.
Put large functions into linux_slab.c instead of declaring them static
inline.

Add support for more memory allocation wrappers like kmalloc_array()
and __vmalloc().

Make sure either the M_WAITOK or the M_NOWAIT flag is set and mask
away unused memory allocation flags before calling FreeBSD's malloc()
routine.

Move kmalloc_node() definition to slab.h where it belongs.

Implement support for the SLAB_DESTROY_BY_RCU feature when creating a
kmem_cache which basically means kmem_cache memory is freed using
call_rcu().

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-08 11:09:27 +00:00
Hans Petter Selasky
0e3bbe9197 Implement eth_zero_addr() in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-08 09:53:20 +00:00
Hans Petter Selasky
a767c1883d Add support for constant pointer constructs to READ_ONCE() in the
LinuxKPI. When the type of the argument is constant the temporary
variable cannot be assigned after the barrier. Instead assign the
temporary variable by initialization.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-07 20:24:34 +00:00
Hans Petter Selasky
249a42207b Implement time_is_after_eq_jiffies() function in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-07 15:37:51 +00:00
Hans Petter Selasky
661a318c83 Fix implementation of the DECLARE_RWSEM() macro in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-07 15:34:49 +00:00
Hans Petter Selasky
dc0d19dd4a Make sure jiffies value is cast to an integer in the LinuxKPI before
doing millisecond conversion. Under FreeBSD jiffies are 32-bit.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-07 15:33:38 +00:00
Hans Petter Selasky
ca2ad6bd77 LinuxKPI workqueue cleanup.
This change makes the workqueue implementation behave more like in
Linux, both functionality wise and structure wise.

All workqueue code has been moved to linux_work.c

Add an atomic based statemachine to the work_struct to ensure proper
operation. Prior to this change struct_work was directly mapped to a
FreeBSD task. When a taskqueue has multiple threads the same task may
end up being executed on more than one worker thread simultaneously.
This might cause problems with code coming from Linux, which expects
serial behaviour, similar to Linux tasklets.

Move all global workqueue function names into the linux_xxx domain to
avoid symbol name clashes in the future.

Implement a few more workqueue related functions and macros.

Create two multithreaded taskqueues for the LinuxKPI during module
load, one for time-consuming callbacks and one for non-time consuming
callbacks.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-07 12:09:14 +00:00
Hans Petter Selasky
def277d3ef Implement add_timer_on() function in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-06 14:56:57 +00:00
Hans Petter Selasky
19bf8ef562 Implement DECLARE_RWSEM() macro in the LinuxKPI to initialize a
Read-Write semaphore during module init time.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-06 12:22:05 +00:00
Hans Petter Selasky
684bcfec89 Give LinuxKPI Read-Write semaphores better debug names when
WITNESS_ALL is defined. The lock name is based on the filename and
line number where the initialisation happens.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-06 12:20:56 +00:00
Hans Petter Selasky
e0db0ddb39 Remove duplicate prototype in the LinuxKPI to fix compilation warning.
Reported by:		emaste @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-04 20:06:47 +00:00
Hans Petter Selasky
1f827dab9e Update the LinuxKPI RCU and SRCU wrappers for the concurrency kit, CK.
- Optimise the RCU implementation to not allocate and free
ck_epoch_records during runtime. Instead allocate two sets of
ck_epoch_records per CPU for general purpose use. The first set is
only used for reader locks and the second set is only used for
synchronization and barriers and is protected with a regular mutex to
prevent simultaneous issues.

- Move the task structure away from the rcu_head structure and into
the per-CPU structures. This allows the size of the rcu_head structure
to be reduced down to the size of two pointers.

- Fix a bug where the linux_rcu_barrier() function only waited for one
per-CPU epoch record to be completed instead of all.

- Use a critical section or a mutex to protect ck_epoch_begin() and
ck_epoch_end() depending on RCU or SRCU type. All the ck_epoch_xxx()
functions, except ck_epoch_register(), ck_epoch_unregister() and
ck_epoch_recycle() are not re-entrant and needs a critical section or
a mutex to operate in the LinuxKPI, after inspecting the CK
implementation of the above mentioned functions. The simultaneous
issues arise from per-CPU epoch records being shared between multiple
threads depending on the amount of taskswitching and how many threads
are involved with the RCU and SRCU operations.

- Properly free all epoch records by using safe list traversal at
LinuxKPI module unload. It turns out the ck_epoch_recycle() always
have the records on an internal list and use a flag in the epoch
record to track allocated and free entries. This would lead to use
after free during module unload.

- Remove redundant synchronize_rcu() call from the
linux_compat_uninit() function. Let the linux_rcu_runtime_uninit()
function do the final rcu_barrier() instead.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-03 16:28:03 +00:00
Hans Petter Selasky
3cfeca84b4 Implement more bit operation functions in the LinuxKPI.
Some minor whitespace nits while at it.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-27 14:38:17 +00:00
Hans Petter Selasky
522f4b2c75 Define __sum16 type in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-27 13:59:02 +00:00
Hans Petter Selasky
0f32531a56 Implement more string functions in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-24 17:36:55 +00:00
Hans Petter Selasky
5a5a8c8a17 Prototype device structure to ensure LinuxKPI header file can be
included standalone.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-24 17:03:14 +00:00
Hans Petter Selasky
959d6165a2 Implement srcu_dereference() macro in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-24 14:40:15 +00:00
Hans Petter Selasky
797046eebb Implement BIT_ULL() macro in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-24 14:23:46 +00:00
Hans Petter Selasky
cffaf933d7 Implement __test_and_clear_bit() and __test_and_set_bit() in the LinuxKPI.
The clang compiler will optimise these functions down to three AMD64
instructions if the bit argument is a constant during compilation.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-23 09:53:54 +00:00
Hans Petter Selasky
72ebbe00b3 Convert magic values into macros in the LinuxKPI scatterlist
implementation.

Suggested by:		cem @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-22 20:24:09 +00:00
Hans Petter Selasky
1cdefd084d Optimise unmapped LinuxKPI page allocations.
When allocating unmapped pages, take advantage of the direct map on
AMD64 to get the virtual address corresponding to a page. Else all
pages allocated must be mapped because sometimes the virtual address
of a page is requested.

Move all page allocation and deallocation code into an own C-file.

Add support for GFP_DMA32, GFP_KERNEL, GFP_ATOMIC and __GFP_ZERO
allocation flags.

Make a clear separation between mapped and unmapped allocations.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-22 19:39:54 +00:00
Hans Petter Selasky
8306998f5b Improve LinuxKPI scatter list support.
The i915kms driver in Linux 4.9 reimplement parts of the scatter list
functions with regards to performance. In other words there is not so
much room for changing structure layouts and functionality if the
i915kms should be built AS-IS. This patch aligns the scatter list
support to what is expected by the i915kms driver. Remove some
comments not needed while at it.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-22 19:31:02 +00:00
Hans Petter Selasky
1a01b4e566 Replace dummy implementation of RCU in the LinuxKPI with one based on
the in-kernel concurrency kit's ck_epoch API. Factor RCU hlist_xxx()
functions into own rculist.h header file.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-21 18:04:21 +00:00
Hans Petter Selasky
e560eab72c Streamline the LinuxKPI spinlock wrappers.
1) Add better spinlock debug names when WITNESS_ALL is defined.

2) Make sure that the calling thread gets bound to the current CPU
while a spinlock is locked. Some Linux kernel code depends on that the
CPU ID doesn't change while a spinlock is locked.

3) Add support for using LinuxKPI spinlocks during a panic().

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-21 14:22:14 +00:00
Hans Petter Selasky
ef23481a79 Add support for LinuxKPI tasklets.
Tasklets are implemented using a taskqueue and a small statemachine on
top. The additional statemachine is required to ensure all LinuxKPI
tasklets get serialized. FreeBSD taskqueues do not guarantee
serialisation of its tasks, except when there is only one worker
thread configured.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-21 13:23:53 +00:00
Hans Petter Selasky
1e3db1de0c Make the LinuxKPI task struct persistent accross system calls.
A set of helper functions have been added to manage the life of the
LinuxKPI task struct. When an external system call or task is invoked,
a check is made to create the task struct by demand. A thread
destructor callback is registered to free the task struct when a
thread exits to avoid memory leaks.

This change lays the ground for emulating the Linux kernel more
closely which is a dependency by the code using the LinuxKPI APIs.

Add new dedicated td_lkpi_task field has been added to struct thread
instead of abusing td_retval[1].

Fix some header file inclusions to make LINT kernel build properly
after this change.

Bump the __FreeBSD_version to force a rebuild of all kernel modules.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-21 12:43:02 +00:00
Hans Petter Selasky
269d8c86e9 Implement GFP_DMA32 flag in the LinuxKPI.
Define all FreeBSD native GFP bits as GFP_NATIVE_MASK.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-17 13:31:11 +00:00
Hans Petter Selasky
ddad2785bc Allow container_of() to be used with constant data pointers.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-16 14:13:36 +00:00
Hans Petter Selasky
622f2291e8 Implement more LinuxKPI atomic functions and macros.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-16 12:56:10 +00:00
Hans Petter Selasky
28a04a26b2 Allow passing a constant atomic_t to atomic_read().
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-16 12:20:57 +00:00
Hans Petter Selasky
13459eb4a3 Whitespace fix.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-02-16 12:08:52 +00:00
Baptiste Daroussin
b4b4b5304b Revert crap accidentally committed 2017-01-28 16:31:23 +00:00
Baptiste Daroussin
814aaaa7da Revert r312923 a better approach will be taken later 2017-01-28 16:30:14 +00:00
Mateusz Guzik
21b737495b Introduce __read_mostly and __exclusive_cache_line macros.
The intended use is to annotate frequently used globals which either rarely
change (and thus can be grouped in the same cacheline) or are an atomic counter
(which means it may benefit from being the only variable in the cacheline).

Linker script support is provided only for amd64. Architectures without it risk
having other variables put in, i.e. as if they were not annotated. This is
harmless from correctness point of view.

Reviewed by:	bde (previous version)
MFC after:	1 month
2017-01-27 14:53:09 +00:00
Hans Petter Selasky
a11bac7379 Implement more list header file functions.
Add definition guard for the list_head structure.

Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-12-26 10:41:51 +00:00
Hans Petter Selasky
70a3cc597a Fix LINT build.
Found by:	mmel @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-12-26 10:03:33 +00:00
Hans Petter Selasky
1125dbc049 Implement register and unregister chrdev in the LinuxKPI.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-12-26 01:18:07 +00:00
Hans Petter Selasky
a1410999f4 Use correct integer type when computing the maximum physical address
for kmem_alloc_contig().

Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-12-25 21:41:40 +00:00
Hans Petter Selasky
03adb29e0d Improve LinuxKPI device support. Only delete own BSD devices and not
the ones obtained through devclass_get_device(). Some minor code
cleanups while at it.

Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-12-25 19:49:09 +00:00
Conrad Meyer
2f02a9e15a linuxkpi: Fix not-found case of linux_pci_find_irq_dev
Linux list_for_each_entry() does not neccessarily end with the iterator
NULL (it may be an offset from NULL if the list member is not the first
element of the member struct).

Reported by:	Coverity
CID:		1366940
Reviewed by:	hselasky@
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D8780
2016-12-13 19:58:21 +00:00
Ed Schouten
d6d7df8aa0 Remove the only user of sysctl_add_oid().
My plan is to change this function's prototype at some point in the
future to add a new label argument, which can be used to export all of
sysctl as metrics that can be scraped by Prometheus. Switch over this
caller to use the macro wrapper counterpart.
2016-12-13 07:58:30 +00:00
Hans Petter Selasky
12af734d32 Add more LinuxKPI PCI definitions.
Obtained from:	kmacy @
Sponsored by:	Mellanox Technologies
MFC after:	1 week
2016-12-09 15:05:09 +00:00
Hans Petter Selasky
1724ded49c Prefer function macros over regular macros in the LinuxKPI.
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-12-09 15:01:37 +00:00
Hans Petter Selasky
be48ab92ac Avoid malloc() warnings when using the LinuxKPI by zero-checking
the allocation flags.

Obtained from:		kmacy @
Sponsored by:           Mellanox Technologies
MFC after:		1 week
2016-12-09 14:06:22 +00:00
Hans Petter Selasky
0a61267a99 MSIX can support more than 256 IRQs. Make sure the invalid IRQ number
set in the LinuxKPI is big enough.

Sponsored by:           Mellanox Technologies
MFC after:		1 week
2016-12-09 13:53:31 +00:00
Hans Petter Selasky
e996b07c72 Prefix some _pci_xxx() functions in the Linux KPI with linux_ and make
sure the IRQ number used by these functions is unsigned.

Sponsored by:           Mellanox Technologies
MFC after:		1 week
2016-12-09 13:47:50 +00:00
Hans Petter Selasky
7fdce5c42b Prefix the Linux KPI's kmem_xxx() functions with linux_ to avoid
conflict with the opensolaris kernel module.

This patch solves a problem where the kernel linker will incorrectly
resolve opensolaris kmem_xxx() functions as linuxkpi ones, which leads
to a panic when these functions are used.

Submitted by:		gallatin @
Sponsored by:           Mellanox Technologies
MFC after:		1 week
2016-12-09 13:41:26 +00:00
Sepherosa Ziehau
6faefea03b linuxkpi: Fix PCI BAR lazy allocation support.
FreeBSD supports lazy allocation of PCI BAR, that is, when a device
driver's attach method is invoked, even if the device's PCI BAR
address wasn't initialized, the invocation of bus_alloc_resource_any()
(the call chain: pci_alloc_resource() -> pci_alloc_multi_resource() ->
pci_reserve_map() -> pci_write_bar()) would allocate a proper address
for the PCI BAR and write this 'lazy allocated' address into the PCI
BAR.

This model works fine for native FreeBSD device drivers, but _not_ for
device drivers shared with Linux (e.g. dev/mlx5/mlx5_core/mlx5_main.c
and ofed/drivers/net/mlx4/main.c.  Both of them use
pci_request_regions(), which doesn't work properly with the PCI BAR
lazy allocation, because pci_resource_type() -> _pci_get_rle() always
returns NULL, so pci_request_regions() doesn't have the opportunity to
invoke bus_alloc_resource_any().  We now use pci_find_bar() in
pci_resource_type(), which is able to locate all available PCI BARs
even if some of them will be lazy allocated.

Submitted by:	Dexuan Cui <decui microsoft com>
Reviewed by:	hps
MFC after:	1 week
Sponsored by:	Microsoft
Differential Revision:	https://reviews.freebsd.org/D8071
2016-09-30 05:51:11 +00:00
Hans Petter Selasky
bdff61f849 The IORESOURCE_XXX defines should resemble a bitmask while SYS_RES_XXX
are not bitmasks. Fix return value of pci_resource_flags() to reflect
this change.

MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-09-29 14:35:32 +00:00
Hans Petter Selasky
d8571d3ea3 Fallback to arc4rand() in the LinuxKPI when read_random() returns
zero. This can happen for virtual machines.

MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-06-07 13:10:13 +00:00
Hans Petter Selasky
8eeb3e1773 The SCHEDULER_STOPPED() macro already contains a predict false statement.
Remove superfluous unlikely() wrapper.

Suggested by:	glebius
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-27 07:33:49 +00:00
Hans Petter Selasky
2bb46d5516 Define ATOMIC_LONG_INIT() in the LinuxKPI.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-26 10:03:22 +00:00
Hans Petter Selasky
06ca64ecf3 Add support for runtime modifiable module parameters in the LinuxKPI.
Linux module parameters have a permissions value. If any write bits
are set we are allowed to modify the module parameter runtime. Reflect
this when creating the static SYSCTL nodes.

Sponsored by:	Mellanox Technologies
MFC after:	1 week
2016-05-26 09:04:14 +00:00
Hans Petter Selasky
707324edee Add more module parameter macros to the LinuxKPI.
Obtained from:	kmacy @
Sponsored by:	Mellanox Technologies
2016-05-26 08:47:06 +00:00
Hans Petter Selasky
0bb3dd300b Add support for boolean module parameters in the LinuxKPI.
Requested by:	kmacy @
Sponsored by:	Mellanox Technologies
2016-05-26 08:44:11 +00:00
Hans Petter Selasky
1d9b99e5e3 Implement Linux module parameters as read-only tunable SYSCTLs.
Bool module parameters are no longer supported, because there is no
equivalent in FreeBSD.

There are two macros available which control the behaviour of the
LinuxKPI module parameters:

- LINUXKPI_PARAM_PARENT allows the consumer to set the SYSCTL parent
where the modules parameters will be created.

- LINUXKPI_PARAM_PREFIX defines a parameter name prefix, which is
  added to all created module parameters.

Sponsored by:	Mellanox Technologies
MFC after:	1 week
2016-05-25 12:12:14 +00:00
Hans Petter Selasky
8571421886 Add checks for SCHEDULER_STOPPED() so that code using the LinuxKPI can
run after a panic(). This for example allows a LinuxKPI based graphics
stack to receive prints during a panic.

Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-25 09:04:06 +00:00
Kevin Lo
8636496407 Add __iowrite32_copy() to the Linux kernel compatibility layer.
Reviewed by:	hselasky
2016-05-24 09:23:04 +00:00
Hans Petter Selasky
3ce1263063 Set "current" for all PCI enumeration callbacks.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-24 07:46:20 +00:00
Hans Petter Selasky
5a6748b2cf Use make_dev_s() instead of make_dev() to avoid race setting
"si_drv1". Convert panic() into regular error while at it.

Suggested by:	jhb @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-24 07:06:04 +00:00
Hans Petter Selasky
44701cf732 Implement "atomic_long_add_unless()" in the LinuxKPI and fix the
implementation of "atomic_long_inc_not_zero()".

Found by:	ngie @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-23 16:19:51 +00:00
Hans Petter Selasky
8b68f2509f A missing definition needed by ktime_to_ms().
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-23 13:19:20 +00:00
Hans Petter Selasky
425da8eb61 Fix some data types and add "inline" keyword for __reg_op() function.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-23 13:18:15 +00:00
Hans Petter Selasky
83cfd83419 Implement ror32() in the LinuxKPI.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-23 12:53:17 +00:00
Hans Petter Selasky
fb2faed84e Define more copy to/from userspace functions in the LinuxKPI.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-23 12:52:22 +00:00
Hans Petter Selasky
aef2a67b83 Add more printf() related functions to the LinuxKPI.
Obtained from:	kmacy @
MFC after:	1 week
Sponsored by:	Mellanox Technologies
2016-05-23 12:35:07 +00:00