Commit Graph

206 Commits

Author SHA1 Message Date
Hans Petter Selasky
dff36e69a1 Implement in_atomic() function in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 15:05:44 +00:00
Hans Petter Selasky
90b30e6560 Properly set the .d_name field in the cdevsw structure for the
LinuxKPI.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 13:11:06 +00:00
Hans Petter Selasky
d56f1ed887 Make sure the VMAP's "vm_file" field is referenced in a Linux
compatible way by the linux_dev_mmap_single() function in the
LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 13:07:05 +00:00
Hans Petter Selasky
cca15f28c5 Remove the VMA handle from its list before calling the LinuxKPI VMA
close operation to prevent other threads from reusing the VM object
handle pointer.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 13:05:54 +00:00
Hans Petter Selasky
68b9f2f00c Don't acquire a reference on the VM-space when allocating the LinuxKPI
task structure to avoid deadlock when tearing down the VM object
during a process exit.

Found by:		markj @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 13:01:27 +00:00
Hans Petter Selasky
ea67550be0 Fix a reference count leak in the LinuxKPI due to calling VM open when
it shouldn't be called.

Background:
The Linux VM open operation is called when a new VMA is
created on top of the current VMA. This is done through either mremap
flow or split_vma, usually due to mlock, madvise, munmap and so
on. This is currently not supported by the LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 12:08:25 +00:00
Hans Petter Selasky
f5a9867b7d Fixes for refcounting "struct linux_file" in the LinuxKPI.
- Allow "struct linux_file" to be refcounted when its "_file" member
  is NULL by using its "f_count" field. The reference counts are
  transferred to the file structure when the file descriptor is
  installed.

- Add missing vdrop() calls for error cases during open().

- Set the "_file" member of "struct linux_file" during open. This
allows use of refcounting through get_file() and fput() with LinuxKPI
character devices.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 12:02:59 +00:00
Hans Petter Selasky
3f743d782a Make sure the thread's priority is restored for all three cases inside
linux_synchronize_rcu_cb() in the LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-31 10:01:15 +00:00
Mark Johnston
cb564d2436 Add some miscellaneous definitions to support DRM drivers.
Reviewed by:	hselasky
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D10985
2017-05-30 17:16:08 +00:00
Mark Johnston
d6c8335623 Add get_cpu() and put_cpu().
MFC after:	1 week
2017-05-21 00:06:36 +00:00
Mark Johnston
02fb845bbf Fix a few uses of kern_yield() in the TTM and the LinuxKPI.
kern_yield(0) effectively causes the calling thread to be rescheduled
immediately since it resets the thread's priority to the highest possible
value. This can cause livelocks when the pattern
"while (!trylock()) kern_yield(0);" is used since the thread holding the
lock may linger on the runqueue for the CPU on which the looping thread is
running.

MFC after:	1 week
2017-05-18 18:35:14 +00:00
Hans Petter Selasky
d8e073a985 Fix init order in the LinuxKPI for RCU support.
CPU_FOREACH() is not available until SI_SUB_CPU at SI_ORDER_ANY
when the LinuxKPI is loaded as part of the kernel.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-09 12:51:42 +00:00
Mark Johnston
6c10623340 Use pmap_invalidate_cache() to implement wbinvd_on_all_cpus().
Suggested by:	jhb
X-MFC with:	r317651
2017-05-05 17:22:00 +00:00
Hans Petter Selasky
6796081682 Fix for use after free in the LinuxKPI.
Background:
The same VM object might be shared by multiple processes and the
mm_struct is usually freed when a process exits.

Grab a reference on the mm_struct while the vmap is in the
linux_vma_head list in case the first process which inserted a VM
object has exited.

Tested by:		kwm @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-05-05 14:09:44 +00:00
Mark Johnston
c12488bbe0 Add on_each_cpu() and wbinvd_on_all_cpus().
Reviewed by:	hselasky
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D10550
2017-05-01 16:32:28 +00:00
Hans Petter Selasky
a8c348db51 Prefer to use real virtual address over direct map address in the
linux_page_address() function in the LinuxKPI. This solves an issue
where the return value from linux_page_address() is passed to
kmem_free().

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-27 14:29:21 +00:00
Mark Johnston
b602c283b3 Drop Giant before sleeping in linux_wait_for_{timeout_,}common().
Reported and tested by:	Pete Wright <pete@nomadlogic.org>
Reviewed by:	hselasky (previous version)
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D10414
2017-04-19 16:12:02 +00:00
Hans Petter Selasky
a1be2ead3a Use __typeof() instead of typeof() in some RCU related macros in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-19 13:04:34 +00:00
Hans Petter Selasky
f3de9af633 Fix problem regarding priority inversion when using the concurrency
kit, CK, in the LinuxKPI.

When threads are pinned to a CPU core or when there is only one CPU,
it can happen that a higher priority thread can call the CK
synchronize function while a lower priority thread holds the read
lock. Because the CK's synchronize is a simple wait loop this can lead
to a deadlock situation. To solve this problem use the recently
introduced CK's wait callback function.

When detecting a CK blocking condition figure out the lowest priority
among the blockers and update the calling thread's priority and
yield. If another CPU core is holding the read lock, pin the thread to
the blocked CPU core and update the priority. The calling threads
priority and CPU bindings are restored before return.

If a thread holding a CK read lock is detected to be sleeping, pause()
will be used instead of yield().

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-19 13:03:29 +00:00
Hans Petter Selasky
7a742c41cf Zero number of CPUs should be translated into the default number of
CPUs when allocating a LinuxKPI workqueue. This also ensures that the
created taskqueue always have a non-zero number of worker threads.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-19 11:38:07 +00:00
Gleb Smirnoff
9ed01c32e0 All these files need sys/vmmeter.h, but now they got it implicitly
included via sys/pcpu.h.
2017-04-17 17:07:00 +00:00
Gleb Smirnoff
6286dc78d4 Remove unneeded include of vm_phys.h. 2017-04-17 16:51:04 +00:00
Olivier Houchard
7e8cd4e1af Import CK as of commit 6b141c0bdd21ce8b3e14147af8f87f22b20ecf32
This brings us changes we needed in ck_epoch.
2017-04-09 21:02:05 +00:00
Hans Petter Selasky
76fe8c9330 Fix compilation of LinuxKPI for PowerPC.
Found by:		emaste @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-09 14:31:41 +00:00
Hans Petter Selasky
22cbd6ef2e Create the LinuxKPI current task structure on the fly if it doesn't
exist when the current macro is used.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-07 14:43:28 +00:00
Hans Petter Selasky
99e690772a The __stringify() macro in the LinuxKPI should expand any macros
before stringifying.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-07 12:27:49 +00:00
Hans Petter Selasky
c9dd0b48c9 Cleanup the bitmap_xxx() functions in the LinuxKPI:
- Move all bitmap related functions from bitops.h to bitmap.h, similar
  to what Linux does.

- Apply some minor code cleanup and simplifications to optimize the
  generated code when using static inline functions.

- Implement the following list of bitmap functions which are needed by
  drm-next and ibcore:
  - bitmap_find_next_zero_area_off()
  - bitmap_find_next_zero_area()
  - bitmap_or()
  - bitmap_and()
  - bitmap_xor()

- Add missing include directives to the qlnxe driver
  (davidcs@ has been notified)

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 13:30:31 +00:00
Hans Petter Selasky
480e2fd3d5 Define VM_READ, VM_WRITE and VM_EXEC in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 10:52:52 +00:00
Hans Petter Selasky
fc51649e31 Implement need_resched() in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 10:29:54 +00:00
Hans Petter Selasky
8402f058ad Fix implementation of task_pid_group_leader() in the LinuxKPI.
In FreeBSD thread IDs and procedure IDs have distinct number
spaces. When asking for the group leader task ID in the LinuxKPI,
return the procedure ID and let this resolve to the first task in the
procedure having a valid LinuxKPI task structure pointer.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 10:26:03 +00:00
Hans Petter Selasky
1ea4c85781 Implement proper support for memory map operations in the LinuxKPI,
like open, close and fault using the character device pager.

Some notes about the implementation:

1) Linux drivers set the vm_ops and vm_private_data fields during a
mmap() call to indicate that the driver wants to use the LinuxKPI VM
operations. Else these operations are not used.

2) The vm_private_data pointer is associated with a VM area structure
and inserted into an internal LinuxKPI list. If the vm_private_data
pointer already exists, the existing VM area structure is used instead
of the allocated one which gets freed.

3) The LinuxKPI's vm_private_data pointer is used as the callback
handle for the FreeBSD VM object. The VM subsystem in FreeBSD has a
similar list to identify equal handles and will only call the
character device pager's close function once.

4) All LinuxKPI VM operations are serialized through the mmap_sem
sempaphore, which is per procedure, which prevents simultaneous access
to the shared VM area structure when receiving page faults.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 09:34:54 +00:00
Hans Petter Selasky
e54b103e70 Before registering a new mm_struct in the LinuxKPI check if other
tasks in the belonging procedure already have a valid mm_struct and
reference that instead.

The mm_struct in the LinuxKPI should be shared among all tasks
belonging to the same procedure. This has to do with with the mmap_sem
semaphore which should serialize all VM operations inside a given
procedure. Linux based drivers depend on this behaviour.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-06 09:07:01 +00:00
Hans Petter Selasky
6e3e654490 Unify error handling when si_drv1 is NULL in the LinuxKPI.
Make sure the character device poll callback function does not return
an error code, but a POLLXXX value, in case of failure.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-05 12:48:24 +00:00
Hans Petter Selasky
0bd68b774d Implement down_write_killable() in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-04-05 12:10:02 +00:00
Hans Petter Selasky
0a475c59a9 Implement vmalloc_32() in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-27 17:18:04 +00:00
Hans Petter Selasky
0791730913 Add more platforms supporting the direct map feature in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-27 17:09:11 +00:00
Hans Petter Selasky
8186b52744 Implement a series of physical page management related functions in
the LinuxKPI for accessing user-space memory in the kernel.

Add functions to hold and wire physical page(s) based on a given range
of user-space virtual addresses.

Add functions to get and put a reference on, wire, hold, mark
accessed, copy and dirty a physical page.

Add new VM related structures and defines as a preparation step for
advancing the memory map capabilities of the LinuxKPI.

Add function to figure out if a virtual address was allocated using
malloc().

Add function to convert a virtual kernel address into its physical
page pointer.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-27 17:04:11 +00:00
Hans Petter Selasky
8f7eee5a63 Use ppsratecheck() for ratelimiting in the LinuxKPI.
Suggested by:		cem @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 16:23:55 +00:00
Hans Petter Selasky
d2f312e0e7 Add proper error checking for the string to number conversion
functions in the LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 16:01:51 +00:00
Hans Petter Selasky
8293738e64 Function macros are preferred in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 13:28:16 +00:00
Hans Petter Selasky
e9db3df254 Add support for ratelimited printouts in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-23 10:48:10 +00:00
Mark Johnston
e5fe3ae2b4 Extend cmpxchg() to support 8- and 16-bit values, and add xchg().
These are needed to support updated revisions of the DRM code.

Reviewed by:	hselasky (previous version)
MFC after:	2 weeks
2017-03-22 17:33:57 +00:00
Hans Petter Selasky
82d0140707 Add full VNET support to the inet_get_local_port_range() function in
the LinuxKPI.

MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-22 15:46:31 +00:00
Hans Petter Selasky
303bd80ad5 Add support for more IPv4 and IPv6 related macros in the LinuxKPI.
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-22 15:44:00 +00:00
Hans Petter Selasky
a0699ebf77 Implement get_pid_task(), pid_task() and some other PID helper
functions in the LinuxKPI. Add a usage atomic to the task_struct
structure to facilitate refcounting the task structure when returned
from get_pid_task(). The get_task_struct() and put_task_struct()
function is used to manage atomic refcounting. After this change the
task_struct should only be freed through put_task_struct().

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-17 15:40:24 +00:00
Hans Petter Selasky
05d4f501dc Implement minimalistic memory mapping structure, struct mm_struct, and
some associated helper functions in the LinuxKPI. Let the existing
linux_alloc_current() function allocate and initialize the new
structure and let linux_free_current() drop the refcount on the memory
mapping structure. When the mm_struct's refcount reaches zero, the
structure is freed.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-17 10:30:06 +00:00
Hans Petter Selasky
0a2f4606bb Add comment describing the use of pagefault_disable() and
pagefault_enable() in the LinuxKPI.

Suggested by:  		rpokala@
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-17 08:02:46 +00:00
Hans Petter Selasky
3803a97f84 Use __LP64__ to detect presence of suword64() to fix linking and
loading of the LinuxKPI on 32-bit platforms.

Reported by:		lwhsu @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 20:39:31 +00:00
Hans Petter Selasky
810ea5b270 The LinuxKPI pagefault disable and enable functions can only be used
pairwise to support the FreeBSD way of pushing and popping the page
fault flags. Ensure this by requiring every occurrence of pagefault
disable function call to have a corresponding pagefault enable call.

Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 16:53:22 +00:00
Hans Petter Selasky
0e05589b39 Implement more userspace memory access functions in the LinuxKPI.
Obtained from:		kmacy @
MFC after:		1 week
Sponsored by:		Mellanox Technologies
2017-03-16 16:49:27 +00:00