2016-07-10 10:17:38 +00:00
|
|
|
.\"
|
|
|
|
|
.\" Copyright (c) 2016 Alexander Motin <mav@FreeBSD.org>
|
|
|
|
|
.\" All rights reserved.
|
|
|
|
|
.\"
|
|
|
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
|
|
|
.\" modification, are permitted provided that the following conditions
|
|
|
|
|
.\" are met:
|
|
|
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
|
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
|
|
|
.\"
|
|
|
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
.\" SUCH DAMAGE.
|
|
|
|
|
.\"
|
|
|
|
|
.\" $FreeBSD$
|
|
|
|
|
.\"
|
2016-07-28 10:48:20 +00:00
|
|
|
.Dd July 28, 2016
|
2016-07-10 10:17:38 +00:00
|
|
|
.Dt NTB_HW 4
|
|
|
|
|
.Os
|
|
|
|
|
.Sh NAME
|
|
|
|
|
.Nm ntb ,
|
|
|
|
|
.Nm ntb_hw
|
|
|
|
|
.Nd Intel(R) Non-Transparent Bridge driver
|
|
|
|
|
.Sh SYNOPSIS
|
|
|
|
|
To compile this driver into your kernel,
|
|
|
|
|
place the following lines in your kernel configuration file:
|
|
|
|
|
.Bd -ragged -offset indent
|
|
|
|
|
.Cd "device ntb_hw"
|
|
|
|
|
.Ed
|
|
|
|
|
.Pp
|
|
|
|
|
Or, to load the driver as a module at boot, place the following line in
|
|
|
|
|
.Xr loader.conf 5 :
|
|
|
|
|
.Bd -literal -offset indent
|
|
|
|
|
ntb_hw_load="YES"
|
|
|
|
|
.Ed
|
|
|
|
|
.Pp
|
|
|
|
|
The following tunables are settable from the
|
|
|
|
|
.Xr loader 8 :
|
|
|
|
|
.Bl -ohang
|
|
|
|
|
.It Va hw.ntb.debug_level
|
|
|
|
|
Driver debug level.
|
|
|
|
|
The default value is 0, higher means more verbose.
|
2016-07-28 10:48:20 +00:00
|
|
|
.It Va hint.ntb_hw. Ns Ar X Ns Va .config
|
|
|
|
|
Configures NTB resources split between several consumer devices.
|
|
|
|
|
Configuration of multiple consumer devices separated by commas.
|
|
|
|
|
Each device can be configured as: "<name>[:<mw>[:<spad>[:<db>]]]", where:
|
|
|
|
|
.Va name
|
|
|
|
|
is a name of the driver which should attach the device (empty means any),
|
|
|
|
|
.Va mw
|
|
|
|
|
is a number of memory windows to allocate (empty means all available),
|
|
|
|
|
.Va spad
|
|
|
|
|
is a number of scratchpad registers to allocate (empty means all available),
|
|
|
|
|
.Va db
|
|
|
|
|
is a number of doorbells to allocate (empty means all available).
|
|
|
|
|
The default configuration is empty string, which means single device
|
|
|
|
|
with all available resources allowing any driver attachment.
|
2016-07-10 10:17:38 +00:00
|
|
|
.El
|
|
|
|
|
.Sh DESCRIPTION
|
|
|
|
|
The NTB allows you to connect two computer systems using a PCIe link if they
|
|
|
|
|
have the correct equipment and connectors.
|
|
|
|
|
The
|
|
|
|
|
.Nm ntb_hw
|
|
|
|
|
driver provides support for the Non-Transparent Bridge (NTB) in the Intel S1200
|
|
|
|
|
and Xeon E3/E5 processor families.
|
|
|
|
|
The
|
|
|
|
|
.Nm
|
|
|
|
|
driver hides hardware details, exposing memory windows, scratchpads and
|
|
|
|
|
doorbells via hardware independent KPI.
|
|
|
|
|
.Pp
|
|
|
|
|
The hardware provides 2-3 memory windows to the other system's memory,
|
|
|
|
|
16 scratchpad registers and 14/34 doorbells to interrupt the other system.
|
|
|
|
|
On Xeon processors one of memory windows is typically consumed by the driver
|
|
|
|
|
to workaround multiple hardware erratas.
|
|
|
|
|
.Sh CONFIGURATION
|
2016-07-28 10:48:20 +00:00
|
|
|
The NTB configuration should be set by BIOS.
|
2016-07-10 10:17:38 +00:00
|
|
|
It includes enabling NTB, choosing between NTB-to-NTB or NTB-to-Root Port mode,
|
|
|
|
|
enabling split BAR mode (one of two 64-bit BARs can be split into two 32-bit
|
|
|
|
|
ones) and configuring BAR sizes in bits (from 12 to 29/39) for both NTB sides.
|
|
|
|
|
.Pp
|
|
|
|
|
The recommended configuration is NTB-to-NTB mode, split bar is enabled and
|
|
|
|
|
all BAR sizes are set to 20 (1 MiB).
|
|
|
|
|
This needs to be done on both systems.
|
|
|
|
|
.Sh SEE ALSO
|
|
|
|
|
.Xr ntb_transport 4 ,
|
|
|
|
|
.Xr if_ntb 4
|
|
|
|
|
.Sh AUTHORS
|
|
|
|
|
.An -nosplit
|
|
|
|
|
The
|
|
|
|
|
.Nm
|
|
|
|
|
driver was developed by Intel and originally written by
|
|
|
|
|
.An Carl Delsey Aq Mt carl@FreeBSD.org .
|
|
|
|
|
Later improvements were done by
|
|
|
|
|
.An Conrad E. Meyer Aq Mt cem@FreeBSD.org
|
|
|
|
|
and
|
|
|
|
|
.An Alexander Motin Aq Mt mav@FreeBSD.org .
|
|
|
|
|
.Sh BUGS
|
|
|
|
|
NTB-to-Root Port mode is not yet supported, but it doesn't look very useful.
|
|
|
|
|
.Pp
|
|
|
|
|
On Xeon v2/v3/v4 processors split BAR mode should be enabled to allow
|
|
|
|
|
SB01BASE_LOCKUP errata workaround to be applied by the driver.
|
|
|
|
|
.Pp
|
|
|
|
|
There is no way to protect your system from malicious behavior on the other
|
|
|
|
|
system once the link is brought up.
|
|
|
|
|
Anyone with root or kernel access on the other system can read or write to
|
|
|
|
|
any location on your system.
|
|
|
|
|
In other words, only connect two systems that completely trust each other.
|
