2017-02-15 12:38:58 +00:00
|
|
|
DPDK Release 17.05
|
|
|
|
==================
|
|
|
|
|
|
|
|
.. **Read this first.**
|
|
|
|
|
|
|
|
The text in the sections below explains how to update the release notes.
|
|
|
|
|
|
|
|
Use proper spelling, capitalization and punctuation in all sections.
|
|
|
|
|
|
|
|
Variable and config names should be quoted as fixed width text:
|
|
|
|
``LIKE_THIS``.
|
|
|
|
|
|
|
|
Build the docs and view the output file to ensure the changes are correct::
|
|
|
|
|
|
|
|
make doc-guides-html
|
|
|
|
|
|
|
|
xdg-open build/doc/html/guides/rel_notes/release_17_05.html
|
|
|
|
|
|
|
|
|
|
|
|
New Features
|
|
|
|
------------
|
|
|
|
|
|
|
|
.. This section should contain new features added in this release. Sample
|
|
|
|
format:
|
|
|
|
|
|
|
|
* **Add a title in the past tense with a full stop.**
|
|
|
|
|
|
|
|
Add a short 1-2 sentence description in the past tense. The description
|
|
|
|
should be enough to allow someone scanning the release notes to
|
|
|
|
understand the new feature.
|
|
|
|
|
|
|
|
If the feature adds a lot of sub-features you can use a bullet list like
|
|
|
|
this:
|
|
|
|
|
|
|
|
* Added feature foo to do something.
|
|
|
|
* Enhanced feature bar to do something else.
|
|
|
|
|
|
|
|
Refer to the previous release notes for examples.
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|
|
|
|
|
2017-03-24 18:55:53 +00:00
|
|
|
* **Added free Tx mbuf on demand API.**
|
|
|
|
|
|
|
|
Added a new function ``rte_eth_tx_done_cleanup()`` which allows an application
|
|
|
|
to request the driver to release mbufs from their Tx ring that are no longer
|
|
|
|
in use, independent of whether or not the ``tx_rs_thresh`` has been crossed.
|
|
|
|
|
2017-03-29 08:36:28 +00:00
|
|
|
* **Added descriptor status ethdev API.**
|
|
|
|
|
|
|
|
Added a new API to get the status of a descriptor.
|
|
|
|
|
|
|
|
For Rx, it is almost similar to the ``rx_descriptor_done`` API, except
|
|
|
|
it differentiates descriptors which are hold by the driver and not
|
|
|
|
returned to the hardware. For Tx, it is a new API.
|
|
|
|
|
2017-03-14 17:17:38 +00:00
|
|
|
* **Increased number of next hops for LPM IPv6 to 2^21.**
|
|
|
|
|
|
|
|
The next_hop field is extended from 8 bits to 21 bits for IPv6.
|
2017-02-15 12:38:58 +00:00
|
|
|
|
2017-03-29 09:54:50 +00:00
|
|
|
* **Added VFIO hotplug support.**
|
|
|
|
|
|
|
|
How hotplug supported with UIO and VFIO drivers.
|
|
|
|
|
2017-03-06 15:04:15 +00:00
|
|
|
* **Added powerpc support in pci probing for vfio-pci devices.**
|
|
|
|
|
|
|
|
sPAPR IOMMU based pci probing enabled for vfio-pci devices.
|
|
|
|
|
2017-03-29 15:56:48 +00:00
|
|
|
* **Kept consistent PMD batching behaviour.**
|
|
|
|
|
|
|
|
Removed the limit of fm10k/i40e/ixgbe TX burst size and vhost RX/TX burst size
|
|
|
|
in order to support the same policy of "make an best effort to RX/TX pkts"
|
|
|
|
for PMDs.
|
|
|
|
|
2017-03-01 06:04:52 +00:00
|
|
|
* **Updated the ixgbe base driver.**
|
|
|
|
|
|
|
|
Updated the ixgbe base driver, including the following changes:
|
|
|
|
|
|
|
|
* Add link block check for KR.
|
|
|
|
* Complete HW initialization even if SFP is not present.
|
|
|
|
* Add VF xcast promiscuous mode.
|
|
|
|
|
2017-02-23 04:39:08 +00:00
|
|
|
* **Added powerpc support for i40e and its vector PMD .**
|
|
|
|
|
|
|
|
i40e PMD and its vector PMD enabled by default in powerpc.
|
|
|
|
|
2017-02-24 03:24:28 +00:00
|
|
|
* **Added VF max bandwidth setting on i40e.**
|
|
|
|
|
|
|
|
i40e HW supports to set the max bandwidth for a VF. Enable this capability.
|
|
|
|
|
2017-02-24 03:24:29 +00:00
|
|
|
* **Added VF TC min bandwidth setting on i40e.**
|
|
|
|
|
|
|
|
i40e HW supports to set the allocated bandwidth for a TC on a VF. Enable this
|
|
|
|
capability.
|
|
|
|
|
2017-02-24 03:24:30 +00:00
|
|
|
* **Added VF TC max bandwidth setting on i40e.**
|
|
|
|
|
|
|
|
i40e HW supports to set the max bandwidth for a TC on a VF. Enable this
|
|
|
|
capability.
|
|
|
|
|
2017-02-24 03:24:31 +00:00
|
|
|
* **Added TC strict priority mode setting on i40e.**
|
|
|
|
|
|
|
|
There're 2 TX scheduling modes supported for TCs by i40e HW, round ribon mode
|
|
|
|
and strict priority mode. By default it's round robin mode. Enable the
|
|
|
|
capability to change the TX scheduling mode for a TC. It's a global setting
|
|
|
|
on a physical port.
|
|
|
|
|
2017-03-30 02:51:51 +00:00
|
|
|
* **Added i40e dynamic device personalization support.**
|
|
|
|
|
|
|
|
* Added dynamic device personalization processing to i40e FW.
|
|
|
|
|
2017-03-06 11:55:45 +00:00
|
|
|
* **Added TSO support for tunneled and non-tunneled packets on mlx5 driver.**
|
|
|
|
|
|
|
|
Added support for Hardware TSO for tunneled and non-tunneled packets.
|
|
|
|
Tunneling protocols supported are GRE and VXLAN.
|
|
|
|
|
2017-03-14 13:03:09 +00:00
|
|
|
* **Added support for Rx interrupts on mlx5 driver.**
|
|
|
|
|
|
|
|
Rx queues can be armed with an interrupt which will trigger on the
|
|
|
|
next packet arrival.
|
|
|
|
|
2017-03-09 15:26:27 +00:00
|
|
|
* **Updated the sfc_efx driver.**
|
|
|
|
|
|
|
|
* Generic flow API support for Ethernet, VLAN, IPv4, IPv6, UDP and TCP
|
|
|
|
pattern items with QUEUE action for ingress traffic.
|
|
|
|
|
2017-03-09 17:23:03 +00:00
|
|
|
* Support virtual functions (VFs)
|
|
|
|
|
2017-03-25 06:24:57 +00:00
|
|
|
* **Added LiquidIO network PMD.**
|
|
|
|
|
|
|
|
Added poll mode driver support for Cavium LiquidIO II server adapter VFs.
|
|
|
|
|
2017-03-28 11:54:09 +00:00
|
|
|
* **Added support for the Wind River Systems AVP PMD.**
|
|
|
|
|
|
|
|
Added a new networking driver for the AVP device type. Theses devices are
|
|
|
|
specific to the Wind River Systems virtualization platforms.
|
|
|
|
|
2017-03-06 22:55:07 +00:00
|
|
|
* **Added vmxnet3 version 3 support.**
|
|
|
|
|
|
|
|
Added support for vmxnet3 version 3 which includes several
|
|
|
|
performance enhancements viz. configurable TX data ring, Receive
|
|
|
|
Data Ring, ability to register memory regions.
|
|
|
|
|
2017-03-27 14:02:21 +00:00
|
|
|
* **Updated the tap driver.**
|
|
|
|
|
|
|
|
* Support MTU modification.
|
|
|
|
* Support packet type for Rx.
|
|
|
|
* Support segmented packets on Rx and Tx.
|
|
|
|
* Speed up Rx on tap when no packets are available.
|
|
|
|
* Support capturing traffic from another netdevice.
|
|
|
|
* Dynamically change link status when the underlying interface state changes.
|
|
|
|
* Generic flow API support for Ethernet, VLAN, IPv4, IPv6, UDP and TCP pattern
|
|
|
|
items with DROP, QUEUE and PASSTHRU actions for ingress traffic.
|
|
|
|
|
2017-03-12 16:34:04 +00:00
|
|
|
* **Added MTU feature support to Virtio and Vhost.**
|
|
|
|
|
|
|
|
Implemented new Virtio MTU feature into Vhost and Virtio:
|
|
|
|
|
|
|
|
* Add ``rte_vhost_mtu_get()`` API to Vhost library.
|
|
|
|
* Enable Vhost PMD's MTU get feature.
|
|
|
|
* Get max MTU value from host in Virtio PMD
|
|
|
|
|
2017-03-31 19:44:56 +00:00
|
|
|
* **Added interrupt mode support for virtio-user.**
|
|
|
|
|
2017-03-31 19:44:58 +00:00
|
|
|
Implemented Rxq interrupt mode and LSC support for virtio-user as a virtual
|
2017-03-31 19:44:56 +00:00
|
|
|
device. Supported cases:
|
|
|
|
|
|
|
|
* Rxq interrupt for virtio-user + vhost-user as the backend.
|
|
|
|
* Rxq interrupt for virtio-user + vhost-kernel as the backend.
|
2017-03-31 19:44:58 +00:00
|
|
|
* LSC interrupt for virtio-user + vhost-user as the backend.
|
2017-03-31 19:44:56 +00:00
|
|
|
|
2017-03-31 14:20:29 +00:00
|
|
|
* **Added event driven programming model library (rte_eventdev).**
|
|
|
|
|
|
|
|
This API introduces event driven programming model.
|
|
|
|
|
|
|
|
In a polling model, lcores poll ethdev ports and associated
|
|
|
|
rx queues directly to look for packet. In an event driven model,
|
|
|
|
by contrast, lcores call the scheduler that selects packets for
|
|
|
|
them based on programmer-specified criteria. Eventdev library
|
|
|
|
added support for event driven programming model, which offer
|
|
|
|
applications automatic multicore scaling, dynamic load balancing,
|
|
|
|
pipelining, packet ingress order maintenance and
|
|
|
|
synchronization services to simplify application packet processing.
|
|
|
|
|
|
|
|
By introducing event driven programming model, DPDK can support
|
|
|
|
both polling and event driven programming models for packet processing,
|
|
|
|
and applications are free to choose whatever model
|
|
|
|
(or combination of the two) that best suits their needs.
|
|
|
|
|
2017-03-30 19:30:49 +00:00
|
|
|
* **Added Software Eventdev PMD.**
|
|
|
|
|
|
|
|
Added support for the software eventdev PMD. The software eventdev is a
|
|
|
|
software based scheduler device that implements the eventdev API. This
|
|
|
|
PMD allows an application to configure a pipeline using the eventdev
|
|
|
|
library, and run the scheduling workload on a CPU core.
|
|
|
|
|
2017-03-24 18:55:53 +00:00
|
|
|
|
2017-02-15 12:38:58 +00:00
|
|
|
Resolved Issues
|
|
|
|
---------------
|
|
|
|
|
|
|
|
.. This section should contain bug fixes added to the relevant
|
|
|
|
sections. Sample format:
|
|
|
|
|
|
|
|
* **code/section Fixed issue in the past tense with a full stop.**
|
|
|
|
|
|
|
|
Add a short 1-2 sentence description of the resolved issue in the past
|
|
|
|
tense.
|
|
|
|
|
|
|
|
The title should contain the code/lib section like a commit message.
|
|
|
|
|
|
|
|
Add the entries in alphabetic order in the relevant sections below.
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|
|
|
|
|
|
|
|
|
|
|
|
EAL
|
|
|
|
~~~
|
|
|
|
|
|
|
|
|
|
|
|
Drivers
|
|
|
|
~~~~~~~
|
|
|
|
|
|
|
|
|
|
|
|
Libraries
|
|
|
|
~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
|
|
Examples
|
|
|
|
~~~~~~~~
|
|
|
|
|
|
|
|
|
|
|
|
Other
|
|
|
|
~~~~~
|
|
|
|
|
|
|
|
|
|
|
|
Known Issues
|
|
|
|
------------
|
|
|
|
|
|
|
|
.. This section should contain new known issues in this release. Sample format:
|
|
|
|
|
|
|
|
* **Add title in present tense with full stop.**
|
|
|
|
|
|
|
|
Add a short 1-2 sentence description of the known issue in the present
|
|
|
|
tense. Add information on any known workarounds.
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|
|
|
|
|
2017-03-31 19:44:58 +00:00
|
|
|
* **LSC interrupt cannot work for virtio-user + vhost-kernel.**
|
|
|
|
|
|
|
|
LSC interrupt cannot be detected when setting the backend, tap device,
|
|
|
|
up/down as we fail to find a way to monitor such event.
|
|
|
|
|
2017-02-15 12:38:58 +00:00
|
|
|
|
|
|
|
API Changes
|
|
|
|
-----------
|
|
|
|
|
|
|
|
.. This section should contain API changes. Sample format:
|
|
|
|
|
|
|
|
* Add a short 1-2 sentence description of the API change. Use fixed width
|
|
|
|
quotes for ``rte_function_names`` or ``rte_struct_names``. Use the past
|
|
|
|
tense.
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|
|
|
|
|
2017-03-14 17:17:38 +00:00
|
|
|
* The LPM ``next_hop`` field is extended from 8 bits to 21 bits for IPv6
|
|
|
|
while keeping ABI compatibility.
|
2017-02-15 12:38:58 +00:00
|
|
|
|
2017-03-29 15:21:17 +00:00
|
|
|
* **Reworked rte_ring library**
|
|
|
|
|
|
|
|
The rte_ring library has been reworked and updated. The following changes
|
|
|
|
have been made to it:
|
|
|
|
|
|
|
|
* removed the build-time setting ``CONFIG_RTE_RING_SPLIT_PROD_CONS``
|
2017-03-29 15:21:20 +00:00
|
|
|
* removed the build-time setting ``CONFIG_RTE_LIBRTE_RING_DEBUG``
|
2017-03-29 15:21:21 +00:00
|
|
|
* removed the build-time setting ``CONFIG_RTE_RING_PAUSE_REP_COUNT``
|
2017-03-29 15:21:22 +00:00
|
|
|
* removed the function ``rte_ring_set_water_mark`` as part of a general
|
|
|
|
removal of watermarks support in the library.
|
2017-03-29 15:21:24 +00:00
|
|
|
* added an extra parameter to the burst/bulk enqueue functions to
|
|
|
|
return the number of free spaces in the ring after enqueue. This can
|
|
|
|
be used by an application to implement its own watermark functionality.
|
2017-03-29 15:21:25 +00:00
|
|
|
* added an extra parameter to the burst/bulk dequeue functions to return
|
|
|
|
the number elements remaining in the ring after dequeue.
|
2017-03-29 15:21:23 +00:00
|
|
|
* changed the return value of the enqueue and dequeue bulk functions to
|
|
|
|
match that of the burst equivalents. In all cases, ring functions which
|
|
|
|
operate on multiple packets now return the number of elements enqueued
|
|
|
|
or dequeued, as appropriate. The updated functions are:
|
|
|
|
|
|
|
|
- ``rte_ring_mp_enqueue_bulk``
|
|
|
|
- ``rte_ring_sp_enqueue_bulk``
|
|
|
|
- ``rte_ring_enqueue_bulk``
|
|
|
|
- ``rte_ring_mc_dequeue_bulk``
|
|
|
|
- ``rte_ring_sc_dequeue_bulk``
|
|
|
|
- ``rte_ring_dequeue_bulk``
|
2017-03-29 15:21:17 +00:00
|
|
|
|
2017-03-29 15:21:25 +00:00
|
|
|
NOTE: the above functions all have different parameters as well as
|
|
|
|
different return values, due to the other listed changes above. This
|
|
|
|
means that all instances of the functions in existing code will be
|
|
|
|
flagged by the compiler. The return value usage should be checked
|
|
|
|
while fixing the compiler error due to the extra parameter.
|
|
|
|
|
2017-04-01 07:22:40 +00:00
|
|
|
* **Reworked rte_vhost library**
|
|
|
|
|
|
|
|
The rte_vhost library has been reworked to make it generic enough so that
|
|
|
|
user could build other vhost-user drivers on top of it. To achieve that,
|
|
|
|
following changes have been made:
|
|
|
|
|
|
|
|
* The following vhost-pmd APIs are removed:
|
|
|
|
|
|
|
|
* ``rte_eth_vhost_feature_disable``
|
|
|
|
* ``rte_eth_vhost_feature_enable``
|
|
|
|
* ``rte_eth_vhost_feature_get``
|
|
|
|
|
2017-04-01 07:22:42 +00:00
|
|
|
* The vhost API ``rte_vhost_driver_callback_register(ops)`` is reworked to
|
|
|
|
be per vhost-user socket file. Thus, it takes one more argument:
|
|
|
|
``rte_vhost_driver_callback_register(path, ops)``.
|
|
|
|
|
2017-04-01 07:22:48 +00:00
|
|
|
* The vhost API ``rte_vhost_get_queue_num`` is deprecated, instead,
|
|
|
|
``rte_vhost_get_vring_num`` should be used.
|
|
|
|
|
2017-04-01 07:22:50 +00:00
|
|
|
* Following macros are removed in ``rte_virtio_net.h``
|
|
|
|
|
|
|
|
* ``VIRTIO_RXQ``
|
|
|
|
* ``VIRTIO_TXQ``
|
|
|
|
* ``VIRTIO_QNUM``
|
|
|
|
|
2017-04-01 07:22:51 +00:00
|
|
|
* Following net specific header files are removed in ``rte_virtio_net.h``
|
|
|
|
|
|
|
|
* ``linux/virtio_net.h``
|
|
|
|
* ``sys/socket.h``
|
|
|
|
* ``linux/if.h``
|
|
|
|
* ``rte_ether.h``
|
|
|
|
|
2017-04-01 07:22:52 +00:00
|
|
|
* The vhost struct ``virtio_net_device_ops`` is renamed to
|
|
|
|
``vhost_device_ops``
|
|
|
|
|
vhost: introduce API to start a specific driver
We used to use rte_vhost_driver_session_start() to trigger the vhost-user
session. It takes no argument, thus it's a global trigger. And it could
be problematic.
The issue is, currently, rte_vhost_driver_register(path, flags) actually
tries to put it into the session loop (by fdset_add). However, it needs
a set of APIs to set a vhost-user driver properly:
* rte_vhost_driver_register(path, flags);
* rte_vhost_driver_set_features(path, features);
* rte_vhost_driver_callback_register(path, vhost_device_ops);
If a new vhost-user driver is registered after the trigger (think OVS-DPDK
that could add a port dynamically from cmdline), the current code will
effectively starts the session for the new driver just after the first
API rte_vhost_driver_register() is invoked, leaving later calls taking
no effect at all.
To handle the case properly, this patch introduce a new API,
rte_vhost_driver_start(path), to trigger a specific vhost-user driver.
To do that, the rte_vhost_driver_register(path, flags) is simplified
to create the socket only and let rte_vhost_driver_start(path) to
actually put it into the session loop.
Meanwhile, the rte_vhost_driver_session_start is removed: we could hide
the session thread internally (create the thread if it has not been
created). This would also simplify the application.
NOTE: the API order in prog guide is slightly adjusted for showing the
correct invoke order.
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
2017-04-01 07:22:56 +00:00
|
|
|
* The vhost API ``rte_vhost_driver_session_start`` is removed. Instead,
|
|
|
|
``rte_vhost_driver_start`` should be used, and no need to create a
|
|
|
|
thread to call it.
|
|
|
|
|
2017-04-01 07:22:57 +00:00
|
|
|
* The vhost public header file ``rte_virtio_net.h`` is renamed to
|
|
|
|
``rte_vhost.h``
|
|
|
|
|
2017-04-01 07:22:40 +00:00
|
|
|
|
2017-02-15 12:38:58 +00:00
|
|
|
ABI Changes
|
|
|
|
-----------
|
|
|
|
|
|
|
|
.. This section should contain ABI changes. Sample format:
|
|
|
|
|
|
|
|
* Add a short 1-2 sentence description of the ABI change that was announced
|
|
|
|
in the previous releases and made in this release. Use fixed width quotes
|
|
|
|
for ``rte_function_names`` or ``rte_struct_names``. Use the past tense.
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|
|
|
|
|
|
|
|
|
2017-02-20 14:30:23 +00:00
|
|
|
Removed Items
|
|
|
|
-------------
|
|
|
|
|
|
|
|
.. This section should contain removed items in this release. Sample format:
|
|
|
|
|
|
|
|
* Add a short 1-2 sentence description of the removed item in the past
|
|
|
|
tense.
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|
|
|
|
|
2017-02-20 14:30:24 +00:00
|
|
|
* KNI vhost support removed.
|
|
|
|
|
2017-02-15 12:38:58 +00:00
|
|
|
|
|
|
|
Shared Library Versions
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
.. Update any library version updated in this release and prepend with a ``+``
|
|
|
|
sign, like this:
|
|
|
|
|
|
|
|
librte_acl.so.2
|
|
|
|
+ librte_cfgfile.so.2
|
|
|
|
librte_cmdline.so.2
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
=========================================================
|
|
|
|
|
|
|
|
|
|
|
|
The libraries prepended with a plus sign were incremented in this version.
|
|
|
|
|
|
|
|
.. code-block:: diff
|
|
|
|
|
|
|
|
librte_acl.so.2
|
|
|
|
librte_cfgfile.so.2
|
|
|
|
librte_cmdline.so.2
|
|
|
|
librte_cryptodev.so.2
|
|
|
|
librte_distributor.so.1
|
|
|
|
librte_eal.so.3
|
|
|
|
librte_ethdev.so.6
|
|
|
|
librte_hash.so.2
|
|
|
|
librte_ip_frag.so.1
|
|
|
|
librte_jobstats.so.1
|
|
|
|
librte_kni.so.2
|
|
|
|
librte_kvargs.so.1
|
|
|
|
librte_lpm.so.2
|
|
|
|
librte_mbuf.so.2
|
|
|
|
librte_mempool.so.2
|
|
|
|
librte_meter.so.1
|
|
|
|
librte_net.so.1
|
|
|
|
librte_pdump.so.1
|
|
|
|
librte_pipeline.so.3
|
|
|
|
librte_pmd_bond.so.1
|
|
|
|
librte_pmd_ring.so.2
|
|
|
|
librte_port.so.3
|
|
|
|
librte_power.so.1
|
|
|
|
librte_reorder.so.1
|
|
|
|
librte_ring.so.1
|
|
|
|
librte_sched.so.1
|
|
|
|
librte_table.so.2
|
|
|
|
librte_timer.so.1
|
|
|
|
librte_vhost.so.3
|
|
|
|
|
|
|
|
|
|
|
|
Tested Platforms
|
|
|
|
----------------
|
|
|
|
|
|
|
|
.. This section should contain a list of platforms that were tested with this
|
|
|
|
release.
|
|
|
|
|
|
|
|
The format is:
|
|
|
|
|
|
|
|
* <vendor> platform with <vendor> <type of devices> combinations
|
|
|
|
|
|
|
|
* List of CPU
|
|
|
|
* List of OS
|
|
|
|
* List of devices
|
|
|
|
* Other relevant details...
|
|
|
|
|
|
|
|
This section is a comment. do not overwrite or remove it.
|
|
|
|
Also, make sure to start the actual text at the margin.
|
|
|
|
=========================================================
|