Commit Graph

3711 Commits

Author SHA1 Message Date
Warner Losh
259fb883db Stupid last minute changes: Add missing } and fi
Pointed out by: Howard Su
2015-12-05 16:28:14 +00:00
Warner Losh
67af7f8bfb New config files for embedded boards.
Build with ../nanobsd.sh -c rpi.cfg, for example.
This can be done as a normal user.

This is a work in progress. It relies on the new nopriv
build stuff committed to nanobsd, but isn't complete yet.
Currently, one must copy files into the DOS partition
in the image. Also, ownership isn't preserved because
this doesn't use the new mtree-dedup.awk yet, but rather
some crazy mtree stuff. The image building bits will
move up into nanobsd when they are ready.

Also includes very preliminary support for building qemu
images for all platforms that we can for qemu. It is
missing aarch64, and we put the image on s2 instead of
s1 and mkimg can't mark s2 as active, so there's some
issues. Oh, and I didn't do it for arm.

Take a look, kick the tires, expect problems.
2015-12-05 04:43:56 +00:00
Warner Losh
35028a6e30 Awk helper script that reads in a mtree METALOG file from installworld
(and soon augmented by nanobsd), performs the actions documented in
the script, and then spits out a new mtree file suitable for feeding
to makefs.

Discussed on: arch@
2015-12-05 01:12:44 +00:00
Warner Losh
c275d9ec22 Setting NANO_NOPRIV_BUILD will now add -DNO_ROOT and METALOG=xxxx as
appropriate. First step in supporting a build w/o root. More to
follow as actions by customization scripts are not (yet) recorded in
the metalog, and duplicate entries in it aren't removed.
2015-12-05 01:10:04 +00:00
Warner Losh
a9d9c49f3f SRCCONF makes no sense in make.conf. Don't set it there. Rely on it
being in the environment. Also filter out the new SRC_ENV_CONF as
well. If you really need these set, set them in your config file,
not in the build environment used to launch nanobsd.

Pointed out by: bdrewery@
2015-12-05 00:54:43 +00:00
Warner Losh
1934c7aa0a Minor cleanup in how we run make:
o Move SRCCONF and __MAKE_CONF into the environment to cope with
  file paths with spaces in them better.
o Move the rest of the variable setting command line args into
  __MAKE_CONF files.
o Trace the commands that we're using to build so they appear at the
  top of the log.
o Be more consistent about quoting paths for cd and similar commands
  to better cope with paths with spaces in them, though some more
  work is likely needed.
o Add some comments about all this.
o Minor formatting tweaks in a couple places

Sponsored by:	Netflix, Inc
2015-12-05 00:15:04 +00:00
Warner Losh
7dfcfa5cf1 Remove commented out junk. 2015-12-04 23:44:34 +00:00
Warner Losh
2b8875e3e1 Since this is an almost identical copy of ALIX_DSK, just include it
and add the bits we need at the end.
2015-12-04 23:22:07 +00:00
Bryan Drewery
eacae6dc66 Fix LDADD/DPADD that should be LIBADD.
Sponsored by:	EMC / Isilon Storage Division
2015-12-04 03:17:47 +00:00
Bryan Drewery
a9dfc956d6 Fully connect cddl/usr.sbin/plockstat.
There seems to be no reason to keep this so private.

Also add missing optional MK_CDDL files.

Sponsored by:	EMC / Isilon Storage Division
2015-12-02 00:49:45 +00:00
Adrian Chadd
36f8b167b5 add missing lib80211 include. 2015-11-30 09:02:28 +00:00
Dimitry Andric
85ab8f98e6 Install the public sanitizer headers. These are useful for programs
that want to directly interface with sanitizer internals.
2015-11-29 16:28:40 +00:00
Hajimu UMEMOTO
f1f871e17d Fix breakage by recent collation change.
LC_ALL has to be exported.
2015-11-28 00:41:37 +00:00
Enji Cooper
3ec75a6ccf Clean up trailing whitespace
MFC after: 1 week
2015-11-26 08:58:13 +00:00
Enji Cooper
a6961d88b4 r291359 was incorrect. Skip over tokens that start with `#' as fgetln can
return more than one '\n' delimited line in a buffer

Handle empty lines too, just in case

MFC after: 3 days
X-MFC with: r291359
2015-11-26 07:58:22 +00:00
Enji Cooper
059d81a6e4 Skip over lines that start with # (comments)
MFC after: 3 days
2015-11-26 07:03:59 +00:00
Enji Cooper
84da2236a4 Remove cnftp.bjpu.edu.cn (it no longer resolves)
Add localhost to resolv/mach for parity with nss/mach

MFC after: 3 days
2015-11-26 07:03:26 +00:00
Hans Petter Selasky
b35ff88448 Update usage:
- Add missing description of "-c" option.
- Add one more usage example.
2015-11-25 14:51:44 +00:00
Hans Petter Selasky
10ac20335c Add simple indent wrapper tool for style(9) checking GIT/SVN patches.
The indent_wrapper tool only accepts full context diffs and works by
identifying the surrounding C-block touched by a diff and passing only
that to indent for styling. In the end a diff is produced or an
external tool like meld can be invoked, to show the styling
differences.
2015-11-25 13:26:42 +00:00
Dag-Erling Smørgrav
1765946ba9 Retire the NONE cipher option. 2015-11-23 12:48:13 +00:00
Warner Losh
d07833f981 Add support for passing TARGET_CPUTYPE into the build. It's more
important for arm and mips than for the more generic x86 systems.
2015-11-20 21:54:05 +00:00
Craig Rodrigues
749f65e3e3 Use print as a function for Python 3 [PEP 3105]
Import print_function from the __future__ module
to activate this for Python 2.  This works as far back as Python 2.6.0a2:
   https://docs.python.org/2/library/__future__.html

[PEP 3105] https://www.python.org/dev/peps/pep-3105/
2015-11-19 00:01:52 +00:00
Craig Rodrigues
aef675d879 Use 'in' instead of 'has_key()' for testing dictionary membership.
In PEP 0290, has_key() was deprecated in Python 2.2 and higher:
https://www.python.org/dev/peps/pep-0290/#testing-dictionary-membership
https://docs.python.org/2.2/whatsnew/node4.html

In Python 3, dict.has_key() was removed:
https://docs.python.org/3.0/whatsnew/3.0.html#builtins
2015-11-18 23:32:29 +00:00
Bryan Drewery
7359fe7af4 Use -n to ln(1) which is compatible with GNU ln(1).
Sponsored by:	EMC / Isilon Storage Division
2015-11-16 23:27:44 +00:00
Bryan Drewery
804baa3822 Revert r290944. It was wrong. 2015-11-16 21:05:38 +00:00
Bryan Drewery
5a16e0b461 Fix error case for bmake to echo 0.
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-11-16 20:31:00 +00:00
Baptiste Daroussin
1d211085ac Bring back a couple of ISO8859-15 locales:
- af_ZA.ISO8859-15
- en_AU.ISO8859-15
- en_CA.ISO8859-15
- en_NZ.ISO8859-15
- en_US.ISO8859-15
- fr_CA.ISO8859-15
2015-11-16 12:58:47 +00:00
Enji Cooper
e742867ba4 Remove directory made obsolete by r290905
MFC after: 1 week
X-MFC with: r290905
X-MFC note: stable/10 only
Sponsored by: EMC / Isilon Storage Division
2015-11-16 06:56:45 +00:00
Enji Cooper
e30a620063 Integrate tools/regression/pipe in to the FreeBSD test suite as
tests/sys/kern/pipe

- Fix style(9) bugs
- Fix compiler warnings
- Use `nitems(x)` instead of `sizeof(x) / sizeof(*x)` pattern

The testcases will be converted over to ATF eventually, but for now will be
integrated in as plain C tests

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-16 05:38:40 +00:00
Baptiste Daroussin
0dc64c20b4 Remove trailing spaces at end of lines 2015-11-15 20:00:00 +00:00
Baptiste Daroussin
99b47c3cc6 Update the list of files to be removed in case world is built WITHOUT_LOCALES 2015-11-15 19:58:05 +00:00
Baptiste Daroussin
66cad538dd Generate in the FreeBSD keyword when generating the Makefiles 2015-11-15 14:34:29 +00:00
Baptiste Daroussin
09712c4216 Add ISO8859-1 everywhere ISO8859-15 exists 2015-11-15 14:19:56 +00:00
Baptiste Daroussin
30b3fca790 Allow to generate the locale when the source directory is not /usr/src 2015-11-15 14:19:08 +00:00
Simon J. Gerraty
32b5dc9a48 Regen src.conf.5 2015-11-14 04:50:28 +00:00
Simon J. Gerraty
948f327ee4 Rename META_MODE option to DIRDEPS_BUILD
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.

Differential Revision:	https://reviews.freebsd.org/D4153
Reviewed by:	bdrewery
2015-11-14 03:24:48 +00:00
Warner Losh
7b77d3ea7f Make the slice names for root configurable. For embedded platforms, we
need s1 to be a FAT partition, s2 to be the config partition and s3
and s4 to be the ping-pong upgrade partitions.

NANO_SLICE_ROOT defaults to s1
NANO_SLICE_ALTROOT defaults to s2
NANO_SLICE_CFG defaults to s3
NANO_SLICE_DATA defaults to s4

All can be overridden in the config file. Some basic sanity checking
is in place, but is no substitute for being careful.
2015-11-12 05:53:32 +00:00
Bryan Drewery
d4ea5890a8 Use explicit filename when creating locale symlinks to avoid creating a
directory symlink when the target directory does not exist.  This will
cause an error instead of a broken setup.

Sponsored by:	EMC / Isilon Storage Division
2015-11-11 18:45:48 +00:00
Enji Cooper
44642b54ec - Move ng_bluetooth.4 under MK_BLUETOOTH != no
- Move all section 5 bluetooth manpages under MK_BLUETOOTH != no

MFC after: 3 days
PR: 193260
Reported by: Philippe Michel <philippe.michel7@sfr.fr>
Sponsored by: EMC / Isilon Storage Division
2015-11-10 13:28:41 +00:00
Baptiste Daroussin
32810cf3c6 Add a README to explain how to generate the locales from CLDR data 2015-11-09 23:23:29 +00:00
Enji Cooper
fbf5b9f8a2 Integrate tools/regression/lib/libc/gen into the FreeBSD test suite
as lib/libc/tests/gen

The code in test-fnmatch that was used for generating:

- bin/sh/tests/builtins/case2.0
- bin/sh/tests/builtins/case3.0

has been left undisturbed. The target `make sh-tests` has been moved over
from tools/regression/lib/libc/gen/Makefile to
lib/libc/tests/gen/Makefile and made into a PHONY target

case2.0 and case3.0 test input generation isn't being done automatically.
This needs additional discussion.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-09 06:24:11 +00:00
Enji Cooper
24732d675d Integrate tools/regression/lib/libc/net into the FreeBSD test suite
as lib/libc/tests/net

Also, fix eui64_aton_test:test_str(..). The test was comparing the result
of eui64_aton to a pointer of the expected result.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-08 23:06:40 +00:00
Baptiste Daroussin
7b71bb4224 Modify the tool that generates the locales so that the generated CJK locales
are padded as expected by the users

Submitted by:	marino
Obtained from:	DragonflyBSD
2015-11-08 19:08:38 +00:00
Enji Cooper
347267dcf8 Integrate tools/regression/lib/libc/string into the FreeBSD test suite
as lib/libc/tests/string

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-08 07:53:31 +00:00
Enji Cooper
98682851f2 Integrate tools/regression/lib/libc/stdlib into the FreeBSD test suite
as lib/libc/tests/stdlib

- Make the code a bit more style(9) compliant
- Convert a sizeof(x)/sizeof(x[0]) to nitems

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-08 07:03:17 +00:00
Enji Cooper
1ee0219205 Integrate tools/regression/lib/libc/stdio into the FreeBSD test suite
as lib/libc/tests/stdio

- Fix some whitespace
- Convert the testcases to ATF
- Convert "/dev/null" to _PATH_DEVNULL

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-08 06:37:50 +00:00
Enji Cooper
9da7d79f25 Integrate tools/regression/lib/libc/locale into the FreeBSD test suite
as lib/libc/tests/locale

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
2015-11-08 02:06:17 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Baptiste Daroussin
07fa68d600 Catchup with latest changes for ObsoleteFiles in case base is built using
WITHOUT_LOCALE knob
2015-11-07 11:28:25 +00:00
Baptiste Daroussin
3e4f384ed2 Merge from head r290483 2015-11-07 11:02:33 +00:00