newvers.sh takes upwards of 4-5 seconds to complete on trees checked
out from github, due to searching the entire history for non-existent
git-svn metadata. Similarly, if one does not check out notes, we
again search the entire history for notes. That makes newvers.sh very
slow for many github users.
To fix this in a fair way, limit the history search to the last 10K
commits: if you're more than 10K commits out of sync, then you've
forked the project, and our SVN rev is no longer very important to you.
Due to how git implements --grep in conjunction with -n, --grep has been
removed for performance reasons (git does not seem to limit its search
to the -n limit in this case, and takes just as long as it did with no
limit).
Reviewed by: emaste, imp
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18745
Perforce no longer offers a FreeBSD client and it not a viable VCS for
FreeBSD development. Remove p4 version logic to simplify newvers.sh in
advance of other changes.
Sponsored by: The FreeBSD Foundation
Consider the case where FreeBSD is checked out via Subversion with a
(perhaps unrelated) .git or .hg directory at a higher level - for
example,
.../.git
.../src/freebsd
Previously newvers obtained the SVN revision information via svnversion,
and then tried to obtain the SVN revision corresponding to the git or hg
commit, overwriting the existing information.
As a short term fix use a different variable for hg-svn or git-svn
information, setting $svn from hg or git info only if not empty.
Reported by: Matthias Apitz
Sponsored by: The FreeBSD Foundation
newvers.sh runs mkfifo which did not exist before this change.
However, I didn't notice before because it is run from a function
where a missing command does cause a noticeable failure.
Reviewed By: emaste, markj
Differential Revision: https://reviews.freebsd.org/D18377
When reproducible build mode is enabled vers.c may be unchanged between
successive builds. In this case avoid changing the file's metadata so
that it does not cause dependent targets to be rebuilt.
Sponsored by: The FreeBSD Foundation
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D17892
Assuming that any output from `git diff-index --name-only` implies
changes in the working tree results in false positives: files with
metadata, but not content, changes are also listed.
Check that content differences exist before adding the -dirty tag to
the git hash.
PR: 229230
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D15968
cycle.
This is expected to be the final ALPHA build of this release
cycle, prior to branching stable/12.
Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation
Building the kernel in Git repositories when git-svn is not available and
the "help.autocorrect" Git parameter is enabled results in Git trying to
replace the "svn" command (it does not know) with "serve". As a result the
output of the "git server" command is appended to the value of the
environmental variable VERINFO, which causes the auto generated vers.c
file to contain invalid C syntax (missing newline escapes):
#define "@(#)FreeBSD 12.0-ALPHA7 r000eversion 2
0015agent=git/2.19.0
000cls-refs
0012fetch=shallow
0012server-option
0000=5e2272613fa(splash-vt)"
#define VERSTR "FreeBSD 12.0-ALPHA7 r000eversion 2
0015agent=git/2.19.0
000cls-refs
0012fetch=shallow
0012server-option
0000=5e2272613fa(splash-vt)\n"
Using `-c help.autocorrect=0` seems to be a good solution as it does not
modify user's environment. I am not sure, however, if we should use
programs (or Git commands), which we are not sure exist (we never check if
git-svn is available on the host), as there may be more unexpected
behaviors like this one.
Reviewed by: eadler, emaste, krion
Approved by: re (gjb), krion (mentor)
Sponsored by: Bally Wulff Games & Entertainment GmbH
Differential Revision: https://reviews.freebsd.org/D17271
In non-reproducible mode we have the kernel ident as a side effect of
including the build directory. Explicitly add it to the ident string in
reproducible mode.
Reported by: mjg
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation
cycle. The i386 build failure appears to be transient, and
now becoming more difficult to reliably reproduce to identify
the cause. I will continue to investigate this, however.
Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation
Previously newvers.sh passed --work-tree=${VCSDIR}/.. when invoking git.
When using git worktree .git is actually a file, not a directory, and
.git/.. is not a valid path. Although it appears git handles this
internally (perhaps it normalizes the path first), it is simple enough
for the script to store both the working tree top-level directory and
the VCS (.git) directory, so do so.
Sponsored by: The FreeBSD Foundation
Mainly focus on files that use BSD 3-Clause license.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
In the case of running newvers.sh on a git tree w/o git-svn-id notes we
previously piped the entire 'git log' to grep. Add --grep to the log
invocation to avoid processing log entries of no interest.
This saves about 2-3 seconds of newvers.sh run time on my SSD laptop.
Later changes will bring further speedups.
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
This prevents incorrect subversion revision detection when "git svn" is
not being used to get the sources but git is available. Previously old
subversion revisions included in commit messages were favoured over the
more recent and correct revisions in git notes.
For example cf1f355747 represents r315395 but was treated as r313908
which is referenced in the commit message. Commits following
r315395/cf1f35574722 but before another commit with a git-svn-id
reference in the commit message would be treated as r313908 as well.
Patch from PR updated to accommodate the initial four space indent in
`git log` ouptut.
PR: 221848
Submitted by: Fabian Keil
Obtained from: ElectroBSD
MFC after: 2 weeks
newvers.sh looks for a .vcs subdirectory (e.g. .git, .svn) to determine
which vcs info tool to run (e.g., git rev-parse, svn info).
(As of r308789 if a .vcs subdirectory is not found at ${TOPDIR} then
newvers.sh walks up successive parent directories, testing for the .vcs
subdirectory at each step. This is done in case the FreeBSD source is
built in a subdirectory as part of some larger project, but either way
newvers.sh still tests for the .vcs subdirectory.)
However, when using git worktree there is no .git subdirectory but
rather a plain text .git file which contains a reference to the main
working tree.
Change findvcs() to test that the .vcs entry exists, regardless of type.
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
The newvers -R option is intended to include build metadata (e.g. user,
host, time) if the build is from an unmodified VCS tree. For subversion
it considered a trailing 'M' as an indication of a modified tree, and
any other version string as modified.
Also include mixed revision checkouts (e.g. 123:126), switched (123S)
and partial (123P) working copies as modified: the revision number is
insufficient to uniquely determine which source was used for the build.
Reported by: gjb
Reviewed by: gjb
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8853
Build metadata (username, hostname, etc.) prevents the FreeBSD kernel
from building reproducibly. Add an option to disable inclusion of that
metadata but retain the release information and SVN/git VCS details.
See https://reproducible-builds.org/ for additional background.
Reviewed by: bapt
Obtained from: NetBSD
MFC after: 1 month
Sponsored by: Reproducible Builds World Summit 2, Berlin
Differential Revision: https://reviews.freebsd.org/D4347
.hg may reside above FreeBSD sources root. Provide function findvcs()
that will climb up and seek for presence of a VCS directory.
Reviewed by: imp (earlier version of the patch)
One reason the kernel does not build reproducibly is that it includes
a timestamp in the version string. SOURCE_DATE_EPOCH provides a standard
method to address this: it should be set to the last modification time
of the source, and build processes use the specified timestamp instead
of the "current" date and time.
This change uses SOURCE_DATE_EPOCH if it is set; how it gets set needs
to be addressed elsewhere.
Reviewed by: bapt
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Because of how osreldate.h was being built with newvers.sh, which always
spat out a vers.c dependent on SVN or git, the meta mode build was
considering osreldate.h to depend on the current git or SVN index. This
would lead to entire tree rebuilds when modifying git's index. There's
no reason to be generating vers.c here so just skip it.
While here, in mk-osreldate.sh rename PARAM_H to proper PARAMFILE (which
newvers.sh already has a default for) and remove unneeded export.
Sponsored by: EMC / Isilon Storage Division
