Commit Graph

271 Commits

Author SHA1 Message Date
Ceri Davies
774bb1c256 periodic: add support for .xz and .zcat compressed logs
Also improve temporary file usage in 200.accounting, add an xref to
zstd(1) to newsyslog.conf.5, and clarify in periodic.conf that
"daily accounting" means process accounting and "monthly accounting"
is login accounting.

PR:		253868
Reviewed by:	allanjude
Approved by:	blackend (mentor)
Differential Revision:	https://reviews.freebsd.org/D29267
2021-06-02 17:28:28 +01:00
Juraj Lutter
c7d27b225d newsyslog(8): Implement a new 'E' flag to not rotate empty log files
Based on an idea from dvl's coworker, László DANIELISZ, implement
a new flag, 'E', that prevents newsyslog(8) from rotating the empty
log files. This 'E' flag ist mostly usable in conjunction with 'B'
flag that instructs newsyslog(8) to not insert an informational
message into the log file after rotation, keeping it still empty.

Reviewed by:	markj, ian, manpages (rpokala)
Approved by:	markj, ian, manpages (rpokala)
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D28940
2021-02-28 23:32:19 +01:00
Eugene Grosbein
01cf0a1724 newsyslog(8): make configuration parser more robust.
Currently newsyslog supports <include> directive that is used
in our default /etc/newsyslog.conf in the following form:

<include> /usr/local/etc/newsyslog.conf.d/*

While this is suitable for ports installing their own rules
for logs rotation, this also makes newsyslog break entire
processing of all files if it encounters single line it cannot parse.
This includes lines referring to nonexistent username/group for log
ownership, so newsyslog stops calling errx() function in the parser.

With this fix, newsyslog uses warnx() instead of errx() in such cases
to print a warning, recover gracefully and continue with execution.

Among other cases, this unbreaks initial creation of log files
having flag "C" at boot time (newsyslog -CN). This is most important
for systems having RAM-based /var file system like nanobsd(8)-based
that rely on newsyslog to bring system log files into existence.

MFC after:	1 month
2020-06-16 17:45:23 +00:00
Mark Johnston
4c8f64714a newsyslog: Add fallthrough comments to appease Coverity.
CID:		1008165, 1008166, 1008167
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2020-03-24 18:17:10 +00:00
Mark Johnston
13d1902439 Add regression tests for newsyslog.conf's p flag.
While here do a bit of cleanup:
- declare local variables as such,
- make tmpdir_create() clean up logfile directories, to handle a
  previously interrupt test run more gracefully.

MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
2020-03-24 18:16:56 +00:00
Mark Johnston
faff7ddb00 newsyslog: Fix stack corruption when initializing a zipwork structure.
This happens when compressing a previously uncompressed already-rotated
file, as happens when handling the 'p' flag in newsyslog.conf.  The file
name is stored in a flexible array member, so these structures cannot be
stack allocated.

Also make sure that we call change_attrs() and do_zipwork() in dry-run
mode; they handle this properly, contrary to the commit log message for
r327451.

CID:		1008168
Github PR:	https://github.com/freebsd/freebsd/pull/427
MFC after:	2 weeks
Submitted by:	Radek Brich (original version)
2020-03-24 18:16:36 +00:00
Ed Maste
13f7dbe822 retire amd(8)
autofs was introduced with FreeBSD 10.1 and is the supported method for
automounting filesystems.  As of r296194 the amd man page claimed that it
is deprecated.  Remove it from base now; the sysutils/am-utils port is
still available if necessary.

Discussed with:	cy
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2020-03-09 20:46:43 +00:00
Steve Wills
6cc4a3c970 Use correct filename in newsyslog.conf
Approved by:		bapt (implicit)
Differential Revision:	https://reviews.freebsd.org/D21561
2019-09-17 20:05:06 +00:00
Steve Wills
cd4b2a3c08 log daemon.info to /var/log/daemon.log by default
log daemon facility now that daemon(8) has syslog support which defaults to
daemon facility, info priority

Reviewed by:		bapt
Approved by:		bapt
Differential Revision:	https://reviews.freebsd.org/D21561
2019-09-17 20:03:20 +00:00
David Bright
1f641a35fd Fix several Coverity-detected issues in newsyslog.
- CID 1394815, CID 1305673: Dereference before null check - memory was
  allocated and the allocation checked for NULL with a call to errx()
  if it failed. Code below that was guaranteed that the pointer was
  non-NULL, but there was another check for NULL at the exit of the
  function (after the memory had already been referenced). Eliminate
  the useless NULL check.

- CID 1007452: Resource leak - Storage intended to be allocated and
  returned to the caller was never freed. This was the result of a
  regression in the function signature introduced in r208648 (2010)
  (thanks for that find, @cem!). Fixed by altering the function
  signature and passing the allocated memory to the caller as
  intended. This also fixes PR158794.

- CID 1008620: Logically dead code in newsyslog.c - This was a direct
  result of CID 1007452. Since the memory allocated as described there
  was not returned to the caller, a subsequent check for the memory
  having been allocated was dead code. Returning the memory
  re-animates the code that is the subject of this CID.

- CID 1006131: Unused value - in parsing a configuration file, a
  pointer to the end of the last field was saved, but not used after
  that. Rewrite to use the pointer value. This could have been fixed
  by avoiding the assignment altogether, but this solutions more
  closely follows the pattern used in the preceding code.

PR:		158794
Reported by:	Coverity, Ken-ichi EZURA <k.ezura@gmail.com> (PR158794)
Reviewed by:	cem, markj
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D19105
2019-02-22 15:31:50 +00:00
David Bright
c7189f3694 Revert r344389 & r343906 - Fix Coverity errors in newsyslog
It was pointed out that a couple of the "memory leak" CIDs that I
fixed were arguably Coverity errors rather than errors in the
newsyslog code and the cure was worse than the disease. Revert both
changes. The first change, which included fixes for other Coverity
errors, will be re-worked to omit the troublesome changes and then
re-committed with the remaining fixes.

Reported by:	bde
Sponsored by:	Dell EMC Isilon
2019-02-22 14:59:40 +00:00
David Bright
2c3e626a0d Complete fix for CID 1007454, CID 1007453: Resource leak in newsyslog
The result of a strdup() was stored in a global variable and not freed
before program exit. This is a follow-up to r343906. That change
attempted to plug these resource leaks but managed to miss a code path
on which the leak still occurs. Plug the leak on that path, too.

MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2019-02-20 22:05:44 +00:00
David Bright
00c4df53b6 Fix several Coverity-detected issues in newsyslog.
- CID 1394815, CID 1305673: Dereference before null check - memory was
  allocated and the allocation checked for NULL with a call to errx()
  if it failed. Code below that was guaranteed that the pointer was
  non-NULL, but there was another check for NULL at the exit of the
  function (after the memory had already been referenced). Eliminate
  the useless NULL check.

- CID 1007454, CID 1007453: Resource leak - The result of a strdup()
  was stored in a global variable and not freed before program exit.

- CID 1007452: Resource leak - Storage intended to be allocated and
  returned to the caller was never freed. This was the result of a
  regression in the function signature introduced in r208648 (2010)
  (thanks for that find, @cem!). Fixed by altering the function
  signature and passing the allocated memory to the caller as
  intended. This also fixes PR158794.

- CID 1008620: Logically dead code in newsyslog.c - This was a direct
  result of CID 1007452. Since the memory allocated as described there
  was not returned to the caller, a subsequent check for the memory
  having been allocated was dead code. Returning the memory
  re-animates the code that is the subject of this CID.

- CID 1006131: Unused value - in parsing a configuration file, a
  pointer to the end of the last field was saved, but not used after
  that. Rewrite to use the pointer value. This could have been fixed
  by avoiding the assignment altogether, but this solutions more
  closely follows the pattern used in the preceding code.

PR:		158794
Reported by:	Coverity, Ken-ichi EZURA <k.ezura@gmail.com> (PR158794)
Reviewed by:	cem, markj
MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2019-02-08 13:54:16 +00:00
Mark Johnston
c2c3992b39 Fix age_old_log() after r337468.
We can no longer use sizeof() to get the path buffer's size.  Apply
a straightforward fix for now with the aim of MFCing soon.

PR:		233633
Submitted by:	Katsuyuki Miyoshi <katsu@miyoshi.matsuyama.ehime.jp>
MFC after:	3 days
2018-11-29 16:49:56 +00:00
Ben Woods
9165316ff6 newsyslog.conf: Restrict included files in default config to [!.]*.conf
The new default config will only include files from the following
directories which end with '.conf' and do not beginning with a '.'
character:
- /etc/newsyslog.conf.d/
- /usr/local/etc/newsyslog.conf.d/

This matches the syslog.conf(5) functionality, and also prevents '.sample' or
'.pkgnew' files being included. This is important for ports which install files
in /usr/local/etc/newsyslog.conf.d/ and also for pkgbase.

Approved by:	eadler
Approved by:	bapt
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D17086
2018-11-10 10:46:38 +00:00
Hans Petter Selasky
ba2c34f4f2 Put a size limit on the opensm.log and use bzip2(1).
Discussed with:		markj@
Sponsored by:		Mellanox Technologies
2018-11-08 17:00:05 +00:00
Hans Petter Selasky
fa895a749e Revert r340246.
Sponsored by:		Mellanox Technologies
2018-11-08 16:23:09 +00:00
Hans Petter Selasky
5e0e7e1846 Add /var/log/opensm.log to list of rotating log files.
MFC after:		3 days
Sponsored by:		Mellanox Technologies
2018-11-08 12:43:13 +00:00
Conrad Meyer
c657f9385b newsyslog(8): Reject configurations that specify setuid or executable logs
Prevent some classes of foot-shooting that may result in permissions
problems.

Reviewed by:	dab, delphij, vangyzen (earlier version)
Relnotes:	yes (behavior change)
Sponsored by:	Dell EMC Isilon
Differential Revision:	D16831
2018-08-21 23:12:46 +00:00
Brad Davis
31ef5c6891 Move all the newsyslog related configs to usr.sbin/newsyslog/
This is related to pkgbase and changes these to use CONFS so that these are
tagged as config files.

Approved by:	AllanJude (mentor)
Sponsored by:	Essen Hackathon
Differential Revision:	https://reviews.freebsd.org/D16694
2018-08-12 13:24:53 +00:00
Mark Johnston
28984f2552 Simplify compression code.
- Remove the compression suffix macros and move them directly into the
  compress_type array.
- Remove the hardcoded sizes on the suffix and compression args arrays.
- Simplify the compression args arrays at the expense of a __DECONST
  when calling execv().
- Rewrite do_zipwork.  The COMPRESS_* macros can directly index the
  compress_types array, so the outer loop is not needed. Convert
  fixed-length strings into asprintf or sbuf calls.

Submitted by:	Dan Nelson <dnelson_1901@yahoo.com>
Reviewed by:	gad
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D16518
2018-08-08 17:26:51 +00:00
Mark Johnston
5c1b641b09 Fix a flag collision introduced in r327451.
PR:		230350
MFC after:	3 days
2018-08-04 15:30:56 +00:00
Mark Johnston
c4c9cd8d68 Remove ARGS_NUM, accidentally left over after r337050. 2018-08-01 18:49:40 +00:00
Mark Johnston
322dacf057 Don't hard-code the number of compression utility arguments.
The zstd invocation constructed by newsyslog contains one more parameter
than invocations for the other supported compression utilities.  However,
the maximum number of arguments was hard-coded, leading to an
out-of-bounds array access when using zstd compression.
2018-08-01 18:45:33 +00:00
Mark Johnston
995cd89924 Adjust the number of tests after r336913. 2018-08-01 18:41:43 +00:00
Baptiste Daroussin
19fe43f796 newsyslog: fix compression arglist construction
Reuse of the index variable in two nested loops resulted in only the first
argument in the list being used (fine for gzip, not fine for zstd). Also
add tests for xz and zstd, and fix the COMPRESS_SUFFIX_MAXLEN macro.

Submitted by:	dnelson_1901_yahoo.com
Differential Revision:	https://reviews.freebsd.org/D16509
2018-07-30 15:46:24 +00:00
Conrad Meyer
7bdbd012c6 newsyslog.8: Remove cutesy nonsense
Sponsored by:	Dell EMC Isilon
2018-07-19 16:03:20 +00:00
Ed Schouten
30dd2da2a3 Use the FQDN in the newsyslog log message when RFC 5424 is enabled.
The RFC 5424 spec mentions that logging FQDNs over short hostnames is
preferred. Alter this code, so that the hostname doesn't get truncated
on startup. Keep track of the length of the short hostname, so that
fprintf() can do the truncation where necessary.

MFC after:	1 month
2018-07-08 10:08:24 +00:00
Eitan Adler
a184696e5a newsyslog: fix typeo for 'zstd'
Reported by:	swildner@DragonFlyBSD.org
MFC After:	1 week
2018-02-03 20:53:21 +00:00
John Baldwin
7de32ca317 Sort the list of flags in newsyslog.conf entries.
Move the 'X' and 'Y' entries into their sorted location in the list
of flags just above 'Z'.

Reviewed by:	bcr
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D13904
2018-01-16 00:47:53 +00:00
Eitan Adler
df76ac9951 newsyslog: implement 'p' flag
Implement the 'p' flag for newsyslog from NetBSD. This flag results in
the first log file for a given file to not be compressed.

While here, don't change file attributes during a no-op run

PR:		162798
Submitted by:	heas@shrubbery.net
MFC After:	1 month
2017-12-31 22:01:36 +00:00
Baptiste Daroussin
64874a795e newsyslog: Fix issues after r326616
When building the command to execute for compression, newsyslog was modifying
the generic arguments array instead of its own copy.
Meaning on the second file to compress with the same arguments, the command line
was not the one expected.
Fix it by creating one copy of the arguments per execution and modifying that
copy.

While here, print the command line executed in verbose mode.

Reported by:	many
2017-12-18 09:35:04 +00:00
Baptiste Daroussin
6f80848e09 Really fix typo and improve wording of the comment 2017-12-06 10:47:50 +00:00
Baptiste Daroussin
69f0fc7877 Fix typo
Reported by:	danfe
2017-12-06 10:20:01 +00:00
Baptiste Daroussin
307845dff2 Allow newsyslog to execute compression commands which
have a semantic different than the traditional gzip(1)

This is done to allow to use zstd(1) as a compression tool without
having to patch it to change its default behavior.
2017-12-06 09:44:35 +00:00
Pedro F. Giffuni
1de7b4b805 various: general adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.
2017-11-27 15:37:16 +00:00
Bryan Drewery
ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Enji Cooper
d511b20a69 Add HAS_TESTS to all Makefiles that are currently using the
`SUBDIR.${MK_TESTS}+= tests` idiom.

This is a follow up to r321912.
2017-08-02 08:50:42 +00:00
Enji Cooper
4b330699f8 Convert traditional ${MK_TESTS} conditional idiom for including test
directories to SUBDIR.${MK_TESTS} idiom

This is being done to pave the way for future work (and homogenity) in
^/projects/make-check-sandbox .

No functional change intended.

MFC after:	1 weeks
2017-08-02 08:35:51 +00:00
Enji Cooper
b024603382 Clean up leading whitespace (convert single column spaces to hard tabs)
MFC after:	now
2017-07-20 00:40:03 +00:00
Enji Cooper
649c9a25d5 Remove TODO for sub testcases added for bug 212160
On closer inspection, the past failures no longer occur on ^/head.

PR:		212160
Sponsored by:	Dell EMC Isilon
2017-06-03 18:20:23 +00:00
Enji Cooper
81341eb139 Fix the testplan after ^/head@r318960
The number of executed testcases is 128, not 126.

MFC after:	18 days
MFC with:	r318960
Sponsored by:	Dell EMC Isilon
2017-06-03 18:12:00 +00:00
Enji Cooper
f54f69008f Don't execute the TODO cases in a subshell
This messes up the testcase counter, as seen in bug 219756.

PR:		212160, 219756
Sponsored by:	Dell EMC Isilon
2017-06-03 18:10:04 +00:00
David Bright
b326fec486 Add newsyslog capability to write RFC5424 compliant rotation message.
This modification adds the capability to newsyslog to write the
rotation message in a format that is compliant with RFC5424. This
capability is enabled on a per-log file basis through a new value
("T") in the flags field in newsyslog.conf. This is useful on systems
that use the RFC5424 format for log files so that the rotation message
format matches that of the other log messages. There has been recent
mention of adding an RFC5424 compliant mode to syslogd and at least
one alternative system log daemon (rsyslogd) that already has the
capability to use that format.

Reviewed by:	vangyzen, ngie
Approved by:	vangyzen (mentor)
MFC after:	2 months
Relnotes:	yes
Sponsored by:	Dell EMC
Differential Revision:	https://reviews.freebsd.org/D10253
2017-05-26 16:36:30 +00:00
Baptiste Daroussin
1af6dc18b9 Add a new Y flag to newsyslog.conf
This makes newsyslog use zstandard to compress log files.

Given Z is already taken for gzip and zstandard compression level stands in
between gzip and xz (which has the X flag) chosing Y sounds ok :)
2017-04-15 20:37:34 +00:00
Bryan Drewery
28323add09 Fix improper use of "its".
Sponsored by:	Dell EMC Isilon
2016-11-08 23:59:41 +00:00
Ed Schouten
23e591f485 Properly patch up dirname()/basename() calls to not clobber ent->log.
It turns out that we had a couple of more calls to dirname()/basename()
in newsyslog(8) that assume the input isn't clobbered. This is bad,
because it apparently breaks log rotation now that the new dirname()
implementation has been merged.

Fix this by first copying the input and then calling
dirname()/basename(). While there, improve the naming of variables in
this function a bit.

Reported by:	Ryan Steinmetz, gjb
Reviewed by:	bdrewery, allanjude
Differential Revision:	https://reviews.freebsd.org/D7838
2016-09-09 07:10:50 +00:00
Enji Cooper
771e2c7753 Expect sub testcases 2-4 in :tests_time_rotate to fail today due to changes
to newsyslog, etc made in the past month.

The issue is being root-caused as part of the bug noted below. This commit
will need to be partially reverted once the issue has been found/fixed

PR:		212160
Reported by:	Jenkins
Sponsored by:	EMC / Isilon Storage Division
2016-08-29 18:46:04 +00:00
Ed Schouten
f0d2919243 Clean up use of basename() and dirname().
Pull copies of the input pathname string before calling basename() and
dirname() to make this comply to POSIX. Free these copies at the end of
this function. While there, remove the duplication of the 's' ->
'logfname' string. There is no need for this.
2016-07-28 16:02:30 +00:00
Eric van Gyzen
c152ba971f newsyslog: Eliminate unnecessary sleep(10) when -R and -s are specified
After going through the signal work list, during which do_sigwork()
is called and essentially does nothing because -s and -R were
specified on the command line, newsyslog will sleep for 10 seconds
as the (verbose) code says: "Pause 10 seconds to allow daemon(s)
to close log file(s)".

However, the man page verbiage for -R (and -s) seems quite clear
that this sleep() is unnecessary because the daemon was expected
to have already closed the log file before calling newsyslog.

PR:		210020
Submitted by:	David A. Bright <david_a_bright@dell.com>
MFC after:	1 week
Sponsored by:	Dell Inc.
Differential Revision:	https://reviews.freebsd.org/D6727
2016-06-06 22:54:08 +00:00