| Age | Commit message (Collapse) | Author |
|---|
|
The only code path which makes a journal durable is via
journal_file_set_offline().
When we perform a rotate the journal's header->state is being set to
STATE_ARCHIVED prior to journal_file_set_offline() being called.
In journal_file_set_offline(), we short-circuit the entire offline when
f->header->state != STATE_ONLINE.
This all results in none of the journal_file_set_offline() fsync() calls
being reached when rotate archives a journal, so archived journals are
never explicitly made durable.
What we do now is instead of setting the f->header->state to
STATE_ARCHIVED directly in journal_file_rotate() prior to
journal_file_close(), we set an archive flag in f->archive for the
journal_file_set_offline() machinery to honor by committing
STATE_ARCHIVED instead of STATE_OFFLINE when set.
Prior to this, rotated journals were never getting fsync() explicitly
performed on them, since journal_file_set_offline() short-circuited.
Obviously this is undesirable, and depends entirely on the underlying
filesystem as to how much durability was achieved when simply closing
the file.
Note that this problem existed prior to the recent asynchronous fsync
changes, but those changes do facilitate our performing this durable
offline on rotate without blocking, regardless of the underlying
filesystem sync-on-close semantics.
|
|
Previously, when we used a bisection table for seeking through a corrupted
file, and the end of the bisection table was corrupted we'd most likely fail
the entire seek operation. Improve the situation: if we encounter invalid
entries in a bisection table, linearly go backwards until we find a working
entry again.
|
|
error like EOF
When we linearly iterate through a corrupted journal file, and we encounter a
read error, don't consider this fatal, but merely as EOF condition (and log
about it).
|
|
Let's make sure EBADMSG is the one error we throw when we encounter corrupted
data, so that we can neatly test for it.
|
|
Specifically, detect early if we try to read from offset 0, i.e. are using
uninitialized offset data.
|
|
rework "journalctl -M"
|
|
Fix endless loops in journalctl --list-boots (closes #617).
|
|
Early in journal_file_set_offline() f->header->state is tested to see if
it's != STATE_ONLINE, and since there's no need to do anything if the
journal isn't online, the function simply returned here.
Since moving part of the offlining process to a separate thread, there
are two problems here:
1. We can't simply check f->header->state, because if there is an
offline thread active it may modify f->header->state.
2. Even if the journal is deemed offline, the thread responsible may
still need joining, so a bare return may leak the thread's resources
like its stack.
To address #1, the helper journal_file_is_offlining() is called prior to
accessing f->header->state.
If journal_file_is_offlining() returns true, f->header->state isn't even
checked, because an offlining journal is obviously online, and we'll
just continue with the normal set offline code path.
If journal_file_is_offlining() returns false, then it's safe to check
f->header->state, because the offline_state is beyond the point of
modifying f->header->state, and there's a memory barrier in the helper.
If we find f->header->state is != STATE_ONLINE, then we call the
idempotent journal_file_set_offline_thread_join() on the way out of the
function, to join a potential lingering offline thread.
|
|
Let's be nice to users, and let's turn the nonsensical "--unit=… --user" into
"--user-unit=…" which the user more likely meant.
Fixes #1621
|
|
Let's document the call as deprecated, since it doesn't cover containers with
directories that aren#t visible to the host properly.
|
|
This way, the switch becomes compatible with nspawn containers using --image=,
and those which only store journal data in /run (i.e. have persistant logs
off).
Fixes: #49
|
|
When appending to a journal file, journald will:
a) first, append the actual entry to the end of the journal file
b) second, add an offset reference to it to the global entry array stored at
the beginning of the file
c) third, add offset references to it to the per-field entry array stored at
various places of the file
The global entry array, maintained by b) is used when iterating through the
journal without matches applied.
The per-field entry array maintained by c) is used when iterating through the
journal with a match for that specific field applied.
In the wild, there are journal files where a) and b) were completed, but c)
was not before the files were abandoned. This means, that in some cases log
entries are at the end of these files that appear in the global entry array,
but not in the per-field entry array of the _BOOT_ID= field. Now, the
"journalctl --list-boots" command alternatingly uses the global entry array
and the per-field entry array of the _BOOT_ID= field. It seeks to the last
entry of a specific _BOOT_ID=field by having the right match installed, and
then jumps to the next following entry with no match installed anymore, under
the assumption this would bring it to the next boot ID. However, if the
per-field entry wasn't written fully, it might actually turn out that the
global entry array might know one more entry with the same _BOOT_ID, thus
resulting in a indefinite loop around the same _BOOT_ID.
This patch fixes that, by updating the boot search logic to always continue
reading entries until the boot ID actually changed from the previous. Thus, the
per-field entry array is used as quick jump index (i.e. as an optimization),
but not trusted otherwise. Only the global entry array is trusted.
This replaces PR #1904, which is actually very similar to this one. However,
this one actually reads the boot ID directly from the entry header, and doesn't
try to read it at all until the read pointer is actually really located on the
first item to read.
Fixes: #617
Replaces: #1904
|
|
Show the various timestamps in hexadecimal too. This is useful for matching the
timestamps included in cursor strings (which are encoded in hex, too), with the
references in the journal header.
|
|
Drop the "read_realtime" parameter. Getting the realtime timestamp from an
entry is cheap, as it is a normal header field, hence let's just get this
unconditionally, and simplify our code a bit.
|
|
Let's store the reference as simple sd_id128_t, since we don't actually need a
BootId for it.
|
|
|
|
With this change a new flag SD_JOURNAL_OS_ROOT is introduced. If specified
while opening the journal with the per-directory calls (specifically:
sd_journal_open_directory() and sd_journal_open_directory_fd()) the passed
directory is assumed to be the root directory of an OS tree, and the journal
files are searched for in /var/log/journal, /run/log/journal relative to it.
This is useful to allow usage of sd-journal on file descriptors returned by the
OpenRootDirectory() call of machined.
|
|
Also, expose this via the "journalctl --file=-" syntax for STDIN. This feature
remains undocumented though, as it is probably not too useful in real-life as
this still requires fds that support mmaping and seeking, i.e. does not work
for pipes, for which reading from STDIN is most commonly used.
|
|
|
|
Let's output the actual error code encountered, and let's not claim this was
purely triggered by files, because it can also be triggered by directories.
|
|
Let's also collect errors returned by readdir() into our set of errors, like we
do this for all other errors from journal files.
|
|
This is slightly nicer, since we actually watch the directories we opened and
enumerate. However, primarily this is preparation for adding support for
opening journal files by fd without specifying any path, to be added in a later
commit.
|
|
This suppresses output of the hostname for messages from the local system.
Fixes: #2342
|
|
This moves the O_TMPFILE handling from the coredumping code into common library
code, and generalizes it as open_tmpfile_linkable() + link_tmpfile(). The
existing open_tmpfile() function (which creates an unlinked temporary file that
cannot be linked into the fs) is renamed to open_tmpfile_unlinkable(), to make
the distinction clear. Thus, code may now choose between:
a) open_tmpfile_linkable() + link_tmpfile()
b) open_tmpfile_unlinkable()
Depending on whether they want a file that may be linked back into the fs later
on or not.
In a later commit we should probably convert fopen_temporary() to make use of
open_tmpfile_linkable().
Followup for: #3065
|
|
|
|
The comments and the log messages are next to one another, so it's easier
to check that the messages match the comments.
The sign was omitted in the check for -ESHUTDOWN, so it was never matched.
|
|
It might be nicer to propagate the error to the caller, but that'd
be a bigger refactoring. This shouldn't really fail, so just add
an assert.
CID #1349697.
|
|
src/journal/compress.c: In function ‘compress_blob_lz4’:
src/journal/compress.c:115:49: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
r = LZ4_compress_limitedOutput(src, dst + 8, src_size, (int) dst_alloc_size - 8);
^
src/journal/compress.c: In function ‘decompress_blob_xz’:
src/journal/compress.c:179:35: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
s.next_out = *dst + used;
^
src/journal/compress.c: In function ‘decompress_blob_lz4’:
src/journal/compress.c:218:37: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
r = LZ4_decompress_safe(src + 8, out, src_size - 8, size);
^
src/journal/compress.c: In function ‘decompress_startswith_xz’:
src/journal/compress.c:294:38: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
s.next_out = *buffer + *buffer_size - s.avail_out;
^
src/journal/compress.c:294:53: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
s.next_out = *buffer + *buffer_size - s.avail_out;
^
src/journal/compress.c: In function ‘decompress_startswith_lz4’:
src/journal/compress.c:327:45: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
r = LZ4_decompress_safe_partial(src + 8, *buffer, src_size - 8,
^
LZ4 and XZ functions use char* and unsigned char*, respectively,
so keep void* in our internal APIs and add casts.
|
|
The patch is not minimal, but a function to parse size_t is probably
going to come in handy in other places, so I think it's nicer to define
a proper parsing function than to open-code the cast.
|
|
Otherwise we would hit an assert in the compression code.
|
|
|
|
tree-wide: merge pager_open_if_enabled() to the pager_open()
|
|
Many subsystems define own pager_open_if_enabled() function which
checks '--no-pager' command line argument and open pager depends
on its value. All implementations of pager_open_if_enabled() are
the same. Let's merger this function with pager_open() from the
shared/pager.c and remove pager_open_if_enabled() from all subsytems
to prevent code duplication.
|
|
|
|
journal: restore offline state on error
|
|
Throughout the tree there's spurious use of spaces separating ++ and --
operators from their respective operands. Make ++ and -- operator
consistent with the majority of existing uses; discard the spaces.
|
|
If we fail to create the thread, technically we should leave the
offline_state as OFFLINE_JOINED, not OFFLINE_SYNCING.
|
|
When we rotate journals, we must set offline and close the current one,
but don't generally need to wait for this to complete.
Instead, we'll initiate an asynchronous offline via
journal_file_set_offline(oldfile, false), and add the file to a
per-server set of deferred closes to be closed later when they
won't block.
There's one complication however; journal_file_open() via
journal_file_verify_header() assumes that any writable journal in the
online state is the product of an unclean shutdown or other form of
corruption.
Thus there's a need for journal_file_open() to be aware of deferred
closes and synchronize with their completion when opening preexisting
journals for writing. To facilitate this the deferred closes set is
supplied to the journal_file_open() function where the deferred closes
may be closed synchronously before verifying the header in such
circumstances.
|
|
This adds a wait flag to journal_file_set_offline(), when false the offline is
performed asynchronously in a separate thread.
When wait is true, if an asynchronous offline is already in-progress it is
restarted and waited for. Otherwise the offline is performed synchronously
without the use of a thread.
journal_file_set_online() cancels or waits for the asynchronous offline to
complete if in-flight, depending on where in the offline process the thread
happens to be. If the thread is in the fsync() phase, it is cancelled and
waiting is unnecessary. Otherwise, the thread is joined before proceeding.
A new offline_state member is added to JournalFile which is used via
atomic operations for communicating between the offline thread and the
journal_file_set_{offline,online}() functions.
|
|
|
|
|
|
Various buffers were lost because finish_item() either consumed
the buffer or allocated a new one (if an entry with the same key existed).
The caller would simply forget the buffer in either case.
Also add a check for the case when a valid identifier is followed by
an empty body. We should not allow this.
Also be more consistent in error handling and always print an error
message.
|
|
Better support of OPENPGPKEY, CAA, TLSA packets and tests
|
|
It's annoying to have the exact same function in three places.
It's stored in src/shared, but it's not added to the library to
avoid the dependency on libgcrypt.
|
|
ISO/IEC 9899:1999 §7.21.1/2 says:
Where an argument declared as size_t n specifies the length of the array
for a function, n can have the value zero on a call to that
function. Unless explicitly stated otherwise in the description of a
particular function in this subclause, pointer arguments on such a call
shall still have valid values, as described in 7.1.4.
In base64_append_width memcpy was called as memcpy(x, NULL, 0). GCC 4.9
started making use of this and assumes This worked fine under -O0, but
does something strange under -O3.
This patch fixes a bug in base64_append_width(), fixes a possible bug in
journal_file_append_entry_internal(), and makes use of the new function
to simplify the code in other places.
|
|
|
|
This should be handled fine now by .dir-locals.el, so need to carry that
stuff in every file.
|
|
Remove the check that triggers rotation of the journal file when the arriving log entry had a monotonic timestamp smaller that the previous log entry. This check causes unnecessary rotations when journal-remote was receiving from multiple senders, therefore monotonicity can not be guaranteed. Also, it does not offer any useful functionality for systemd-journald.
|
|
treewide: fix typos and spacing
|
|
journal: miscellaneous cleanups
|
