summaryrefslogtreecommitdiff
path: root/src/journal/sd-journal.c
AgeCommit message (Collapse)Author
2015-11-03journalctl: when we fail to open a journal file, print whyLennart Poettering
When we enumerate journal files and encounter an invalid one, remember which this, and show it to the user. Note the possibly slightly surprising logic here: we store only one path per error code. This means we show all error kinds but not every actual error we encounter. This has the benefit of not requiring us to keep a potentially unbounded list of errors with their sources around, but can still provide a pretty complete overview on the errors we encountered. Fixes #1669.
2015-11-03sd-journal: various clean-ups and modernizationsLennart Poettering
- Always print a debug log message about files and directories we cannot open right when it happens instead of the caller, thus reducing the number of places where we need to generate the debug message. - Always push the errors we encounter immediately into the error set, when we run into them, instead of in the caller. Thus, we never forget to push them in. - Use stack instead of heap memory where we can. - Make remove_file() void, since it cannot fail anyway and always returned 0. - Make local machine check of journal directories explicit in a function, to make things more readable. - Port to all directory listing loops FOREACH_DIRENT_ALL() - sd-daemon is library code, hence never log at higher log levels than LOG_DEBUG.
2015-11-01sd-journal.c: port to extract_first_wordSusant Sahani
2015-10-27util-lib: move inotify-related definitions to fs-util.[ch]Lennart Poettering
2015-10-27util-lib: split out allocation calls into alloc-util.[ch]Lennart Poettering
2015-10-27util-lib: split out printf() helpers to stdio-util.hLennart Poettering
2015-10-27util-lib: introduce dirent-util.[ch] for directory entry callsLennart Poettering
Also, move a couple of more path-related functions to path-util.c.
2015-10-26util-lib: split out IO related calls to io-util.[ch]Lennart Poettering
2015-10-25util-lib: split out fd-related operations into fd-util.[ch]Lennart Poettering
There are more than enough to deserve their own .c file, hence move them over.
2015-10-24util-lib: split our string related calls from util.[ch] into its own file ↵Lennart Poettering
string-util.[ch] There are more than enough calls doing string manipulations to deserve its own files, hence do something about it. This patch also sorts the #include blocks of all files that needed to be updated, according to the sorting suggestions from CODING_STYLE. Since pretty much every file needs our string manipulation functions this effectively means that most files have sorted #include blocks now. Also touches a few unrelated include files.
2015-08-24util: make machine_name_is_valid() a macro and move it to hostname-util.hLennart Poettering
As it turns out machine_name_is_valid() does the exact same thing as hostname_is_valid() these days, as it just invoked that and checked the name length was < 64. However, hostname_is_valid() checks the length against HOST_NAME_MAX anyway (which is 64 on Linux), hence any additional check is redundant. We hence replace machine_name_is_valid() by a macro that simply maps it to hostname_is_valid() but sets the allow_trailing_dot parameter to false. We also move this this call to hostname-util.h, to the same place as the hostname_is_valid() declaration.
2015-08-18journal: remove error check that never happensThomas Hindoe Paaboel Andersen
remove_directory will always return 0 so this can never happen. Besides that, d->path and d are freed so we would end up with a null pointer dereference anyway.
2015-04-10shared: add formats-util.hRonny Chevalier
2015-03-08sd-journal: return error when we cannot open a fileZbigniew Jędrzejewski-Szmek
Lack of this caused journalctl not to display a hint about missing groups properly when the user lacks permissions.
2015-02-25journal: make skipping of exhausted journal files effective againMichal Schmidt
Commit 668c965af "journal: skipping of exhausted journal files is bad if direction changed" fixed a correctness issue, but it also significantly limited the cases where the optimization that skips exhausted journal files could apply. As a result, some journalctl queries are much slower in v219 than in v218. (e.g. queries where a "--since" cutoff should have quickly eliminated older journal files from consideration, but didn't.) If already in the initial iteration find_location_with_matches() finds no entry, the journal file's location is not updated. This is fine, except that: - We must update at least f->last_direction. The optimization relies on it. Let's separate that from journal_file_save_location() and update it immediately after the direction checks. - The optimization was conditional on "f->current_offset > 0", but it would always be 0 in this scenario. This check is unnecessary for the optimization.
2015-02-12include <poll.h> instead of <sys/poll.h>Thomas Hindoe Paaboel Andersen
include-what-you-use automatically does this and it makes finding unnecessary harder to spot. The only content of poll.h is a include of sys/poll.h so should be harmless.
2015-02-03util: rework strappenda(), and rename it strjoina()Lennart Poettering
After all it is now much more like strjoin() than strappend(). At the same time, add support for NULL sentinels, even if they are normally not necessary.
2015-02-01Add a snprinf wrapper which checks that the buffer was big enoughZbigniew Jędrzejewski-Szmek
If we scale our buffer to be wide enough for the format string, we should expect that the calculation was correct. char_array_0() invocations are removed, since snprintf nul-terminates the output in any case. A similar wrapper is used for strftime calls, but only in timedatectl.c.
2015-01-29Revert "journal: do not check for number of files"Lennart Poettering
This reverts commit b914ea8d379b446c4c9fac4ba181771676ef38cd. We really need to put a limit on all our resources, everywhere, and in particular if we operate on external data. Hence, let's reintroduce the limit, but bump it substantially, so that it is guaranteed to be higher than any realistic RLIMIT_NOFILE setting.
2015-01-09journal: do not check for number of filesZbigniew Jędrzejewski-Szmek
Now that we bump rlimit, we do not really know how many files we can open. Remove the check. https://bugzilla.redhat.com/show_bug.cgi?id=1179980
2014-12-24util: fix strict aliasing violations in use of struct inotify_event v5Shawn Paul Landden
There is alot of cleanup that will have to happen to turn on -fstrict-aliasing, but I think our code should be "correct" to the rule.
2014-12-19journal: skipping of exhausted journal files is bad if direction changedMichal Schmidt
EOF is meaningless if the direction of iteration changes. Move the EOF optimization under the direction check. This fixes test-journal-interleaving for me. Thanks to Filipe Brandenburger for telling me about the failure.
2014-12-19journal: make next_with_matches() always use f->current_offsetMichal Schmidt
next_with_matches() is odd in that its "unit64_t *offset" parameter is both input and output. In other it's purely for output. The function is called from two places in next_beyond_location(). In both of them "&cp" is used as the argument and in both cases cp is guaranteed to equal f->current_offset. Let's just have next_with_matches() ignore "*offset" on input and operate with f->current_offset. I did not investigate why it is, but it makes my usual benchmark run reproducibly faster: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m4.032s user 0m3.896s sys 0m0.135s (Compare to preceding commit, where real was 4.4s.)
2014-12-19journal: fix skipping of duplicate entries in iterationMichal Schmidt
I accidentally broke the detection of duplicate entries in 7943f42275 "journal: optimize iteration by returning previously found candidate entry". When we have a known location of a candidate entry, we must not return from next_beyond_location() immediately. We must go through the duplicates detection to make sure the candidate differs from the already iterated entry. This fix slows down iteration a bit, but it's still faster than it was before the rework. $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m4.448s user 0m4.298s sys 0m0.149s (Compare with results from commit 7943f42275, where real was 5.3s before the rework.)
2014-12-18journal: next_with_matches() now does not need a mapped object as inputMichal Schmidt
Now that journal_file_next_entry() does not need a pointer to the current object, next_with_matches() does not need it either.
2014-12-18journal: journal_file_next_entry() does not need pointer to current ObjectMichal Schmidt
The current offset is sufficient information.
2014-12-18journal: optimize iteration by returning previously found candidate entryMichal Schmidt
In next_beyond_location() when the JournalFile's location type is LOCATION_SEEK, it means there's nothing to do, because we already have the location of the candidate entry. Do an early return. Note that now next_beyond_location() does not anymore guarantee on return that the entry is mapped, but previous patches made sure the caller does not care. This optimization is at least as good as "journal: optimize iteration: skip files that cannot improve current candidate entry" was. Timing results on my workstation, using: $ time ./journalctl -q --since=2014-06-01 --until=2014-07-01 > /dev/null Before "Revert "journal: optimize iteration: skip files that cannot improve current candidate entry": real 0m5.349s user 0m5.166s sys 0m0.181s Now: real 0m3.901s user 0m3.724s sys 0m0.176s
2014-12-18journal: optimize iteration by skipping exhausted filesMichal Schmidt
If from a previous iteration we know we are at the end of a journal file, don't bother looking into the file again. This is complicated by the fact that the EOF does not have to be permanent (think of "journalctl -f"). So we also check if the number of entries in the journal file changed. This optimization has a similar effect as "journal: optimize iteration: skip whole files behind current location" had.
2014-12-18journal: drop unnecessary parameters of next_beyond_location()Michal Schmidt
offset is redundant, because the caller can rely on f->current_offset. The object pointer the function saves in *ret is thrown away by the caller.
2014-12-18journal: remove redundant variable new_offsetMichal Schmidt
The file's current_offset is already updated at this point, so let's use it.
2014-12-18journal: compare candidate entries using JournalFiles' locationsMichal Schmidt
When comparing the locations of candidate entries, we can rely on the location information stored in struct JournalFile.
2014-12-18journal: simplify set_location()Michal Schmidt
set_location() is called from real_journal_next() when a winning entry has been picked from among the candidates in journal files. The location type is always set to LOCATION_DISCRETE. No need to pass it as a parameter. The per-JournalFile location information is already updated at this point. No need for having the direction and offset here.
2014-12-18journal: keep per-JournalFile location info during iterationMichal Schmidt
In next_beyond_location() when we find a candidate entry in a journal file, save its location information in struct JournalFile. The purpose of remembering the locations of candidate entries is to be able to save work in the next iteration. This patch does only the remembering part. LOCATION_SEEK means the location identifies a candidate entry. When a winner is picked from among candidates, it becomes LOCATION_DISCRETE. LOCATION_TAIL here signifies we've iterated the file to the end (or the beginning in the case of reversed direction).
2014-12-18journal: abstract the resetting of JournalFile's locationMichal Schmidt
2014-12-18Revert "journal: optimize iteration: skip whole files behind current location"Michal Schmidt
This reverts commit b7c88ab8cc7d55a43450bf3dea750f95f2e910d6. This optimization will be made redundant by the following patches.
2014-12-18Revert "journal: optimize iteration: skip files that cannot improve current ↵Michal Schmidt
candidate entry" This reverts commit f8b5a3b75fb55f0acb85c21424b3893c822742e9. This optimization will be made redundant by the following patches.
2014-12-13journal: consistently use OBJECT_<type> names instead of numbersMichal Schmidt
Note that numbers 0 and -1 are both replaced with OBJECT_UNUSED, because they are treated the same everywhere (e.g. type_to_context() translates them both to 0).
2014-12-13journal: remove journal_file_object_keep/release functionsMichal Schmidt
The only user is sd_journal_enumerate_unique() and, as explained in the previous commit (fed67c38e3 "journal: map objects to context set by caller, not by actual object type"), the use of them there is now superfluous. Let's remove them. This reverts major parts of commits: ae97089d49 journal: fix access to munmapped memory in sd_journal_enumerate_unique 06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values Tested with an "--enable-debug" build and "journalctl --list-boots". It gives the expected number of results. Additionally, if I then revert the previous commit ("journal: map objects to context set by caller, not to actual object type"), it crashes with SIGSEGV, as expected.
2014-12-10core: unify how we iterate over inotify eventsLennart Poettering
Let's add some syntactic sugar for iterating through inotify events, and use it everywhere.
2014-12-09journal: optimize iteration: skip files that cannot improve current ↵Michal Schmidt
candidate entry Suppose that while iterating we have already looked into a journal file and got a candidate for the next entry. And we are considering to look into another journal file because it may contain an entry that is nearer to the current location than the candidate. We should skip the whole journal file if we can tell by looking at its header that none of its entries can precede the candidate. Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m20.518s user 0m19.989s sys 0m0.328s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m9.445s user 0m9.228s sys 0m0.213s
2014-12-09journal: optimize iteration: skip whole files behind current locationMichal Schmidt
Interleaving of entries from many journal files is expensive. But there is room for optimization. We can skip looking into journal files whose entries all lie before the current iterating location. We can tell if that's the case from looking at the journal file header. This saves a huge amount of work if one has many of mostly not interleaved journal files. On my workstation with 90 journal files in /var/log/journal/ID/ totalling 3.4 GB I get these results: Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 5m54.258s user 2m4.263s sys 3m48.965s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m20.518s user 0m19.989s sys 0m0.328s The high "sys" time in the original was caused by putting more stress on the mmap-cache than it could handle. With the patch the working set now consists of fewer mmap windows and mmap-cache is not thrashing.
2014-11-28treewide: use log_*_errno whenever %m is in the format stringMichal Schmidt
If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.
2014-11-28treewide: more log_*_errno() conversions, multiline callsMichal Schmidt
Basically: find . -name '*.[ch]' | while read f; do perl -i.mmm -e \ 'local $/; local $_=<>; s/log_(debug|info|notice|warning|error|emergency)\("([^"]*)%s"([^;]*),\s*strerror\(-?([->a-zA-Z_]+)\)\);/log_\1_errno(\4, "\2%m"\3);/gms;print;' \ $f; done Plus manual indentation fixups.
2014-11-28treewide: no need to negate errno for log_*_errno()Michal Schmidt
It corrrectly handles both positive and negative errno values.
2014-11-28treewide: auto-convert the simple cases to log_*_errno()Michal Schmidt
As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().
2014-10-23journal: make sd_journal::files a OrderedHashmapMichal Schmidt
Anything that uses hashmap_next() almost certainly cares about the order and needs to be an OrderedHashmap.
2014-10-22machine: validate machine names using machine_name_is_valid() instead of ↵Lennart Poettering
string_is_safe() After all, we know have this as generic validator, so let's be correct and use it wherver applicable.
2014-10-20sd-journal: consistently use ternary operator for all direction checksChristian Hesse
2014-10-09sd-journal: fix sd_journal_enumerate_unique skipping valuesJan Janssen
sd_journal_enumerate_unique will lock its mmap window to prevent it from being released by calling mmap_cache_get with keep_always=true. This call may return windows that are wider, but compatible with the parameters provided to it. This can result in a mismatch where the window to be released cannot properly be selected, because we have more than one window matching the parameters of mmap_cache_release. Therefore, introduce a release_cookie to be used when releasing the window. https://bugs.freedesktop.org/show_bug.cgi?id=79380
2014-10-09sd-journal: do not reset sd_j_enumerate_unique position on errorZbigniew Jędrzejewski-Szmek
systemctl would call sd_j_enumerate_unique() interleaved with sd_j_next(). But the latter can remove a file if it detects an error in it. In those circumstances sd_j_enumerate_unique would restart with the first file in hashmap. With many corrupted files sd_j_enumerate_unique might iterate over the list multiple times. Avoid this by jumping to the next file in unique list if possible, or setting a flag that tells sd_j_enumerate_unique that it is done otherwise.