summaryrefslogtreecommitdiff
path: root/src/journal/sd-journal.c
AgeCommit message (Collapse)Author
2015-04-10shared: add formats-util.hRonny Chevalier
2015-03-08sd-journal: return error when we cannot open a fileZbigniew Jędrzejewski-Szmek
Lack of this caused journalctl not to display a hint about missing groups properly when the user lacks permissions.
2015-02-25journal: make skipping of exhausted journal files effective againMichal Schmidt
Commit 668c965af "journal: skipping of exhausted journal files is bad if direction changed" fixed a correctness issue, but it also significantly limited the cases where the optimization that skips exhausted journal files could apply. As a result, some journalctl queries are much slower in v219 than in v218. (e.g. queries where a "--since" cutoff should have quickly eliminated older journal files from consideration, but didn't.) If already in the initial iteration find_location_with_matches() finds no entry, the journal file's location is not updated. This is fine, except that: - We must update at least f->last_direction. The optimization relies on it. Let's separate that from journal_file_save_location() and update it immediately after the direction checks. - The optimization was conditional on "f->current_offset > 0", but it would always be 0 in this scenario. This check is unnecessary for the optimization.
2015-02-12include <poll.h> instead of <sys/poll.h>Thomas Hindoe Paaboel Andersen
include-what-you-use automatically does this and it makes finding unnecessary harder to spot. The only content of poll.h is a include of sys/poll.h so should be harmless.
2015-02-03util: rework strappenda(), and rename it strjoina()Lennart Poettering
After all it is now much more like strjoin() than strappend(). At the same time, add support for NULL sentinels, even if they are normally not necessary.
2015-02-01Add a snprinf wrapper which checks that the buffer was big enoughZbigniew Jędrzejewski-Szmek
If we scale our buffer to be wide enough for the format string, we should expect that the calculation was correct. char_array_0() invocations are removed, since snprintf nul-terminates the output in any case. A similar wrapper is used for strftime calls, but only in timedatectl.c.
2015-01-29Revert "journal: do not check for number of files"Lennart Poettering
This reverts commit b914ea8d379b446c4c9fac4ba181771676ef38cd. We really need to put a limit on all our resources, everywhere, and in particular if we operate on external data. Hence, let's reintroduce the limit, but bump it substantially, so that it is guaranteed to be higher than any realistic RLIMIT_NOFILE setting.
2015-01-09journal: do not check for number of filesZbigniew Jędrzejewski-Szmek
Now that we bump rlimit, we do not really know how many files we can open. Remove the check. https://bugzilla.redhat.com/show_bug.cgi?id=1179980
2014-12-24util: fix strict aliasing violations in use of struct inotify_event v5Shawn Paul Landden
There is alot of cleanup that will have to happen to turn on -fstrict-aliasing, but I think our code should be "correct" to the rule.
2014-12-19journal: skipping of exhausted journal files is bad if direction changedMichal Schmidt
EOF is meaningless if the direction of iteration changes. Move the EOF optimization under the direction check. This fixes test-journal-interleaving for me. Thanks to Filipe Brandenburger for telling me about the failure.
2014-12-19journal: make next_with_matches() always use f->current_offsetMichal Schmidt
next_with_matches() is odd in that its "unit64_t *offset" parameter is both input and output. In other it's purely for output. The function is called from two places in next_beyond_location(). In both of them "&cp" is used as the argument and in both cases cp is guaranteed to equal f->current_offset. Let's just have next_with_matches() ignore "*offset" on input and operate with f->current_offset. I did not investigate why it is, but it makes my usual benchmark run reproducibly faster: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m4.032s user 0m3.896s sys 0m0.135s (Compare to preceding commit, where real was 4.4s.)
2014-12-19journal: fix skipping of duplicate entries in iterationMichal Schmidt
I accidentally broke the detection of duplicate entries in 7943f42275 "journal: optimize iteration by returning previously found candidate entry". When we have a known location of a candidate entry, we must not return from next_beyond_location() immediately. We must go through the duplicates detection to make sure the candidate differs from the already iterated entry. This fix slows down iteration a bit, but it's still faster than it was before the rework. $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m4.448s user 0m4.298s sys 0m0.149s (Compare with results from commit 7943f42275, where real was 5.3s before the rework.)
2014-12-18journal: next_with_matches() now does not need a mapped object as inputMichal Schmidt
Now that journal_file_next_entry() does not need a pointer to the current object, next_with_matches() does not need it either.
2014-12-18journal: journal_file_next_entry() does not need pointer to current ObjectMichal Schmidt
The current offset is sufficient information.
2014-12-18journal: optimize iteration by returning previously found candidate entryMichal Schmidt
In next_beyond_location() when the JournalFile's location type is LOCATION_SEEK, it means there's nothing to do, because we already have the location of the candidate entry. Do an early return. Note that now next_beyond_location() does not anymore guarantee on return that the entry is mapped, but previous patches made sure the caller does not care. This optimization is at least as good as "journal: optimize iteration: skip files that cannot improve current candidate entry" was. Timing results on my workstation, using: $ time ./journalctl -q --since=2014-06-01 --until=2014-07-01 > /dev/null Before "Revert "journal: optimize iteration: skip files that cannot improve current candidate entry": real 0m5.349s user 0m5.166s sys 0m0.181s Now: real 0m3.901s user 0m3.724s sys 0m0.176s
2014-12-18journal: optimize iteration by skipping exhausted filesMichal Schmidt
If from a previous iteration we know we are at the end of a journal file, don't bother looking into the file again. This is complicated by the fact that the EOF does not have to be permanent (think of "journalctl -f"). So we also check if the number of entries in the journal file changed. This optimization has a similar effect as "journal: optimize iteration: skip whole files behind current location" had.
2014-12-18journal: drop unnecessary parameters of next_beyond_location()Michal Schmidt
offset is redundant, because the caller can rely on f->current_offset. The object pointer the function saves in *ret is thrown away by the caller.
2014-12-18journal: remove redundant variable new_offsetMichal Schmidt
The file's current_offset is already updated at this point, so let's use it.
2014-12-18journal: compare candidate entries using JournalFiles' locationsMichal Schmidt
When comparing the locations of candidate entries, we can rely on the location information stored in struct JournalFile.
2014-12-18journal: simplify set_location()Michal Schmidt
set_location() is called from real_journal_next() when a winning entry has been picked from among the candidates in journal files. The location type is always set to LOCATION_DISCRETE. No need to pass it as a parameter. The per-JournalFile location information is already updated at this point. No need for having the direction and offset here.
2014-12-18journal: keep per-JournalFile location info during iterationMichal Schmidt
In next_beyond_location() when we find a candidate entry in a journal file, save its location information in struct JournalFile. The purpose of remembering the locations of candidate entries is to be able to save work in the next iteration. This patch does only the remembering part. LOCATION_SEEK means the location identifies a candidate entry. When a winner is picked from among candidates, it becomes LOCATION_DISCRETE. LOCATION_TAIL here signifies we've iterated the file to the end (or the beginning in the case of reversed direction).
2014-12-18journal: abstract the resetting of JournalFile's locationMichal Schmidt
2014-12-18Revert "journal: optimize iteration: skip whole files behind current location"Michal Schmidt
This reverts commit b7c88ab8cc7d55a43450bf3dea750f95f2e910d6. This optimization will be made redundant by the following patches.
2014-12-18Revert "journal: optimize iteration: skip files that cannot improve current ↵Michal Schmidt
candidate entry" This reverts commit f8b5a3b75fb55f0acb85c21424b3893c822742e9. This optimization will be made redundant by the following patches.
2014-12-13journal: consistently use OBJECT_<type> names instead of numbersMichal Schmidt
Note that numbers 0 and -1 are both replaced with OBJECT_UNUSED, because they are treated the same everywhere (e.g. type_to_context() translates them both to 0).
2014-12-13journal: remove journal_file_object_keep/release functionsMichal Schmidt
The only user is sd_journal_enumerate_unique() and, as explained in the previous commit (fed67c38e3 "journal: map objects to context set by caller, not by actual object type"), the use of them there is now superfluous. Let's remove them. This reverts major parts of commits: ae97089d49 journal: fix access to munmapped memory in sd_journal_enumerate_unique 06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values Tested with an "--enable-debug" build and "journalctl --list-boots". It gives the expected number of results. Additionally, if I then revert the previous commit ("journal: map objects to context set by caller, not to actual object type"), it crashes with SIGSEGV, as expected.
2014-12-10core: unify how we iterate over inotify eventsLennart Poettering
Let's add some syntactic sugar for iterating through inotify events, and use it everywhere.
2014-12-09journal: optimize iteration: skip files that cannot improve current ↵Michal Schmidt
candidate entry Suppose that while iterating we have already looked into a journal file and got a candidate for the next entry. And we are considering to look into another journal file because it may contain an entry that is nearer to the current location than the candidate. We should skip the whole journal file if we can tell by looking at its header that none of its entries can precede the candidate. Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m20.518s user 0m19.989s sys 0m0.328s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m9.445s user 0m9.228s sys 0m0.213s
2014-12-09journal: optimize iteration: skip whole files behind current locationMichal Schmidt
Interleaving of entries from many journal files is expensive. But there is room for optimization. We can skip looking into journal files whose entries all lie before the current iterating location. We can tell if that's the case from looking at the journal file header. This saves a huge amount of work if one has many of mostly not interleaved journal files. On my workstation with 90 journal files in /var/log/journal/ID/ totalling 3.4 GB I get these results: Before: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 5m54.258s user 2m4.263s sys 3m48.965s After: $ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null real 0m20.518s user 0m19.989s sys 0m0.328s The high "sys" time in the original was caused by putting more stress on the mmap-cache than it could handle. With the patch the working set now consists of fewer mmap windows and mmap-cache is not thrashing.
2014-11-28treewide: use log_*_errno whenever %m is in the format stringMichal Schmidt
If the format string contains %m, clearly errno must have a meaningful value, so we might as well use log_*_errno to have ERRNO= logged. Using: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/' Plus some whitespace, linewrap, and indent adjustments.
2014-11-28treewide: more log_*_errno() conversions, multiline callsMichal Schmidt
Basically: find . -name '*.[ch]' | while read f; do perl -i.mmm -e \ 'local $/; local $_=<>; s/log_(debug|info|notice|warning|error|emergency)\("([^"]*)%s"([^;]*),\s*strerror\(-?([->a-zA-Z_]+)\)\);/log_\1_errno(\4, "\2%m"\3);/gms;print;' \ $f; done Plus manual indentation fixups.
2014-11-28treewide: no need to negate errno for log_*_errno()Michal Schmidt
It corrrectly handles both positive and negative errno values.
2014-11-28treewide: auto-convert the simple cases to log_*_errno()Michal Schmidt
As a followup to 086891e5c1 "log: add an "error" parameter to all low-level logging calls and intrdouce log_error_errno() as log calls that take error numbers", use sed to convert the simple cases to use the new macros: find . -name '*.[ch]' | xargs sed -r -i -e \ 's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/' Multi-line log_*() invocations are not covered. And we also should add log_unit_*_errno().
2014-10-23journal: make sd_journal::files a OrderedHashmapMichal Schmidt
Anything that uses hashmap_next() almost certainly cares about the order and needs to be an OrderedHashmap.
2014-10-22machine: validate machine names using machine_name_is_valid() instead of ↵Lennart Poettering
string_is_safe() After all, we know have this as generic validator, so let's be correct and use it wherver applicable.
2014-10-20sd-journal: consistently use ternary operator for all direction checksChristian Hesse
2014-10-09sd-journal: fix sd_journal_enumerate_unique skipping valuesJan Janssen
sd_journal_enumerate_unique will lock its mmap window to prevent it from being released by calling mmap_cache_get with keep_always=true. This call may return windows that are wider, but compatible with the parameters provided to it. This can result in a mismatch where the window to be released cannot properly be selected, because we have more than one window matching the parameters of mmap_cache_release. Therefore, introduce a release_cookie to be used when releasing the window. https://bugs.freedesktop.org/show_bug.cgi?id=79380
2014-10-09sd-journal: do not reset sd_j_enumerate_unique position on errorZbigniew Jędrzejewski-Szmek
systemctl would call sd_j_enumerate_unique() interleaved with sd_j_next(). But the latter can remove a file if it detects an error in it. In those circumstances sd_j_enumerate_unique would restart with the first file in hashmap. With many corrupted files sd_j_enumerate_unique might iterate over the list multiple times. Avoid this by jumping to the next file in unique list if possible, or setting a flag that tells sd_j_enumerate_unique that it is done otherwise.
2014-09-15hashmap: introduce hash_ops to make struct Hashmap smallerMichal Schmidt
It is redundant to store 'hash' and 'compare' function pointers in struct Hashmap separately. The functions always comprise a pair. Store a single pointer to struct hash_ops instead. systemd keeps hundreds of hashmaps, so this saves a little bit of memory.
2014-08-27sd-journal: verify that object start with the field nameZbigniew Jędrzejewski-Szmek
If the journal is corrupted, we might return an object that does not start with the expected field name and/or is shorter than it should.
2014-08-26sd-journal: never log anything by default from a libraryLennart Poettering
2014-08-03Fix misuse of uint64_t as size_tZbigniew Jędrzejewski-Szmek
They have different size on 32 bit, so they are really not interchangable.
2014-07-31Reject invalid quoted stringsZbigniew Jędrzejewski-Szmek
String which ended in an unfinished quote were accepted, potentially with bad memory accesses. Reject anything which ends in a unfished quote, or contains non-whitespace characters right after the closing quote. _FOREACH_WORD now returns the invalid character in *state. But this return value is not checked anywhere yet. Also, make 'word' and 'state' variables const pointers, and rename 'w' to 'word' in various places. Things are easier to read if the same name is used consistently. mbiebl_> am I correct that something like this doesn't work mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-passwd "Unlock EncFS"' mbiebl_> systemd seems to strip of the quotes mbiebl_> systemctl status shows mbiebl_> ExecStart=/usr/bin/encfs --extpass='/bin/systemd-ask-password Unlock EncFS $RootDir $MountPoint mbiebl_> which is pretty weird
2014-07-18man,journal: add note about sd_journal_get_cutoff_monotonic_usec return valueZbigniew Jędrzejewski-Szmek
Also modify the function itself to be a bit simpler to read.
2014-07-11Fix build without any compression enabledZbigniew Jędrzejewski-Szmek
2014-07-07journal/compress: fix calls to decompress_blobZbigniew Jędrzejewski-Szmek
2014-07-06journal: add LZ4 as optional compressorZbigniew Jędrzejewski-Szmek
Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.
2014-05-25Use %m instead of strerror(errno) where appropiateCristian Rodríguez
2014-05-15Remove unnecessary casts in printfsZbigniew Jędrzejewski-Szmek
No functional change expected :)
2014-03-18util: replace close_nointr_nofail() by a more useful safe_close()Lennart Poettering
safe_close() automatically becomes a NOP when a negative fd is passed, and returns -1 unconditionally. This makes it easy to write lines like this: fd = safe_close(fd); Which will close an fd if it is open, and reset the fd variable correctly. By making use of this new scheme we can drop a > 200 lines of code that was required to test for non-negative fds or to reset the closed fd variable afterwards.