| Age | Commit message (Collapse) | Author |
|---|
|
Define DATA_SIZE_MAX to mean the maximum size of a single
field, and ENTRY_SIZE_MAX to mean the size of the whole
entry, with some rough calculation of overhead over the payload.
Check if entries are not too big when processing native journal
messages.
|
|
|
|
|
|
Also fix an infinite loop on E2BIG.
Remember what range we already scanned for '\n', to avoid
quadratic behaviour on long "text" fields.
|
|
Directory src/journal has become one of the largest directories,
and since systemd-journal-gatewayd, systemd-journal-remote, and
forthcoming systemd-journal-upload are all closely related, create
a separate directory for them.
|
|
|
|
If a file was opened for writing, and then closed immediately without
actually writing any entries, on subsequent opening, it would be
considered "corrupted". This should be totally fine, and even in
read mode, an empty file can become non-empty later on.
|
|
|
|
After all, rsyslog and friends nowadays read their data directly from
the journal, hence the forwarding is unnecessary in most cases.
|
|
The new lzma2 compression options at the top of compress_blob_xz are
equivalent to using preset "0", exept for using a 1 MiB dictionary
(the same as preset "1"). This makes the memory usage at most 7.5 MiB
in the compressor, and 1 MiB in the decompressor, instead of the
previous 92 MiB in the compressor and 8 MiB in the decompressor.
According to test-compress-benchmark this commit makes XZ compression
20 times faster, with no increase in compressed data size.
Using more realistic test data (an ELF binary rather than repeating
ASCII letters 'a' through 'z' in order) it only provides a factor 10
speedup, and at a cost if a 10% increase in compressed data size.
But that is still a worthwhile trade-off.
According to test-compress-benchmark XZ compression is still 25 times
slower than LZ4, but the compressed data is one eighth the size.
Using more realistic test data XZ compression is only 18 times slower
than LZ4, and the compressed data is only one quarter the size.
$ ./test-compress-benchmark
XZ: compressed & decompressed 2535300963 bytes in 42.30s (57.15MiB/s), mean compresion 99.95%, skipped 3570 bytes
LZ4: compressed & decompressed 2535303543 bytes in 1.60s (1510.60MiB/s), mean compresion 99.60%, skipped 990 bytes
|
|
|
|
|
|
|
|
This is useful to test the behaviour of the compressor for various buffer
sizes.
Time is limited to a minute per compression, since otherwise, when LZ4
takes more than a second which is necessary to reduce the noise, XZ
takes more than 10 minutes.
% build/test-compress-benchmark (without time limit)
XZ: compressed & decompressed 2535300963 bytes in 794.57s (3.04MiB/s), mean compresion 99.95%, skipped 3570 bytes
LZ4: compressed & decompressed 2535303543 bytes in 1.56s (1550.07MiB/s), mean compresion 99.60%, skipped 990 bytes
% build/test-compress-benchmark (with time limit)
XZ: compressed & decompressed 174321481 bytes in 60.02s (2.77MiB/s), mean compresion 99.76%, skipped 3570 bytes
LZ4: compressed & decompressed 2535303543 bytes in 1.63s (1480.83MiB/s), mean compresion 99.60%, skipped 990 bytes
It appears that there's a bug in lzma_end where it leaks 32 bytes.
|
|
Add liblz4 as an optional dependency when requested with --enable-lz4,
and use it in preference to liblzma for journal blob and coredump
compression. To retain backwards compatibility, XZ is used to
decompress old blobs.
Things will function correctly only with lz4-119.
Based on the benchmarks found on the web, lz4 seems to be the best
choice for "quick" compressors atm.
For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.
|
|
uncompress_startswith would always decode the whole stream, even
if it did not start with the given prefix.
Reallocation policy was also strange.
|
|
|
|
journald.conf(5) states that the default for MaxFileSec is one month,
but the code didn't respect that.
|
|
This also make sure we remove the original coredump temporary file if we
successfully managed to compress the coredump.
|
|
Let's move things closer to journald's configuration settings, which
knows Compress= already, as a boolean. This makes things more uniform,
but also gives us more freedom to possibly swap out the used compression
algorithm one day.
|
|
This sounds overly low-level and implementation-detaily. Let's just
use the default level XZ suggests. This gives us more room to possibly
swap out the compression algorithm used, as the compression level range
will not leak into user configuration.
|
|
|
|
while we work on it
|
|
|
|
When disk space taken up by coredumps grows beyond a configured limit
start removing the oldest coredump of the user with the most coredumps,
until we get below the limit again.
|
|
|
|
reorder the code so the fstat is done before we can jump to
uncompressed
|
|
|
|
typo from 347272731e15d3c4a70fad7ccd7185e8e8059d01
|
|
|
|
Fixes the following build error:
CCLD coredumpctl
src/journal/coredumpctl.o: In function `save_core':
/src/systemd-master/src/journal/coredumpctl.c:656:
undefined reference to `decompress_stream'
collect2: error: ld returned 1 exit status
make[2]: *** [coredumpctl] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
|
|
|
|
|
|
Add Compression={none,xz} and CompressionLevel=0-9 settings. Defaults
are xz/6.
Compression=filesystem may be added later.
I picked "xz" for the compression "type", since we might want to add
different compressors later on. XZ is fairly memory and CPU intensive, and
embedded users will likely want to use LZO or some other lightweight compression
mechanism.
|
|
Unfortunately the core is first written uncompressed, then compressed
by reading from disk and writing to the output file. This is ugly and
slow, but I don't see a way around, if we want to get the backtrace
without keeping everything in memory.
|
|
|
|
|
|
|
|
Journal might be functional even if we cannot write to
/var/lib/systemd/coredump.
|
|
Kernel mangles comm information in an irreversible way when comm
constains repeated spaces. Retrieve comm information from /proc, and
only fallback to the information provided on the commandline when
retrieving information from /proc fails.
Add exe information to the list of saved xattr.
https://bugs.freedesktop.org/show_bug.cgi?id=62043
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The correct path is now <sys/xattr.h> (from glibc-headers) and no longer
<attr/xattr.h> (from libattr-devel.)
Fixes: 34c10968cbe3b5591b3c0ce225b8694edd9709d0
|
|
|
|
|
|
|
