summaryrefslogtreecommitdiff
path: root/src/journal/compress.c
AgeCommit message (Collapse)Author
2015-09-10tree-wide: never use the off_t unless glibc makes us use itLennart Poettering
off_t is a really weird type as it is usually 64bit these days (at least in sane programs), but could theoretically be 32bit. We don't support off_t as 32bit builds though, but still constantly deal with safely converting from off_t to other types and back for no point. Hence, never use the type anymore. Always use uint64_t instead. This has various benefits, including that we can expose these values directly as D-Bus properties, and also that the values parse the same in all cases.
2015-03-09Introduce loop_read_exact helperZbigniew Jędrzejewski-Szmek
Usually when using loop_read(), we want to read the full buffer. Add a helper that mirrors loop_write(), and returns 0 when full buffer was read, and an error otherwise. Use -ENODATA for the short read, to distinguish it from a read error.
2015-02-23remove unused includesThomas Hindoe Paaboel Andersen
This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.
2015-01-22Assorted format fixesZbigniew Jędrzejewski-Szmek
Types used for pids and uids in various interfaces are unpredictable. Too bad.
2014-12-09treewide: sanitize loop_writeZbigniew Jędrzejewski-Szmek
loop_write() didn't follow the usual systemd rules and returned status partially in errno and required extensive checks from callers. Some of the callers dealt with this properly, but many did not, treating partial writes as successful. Simplify things by conforming to usual rules.
2014-08-30journal/compress: use LZ4_compress_continue()Evangelos Foutras
We can't use LZ4_compress_limitedOutput_continue() because in the worst-case scenario the compressed output can be slightly bigger than the input block. This generally affects very few blocks and is no reason to abort the compression process. I ran into this when I noticed that Chromium core dumps weren't being compressed. After switching to LZ4_compress_continue() a ~330MB Chromium core dump gets compressed to ~17M.
2014-08-03Fix misuse of uint64_t as size_tZbigniew Jędrzejewski-Szmek
They have different size on 32 bit, so they are really not interchangable.
2014-07-18compress: fix return valueZbigniew Jędrzejewski-Szmek
2014-07-11Fix build without any compression enabledZbigniew Jędrzejewski-Szmek
2014-07-08journal/compress: improve xz compression performanceJon Severinsson
The new lzma2 compression options at the top of compress_blob_xz are equivalent to using preset "0", exept for using a 1 MiB dictionary (the same as preset "1"). This makes the memory usage at most 7.5 MiB in the compressor, and 1 MiB in the decompressor, instead of the previous 92 MiB in the compressor and 8 MiB in the decompressor. According to test-compress-benchmark this commit makes XZ compression 20 times faster, with no increase in compressed data size. Using more realistic test data (an ELF binary rather than repeating ASCII letters 'a' through 'z' in order) it only provides a factor 10 speedup, and at a cost if a 10% increase in compressed data size. But that is still a worthwhile trade-off. According to test-compress-benchmark XZ compression is still 25 times slower than LZ4, but the compressed data is one eighth the size. Using more realistic test data XZ compression is only 18 times slower than LZ4, and the compressed data is only one quarter the size. $ ./test-compress-benchmark XZ: compressed & decompressed 2535300963 bytes in 42.30s (57.15MiB/s), mean compresion 99.95%, skipped 3570 bytes LZ4: compressed & decompressed 2535303543 bytes in 1.60s (1510.60MiB/s), mean compresion 99.60%, skipped 990 bytes
2014-07-06journal: add LZ4 as optional compressorZbigniew Jędrzejewski-Szmek
Add liblz4 as an optional dependency when requested with --enable-lz4, and use it in preference to liblzma for journal blob and coredump compression. To retain backwards compatibility, XZ is used to decompress old blobs. Things will function correctly only with lz4-119. Based on the benchmarks found on the web, lz4 seems to be the best choice for "quick" compressors atm. For pkg-config status, see http://code.google.com/p/lz4/issues/detail?id=135.
2014-07-06journal/compress: return early in uncompress_startswithZbigniew Jędrzejewski-Szmek
uncompress_startswith would always decode the whole stream, even if it did not start with the given prefix. Reallocation policy was also strange.
2014-06-26coredump: make compression configurableZbigniew Jędrzejewski-Szmek
Add Compression={none,xz} and CompressionLevel=0-9 settings. Defaults are xz/6. Compression=filesystem may be added later. I picked "xz" for the compression "type", since we might want to add different compressors later on. XZ is fairly memory and CPU intensive, and embedded users will likely want to use LZO or some other lightweight compression mechanism.
2014-06-26journal/compress: add stream compression/decompression functionsZbigniew Jędrzejewski-Szmek
2014-06-26journal/compress: simplify compress_blobZbigniew Jędrzejewski-Szmek
2012-11-21journal: by default do not decompress dat objects larger than 64KLennart Poettering
This introduces a new data threshold setting for sd_journal objects which controls the maximum size of objects to decompress. This is relieves the library from having to decompress full data objects even if a client program is only interested in the initial part of them. This speeds up "systemd-coredumpctl" drastically when invoked without parameters.
2012-04-12relicense to LGPLv2.1 (with exceptions)Lennart Poettering
We finally got the OK from all contributors with non-trivial commits to relicense systemd from GPL2+ to LGPL2.1+. Some udev bits continue to be GPL2+ for now, but we are looking into relicensing them too, to allow free copy/paste of all code within systemd. The bits that used to be MIT continue to be MIT. The big benefit of the relicensing is that closed source code may now link against libsystemd-login.so and friends.
2011-12-21journal: add missing compress.[ch]Lennart Poettering