journal: set STATE_ARCHIVED as part of offlining (#2740)

The only code path which makes a journal durable is via
journal_file_set_offline().

When we perform a rotate the journal's header->state is being set to
STATE_ARCHIVED prior to journal_file_set_offline() being called.

In journal_file_set_offline(), we short-circuit the entire offline when
f->header->state != STATE_ONLINE.

This all results in none of the journal_file_set_offline() fsync() calls
being reached when rotate archives a journal, so archived journals are
never explicitly made durable.

What we do now is instead of setting the f->header->state to
STATE_ARCHIVED directly in journal_file_rotate() prior to
journal_file_close(), we set an archive flag in f->archive for the
journal_file_set_offline() machinery to honor by committing
STATE_ARCHIVED instead of STATE_OFFLINE when set.

Prior to this, rotated journals were never getting fsync() explicitly
performed on them, since journal_file_set_offline() short-circuited.
Obviously this is undesirable, and depends entirely on the underlying
filesystem as to how much durability was achieved when simply closing
the file.

Note that this problem existed prior to the recent asynchronous fsync
changes, but those changes do facilitate our performing this durable
offline on rotate without blocking, regardless of the underlying
filesystem sync-on-close semantics.

2 files changed, 9 insertions, 2 deletions

diff --git a/src/journal/journal-file.c b/src/journal/journal-file.c
index ff01e5aa94..35f4abab1d 100644
--- a/src/journal/journal-file.c
+++ b/src/journal/journal-file.c
@@ -120,7 +120,7 @@ static void journal_file_set_offline_internal(JournalFile *f) {
                         if (!__sync_bool_compare_and_swap(&f->offline_state, OFFLINE_SYNCING, OFFLINE_OFFLINING))
                                 continue;
 
-                        f->header->state = STATE_OFFLINE;
+                        f->header->state = f->archive ? STATE_ARCHIVED : STATE_OFFLINE;
                         (void) fsync(f->fd);
                         break;
 
@@ -3177,7 +3177,13 @@ int journal_file_rotate(JournalFile **f, bool compress, bool seal, Set *deferred
         if (r < 0 && errno != ENOENT)
                 return -errno;
 
-        old_file->header->state = STATE_ARCHIVED;
+        /* Set as archive so offlining commits w/state=STATE_ARCHIVED.
+         * Previously we would set old_file->header->state to STATE_ARCHIVED directly here,
+         * but journal_file_set_offline() short-circuits when state != STATE_ONLINE, which
+         * would result in the rotated journal never getting fsync() called before closing.
+         * Now we simply queue the archive state by setting an archive bit, leaving the state
+         * as STATE_ONLINE so proper offlining occurs. */
+        old_file->archive = true;
 
         /* Currently, btrfs is not very good with out write patterns
          * and fragments heavily. Let's defrag our journal files when
diff --git a/src/journal/journal-file.h b/src/journal/journal-file.h
index a0ec8c284b..564e1a8179 100644
--- a/src/journal/journal-file.h
+++ b/src/journal/journal-file.h
@@ -86,6 +86,7 @@ typedef struct JournalFile {
         bool seal:1;
         bool defrag_on_close:1;
         bool close_fd:1;
+        bool archive:1;
 
         bool tail_entry_monotonic_valid:1;