Merge pull request #3126 from poettering/small-fixes

fsync directory when creating or rotating journal files and other small fixes,
most importantly for the DHCP DUID code.

10 files changed, 115 insertions, 95 deletions

diff --git a/.vimrc b/.vimrc
index 7b436bd377..284bf88494 100644
--- a/.vimrc
+++ b/.vimrc
@@ -16,5 +16,5 @@ set shiftwidth=8
 set expandtab
 set makeprg=GCC_COLORS=\ make
 set tw=79
-au FileType xml set tw=119
+au BufRead,BufNewFile *.xml set tw=119 shiftwidth=2 smarttab
 au FileType c set tw=119
diff --git a/TODO b/TODO
index ad72ba6076..f088cac5b7 100644
--- a/TODO
+++ b/TODO
@@ -33,6 +33,10 @@ Janitorial Clean-ups:
 
 Features:
 
+* journalctl: make sure -f ends when the container indicated by -M terminates
+
+* make "machinectl clone" properly async, and add fallback for non-tmpfs
+
 * rework fopen_temporary() to make use of open_tmpfile_linkable() (problem: the
   kernel doesn't support linkat() that replaces existing files, currently)
 
@@ -99,9 +103,6 @@ Features:
 
 * install: include generator dirs in unit file search paths
 
-* rework C11 utf8.[ch] to use char32_t instead of uint32_t when referring
-  to unicode chars, to make things more expressive.
-
 * fstab-generator: default to tmpfs-as-root if only usr= is specified on the kernel cmdline
 
 * docs: bring http://www.freedesktop.org/wiki/Software/systemd/MyServiceCantGetRealtime up to date
@@ -151,8 +152,6 @@ Features:
 * as soon as we have kdbus, and sender timestamps, revisit coalescing multiple parallel daemon reloads:
   http://lists.freedesktop.org/archives/systemd-devel/2014-December/025862.html
 
-* the install state probably shouldn't get confused by generated units, think dbus1/kdbus compat!
-
 * in systemctl list-unit-files: show the install value the presets would suggest for a service in a third column
 
 * figure out when we can use the coarse timers
@@ -164,8 +163,6 @@ Features:
 
 * firstboot: make it useful to be run immediately after yum --installroot to set up a machine. (most specifically, make --copy-root-password work even if /etc/passwd already exists
 
-* add infrastructure to allocate dynamic/transient users and UID ranges, for use in user-namespaced containers, per-seat gdm login screens and gdm guest sessions
-
 * maybe add support for specifier expansion in user.conf, specifically DefaultEnvironment=
 
 * introduce systemd-timesync-wait.service or so to sync on an NTP fix?
@@ -252,7 +249,7 @@ Features:
   CAP_NET_ADMIN is set, more than the loopback device is defined, even
   when it is otherwise off
 
-* MessageQueueMessageSize= and RLimitFSIZE= (and suchlike) should use parse_iec_size().
+* MessageQueueMessageSize= should use parse_iec_size().
 
 * "busctl status" works only as root on dbus1, since we cannot read
   /proc/$PID/exe
@@ -267,7 +264,7 @@ Features:
   and passes this back to PID1 via SCM_RIGHTS. This also could be used
   to allow Chown/chgrp on sockets without requiring NSS in PID 1.
 
-* New service property: maximum CPU and wallclock runtime for a service
+* New service property: maximum CPU runtime for a service
 
 * introduce bus call FreezeUnit(s, b), as well as "systemctl freeze
   $UNIT" and "systemctl thaw $UNIT" as wrappers around this. The calls
@@ -509,8 +506,6 @@ Features:
     written to as FAIL, but instead show that their are being written to.
   - add journalctl -H that talks via ssh to a remote peer and passes through
     binary logs data
-  - change journalctl -M to acquire fd to journal directory via machined, and
-    then operate on that via openat() instead of absolute paths
   - add a version of --merge which also merges /var/log/journal/remote
   - log accumulated resource usage after each service invocation
   - journalctl: -m should access container journals directly by enumerating
@@ -549,7 +544,6 @@ Features:
 * unit install:
   - "systemctl mask" should find all names by which a unit is accessible
     (i.e. by scanning for symlinks to it) and link them all to /dev/null
-  - systemctl list-unit-files should list generated files (and probably with a new state "generated" for them, or so)
 
 * timer units:
   - timer units should get the ability to trigger when:
@@ -596,8 +590,6 @@ Features:
   - should send out sd_notify("WATCHDOG=1") messages
   - optionally automatically add FORWARD rules to iptables whenever nspawn is
     running, remove them when shut down.
-  - add a logic for cleaning up read-only, hidden container images in
-    /var/lib/machines that are not ancestors of any non-hidden containers
   - Improve error message when --bind= is used on a non-existing source
     directory
   - maybe make copying of /etc/resolv.conf optional, and skip it if --read-only
@@ -641,8 +633,6 @@ Features:
 
 * initialize the hostname from the fs label of /, if /etc/hostname does not exist?
 
-* rename "userspace" to "core-os"
-
 * udev:
   - move to LGPL
   - kill scsi_id
@@ -757,11 +747,6 @@ Features:
   - Allow multiple ExecStart= for all Type= settings, so that we can cover rescue.service nicely
   - consider adding RuntimeDirectoryUser= + RuntimeDirectoryGroup=
 
-* systemd-python:
-   - figure out a simple way to wait for journal events in a way that
-     works with ^C
-   - add documentation to systemd.daemon
-
 * udev-link-config:
    - Make sure ID_PATH is always exported and complete for
      network devices where possible, so we can safely rely
diff --git a/man/systemd.exec.xml b/man/systemd.exec.xml
index fea42ebd31..2d0fb63f1d 100644
--- a/man/systemd.exec.xml
+++ b/man/systemd.exec.xml
@@ -94,10 +94,9 @@
     required to access <filename>/tmp</filename> and
     <filename>/var/tmp</filename>.</para>
 
-    <para>Units whose output standard output or error output is connected to <option>journal</option>,
-    <option>syslog</option> or <option>kmsg</option> (or their combinations with console output, see below)
-    automatically acquire dependencies of type <varname>After=</varname> on
-    <filename>systemd-journald.socket</filename>.</para>
+    <para>Units whose standard output or error output is connected to <option>journal</option>, <option>syslog</option>
+    or <option>kmsg</option> (or their combinations with console output, see below) automatically acquire dependencies
+    of type <varname>After=</varname> on <filename>systemd-journald.socket</filename>.</para>
   </refsect1>
 
   <refsect1>
@@ -469,8 +468,8 @@
         similar to the same option of
         <varname>StandardInput=</varname>.</para>
 
-        <para>If the standard output (or error output, see below) of a unit is connected with the journal, syslog or
-        the kernel log buffer the unit will implicitly gain a dependency of type <varname>After=</varname> on
+        <para>If the standard output (or error output, see below) of a unit is connected to the journal, syslog or the
+        kernel log buffer, the unit will implicitly gain a dependency of type <varname>After=</varname> on
         <filename>systemd-journald.socket</filename> (also see the automatic dependencies section above).</para>
 
         <para>This setting defaults to the value set with
diff --git a/man/systemd.network.xml b/man/systemd.network.xml
index 9bf1b198ad..2a20748376 100644
--- a/man/systemd.network.xml
+++ b/man/systemd.network.xml
@@ -815,9 +815,8 @@
         <varlistentry>
           <term><varname>ClientIdentifier=</varname></term>
           <listitem>
-            <para>DHCP client identifier to use. Either <literal>mac</literal>
-            to use the MAC address of the link or <literal>duid</literal>
-            (the default) to use a RFC4361-compliant Client ID.</para>
+            <para>The DHCPv4 client identifier to use. Either <literal>mac</literal> to use the MAC address of the link
+            or <literal>duid</literal> (the default, see below) to use a RFC4361-compliant Client ID.</para>
           </listitem>
         </varlistentry>
         <varlistentry>
@@ -828,6 +827,25 @@
           </listitem>
         </varlistentry>
         <varlistentry>
+          <term><varname>DUIDRawData=</varname></term>
+          <listitem><para>Specifies the DHCP DUID bytes as a single newline-terminated, hexadecimal string, with each
+          byte separated by a ':'.  A DHCPv6 client sends the DHCP Unique Identifier (DUID) and the interface Identity
+          Association Identifier (IAID) to a DHCP server when acquiring a dynamic IPv6 address. Similar, DHCPv4 clients
+          send the IAID and DUID to the DHCP server when acquiring a dynamic IPv4 address if
+          <option>ClientIdentifier=duid</option>. IAID and DUID allows a DHCP server to uniquely identify the machine
+          and the interface requesting a DHCP IP address.</para>
+
+          <para>The DUID value specified here takes precedence over the DUID that systemd-networkd generates
+          using the machine-id from the <filename>/etc/machine-id</filename> file, as well as the
+          global DUID that may be specified in <citerefentry><refentrytitle>networkd.conf
+          </refentrytitle><manvolnum>5</manvolnum></citerefentry>.</para>
+
+          <para>The configured DHCP DUID should conform to the specification in
+          <ulink url="http://tools.ietf.org/html/rfc3315#section-9">RFC 3315</ulink>,
+          <ulink url="http://tools.ietf.org/html/rfc6355">RFC 6355</ulink>.</para>
+          </listitem>
+        </varlistentry>
+        <varlistentry>
           <term><varname>RequestBroadcast=</varname></term>
           <listitem>
             <para>Request the server to use broadcast messages before
@@ -846,40 +864,7 @@
           </listitem>
         </varlistentry>
       </variablelist>
-
-  </refsect1>
-
-  <refsect1>
-    <title>[DUID] Section Options</title>
-
-    <para>This section configures the DHCP Unique Identifier (DUID) value used by DHCP
-    protocol. DHCPv6 client protocol sends the DHCP Unique Identifier and the interface
-    Identity Association Identifier (IAID) to a DHCP server when acquiring a dynamic IPv6
-    address. DHCPv4 client protocol sends IAID and DUID to the DHCP server when acquiring
-    a dynamic IPv4 address if <option>ClientIdentifier=duid</option>. IAID and DUID allows a
-    DHCP server to uniquely identify the machine and the interface requesting a DHCP IP.</para>
-
-    <para>The DUID value specified here overrides the DUID that systemd-networkd generates
-    using the machine-id from the <filename>/etc/machine-id</filename> file, as well as the
-    global DUID that may be specified in <citerefentry><refentrytitle>networkd.conf
-    </refentrytitle><manvolnum>5</manvolnum></citerefentry>.</para>
-
-    <para>The configured DHCP DUID should conform to the specification in
-    <ulink url="http://tools.ietf.org/html/rfc3315#section-9">RFC 3315</ulink>,
-    <ulink url="http://tools.ietf.org/html/rfc6355">RFC 6355</ulink>.</para>
-
-    <para>The following options are available in <literal>[DUID]</literal> section:</para>
-
-    <variablelist class='network-directives'>
-
-      <varlistentry>
-        <term><varname>RawData=</varname></term>
-        <listitem><para>Specifies the DUID bytes as a single newline-terminated, hexadecimal
-        string, with each byte separated by a ':'.</para></listitem>
-      </varlistentry>
-
-    </variablelist>
-  </refsect1>
+    </refsect1>
 
   <refsect1>
     <title>[DHCPServer] Section Options</title>
diff --git a/src/basic/fd-util.c b/src/basic/fd-util.c
index 9130d023d7..8b466cff15 100644
--- a/src/basic/fd-util.c
+++ b/src/basic/fd-util.c
@@ -361,8 +361,14 @@ bool fdname_is_valid(const char *s) {
 
 int fd_get_path(int fd, char **ret) {
         char procfs_path[strlen("/proc/self/fd/") + DECIMAL_STR_MAX(int)];
+        int r;
 
         xsprintf(procfs_path, "/proc/self/fd/%i", fd);
 
-        return readlink_malloc(procfs_path, ret);
+        r = readlink_malloc(procfs_path, ret);
+
+        if (r == -ENOENT) /* If the file doesn't exist the fd is invalid */
+                return -EBADF;
+
+        return r;
 }
diff --git a/src/journal/journal-file.c b/src/journal/journal-file.c
index 35f4abab1d..c9ce5c73be 100644
--- a/src/journal/journal-file.c
+++ b/src/journal/journal-file.c
@@ -439,6 +439,39 @@ static int journal_file_init_header(JournalFile *f, JournalFile *template) {
         return 0;
 }
 
+static int fsync_directory_of_file(int fd) {
+        _cleanup_free_ char *path = NULL, *dn = NULL;
+        _cleanup_close_ int dfd = -1;
+        struct stat st;
+        int r;
+
+        if (fstat(fd, &st) < 0)
+                return -errno;
+
+        if (!S_ISREG(st.st_mode))
+                return -EBADFD;
+
+        r = fd_get_path(fd, &path);
+        if (r < 0)
+                return r;
+
+        if (!path_is_absolute(path))
+                return -EINVAL;
+
+        dn = dirname_malloc(path);
+        if (!dn)
+                return -ENOMEM;
+
+        dfd = open(dn, O_RDONLY|O_CLOEXEC|O_DIRECTORY);
+        if (dfd < 0)
+                return -errno;
+
+        if (fsync(dfd) < 0)
+                return -errno;
+
+        return 0;
+}
+
 static int journal_file_refresh_header(JournalFile *f) {
         sd_id128_t boot_id;
         int r;
@@ -464,6 +497,9 @@ static int journal_file_refresh_header(JournalFile *f) {
         /* Sync the online state to disk */
         (void) fsync(f->fd);
 
+        /* We likely just created a new file, also sync the directory this file is located in. */
+        (void) fsync_directory_of_file(f->fd);
+
         return r;
 }
 
@@ -3177,6 +3213,9 @@ int journal_file_rotate(JournalFile **f, bool compress, bool seal, Set *deferred
         if (r < 0 && errno != ENOENT)
                 return -errno;
 
+        /* Sync the rename to disk */
+        (void) fsync_directory_of_file(old_file->fd);
+
         /* Set as archive so offlining commits w/state=STATE_ARCHIVED.
          * Previously we would set old_file->header->state to STATE_ARCHIVED directly here,
          * but journal_file_set_offline() short-circuits when state != STATE_ONLINE, which
diff --git a/src/network/networkd-conf.c b/src/network/networkd-conf.c
index 73a8d16b58..70f0121d6d 100644
--- a/src/network/networkd-conf.c
+++ b/src/network/networkd-conf.c
@@ -31,7 +31,7 @@ int manager_parse_config_file(Manager *m) {
 
         return config_parse_many(PKGSYSCONFDIR "/networkd.conf",
                                  CONF_PATHS_NULSTR("systemd/networkd.conf.d"),
-                                 "DUID\0",
+                                 "DHCP\0",
                                  config_item_perf_lookup, networkd_gperf_lookup,
                                  false, m);
 }
@@ -57,7 +57,8 @@ int config_parse_duid_rawdata(
                 const char *rvalue,
                 void *data,
                 void *userdata) {
-        int r, n1, n2, byte;
+
+        int r;
         char *cbyte;
         const char *pduid = rvalue;
         Manager *m = userdata;
@@ -72,71 +73,78 @@ int config_parse_duid_rawdata(
         assert(rvalue);
         assert(userdata);
 
-        duidtype = (ltype == DUID_CONFIG_SOURCE_GLOBAL) ? m->duid_type
-                                                        : n->duid_type;
+        duidtype = (ltype == DUID_CONFIG_SOURCE_GLOBAL) ? m->duid_type : n->duid_type;
 
         if (duidtype == _DUID_TYPE_INVALID)
                 duidtype = DUID_TYPE_RAW;
 
         switch (duidtype) {
+
         case DUID_TYPE_LLT:
                 /* RawData contains DUID-LLT link-layer address (offset 6) */
                 duid_start_offset = 6;
                 break;
+
         case DUID_TYPE_EN:
                 /* RawData contains DUID-EN identifier (offset 4) */
                 duid_start_offset = 4;
                 break;
+
         case DUID_TYPE_LL:
                 /* RawData contains DUID-LL link-layer address (offset 2) */
                 duid_start_offset = 2;
                 break;
+
         case DUID_TYPE_UUID:
                 /* RawData specifies UUID (offset 0) - fall thru */
+
         case DUID_TYPE_RAW:
                 /* First two bytes of RawData is DUID Type - fall thru */
+
         default:
                 break;
         }
 
         if (duidtype != DUID_TYPE_RAW)
-                dhcp_duid_type = (uint16_t)duidtype;
+                dhcp_duid_type = (uint16_t) duidtype;
 
         /* RawData contains DUID in format " NN:NN:NN... " */
         for (;;) {
+                int n1, n2;
+                uint32_t byte;
+
                 r = extract_first_word(&pduid, &cbyte, ":", 0);
                 if (r < 0) {
-                        log_syntax(unit, LOG_ERR, filename, line, r,
-                                   "Failed to read DUID, ignoring assignment: %s.", rvalue);
-                        goto exit;
+                        log_syntax(unit, LOG_ERR, filename, line, r, "Failed to read DUID, ignoring assignment: %s.", rvalue);
+                        return 0;
                 }
                 if (r == 0)
                         break;
-                if ((duid_start_offset + dhcp_duid_len) >= MAX_DUID_LEN) {
-                        log_syntax(unit, LOG_ERR, filename, line, 0,
-                                   "Max DUID length exceeded, ignoring assignment: %s.", rvalue);
-                        goto exit;
+                if (duid_start_offset + dhcp_duid_len >= MAX_DUID_LEN) {
+                        log_syntax(unit, LOG_ERR, filename, line, 0, "Max DUID length exceeded, ignoring assignment: %s.", rvalue);
+                        return 0;
                 }
 
                 len = strlen(cbyte);
-                if ((len == 0) || (len > 2)) {
-                        log_syntax(unit, LOG_ERR, filename, line, 0,
-                                   "Invalid length - DUID byte: %s, ignoring assignment: %s.", cbyte, rvalue);
-                        goto exit;
+                if (len != 1 && len != 2) {
+                        log_syntax(unit, LOG_ERR, filename, line, 0, "Invalid length - DUID byte: %s, ignoring assignment: %s.", cbyte, rvalue);
+                        return 0;
                 }
-                n2 = 0;
                 n1 = unhexchar(cbyte[0]);
                 if (len == 2)
                         n2 = unhexchar(cbyte[1]);
-                if ((n1 < 0) || (n2 < 0)) {
-                        log_syntax(unit, LOG_ERR, filename, line, 0,
-                                   "Invalid DUID byte: %s. Ignoring assignment: %s.", cbyte, rvalue);
-                        goto exit;
+                else
+                        n2 = 0;
+
+                if (n1 < 0 || n2 < 0) {
+                        log_syntax(unit, LOG_ERR, filename, line, 0, "Invalid DUID byte: %s. Ignoring assignment: %s.", cbyte, rvalue);
+                        return 0;
                 }
-                byte = (n1 << (4 * (len-1))) | n2;
+
+                byte = ((uint8_t) n1 << (4 * (len-1))) | (uint8_t) n2;
 
                 /* If DUID_TYPE_RAW, first two bytes hold DHCP DUID type code */
-                if ((duidtype == DUID_TYPE_RAW) && (count < 2)) {
+                if (duidtype == DUID_TYPE_RAW && count < 2) {
                         dhcp_duid_type |= (byte << (8 * (1 - count)));
                         count++;
                         continue;
@@ -159,6 +167,5 @@ int config_parse_duid_rawdata(
                 memcpy(&n->dhcp_duid[duid_start_offset], dhcp_duid, dhcp_duid_len);
         }
 
-exit:
         return 0;
 }
diff --git a/src/network/networkd-gperf.gperf b/src/network/networkd-gperf.gperf
index 0625fb335b..afc71b4cb8 100644
--- a/src/network/networkd-gperf.gperf
+++ b/src/network/networkd-gperf.gperf
@@ -14,5 +14,5 @@ struct ConfigPerfItem;
 %struct-type
 %includes
 %%
-DUID.Type,              config_parse_duid_type,                 0,                                  offsetof(Manager, duid_type)
-DUID.RawData,           config_parse_duid_rawdata,              DUID_CONFIG_SOURCE_GLOBAL,          offsetof(Manager, dhcp_duid)
+DHCP.DUIDType,              config_parse_duid_type,                 0,                                  offsetof(Manager, duid_type)
+DHCP.DUIDRawData,           config_parse_duid_rawdata,              DUID_CONFIG_SOURCE_GLOBAL,          offsetof(Manager, dhcp_duid)
diff --git a/src/network/networkd-network-gperf.gperf b/src/network/networkd-network-gperf.gperf
index 1da99cd5bc..654d6a0316 100644
--- a/src/network/networkd-network-gperf.gperf
+++ b/src/network/networkd-network-gperf.gperf
@@ -28,8 +28,6 @@ Match.Architecture,                     config_parse_net_condition,
 Link.MACAddress,                        config_parse_hwaddr,                            0,                             offsetof(Network, mac)
 Link.MTUBytes,                          config_parse_iec_size,                          0,                             offsetof(Network, mtu)
 Link.IAID,                              config_parse_iaid,                              0,                             offsetof(Network, iaid)
-DUID.Type,                              config_parse_duid_type,                         0,                             offsetof(Network, duid_type)
-DUID.RawData,                           config_parse_duid_rawdata,                      DUID_CONFIG_SOURCE_NETWORK,    offsetof(Network, dhcp_duid)
 Network.Description,                    config_parse_string,                            0,                             offsetof(Network, description)
 Network.Bridge,                         config_parse_netdev,                            0,                             offsetof(Network, bridge)
 Network.Bond,                           config_parse_netdev,                            0,                             offsetof(Network, bond)
@@ -85,6 +83,8 @@ DHCP.Hostname,                          config_parse_hostname,
 DHCP.RequestBroadcast,                  config_parse_bool,                              0,                             offsetof(Network, dhcp_broadcast)
 DHCP.CriticalConnection,                config_parse_bool,                              0,                             offsetof(Network, dhcp_critical)
 DHCP.VendorClassIdentifier,             config_parse_string,                            0,                             offsetof(Network, dhcp_vendor_class_identifier)
+DHCP.DUIDType,                          config_parse_duid_type,                         0,                             offsetof(Network, duid_type)
+DHCP.DUIDRawData,                       config_parse_duid_rawdata,                      DUID_CONFIG_SOURCE_NETWORK,    offsetof(Network, dhcp_duid)
 DHCP.RouteMetric,                       config_parse_unsigned,                          0,                             offsetof(Network, dhcp_route_metric)
 DHCP.UseTimezone,                       config_parse_bool,                              0,                             offsetof(Network, dhcp_use_timezone)
 DHCPServer.MaxLeaseTimeSec,             config_parse_sec,                               0,                             offsetof(Network, dhcp_server_max_lease_time_usec)
diff --git a/src/network/networkd-network.c b/src/network/networkd-network.c
index 07f8fb028f..2ebcdfa744 100644
--- a/src/network/networkd-network.c
+++ b/src/network/networkd-network.c
@@ -137,7 +137,6 @@ static int network_load_one(Manager *manager, const char *filename) {
         r = config_parse(NULL, filename, file,
                          "Match\0"
                          "Link\0"
-                         "DUID\0"
                          "Network\0"
                          "Address\0"
                          "Route\0"