service: add 'debug' option to RestartMode=

One of the major pait points of managing fleets of headless nodes is
that when something fails at startup, unless debug level was already
enabled (which usually isn't, as it's a firehose), one needs to manually
enable it and pray the issue can be reproduced, which often is really
hard and time consuming, just to get extra info. Usually the extra log
messages are enough to triage an issue.

This new option makes it so that when a service fails and is restarted
due to Restart=, log level for that unit is set to debug, so that all
setup code in pid1 and sd-executor logs at debug level, and also a new
DEBUG_INVOCATION=1 env var is passed to the service itself, so that it
knows it should start with a higher log level. Once the unit succeeds
or reaches the rate limit the original level is restored.

1 files changed, 12 insertions, 0 deletions

diff --git a/man/org.freedesktop.systemd1.xml b/man/org.freedesktop.systemd1.xml
index b9120cc222..d445c138fa 100644
--- a/man/org.freedesktop.systemd1.xml
+++ b/man/org.freedesktop.systemd1.xml
@@ -2115,6 +2115,8 @@ node /org/freedesktop/systemd1/unit/avahi_2ddaemon_2eservice {
       @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
       readonly as Refs = ['...', ...];
       readonly a(ss) ActivationDetails = [...];
+      @org.freedesktop.DBus.Property.EmitsChangedSignal("false")
+      readonly b DebugInvocation = ...;
   };
   interface org.freedesktop.DBus.Peer { ... };
   interface org.freedesktop.DBus.Introspectable { ... };
@@ -2458,6 +2460,8 @@ node /org/freedesktop/systemd1/unit/avahi_2ddaemon_2eservice {
 
     <variablelist class="dbus-property" generated="True" extra-ref="ActivationDetails"/>
 
+    <variablelist class="dbus-property" generated="True" extra-ref="DebugInvocation"/>
+
     <!--End of Autogenerated section-->
 
     <refsect2>
@@ -2644,6 +2648,13 @@ node /org/freedesktop/systemd1/unit/avahi_2ddaemon_2eservice {
       <citerefentry><refentrytitle>systemd.exec</refentrytitle><manvolnum>5</manvolnum></citerefentry>. Note
       that new key value pair may be added at any time in future versions. Existing entries will not be
       removed.</para>
+
+      <para><varname>DebugInvocation</varname> contains a boolean that will change to
+      <constant>true</constant> when the unit is restarted with log level set to debug due to an earlier
+      failure, and will change back to <constant>false</constant> when either the unit successfully starts
+      or the restart rate limit is reached. See the <literal>RestartMode=</literal> section in
+      <citerefentry><refentrytitle>systemd.service</refentrytitle><manvolnum>5</manvolnum></citerefentry>
+      for more details.</para>
     </refsect2>
 
     <refsect2>
@@ -12079,6 +12090,7 @@ $ gdbus introspect --system --dest org.freedesktop.systemd1 \
       <para><function>QueueSignal()</function> was added in version 254.</para>
       <para><varname>SurviveFinalKillSignal</varname> was added in version 255.</para>
       <para><varname>WantsMountsFor</varname> was added in version 256.</para>
+      <para><varname>DebugInvocation</varname> was added in version 257.</para>
     </refsect2>
     <refsect2>
       <title>Service Unit Objects</title>