resolved: rework dnssec validation results

This adds a new validation result DNSSEC_UNSUPPORTED_ALGORITHM which is
returned when we encounter an unsupported crypto algorithm when trying
to validate RRSIG/DNSKEY combinations. Previously we'd return ENOTSUPP
in this case, but it's better to consider this a non-error DNSSEC
validation result, since our reaction to this case needs to be more like
in cases such as expired or missing keys: we need to keep continue
validation looking for another RRSIG/DNSKEY combination that works
better for us.

This also reworks how dnssec_validate_rrsig_search() propagates errors
from dnssec_validate_rrsig(). Previously, errors such as unsupported
algorithms or expired signatures would not be propagated, but simply be
returned as "missing-key".

1 files changed, 8 insertions, 2 deletions

diff --git a/src/resolve/resolved-dns-dnssec.h b/src/resolve/resolved-dns-dnssec.h
index f0825ba23f..f33abe3e11 100644
--- a/src/resolve/resolved-dns-dnssec.h
+++ b/src/resolve/resolved-dns-dnssec.h
@@ -43,12 +43,18 @@ enum DnssecMode {
 };
 
 enum DnssecResult {
+        /* These four are returned by dnssec_verify_rrset() */
         DNSSEC_VALIDATED,
         DNSSEC_INVALID,
-        DNSSEC_UNSIGNED,
+        DNSSEC_SIGNATURE_EXPIRED,
+        DNSSEC_UNSUPPORTED_ALGORITHM,
+
+        /* These two are added by dnssec_verify_rrset_search() */
         DNSSEC_NO_SIGNATURE,
         DNSSEC_MISSING_KEY,
-        DNSSEC_SIGNATURE_EXPIRED,
+
+        /* These two are added by the DnsTransaction logic */
+        DNSSEC_UNSIGNED,
         DNSSEC_FAILED_AUXILIARY,
         _DNSSEC_RESULT_MAX,
         _DNSSEC_RESULT_INVALID = -1