Add DMARCbis aggregate/failure report support; rename forensic→failure project-wide

[Copilot]

Summary

• Adds parsing support for DMARCbis aggregate and failure reports per the new IETF drafts (draft-ietf-dmarc-aggregate-reporting-32, draft-ietf-dmarc-failure-reporting-24, draft-ietf-dmarc-dmarcbis-41) while maintaining full RFC 7489 backward compatibility.
• Renames all "forensic" references to "failure" across the entire project per draft-ietf-dmarc-failure-reporting-24, with backward-compatible aliases preserved.

DMARCbis aggregate fields

New fields from the XSD schema added to types, parsing, validation, CSV output, and Elasticsearch/OpenSearch mappings:

• np — non-existent subdomain policy (none/quarantine/reject)
• testing — testing mode flag (n/y), replaces RFC 7489 pct
• discovery_method — policy discovery method (psl/treewalk)
• generator — report generator software identifier (metadata)
• human_result — optional descriptive text on DKIM/SPF auth results

Field values are validated against XSD enumerations. pass is now a valid disposition value per DMARCbis ActionDispositionType. pct and fo default to None when absent (DMARCbis drops these). Namespaced XML without <?xml declaration is handled correctly.

Forensic → failure rename

All references renamed project-wide:

• Core: types.py, __init__.py — ForensicReport→FailureReport, parse_forensic_report→parse_failure_report
• Output modules: elastic.py, opensearch.py, splunk.py, kafkaclient.py, syslog.py, gelf.py, webhook.py, loganalytics.py, s3.py
• CLI: cli.py — args, config keys, index names (dmarc_failure)
• Samples: samples/forensic/ → samples/failure/
• Docs & dashboards: all markdown, Grafana JSON, Kibana NDJSON, Splunk XML

Why

• The IETF DMARCbis working group has introduced new aggregate report fields and renamed "forensic" reports to "failure" reports in draft-ietf-dmarc-failure-reporting-24.
• parsedmarc needs to parse these new fields and align terminology with the upcoming RFCs while maintaining backward compatibility with existing RFC 7489 deployments and data already indexed under the old names.

Testing

• 177 tests total, all passing. Includes DMARCbis draft sample validation (exact sample from aggregate draft appendix), RFC 7489 backward compatibility, explicit DMARCbis field parsing, function alias identity checks, failure report CSV round-trip, output module mocks (webhook, Kafka, Splunk, syslog, LogAnalytics, S3, GELF), CLI config parsing, and edge cases for extract_report.
• ruff check and ruff format --check both pass clean.
• AGENTS.md updated to reflect all changes.

Backward Compatibility / Risk

• Old function/type names preserved as aliases (parse_forensic_report = parse_failure_report, ForensicReport = FailureReport, etc.)
• CLI config accepts both old (save_forensic, forensic_topic) and new keys (save_failure, failure_topic)
• Dashboard queries match both old and new index names:
  • Kibana: dmarc_f* matches both dmarc_forensic* and dmarc_failure*
  • Splunk: (sourcetype="dmarc:failure" OR sourcetype="dmarc:forensic")
  • Elasticsearch/OpenSearch: duplicate-check queries search across both dmarc_failure* and dmarc_forensic* index patterns
• RFC 7489 reports parse with None for DMARCbis-only fields; pct/fo are None when absent (ES/OS omit None fields gracefully)

Related Issue

Checklist

• Tests added or updated if behavior changed
• Docs updated if config or user-facing behavior changed

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[seanthegeek]

@copilot Let's make to dashboard queries backwards compatible so they show data before and after the forensic to failure rename,

[Copilot]

@copilot Let's make to dashboard queries backwards compatible so they show data before and after the forensic to failure rename,

Updated all dashboard queries to be backward compatible in 2ce864f:

• Kibana: Index pattern changed from dmarc_failure* to dmarc_f* — matches both dmarc_forensic* (old data) and dmarc_failure* (new data)
• Splunk: Base search updated to (sourcetype="dmarc:failure" OR sourcetype="dmarc:forensic") so both old and new sourcetypes are queried
• Elasticsearch/OpenSearch: Duplicate-check search queries in save_failure_report_to_elasticsearch / save_failure_report_to_opensearch now search across dmarc_failure*,dmarc_forensic* (both index patterns, with proper prefix handling)

[seanthegeek]

@copilot fix the ruff check

F401 parsedmarc.types.ForensicReport imported but unused; consider removing, adding to __all__, or using a redundant alias
--> parsedmarc/init.py:52:5
|
50 | AggregateReport,
51 | FailureReport,
52 | ForensicReport,
| ^^^^^^^^^^^^^^
53 | ParsedReport,
54 | ParsingResults,
|

[Copilot]

@copilot fix the ruff check

F401 parsedmarc.types.ForensicReport imported but unused; consider removing, adding to __all__, or using a redundant alias
...

Fixed in 970a200 — used a redundant alias (ForensicReport as ForensicReport) so ruff recognizes it as an intentional re-export for backward compatibility.

[seanthegeek]

@copilot Let's make pct and fo None on DMARCbis reports, where they are not used/provided.

[Copilot]

@copilot Let's make pct and fo None on DMARCbis reports, where they are not used/provided.

Done in bb4be6e. Changed pct and fo to default to None instead of "100"/"0" when not present in the XML. DMARCbis reports that don't include these fields will now have None, while RFC 7489 reports that explicitly include <pct> and <fo> continue to parse their values correctly.

[seanthegeek]

@copilot will that cause problems with Elasticsearch/OpenSearch document mappings and saving documents?

[Copilot]

@copilot will that cause problems with Elasticsearch/OpenSearch document mappings and saving documents?

No, it won't cause problems. In the ES/OS _PublishedPolicy mapping, pct is Integer() and fo is Text() — both handle None gracefully. When a field value is None, Elasticsearch/OpenSearch simply omits the field from the stored document rather than raising an error. This is the same behavior already used for the other optional DMARCbis fields (np, testing, discovery_method) which are also None when not provided.

[Copilot]

Pull request overview

This PR adds support for DMARCbis aggregate/failure reports and renames "forensic" to "failure" throughout the codebase, aligning with RFC terminology.

Changes:

• Adds DMARCbis fields: np, testing, discovery_method, generator, human_result
• Renames all "forensic" references to "failure" with backward-compatible aliases
• Updates dashboards, documentation, and sample files

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests.py	Added DMARCbis tests and backward compatibility tests
parsedmarc/types.py	Added new field types and renamed FailureReport with alias
parsedmarc/init.py	Core parsing logic for DMARCbis fields and renaming
parsedmarc/elastic.py, opensearch.py	Index mappings for new fields, renamed functions
parsedmarc/splunk.py, webhook.py, etc.	Renamed methods with backward-compatible aliases
splunk/, kibana/, grafana/	Dashboard updates with backward-compatible queries
samples/aggregate/*.xml	New DMARCbis sample reports
docs/	Documentation terminology updates

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[seanthegeek]

@copilot Can you also add tests to cover all code that is not already covered by tests?

[Copilot]

@copilot Can you also add tests to cover all code that is not already covered by tests?

Added 89 new tests (104 total, all passing) covering core parsing functions (_bucket_interval_by_day, _append_parsed_record, _parse_report_record, SMTP TLS parsing, aggregate XML parsing), utility functions (timestamps, IP geo, reverse DNS, email parsing, filename safety), and output modules with mocks (webhook, Kafka, Splunk, syslog, LogAnalytics). Ruff passes clean.

[seanthegeek]

@copilot revise AGENTS.md to properly reflect the changes in this PR.

[Copilot]

@copilot revise AGENTS.md to properly reflect the changes in this PR.

Updated in 72fb0d4:

• Project overview now mentions failure/forensic (RUF) and DMARCbis draft references
• Key modules section reflects parse_failure_report(), FailureReport, InvalidFailureReport with notes about legacy aliases
• CLI description notes backward-compat config key support (save_forensic → save_failure)
• New "DMARCbis support" subsection documents the new fields (np, testing, discovery_method, generator, human_result), None defaults for pct/fo, and namespace handling
• Report type system updated to "failure" with InvalidForensicReport alias noted

[seanthegeek]

@copilot

• Rename the samples/forensic to samples/failure and update any references to it
• Remove any reference to DMARC 2.0 (That's not what DMARCbis is called. and I will replace references to DMARCbis with the actual RFC name once it it published prior to merging this PR

[Copilot]

@copilot

• Rename the samples/forensic to samples/failure and update any references to it
  ...

Done in aea6c10:

• Renamed samples/forensic/ → samples/failure/ and updated all references in tests.py, .github/workflows/python-tests.yml, AGENTS.md, and docs/source/output.md
• Removed all "DMARC 2.0" text from AGENTS.md (the urn:ietf:params:xml:ns:dmarc-2.0 XML namespace URI in the sample XML is the official namespace from the XSD spec and remains as-is)

All 113 tests pass and ruff is clean.

[seanthegeek]

@copilot Fix ruff formatting errors

[Copilot]

@copilot Fix ruff formatting errors

Fixed in fce8e22:

• Applied ruff format to 6 files (line-wrapping adjustments in __init__.py, cli.py, elastic.py, opensearch.py, webhook.py, tests.py)
• Removed duplicate from glob import glob import in tests.py (ruff F811)
• Fixed test mock return dicts using old forensic_reports key → failure_reports (was causing 2 test failures)

All 146 tests pass, ruff check and ruff format --check both clean.

[seanthegeek]

@copilot Why are the tests failing after my most recent merge. I want to keep the quality of the upstream tests.

[Copilot]

@copilot Why are the tests failing after my most recent merge. I want to keep the quality of the upstream tests.

Fixed in a12fe81. The merge brought in 5 upstream tests that construct mock return dicts with "forensic_reports" key, but cli.py (renamed in this PR) expects "failure_reports". Updated all mock return dicts and @patch targets to use the new names so they match what cli.py actually calls. All 162 tests pass, ruff clean.

[seanthegeek]

@copilot What did I break in the most recent merge?

[Copilot]

@copilot What did I break in the most recent merge?

The merge brought in one new upstream test (testCliPassesMsGraphCertificateAuthSettings from PR #692) that uses "forensic_reports" as the mock return dict key. Since cli.py in this branch expects "failure_reports", the mock didn't match and caused a KeyError. This is the same pattern as the previous merge — any upstream test that constructs a mock return dict for get_dmarc_reports_from_mailbox() needs the key updated from forensic_reports → failure_reports to match the renamed code.

Fixed in a6ae640 — also applied ruff format to 3 files that had formatting drift from the merge.