Feat/nfstream migration#1
Open
cod3SK wants to merge 53 commits into
Open
Conversation
Fail closed on the iptables-backed response endpoints and stop shipping credentials in source: - Gate /block_ip, /unblock_ip, and POST /killswitch behind ADNS_ADMIN_TOKEN (disabled when unset; constant-time bearer/X-Admin-Token check when set). GET /killswitch stays open since it only reads state. - Drive Postgres credentials and the SQLAlchemy DSN from POSTGRES_USER/ PASSWORD/DB via Compose default-substitution, keeping out-of-the-box runs working without committing real secrets. - Document the new knobs in .env.example and collapse the stale api/.env.example into a pointer to the canonical root file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a fast, dependency-light pytest suite and wire up GitHub Actions: - 26 tests covering endpoints, payload validation, the admin-token gate, simulation, blocked-IP filtering, the heuristic scorer, and the meta feature builder. Tests run against SQLite in heuristic mode (model artifacts pointed off, rDNS/nsenter disabled, no Redis), so they need no external services and finish in seconds. - requirements-test.txt pins the lightweight deps; pytest.ini configures discovery. - .github/workflows/ci.yml runs the API tests plus the frontend lint/build on every push and pull request. - Ignore .venv-test/ and .pytest_cache/; update AGENTS.md testing notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the project's design rationale and current state: - design-decisions/ holds 9 Architecture Decision Records grouped by phase (foundational architecture, hardening, quality/CI), each covering context, decision, and consequences, with an index README. - ml/model_card.md documents both detectors (meta ensemble and the lightweight flow detector) and the heuristic fallback: training data, features, metrics, intended use, and honest limitations (train/serve skew, label-space mismatch, missing meta metrics). - README gains a Design decisions summary linking the ADRs, a Testing section, Security notes, a CI badge, and corrected /simulate examples (the documented botnet_flood/data_exfiltration types never existed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- launcher.py + ADNS.spec + installer.iss: pywebview desktop app bundled with PyInstaller and packaged by Inno Setup; _StripApiPrefix middleware maps /api/* React paths to Flask routes without changing Flask - .github/workflows/build-installer.yml: CI builds ADNS_installer.exe on tag push and uploads it as a release artifact - requirements-desktop.txt: pinned desktop-only deps (excludes psycopg2/alembic) - api/app.py: Windows firewall support via netsh advfirewall; _STREAM_LOCK prevents concurrent streaming threads (OOM fix); static file serving route for ADNS_FRONTEND_DIST; block_ip now records to DB without requiring a token - core/attack_generator.py: stdlib-only CLI replacing dashboard sim buttons - frontend/src/App.jsx: removed attack simulation UI controls - design-decisions/0006: ADR revised to reflect CLI-only simulation approach - tests: updated to match new block/unblock semantics (8 passing) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add Desktop App section with non-technical install steps (download, run installer, launch) and a developer build guide listing the three prerequisite tools - Fix component table: remove stale "attack simulation buttons" reference - Fix Security notes: reflect new block/unblock semantics (DB always updated; OS firewall only fires with a valid token) and add Windows netsh advfirewall alongside iptables Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The button was broken in two ways: the POST endpoint required ADNS_ADMIN_TOKEN which is unset by default (returning 403), and the iptables rules only blocked a single configured interface rather than all network traffic. - Remove admin token requirement from /killswitch POST; it is a first-responder dashboard action that must work without extra config. block_ip/unblock_ip retain their token gate unchanged. - Replace single-interface iptables rules with ! -o lo / ! -i lo rules that drop all non-loopback traffic, keeping the API reachable on localhost so the dashboard can still send the disable command. - Add Windows path via netsh advfirewall (ADNS Killswitch IN/OUT rules). - Drop the now-unused KILL_SWITCH_INTERFACE / ADNS_KILLSWITCH_INTERFACE. - Add design-decisions/architecture.md: full pipeline map with component call graph, batch sizes at every level, per-stage latency table, and env var reference. - Update design-decisions/README.md to surface architecture.md first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ADR 0007: add Amendment section explaining why /killswitch was removed from the token gate and why scope expanded from one interface to all non-loopback. Block/unblock IP gate rationale unchanged. - README Design decisions: correct the ADR 0007 summary line. - README Security notes: rewrite to accurately describe the two-tier model — killswitch is ungated and blocks all non-loopback interfaces; block_ip/unblock_ip remain token-gated and fail closed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test_killswitch_post_disabled_without_token expected HTTP 403 after the endpoint was made token-free, causing CI to fail. - Replace the 403 assertion with a test that the POST succeeds without a token (200, returns enabled state). - Patch ensure_killswitch_rules_enabled to a no-op so the test does not shell out to iptables/netsh. - Use monkeypatch.setitem on KILL_SWITCH_STATE so the enabled=True write does not bleed into the subsequent GET test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The button appeared pressed even when netsh/iptables failed silently (most commonly: API not running as Administrator on Windows, or without NET_ADMIN on Linux). API: - ensure_killswitch_rules_enabled and _ensure_killswitch_windows now return bool — True only if every firewall rule was applied. - /killswitch POST only updates KILL_SWITCH_STATE if the OS action succeeded, and includes os_action: "ok"|"failed" in the response. Frontend: - toggleKillSwitch checks os_action; on "failed" it leaves the button in its previous state and shows a message explaining the privilege requirement (Administrator on Windows, NET_ADMIN on Linux). Tests: - Update mock to return True so the success path works correctly. - Add test_killswitch_post_reports_os_failure covering the failed path: os_action == "failed" and state unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The killswitch cannot affect a Windows host's network when the API runs inside Docker Desktop. sys.platform in the container is "linux" so the netsh path is never taken; iptables rules land in the container's own network namespace. Even with NET_ADMIN/SYS_ADMIN and nsenter available, Docker Desktop routes through a WSL2 VM — escaping the container reaches the VM, not Windows's actual network adapters. The killswitch is effective on a native Linux deployment. Document this constraint in README Security notes and in ADR 0007. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes the external Redis dependency so the stack runs natively on Windows (and anywhere else) without a queue sidecar. - api/task_queue.py: rewritten to submit chunks to a concurrent.futures.ThreadPoolExecutor (ADNS_SCORER_WORKERS=2). Public interface (enqueue_flow_scoring) is unchanged. - api/worker.py: deleted -- RQ bootstrap no longer needed. - api/requirements.txt: dropped redis and rq. - api/app.py: removed the two-level inline-fallback in /ingest; ThreadPoolExecutor path is the only path now. - api/tests/conftest.py: autouse no_background_scoring fixture monkeypatches enqueue_flow_scoring to a no-op so SQLite tests are not raced by background scorer threads. - agent/capture.py: Windows-aware tshark default path. - docker-compose.yml: removed redis and worker services; added ADNS_SCORER_WORKERS / ADNS_SCORING_BATCH_SIZE env vars. - design-decisions/0002-async-scoring-redis-rq.md: Amendment added. - design-decisions/architecture.md: pipeline, latency table, batch-size table, and env-var table updated for thread pool. - README.md: removed all Redis/RQ references throughout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without SQLALCHEMY_DATABASE_URI set, Flask was connecting to PostgreSQL (which isn't running on Windows) while data was written to a SQLite file via a previous session. get_recent_flows() returned [] and /flows fell back to demo data every time. - DEFAULT_DB_URI now resolves to an absolute path at api/instance/adns_demo.db so Flask always reads from the same file regardless of launch directory. - check_same_thread=False added to engine options so ThreadPoolExecutor scorer threads can share the SQLite connection pool. - WAL mode enabled in init_db() so background writes don't block dashboard reads. Launcher (desktop) and production deployments still override via env var. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ADNS.spec: drop rq and redis hidden imports (no longer used after ThreadPoolExecutor refactor). requirements-desktop.txt: same removal, pin pywebview>=5.1. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Excludes SQLite dev DB (api/instance/), PyInstaller and Vite build outputs (dist/, build/, frontend/adns-frontend/dist/), and other generated artifacts. Adds check_db.py for quickly verifying DB table structure and row counts during development. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Embeds tshark capture pipeline directly in Flask: _CaptureAgent spawns tshark as a subprocess, reads stdout in a daemon thread, parses packets using ported capture.py logic, and ingests batches straight into the DB (no separate agent process needed — works in both dev and the bundled exe). New API endpoints: GET /interfaces (tshark -D), GET /agent/status, POST /agent/start, POST /agent/stop. Dashboard sidebar gains a "Capture pipeline" panel with interface dropdown, Start/Stop button, and live health indicators (tshark found, running state, flows captured, last packet time, uptime). ADNS.spec updated to bundle tshark.exe + Wireshark DLLs into the exe under a tshark/ subdirectory; launcher falls back to system tshark if bundle is absent. Npcap must still be installed on the target machine. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A stale Flask process or another app holding port 5000 causes the launcher to connect the webview to the wrong server (no ADNS_FRONTEND_DIST), showing raw JSON instead of the dashboard. Now checks the port before spawning Flask and shows an actionable error dialog instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bundled tshark.exe lacks the registry entries and Npcap paths set up by the Wireshark installer, so tshark -D returned empty output when running from _internal/tshark/. System Wireshark (required anyway for Npcap) always works correctly. Bundled copy is kept as a fallback for machines without a Wireshark system install. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- launcher.py: elevate to admin on startup (ShellExecuteW runas), detect/install Npcap via registry check, bundle npcap-installer.exe if present in repo root - api/app.py: revert _find_tshark() to prefer bundled copy (safe now that admin + Npcap are guaranteed); add _tshark_env() helper that prepends tshark dir to PATH and passes cwd so bundled DLLs resolve - ADNS.spec: conditionally bundle npcap-installer.exe from repo root Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- tests/conftest.py: session-scoped Flask + SQLite in-memory fixtures - tests/test_tshark.py: 10 tests for parsing helpers (_ts_safe_float, _ts_safe_int, _ts_proto, _ts_service, _parse_tshark_line, _build_tshark_cmd, _tshark_env) - tests/test_db.py: 10 tests for Flow/Prediction/BlockedIP models, flow_to_dict, get_recent_flows, is_anomalous_flow, enforce_flow_retention - tests/test_api.py: 10 tests for Flask endpoints (health, flows, ingest, anomalies, killswitch, block_ip, blocked_ips, unblock_ip) - tests/test_scorer.py: 10 tests for FlowScorer heuristic and MetaFeatureBuilder helpers - tests/test_launcher.py: 10 tests for launcher utilities (_port_in_use, _is_admin, _npcap_installed, resource_path, _data_dir, _StripApiPrefix) - frontend/adns-frontend/src/__tests__/app.test.jsx: 13 Vitest tests for formatLabel, severityFromLabel, and App component rendering (mocked axios) - Vitest configured in vite.config.js; run with: npm test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tshark delegates -D (interface enumeration) and packet capture to dumpcap.exe — without it tshark exits with code 13 and no output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use catch without binding + comment to satisfy both no-empty and no-unused-vars in fetchAgentStatus and fetchInterfaces - Remove unused userEvent import from test file Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers interface line parsing regex, /interfaces endpoint (200/503/504/500 paths, empty stdout, field structure, Wi-Fi detection), and agent/status tshark_found field — all subprocess calls mocked so no real binary needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace scrolling single-page dashboard with left nav rail + four tabs (Dashboard, Flows, Flows Manager, Settings) - Fix installer update safety: add AppId GUID, wire version param through iscc /D, add CloseApplications=yes; bump default version to 0.0.1 - Add CHANGELOG.md documenting 0.0.1 changes - Add ADRs 0010 (desktop packaging), 0011 (tabbed layout), 0012 (installer versioning); amend 0001 and 0009 to remove stale content Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add assets/icon.ico and assets/icon.svg - Copy icon.ico as favicon for the web UI - Wire icon.ico into PyInstaller exe and ADNS.spec datas - Enable SetupIconFile and desktop shortcut IconFilename in installer.iss - Remove stale sqlalchemy.pool.manage hidden import from spec Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- launcher: switch from run_simple to make_server so shutdown()/server_close() can be called explicitly on quit, releasing port 5000 immediately instead of waiting for Windows TIME_WAIT - installer: drop IconFilename from desktop shortcut — path was wrong for PyInstaller 5+ (_internal/ layout); exe already has icon embedded Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ortcuts
- PrivilegesRequired=admin so UAC prompts on launch (needed for Npcap install
and firewall rules); install target moved to {autopf} to match admin context
- Desktop/Start Menu shortcuts switched to {commondesktop}/{commonprograms}
(all-users paths, correct for admin installs — avoids Inno Setup warning)
- build_installer.ps1: skip npm steps and reuse existing dist/ if npm is not
on PATH, failing with a clear message only when dist is also absent
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Npcap's installer can refuse fully-silent operation when Windows driver-signing policy requires user confirmation. Removing /S lets Npcap's own wizard handle the install in both the Inno Setup [Run] step and the in-app first-run prompt. Timeout increased to 300 s to give the user time to click through the wizard. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… names - agent/batch_capture.py: ring-buffer tshark capture with two-pass pcap processing (conv stats for real flow metrics + field pass for app-layer), posts to /ingest_batch - api/app.py: Flow.source column, /ingest_batch endpoint, /batch_summary (10m/15m/1h windows with timeseries + top IPs + proto breakdown), enforce_batch_flow_retention (65-min window), _LIVE_FLOW_FILTER to keep batch flows out of live dashboard queries; /interfaces now decodes tshark and PowerShell output as explicit UTF-8 (fixes garbled Cyrillic/non-Latin adapter names on non-English Windows locales) - api/model_runner.py: extended DNS flag fields in MetaFeatureBuilder - frontend: Batch Analysis tab with window selector, area chart, donut, top IP tables; CSS for batch layout components - tests: update subprocess mock to return bytes (matches new decode path) - docs: AGENTS.md and architecture.md updated for batch pipeline, new endpoints, env vars, and Windows interface enrichment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- launcher.py: POST /capture/autostart immediately after Flask is ready; both live and batch capture begin without any user interaction; atexit stops both agents on quit - api/app.py: _auto_detect_interface() uses PowerShell Get-NetRoute to find the default-gateway adapter and map its GUID to a tshark device path; _BatchCaptureAgent class embeds the ring-buffer + two-pass pcap logic in-process (no HTTP round-trip, no separate subprocess); POST /capture/autostart and GET /capture_status endpoints; removed manual POST /agent/start and POST /agent/stop - api/model_runner.py: removed FlowModel (sklearn pipeline) and heuristic FlowScorer from DetectionEngine cascade; MetaEnsembleModel is the only scorer; returns (0.0, "normal") when artifact is absent - frontend: Settings tab replaced interface selector + start/stop buttons with read-only indicators (interface name, tshark found, live capture running/flows/last packet, batch capture running/batches/last batch); removed interfaces/selectedIface/agentBusy state and related callbacks - tests: replaced removed /agent/start test with /capture_status key check - docs: AGENTS.md and architecture.md updated for auto-start, embedded batch agent, single-model detection, and new endpoint list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_auto_detect_interface() was only filtering loopback from the fallback selection, so Microsoft virtual adapters (Wi-Fi Direct, Hyper-V, Mobile Hotspot) named with * could win as first_valid. Added _is_virtual() predicate that skips *-named adapters plus bluetooth/miniport/tunnel entries. Physical adapters are now tracked separately as first_physical and the GUID match is also guarded, so a Get-NetRoute hit on a virtual GUID correctly falls through to the real NIC. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tshark defaults to pcapng format in modern builds, naming ring-buffer files cap_*.pcapng. The glob only looked for cap_*.pcap so no files were ever picked up and zero batch flows appeared. Now collects both extensions before sorting by mtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Version: introduce VERSION file (0.00.001 -> increments each build),
api/_version.py written by build script before PyInstaller runs,
exposed via /capture_status and shown in Settings tab. Installer output
is now ADNS_Installer_v{version}.exe and the wizard header shows the
version number.
Batch capture: replace ring-buffer approach with fixed-duration sequential
captures (tshark -a duration:15 -F pcap). Ring-buffer file naming varied
by tshark version (pcap vs pcapng) and required a glob to discover completed
files. The new approach runs one tshark process per 15s window, waits for
it to finish, processes the single output file, then repeats -- no naming
ambiguity, no missed files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pName Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rors Scoring: launcher.py inserts _MEIPASS/api into sys.path so model_runner loads from there, making Path(__file__).parent resolve to _MEIPASS/api/. Model artifacts are bundled to _MEIPASS/model_artifacts/ (one level up), so the computed DEFAULT_META_MODEL_PATH never existed in the bundle and every flow scored 0.000. Fix: use sys._MEIPASS as BASE_DIR when frozen. Batch: tshark stderr was sent to DEVNULL, making any launch error invisible. Now captured and included in last_error so the Settings tab shows the actual tshark message (wrong interface, permission denied, etc). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AppVersion was required by Inno Setup; replaced with AppVerName which satisfies the requirement without adding a separate "Version:" line to the wizard header -- the version is already in AppName. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If ADNS is killed hard (Task Manager, crash), atexit never fires and Flask keeps port 5000. Instead of showing a fatal error, try to kill the owning process via netstat + taskkill and retry. Only show the error dialog if the port is still occupied after the kill attempt, meaning a genuinely different application owns it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
netstat -ano output parsing was fragile. PowerShell Get-NetTCPConnection gives the owning PID directly. Also wait up to 3s in 0.5s increments for the OS to actually release the socket after the kill. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous _kill_port failed when Get-NetTCPConnection returned multiple
objects (LISTEN + ESTABLISHED on port 5000) -- $c.OwningProcess became
an array and Stop-Process silently did nothing.
New _reclaim_port runs two PowerShell passes:
1. Kill all ADNS.exe except ourselves (handles the tray-leftover case
where closing the window hides it but keeps Flask bound to 5000).
2. Kill whatever owns the port specifically (fallback for non-ADNS).
Then polls up to 5 s for the OS to release the socket before giving up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_start_flask ran in a daemon thread with no error handling — any crash
(import error, init_db failure, bind error) was swallowed and the user
saw a generic "port 5000" message after a 15-second timeout.
Now:
- _flask_start_error captures the full traceback on failure
- _wait_for_api checks thread.is_alive() every poll cycle and returns
immediately when the thread dies (no more 15-30s wait)
- _fatal shows the real traceback so the actual error is visible
- timeout raised to 30s for slow machines
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
collect_all("xgboost") collects Python files but skips xgboost/lib/xgboost.dll
(the native C++ library). At runtime xgboost raises XGBoostLibraryNotFound,
crashing Flask on import before it can bind port 5000.
Dynamically locate the DLL relative to xgboost.__file__ and add it as an
explicit binary so PyInstaller copies it to _internal/xgboost/lib/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tasks.py imports `from sqlalchemy.dialects.postgresql import insert` at
module level. PyInstaller did not include this dialect (only sqlite was
in hiddenimports), so the deferred `from tasks import score_flow_batch`
in task_queue._run_batch raised ImportError, was silently caught, and
every flow was left unscored (0.000).
Two-part fix:
1. Add sqlalchemy.dialects.postgresql{,.base} to hiddenimports so the
module is present in the bundle.
2. Guard the import with try/except in tasks.py so the SQLite fallback
path still works even if the dialect is absent (belt-and-suspenders).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- model_runner: catch both predict_proba and predict failures per estimator and skip with a warning instead of propagating the AttributeError that was silently killing the background scorer - app: add GET /model_status endpoint that probes each estimator with a dummy call and reports ok/broken/degraded/absent - frontend: Model Health section in Settings with per-estimator dot indicators (XGBoost, ExtraTrees) and combined Meta-model status; amber dot-warn CSS class for degraded state - tests: 53-test scoring pipeline suite covering MetaFeatureBuilder edge cases, MetaEnsembleModel loading/scoring/broken-estimator skip, DetectionEngine fallback and reload, _insert_predictions, score_flow_batch end-to-end, enqueue_flow_scoring chunking, API /flows and /anomalous_flows; 28-test React suite covering score formatting and severity logic; fix stale app.test.jsx assertions for removed UI text Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… health UI
- ADNS.spec: switch to collect_all("xgboost") + explicit native DLL; previous
hiddenimports-only approach left the package directory out of the bundle
- launcher.py: replace window.destroy() with os._exit(0) in on_quit so the
process terminates reliably from the pystray thread on Windows EdgeChromium
- scripts/build_installer.ps1: auto-kill running ADNS before PyInstaller step
to prevent PermissionError on locked dist\ADNS\_internal\ files
- api/app.py: rewrite _BATCH_CONV_RE + add _parse_tshark_bytes for tshark 4.x
format (no pipe separators, human-readable byte units like "85 kB")
- AGENTS.md: full rewrite — remove Redis/RQ/worker.py references, update to
ThreadPoolExecutor, document 5-tab layout, batch conv format caveat, Windows
dev commands
- design-decisions/0013-tshark-4x-batch-conv-format.md: new ADR documenting
the format change and regex fix
- design-decisions/README.md: add ADR-0013 to Phase 3 index
- VERSION / api/_version.py: bump to 0.01.000
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- gotham_labels.py: PCAP-level label adapter (directory-encoded ground
truth, no timestamps); maps 5 attack dirs to internal attack_cat names
with per-category attacker IP sets
- build_corpus.py: _apply_labels_gotham + build_corpus_gotham; graceful
pass-B timeout fallback (window_sec=570 -> 600s) for large DDoS PCAPs
(Mirai flood files up to 3.3GB); --dataset {unsw,gotham} CLI flag
- cross_eval.py: three-way evaluation (A in-domain, B cross-domain,
C pooled) with PR-AUC, per-attack-cat recall, benign FPR, and benign
feature-distribution diagnostic; --unsw optional for Gotham-only runs
- __init__.py: export new gotham symbols
Gotham build result: 14,024,188 rows, 32/32 attack PCAPs, 0 dropped
Config A: PR-AUC=1.0000, benign FPR=0.00, overall attack recall=0.9995
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…etrics
Pass-B correctness (no silent zero-fill):
- run_pass_b_chunked(): split large PCAPs via editcap, run pass B on each
500K-packet chunk, merge flag counts; identical to single-pass by construction
- Pass A runs first (600s timeout); pass B normal (90s) then chunked; if
chunked also fails: quarantine all flows as REASON_FLAGS_UNEXTRACTABLE
(never fabricate zeros that masquerade as clean observed-zero flags)
- 3 Mirai DDoS PCAPs (2.1-3.3GB) previously silently zero-filled TCP flags;
all 3 now fully extracted via chunking: 0% all-zero-flag TCP flows in v2
Flood-cap sampling (apply_flood_cap):
- Caps degenerate one-sided flows (src_pkts<=1, dst_pkts==0, label=1) at
DEFAULT_FLOOD_CAP=3000 per source IP with fixed seed=42 for reproducibility
- v1: 11.2M degenerate flows (80% of attack); v2: 24K (0.9%)
- Corpus shrinks from 14M->2.87M rows; attack prevalence 99.6%->98.2%
Honest metrics (cross_eval.py):
- Prevalence (trivial always-attack PR-AUC baseline) now reported per block
- Precision + Recall at fixed threshold=0.5 replace PR-AUC-only reporting
- Per-source-host attacker recall and benign-host FPR (a flood is one event;
"did we catch the host" matters more than scoring each SYN independently)
- _to_xy() now returns src_ips; _evaluate() accepts test_src_ips
Test: test_chunked_pass_b_equals_single_pass (chunk_size=1 forces max stress)
v2 Config A (2.87M rows, flood-capped, real TCP flags):
prevalence=0.982 PR-AUC=1.000 prec@0.5=1.000 recall@0.5=0.998
benign-FPR=0.000 attacker-host-recall=0.9999 benign-host-FPR=0.000
per-cat: scanning=0.888 mirai_infection=0.944 coap_amp=0.921
merlin_dos=0.983 mirai_dos=0.999
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 0 (2026-06-20): Fixed fork-bomb crash (WinError 1455) in frozen NFStream exe by adding multiprocessing.freeze_support() as the first call in __main__. Added run_frozen_guarded.ps1 safety guard (child-process limit + wall-clock timeout). Constrained serving config: n_meters=1, n_dissections=0. STEP 3 (pcap read) and STEP 4 (90s live capture, 135 flows, exactly 1 child for all 18 samples) both passed — Phase 0 verdict: GO. Phase 1: NFStream extractor behind the existing feature contract. - ml/adns_flows/nfstream_config.py: canonical SSoT config (statistical_analysis=True required for 6/21 TCP flag features; idle_timeout=120; active_timeout=1800). - ml/adns_flows/extract_nfstream.py: single-pass extractor emitting identical Flow objects via canonicalize_orientation() + flow_to_row(); validate_matrix() enforces schema on output DataFrame. - ml/adns_flows/tests/test_nfstream_parity.py: 15 parity tests (all pass): hand-fixture byte/flag/orientation checks; n_meters config-parity; determinism; fwd-vs-rev orientation invariance; cross-extractor flag+byte equality vs tshark. Bonus finding: both NFStream and tshark -z conv count Ethernet frame bytes (L2) — they agree exactly, no per-packet offset. - 86/86 adns_flows tests pass. Also adds CLAUDE.md (project context file) and the full ml/adns_flows/ module (schema, tshark extractor, tests) which was untracked from earlier sessions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_corpus.py: NFStream single-pass path for all three datasets (UNSW,
Gotham, CIC); tshark two-pass preserved. New helpers: _reorient_flow,
_apply_labels_nf, _apply_labels_gotham_nf, _apply_labels_cic_nf,
_cmd_probe_attack_nf. CLI: --extractor {nfstream,tshark} (default nfstream),
--n-meters N.
test_labeling_nf.py: 27 unit tests for NFStream labeling helpers (all pass).
run_unsw_day{1,2}_nfstream.py + combine_unsw_nfstream.py: UNSW build scripts
with freeze_support() guard (required on Windows/spawn multiprocessing).
Three-corpus rebuild results (tshark corpora untouched):
- CIC: 308,349 rows, 2.26% attack (+45% rows, grain-driven)
- Gotham: 2,331,227 rows, 97.05% attack (-18.9% rows, flood-cap/Mirai)
- UNSW: 2,058,890 rows, 3.22% attack (+0.1% rows, essentially identical)
All 172 ML suite tests pass. Determinism check: byte-identical. No all-zero rows.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Label-integrity gate PASS: - CIC: FTP/SSH 100% in-window; +45% rows are benign splits only - Gotham: -18.9% explained by NFStream counting fewer UDP flows for Mirai DoS (tshark conv,udp vs NFStream: 925k vs 361k UDP flows); TCP intact; label correct NFStream cross-eval vs tshark-era (qualitative story unchanged): - A in-domain: near-perfect (PR-AUC >= 0.9998, FPR <= 0.05%) - B/C cross-domain: FPR collapse persists (44-47% vs 57-84%), modestly better - D pooled: PR-AUC 1.000, FPR 0.05% - E1 UNSW+Gotham->CIC: FPR 36% (was 51%), still fails -- same root cause - E3 three-way pool: PR-AUC 1.000, FPR 0.05% -- healthy All findings reproduce. Phase 3 next: live scoring path migration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…shark
NFStream is now the only extractor for corpus building, live capture, and
model scoring. tshark is used only for interface enumeration (tshark -D).
Extraction changes:
- Deleted ml/adns_flows/extract.py + assemble.py (tshark two-pass extractor)
- New ml/adns_flows/extract_nfstream.py: single-pass, canonicalize_orientation(),
validate_matrix() gate on every DataFrame
- New api/serving_nfstream.py: flow_to_extra(), NfstreamScorer (validate_matrix
gates every predict call — no _match_shape)
- ADNS.spec: tshark DLLs removed, collect_all("nfstream") + Npcap DLL hook added
- pyi_hooks/rthook_nfstream_npcap.py: runtime hook for frozen NFStream
Serving changes:
- api/model_runner.py: MetaFeatureBuilder/DetectionEngine removed; only
NfstreamDetectionEngine remains
- api/app.py: _NfstreamCaptureAgent with direct NFStream live capture
(idle_timeout=120s, active_timeout=1800s matching corpus grain);
Windows Job Object for forced-shutdown orphan protection; --headless mode
- api/tasks.py: direct nfstream_detector.score_many(), no routing fork
- api/model_artifacts/nfstream_model.joblib: XGBoost E3 pooled (21 features,
PR-AUC 1.000, recall 99.66%, benign FPR 0.05%)
Corpus changes:
- ml/corpus/build_corpus.py: tshark branches and _apply_labels* removed;
build_corpus/build_corpus_gotham/build_corpus_cic all call
extract_flows_nfstream() directly
- NFStream corpora promoted to canonical names (archive/ holds tshark-era files)
- ml/train_nfstream.py: E3 three-corpus training script
Test changes:
- Deleted test_extract.py, test_assemble.py, test_parity.py (tshark tests)
- Deleted api/tests/test_scoring_and_features.py, test_scoring_pipeline.py
(tested removed MetaFeatureBuilder/DetectionEngine)
- New: test_live_equals_training_nfstream.py (8 tests: corpus==serving path),
test_windowing_grain.py (7 tests: direct capture matches corpus grain),
test_labeling_nf.py (27 tests), test_nfstream_parity.py (14 tests)
- 131 tests pass
Frozen exe verified: step4_smoke_test.py 11/11 PASS on dist/ADNS/ADNS.exe
- Startup, model health, live capture (no DLL error), forced-shutdown
(0 orphan meter workers after taskkill /F /T), detection + non-zero scores
Core invariant: ml/adns_flows/ is the only code path producing model inputs.
Train == serve is proven by test_live_equals_training_nfstream.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… rebuild path - NfstreamDetectionEngine now logs at ERROR (not INFO) when model is missing and exposes is_model_loaded / model_error properties - /capture/autostart returns 503 when model absent — silent no-scoring eliminated - 10 new tests in api/tests/test_absent_model.py (unit + integration); suite: 141 - api/model_artifacts/nfstream_model.joblib tracked via Git LFS (.gitattributes) - .gitignore no longer excludes *.joblib (LFS handles it) - CLAUDE.md: model versioning section + exact reproducible rebuild commands - ADNS.spec: comment updated to name the 503 behaviour - gotham_build7.log: build evidence for Phase 6 frozen-exe gate Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.