feat(config): DJ_STORES env var + DJ_IGNORE_CONFIG_FILE flag + arbitrary .secrets/ attrs — 2.3

[dimitri-yatsenko]

Summary

Closes the env-var gap for object-store configuration. Until now the DataJoint platform (and any deployment that ships container images without datajoint.json) could inject DJ_HOST / DJ_USER / DJ_PASS but had no env-var path for stores.<name>.*. Plugin-registered adapters (Databricks Unity Catalog Volumes, custom HTTP-based protocols, lab archive systems) with credential fields like token, api_key, workspace_url couldn't be used on the platform at all.

This PR adds two env vars and broadens the file-based secrets loader, all targeting DataJoint 2.3:

• DJ_STORES — JSON-encoded copy of the stores dict, same shape as in datajoint.json. Replaces the file's stores block when set. A single env var avoids per-field delimiter / underscore-stripping ambiguity (access_key vs accesskey, raw_data vs rawdata) and accepts arbitrary adapter-defined attr names without negotiating an env-var schema.
• DJ_IGNORE_CONFIG_FILE (default false) — when true, skips datajoint.json, the project .secrets/, and /run/secrets/datajoint/ entirely. Hard guarantee that no file on disk leaks into config.
• .secrets/stores.<name>.<attr> now accepts any attribute, not only access_key / secret_key. (This was the original scope of the PR; kept and expanded.)

Motivation

A plugin that wraps Databricks Unity Catalog Volumes authenticates via a Bearer token — its store spec wants a token field, not access_key / secret_key. With the previous whitelist, .secrets/stores.uc.token was silently ignored, and there was no env-var path for token at all. Users had to either rename to a misleading access_key or set the value programmatically at notebook startup — neither of which works on the DataJoint platform, which has no datajoint.json and no .secrets/ on disk.

After this PR, the same adapter is configured identically across:

Deployment	Source
Local notebook	datajoint.json + .secrets/stores.uc.token
Customer-managed VM	datajoint.json + env vars for credentials
DataJoint Platform / Kubernetes / Lambda	DJ_IGNORE_CONFIG_FILE=true + DJ_STORES='{"uc": {...}}'

What changed

src/datajoint/settings.py

• Added Config.ignore_config_file: bool field (env: DJ_IGNORE_CONFIG_FILE, default False) — pydantic-settings auto-binds it via validation_alias.
• Added a validation_alias placeholder on the stores field so pydantic-settings does not auto-bind DJ_STORES at Config() time. Otherwise its built-in JSON parser intercepts before precedence logic runs and reports SettingsError instead of a clean ValueError.
• Added Config._apply_stores_env() — parses DJ_STORES JSON, replaces self.stores wholesale, raises ValueError on bad JSON or non-object payloads.
• Restructured _create_config(): skip file + secrets when ignore_config_file is set; apply DJ_STORES between file load and secrets fill so env wins over file and secrets only fill gaps.
• Removed the access_key / secret_key whitelist from _load_secrets()'s per-store branch — any stores.<name>.<attr> file is loaded.
• Updated save_template() docstring to mention DJ_STORES / DJ_IGNORE_CONFIG_FILE.

src/datajoint/storage_adapter.py

Incidental cleanup (unchanged from the prior PR scope): removed a try/except TypeError fallback around entry_points(group=...) for Python <3.10. requires-python = ">=3.10", so the branch was dead code (and was producing a stale mypy error).

tests/unit/test_settings.py

New TestStoreEnv class with 8 tests:

1. test_dj_stores_sets_stores_dict — happy-path JSON parse with no config file.
2. test_dj_stores_overrides_config_file — env wins over file.
3. test_dj_stores_invalid_json_raises — clear ValueError on bad JSON.
4. test_dj_stores_non_object_raises — clear ValueError on non-object JSON.
5. test_dj_stores_plus_secrets_dir — secrets fill attrs DJ_STORES omits.
6. test_ignore_config_file_skips_json — DJ_IGNORE_CONFIG_FILE=true blocks datajoint.json.
7. test_ignore_config_file_skips_secrets — same blocks .secrets/.
8. test_ignore_config_file_default_loads_both — default behavior preserved.

Existing TestStoreSecrets tests (incl. the new test_load_store_arbitrary_attr for the whitelist removal) and all 23 test_storage_adapter tests still pass.

Functional precedence

Source	Priority
dj.config["stores"][...] (programmatic)	1 (highest)
DJ_STORES env var	2
stores block of datajoint.json	3
.secrets/stores.<name>.<attr> files	4 (fills missing attrs only)

When DJ_IGNORE_CONFIG_FILE=true, sources 3 and 4 are skipped entirely.

Companion PR

• Documentation: datajoint/datajoint-docs#172 (draft) — release notes for 2.3, updated env-var reference, new Storage Adapter API spec.

Test plan

• 80 tests pass in tests/unit/test_settings.py (including the 8 new TestStoreEnv tests)
• 23 tests pass in tests/unit/test_storage_adapter.py
• Pre-commit hooks green (ruff, ruff-format, mypy, codespell)
• Manual: DJ_IGNORE_CONFIG_FILE=true DJ_STORES='{...}' python -c "import datajoint as dj; print(dj.config['stores'])" from an empty directory shows the env-supplied stores and emits no "No datajoint.json found" warning.
• Manual: with DJ_IGNORE_CONFIG_FILE=true and a datajoint.json adjacent to cwd containing {"database": {"host": "should-not-load"}}, dj.config["database.host"] is the default (localhost), not should-not-load.

Release notes (2.3)

• DJ_STORES env var — JSON-encoded stores dict
• DJ_IGNORE_CONFIG_FILE env var — skip file-based config sources (default false)
• .secrets/stores.<name>.<attr> accepts any attribute, not only access_key / secret_key
• Storage Adapter API (datajoint.storage entry-point group) formalized as part of the public API; see the spec in the companion docs PR.