Enable local testing for all cachier backends with Docker (#287)

* Add a script for running all cores locallly

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* shorten some line

* safe import of optional dependencies in mongodb core tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* safer import of opt deps in mongo tests

* fix test dependencies

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* more stupid docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move test_missing_mongetter and mark it as a mongodb core test

* fixes and readme updates

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* shorter comment

* no need for this

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: shaypal5

.github/workflows/ci-test.yml

@@ -90,6 +90,11 @@ jobs:
           # show running containers
           docker ps -a
 
+      - name: Install MongoDB core test dependencies
+        if: matrix.backend == 'mongodb'
+        run: |
+          python -m pip install -e . -r tests/mongodb_requirements.txt
+
       - name: Unit tests (DB)
         if: matrix.backend == 'mongodb'
         run: pytest -m "mongo" --cov=cachier --cov-report=term --cov-report=xml:cov.xml

CLAUDE.md

@@ -161,8 +161,9 @@ ______________________________________________________________________
   pip install -e .
   pip install -r tests/requirements.txt
   # For specific backends:
-  pip install -r tests/sql_requirements.txt
+  pip install -r tests/mongodb_requirements.txt
   pip install -r tests/redis_requirements.txt
+  pip install -r tests/sql_requirements.txt
   ```
 - **Run all tests:** `pytest`
 - **Run backend-specific tests:** `pytest -m <backend>` (mongo, redis, sql, memory, pickle, maxage)
@@ -177,41 +178,54 @@ ______________________________________________________________________
 - **Run example:** `python examples/redis_example.py`
 - **Update requirements:** Edit `tests/*_requirements.txt` as needed (sql_requirements.txt, redis_requirements.txt).
 
-### MongoDB Local Testing
+### Local Testing with Docker
 
-**Quick Start for MongoDB Testing:**
+**Quick Start - Test Any Backend Locally:**
 
 ```bash
-# Option 1: Using the shell script (recommended)
-./scripts/test-mongo-local.sh                    # MongoDB tests only
-./scripts/test-mongo-local.sh --mode also-local # MongoDB + memory, pickle, maxage tests
-
-# Option 2: Using make
-make test-mongo-local
-
-# Option 3: Using docker-compose
-docker-compose -f scripts/docker-compose.mongodb.yml up -d
-CACHIER_TEST_HOST=localhost CACHIER_TEST_PORT=27017 CACHIER_TEST_VS_DOCKERIZED_MONGO=true pytest -m mongo
-docker-compose -f scripts/docker-compose.mongodb.yml down
+# Test single backend
+./scripts/test-local.sh mongo
+./scripts/test-local.sh redis
+./scripts/test-local.sh sql
+
+# Test multiple backends
+./scripts/test-local.sh mongo redis
+./scripts/test-local.sh external  # All external (mongo, redis, sql)
+./scripts/test-local.sh all       # All backends
+
+# Test with options
+./scripts/test-local.sh mongo redis -v -k  # Verbose, keep containers running
 ```
 
-**Available Options:**
-
-- `./scripts/test-mongo-local.sh` - Run MongoDB tests only (default)
-- `./scripts/test-mongo-local.sh --mode also-local` - Include memory, pickle, and maxage tests
-- `./scripts/test-mongo-local.sh --keep-running` - Keep MongoDB running after tests
-- `./scripts/test-mongo-local.sh --verbose` - Show verbose output
-- `./scripts/test-mongo-local.sh --coverage-html` - Generate HTML coverage report
-
 **Make Targets:**
 
-- `make test-mongo-local` - Run MongoDB tests with Docker
-- `make test-mongo-inmemory` - Run with in-memory MongoDB (default)
-- `make mongo-start` - Start MongoDB container
-- `make mongo-stop` - Stop MongoDB container
-- `make mongo-logs` - View MongoDB logs
-
-**Note:** By default, MongoDB tests use `pymongo_inmemory` which doesn't require Docker. The above commands let you test against a real MongoDB instance matching the CI environment.
+- `make test-local CORES="mongo redis"` - Test specified cores
+- `make test-all-local` - Test all backends with Docker
+- `make test-external` - Test all external backends
+- `make test-mongo-local` - Test MongoDB only
+- `make test-redis-local` - Test Redis only
+- `make test-sql-local` - Test SQL only
+- `make services-start` - Start all Docker containers
+- `make services-stop` - Stop all Docker containers
+
+**Available Cores:**
+
+- `mongo` - MongoDB backend
+- `redis` - Redis backend
+- `sql` - PostgreSQL backend
+- `memory` - Memory backend (no Docker)
+- `pickle` - Pickle backend (no Docker)
+- `all` - All backends
+- `external` - All external backends (mongo, redis, sql)
+- `local` - All local backends (memory, pickle)
+
+**Options:**
+
+- `-v, --verbose` - Verbose pytest output
+- `-k, --keep-running` - Keep containers running after tests
+- `-h, --html-coverage` - Generate HTML coverage report
+
+**Note:** External backends (MongoDB, Redis, SQL) require Docker. Memory and pickle backends work without Docker.
 
 ______________________________________________________________________
 

Makefile

@@ -1,8 +1,11 @@
 # Cachier Makefile
 # Development tasks and shortcuts
 
-.PHONY: help test test-all test-mongo-local test-mongo-docker test-mongo-inmemory \
-        test-mongo-also-local mongo-start mongo-stop mongo-logs lint type-check format clean \
+.PHONY: help test test-all test-local test-all-local test-external \
+        test-mongo-local test-mongo-docker test-mongo-inmemory test-mongo-also-local \
+        test-redis-local test-sql-local \
+        services-start services-stop services-logs \
+        mongo-start mongo-stop mongo-logs lint type-check format clean \
         install install-dev install-all
 
 # Default target
@@ -11,9 +14,16 @@ help:
 	@echo ""
 	@echo "Testing:"
 	@echo "  make test               - Run all tests"
+	@echo "  make test-local CORES='...' - Run tests for specified cores with Docker"
+	@echo "  make test-all-local     - Run all backend tests with Docker"
+	@echo "  make test-external      - Run all external backend tests (mongo, redis, sql)"
 	@echo "  make test-mongo-local   - Run MongoDB tests with Docker"
+	@echo "  make test-redis-local   - Run Redis tests with Docker"
+	@echo "  make test-sql-local     - Run SQL tests with Docker"
 	@echo "  make test-mongo-also-local - Run MongoDB + memory, pickle, maxage tests with Docker"
 	@echo "  make test-mongo-inmemory - Run MongoDB tests with in-memory backend"
+	@echo "  make services-start     - Start all test containers"
+	@echo "  make services-stop      - Stop all test containers"
 	@echo "  make mongo-start        - Start MongoDB container"
 	@echo "  make mongo-stop         - Stop MongoDB container"
 	@echo "  make mongo-logs         - View MongoDB container logs"
@@ -42,8 +52,9 @@ install-dev:
 install-all:
 	pip install -e .[all]
 	pip install -r tests/requirements.txt
-	pip install -r tests/sql_requirements.txt
+	pip install -r tests/mongodb_requirements.txt
 	pip install -r tests/redis_requirements.txt
+	pip install -r tests/sql_requirements.txt
 
 # Testing targets
 test:
@@ -66,6 +77,39 @@ test-mongo-also-local:
 	@echo "Running MongoDB tests with local core tests..."
 	./scripts/test-mongo-local.sh --mode also-local
 
+# New unified testing targets
+test-local:
+	@echo "Running tests for cores: $(CORES)"
+	./scripts/test-local.sh $(CORES)
+
+test-all-local:
+	@echo "Running all backend tests with Docker..."
+	./scripts/test-local.sh all
+
+test-external:
+	@echo "Running all external backend tests..."
+	./scripts/test-local.sh external
+
+test-redis-local:
+	@echo "Running Redis tests with Docker..."
+	./scripts/test-local.sh redis
+
+test-sql-local:
+	@echo "Running SQL tests with Docker..."
+	./scripts/test-local.sh sql
+
+# Service management
+services-start:
+	@echo "Starting all test services..."
+	docker-compose -f scripts/docker-compose.all-cores.yml up -d
+
+services-stop:
+	@echo "Stopping all test services..."
+	docker-compose -f scripts/docker-compose.all-cores.yml down
+
+services-logs:
+	docker-compose -f scripts/docker-compose.all-cores.yml logs -f
+
 # MongoDB container management
 mongo-start:
 	@echo "Starting MongoDB container..."

README.rst

@@ -541,22 +541,25 @@ Running MongoDB tests against a live MongoDB instance
 
 .. code-block:: bash
 
-  ./scripts/test-mongo-local.sh                    # MongoDB tests only (default)
-  ./scripts/test-mongo-local.sh --mode also-local # MongoDB + memory, pickle, maxage tests
+  # Test MongoDB only
+  ./scripts/test-local.sh mongo
+
+  # Test MongoDB with local backends
+  ./scripts/test-local.sh mongo memory pickle
 
 This script automatically handles Docker container lifecycle, environment variables, and cleanup. Additional options:
 
-- ``--mode also-local``: Include memory, pickle, and maxage tests alongside MongoDB tests
-- ``--keep-running``: Keep MongoDB container running after tests
-- ``--verbose``: Show verbose output
-- ``--coverage-html``: Generate HTML coverage report
+- ``-v, --verbose``: Show verbose output
+- ``-k, --keep-running``: Keep containers running after tests
+- ``-h, --html-coverage``: Generate HTML coverage report
 
 **Option 2: Using Make**
 
 .. code-block:: bash
 
-  make test-mongo-local  # Run tests with Docker MongoDB
-  make test-mongo-inmemory  # Run tests with in-memory MongoDB (default)
+  make test-mongo-local     # Run MongoDB tests with Docker
+  make test-all-local       # Run all backends with Docker
+  make test-mongo-inmemory  # Run with in-memory MongoDB (default)
 
 **Option 3: Manual setup**
 
@@ -578,6 +581,28 @@ Contributors are encouraged to test against a real MongoDB instance before submi
 **HOWEVER, the tests run against a live MongoDB instance when you submit a PR are the determining tests for deciding whether your code functions correctly against MongoDB.**
 
 
+Testing all backends locally
+-----------------------------
+
+To test all cachier backends (MongoDB, Redis, SQL, Memory, Pickle) locally with Docker:
+
+.. code-block:: bash
+
+  # Test all backends at once
+  ./scripts/test-local.sh all
+
+  # Test only external backends (MongoDB, Redis, SQL)
+  ./scripts/test-local.sh external
+
+  # Test specific combinations
+  ./scripts/test-local.sh mongo redis
+
+  # Keep containers running for debugging
+  ./scripts/test-local.sh all -k
+
+The unified test script automatically manages Docker containers, installs required dependencies, and runs the appropriate test suites. See ``scripts/README-local-testing.md`` for detailed documentation.
+
+
 Adding documentation
 --------------------
 

pyproject.toml

@@ -75,7 +75,7 @@ line-length = 79
 # --- ruff ---
 
 [tool.ruff]
-target-version = "py38"
+target-version = "py39"
 line-length = 79
 # Exclude a variety of commonly ignored directories.
 exclude = [