Add an SQL core (#267)

shaypal5 · pre-commit-ci[bot] · Copilot · web-flow · commit 410543c09be2 · 2025-07-03T22:29:12.000+03:00
* Add an SQL core * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * black and flake8 * tests for sql core * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * only test sql backend on ubuntu currently * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more linting nightmare * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more error fixin * install sql deps only for sql backend tests * no import warning on sql core imports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more sql core tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes and linting * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test fixes and more copilot instructions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix pytest arg to run only local tests * fix var name * a couple more sql core tests * addressing copilot comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove evil lines from pyproject.toml * fix test_sqlcore_set_entry_fallback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * readme note fix * pylint and flake8 be gone * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update README.rst Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
@@ -1,12 +1,59 @@
-# GitHub Copilot Custom Instructions for Cachier
+# Copilot Instructions for Cachier
 
-- Cachier is a Python package providing persistent, stale-free memoization decorators for Python functions, supporting local (pickle), cross-machine (MongoDB), and in-memory caching backends.
-- Always refer to the main decorator as `@cachier`, and note that it can be configured via parameters such as `stale_after`, `backend`, `mongetter`, `cache_dir`, `pickle_reload`, `separate_files`, `wait_for_calc_timeout`, and `allow_none`.
-- Arguments to cached functions must be hashable; custom hash functions can be provided via the `hash_func` parameter for unhashable arguments.
+Welcome to the Cachier codebase! Please follow these guidelines to ensure code suggestions, reviews, and contributions are robust, maintainable, and compatible with our multi-backend architecture.
+
+## 1. Decorator and API Usage
+
+- The main decorator is `@cachier`. It supports parameters such as `stale_after`, `backend`, `mongetter`, `cache_dir`, `pickle_reload`, `separate_files`, `wait_for_calc_timeout`, `allow_none`, and `hash_func`.
+- Arguments to cached functions must be hashable; for unhashable arguments, provide a custom hash function via the `hash_func` parameter.
 - The default backend is pickle-based, storing cache files in `~/.cachier/` unless otherwise specified. MongoDB and memory backends are also supported.
 - Cachier is thread-safe and supports per-function cache clearing via the `clear_cache()` method on decorated functions.
-- Global configuration is possible via `set_default_params` and `enable_caching`/`disable_caching` functions.
-- When reviewing code, ensure new features or bugfixes maintain compatibility with Python 3.9+, preserve thread safety, and follow the numpy docstring conventions for documentation.
-- Tests are located in the `tests/` directory and should be run with `pytest`. MongoDB-related tests require either a mocked or live MongoDB instance.
-- When discussing or generating code, prefer concise, readable, and well-documented Python code, and follow the established conventions in the codebase and README.
+- Global configuration is possible via `set_default_params`, `set_global_params`, and `enable_caching`/`disable_caching`.
+
+## 2. Optional Dependencies and Backends
+
+- Cachier supports multiple backends: `pickle`, `memory`, `mongo`, and `sql`.
+- Not all dependencies are required for all backends. Code and tests for optional backends (e.g., MongoDB, SQL/SQLAlchemy) **must gracefully handle missing dependencies** and should not break import or test collection for other backends.
+- Only raise errors or warnings for missing dependencies when the relevant backend is actually used (not at import time).
+
+## 3. Testing Matrix and Markers
+
+- Tests are located in the `tests/` directory and should be run with `pytest`.
+- Tests are marked with `@pytest.mark.<backend>` (e.g., `@pytest.mark.sql`, `@pytest.mark.mongo`, `@pytest.mark.local`).
+- The CI matrix runs different backends on different OSes. Do **not** assume all tests run on all platforms.
+- MongoDB-related tests require either a mocked or live MongoDB instance.
+- When adding new backends that require external services (e.g., databases), update the CI matrix and use Dockerized services as in the current MongoDB and PostgreSQL setup. Exclude backends from OSes where they are not supported.
+
+## 4. Coverage, Linting, and Typing
+
+- Code must pass `mypy`, `ruff`, and `pytest`.
+- Use per-file or per-line ignores for known, justified issues (e.g., SQLAlchemy model base class typing, intentional use of `pickle`).
+- All new code must include full type annotations and docstrings matching the style of the existing codebase.
+- All docstrings should follow numpy docstring conventions.
+
+## 5. Error Handling and Warnings
+
+- Do **not** emit warnings at import time for missing optional dependencies. Only raise errors or warnings when the relevant backend is actually used.
+
+## 6. Backward Compatibility
+
+- Maintain backward compatibility for public APIs unless a breaking change is explicitly approved.
+- Cachier supports Python 3.9+.
+
+## 7. Documentation and Examples
+
+- When adding a new backend or feature, provide:
+  - Example usage in the README
+  - At least one test for each public method
+  - Documentation of any new configuration options
 - For documentation, follow numpy docstring conventions and validate changes to `README.rst` with `python setup.py checkdocs`.
+
+## 8. General Style
+
+- Prefer concise, readable, and well-documented Python code.
+- Follow the existing code style and conventions for imports, docstrings, and type annotations.
+- Prefer explicit, readable code over cleverness.
+
+______________________________________________________________________
+
+Thank you for contributing to Cachier! These guidelines help ensure a robust, maintainable, and user-friendly package for everyone.
diff --git a/.github/workflows/ci-test.yml b/.github/workflows/ci-test.yml
@@ -22,11 +22,13 @@ jobs:
       matrix:
         python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
         os: ["ubuntu-latest", "macOS-latest", "windows-latest"]
-        backend: ["local", "db"]
+        backend: ["local", "mongodb", "postgres"]
         exclude:
           # ToDo: take if back when the connection become stable
           #  or resolve using `InMemoryMongoClient`
-          - { os: "macOS-latest", backend: "db" }
+          - { os: "macOS-latest", backend: "mongodb" }
+          - { os: "macOS-latest", backend: "postgres" }
+          - { os: "windows-latest", backend: "postgres" }
     env:
       CACHIER_TEST_HOST: "localhost"
       CACHIER_TEST_PORT: "27017"
@@ -50,37 +52,63 @@ jobs:
 
       - name: Unit tests (local)
         if: matrix.backend == 'local'
-        run: pytest -m "not mongo" --cov=cachier --cov-report=term --cov-report=xml:cov.xml
+        run: pytest -m "not mongo and not sql" --cov=cachier --cov-report=term --cov-report=xml:cov.xml
 
       - name: Setup docker (missing on MacOS)
-        if: runner.os == 'macOS' && matrix.backend == 'db'
+        if: runner.os == 'macOS' && matrix.backend == 'mongodb'
         run: |
           brew install docker
           colima start
           # For testcontainers to find the Colima socket
           sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock
       # ToDo: find a way to cache docker images
       #- name: Cache Container Images
-      #  if: matrix.backend == 'db'
+      #  if: matrix.backend == 'mongodb'
       #  uses: borda/cache-container-images-action@b32a5e804cb39af3c3d134fc03ab76eac0bfcfa9
       #  with:
       #    prefix-key: "mongo-db"
       #    images: mongo:latest
+
       - name: Start MongoDB in docker
-        if: matrix.backend == 'db'
+        if: matrix.backend == 'mongodb'
         run: |
           # start MongoDB in a container
           docker run -d -p ${{ env.CACHIER_TEST_PORT }}:27017 --name mongodb mongo:latest
           # wait for MongoDB to start, which is in average 5 seconds
           sleep 5
           # show running containers
           docker ps -a
+
       - name: Unit tests (DB)
-        if: matrix.backend == 'db'
+        if: matrix.backend == 'mongodb'
         run: pytest -m "mongo" --cov=cachier --cov-report=term --cov-report=xml:cov.xml
       - name: Speed eval
         run: python tests/speed_eval.py
 
+      - name: Start PostgreSQL in docker
+        if: matrix.backend == 'postgres'
+        run: |
+          docker run -d \
+            -e POSTGRES_USER=testuser \
+            -e POSTGRES_PASSWORD=testpass \
+            -e POSTGRES_DB=testdb \
+            -p 5432:5432 \
+            --name postgres postgres:15
+          # wait for PostgreSQL to start
+          sleep 10
+          docker ps -a
+
+      - name: Install SQL core test dependencies (SQL/Postgres)
+        if: matrix.backend == 'postgres'
+        run: |
+          python -m pip install -e . -r tests/sql_requirements.txt
+
+      - name: Unit tests (SQL/Postgres)
+        if: matrix.backend == 'postgres'
+        env:
+          SQLALCHEMY_DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb
+        run: pytest -m sql --cov=cachier --cov-report=term --cov-report=xml:cov.xml
+
       - name: Upload coverage to Codecov
         continue-on-error: true
         uses: codecov/codecov-action@v5
diff --git a/README.rst b/README.rst
@@ -342,6 +342,64 @@ You can set an in-memory cache by assigning the ``backend`` parameter with ``'me
 
 Note, however, that ``cachier``'s in-memory core is simple, and has no monitoring or cap on cache size, and can thus lead to memory errors on large return values - it is mainly intended to be used with future multi-core functionality. As a rule, Python's built-in ``lru_cache`` is a much better stand-alone solution.
 
+SQLAlchemy (SQL) Core
+---------------------
+
+**Note:** The SQL core requires SQLAlchemy to be installed. It is not installed by default with cachier. To use the SQL backend, run::
+
+    pip install SQLAlchemy
+
+Cachier supports a generic SQL backend via SQLAlchemy, allowing you to use SQLite, PostgreSQL, MySQL, and other databases.
+
+**Usage Example (SQLite in-memory):**
+
+.. code-block:: python
+
+    from cachier import cachier
+
+    @cachier(backend="sql", sql_engine="sqlite:///:memory:")
+    def my_func(x):
+        return x * 2
+
+**Usage Example (PostgreSQL):**
+
+.. code-block:: python
+
+    @cachier(backend="sql", sql_engine="postgresql://user:pass@localhost/dbname")
+    def my_func(x):
+        return x * 2
+
+**Usage Example (MySQL):**
+
+.. code-block:: python
+
+    @cachier(backend="sql", sql_engine="mysql+pymysql://user:pass@localhost/dbname")
+    def my_func(x):
+        return x * 2
+
+**Configuration Options:**
+
+- ``sql_engine``: SQLAlchemy connection string, Engine, or callable returning an Engine.
+- All other standard cachier options are supported.
+
+**Table Schema:**
+
+- ``function_id``: Unique identifier for the cached function
+- ``key``: Cache key
+- ``value``: Pickled result
+- ``timestamp``: Datetime of cache entry
+- ``stale``: Boolean, is value stale
+- ``processing``: Boolean, is value being calculated
+- ``completed``: Boolean, is value calculation completed
+
+**Limitations & Notes:**
+
+- Requires SQLAlchemy (install with ``pip install SQLAlchemy``)
+- For production, use a persistent database (not ``:memory:``)
+- Thread/process safety is handled via transactions and row-level locks
+- Value serialization uses ``pickle``. **Warning:** `pickle` can execute arbitrary code during deserialization if the cache database is compromised. Ensure the cache is stored securely and consider using safer serialization methods like `json` if security is a concern.
+- For best performance, ensure your DB supports row-level locking
+
 
 Contributing
 ============
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,3 +1,5 @@
+# === Metadata & Build System ===
+
 [build-system]
 requires = [
   "setuptools",
@@ -46,6 +48,8 @@ dependencies = [
   "watchdog>=2.3.1",
 ]
 urls.Source = "https://github.com/python-cachier/cachier"
+# --- setuptools ---
+
 scripts.cachier = "cachier.__main__:cli"
 
 [tool.setuptools]
@@ -63,6 +67,13 @@ include = [
 ] # package names should match these glob patterns (["*"] by default)
 namespaces = false # to disable scanning PEP 420 namespaces (true by default)
 
+# === Linting & Formatting ===
+
+[tool.black]
+line-length = 79
+
+# --- ruff ---
+
 [tool.ruff]
 target-version = "py38"
 line-length = 79
@@ -97,13 +108,16 @@ lint.extend-select = [
 lint.ignore = [
   "C901",
   "E203",
+  "F824",
+  "W503",
 ]
 lint.per-file-ignores."src/**/__init__.py" = [
   "D104",
 ]
 lint.per-file-ignores."src/cachier/config.py" = [
   "D100",
 ]
+lint.per-file-ignores."src/cachier/cores/sql.py" = [ "S301" ]
 lint.per-file-ignores."tests/**" = [
   "D100",
   "D101",
@@ -119,6 +133,7 @@ lint.unfixable = [
   "F401",
 ]
 
+# --- flake8 ---
 #[tool.ruff.pydocstyle]
 ## Use Google-style docstrings.
 #convention = "google"
@@ -134,6 +149,10 @@ wrap-summaries = 79
 wrap-descriptions = 79
 blank = true
 
+# === Testing ===
+
+# --- pytest ---
+
 [tool.pytest.ini_options]
 testpaths = [
   "cachier",
@@ -154,8 +173,11 @@ markers = [
   "mongo: test the MongoDB core",
   "memory: test the memory core",
   "pickle: test the pickle core",
+  "sql: test the SQL core",
 ]
 
+# --- coverage ---
+
 [tool.coverage.run]
 branch = true
 # dynamic_context = "test_function"
diff --git a/src/cachier/__init__.py b/src/cachier/__init__.py
@@ -1,4 +1,4 @@
-from ._version import *  # noqa: F403
+from ._version import __version__
 from .config import (
     disable_caching,
     enable_caching,
@@ -17,4 +17,5 @@
     "get_global_params",
     "enable_caching",
     "disable_caching",
+    "__version__",
 ]
diff --git a/src/cachier/_version.py b/src/cachier/_version.py
@@ -18,7 +18,8 @@
 def _get_git_sha() -> str:
     from subprocess import DEVNULL, check_output
 
-    out = check_output(["git", "rev-parse", "--short", "HEAD"], stderr=DEVNULL)  # noqa: S603, S607
+    args = ["git", "rev-parse", "--short", "HEAD"]
+    out = check_output(args, stderr=DEVNULL)  # noqa: S603
     return out.decode("utf-8").strip()
 
 
diff --git a/src/cachier/core.py b/src/cachier/core.py
@@ -14,7 +14,7 @@
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timedelta
 from functools import wraps
-from typing import Any, Optional, Union
+from typing import Any, Callable, Optional, Union
 from warnings import warn
 
 from .config import (
@@ -27,6 +27,7 @@
 from .cores.memory import _MemoryCore
 from .cores.mongo import _MongoCore
 from .cores.pickle import _PickleCore
+from .cores.sql import _SQLCore
 
 MAX_WORKERS_ENVAR_NAME = "CACHIER_MAX_WORKERS"
 DEFAULT_MAX_WORKERS = 8
@@ -107,6 +108,7 @@ def cachier(
     hash_params: Optional[HashFunc] = None,
     backend: Optional[Backend] = None,
     mongetter: Optional[Mongetter] = None,
+    sql_engine: Optional[Union[str, Any, Callable[[], Any]]] = None,
     stale_after: Optional[timedelta] = None,
     next_time: Optional[bool] = None,
     cache_dir: Optional[Union[str, os.PathLike]] = None,
@@ -134,13 +136,16 @@ def cachier(
     hash_params : callable, optional
     backend : str, optional
         The name of the backend to use. Valid options currently include
-        'pickle', 'mongo' and 'memory'. If not provided, defaults to
+        'pickle', 'mongo', 'memory', and 'sql'. If not provided, defaults to
         'pickle' unless the 'mongetter' argument is passed, in which
         case the mongo backend is automatically selected.
     mongetter : callable, optional
         A callable that takes no arguments and returns a pymongo.Collection
         object with writing permissions. If unset a local pickle cache is used
         instead.
+    sql_engine : str, Engine, or callable, optional
+        SQLAlchemy connection string, Engine, or callable returning an Engine.
+        Used for the SQL backend.
     stale_after : datetime.timedelta, optional
         The time delta after which a cached result is considered stale. Calls
         made after the result goes stale will trigger a recalculation of the
@@ -208,6 +213,12 @@ def cachier(
         core = _MemoryCore(
             hash_func=hash_func, wait_for_calc_timeout=wait_for_calc_timeout
         )
+    elif backend == "sql":
+        core = _SQLCore(
+            hash_func=hash_func,
+            sql_engine=sql_engine,
+            wait_for_calc_timeout=wait_for_calc_timeout,
+        )
     else:
         raise ValueError("specified an invalid core: %s" % backend)
 
diff --git a/src/cachier/cores/base.py b/src/cachier/cores/base.py
@@ -1,4 +1,5 @@
 """Defines the interface of a cachier caching core."""
+
 # This file is part of Cachier.
 # https://github.com/python-cachier/cachier
 
diff --git a/src/cachier/cores/mongo.py b/src/cachier/cores/mongo.py
@@ -75,7 +75,9 @@ def get_entry_by_key(self, key: str) -> Tuple[str, Optional[CacheEntry]]:
         )
         if not res:
             return key, None
-        val = pickle.loads(res["value"]) if "value" in res else None  # noqa: S301
+        val = None
+        if "value" in res:
+            val = pickle.loads(res["value"])  # noqa: S301
         entry = CacheEntry(
             value=val,
             time=res.get("time", None),
diff --git a/src/cachier/cores/sql.py b/src/cachier/cores/sql.py
diff --git a/tests/sql_requirements.txt b/tests/sql_requirements.txt
diff --git a/tests/test_general.py b/tests/test_general.py
diff --git a/tests/test_sql_core.py b/tests/test_sql_core.py

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,5 @@`
`1`	`1`	`"""Defines the interface of a cachier caching core."""`
	`2`	`+`
`2`	`3`	`# This file is part of Cachier.`
`3`	`4`	`# https://github.com/python-cachier/cachier`
`4`	`5`