test: add unit tests for granite formatters (#812) (#818)

* test: add 199 unit tests for granite formatters (#812)

Unit tests for Granite 3.2 and 3.3 input/output processors, shared
utilities, IntrinsicsResultProcessor canned data regression, and
OpenAI SDK compatibility. No GPU, network, or model downloads required.

- test_granite3_shared.py: find_substring_in_text, create_dict,
  parse_hallucinations_text, hallucination/citation span helpers
- test_granite32_output.py: citation parsing, model output splitting,
  validation, transform pipeline
- test_granite33_output.py: JSON-delimited citations/hallucinations,
  think/response extraction, controls cleanup
- test_granite32_input.py: system message matrix, sanitize, transform
- test_granite33_input.py: available_tools role, per-document roles
- test_intrinsics_canned_output.py: canned model outputs through
  IntrinsicsResultProcessor, Pydantic schema validation of fixtures
- test_openai_compat.py: ChatCompletion round-trip through OpenAI SDK

Closes #812

* test: remove test_empty_substring (tests stdlib not our code)

* fix: nltk_check() now catches LookupError for missing punkt_tab data

nltk_check() only caught ImportError (package not installed) but missed
LookupError (package installed, data not downloaded). Users with nltk
present via transitive deps (rouge_score) got an unhelpful ValueError
instead of install instructions when punkt_tab was missing.

Also adds punkt_tab download to CI workflow — same pattern as Ollama
model pulls.

* fix: declare nltk as explicit dependency

nltk is required by granite citation/hallucination parsing
(nltk.sent_tokenize) but was only present as a transitive dependency
of rouge_score. Pin >=3.9 for punkt_tab support (security fix over
pickle-based punkt).

* fix: code review cleanups for granite formatter tests

- Fix granite_io → mellea in nltk_check error message (copy-paste from upstream)
- Consolidate duplicate nltk_check into optional.py (single source of truth)
- Remove unused imports in test_granite32_output.py
- Fix assertion logic in test_balanced_tags_no_warnings (or → and)

* test: harden granite test robustness

- Fix invalid TCP port 98765 → 127.0.0.1:1 in OpenAI compat test
- Add assertions that testdata directories are non-empty (fail loudly
  instead of silently collecting zero test cases)
- Add unit tests for nltk_check LookupError handling

* fix: address PR #818 review comments

- Replace stale granite_io reference with mellea in optional.py docstring
- Use record-level caplog assertions in granite32 output tests for
  consistency with granite33 tests

* fix: address code review findings

- Fix stale granite_io reference in util.py import_optional docstring
- Tighten test_missing_colon_skipped assertion to verify empty result

Author: planetf1

.github/workflows/quality.yml

@@ -41,6 +41,8 @@ jobs:
       - name: Send failure message pre-commit
         if: failure() # This step will only run if a previous step failed
         run: echo "The quality verification failed. Please run precommit "
+      - name: Download NLTK data
+        run: uv run python -m nltk.downloader punkt_tab
       - name: Install Ollama
         run: curl -fsSL https://ollama.com/install.sh | sh
       - name: Start serving ollama

mellea/formatters/granite/base/optional.py

@@ -27,7 +27,7 @@ def import_optional(extra_name: str):
 
     Args:
         extra_name: Package extra to suggest in the install hint
-            (e.g. ``pip install granite_io[extra_name]``).
+            (e.g. ``pip install mellea[extra_name]``).
     """
     try:
         yield
@@ -49,14 +49,21 @@ def nltk_check(feature_name: str):
         feature_name: Name of the feature that requires NLTK, used in the error message.
 
     Raises:
-        ImportError: If the ``nltk`` package is not installed, re-raised with
-            a descriptive message and installation instructions.
+        ImportError: If the ``nltk`` package is not installed or required
+            NLTK data (e.g. ``punkt_tab``) has not been downloaded,
+            re-raised with a descriptive message and installation
+            instructions.
     """
     try:
         yield
     except ImportError as err:
         raise ImportError(
             f"'nltk' package not installed. This package is required for "
-            f"{feature_name} in the 'granite_io' library."
+            f"{feature_name} in the 'mellea' library."
+            f"{_NLTK_INSTALL_INSTRUCTIONS}"
+        ) from err
+    except LookupError as err:
+        raise ImportError(
+            f"NLTK data required for {feature_name} is not installed."
             f"{_NLTK_INSTALL_INSTRUCTIONS}"
         ) from err

mellea/formatters/granite/base/util.py

@@ -23,22 +23,14 @@
 # First Party
 from .types import ChatCompletionResponse, ChatCompletionResponseChoice
 
-NLTK_INSTALL_INSTRUCTIONS = """
-Please install nltk with:
-    pip install nltk
-In some environments you may also need to manually download model weights with:
-    python -m nltk.downloader punkt_tab
-See https://www.nltk.org/install.html#installing-nltk-data for more detailed
-instructions."""
-
 
 @contextlib.contextmanager
 def import_optional(extra_name: str):
     """Handle optional imports.
 
     Args:
         extra_name: Package extra to suggest in the install hint
-            (e.g. ``pip install granite_io[extra_name]``).
+            (e.g. ``pip install mellea[extra_name]``).
     """
     try:
         yield
@@ -52,27 +44,6 @@ def import_optional(extra_name: str):
         raise
 
 
-@contextlib.contextmanager
-def nltk_check(feature_name: str):
-    """Variation on import_optional for nltk.
-
-    Args:
-        feature_name: Name of the feature that requires NLTK, used in the error message.
-
-    Raises:
-        ImportError: If the ``nltk`` package is not installed, re-raised with
-            a descriptive message and installation instructions.
-    """
-    try:
-        yield
-    except ImportError as err:
-        raise ImportError(
-            f"'nltk' package not installed. This package is required for "
-            f"{feature_name} in the 'granite_io' library."
-            f"{NLTK_INSTALL_INSTRUCTIONS}"
-        ) from err
-
-
 def find_substring_in_text(substring: str, text: str) -> list[dict]:
     """Find all substring matches in text.
 

mellea/formatters/granite/granite3/granite32/output.py

@@ -40,8 +40,9 @@
 
 # First Party
 from ...base.io import OutputProcessor
+from ...base.optional import nltk_check
 from ...base.types import AssistantMessage, ChatCompletion, Document, ToolCall
-from ...base.util import find_substring_in_text, nltk_check, random_uuid
+from ...base.util import find_substring_in_text, random_uuid
 from ...granite3.output import (
     add_citation_context_spans,
     add_hallucination_response_spans,

mellea/formatters/granite/granite3/granite33/output.py

@@ -25,8 +25,9 @@
 
 # First Party
 from ...base.io import OutputProcessor
+from ...base.optional import nltk_check
 from ...base.types import AssistantMessage, ChatCompletion, ToolCall
-from ...base.util import nltk_check, random_uuid
+from ...base.util import random_uuid
 from ...granite3.output import (
     add_citation_context_spans,
     add_hallucination_response_spans,

pyproject.toml

@@ -28,6 +28,7 @@ dependencies = [
     "mistletoe>=1.4.0",
     "pillow", # Needed for Intrinsics (HF and OpenAI Backends).
     "math_verify", # Needed for Majority Voting Sampling Strategies.
+    "nltk>=3.9", # Needed for sentence tokenization in granite citation parsing.
     "rouge_score", # Needed for Majority Voting Sampling Strategies.
     "PyYAML", # Needed for backends/adapters and granite formatters.
     "packaging", # Needed for server type helpers.