feat(import) Add pagination, rate limiting, retry, and truncation warnings (#518)

## Summary

- **`--limit 0` means "no limit"**: Common CLI convention so users don't have to guess a large number when importing many repos
- **Retry with exponential backoff on HTTP 429**: Rate-limited requests now automatically retry (up to 3 times) using `Retry-After` header or exponential backoff, instead of failing immediately
- **GitLab rate limit header logging**: Log `ratelimit-remaining`/`ratelimit-limit` headers after each API request (GitHub already had this)
- **Truncation warnings when `--limit` caps results**: Both GitLab and GitHub importers now warn when results are silently truncated, showing "Showing N of M repositories"

## Changes

- `base.py`: Add `max_retries`, `retry_base_delay` to `HTTPClient`; retry loop with `_calculate_retry_delay()` for 429s
- `gitlab.py`: Add `_log_rate_limit()`, `_warn_truncation()`; capture response headers in pagination methods
- `github.py`: Add truncation detection using `total_count` (search) and mid-page limit hit (user/org)
- `cli/import_cmd/_common.py`: Allow `limit=0` → `sys.maxsize` in `ImportOptions`

Author: tony

CHANGES

@@ -33,6 +33,41 @@ $ uvx --from 'vcspull' --prerelease allow vcspull
 _Notes on upcoming releases will be added here_
 <!-- END PLACEHOLDER - ADD NEW CHANGELOG ENTRIES BELOW THIS LINE -->
 
+### Bug fixes
+
+#### `vcspull import`: Fix silent truncation of GitLab/GitHub results (#518)
+
+Previously, `--limit 100` (the default) would silently discard repositories
+beyond the cap with no indication that more were available.
+
+- GitLab: Read `x-total` and `x-next-page` response headers to detect
+  truncation and warn users
+- GitHub search: Use `total_count` from the JSON body to detect truncation
+- GitHub user/org: Detect mid-page limit hit as a "more available" signal
+- All providers now warn when results are capped by `--limit`
+
+#### `vcspull import`: Fix HTTP 429 rate-limit failures on large imports (#518)
+
+Rate-limited API requests previously failed immediately with an unrecoverable
+error. Large imports that triggered rate limits had to be manually restarted.
+
+- Add automatic retry with exponential backoff (up to 3 attempts) on HTTP 429
+- Honor the `Retry-After` response header when present (capped at 120 s)
+- Fall back to exponential backoff with jitter when the header is absent
+
+### New features
+
+#### `vcspull import`: `--limit 0` means "no limit" (#518)
+
+`--limit 0` now fetches every repository instead of raising a validation
+error. This follows the common CLI convention where 0 means unlimited.
+
+#### `vcspull import`: GitLab rate-limit header logging (#518)
+
+GitLab `ratelimit-remaining` / `ratelimit-limit` headers are now logged
+after each API request, matching the existing GitHub rate-limit logging.
+A warning is emitted when fewer than 10 requests remain.
+
 ## vcspull v1.56.0 (2026-02-15)
 
 ### New features

docs/cli/import/gitlab.md

@@ -14,6 +14,24 @@ Import repositories from GitLab or a self-hosted GitLab instance.
     :path: import gitlab
 ```
 
+## Subgroup targeting
+
+Use slash notation to target a specific subgroup or sub-subgroup directly:
+
+```console
+$ vcspull import gl my-group/my-subgroup \
+    --mode org \
+    --workspace ~/code/
+```
+
+```console
+$ vcspull import gl my-group/my-subgroup/my-leaf \
+    --mode org \
+    --workspace ~/code/
+```
+
+The `TARGET` argument accepts any depth of slash-separated group path.
+
 ## Group flattening
 
 When importing a GitLab group with `--mode org`, vcspull preserves subgroup
@@ -27,6 +45,22 @@ $ vcspull import gl my-group \
     --flatten-groups
 ```
 
+### Workspace structure by target and flag
+
+Given a group tree `my-group → sub → leaf`, importing from `~/code/`:
+
+| Target | `--flatten-groups` | Workspace sections written |
+|--------|:-----------------:|---------------------------|
+| `my-group` | no | `~/code/`, `~/code/sub/`, `~/code/sub/leaf/` |
+| `my-group` | yes | `~/code/` only |
+| `my-group/sub` | no | `~/code/`, `~/code/leaf/` |
+| `my-group/sub` | yes | `~/code/` only |
+| `my-group/sub/leaf` | no | `~/code/` only (leaf — no further nesting) |
+| `my-group/sub/leaf` | yes | `~/code/` only |
+
+When the target is already the deepest group (a leaf), `--flatten-groups` has
+no effect — all repositories already land in the base workspace.
+
 ## Authentication
 
 - **Env vars**: `GITLAB_TOKEN` (primary), `GL_TOKEN` (fallback)

src/vcspull/_internal/remotes/base.py

@@ -7,6 +7,9 @@
 import json
 import logging
 import os
+import random
+import sys
+import time
 import typing as t
 import urllib.error
 import urllib.parse
@@ -251,14 +254,21 @@ def __post_init__(self) -> None:
         >>> opts.limit
         10
 
-        >>> ImportOptions(limit=0)
+        >>> import sys
+        >>> opts = ImportOptions(limit=0)
+        >>> opts.limit == sys.maxsize
+        True
+
+        >>> ImportOptions(limit=-1)
         Traceback (most recent call last):
             ...
-        ValueError: limit must be >= 1, got 0
+        ValueError: limit must be >= 0, got -1
         """
-        if self.limit < 1:
-            msg = f"limit must be >= 1, got {self.limit}"
+        if self.limit < 0:
+            msg = f"limit must be >= 0, got {self.limit}"
             raise ValueError(msg)
+        if self.limit == 0:
+            self.limit = sys.maxsize
 
 
 class HTTPClient:
@@ -273,6 +283,8 @@ def __init__(
         auth_prefix: str = "Bearer",
         user_agent: str = "vcspull",
         timeout: int = 30,
+        max_retries: int = 3,
+        retry_base_delay: float = 1.0,
     ) -> None:
         """Initialize the HTTP client.
 
@@ -290,6 +302,10 @@ def __init__(
             User-Agent header value
         timeout : int
             Request timeout in seconds
+        max_retries : int
+            Maximum number of retries on HTTP 429 (rate limit) responses
+        retry_base_delay : float
+            Base delay in seconds for exponential backoff
 
         Examples
         --------
@@ -309,6 +325,8 @@ def __init__(
         self.auth_prefix = auth_prefix
         self.user_agent = user_agent
         self.timeout = timeout
+        self.max_retries = max_retries
+        self.retry_base_delay = retry_base_delay
 
     def _build_headers(self) -> dict[str, str]:
         """Build request headers.
@@ -368,7 +386,7 @@ def get(
         AuthenticationError
             When authentication fails (401)
         RateLimitError
-            When rate limit is exceeded (403/429)
+            When rate limit is exceeded (403/429) after retries exhausted
         NotFoundError
             When resource is not found (404)
         ServiceUnavailableError
@@ -392,24 +410,74 @@ def get(
 
         log.debug("GET %s", url)
 
-        try:
-            with urllib.request.urlopen(request, timeout=self.timeout) as response:
-                body = response.read().decode("utf-8")
-                response_headers = {k.lower(): v for k, v in response.getheaders()}
-                return json.loads(body), response_headers
-        except urllib.error.HTTPError as exc:
-            self._handle_http_error(exc, service_name)
-        except urllib.error.URLError as exc:
-            msg = f"Connection error: {exc.reason}"
-            raise ServiceUnavailableError(msg, service=service_name) from exc
-        except json.JSONDecodeError as exc:
-            msg = f"Invalid JSON response from {service_name}"
-            raise ServiceUnavailableError(msg, service=service_name) from exc
+        for attempt in range(self.max_retries + 1):
+            try:
+                with urllib.request.urlopen(request, timeout=self.timeout) as response:
+                    body = response.read().decode("utf-8")
+                    response_headers = {k.lower(): v for k, v in response.getheaders()}
+                    return json.loads(body), response_headers
+            except urllib.error.HTTPError as exc:  # noqa: PERF203
+                if exc.code == 429 and attempt < self.max_retries:
+                    delay = self._calculate_retry_delay(exc, attempt)
+                    log.warning(
+                        "Rate limited by %s, retrying in %.1fs (attempt %d/%d)",
+                        service_name,
+                        delay,
+                        attempt + 1,
+                        self.max_retries,
+                    )
+                    time.sleep(delay)
+                    continue
+                self._handle_http_error(exc, service_name)
+            except urllib.error.URLError as exc:
+                msg = f"Connection error: {exc.reason}"
+                raise ServiceUnavailableError(msg, service=service_name) from exc
+            except json.JSONDecodeError as exc:
+                msg = f"Invalid JSON response from {service_name}"
+                raise ServiceUnavailableError(msg, service=service_name) from exc
 
-        # Should never reach here, but for type checker
         msg = "Unexpected error"
         raise ServiceUnavailableError(msg, service=service_name)
 
+    def _calculate_retry_delay(
+        self,
+        exc: urllib.error.HTTPError,
+        attempt: int,
+    ) -> float:
+        """Calculate delay before retrying a rate-limited request.
+
+        Uses the ``Retry-After`` header if present (capped at 120s),
+        otherwise falls back to exponential backoff with jitter.
+
+        Parameters
+        ----------
+        exc : urllib.error.HTTPError
+            The 429 HTTP error response
+        attempt : int
+            Zero-based attempt number
+
+        Returns
+        -------
+        float
+            Delay in seconds before the next retry
+        """
+        retry_after = None
+        if exc.headers:
+            retry_after = exc.headers.get("Retry-After")
+
+        if retry_after is not None:
+            try:
+                delay = min(float(retry_after), 120.0)
+                return max(delay, 0.0)
+            except (ValueError, TypeError):
+                pass
+
+        # Exponential backoff: 2^attempt * base_delay, capped at 60s
+        backoff_delay = float(min(2**attempt * self.retry_base_delay, 60.0))
+        # Add jitter: 0 to 50% of the delay
+        jitter = random.uniform(0, 0.5 * backoff_delay)
+        return backoff_delay + jitter
+
     def _handle_http_error(
         self,
         exc: urllib.error.HTTPError,

src/vcspull/_internal/remotes/github.py

@@ -191,6 +191,7 @@ def _fetch_search(self, options: ImportOptions) -> t.Iterator[RemoteRepo]:
         endpoint = "/search/repositories"
         page = 1
         count = 0
+        total_available: int | None = None
 
         while count < options.limit:
             # Always use DEFAULT_PER_PAGE to maintain consistent pagination offset.
@@ -212,12 +213,14 @@ def _fetch_search(self, options: ImportOptions) -> t.Iterator[RemoteRepo]:
             self._log_rate_limit(headers)
 
             total_count = data.get("total_count", 0)
-            if page == 1 and total_count > 1000:
-                log.warning(
-                    "GitHub search returned %d total results but API limits "
-                    "to 1000; consider narrowing your query",
-                    total_count,
-                )
+            if page == 1:
+                total_available = total_count
+                if total_count > 1000:
+                    log.warning(
+                        "GitHub search returned %d total results but API limits "
+                        "to 1000; consider narrowing your query",
+                        total_count,
+                    )
 
             items = data.get("items", [])
             if not items:
@@ -242,6 +245,18 @@ def _fetch_search(self, options: ImportOptions) -> t.Iterator[RemoteRepo]:
 
             page += 1
 
+        # Warn if results were truncated by --limit
+        if (
+            count >= options.limit
+            and total_available is not None
+            and total_available > count
+        ):
+            log.warning(
+                "Showing %d of %d repositories (use --limit 0 to fetch all)",
+                count,
+                total_available,
+            )
+
     def _paginate_repos(
         self,
         endpoint: str,
@@ -263,6 +278,7 @@ def _paginate_repos(
         """
         page = 1
         count = 0
+        more_available = False
 
         while count < options.limit:
             # Always use DEFAULT_PER_PAGE to maintain consistent pagination offset.
@@ -285,21 +301,39 @@ def _paginate_repos(
             if not data:
                 break
 
-            for item in data:
+            for idx, item in enumerate(data):
                 if count >= options.limit:
+                    # Remaining items on this page or a full page = more exist
+                    more_available = (
+                        idx < len(data) - 1 or len(data) == DEFAULT_PER_PAGE
+                    )
                     break
 
                 repo = self._parse_repo(item)
                 if filter_repo(repo, options):
                     yield repo
                     count += 1
 
+            # Boundary: limit reached on the last item of a full page
+            if count >= options.limit and len(data) == DEFAULT_PER_PAGE:
+                more_available = True
+                break
+
             # Check if there are more pages
             if len(data) < DEFAULT_PER_PAGE:
                 break
 
             page += 1
 
+        # Warn if results were truncated by --limit
+        # GitHub user/org endpoints don't return total count
+        if count >= options.limit and more_available:
+            log.warning(
+                "Showing %d repositories; more may be available "
+                "(use --limit 0 to fetch all)",
+                count,
+            )
+
     def _parse_repo(self, data: dict[str, t.Any]) -> RemoteRepo:
         """Parse GitHub API response into RemoteRepo.