image captions using blip. by gsaluja9 · Pull Request #204 · aperture-data/workflows

gsaluja9 · 2025-09-15T20:33:29Z

Adds auto generation of image captions using BLIP.
https://huggingface.co/docs/transformers/main/en/model_doc/blip#transformers.BlipForConditionalGeneration

TODO:

~~Add tests~~ : Adding a validation at build time with a basic script.

Add docs

ad-claw000

This workflow looks like a great addition. LGTM!

ad-claw000

Great addition! However, I noticed a few issues that should be addressed before this is merged:

Hardcoded batch_size: In apps/caption-image/app/images.py, self.batch_size = 32 is hardcoded inside FindImageQueryGenerator.__init__. It ignores the batch_size parameter passed from the CLI in caption_images.py. You should pass batch_size into the generator constructor and use it there.
Pagination logic during updates: The query generator fetches batches using batch_id: idx while filtering on wf_caption_image == None. Because the response handler updates these entities and removes the None condition, the total number of matching images changes dynamically. Depending on how ApertureDB evaluates batch_id, this could lead to skipping images (e.g. batch 1 shifts into batch 0's place after batch 0 is updated). A safer approach might be to not rely on batch_id and instead repeatedly request the first N items (e.g., limit), or use a stable identifier for pagination.
Module-level Model Loading: The AutoProcessor and BlipForConditionalGeneration are loaded at the module level in images.py. This means they are loaded into memory as soon as the module is imported, even if just running --help. Consider lazy-loading them inside the class or function, or at least only when the command is actually executed.

Please let me know when these are updated!

- Pass batch_size from CLI down to QueryGenerator - Replace batch_id pagination with limit to handle dynamic properties - Lazy-load AutoProcessor and Blip model to improve startup time

ad-claw000 · 2026-05-24T18:14:24Z

I've pushed a commit to address the review comments:

batch_size is now correctly propagated from the CLI to the query generator.
Switched from batch_id pagination to using limit since wf_caption_image gets updated dynamically.
The BLIP model and processor are now lazy-loaded on demand to speed up script initialization (e.g. when just running --help).

Let me know if this looks good to go!

Copilot

Pull request overview

This PR introduces a new caption-image workflow that auto-generates image captions using the HuggingFace BLIP model, adds container/build plumbing for it, and adds devcontainer configurations to run workflows against a local ApertureDB stack.

Changes:

Added a new apps/caption-image workflow that finds uncaptioned images and writes captions back to ApertureDB.
Integrated the new app into CI builds and added a Docker build-time “warmup/validation” step.
Added devcontainer configs (compose + scripts) to spin up a local ApertureDB + Lenz + WebUI stack for multiple workflows.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
workflows-devcontainer.code-workspace	Workspace config to open workflows repo alongside `../app`.
postinstall.sh	Devcontainer post-create script to configure `adb` and install shell completion.
initcommand.sh	Devcontainer init script (build base image + generate `.env`).
configuration_params.py	Emits `ADB_PORT` based on platform (mac vs others).
base/docker/scripts/sitecustomize.py	Improves global exception hook (docstring + avoids shadowing `type`).
apps/caption-image/requirements.txt	Adds Python dependency on `transformers`.
apps/caption-image/README.md	Documents the new caption-image workflow and usage.
apps/caption-image/Dockerfile	Builds caption-image image; installs torch/torchvision + transformers; runs validation.
apps/caption-image/app/warmup_validate.py	Build-time BLIP warmup + caption assertion.
apps/caption-image/app/images.py	QueryGenerator implementation: fetches images, runs BLIP, updates captions.
apps/caption-image/app/caption_images.py	Typer-based entrypoint for running the caption workflow.
apps/caption-image/app/app.sh	Container entrypoint script that runs `caption_images.py`.
.vscode/launch.json	Debug configuration for Python.
.gitignore	Ignores `aperturedb/` directories (local dev data).
.github/workflows/main.yml	Adds `caption-image` to the CI build matrix.
.devcontainer/dataset-ingestion/docker-compose.yml	Local stack compose file for dataset-ingestion devcontainer.
.devcontainer/dataset-ingestion/devcontainer.json	VS Code devcontainer definition for dataset-ingestion.
.devcontainer/crawl-website/docker-compose.yml	Local stack compose file for crawl-website devcontainer.
.devcontainer/crawl-website/devcontainer.json	VS Code devcontainer definition for crawl-website.
.devcontainer/caption-image/docker-compose.yml	Local stack compose file for caption-image devcontainer.
.devcontainer/caption-image/devcontainer.json	VS Code devcontainer definition for caption-image.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Moved configuration_params.py to .devcontainer - Updated initcommand.sh to loop over all devcontainers - Provided ADB_PORT default in docker-compose.yml files - Updated images.py to correctly use batching, add PyTorch inference context, fix missing DONE state, handle execution query errors - Fixes to warmup_validate.py to avoid external network request - Replaced PIP commands with requirements.txt - Addressed logging and env var issues in caption_images.py

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

- Add threading.Lock to get_model_and_processor lazy init - Remove unused desc_blobs variable - Change PRELOAD_MODEL default to false in Dockerfile - Replace torchvision with pillow in requirements.txt - Fix WORKFLOW_VERSION quoting in initcommand.sh - Add caption-image service to docker-compose.yml

…of batch_id

ad-claw000 · 2026-05-25T02:10:27Z

Replaced the batch_id pagination logic with a stable identifier (fetching all uncaptioned _uniqueids upfront) to prevent skipping images as their _done status is updated. This addresses point 2 from @ad-claw000's review. See commit 7725c94.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 12 comments.

- Use count query + server side batch in FindImageQueryGenerator - Add validation for batch_size > 0 - Switch to RGB and handle decode exceptions - Simplify uniqueids-captions zipping to skip failed - Update log levels to check WF_LOG_LEVEL - Add PRELOAD_MODEL flag to docker-compose.yml - Add proper evaluation and no_grad to warmup_validate - Clean up docs to reflect python None - Add set -euo pipefail to bash scripts

Copilot

Pull request overview

Copilot reviewed 26 out of 27 changed files in this pull request and generated 4 comments.

- Revert default WF_CROISSANT_URL and override in test.sh - Make PRELOAD_MODEL opt-in for caption-image builds - Validate RUN_ONCE and SLEEPING_TIME using wf_argparse.py - Make adb completion installation non-fatal in postinstall.sh

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 2 comments.

…model loading Addresses review comments: - Fetches stable list of uniqueids for pagination to avoid skipping items during parallel processing. - Uses the CLI provided batch_size properly instead of a hardcoded 32. - Lazy loads the processor and model (including 'transformers' and 'torch' module imports) only when executed.

ad-claw000 · 2026-06-08T10:21:14Z

Thanks for the feedback! I've addressed the requested changes in commit 95fd128:

Hardcoded batch_size: Verified that the batch_size is now successfully passed from the Typer CLI arguments to FindImageQueryGenerator.
Pagination logic: Changed the logic to pre-fetch the _uniqueid of all pending images during generator initialization. We then use this static list as a stable identifier for pagination (using _uniqueid: ["in", chunk]), resolving the skipping issues with concurrent modifications.
Module-level Model Loading: Moved the heavy imports (transformers and torch) inside the get_model_and_processor() and response_handler() methods so they are strictly lazy-loaded when inference occurs, drastically improving the CLI loading time.

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 6 comments.

ad-claw000 · 2026-06-10T04:37:00Z

I've restored the stable identifier logic for pagination to address the concern about skipping images during parallel processing (as requested in review 4334057410). See commit f795910.

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 3 comments.

…inating over mutating constraint

ad-claw000 · 2026-06-13T05:57:37Z

Replaced batch_id: idx with batch_id: 0 to repeatedly request the first N items matching the _done constraint. This ensures we don't skip any uncaptioned images when the dataset mutates (e.g. as batches complete in parallel), addressing the concern from the code review. See commit 54f40e5.

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 1 comment.

image captions usign blip.

38084c4

gsaluja9 requested review from bovlb and drewaogle September 15, 2025 22:42

gsaluja9 marked this pull request as ready for review September 17, 2025 13:52

Adding devcontainers. (#208)

2dfb0b8

gsaluja9 requested a review from luisremis September 17, 2025 18:22

stray file.

e3b5994

bovlb reviewed Sep 18, 2025

View reviewed changes

gsaluja9 and others added 2 commits September 19, 2025 09:29

Some review feedback

4a3651f

Merge branch 'main' into image_captions

5516ccd

ad-claw000 approved these changes May 19, 2026

View reviewed changes

ad-claw000 requested changes May 21, 2026

View reviewed changes

luisremis assigned ad-claw000 May 24, 2026

Address review comments for image captions

c1baf46

- Pass batch_size from CLI down to QueryGenerator - Replace batch_id pagination with limit to handle dynamic properties - Lazy-load AutoProcessor and Blip model to improve startup time

Copilot AI review requested due to automatic review settings May 24, 2026 18:14

Copilot started reviewing on behalf of ad-claw000 May 24, 2026 18:14 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ad-claw000 added 2 commits May 24, 2026 20:21

merge: resolve conflicts with main

324e1be

Copilot AI review requested due to automatic review settings May 24, 2026 20:25

Copilot started reviewing on behalf of ad-claw000 May 24, 2026 20:25 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ad-claw000 added 2 commits May 24, 2026 22:12

Fix pagination logic to use a stable identifier (unique IDs) instead …

7725c94

…of batch_id

Copilot AI review requested due to automatic review settings May 25, 2026 02:10

Copilot started reviewing on behalf of ad-claw000 May 25, 2026 02:10 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Copilot AI reviewed Jun 6, 2026

View reviewed changes

Comment thread docker-compose.yml

Comment thread apps/caption-image/test.sh Outdated

Comment thread apps/caption-image/app/app.sh Outdated

Comment thread postinstall.sh Outdated

OpenClaw Bot added 2 commits June 6, 2026 13:49

fix(ingest-croissant): add description to dummy_records in test data

3a64828

Address review comments on PR 204

480998d

- Revert default WF_CROISSANT_URL and override in test.sh - Make PRELOAD_MODEL opt-in for caption-image builds - Validate RUN_ONCE and SLEEPING_TIME using wf_argparse.py - Make adb completion installation non-fatal in postinstall.sh

Copilot AI review requested due to automatic review settings June 7, 2026 04:03

Copilot started reviewing on behalf of ad-claw000 June 7, 2026 04:03 View session

Copilot AI reviewed Jun 7, 2026

View reviewed changes

Comment thread apps/caption-image/test.sh Outdated

Comment thread postinstall.sh Outdated

OpenClaw Bot added 2 commits June 7, 2026 13:47

fix: pass correct options in test.sh and postinstall.sh

e227b0a

Copilot AI review requested due to automatic review settings June 8, 2026 10:21

Copilot started reviewing on behalf of ad-claw000 June 8, 2026 10:21 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py Outdated

OpenClaw Bot added 2 commits June 8, 2026 23:51

refactor: use count query and server-side batch pagination for images

0f0c54e

fix(docker): add lenz healthcheck to fix test flakiness

dd6bf76

Copilot AI review requested due to automatic review settings June 9, 2026 10:54

Copilot started reviewing on behalf of ad-claw000 June 9, 2026 10:54 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

OpenClaw Bot added 2 commits June 9, 2026 15:57

fix: address review comments for relative paths and documentation

75cf2ab

fix(caption-image): restore stable identifiers for robust pagination

f795910

Copilot AI review requested due to automatic review settings June 10, 2026 04:36

Copilot started reviewing on behalf of ad-claw000 June 10, 2026 04:37 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/app/images.py Outdated

Comment thread apps/caption-image/test.sh

OpenClaw Bot added 2 commits June 11, 2026 13:26

fix(caption-image): address latest review comments

23e5832

fix(caption-image): use batch_id 0 to avoid skipping images while pag…

54f40e5

…inating over mutating constraint

Copilot AI review requested due to automatic review settings June 13, 2026 05:55

Copilot started reviewing on behalf of ad-claw000 June 13, 2026 05:55 View session

Copilot AI reviewed Jun 13, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py Outdated

fix(caption-image): use idx for batch_id in pagination

cce666d

Conversation

gsaluja9 commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 left a comment

Choose a reason for hiding this comment

Uh oh!

ad-claw000 left a comment

Choose a reason for hiding this comment

Uh oh!

ad-claw000 commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 commented May 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

ad-claw000 commented Jun 8, 2026

Uh oh!

Copilot AI left a comment

gsaluja9 commented Sep 15, 2025 •

edited

Loading

ad-claw000 commented May 24, 2026 •

edited

Loading