Skip to content

feat: add asynchronous schema tracking export with ZIP segments and job status API#140

Merged
devlead merged 2 commits intoWCOMAB:mainfrom
devlead:feature/orderSyncData
Apr 30, 2026
Merged

feat: add asynchronous schema tracking export with ZIP segments and job status API#140
devlead merged 2 commits intoWCOMAB:mainfrom
devlead:feature/orderSyncData

Conversation

@devlead
Copy link
Copy Markdown
Member

@devlead devlead commented Apr 30, 2026

Introduce end-to-end export of change-tracking data for a configured sync table: full rows for inserts/updates and primary keys for deletes, written as streamed JSON-in-ZIP to blob storage with Azure Table coordination and queue-based orchestration.

HTTP (Function auth):

  • POST .../schema/tracking/{tableId}/export with author, referenceId, purpose; persists blobs and exportjobs row, returns 202 with Location to status.
  • GET .../export/status/{*correlationId} for job detail (optional result SAS when result.json exists) or list when correlation segment is empty.

Workers and storage:

  • Queues: exportjob dispatch; exportjob-{updated,inserted,deleted} segments; matching -done completion queues and -error dead-letter style correlation messages on segment failure.
  • Blobs: export container (request.json, job.json, response/*.zip, result.json).
  • Tables: exportjobs partition area_jobId_tableId (sanitized), row key v7 GUID.

Implementation:

  • SchemaTrackingExportService for create, list, status, dispatch, segment processing, optimistic ETag merges, finalize with 7-day read SAS URIs.
  • SchemaTrackingExportStreamingZip streams SqlDataReader into one ZIP entry.
  • SqlStatementExtensions: CHANGETABLE segment SELECTs; Guid.CreateVersion7 for new identifiers where applicable.

Also: README schema tracking export section; Azure.Data.Tables dependency; Directory.Packages.props bumps (Functions worker, OpenTelemetry, SqlClient, Cronos, etc.).

devlead added 2 commits April 30, 2026 16:53
…ob status API

Introduce end-to-end export of change-tracking data for a configured sync
table: full rows for inserts/updates and primary keys for deletes, written
as streamed JSON-in-ZIP to blob storage with Azure Table coordination and
queue-based orchestration.

HTTP (Function auth):
- POST .../schema/tracking/{tableId}/export with author, referenceId, purpose;
  persists blobs and exportjobs row, returns 202 with Location to status.
- GET .../export/status/{*correlationId} for job detail (optional result SAS
  when result.json exists) or list when correlation segment is empty.

Workers and storage:
- Queues: exportjob dispatch; exportjob-{updated,inserted,deleted} segments;
  matching -done completion queues and -error dead-letter style correlation
  messages on segment failure.
- Blobs: export container (request.json, job.json, response/*.zip, result.json).
- Tables: exportjobs partition area_jobId_tableId (sanitized), row key v7 GUID.

Implementation:
- SchemaTrackingExportService for create, list, status, dispatch, segment
  processing, optimistic ETag merges, finalize with 7-day read SAS URIs.
- SchemaTrackingExportStreamingZip streams SqlDataReader into one ZIP entry.
- SqlStatementExtensions: CHANGETABLE segment SELECTs; Guid.CreateVersion7 for
  new identifiers where applicable.

Also: README schema tracking export section; Azure.Data.Tables dependency;
Directory.Packages.props bumps (Functions worker, OpenTelemetry, SqlClient,
Cronos, etc.).
@devlead devlead merged commit cadc930 into WCOMAB:main Apr 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant