feat(deploy): allow excluding files from deploys via .astro/deployignore#2168
Open
tayloramurphy wants to merge 3 commits into
Open
Conversation
Adds a first-class ignore mechanism for deploys. Patterns in .astro/deployignore (.dockerignore syntax, anchored at the project root) are excluded from all deploy artifacts: - DAG bundles (astro deploy / astro deploy --dags): a skip predicate is threaded through UploadBundle and fileutil.Tar; bundle paths are re-anchored at the project root so patterns like dags/docs/ work the same across deploy types - dbt bundles (astro dbt deploy): patterns are read from the dbt project's own .astro/deployignore - image builds (astro deploy): patterns are temporarily appended to .dockerignore during the build and restored byte-for-byte, reusing the buildImageWithoutDags machinery (now buildImageWithIgnorePatterns) - client/remote image deploys (astro remote deploy): patterns are merged into the build-context copy filter alongside .dockerignore The file lives in the committable .astro directory so the ignore list can be version controlled independently of .gitignore.
- buildImageWithIgnorePatterns: append all patterns without deduplication. Dedup by exact line broke .dockerignore last-match-wins ordering in two ways: a restated pattern (normalized 'docs' vs raw 'docs/') was appended after an existing '!' negation and flipped it, and a deduped negation lost its re-include. Plain concatenation matches how dockerignoreSkipFunc orders the combined pattern list. Also merged the single-caller withDockerignoreAdditions wrapper into it and skip touching the file when there is nothing to append. - deployIgnoreSkipFunc: make root and bundle paths absolute before filepath.Rel, so a relative --dags-path no longer silently disables project-root anchoring; fix the escape check to not misclassify directories whose names start with '..'; warn when the bundle lies outside the project root and patterns fall back to bundle-relative matching. - deployDags: never let user patterns exclude the CLI-injected astronomer_monitoring_dag.py from hybrid deploys. - fileutil.Tar: consult the skip predicate before symlink/header work so excluded irregular files (e.g. sockets in an ignored dbt target/) cannot fail the deploy; return the os.Stat error instead of swallowing it (previously a missing source dir uploaded an empty bundle, wiping deployed DAGs); replace the excludePathPrefixes parameter with the skip predicate — it matched post-prepend tar paths, so the '.git/' exclusion silently never fired for DAG bundles (prependBaseDir=true). - UploadBundle: exclude .git via the source-relative skip predicate, which now also covers DAG bundles. - Extract readIgnorePatternsFile shared by .dockerignore and deployignore parsing; unexport DeployIgnoreFilePath; attribute ignore-file errors to the right file in the client build-context path.
Coverage Report for CI Build 27626050549Coverage increased (+0.04%) to 45.13%Details
Uncovered Changes
Coverage Regressions1 previously-covered line in 1 file lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Users often have files that belong in version control but not in deploy artifacts - internal docs, local
.venvdirectories, scratch files. Today there is no way to keep them out of the DAG bundle:.dockerignoreonly affects image builds, and.gitignoreis the wrong tool because the files should stay in git. The common workaround is custom CI/CD logic that copies the project and prunes it before deploying.This adds an optional
.astro/deployignorefile at the project root. It uses.dockerignoresyntax (globs,**,!negation, comments) and applies to every deploy artifact:astro deploy,astro deploy --dagsastro dbt deploy(read from the dbt project's own.astro/deployignore)astro deploy(patterns are appended to.dockerignorefor the duration of the build; the file is restored byte-for-byte afterward)astro remote deploy(merged into the build-context filter alongside.dockerignore)Example
Behavior notes
dags/internal_docs/means the same thing in the image and in the DAG bundle. If the bundled directory is outside the project (e.g.--dags-path ../shared), patterns match relative to the bundle root and the CLI prints a warning..dockerignore, so their!negations take precedence..git/is now excluded from all bundles (previously the exclusion silently did not apply to DAG bundles).fileutil.Tarto silently upload an empty bundle — deleting all deployed DAGs — instead of returning an error.Testing
Unit tests cover pattern parsing, project-root anchoring (including relative
--dags-pathand outside-root fallback),!negation, invalid patterns,.dockerignoreappend/restore (LF/CRLF, created-then-removed), and an end-to-end bundle deploy asserting the uploaded tarball contents.