Skip to content

Commit 76a5796

Browse files
sarsharmaSarath chandra Bussakumaraksh1CodingIsBliss
authored
ACR SDK dist: Minor fix and add more tests (#2899)
* feat: Oryx SDK regional distribution — pull SDKs from ACR * feat: Add ACR-based SDK distribution with feature flag Add two separate ACR SDK providers behind ORYX_ENABLE_ACR_SDK_PROVIDER: 1. ExternalAcrSdkProvider - communicates with LWASv2 via Unix socket to pull SDK images from WAWS Images ACR 2. AcrSdkProvider (direct) - downloads SDKs from Oryx ACR (oryxsdks.azurecr.io) using OCI Distribution API New files: - IExternalAcrSdkProvider.cs / ExternalAcrSdkProvider.cs - OciRegistryClient.cs - HTTP client for OCI Distribution API - AcrVersionProviderBase.cs - base class for ACR version discovery - Per-platform ACR version providers (Node, Python, PHP, DotNetCore) - publishSdkImageToAcr.sh / publishSdkToAcr.yml - ACR publish pipeline Modified files: - Platform install decisions (NodePlatform, PythonPlatform, PhpPlatform, DotNetCorePlatform) with ACR branch before existing DynamicInstall - Version provider orchestrators with ACR provider chain - PlatformInstallerBase with GetAcrInstallerScriptSnippet() - Go startup script generator with ACR download path - DI registrations for all new providers - Constants, options, settings keys for ACR configuration All changes are additive and behind feature flags. Existing blob storage code paths are completely untouched. * refactor: Simplify ExternalAcrSdkProvider - let LWASv2 handle image resolution Remove ACR repository/tag construction from ExternalAcrSdkProvider. Now sends only platform, version, and debianFlavor to LWASv2. LWASv2 resolves the SDK companion image from LinuxAssets and handles pinning, containerd pull, mount, and extraction. * refactor: Remove script-based ACR publish (moved to AAPT-Antares-Oryx OneBranch pipeline) Remove publishSdkImageToAcr.sh and publishSdkToAcr.yml since SDK images are now built and pushed using onebranch.pipeline.imagebuildinfo in the AAPT-Antares-Oryx pipeline, not via custom scripts. * fix: Default ACR SDK registry to oryxacr.azurecr.io Use the same ACR that AAPT-Antares-Oryx publishes SDK images to. Updated both C# constant and Go constant. * fix: Resolve StyleCop analyzer errors in ACR SDK provider files - SA1204: Move static GetFirstLayerDigest before instance members (OciRegistryClient.cs) - SA1124: Remove #region/#endregion around OCI JSON models (OciRegistryClient.cs) - SA1202: Move protected GetAcrInstallerScriptSnippet before private methods (PlatformInstallerBase.cs) - SA1116/SA1117: Place multi-line parameters each on own line (ExternalAcrSdkProvider.cs) - SA1515: Add blank line before single-line comment (DotNetCoreAcrVersionProvider.cs) * fix: Resolve StyleCop analyzer errors in ACR SDK provider files - SA1204: Move static GetFirstLayerDigest before instance members (OciRegistryClient.cs) - SA1124: Remove #region/#endregion around OCI JSON models (OciRegistryClient.cs) - SA1202: Move protected GetAcrInstallerScriptSnippet before private methods (PlatformInstallerBase.cs) - SA1116/SA1117: Place multi-line parameters each on own line (ExternalAcrSdkProvider.cs) - SA1515: Add blank line before single-line comment (DotNetCoreAcrVersionProvider.cs) * Feature/oryx sdk acr distribution sdks resolver (#2842) * fix: Resolve StyleCop analyzer errors in ACR SDK provider files - SA1204: Move static GetFirstLayerDigest before instance members (OciRegistryClient.cs) - SA1124: Remove #region/#endregion around OCI JSON models (OciRegistryClient.cs) - SA1202: Move protected GetAcrInstallerScriptSnippet before private methods (PlatformInstallerBase.cs) - SA1116/SA1117: Place multi-line parameters each on own line (ExternalAcrSdkProvider.cs) - SA1515: Add blank line before single-line comment (DotNetCoreAcrVersionProvider.cs) * fix sdk providers logic * refactor pythonPlatform * refctor for dotnet,php and node --------- Co-authored-by: Sarath chandra Bussa <sbussa@microsoft.com> * fix build and tests (#2843) * fix to fetch from acr (#2844) * Update version providers (#2845) * Refactor version providers (#2846) * Fix acr version info (#2848) * Changes for php composer (#2849) * Clean dead code (#2850) * Fix priroity in sdk resolver (#2851) * Fix Uts and version providers (#2852) * ACR Sdk provider refactor (#2854) * registryclient improvements * fix build issue * Add changes foe external ACR provider (#2855) * Change flag name for external acr case (#2856) * Handle multiplatform (#2857) * Revert "Handle multiplatform (#2857)" (#2858) This reverts commit 77d9f99. * Disable external ACR for multiplatform (#2859) * Fix test case (#2860) * Refactor external acr provider (#2862) * update externalacrsdkprovider contract * Align ExternalAcrSdkProvider and ExternalAcrVersionProviderBase with LWASv2 socket contract - Fix socket path to /var/sdk-image-sockets/oryx-pull-sdk-image.socket (ACR socket) - Add ExternalAcrSdksStorageDir constant (/var/OryxAcrSdks) for ACR SDK cache - Add top-level Action field (pull-sdk / get-version) to request DTOs - Remove BlobName and UrlParameters from ExternalAcrSdkProvider request - Accept server-returned filename as success response instead of expecting 'Success$' * update logic for dotnet and default versions * update logic for dotnet and default versions (#2864) * Refactor code for new acr way (#2863) * nit fix (#2865) * Refactor ACR SDK provider to return tarball path * ACR SDK distribution: Refactor direct acr sdk fetching logic (#2866) * update logic for dotnet and default versions * Refactor ACR SDK provider to return tarball path * Cleanup startupscriptgen and refactor ACR SDK distribution constants * ACR Sdk distribution: Cleanup startupscriptgenerator and fix image repo constants (#2867) * update logic for dotnet and default versions * Refactor ACR SDK provider to return tarball path * Cleanup startupscriptgen and refactor ACR SDK distribution constants * Refactor ACR SDK provider methods to return boolean status and update related logic across platforms * Update version provider logic to check for external SDK provider option * ACR Sdk distribution: Fix failure path for acr sdk fetching (#2868) * update logic for dotnet and default versions * Refactor ACR SDK provider to return tarball path * Cleanup startupscriptgen and refactor ACR SDK distribution constants * Refactor ACR SDK provider methods to return boolean status and update related logic across platforms * Update version provider logic to check for external SDK provider option * Update version provider fallback (#2872) * Update fallback logic for sdks (#2873) * version map fix * fix externalacrprovider for dotnet * ACR Sdk distribution: fix external ACR sdk provider for dotnet (#2874) * update logic for dotnet and default versions * Refactor ACR SDK provider to return tarball path * Cleanup startupscriptgen and refactor ACR SDK distribution constants * Refactor ACR SDK provider methods to return boolean status and update related logic across platforms * Update version provider logic to check for external SDK provider option * version map fix * fix externalacrprovider for dotnet * Feature/oryx sdk acr distribution fix phpand composer (#2875) * Update PHP and PHP-Composer Platform * update php and composer * Enhance OciRegistryClient to support anonymous token acquisition for public repositories * add logging * Acr SDK Distribution: Add token for listing sdk images (#2876) * update logic for dotnet and default versions * Refactor ACR SDK provider to return tarball path * Cleanup startupscriptgen and refactor ACR SDK distribution constants * Refactor ACR SDK provider methods to return boolean status and update related logic across platforms * Update version provider logic to check for external SDK provider option * version map fix * fix externalacrprovider for dotnet * Enhance OciRegistryClient to support anonymous token acquisition for public repositories * add logging * fix build * suppress auth header from logs * fix socket paths * Add a method to fetch all versions for a platform (#2877) * fix fallback flow * logging fixes * logging fixes (#2878) * fix tar extraction and add more logging * fix dotnet image pull * handle gzip stream * fix php installationscript duplication * ACR Sdk dist: Enhancements for cache management (#2889) * add fallback for no versions returned * improve image caching logic * handle mcr repo * Feature/oryx sdk acr distribution pr review 1 (#2888) * nit fixes in pr review * nit fixes 2.0 * refactor * Fix UTs * ACR SDK dist: Add tests, refactoring and some fixes (#2892) * Fix composer logic (#2893) * Fix composer logic * nit * fix build (#2894) * Add .NET SDK versions and composerVersion to constants.yml Add DOTNET_SDK_80, DOTNET_SDK_90, DOTNET_SDK_100 variables to pair with existing runtime version variables. Add composerVersion for PHP Composer. These are consumed by the Official and Buddy pipelines for SDK ACR publishing. * ACR SDK dist: Refactor version provider and bump up dependencies (#2895) * fix node logic and some other bugs * more fixes * socket helper refactor * more fixes * add tests * refactor version provider * fix build * bump version * ACR SDK dist: Additional test cases (#2896) * fix node logic and some other bugs * more fixes * socket helper refactor * more fixes * add tests * refactor version provider * fix build * bump version * add tests * follow existing pattern for php composer, add more tests * Revert "Add .NET SDK versions and composerVersion to constants.yml" This reverts commit 2f2b67d. * update comment * Add VERSIONS_TO_BUILD_OVERRIDE to force-build specific SDK versions (#2897) - Modify buildPlatform() in __common.sh: when VERSIONS_TO_BUILD_OVERRIDE is set, filter versionsToBuild.txt to only matching versions and set OVERWRITE_EXISTING_SDKS=true to bypass blob existence checks - Add ARG/ENV VERSIONS_TO_BUILD_OVERRIDE to nodejs, python, php, and php-composer Dockerfiles so the env var reaches build containers via --build-arg Co-authored-by: Sarath chandra Bussa <sbussa@microsoft.com> * resolve comments * nit * fix comment * add tests --------- Co-authored-by: Sarath chandra Bussa <sbussa@microsoft.com> Co-authored-by: Akshay Kumar <kumaraksh@microsoft.com> Co-authored-by: Akshay Kumar <112485097+kumaraksh1@users.noreply.github.com> Co-authored-by: Sarath Chandra Bussa <bussasarath2@gmail.com>
1 parent 397be11 commit 76a5796

4 files changed

Lines changed: 477 additions & 7 deletions

File tree

‎src/BuildScriptGenerator/DotNetCore/DotnetCorePlatform.cs‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -327,14 +327,23 @@ public void ResolveVersions(RepositoryContext context, PlatformDetectorResult de
327327
// which SDK companion image to use, so we trust its SDK version.
328328
if (this.commonOptions.EnableExternalAcrSdkProvider)
329329
{
330-
var dictatedSdk = this.externalAcrVersionProvider.GetSdkVersion();
331-
if (!string.IsNullOrEmpty(dictatedSdk))
330+
try
332331
{
333-
this.logger.LogInformation(
334-
"External ACR provider returned .NET SDK version {Version}. Using it directly.",
335-
dictatedSdk);
336-
dotNetCorePlatformDetectorResult.SdkVersion = dictatedSdk;
337-
return;
332+
var dictatedSdk = this.externalAcrVersionProvider.GetSdkVersion();
333+
if (!string.IsNullOrEmpty(dictatedSdk))
334+
{
335+
this.logger.LogInformation(
336+
"External ACR provider returned .NET SDK version {Version}. Using it directly.",
337+
dictatedSdk);
338+
dotNetCorePlatformDetectorResult.SdkVersion = dictatedSdk;
339+
return;
340+
}
341+
}
342+
catch (Exception ex)
343+
{
344+
this.logger.LogError(
345+
ex,
346+
"Error getting SDK version from external ACR provider. Falling back to next provider for SDK resolution.");
338347
}
339348
}
340349

‎tests/BuildScriptGenerator.Tests/ExternalAcrVersionProviderBaseTest.cs‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
// --------------------------------------------------------------------------------------------
2+
// Copyright (c) Microsoft Corporation. All rights reserved.
3+
// Licensed under the MIT license.
4+
// --------------------------------------------------------------------------------------------
5+
6+
using Microsoft.Extensions.Logging.Abstractions;
7+
using Microsoft.Extensions.Options;
8+
using Microsoft.Oryx.BuildScriptGenerator.DotNetCore;
9+
using Xunit;
10+
11+
namespace Microsoft.Oryx.BuildScriptGenerator.Tests
12+
{
13+
/// <summary>
14+
/// Tests for <see cref="ExternalAcrVersionProviderBase"/> exercised through
15+
/// <see cref="DotNetCoreExternalAcrVersionProvider"/>, the thinnest concrete subclass.
16+
/// Validates the DebianFlavor guard and the never-throws contract.
17+
/// </summary>
18+
public class ExternalAcrVersionProviderBaseTest
19+
{
20+
[Theory]
21+
[InlineData(null)]
22+
[InlineData("")]
23+
public void GetSdkVersion_ReturnsNull_WhenDebianFlavorIsNullOrEmpty(string debianFlavor)
24+
{
25+
var provider = CreateProvider(debianFlavor: debianFlavor);
26+
27+
var version = provider.GetSdkVersion();
28+
29+
Assert.Null(version);
30+
}
31+
32+
[Fact]
33+
public void GetSdkVersion_NeverThrows_WhenSocketIsUnavailable()
34+
{
35+
// GetCompanionSdkVersion has try/catch around SendRequestAsync.
36+
// No exception should ever propagate to the caller.
37+
// DotNetCorePlatform.ResolveVersions() also has a try/catch around
38+
// GetSdkVersion() for defense-in-depth — but GetSdkVersion itself
39+
// must never throw, matching the contract of other version providers.
40+
var provider = CreateProvider(debianFlavor: "bookworm");
41+
42+
var ex = Record.Exception(() => provider.GetSdkVersion());
43+
44+
Assert.Null(ex);
45+
}
46+
47+
private static DotNetCoreExternalAcrVersionProvider CreateProvider(string debianFlavor)
48+
{
49+
var options = Options.Create(new BuildScriptGeneratorOptions
50+
{
51+
DebianFlavor = debianFlavor,
52+
});
53+
54+
return new DotNetCoreExternalAcrVersionProvider(
55+
options,
56+
NullLoggerFactory.Instance,
57+
new DefaultStandardOutputWriter());
58+
}
59+
}
60+
}

‎tests/BuildScriptGenerator.Tests/PlatformInstallerBaseTest.cs‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
// --------------------------------------------------------------------------------------------
2+
// Copyright (c) Microsoft Corporation. All rights reserved.
3+
// Licensed under the MIT license.
4+
// --------------------------------------------------------------------------------------------
5+
6+
using Microsoft.Extensions.Logging.Abstractions;
7+
using Microsoft.Extensions.Options;
8+
using Microsoft.Oryx.BuildScriptGenerator.Common;
9+
using Microsoft.Oryx.BuildScriptGenerator.Node;
10+
using Xunit;
11+
12+
namespace Microsoft.Oryx.BuildScriptGenerator.Tests
13+
{
14+
/// <summary>
15+
/// Tests for <see cref="PlatformInstallerBase.GetInstallerScriptSnippet"/> exercised
16+
/// through <see cref="NodePlatformInstaller"/>.
17+
/// </summary>
18+
public class PlatformInstallerBaseTest
19+
{
20+
21+
[Fact]
22+
public void GetInstallerScriptSnippet_WhenSkipDownload_ChecksExternalSdkDir()
23+
{
24+
var installer = CreateInstaller(debianFlavor: "bookworm");
25+
26+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: true);
27+
28+
Assert.Contains("/var/OryxSdks", snippet);
29+
}
30+
31+
[Fact]
32+
public void GetInstallerScriptSnippet_WhenSkipDownload_ChecksExternalAcrSdkDir()
33+
{
34+
var installer = CreateInstaller(debianFlavor: "bookworm");
35+
36+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: true);
37+
38+
Assert.Contains("/var/OryxAcrSdks", snippet);
39+
}
40+
41+
[Fact]
42+
public void GetInstallerScriptSnippet_WhenSkipDownload_ExitsOnMissingTarball()
43+
{
44+
var installer = CreateInstaller(debianFlavor: "bookworm");
45+
46+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: true);
47+
48+
Assert.Contains("exit 1", snippet);
49+
}
50+
51+
[Fact]
52+
public void GetInstallerScriptSnippet_WhenSkipDownload_DoesNotContainSdkDownloadCurl()
53+
{
54+
var installer = CreateInstaller(debianFlavor: "bookworm");
55+
56+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: true);
57+
58+
// Should not contain the SDK binary download curl
59+
Assert.DoesNotContain("curl --connect-timeout", snippet);
60+
}
61+
62+
[Fact]
63+
public void GetInstallerScriptSnippet_WhenSkipDownload_ChecksDynamicInstallDir()
64+
{
65+
var dynamicDir = "/opt/oryx-dynamic";
66+
var installer = CreateInstaller(debianFlavor: "bookworm", dynamicInstallRootDir: dynamicDir);
67+
68+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: true);
69+
70+
Assert.Contains(dynamicDir, snippet);
71+
}
72+
73+
[Fact]
74+
public void GetInstallerScriptSnippet_WhenSkipDownload_UsesFlavorInTarballName()
75+
{
76+
var installer = CreateInstaller(debianFlavor: "bookworm");
77+
78+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: true);
79+
80+
Assert.Contains("nodejs-bookworm-20.0.0.tar.gz", snippet);
81+
}
82+
83+
[Fact]
84+
public void GetInstallerScriptSnippet_WhenSkipDownload_Stretch_UsesLegacyTarballName()
85+
{
86+
var installer = CreateInstaller(debianFlavor: OsTypes.DebianStretch);
87+
88+
var snippet = installer.GetInstallerScriptSnippet("14.0.0", skipSdkBinaryDownload: true);
89+
90+
// Stretch uses the legacy format: {platform}-{version}.tar.gz (no flavor in tarball name)
91+
Assert.Contains("nodejs-14.0.0.tar.gz", snippet);
92+
Assert.DoesNotContain("nodejs-stretch-14.0.0.tar.gz", snippet);
93+
}
94+
95+
96+
[Fact]
97+
public void GetInstallerScriptSnippet_WhenNotSkipDownload_UsesCurl()
98+
{
99+
var installer = CreateInstaller(debianFlavor: "bookworm");
100+
101+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: false);
102+
103+
Assert.Contains("curl", snippet);
104+
}
105+
106+
[Fact]
107+
public void GetInstallerScriptSnippet_WhenNotSkipDownload_ContainsPrimaryUrl()
108+
{
109+
var installer = CreateInstaller(debianFlavor: "bookworm", sdkStorageUrl: "https://primary.example.com");
110+
111+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: false);
112+
113+
Assert.Contains("https://primary.example.com", snippet);
114+
}
115+
116+
[Fact]
117+
public void GetInstallerScriptSnippet_WhenNotSkipDownload_ContainsBackupUrl()
118+
{
119+
var installer = CreateInstaller(
120+
debianFlavor: "bookworm",
121+
sdkStorageUrl: "https://primary.example.com",
122+
sdkStorageBackupUrl: "https://backup.example.com");
123+
124+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: false);
125+
126+
Assert.Contains("https://backup.example.com", snippet);
127+
}
128+
129+
[Fact]
130+
public void GetInstallerScriptSnippet_WhenNotSkipDownload_EmbedsDebianFlavorInUrl()
131+
{
132+
var installer = CreateInstaller(debianFlavor: "bookworm");
133+
134+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: false);
135+
136+
// The DEBIAN_FLAVOR variable is exported, and the curl line references $DEBIAN_FLAVOR
137+
Assert.Contains("DEBIAN_FLAVOR=bookworm", snippet);
138+
Assert.Contains("nodejs-$DEBIAN_FLAVOR-20.0.0.tar.gz", snippet);
139+
}
140+
141+
[Fact]
142+
public void GetInstallerScriptSnippet_WhenNotSkipDownload_Stretch_UsesLegacyUrlPattern()
143+
{
144+
var installer = CreateInstaller(debianFlavor: OsTypes.DebianStretch);
145+
146+
var snippet = installer.GetInstallerScriptSnippet("14.0.0", skipSdkBinaryDownload: false);
147+
148+
// Stretch path: {platform}-{version}.tar.gz (no $DEBIAN_FLAVOR)
149+
Assert.Contains("nodejs-14.0.0.tar.gz", snippet);
150+
}
151+
152+
[Fact]
153+
public void GetInstallerScriptSnippet_WhenNotSkipDownload_DoesNotContainExitOne()
154+
{
155+
// The curl path handles failures via the backup URL, not exit 1 from tarball lookup
156+
var installer = CreateInstaller(debianFlavor: "bookworm");
157+
158+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: false);
159+
160+
// Should not have the "Could not find cached tarball" exit 1
161+
Assert.DoesNotContain("Could not find cached tarball", snippet);
162+
}
163+
164+
[Fact]
165+
public void GetInstallerScriptSnippet_WritesSentinelFile()
166+
{
167+
var installer = CreateInstaller(debianFlavor: "bookworm");
168+
169+
var snippet = installer.GetInstallerScriptSnippet("20.0.0", skipSdkBinaryDownload: false);
170+
171+
Assert.Contains(SdkStorageConstants.SdkDownloadSentinelFileName, snippet);
172+
}
173+
174+
175+
private static NodePlatformInstaller CreateInstaller(
176+
string debianFlavor,
177+
string dynamicInstallRootDir = "/opt/oryx",
178+
string sdkStorageUrl = "https://sdk.example.com",
179+
string sdkStorageBackupUrl = "https://sdk-backup.example.com")
180+
{
181+
var options = Options.Create(new BuildScriptGeneratorOptions
182+
{
183+
DebianFlavor = debianFlavor,
184+
DynamicInstallRootDir = dynamicInstallRootDir,
185+
OryxSdkStorageBaseUrl = sdkStorageUrl,
186+
OryxSdkStorageBackupBaseUrl = sdkStorageBackupUrl,
187+
});
188+
189+
return new NodePlatformInstaller(options, NullLoggerFactory.Instance);
190+
}
191+
}
192+
}

0 commit comments

Comments
 (0)