Skip to content

Commit 1cb37f2

Browse files
Annelotte-Monsstijnmoreelspim-simons
authored
feat: Added "Set Entities" script for Azure Table storage (#222)
Co-authored-by: Stijn Moreels <9039753+stijnmoreels@users.noreply.github.com> Co-authored-by: Pim Simons <pim.simons@codit.eu> Co-authored-by: Pim Simons <32359437+pim-simons@users.noreply.github.com>
1 parent 12902f5 commit 1cb37f2

15 files changed

Lines changed: 482 additions & 2 deletions

‎build/templates/run-pester-tests.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@ parameters:
22
projectName: ''
33

44
steps:
5+
- pwsh: 'wget -O - https://aka.ms/install-powershell.sh | sudo bash'
6+
displayName: 'Update Powershell on Linux'
7+
condition: eq( variables['Agent.OS'], 'Linux' )
8+
- powershell: 'Invoke-Expression -Command "& { $(Invoke-RestMethod -Uri ''https://aka.ms/install-powershell.ps1'') } -UseMSI -quiet" '
9+
displayName: 'Update Powershell on Windows'
10+
condition: eq( variables['Agent.OS'], 'Windows_NT' )
511
- bash: |
612
if [ -z "$PROJECT_NAME" ]; then
713
echo "##vso[task.logissue type=error;]Missing template parameter \"projectName\""
@@ -37,7 +43,7 @@ steps:
3743
Install-Module -Name Microsoft.Graph.Applications -Force -SkipPublisherCheck -MaximumVersion 1.15.0
3844
Write-Host "Done installing, start importing modules"
3945
displayName: 'Install Pester test framework and Az required modules'
40-
- powershell: |
46+
- pwsh: |
4147
Import-Module ./src/Arcus.Scripting.Security
4248
Get-ChildItem -Path ./src -Filter *.psm1 -Recurse -Exclude "*Arcus.Scripting.Security*", "*.All.psm1" |
4349
% { Write-Host "Import $($_.DirectoryName) module"

‎docs/preview/03-Features/powershell/azure-storage/azure-storage-table.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,81 @@ PS> Create-AzStorageTable `
5353
# Failed to re-create the Azure storage table 'products' in Azure storage account 'admin', retrying in 5 seconds...
5454
# Azure storage table 'products' created in Azure storage account 'admin'
5555
```
56+
57+
58+
## Set the entities in a table of an Azure Storage Account
59+
60+
Deletes all entities of a specified table in an Azure Storage Account and creates new entities based on a configuration file.
61+
62+
| Parameter | Mandatory | Description |
63+
| ---------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------- |
64+
| `ResourceGroupName` | yes | The resource group where the Azure Storage Account is located |
65+
| `StorageAccountName` | yes | The name of the Azure Storage Account that contains the table |
66+
| `TableName` | yes | The name of the table in which the entities should be set |
67+
| `ConfigurationFile` | yes | Path to the JSON Configuration file containing all the entities to be set |
68+
69+
**Configuration File**
70+
71+
The configuration file is a simple JSON file that contains all of the entities that should be set on the specified table, the JSON file consists of an array of JSON objects (= your entities). Each object contains simple name-value pairs (string-string).
72+
73+
Defining the PartitionKey and/or RowKey are optional, if not provided a random GUID will be set for these.
74+
75+
The file needs to adhere to the following JSON schema:
76+
77+
``` json
78+
{
79+
"definitions": {},
80+
"$schema": "https://json-schema.org/draft/2020-12/schema",
81+
"$id": "https://scripting.arcus-azure.net/Features/powershell/azure-storage/azure-storage-table/config.json",
82+
"type": "array",
83+
"title": "The configuration JSON schema",
84+
"items": [{
85+
"type": "object",
86+
"patternProperties": {
87+
"^.*$": {
88+
"anyOf": [{
89+
"type": "string"
90+
}, {
91+
"type": "null"
92+
}
93+
]
94+
}
95+
},
96+
"additionalProperties": false
97+
}
98+
]
99+
}
100+
```
101+
102+
**Example Configuration File**
103+
104+
```json
105+
[
106+
{
107+
"PartitionKey": "SystemA",
108+
"RowKey": "100",
109+
"ReadPath": "/home/in",
110+
"ReadIntervalInSeconds": "30"
111+
},
112+
{
113+
"PartitionKey": "SystemA",
114+
"RowKey": "200",
115+
"ReadPath": "/data/in",
116+
"ReadIntervalInSeconds": "10",
117+
"HasSubdirectories": "true"
118+
}
119+
]
120+
```
121+
122+
**Example**
123+
124+
```powershell
125+
PS> Set-AzTableStorageEntities `
126+
-ResourceGroupName "someresourcegroup" `
127+
-StorageAccountName "somestorageaccount" `
128+
-TableName "sometable" `
129+
-ConfigurationFile ".\config.json"
130+
# Deleting all existing entities in Azure storage table 'sometable' for Azure storage account 'somestorageaccount' in resource group 'someresourcegroup'...
131+
# Successfully deleted all existing entities in Azure storage table 'sometable' for Azure storage account 'somestorageaccount' in resource group 'someresourcegroup'
132+
# Successfully added all entities in Azure storage table 'sometable' for Azure storage account 'somestorageaccount' in resource group 'someresourcegroup'
133+
```

‎src/Arcus.Scripting.Storage.Table/Arcus.Scripting.Storage.Table.psd1‎

82 Bytes
Binary file not shown.

‎src/Arcus.Scripting.Storage.Table/Arcus.Scripting.Storage.Table.psm1‎

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,4 +51,48 @@ function Create-AzStorageTable {
5151
}
5252
}
5353

54-
Export-ModuleMember -Function Create-AzStorageTable
54+
Export-ModuleMember -Function Create-AzStorageTable
55+
56+
57+
58+
<#
59+
.Synopsis
60+
Sets the entities in a specified table storage.
61+
62+
.Description
63+
Sets (Delete & Add) the entities in a specified table storage to the values specified in a (JSON) configuration file.
64+
65+
.Parameter ResourceGroupName
66+
The resource group where the Azure Storage Account is located.
67+
68+
.Parameter StorageAccountName
69+
The name of the Azure Storage Account which holds the Azure table storage.
70+
71+
.Parameter TableName
72+
The name of the table on the Azure Storage Account.
73+
74+
.Parameter ConfigurationFile
75+
The Configuration File (JSON) specifying the entities that have to be set in the table storage. The JSON file has a single property "entities" which is an array of objects.
76+
77+
.Parameter RetryIntervalSeconds
78+
The optional amount of seconds to wait each retry-run when a failure occures during the re-creating process.
79+
80+
.Parameter MaxRetryCount
81+
The optional maximum amount of retry-runs should happen when a failure occurs during the re-creating process.
82+
#>
83+
function Set-AzTableStorageEntities {
84+
param(
85+
[Parameter(Mandatory = $true)][string] $ResourceGroupName = $(throw "Name of resource group is required"),
86+
[Parameter(Mandatory = $true)][string] $StorageAccountName = $(throw "Name of Azure storage account is required"),
87+
[Parameter(Mandatory = $true)][string] $TableName = $(throw "Name of Azure table is required"),
88+
[Parameter(Mandatory = $true)][string] $ConfigurationFile = $(throw "Path to the configuration file is required")
89+
)
90+
91+
. $PSScriptRoot\Scripts\Set-AzTableStorageEntities.ps1 `
92+
-ResourceGroupName $ResourceGroupName `
93+
-StorageAccountName $StorageAccountName `
94+
-TableName $TableName `
95+
-ConfigurationFile $ConfigurationFile
96+
}
97+
98+
Export-ModuleMember -Function Set-AzTableStorageEntities

‎src/Arcus.Scripting.Storage.Table/Scripts/Set-AzTableStorageEntities.ps1‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
param(
2+
[Parameter(Mandatory = $true)][string] $ResourceGroupName = $(throw "Name of resource group is required"),
3+
[Parameter(Mandatory = $true)][string] $StorageAccountName = $(throw "Name of Azure storage account is required"),
4+
[Parameter(Mandatory = $true)][string] $TableName = $(throw "Name of Azure table is required"),
5+
[Parameter(Mandatory = $true)][string] $ConfigurationFile = $(throw "Path to the configuration file is required")
6+
)
7+
8+
if (-not (Test-Path -Path $ConfigurationFile)) {
9+
throw "Cannot re-create entities based on JSON configuration file because no file was found at: '$ConfigurationFile'"
10+
}
11+
if ((Get-Content -Path $ConfigurationFile -Raw) -eq $null) {
12+
throw "Cannot re-create entities based on JSON configuration file because the file is empty."
13+
}
14+
15+
$schema = @'
16+
{
17+
"definitions": {},
18+
"$schema": "https://json-schema.org/draft/2020-12/schema",
19+
"$id": "https://scripting.arcus-azure.net/Features/powershell/azure-storage/azure-storage-table/config.json",
20+
"type": "array",
21+
"title": "The configuration JSON schema",
22+
"items": [{
23+
"type": "object",
24+
"patternProperties": {
25+
"^.*$": {
26+
"anyOf": [{
27+
"type": "string"
28+
}, {
29+
"type": "null"
30+
}
31+
]
32+
}
33+
},
34+
"additionalProperties": false
35+
}
36+
]
37+
}
38+
'@
39+
40+
if (-not (Get-Content -Path $ConfigurationFile -Raw | Test-Json -Schema $schema -ErrorAction SilentlyContinue)) {
41+
throw "Cannot re-create entities based on JSON configuration file because the file does not contain a valid JSON configuration file."
42+
}
43+
44+
Write-Verbose "Retrieving Azure storage account context for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'..."
45+
$storageAccount = Get-AzStorageAccount -ResourceGroupName $ResourceGroupName -Name $StorageAccountName
46+
if ($storageAccount -eq $null) {
47+
throw "Retrieving Azure storage account context for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName' failed."
48+
}
49+
$ctx = $storageAccount.Context
50+
Write-Verbose "Azure storage account context has been retrieved for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'"
51+
52+
Write-Verbose "Retrieving Azure storage table '$TableName' for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'..."
53+
$storageTable = Get-AzStorageTable -Name $TableName -Context $ctx
54+
if ($storageTable -eq $null) {
55+
throw "Retrieving Azure storage table '$TableName' for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName' failed."
56+
}
57+
$cloudTable = ($storageTable).CloudTable
58+
Write-Verbose "Azure storage table '$TableName' has been retrieved for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'"
59+
60+
Write-Host "Deleting all existing entities in Azure storage table '$TableName' for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'..."
61+
$entitiesToDelete = Get-AzTableRow -table $cloudTable
62+
$deletedEntities = $entitiesToDelete | Remove-AzTableRow -table $cloudTable
63+
Write-Host "Successfully deleted all existing entities in Azure storage table '$TableName' for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'"
64+
65+
$configFile = Get-Content -Path $ConfigurationFile | ConvertFrom-Json
66+
foreach ($entityToAdd in $configFile) {
67+
if ($entityToAdd.PartitionKey) {
68+
$partitionKey = $entityToAdd.PartitionKey
69+
$entityToAdd.PSObject.Properties.Remove('PartitionKey')
70+
} else {
71+
$partitionKey = New-Guid
72+
}
73+
74+
if ($entityToAdd.RowKey) {
75+
$rowKey = $entityToAdd.RowKey
76+
$entityToAdd.PSObject.Properties.Remove('RowKey')
77+
} else {
78+
$rowKey = New-Guid
79+
}
80+
81+
$entityHash = @{}
82+
$entityToAdd.PSObject.Properties | foreach { $entityHash[$_.Name] = $_.Value }
83+
84+
$addedRow = Add-AzTableRow `
85+
-table $cloudTable `
86+
-partitionKey $partitionKey `
87+
-rowKey $rowKey `
88+
-property $entityHash
89+
90+
Write-Verbose "Successfully added row with PartitionKey '$partitionKey' and RowKey '$rowKey' to Azure storage table '$TableName' for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'"
91+
}
92+
93+
Write-Host "Successfully added all entities in Azure storage table '$TableName' for Azure storage account '$StorageAccountName' in resource group '$ResourceGroupName'"

‎src/Arcus.Scripting.Tests.Integration/Arcus.Scripting.Storage.Table.tests.ps1‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,5 +87,57 @@ InModuleScope Arcus.Scripting.Storage.Table {
8787
}
8888
}
8989
}
90+
Context "Setting Azure Table Storage Entities" {
91+
It "Setting entities in an Azure Table Storage account" {
92+
# Arrange
93+
$resourceGroup = $config.Arcus.ResourceGroupName
94+
$storageAccountName = $config.Arcus.Storage.StorageAccount.Name
95+
$tableName = "SetEntityTable"
96+
$configFile = "$PSScriptRoot\Files\TableStorage\set-aztablestorageentities-config.json"
97+
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccountName
98+
99+
try {
100+
# Act
101+
Create-AzStorageTable `
102+
-ResourceGroupName $config.Arcus.ResourceGroupName `
103+
-StorageAccountName $config.Arcus.Storage.StorageAccount.Name `
104+
-Table $tableName
105+
106+
Set-AzTableStorageEntities -ResourceGroupName $resourceGroup -StorageAccountName $storageAccountName -TableName $tableName -ConfigurationFile $configFile
107+
108+
# Assert
109+
$storageTable = Get-AzStorageTable –Name $tableName –Context $storageAccount.Context
110+
(Get-AzTableRow -table $storageTable.CloudTable | measure).Count |
111+
Should -Be 2
112+
} finally {
113+
Remove-AzStorageTable -Name $tableName -Context $storageAccount.Context -Force -ErrorAction SilentlyContinue
114+
}
115+
}
116+
It "Setting entities in an Azure Table Storage account without specifying PartitionKey and RowKey" {
117+
# Arrange
118+
$resourceGroup = $config.Arcus.ResourceGroupName
119+
$storageAccountName = $config.Arcus.Storage.StorageAccount.Name
120+
$tableName = "SetEntityTableNoKeys"
121+
$configFile = "$PSScriptRoot\Files\TableStorage\set-aztablestorageentities-config-nokeys.json"
122+
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccountName
123+
124+
try {
125+
# Act
126+
Create-AzStorageTable `
127+
-ResourceGroupName $config.Arcus.ResourceGroupName `
128+
-StorageAccountName $config.Arcus.Storage.StorageAccount.Name `
129+
-Table $tableName
130+
131+
Set-AzTableStorageEntities -ResourceGroupName $resourceGroup -StorageAccountName $storageAccountName -TableName $tableName -ConfigurationFile $configFile
132+
133+
# Assert
134+
$storageTable = Get-AzStorageTable –Name $tableName –Context $storageAccount.Context
135+
(Get-AzTableRow -table $storageTable.CloudTable | measure).Count |
136+
Should -Be 2
137+
} finally {
138+
Remove-AzStorageTable -Name $tableName -Context $storageAccount.Context -Force -ErrorAction SilentlyContinue
139+
}
140+
}
141+
}
90142
}
91143
}

‎src/Arcus.Scripting.Tests.Integration/Arcus.Scripting.Tests.Integration.pssproj‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@
6161
<Folder Include="Files\IntegrationAccount\Partners\" />
6262
<Folder Include="Files\IntegrationAccount\Schemas\" />
6363
<Folder Include="Files\IntegrationAccount\Schemas\Common\" />
64+
<Folder Include="Files\TableStorage\" />
6465
<Folder Include="SqlScripts\" />
6566
<Folder Include="SqlScripts\MigrationScriptsAreSuccessfullyExecuted\" />
6667
<Folder Include="SqlScripts\MigrationStopsOnError\" />
@@ -94,6 +95,8 @@
9495
<Content Include="Files\IntegrationAccount\Schemas\NestedSchema.xsd" />
9596
<Content Include="Files\IntegrationAccount\Schemas\Acknowledgement.xsd" />
9697
<Content Include="Files\IntegrationAccount\Schemas\Common\NestedSchemaRoot.xsd" />
98+
<Content Include="Files\TableStorage\set-aztablestorageentities-config-nokeys.json" />
99+
<Content Include="Files\TableStorage\set-aztablestorageentities-config.json" />
97100
<Content Include="SqlScripts\1_EmptyMigration.sql" />
98101
<Content Include="SqlScripts\MigrationScriptsAreSuccessfullyExecuted\0.0.1_baseline.sql" />
99102
<Content Include="SqlScripts\MigrationScriptsAreSuccessfullyExecuted\0.1.0_AddCustomerTable.sql" />

‎src/Arcus.Scripting.Tests.Integration/Files/TableStorage/set-aztablestorageentities-config-nokeys.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[
2+
{
3+
"ReadPath": "/home/in",
4+
"ReadIntervalInSeconds": "30"
5+
},
6+
{
7+
"ReadPath": "/data/in",
8+
"ReadIntervalInSeconds": "10",
9+
"HasSubdirectories": "true"
10+
}
11+
]

‎src/Arcus.Scripting.Tests.Integration/Files/TableStorage/set-aztablestorageentities-config.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[
2+
{
3+
"PartitionKey": "SystemA",
4+
"RowKey": "100",
5+
"ReadPath": "/home/in",
6+
"ReadIntervalInSeconds": "30"
7+
},
8+
{
9+
"PartitionKey": "SystemA",
10+
"RowKey": "200",
11+
"ReadPath": "/data/in",
12+
"ReadIntervalInSeconds": "10",
13+
"HasSubdirectories": "true"
14+
}
15+
]

0 commit comments

Comments
 (0)