diff --git a/tuts/116-aws-xray-gs/README.md b/tuts/116-aws-xray-gs/README.md new file mode 100644 index 00000000..0e444ebb --- /dev/null +++ b/tuts/116-aws-xray-gs/README.md @@ -0,0 +1,56 @@ +# X-Ray: Send traces and query them + +Send trace segments to AWS X-Ray, query trace summaries, retrieve full traces, create a trace group, and view the service graph. + +## Source + +https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html + +## Use case + +- **ID**: xray/getting-started +- **Level**: intermediate +- **Core actions**: `xray:PutTraceSegments`, `xray:GetTraceSummaries`, `xray:BatchGetTraces`, `xray:CreateGroup` + +## Steps + +1. Send a trace segment and subsegment +2. Get trace summaries +3. Get full trace details +4. Create a trace group with a filter expression +5. Get the service graph + +## Resources created + +| Resource | Type | +|----------|------| +| `tut-group-` | X-Ray group | + +## Duration + +~10 seconds + +## Cost + +No charge. X-Ray provides a free tier of 100,000 traces recorded and 1,000,000 traces scanned per month. + +## Related docs + +- [Sending trace data to X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html) +- [Retrieving trace data](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-gettingdata.html) +- [Using groups in X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-groups.html) +- [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/) + +--- + +## Appendix + +| Field | Value | +|-------|-------| +| Date | 2026-04-14 | +| Script lines | 87 | +| Exit code | 0 | +| Runtime | 10s | +| Steps | 5 | +| Issues | Fixed parent_id extraction quoting | +| Version | v1 | diff --git a/tuts/116-aws-xray-gs/REVISION-HISTORY.md b/tuts/116-aws-xray-gs/REVISION-HISTORY.md new file mode 100644 index 00000000..9042e533 --- /dev/null +++ b/tuts/116-aws-xray-gs/REVISION-HISTORY.md @@ -0,0 +1,8 @@ +# Revision History: 116-aws-xray-gs + +## Shell (CLI script) + +### 2026-04-14 v1 published +- Type: functional +- Initial version + diff --git a/tuts/116-aws-xray-gs/aws-xray-gs.md b/tuts/116-aws-xray-gs/aws-xray-gs.md new file mode 100644 index 00000000..12f571cb --- /dev/null +++ b/tuts/116-aws-xray-gs/aws-xray-gs.md @@ -0,0 +1,113 @@ +# Send traces and query them with AWS X-Ray + +## Overview + +In this tutorial, you use the AWS CLI to send trace segments to AWS X-Ray, query trace summaries, retrieve full trace details, create a trace group with a filter expression, and view the service graph. You then delete the group during cleanup. + +## Prerequisites + +- AWS CLI installed and configured with appropriate permissions. +- Python 3 installed (used to generate trace IDs and timestamps). +- An IAM principal with permissions for `xray:PutTraceSegments`, `xray:GetTraceSummaries`, `xray:BatchGetTraces`, `xray:CreateGroup`, `xray:DeleteGroup`, and `xray:GetServiceGraph`. + +## Step 1: Send a trace segment + +Generate a trace ID and send a segment representing an HTTP request, followed by a subsegment representing a downstream call. + +```bash +TRACE_ID=$(python3 -c "import time,random;print(f'1-{int(time.time()):08x}-{random.randbytes(12).hex()}')") +NOW=$(python3 -c "import time;print(time.time())") +END=$(python3 -c "import time;print(time.time()+0.5)") + +SEGMENT="{\"trace_id\":\"$TRACE_ID\",\"id\":\"$(openssl rand -hex 8)\",\"name\":\"tutorial-service\",\"start_time\":$NOW,\"end_time\":$END,\"http\":{\"request\":{\"method\":\"GET\",\"url\":\"https://example.com/api/items\"},\"response\":{\"status\":200}}}" + +aws xray put-trace-segments --trace-segment-documents "$SEGMENT" +``` + +X-Ray trace IDs follow the format `1--<96-bit random hex>`. Each segment needs a unique 64-bit hex ID, a name, and start/end times as Unix epoch floats. + +Send a subsegment linked to the parent segment: + +```bash +PARENT_ID=$(echo "$SEGMENT" | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])") +SUBSEG="{\"trace_id\":\"$TRACE_ID\",\"id\":\"$(openssl rand -hex 8)\",\"name\":\"database-query\",\"start_time\":$NOW,\"end_time\":$END,\"parent_id\":\"$PARENT_ID\"}" +aws xray put-trace-segments --trace-segment-documents "$SUBSEG" +``` + +Subsegments use `parent_id` to link to their parent segment. This creates the hierarchy visible in the X-Ray trace map. + +## Step 2: Get trace summaries + +Query recent trace summaries. X-Ray takes a few seconds to index new traces. + +```bash +START_TIME=$(python3 -c "import time;print(int(time.time()-60))") +END_TIME=$(python3 -c "import time;print(int(time.time()))") + +aws xray get-trace-summaries \ + --start-time "$START_TIME" --end-time "$END_TIME" \ + --query 'TraceSummaries[:3].{TraceId:Id,Duration:Duration,Status:Http.HttpStatus,Method:Http.HttpMethod}' \ + --output table +``` + +Trace summaries include the trace ID, duration, HTTP status, and response time. Use `--filter-expression` to narrow results by service name, status code, or annotation. + +## Step 3: Get full trace details + +Retrieve the complete trace including all segments and subsegments. + +```bash +aws xray batch-get-traces --trace-ids "$TRACE_ID" \ + --query 'Traces[0].Segments[].{Id:Id}' --output table +``` + +`batch-get-traces` returns the raw segment documents. You can retrieve up to 5 trace IDs per call. + +## Step 4: Create a trace group + +Create a group that filters traces by service name. + +```bash +GROUP_NAME="tut-group-$(openssl rand -hex 4)" + +GROUP_ARN=$(aws xray create-group --group-name "$GROUP_NAME" \ + --filter-expression 'service("tutorial-service")' \ + --query 'Group.GroupARN' --output text) +echo "Group ARN: $GROUP_ARN" +``` + +Groups let you organize traces by filter expression. Each group generates its own service graph and CloudWatch metrics. + +## Step 5: Get the service graph + +View the service graph for the time window containing your traces. + +```bash +aws xray get-service-graph \ + --start-time "$START_TIME" --end-time "$END_TIME" \ + --query 'Services[].{Name:Name,Type:Type,Edges:Edges|length(@)}' \ + --output table +``` + +The service graph shows services as nodes and their connections as edges. It may take a few seconds after sending traces for the graph to populate. + +## Cleanup + +Delete the trace group. Trace data itself expires automatically based on your X-Ray retention settings (default 30 days). + +```bash +aws xray delete-group --group-arn "$GROUP_ARN" +``` + +The script automates all steps including cleanup: + +```bash +bash aws-xray-gs.sh +``` + +## Related resources + +- [Sending trace data to X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html) +- [Retrieving trace data](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-gettingdata.html) +- [Using groups in X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/xray-console-groups.html) +- [AWS X-Ray pricing](https://aws.amazon.com/xray/pricing/) diff --git a/tuts/116-aws-xray-gs/aws-xray-gs.sh b/tuts/116-aws-xray-gs/aws-xray-gs.sh new file mode 100644 index 00000000..df35b983 --- /dev/null +++ b/tuts/116-aws-xray-gs/aws-xray-gs.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# Tutorial: Send traces and query them with AWS X-Ray +# Source: https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html + +WORK_DIR=$(mktemp -d) +LOG_FILE="$WORK_DIR/xray-$(date +%Y%m%d-%H%M%S).log" +exec > >(tee -a "$LOG_FILE") 2>&1 + +REGION=${AWS_DEFAULT_REGION:-${AWS_REGION:-$(aws configure get region 2>/dev/null)}} +if [ -z "$REGION" ]; then + echo "ERROR: No AWS region configured. Set one with: export AWS_DEFAULT_REGION=us-east-1" + exit 1 +fi +export AWS_DEFAULT_REGION="$REGION" +echo "Region: $REGION" + +RANDOM_ID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1) +GROUP_NAME="tut-group-${RANDOM_ID}" + +handle_error() { echo "ERROR on line $1"; trap - ERR; cleanup; exit 1; } +trap 'handle_error $LINENO' ERR + +cleanup() { + echo "" + echo "Cleaning up resources..." + [ -n "$GROUP_ARN" ] && aws xray delete-group --group-arn "$GROUP_ARN" 2>/dev/null && echo " Deleted group $GROUP_NAME" + rm -rf "$WORK_DIR" + echo "Cleanup complete." +} + +# Step 1: Send a trace segment +echo "Step 1: Sending trace segments" +TRACE_ID=$(python3 -c "import time,random;print(f'1-{int(time.time()):08x}-{random.randbytes(12).hex()}')") +NOW=$(python3 -c "import time;print(time.time())") +END=$(python3 -c "import time;print(time.time()+0.5)") + +SEGMENT="{\"trace_id\":\"$TRACE_ID\",\"id\":\"$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1)\",\"name\":\"tutorial-service\",\"start_time\":$NOW,\"end_time\":$END,\"http\":{\"request\":{\"method\":\"GET\",\"url\":\"https://example.com/api/items\"},\"response\":{\"status\":200}}}" + +aws xray put-trace-segments --trace-segment-documents "$SEGMENT" \ + --query 'UnprocessedTraceSegments' --output text +echo " Trace ID: $TRACE_ID" + +# Send a subsegment +PARENT_ID=$(echo "$SEGMENT" | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])") +SUBSEG="{\"trace_id\":\"$TRACE_ID\",\"id\":\"$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1)\",\"name\":\"database-query\",\"start_time\":$NOW,\"end_time\":$END,\"parent_id\":\"$PARENT_ID\"}" +aws xray put-trace-segments --trace-segment-documents "$SUBSEG" > /dev/null 2>&1 +echo " Sent parent segment + subsegment" + +# Step 2: Get trace summaries +echo "Step 2: Getting trace summaries" +sleep 5 +START_TIME=$(python3 -c "import time;print(int(time.time()-60))") +END_TIME=$(python3 -c "import time;print(int(time.time()))") +aws xray get-trace-summaries \ + --start-time "$START_TIME" --end-time "$END_TIME" \ + --query 'TraceSummaries[:3].{TraceId:Id,Duration:Duration,Status:Http.HttpStatus,Method:Http.HttpMethod}' --output table 2>/dev/null || \ + echo " No traces found yet (indexing can take a few seconds)" + +# Step 3: Get full trace +echo "Step 3: Getting full trace" +aws xray batch-get-traces --trace-ids "$TRACE_ID" \ + --query 'Traces[0].Segments[].{Id:Id}' --output table 2>/dev/null || \ + echo " Trace not yet indexed" + +# Step 4: Create a group +echo "Step 4: Creating trace group: $GROUP_NAME" +GROUP_ARN=$(aws xray create-group --group-name "$GROUP_NAME" \ + --filter-expression 'service("tutorial-service")' \ + --query 'Group.GroupARN' --output text) +echo " Group ARN: $GROUP_ARN" + +# Step 5: Get service graph +echo "Step 5: Getting service graph" +aws xray get-service-graph \ + --start-time "$START_TIME" --end-time "$END_TIME" \ + --query 'Services[].{Name:Name,Type:Type,Edges:Edges|length(@)}' --output table 2>/dev/null || \ + echo " No service graph available yet" + +echo "" +echo "Tutorial complete." +echo "Do you want to clean up all resources? (y/n): " +read -r CHOICE +if [[ "$CHOICE" =~ ^[Yy]$ ]]; then + cleanup +else + echo "Manual cleanup:" + echo " aws xray delete-group --group-arn $GROUP_ARN" +fi diff --git a/tuts/119-aws-glue-gs/README.md b/tuts/119-aws-glue-gs/README.md new file mode 100644 index 00000000..42536841 --- /dev/null +++ b/tuts/119-aws-glue-gs/README.md @@ -0,0 +1,58 @@ +# Glue: Create a Data Catalog database and table + +Create an AWS Glue Data Catalog database, upload sample JSON data to S3, create an external table pointing to the data, and query the catalog. + +## Source + +https://docs.aws.amazon.com/glue/latest/dg/getting-started.html + +## Use case + +- **ID**: glue/getting-started +- **Level**: beginner +- **Core actions**: `glue:CreateDatabase`, `glue:CreateTable`, `glue:GetTable`, `glue:GetDatabases`, `glue:GetTables` + +## Steps + +1. Create a Glue database +2. Create an S3 bucket and upload sample data +3. Create an external table with JSON SerDe +4. Describe the table +5. List databases and tables + +## Resources created + +| Resource | Type | +|----------|------| +| `tut_db_` | Glue database | +| `tut_events` | Glue table | +| `glue-tut--` | S3 bucket (with sample data) | + +## Duration + +~12 seconds + +## Cost + +No charge. The Glue Data Catalog provides a free tier of 1 million objects stored and 1 million requests per month. + +## Related docs + +- [Getting started with AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/getting-started.html) +- [Defining databases in the Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/define-database.html) +- [Defining tables in the Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html) +- [AWS Glue pricing](https://aws.amazon.com/glue/pricing/) + +--- + +## Appendix + +| Field | Value | +|-------|-------| +| Date | 2026-04-14 | +| Script lines | 104 | +| Exit code | 0 | +| Runtime | 12s | +| Steps | 5 | +| Issues | None | +| Version | v1 | diff --git a/tuts/119-aws-glue-gs/REVISION-HISTORY.md b/tuts/119-aws-glue-gs/REVISION-HISTORY.md new file mode 100644 index 00000000..ec6c87b8 --- /dev/null +++ b/tuts/119-aws-glue-gs/REVISION-HISTORY.md @@ -0,0 +1,8 @@ +# Revision History: 119-aws-glue-gs + +## Shell (CLI script) + +### 2026-04-14 v1 published +- Type: functional +- Initial version + diff --git a/tuts/119-aws-glue-gs/aws-glue-gs.md b/tuts/119-aws-glue-gs/aws-glue-gs.md new file mode 100644 index 00000000..9e2f5d81 --- /dev/null +++ b/tuts/119-aws-glue-gs/aws-glue-gs.md @@ -0,0 +1,124 @@ +# Create a Data Catalog database and table with AWS Glue + +## Overview + +In this tutorial, you use the AWS CLI to create an AWS Glue Data Catalog database, upload sample JSON event data to S3, create an external table that points to the data, and query the catalog. You then delete all resources during cleanup. + +## Prerequisites + +- AWS CLI installed and configured with appropriate permissions. +- An IAM principal with permissions for `glue:CreateDatabase`, `glue:CreateTable`, `glue:GetTable`, `glue:GetDatabases`, `glue:GetTables`, `glue:DeleteTable`, `glue:DeleteDatabase`, `s3:CreateBucket`, `s3:PutObject`, `s3:DeleteObject`, `s3:DeleteBucket`, and `sts:GetCallerIdentity`. + +## Step 1: Create a database + +Create a Glue Data Catalog database to hold table definitions. + +```bash +RANDOM_ID=$(openssl rand -hex 4) +DB_NAME="tut_db_${RANDOM_ID}" + +aws glue create-database \ + --database-input "{\"Name\":\"$DB_NAME\",\"Description\":\"Tutorial database for event data\"}" +echo "Database: $DB_NAME" +``` + +A Glue database is a logical container for tables. It stores metadata only — the actual data lives in S3 or another data store. + +## Step 2: Create an S3 bucket and upload sample data + +Create a bucket and upload sample JSON Lines event data. + +```bash +ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text) +BUCKET_NAME="glue-tut-${RANDOM_ID}-${ACCOUNT_ID}" + +aws s3api create-bucket --bucket "$BUCKET_NAME" + +cat > /tmp/events.jsonl << 'EOF' +{"event_id":"e001","event_type":"click","timestamp":"2026-04-14T10:00:00Z","user_id":"u123"} +{"event_id":"e002","event_type":"view","timestamp":"2026-04-14T10:01:00Z","user_id":"u456"} +{"event_id":"e003","event_type":"purchase","timestamp":"2026-04-14T10:02:00Z","user_id":"u123"} +EOF + +aws s3 cp /tmp/events.jsonl "s3://$BUCKET_NAME/events/events.jsonl" --quiet +``` + +For regions other than `us-east-1`, add `--create-bucket-configuration LocationConstraint=$REGION`. + +## Step 3: Create a table + +Create an external table that maps the JSON fields to columns using the JSON SerDe. + +```bash +TABLE_NAME="tut_events" + +aws glue create-table --database-name "$DB_NAME" --table-input "{ + \"Name\":\"$TABLE_NAME\", + \"StorageDescriptor\":{ + \"Columns\":[ + {\"Name\":\"event_id\",\"Type\":\"string\"}, + {\"Name\":\"event_type\",\"Type\":\"string\"}, + {\"Name\":\"timestamp\",\"Type\":\"string\"}, + {\"Name\":\"user_id\",\"Type\":\"string\"} + ], + \"Location\":\"s3://$BUCKET_NAME/events/\", + \"InputFormat\":\"org.apache.hadoop.mapred.TextInputFormat\", + \"OutputFormat\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\", + \"SerdeInfo\":{\"SerializationLibrary\":\"org.openx.data.jsonserde.JsonSerDe\"} + }, + \"TableType\":\"EXTERNAL_TABLE\" +}" +echo "Table: $TABLE_NAME" +``` + +`EXTERNAL_TABLE` means Glue doesn't manage the data lifecycle — deleting the table doesn't delete the S3 data. The JSON SerDe deserializes each line as a JSON object. + +## Step 4: Describe the table + +View the table metadata including column definitions and data location. + +```bash +aws glue get-table --database-name "$DB_NAME" --name "$TABLE_NAME" \ + --query 'Table.{Name:Name,Database:DatabaseName,Location:StorageDescriptor.Location,Columns:StorageDescriptor.Columns|length(@)}' \ + --output table +``` + +## Step 5: List databases and tables + +List databases and tables in the catalog. + +```bash +aws glue get-databases \ + --query 'DatabaseList[?starts_with(Name, `tut_`)].{Name:Name,Description:Description}' \ + --output table + +aws glue get-tables --database-name "$DB_NAME" \ + --query 'TableList[].{Name:Name,Type:TableType,Columns:StorageDescriptor.Columns|length(@)}' \ + --output table +``` + +## Cleanup + +Delete the table, database, and S3 bucket. + +```bash +aws glue delete-table --database-name "$DB_NAME" --name "$TABLE_NAME" +aws glue delete-database --name "$DB_NAME" +aws s3 rm "s3://$BUCKET_NAME" --recursive --quiet +aws s3 rb "s3://$BUCKET_NAME" +``` + +Delete the table before the database. The database cannot be deleted while it contains tables. + +The script automates all steps including cleanup: + +```bash +bash aws-glue-gs.sh +``` + +## Related resources + +- [Getting started with AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/getting-started.html) +- [Defining databases in the Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/define-database.html) +- [Defining tables in the Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/tables-described.html) +- [AWS Glue pricing](https://aws.amazon.com/glue/pricing/) diff --git a/tuts/119-aws-glue-gs/aws-glue-gs.sh b/tuts/119-aws-glue-gs/aws-glue-gs.sh new file mode 100644 index 00000000..6e365765 --- /dev/null +++ b/tuts/119-aws-glue-gs/aws-glue-gs.sh @@ -0,0 +1,104 @@ +#!/bin/bash +# Tutorial: Create a Glue Data Catalog database and table +# Source: https://docs.aws.amazon.com/glue/latest/dg/getting-started.html + +WORK_DIR=$(mktemp -d) +LOG_FILE="$WORK_DIR/glue-$(date +%Y%m%d-%H%M%S).log" +exec > >(tee -a "$LOG_FILE") 2>&1 + +REGION=${AWS_DEFAULT_REGION:-${AWS_REGION:-$(aws configure get region 2>/dev/null)}} +if [ -z "$REGION" ]; then + echo "ERROR: No AWS region configured. Set one with: export AWS_DEFAULT_REGION=us-east-1" + exit 1 +fi +export AWS_DEFAULT_REGION="$REGION" +ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text) +echo "Region: $REGION" + +RANDOM_ID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1) +DB_NAME="tut_db_${RANDOM_ID}" +TABLE_NAME="tut_events" +BUCKET_NAME="glue-tut-${RANDOM_ID}-${ACCOUNT_ID}" + +handle_error() { echo "ERROR on line $1"; trap - ERR; cleanup; exit 1; } +trap 'handle_error $LINENO' ERR + +cleanup() { + echo "" + echo "Cleaning up resources..." + aws glue delete-table --database-name "$DB_NAME" --name "$TABLE_NAME" 2>/dev/null && echo " Deleted table $TABLE_NAME" + aws glue delete-database --name "$DB_NAME" 2>/dev/null && echo " Deleted database $DB_NAME" + if aws s3 ls "s3://$BUCKET_NAME" > /dev/null 2>&1; then + aws s3 rm "s3://$BUCKET_NAME" --recursive --quiet 2>/dev/null + aws s3 rb "s3://$BUCKET_NAME" 2>/dev/null && echo " Deleted bucket $BUCKET_NAME" + fi + rm -rf "$WORK_DIR" + echo "Cleanup complete." +} + +# Step 1: Create a database +echo "Step 1: Creating Glue database: $DB_NAME" +aws glue create-database --database-input "{\"Name\":\"$DB_NAME\",\"Description\":\"Tutorial database for event data\"}" +echo " Database created" + +# Step 2: Create an S3 bucket for data +echo "Step 2: Creating S3 bucket for data: $BUCKET_NAME" +if [ "$REGION" = "us-east-1" ]; then + aws s3api create-bucket --bucket "$BUCKET_NAME" > /dev/null +else + aws s3api create-bucket --bucket "$BUCKET_NAME" \ + --create-bucket-configuration LocationConstraint="$REGION" > /dev/null +fi + +# Upload sample data +echo '{"event_id":"e001","event_type":"click","timestamp":"2026-04-14T10:00:00Z","user_id":"u123"} +{"event_id":"e002","event_type":"view","timestamp":"2026-04-14T10:01:00Z","user_id":"u456"} +{"event_id":"e003","event_type":"purchase","timestamp":"2026-04-14T10:02:00Z","user_id":"u123"}' > "$WORK_DIR/events.jsonl" +aws s3 cp "$WORK_DIR/events.jsonl" "s3://$BUCKET_NAME/events/events.jsonl" --quiet +echo " Uploaded sample data (3 events)" + +# Step 3: Create a table +echo "Step 3: Creating table: $TABLE_NAME" +aws glue create-table --database-name "$DB_NAME" --table-input "{ + \"Name\":\"$TABLE_NAME\", + \"StorageDescriptor\":{ + \"Columns\":[ + {\"Name\":\"event_id\",\"Type\":\"string\"}, + {\"Name\":\"event_type\",\"Type\":\"string\"}, + {\"Name\":\"timestamp\",\"Type\":\"string\"}, + {\"Name\":\"user_id\",\"Type\":\"string\"} + ], + \"Location\":\"s3://$BUCKET_NAME/events/\", + \"InputFormat\":\"org.apache.hadoop.mapred.TextInputFormat\", + \"OutputFormat\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\", + \"SerdeInfo\":{\"SerializationLibrary\":\"org.openx.data.jsonserde.JsonSerDe\"} + }, + \"TableType\":\"EXTERNAL_TABLE\" +}" +echo " Table created (JSON SerDe, external)" + +# Step 4: Describe the table +echo "Step 4: Table details" +aws glue get-table --database-name "$DB_NAME" --name "$TABLE_NAME" \ + --query 'Table.{Name:Name,Database:DatabaseName,Location:StorageDescriptor.Location,Columns:StorageDescriptor.Columns|length(@)}' --output table + +# Step 5: List databases and tables +echo "Step 5: Listing databases" +aws glue get-databases --query 'DatabaseList[?starts_with(Name, `tut_`)].{Name:Name,Description:Description}' --output table + +echo "Step 5b: Listing tables in $DB_NAME" +aws glue get-tables --database-name "$DB_NAME" \ + --query 'TableList[].{Name:Name,Type:TableType,Columns:StorageDescriptor.Columns|length(@)}' --output table + +echo "" +echo "Tutorial complete." +echo "Do you want to clean up all resources? (y/n): " +read -r CHOICE +if [[ "$CHOICE" =~ ^[Yy]$ ]]; then + cleanup +else + echo "Manual cleanup:" + echo " aws glue delete-table --database-name $DB_NAME --name $TABLE_NAME" + echo " aws glue delete-database --name $DB_NAME" + echo " aws s3 rm s3://$BUCKET_NAME --recursive && aws s3 rb s3://$BUCKET_NAME" +fi diff --git a/tuts/126-amazon-firehose-gs/README.md b/tuts/126-amazon-firehose-gs/README.md new file mode 100644 index 00000000..dbeb98c1 --- /dev/null +++ b/tuts/126-amazon-firehose-gs/README.md @@ -0,0 +1,44 @@ +# Amazon Firehose Gs + +An AWS CLI tutorial that demonstrates Firehose operations. + +## Running + +```bash +bash amazon-firehose-gs.sh +``` + +To auto-run with cleanup: + +```bash +echo 'y' | bash amazon-firehose-gs.sh +``` + +## What it does + +1. Creating S3 bucket: $BUCKET +2. Creating IAM role +3. Creating delivery stream: $STREAM_NAME +4. Sending records +5. Describing stream + +## Resources created + +- Bucket +- Delivery Stream +- Role +- Record +- Record Batch + +The script prompts you to clean up resources when it finishes. + +## Cost + +Free tier eligible for most operations. Clean up resources after use to avoid charges. + +## Related docs + +- [AWS CLI firehose reference](https://docs.aws.amazon.com/cli/latest/reference/firehose/index.html) +- [AWS CLI iam reference](https://docs.aws.amazon.com/cli/latest/reference/iam/index.html) +- [AWS CLI s3 reference](https://docs.aws.amazon.com/cli/latest/reference/s3/index.html) + diff --git a/tuts/126-amazon-firehose-gs/REVISION-HISTORY.md b/tuts/126-amazon-firehose-gs/REVISION-HISTORY.md new file mode 100644 index 00000000..26596d54 --- /dev/null +++ b/tuts/126-amazon-firehose-gs/REVISION-HISTORY.md @@ -0,0 +1,8 @@ +# Revision History: 126-amazon-firehose-gs + +## Shell (CLI script) + +### 2026-04-14 v1 published +- Type: functional +- Initial version + diff --git a/tuts/126-amazon-firehose-gs/amazon-firehose-gs.md b/tuts/126-amazon-firehose-gs/amazon-firehose-gs.md new file mode 100644 index 00000000..75580ce0 --- /dev/null +++ b/tuts/126-amazon-firehose-gs/amazon-firehose-gs.md @@ -0,0 +1,31 @@ +# Amazon Firehose Gs + +## Prerequisites + +1. AWS CLI installed and configured (`aws configure`) +2. Appropriate IAM permissions for the AWS services used + +## Step 1: Creating S3 bucket: $BUCKET + +The script handles this step automatically. See `amazon-firehose-gs.sh` for the exact CLI commands. + +## Step 2: Creating IAM role + +The script handles this step automatically. See `amazon-firehose-gs.sh` for the exact CLI commands. + +## Step 3: Creating delivery stream: $STREAM_NAME + +The script handles this step automatically. See `amazon-firehose-gs.sh` for the exact CLI commands. + +## Step 4: Sending records + +The script handles this step automatically. See `amazon-firehose-gs.sh` for the exact CLI commands. + +## Step 5: Describing stream + +The script handles this step automatically. See `amazon-firehose-gs.sh` for the exact CLI commands. + +## Cleanup + +The script prompts you to clean up all created resources. If you need to clean up manually, check the script log for the resource names that were created. + diff --git a/tuts/126-amazon-firehose-gs/amazon-firehose-gs.sh b/tuts/126-amazon-firehose-gs/amazon-firehose-gs.sh new file mode 100644 index 00000000..efe597b7 --- /dev/null +++ b/tuts/126-amazon-firehose-gs/amazon-firehose-gs.sh @@ -0,0 +1,37 @@ +#!/bin/bash +WORK_DIR=$(mktemp -d) +exec > >(tee -a "$WORK_DIR/firehose-$(date +%Y%m%d-%H%M%S).log") 2>&1 +REGION=${AWS_DEFAULT_REGION:-${AWS_REGION:-$(aws configure get region 2>/dev/null))} +[ -z "$REGION" ] && echo "ERROR: No region" && exit 1 +export AWS_DEFAULT_REGION="$REGION" +ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text) +echo "Region: $REGION" +RANDOM_ID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1) +STREAM_NAME="tut-firehose-${RANDOM_ID}" +BUCKET="firehose-tut-${RANDOM_ID}-${ACCOUNT}" +ROLE_NAME="firehose-tut-role-${RANDOM_ID}" +handle_error() { echo "ERROR on line $1"; trap - ERR; cleanup; exit 1; } +trap 'handle_error $LINENO' ERR +cleanup() { echo ""; echo "Cleaning up..."; aws firehose delete-delivery-stream --delivery-stream-name "$STREAM_NAME" 2>/dev/null && echo " Deleted stream"; sleep 5; aws iam detach-role-policy --role-name "$ROLE_NAME" --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess 2>/dev/null; aws iam delete-role --role-name "$ROLE_NAME" 2>/dev/null && echo " Deleted role"; if aws s3 ls "s3://$BUCKET" > /dev/null 2>&1; then aws s3 rm "s3://$BUCKET" --recursive --quiet; aws s3 rb "s3://$BUCKET" && echo " Deleted bucket"; fi; rm -rf "$WORK_DIR"; echo "Done."; } +echo "Step 1: Creating S3 bucket: $BUCKET" +if [ "$REGION" = "us-east-1" ]; then aws s3api create-bucket --bucket "$BUCKET" > /dev/null; else aws s3api create-bucket --bucket "$BUCKET" --create-bucket-configuration LocationConstraint="$REGION" > /dev/null; fi +echo "Step 2: Creating IAM role" +ROLE_ARN=$(aws iam create-role --role-name "$ROLE_NAME" --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"firehose.amazonaws.com"},"Action":"sts:AssumeRole"}]}' --query 'Role.Arn' --output text) +aws iam attach-role-policy --role-name "$ROLE_NAME" --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess +echo " Role: $ROLE_ARN" +sleep 10 +echo "Step 3: Creating delivery stream: $STREAM_NAME" +aws firehose create-delivery-stream --delivery-stream-name "$STREAM_NAME" --s3-destination-configuration "{\"RoleARN\":\"$ROLE_ARN\",\"BucketARN\":\"arn:aws:s3:::$BUCKET\",\"BufferingHints\":{\"SizeInMBs\":1,\"IntervalInSeconds\":60}}" --query 'DeliveryStreamARN' --output text +echo " Waiting for stream..." +for i in $(seq 1 20); do STATUS=$(aws firehose describe-delivery-stream --delivery-stream-name "$STREAM_NAME" --query 'DeliveryStreamDescription.DeliveryStreamStatus' --output text); echo " $STATUS"; [ "$STATUS" = "ACTIVE" ] && break; sleep 5; done +echo "Step 4: Sending records" +aws firehose put-record --delivery-stream-name "$STREAM_NAME" --record '{"Data":"SGVsbG8gZnJvbSBGaXJlaG9zZQo="}' --query 'RecordId' --output text > /dev/null +aws firehose put-record-batch --delivery-stream-name "$STREAM_NAME" --records '[{"Data":"UmVjb3JkIDEK"},{"Data":"UmVjb3JkIDIK"}]' > /dev/null +echo " Sent 3 records (1 individual + 2 batch)" +echo "Step 5: Describing stream" +aws firehose describe-delivery-stream --delivery-stream-name "$STREAM_NAME" --query 'DeliveryStreamDescription.{Name:DeliveryStreamName,Status:DeliveryStreamStatus,Destination:Destinations[0].S3DestinationDescription.BucketARN}' --output table +echo "" +echo "Tutorial complete." +echo "Do you want to clean up? (y/n): " +read -r CHOICE +[[ "$CHOICE" =~ ^[Yy]$ ]] && cleanup || echo "Manual: aws firehose delete-delivery-stream --delivery-stream-name $STREAM_NAME" diff --git a/tuts/132-amazon-ivs-gs/README.md b/tuts/132-amazon-ivs-gs/README.md new file mode 100644 index 00000000..68ad1647 --- /dev/null +++ b/tuts/132-amazon-ivs-gs/README.md @@ -0,0 +1,37 @@ +# Amazon Ivs Gs + +An AWS CLI tutorial that demonstrates Ivs operations. + +## Running + +```bash +bash amazon-ivs-gs.sh +``` + +To auto-run with cleanup: + +```bash +echo 'y' | bash amazon-ivs-gs.sh +``` + +## What it does + +1. Creating channel: $CHANNEL_NAME +2. Getting channel details +3. Listing channels +4. Getting stream key + +## Resources created + +- Channel + +The script prompts you to clean up resources when it finishes. + +## Cost + +Free tier eligible for most operations. Clean up resources after use to avoid charges. + +## Related docs + +- [AWS CLI ivs reference](https://docs.aws.amazon.com/cli/latest/reference/ivs/index.html) + diff --git a/tuts/132-amazon-ivs-gs/REVISION-HISTORY.md b/tuts/132-amazon-ivs-gs/REVISION-HISTORY.md new file mode 100644 index 00000000..f9cb977a --- /dev/null +++ b/tuts/132-amazon-ivs-gs/REVISION-HISTORY.md @@ -0,0 +1,8 @@ +# Revision History: 132-amazon-ivs-gs + +## Shell (CLI script) + +### 2026-04-14 v1 published +- Type: functional +- Initial version + diff --git a/tuts/132-amazon-ivs-gs/amazon-ivs-gs.md b/tuts/132-amazon-ivs-gs/amazon-ivs-gs.md new file mode 100644 index 00000000..78445df2 --- /dev/null +++ b/tuts/132-amazon-ivs-gs/amazon-ivs-gs.md @@ -0,0 +1,27 @@ +# Amazon Ivs Gs + +## Prerequisites + +1. AWS CLI installed and configured (`aws configure`) +2. Appropriate IAM permissions for the AWS services used + +## Step 1: Creating channel: $CHANNEL_NAME + +The script handles this step automatically. See `amazon-ivs-gs.sh` for the exact CLI commands. + +## Step 2: Getting channel details + +The script handles this step automatically. See `amazon-ivs-gs.sh` for the exact CLI commands. + +## Step 3: Listing channels + +The script handles this step automatically. See `amazon-ivs-gs.sh` for the exact CLI commands. + +## Step 4: Getting stream key + +The script handles this step automatically. See `amazon-ivs-gs.sh` for the exact CLI commands. + +## Cleanup + +The script prompts you to clean up all created resources. If you need to clean up manually, check the script log for the resource names that were created. + diff --git a/tuts/132-amazon-ivs-gs/amazon-ivs-gs.sh b/tuts/132-amazon-ivs-gs/amazon-ivs-gs.sh new file mode 100644 index 00000000..e84de0cb --- /dev/null +++ b/tuts/132-amazon-ivs-gs/amazon-ivs-gs.sh @@ -0,0 +1,23 @@ +#!/bin/bash +WORK_DIR=$(mktemp -d) +exec > >(tee -a "$WORK_DIR/ivs-$(date +%Y%m%d-%H%M%S).log") 2>&1 +REGION=${AWS_DEFAULT_REGION:-${AWS_REGION:-$(aws configure get region 2>/dev/null))}; [ -z "$REGION" ] && echo "ERROR: No region" && exit 1; export AWS_DEFAULT_REGION="$REGION"; echo "Region: $REGION" +RANDOM_ID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1) +CHANNEL_NAME="tut-channel-${RANDOM_ID}" +handle_error() { echo "ERROR on line $1"; trap - ERR; cleanup; exit 1; }; trap 'handle_error $LINENO' ERR +cleanup() { echo ""; echo "Cleaning up..."; [ -n "$CHANNEL_ARN" ] && aws ivs delete-channel --arn "$CHANNEL_ARN" 2>/dev/null && echo " Deleted channel"; rm -rf "$WORK_DIR"; echo "Done."; } +echo "Step 1: Creating channel: $CHANNEL_NAME" +RESULT=$(aws ivs create-channel --name "$CHANNEL_NAME" --type STANDARD) +CHANNEL_ARN=$(echo "$RESULT" | python3 -c "import sys,json;print(json.load(sys.stdin)['channel']['arn'])") +STREAM_KEY=$(echo "$RESULT" | python3 -c "import sys,json;print(json.load(sys.stdin)['streamKey']['value'])") +echo " Channel ARN: $CHANNEL_ARN" +echo " Stream key: ${STREAM_KEY:0:10}..." +echo "Step 2: Getting channel details" +aws ivs get-channel --arn "$CHANNEL_ARN" --query 'channel.{Name:name,Type:type,Latency:latencyMode,Endpoint:ingestEndpoint}' --output table +echo "Step 3: Listing channels" +aws ivs list-channels --query 'channels[?starts_with(name, `tut-`)].{Name:name,ARN:arn}' --output table +echo "Step 4: Getting stream key" +aws ivs list-stream-keys --channel-arn "$CHANNEL_ARN" --query 'streamKeys[0].arn' --output text > /dev/null +echo " Stream key retrieved (use with OBS or ffmpeg to stream)" +echo ""; echo "Tutorial complete." +echo "Do you want to clean up? (y/n): "; read -r CHOICE; [[ "$CHOICE" =~ ^[Yy]$ ]] && cleanup diff --git a/tuts/137-amazon-location-gs/README.md b/tuts/137-amazon-location-gs/README.md new file mode 100644 index 00000000..bb27a0b9 --- /dev/null +++ b/tuts/137-amazon-location-gs/README.md @@ -0,0 +1,39 @@ +# Amazon Location Gs + +An AWS CLI tutorial that demonstrates Location operations. + +## Running + +```bash +bash amazon-location-gs.sh +``` + +To auto-run with cleanup: + +```bash +echo 'y' | bash amazon-location-gs.sh +``` + +## What it does + +1. Creating map: $MAP_NAME +2. Creating place index: $INDEX_NAME +3. Searching for a place +4. Reverse geocoding +5. Listing maps + +## Resources created + +- Map +- Place Index + +The script prompts you to clean up resources when it finishes. + +## Cost + +Free tier eligible for most operations. Clean up resources after use to avoid charges. + +## Related docs + +- [AWS CLI location reference](https://docs.aws.amazon.com/cli/latest/reference/location/index.html) + diff --git a/tuts/137-amazon-location-gs/REVISION-HISTORY.md b/tuts/137-amazon-location-gs/REVISION-HISTORY.md new file mode 100644 index 00000000..9675961e --- /dev/null +++ b/tuts/137-amazon-location-gs/REVISION-HISTORY.md @@ -0,0 +1,8 @@ +# Revision History: 137-amazon-location-gs + +## Shell (CLI script) + +### 2026-04-14 v1 published +- Type: functional +- Initial version + diff --git a/tuts/137-amazon-location-gs/amazon-location-gs.md b/tuts/137-amazon-location-gs/amazon-location-gs.md new file mode 100644 index 00000000..ac960b43 --- /dev/null +++ b/tuts/137-amazon-location-gs/amazon-location-gs.md @@ -0,0 +1,31 @@ +# Amazon Location Gs + +## Prerequisites + +1. AWS CLI installed and configured (`aws configure`) +2. Appropriate IAM permissions for the AWS services used + +## Step 1: Creating map: $MAP_NAME + +The script handles this step automatically. See `amazon-location-gs.sh` for the exact CLI commands. + +## Step 2: Creating place index: $INDEX_NAME + +The script handles this step automatically. See `amazon-location-gs.sh` for the exact CLI commands. + +## Step 3: Searching for a place + +The script handles this step automatically. See `amazon-location-gs.sh` for the exact CLI commands. + +## Step 4: Reverse geocoding + +The script handles this step automatically. See `amazon-location-gs.sh` for the exact CLI commands. + +## Step 5: Listing maps + +The script handles this step automatically. See `amazon-location-gs.sh` for the exact CLI commands. + +## Cleanup + +The script prompts you to clean up all created resources. If you need to clean up manually, check the script log for the resource names that were created. + diff --git a/tuts/137-amazon-location-gs/amazon-location-gs.sh b/tuts/137-amazon-location-gs/amazon-location-gs.sh new file mode 100644 index 00000000..91297e41 --- /dev/null +++ b/tuts/137-amazon-location-gs/amazon-location-gs.sh @@ -0,0 +1,18 @@ +#!/bin/bash +WORK_DIR=$(mktemp -d); exec > >(tee -a "$WORK_DIR/location.log") 2>&1 +REGION=${AWS_DEFAULT_REGION:-${AWS_REGION:-$(aws configure get region 2>/dev/null))}; [ -z "$REGION" ] && echo "ERROR: No region" && exit 1; export AWS_DEFAULT_REGION="$REGION"; echo "Region: $REGION" +RANDOM_ID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1); MAP_NAME="tut-map-${RANDOM_ID}"; INDEX_NAME="tut-index-${RANDOM_ID}" +handle_error() { echo "ERROR on line $1"; trap - ERR; cleanup; exit 1; }; trap 'handle_error $LINENO' ERR +cleanup() { echo ""; echo "Cleaning up..."; aws location delete-map --map-name "$MAP_NAME" 2>/dev/null && echo " Deleted map"; aws location delete-place-index --index-name "$INDEX_NAME" 2>/dev/null && echo " Deleted place index"; rm -rf "$WORK_DIR"; echo "Done."; } +echo "Step 1: Creating map: $MAP_NAME" +aws location create-map --map-name "$MAP_NAME" --configuration '{"Style":"VectorEsriStreets"}' --query 'MapArn' --output text +echo "Step 2: Creating place index: $INDEX_NAME" +aws location create-place-index --index-name "$INDEX_NAME" --data-source Here --query 'IndexArn' --output text +echo "Step 3: Searching for a place" +aws location search-place-index-for-text --index-name "$INDEX_NAME" --text "Seattle" --query 'Results[:3].{Label:Place.Label,Lat:Place.Geometry.Point[1],Lon:Place.Geometry.Point[0]}' --output table +echo "Step 4: Reverse geocoding" +aws location search-place-index-for-position --index-name "$INDEX_NAME" --position '[-122.3321,47.6062]' --query 'Results[0].Place.{Label:Label,Country:Country}' --output table +echo "Step 5: Listing maps" +aws location list-maps --query 'Entries[?starts_with(MapName, `tut-`)].{Name:MapName,DataSource:DataSource}' --output table +echo ""; echo "Tutorial complete." +echo "Do you want to clean up? (y/n): "; read -r CHOICE; [[ "$CHOICE" =~ ^[Yy]$ ]] && cleanup