Lab — Migrate Local State to S3 and DynamoDB with Locking — Infrastructure as Code

What You'll Build

This lab walks you through one of the most common real-world Terraform operations: migrating an existing Terraform configuration from local state (terraform.tfstate on disk) to a remote S3 backend with DynamoDB locking. You will start with the EC2 configuration from M1 Lesson 7 (or a simple configuration you write), verify it has local state with resources deployed, bootstrap the S3 + DynamoDB backend infrastructure, reconfigure the Terraform backend in the configuration, run 'terraform init -migrate-state', verify the migration was successful, and simulate a state lock to understand the locking mechanism. This migration scenario is one that every Terraform engineer encounters when a solo project grows into a team project or when a project is adopted by a platform team that requires standardised backend configuration.

Analogy🏏Cricket

🏏 Think of it like cricket: You are designing the BCCI's standard cricket ground specification module — a parameterisable blueprint that can be instantiated for any venue in the country without redrawing it from scratch. The blueprint fixes the non-negotiables every certified ground shares: a boundary perimeter (the VPC and its CIDR), spectator zones with public access (public subnets with an Internet Gateway route), and restricted player-and-officials zones behind accreditation checks (private subnets routing through NAT). What varies per venue arrives as parameters: how many stands to build (var.azs and subnet counts), whether this is a full international stadium or a modest district ground (var.single_nat_gateway as the cost lever — one shared service corridor instead of one per stand), and the venue's name and signage (tags). Wankhede and a Ranchi district ground are instantiated from the same drawing with different inputs — and when the board later improves the blueprint (adds flow logs, tightens NACLs), every venue inherits the improvement on its next renovation (module version bump) instead of each ground hand-patching its own architecture. That is the entire economics of module authorship: design once, rigorously, then stamp out consistent grounds forever.

Prerequisites

AWS CLI configured with permissions for S3, DynamoDB and EC2
Terraform 1.6+ installed
An existing Terraform configuration with local state (complete the M1 L7 lab first, or use the simple configuration in Step 0 below)
The existing local state should have at least one resource deployed so the migration has something to transfer

Step 0 — Create a Starting Configuration with Local State

If you completed the M1 Lab 7 exercise, use that configuration. Otherwise, create a minimal starting configuration that provisions a single S3 bucket with local state — the resource itself is unimportant; what matters is that Terraform has created a local terraform.tfstate file with a tracked resource.

Analogy🏏Cricket

🏏 Think of it like cricket: before a club can move its scoring to the stadium's official score room, there has to be a scorebook with something written in it. This step is a net session with the scorer's book kept in your own kit bag: you play one simple stroke — provisioning a single S3 bucket — purely so the scorer records a real entry. Just as it does not matter whether the practice shot was a drive or a cut, the resource itself is unimportant; what matters is that Terraform has produced a local terraform.tfstate file with a tracked resource in it, the way the scorebook now holds a genuine, verifiable record. Just as you cannot rehearse handing the scorebook over to the official score room if its pages are blank, you cannot practise a state migration without an existing local state to migrate. The payoff is a safe, throwaway starting point: a real state file whose journey to S3 you can watch end-to-end without risking anything that matters.

bash

#!/bin/bash
# Step 0: Create a simple configuration with local state
mkdir -p ~/state_migration_lab && cd ~/state_migration_lab

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

cat > main.tf << 'HCL'
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
  # NOTE: No backend block — uses local state by default
}

provider "aws" { region = "ap-south-1" }

# Simple resource to track in state
resource "aws_s3_bucket" "cricket_lab" {
  bucket = "cricket-state-migration-lab-${data.aws_caller_identity.current.account_id}"
  tags   = { Purpose = "state-migration-lab", ManagedBy = "terraform" }
}

data "aws_caller_identity" "current" {}

output "bucket_name" {
  value = aws_s3_bucket.cricket_lab.bucket
}
HCL

echo '=== Apply with local state ==='
terraform init
terraform apply -auto-approve

echo
echo '=== Verify local state exists ==='
ls -la terraform.tfstate
cat terraform.tfstate | python3 -m json.tool | grep -E 'lineage|serial|bucket'

echo
echo 'Local state confirmed — ready for migration'

Step 1 — Bootstrap the Remote Backend Infrastructure

Before migrating, the S3 bucket and DynamoDB table that will host the remote state must exist. This bootstrapping step uses the AWS CLI rather than Terraform to avoid the chicken-and-egg problem. All security settings (versioning, encryption, public access block) are applied during bootstrap so the state file is secure from the first write.

Analogy🏏Cricket

🏏 Think of it like cricket: before the match record can move into the stadium, the stadium's score room must be built — and you cannot use match-day scoring rules to build the room that scoring depends on. That is the chicken-and-egg problem: Terraform cannot store its state in an S3 bucket that Terraform itself has not yet been able to record, so the groundstaff — the AWS CLI — construct the S3 bucket and DynamoDB lock table directly. Just as a well-run ground installs the locks, the fire-safe cabinet and the duplicate-scorebook procedure before the first ball is bowled, versioning, encryption and the public access block are applied during bootstrap so the state file is protected from its very first write, not retrofitted after an exposure. And just as the score room's booking board stops two scorers overwriting each other's entries, the DynamoDB table exists to hold locks before any migration begins. The payoff: when the state arrives, it lands in a venue that was secure from day one.

bash

#!/bin/bash
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION='ap-south-1'
STATE_BUCKET="cricket-terraform-state-${ACCOUNT_ID}"
LOCK_TABLE='cricket-terraform-locks'

echo '=== Bootstrap: S3 state bucket ==='

# Create bucket
aws s3api create-bucket \
  --bucket "$STATE_BUCKET" \
  --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION" 2>/dev/null || \
  echo 'Bucket already exists'

# Block all public access
aws s3api put-public-access-block \
  --bucket "$STATE_BUCKET" \
  --public-access-block-configuration \
  'BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true'

# Enable versioning — critical for state recovery
aws s3api put-bucket-versioning \
  --bucket "$STATE_BUCKET" \
  --versioning-configuration Status=Enabled

# Enable encryption
aws s3api put-bucket-encryption \
  --bucket "$STATE_BUCKET" \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

echo "S3 bucket: s3://${STATE_BUCKET}"

echo
echo '=== Bootstrap: DynamoDB lock table ==='
aws dynamodb create-table \
  --table-name "$LOCK_TABLE" \
  --billing-mode PAY_PER_REQUEST \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --region "$REGION" 2>/dev/null || echo 'Table already exists'

# Wait for table to be active
aws dynamodb wait table-exists --table-name "$LOCK_TABLE" --region "$REGION"
echo "DynamoDB table: ${LOCK_TABLE} — ACTIVE"

echo
echo "Bootstrap complete. State bucket: ${STATE_BUCKET}"
export STATE_BUCKET LOCK_TABLE

Step 2 — Add Backend Configuration and Migrate

Update the Terraform configuration to declare the S3 backend, then run 'terraform init -migrate-state'. Terraform detects the existing local state file, prompts to copy it to the new backend, and verifies the migration. The existing resources remain unchanged in AWS — only the state's storage location changes.

Analogy🏏Cricket

🏏 Think of it like cricket: this is the moment the scorer carries the club's scorebook from a kit bag into the stadium's official score room. First you photocopy the book — backing up terraform.tfstate — because no sensible scorer surrenders the only copy of a match record. Then you tell the club where scoring now lives by declaring the S3 backend, and 'terraform init -migrate-state' acts like the match referee: it notices the existing local book, asks you to confirm the handover, copies every entry to the new room, and verifies the copy is faithful before the old book is retired. Crucially, just as moving the scorebook changes nothing on the field — the batters keep batting, the score itself is untouched — the migration leaves every AWS resource exactly as it was; only the state's storage location changes. The payoff is a record that now lives where the whole team can read it, with locking to stop two scorers writing at once, achieved without interrupting a single ball of play.

bash

#!/bin/bash
cd ~/state_migration_lab

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
STATE_BUCKET="cricket-terraform-state-${ACCOUNT_ID}"

echo '=== Step 2a: Back up local state before migration ==='
cp terraform.tfstate "terraform.tfstate.backup.$(date +%Y%m%d_%H%M%S)"
echo 'Local state backed up'

echo
echo '=== Step 2b: Add S3 backend to versions.tf ==='
# Update main.tf to include the backend block
cat > main.tf << HCLDOC
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }

  # Remote backend — replacing local state
  backend "s3" {
    bucket         = "${STATE_BUCKET}"
    key            = "cricket-analytics/lab/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "cricket-terraform-locks"
    encrypt        = true
  }
}

provider "aws" { region = "ap-south-1" }

resource "aws_s3_bucket" "cricket_lab" {
  bucket = "cricket-state-migration-lab-\${data.aws_caller_identity.current.account_id}"
  tags   = { Purpose = "state-migration-lab", ManagedBy = "terraform" }
}

data "aws_caller_identity" "current" {}

output "bucket_name" {
  value = aws_s3_bucket.cricket_lab.bucket
}
HCLDOC

echo
echo '=== Step 2c: Migrate state — the key command ==='
# -migrate-state: Terraform detects existing local state and offers to copy it
terraform init -migrate-state
# Terraform will prompt:
# 'Do you want to copy existing state to the new backend?'
# Type: yes

echo
echo '=== Step 2d: Verify migration success ==='

# 1. Remote state should show the same resources as the local state did
terraform state list

# 2. Plan should show NO changes (resources are unchanged in AWS)
terraform plan
# Expected: No changes. Infrastructure is up-to-date.

# 3. Verify state file exists in S3
aws s3 ls "s3://${STATE_BUCKET}/cricket-analytics/lab/"

# 4. Verify state content matches
terraform state pull | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Resources: {len(d[\"resources\"])}, Serial: {d[\"serial\"]}')"

echo 'Migration verified — state is now in S3'

Step 3 — Simulate State Locking

Understand DynamoDB locking by examining what happens when a lock is held. This simulation shows the lock record structure and demonstrates the error a concurrent apply would receive. This understanding is critical for on-call scenarios where a lock must be force-released after a crashed Terraform process.

bash

#!/bin/bash
cd ~/state_migration_lab

echo '=== Step 3: Understand state locking ==='

# Start an apply in the background (it will complete quickly since no changes are needed)
# We examine the DynamoDB lock while it runs
(terraform apply -auto-approve &)
AP_PID=$!

# Immediately check for lock record in DynamoDB
sleep 2
LOCK_TABLE='cricket-terraform-locks'
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

echo 'Checking DynamoDB for lock record:'
aws dynamodb scan \
  --table-name "$LOCK_TABLE" \
  --query 'Items' \
  --output json | python3 -m json.tool

# Wait for the apply to complete
wait $AP_PID 2>/dev/null || true

# After apply completes, lock should be released (no items in table)
sleep 2
echo
echo 'After apply completes — lock should be released:'
LOCK_COUNT=$(aws dynamodb scan --table-name "$LOCK_TABLE" --query 'Count' --output text)
echo "Active locks: ${LOCK_COUNT}"

echo
echo '=== Simulate a stuck lock ==='
# Manually write a fake lock to DynamoDB
LOCK_ID=$(cat /proc/sys/kernel/random/uuid 2>/dev/null || uuidgen)
STATE_KEY="cricket-analytics/lab/terraform.tfstate"

aws dynamodb put-item \
  --table-name "$LOCK_TABLE" \
  --item "{
    \"LockID\": {\"S\": \"${STATE_BUCKET}/${STATE_KEY}\"},
    \"Info\": {\"S\": \"{\\\"ID\\\":\\\"${LOCK_ID}\\\",\\\"Operation\\\":\\\"OperationTypeApply\\\",\\\"Who\\\":\\\"nagarajarao@cricket-laptop\\\",\\\"Created\\\":\\\"2024-04-15T10:00:00.000Z\\\"}\"}
  }"

echo 'Fake lock written. Attempting terraform plan:'
terraform plan 2>&1 | grep -A10 'Error.*lock\|lock.*Error' | head -15

echo
echo '=== Force-unlock (only safe when original process is confirmed dead) ==='
terraform force-unlock -force "$LOCK_ID"
echo 'Lock released. Terraform plan should now succeed:'
terraform plan 2>&1 | tail -3

Analogy🏏Cricket

🏏 Think of it like cricket: The DynamoDB lock you just examined is the rule that only one scorer may hold the official pen at a time. When a scorer sits down to record an over (terraform apply starts), they first sign the pen out of the register — writing their name, the time, and which match they are scoring (the lock record's ID, operation, timestamp and who fields you saw in the LockID item). Any second scorer arriving mid-over does not get a second pen and a second book; they get shown the register entry — 'Error acquiring the state lock: locked by <name> at <time>' — and must wait. This is what prevents the catastrophic double-entry problem where two scorers each record a different delivery as ball three of the over (two applies interleaving writes and corrupting state). The force-unlock command is the match referee's override for when a scorer collapses mid-innings without signing the pen back in (a CI job killed mid-apply): the referee verifies the scorer is genuinely gone — not just slow — before breaking the register entry, because force-unlocking while the original apply is actually still running recreates the exact double-pen disaster the register exists to prevent.

Step 4 — State Versioning Recovery Test

bash

#!/bin/bash
cd ~/state_migration_lab

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
STATE_BUCKET="cricket-terraform-state-${ACCOUNT_ID}"
STATE_KEY='cricket-analytics/lab/terraform.tfstate'

echo '=== Step 4: Verify S3 versioning enables recovery ==='

# Show all state versions in S3
echo 'S3 state file versions:'
aws s3api list-object-versions \
  --bucket "$STATE_BUCKET" \
  --prefix "$STATE_KEY" \
  --query 'Versions[*].{Version:VersionId, Modified:LastModified, Size:Size}' \
  --output table

# Download the most recent version
terraform state pull > /tmp/current_state.json
python3 -c "import json; d=json.load(open('/tmp/current_state.json')); print(f'Current serial: {d[\"serial\"]}  Resources: {len(d[\"resources\"])}')"

# Simulate corruption by modifying the state serial
python3 -c "
import json
with open('/tmp/current_state.json') as f:
    d = json.load(f)
print(f'Original serial: {d[\"serial\"]}  Lineage: {d[\"lineage\"]}')
"

# Recovery procedure:
echo
echo 'Recovery procedure (if state is corrupted):'
echo '  1. Identify the last-known-good version ID from S3:'
echo '     aws s3api list-object-versions --bucket BUCKET --prefix KEY'
echo '  2. Download the version:'
echo '     aws s3api get-object --bucket BUCKET --key KEY --version-id VERSION_ID recovered_state.json'
echo '  3. Restore using state push:'
echo '     terraform state push recovered_state.json'
echo '  4. Verify with terraform plan (should show no changes if recovery was correct)'

echo
echo '=== Cleanup ==='
echo 'Resources to clean up:'
echo '  1. Lab S3 bucket: terraform destroy -auto-approve'
terraform destroy -auto-approve
echo '  2. State bucket and DynamoDB (manually — state infrastructure persists):'
echo "     aws s3 rb s3://${STATE_BUCKET} --force"
echo "     aws dynamodb delete-table --table-name cricket-terraform-locks"

Warning: After running 'terraform init -migrate-state', do NOT delete the local terraform.tfstate file immediately. Keep it as a backup for at least 24 hours while verifying the remote state is correct and all team members can successfully run 'terraform plan' from the remote state. Only after all team members have confirmed that 'terraform plan' shows no changes (meaning the remote state matches the actual infrastructure) should the local state backup be deleted. The migration is a copy, not a move — both local and remote state contain identical content after migration, providing a recovery path if something goes wrong in the first day of remote state usage.

The state migration procedure is: backup local state → add backend block to configuration → run 'terraform init -migrate-state' → verify with 'terraform plan' showing no changes → verify state in S3 → verify DynamoDB lock table.
Bootstrap the S3 bucket and DynamoDB table with the AWS CLI before running Terraform init — the state backend infrastructure cannot be managed by the same Terraform configuration that uses it.
S3 object versioning on the state bucket enables recovery from accidental state overwrites — list versions with 'aws s3api list-object-versions', download a specific version, and restore with 'terraform state push'.
DynamoDB lock records contain the lock ID, operation type, who acquired the lock, and when — use this information to verify that a lock is truly stale before force-unlocking, and only force-unlock after confirming the original process has terminated.
After migration, keep the local terraform.tfstate as a backup for at least 24 hours — only delete it after verifying that 'terraform plan' from the remote state shows no changes for all team members.
The state bucket key should be structured as 'project/environment/component/terraform.tfstate' — this organises multiple configurations in the same bucket with self-documenting paths that make state files easy to locate during incident response.