Lab — S3 cross-region replication and Route 53 DNS failover drill

This lab implements S3 cross-region replication from ap-south-1 (Mumbai) to ap-southeast-1 (Singapore) for the IPL match data archive, and then conducts a Route 53 DNS failover drill that simulates a regional outage in Mumbai and verifies that DNS traffic routes to the Singapore secondary endpoint. The lab validates two components of the Backup and Restore DR strategy: that data is continuously replicated to the DR region, and that DNS failover responds within the target RTO when the primary health check fails.

Analogy🏏Cricket

🏏 Think of it like cricket: In Test cricket, the ICC publishes playing conditions — governing over rates, DRS quotas, pitch inspection protocols, and player conduct — that both captains sign before the first session, whether the match is at Lord’s, the MCG, or Eden Gardens. Just as the playing conditions give umpires a single authoritative standard so every ruling references the same document rather than personal judgement, the Well-Architected Framework gives architects a shared evaluation language so every workload is measured against the same six pillars rather than each engineer’s intuition. Just as a team posting a slow over rate incurs penalties regardless of their score, a workload with Security or Reliability gaps carries structural risk regardless of how quickly it shipped. Just as every specialist role — opener, keeper, tail — has defined performance expectations against which selectors evaluate each player, every workload component is evaluated against pillar-specific best-practice questions. This reveals why the framework must precede any advanced architectural decision: a shared, evidence-based standard transforms subjective trade-offs into structured, auditable risk assessments that hold across teams, accounts, and regions.

The lab uses Boto3 throughout rather than the AWS Console, because all DR procedures must be automatable and repeatable. The S3 replication configuration and the Route 53 health check and routing records are the same resources that would be deployed by the IaC template in a production landing zone. Implementing them manually in the lab builds the understanding needed to review and validate IaC-generated configurations during architecture reviews.

Prerequisites: AWS CLI configured with an IAM user or role that has s3:*, iam:CreateRole, iam:PutRolePolicy, route53:*, and cloudwatch:* permissions. A Route 53 hosted zone for a test domain is required for the DNS failover drill. Two test ALB or EC2 endpoints in ap-south-1 and ap-southeast-1 represent primary and secondary application endpoints. Python and Boto3 must be installed; verify with `python3 -c "import boto3; print(boto3.__version__)"`.

Prerequisites

AWS CLI configured with an IAM user or role that has s3:*, iam:CreateRole, iam:PutRolePolicy, route53:*, and cloudwatch:* permissions.
Route 53 hosted zone for a test domain (can use a subdomain of an existing domain).
Two test ALB or EC2 endpoints in ap-south-1 and ap-southeast-1 to represent primary and secondary application endpoints for the Route 53 failover test.
Python and Boto3 installed; verify with `python3 -c "import boto3; print(boto3.__version__)"`.

Step 1 — S3 Cross-Region Replication

Create source and destination S3 buckets with versioning enabled, configure the IAM replication role, and enable cross-region replication from Mumbai to Singapore. Versioning is required on both buckets before replication can be enabled. The replication rule can filter by prefix, enabling selective replication of only the match data prefix rather than all bucket content. Upload a test object to the source bucket and verify it appears in the destination bucket within 15 minutes.

Analogy🏏Cricket

🏏 Think of it like cricket: Setting up a live duplicate of the official match records in a second city requires a few prerequisites in order — both record rooms must keep dated versions of every document, an accredited courier must be authorised to carry copies, and you specify that only the match-data files travel, not every scrap of paper. Just as versioned records are the precondition for reliable copying, this step enables versioning on both the source and destination buckets before replication can be turned on. Just as the courier is granted a specific mandate rather than free run of the archive, you create an IAM replication role scoped to read from Mumbai and write to Singapore. Just as you replicate only the official match files by tagging them, the replication rule filters by the match-data/ prefix rather than the whole bucket. Just as you confirm the courier system works by sending one document and checking it arrives, you upload a test object and verify it appears in Singapore within 15 minutes. The payoff: continuous, selective, authorised replication of exactly the DR-critical data to the secondary region.

python

import boto3, json, time

s3_mum = boto3.client('s3', region_name='ap-south-1')     # Mumbai (source)
s3_sin = boto3.client('s3', region_name='ap-southeast-1') # Singapore (destination)
iam    = boto3.client('iam')

SOURCE_BUCKET = 'ipl-match-data-ap-south-1'
DEST_BUCKET   = 'ipl-match-data-ap-southeast-1'
ACCOUNT_ID    = boto3.client('sts').get_caller_identity()['Account']

# Create source and destination buckets with versioning enabled.
for bucket, client, region in [(SOURCE_BUCKET, s3_mum, 'ap-south-1'),
                                 (DEST_BUCKET,   s3_sin, 'ap-southeast-1')]:
    client.create_bucket(
        Bucket=bucket,
        CreateBucketConfiguration={'LocationConstraint': region},
    )
    client.put_bucket_versioning(
        Bucket=bucket,
        VersioningConfiguration={'Status': 'Enabled'},
    )
    print(f'Bucket created with versioning: {bucket}')

# Create IAM role for S3 replication.
replication_role = iam.create_role(
    RoleName='ipl-s3-replication-role',
    AssumeRolePolicyDocument=json.dumps({'Version':'2012-10-17','Statement':[{
        'Effect':'Allow','Principal':{'Service':'s3.amazonaws.com'},
        'Action':'sts:AssumeRole',
    }]}),
)['Role']['Arn']

iam.put_role_policy(
    RoleName='ipl-s3-replication-role',
    PolicyName='ipl-s3-replication-policy',
    PolicyDocument=json.dumps({'Version':'2012-10-17','Statement':[
        {'Effect':'Allow','Action':['s3:GetObjectVersionForReplication',
                                    's3:GetObjectVersionAcl',
                                    's3:GetObjectVersionTagging',
                                    's3:ListBucket','s3:GetReplicationConfiguration'],
         'Resource':[f'arn:aws:s3:::{SOURCE_BUCKET}',f'arn:aws:s3:::{SOURCE_BUCKET}/*']},
        {'Effect':'Allow','Action':['s3:ReplicateObject','s3:ReplicateDelete',
                                    's3:ReplicateTags'],
         'Resource':f'arn:aws:s3:::{DEST_BUCKET}/*'},
    ]}),
)

# Configure cross-region replication.
s3_mum.put_bucket_replication(
    Bucket=SOURCE_BUCKET,
    ReplicationConfiguration={
        'Role': replication_role,
        'Rules': [{
            'ID':     'ipl-match-data-replication',
            'Status': 'Enabled',
            'Filter': {'Prefix': 'match-data/'},  # replicate only match-data/ prefix
            'Destination': {'Bucket': f'arn:aws:s3:::{DEST_BUCKET}'},
        }],
    },
)
print(f'Replication: {SOURCE_BUCKET} → {DEST_BUCKET} for prefix match-data/')

Step 2 — Verify Replication

Upload a test IPL match data file to the source bucket under the match-data/ prefix and wait for it to appear in the destination bucket. S3 cross-region replication is asynchronous with a typical replication time of under 15 minutes for objects under 100 MB. S3 Replication Time Control (RTC) provides a 15-minute SLA with 99.99% of objects replicated within 15 minutes. Verify replication by listing the destination bucket after 15 minutes and confirming the test object is present with the same checksum.

Analogy🏏Cricket

🏏 Think of it like cricket: Sending a copy of the scorecard to the second city is not instant — the courier travels asynchronously, so you confirm arrival rather than assume it, and you check the copy matches the original letter for letter. Just as you would not trust that the duplicate reached Singapore without physically confirming it, this step uploads a test match-data file and waits, because S3 cross-region replication is asynchronous — typically under 15 minutes for objects below 100MB. Just as a formal courier contract with a guaranteed delivery window is stronger than 'it usually arrives soon', S3 Replication Time Control commits to a 15-minute SLA replicating 99.99% of objects in time. Just as an official checks the received scorecard's every figure against the original before filing it, you verify replication by listing the destination and confirming the object is present with a matching checksum. The payoff: proving — not assuming — that the copy arrived intact turns a hopeful backup into a verified recovery point the DR plan can actually rely on.

python

import boto3, hashlib, time

s3_mum = boto3.client('s3', region_name='ap-south-1')
s3_sin = boto3.client('s3', region_name='ap-southeast-1')

SOURCE_BUCKET = 'ipl-match-data-ap-south-1'
DEST_BUCKET   = 'ipl-match-data-ap-southeast-1'
TEST_KEY      = 'match-data/IPL2024FINAL.json'

# Upload test match data to source bucket.
test_content = json.dumps({
    'matchId':     'IPL2024FINAL',
    'battingTeam': 'KolkataKnightRiders',
    'runs': 222, 'wickets': 3, 'overs': '20.0',
}).encode()

s3_mum.put_object(Bucket=SOURCE_BUCKET, Key=TEST_KEY, Body=test_content)
source_md5 = hashlib.md5(test_content).hexdigest()
print(f'Uploaded {TEST_KEY} to source bucket. MD5: {source_md5}')
print('Waiting 15 minutes for replication...')
time.sleep(900)  # wait for replication (in lab, can reduce and verify manually)

# Verify object in destination bucket.
try:
    dest_obj = s3_sin.get_object(Bucket=DEST_BUCKET, Key=TEST_KEY)
    dest_content = dest_obj['Body'].read()
    dest_md5 = hashlib.md5(dest_content).hexdigest()
    match = 'MATCH' if source_md5 == dest_md5 else 'MISMATCH'
    print(f'Destination object MD5: {dest_md5} — {match}')
    print(f'Replication verified: {TEST_KEY} replicated to {DEST_BUCKET}')
except s3_sin.exceptions.NoSuchKey:
    print('Object not yet replicated. Check replication metrics in S3 console.')

Step 3 — Route 53 Failover Configuration

Configure Route 53 health checks for the Mumbai and Singapore application endpoints, and create failover routing records that point api.ipl.srihayavadhana.in to Mumbai as PRIMARY and Singapore as SECONDARY. The health check monitors the endpoint’s /health path every 30 seconds; after three consecutive failures, the health check marks the endpoint unhealthy and Route 53 stops returning the PRIMARY record, automatically returning the SECONDARY record to all DNS queries.

Analogy🏏Cricket

🏏 Think of it like cricket: A match official continuously watches whether the main venue is fit to play — checking the ground every few minutes — and holds a standing instruction to switch the fixture to the reserve venue the moment the main one is confirmed unplayable. Just as that official inspects on a fixed cadence, this step's Route 53 health check probes Mumbai's /health path every 30 seconds. Just as one brief shower does not abandon the match but three confirmed failed inspections do, the health check marks the endpoint unhealthy only after three consecutive failures, avoiding false alarms. Just as the standing instruction names the main ground as primary and the reserve as the automatic fallback, you create failover records pointing api.ipl.srihayavadhana.in at Mumbai as PRIMARY and Singapore as SECONDARY. Just as the switch happens automatically once the main venue is ruled out, Route 53 stops returning the PRIMARY record and serves SECONDARY to every query. The payoff: automatic, false-alarm-resistant DNS failover that reroutes fans to Singapore without any human in the loop.

python

import boto3

r53 = boto3.client('route53')

MUMBAI_ALB    = 'ipl-scorecard-alb.ap-south-1.elb.amazonaws.com'
SINGAPORE_ALB = 'ipl-scorecard-alb.ap-southeast-1.elb.amazonaws.com'
HOSTED_ZONE   = '/hostedzone/YOUR_HOSTED_ZONE_ID'

# Health check for Mumbai primary.
mumbai_hc = r53.create_health_check(
    CallerReference='ipl-mumbai-hc-001',
    HealthCheckConfig={
        'Type':               'HTTPS',
        'FullyQualifiedDomainName': MUMBAI_ALB,
        'ResourcePath':       '/health',
        'RequestInterval':    30,
        'FailureThreshold':   3,
        'EnableSNI':          True,
    },
)['HealthCheck']['Id']
print(f'Mumbai health check: {mumbai_hc}')

# Health check for Singapore secondary.
singapore_hc = r53.create_health_check(
    CallerReference='ipl-singapore-hc-001',
    HealthCheckConfig={
        'Type':               'HTTPS',
        'FullyQualifiedDomainName': SINGAPORE_ALB,
        'ResourcePath':       '/health',
        'RequestInterval':    30,
        'FailureThreshold':   3,
        'EnableSNI':          True,
    },
)['HealthCheck']['Id']

# Create failover routing records.
r53.change_resource_record_sets(
    HostedZoneId=HOSTED_ZONE,
    ChangeBatch={'Changes': [
        {'Action':'UPSERT','ResourceRecordSet':{
            'Name':'api.ipl.srihayavadhana.in','Type':'A',
            'SetIdentifier':'mumbai-primary',
            'Failover':'PRIMARY',
            'HealthCheckId': mumbai_hc,
            'AliasTarget':{'HostedZoneId':'ZP97RAFLXTNZK',
                           'DNSName': MUMBAI_ALB, 'EvaluateTargetHealth': True},
        }},
        {'Action':'UPSERT','ResourceRecordSet':{
            'Name':'api.ipl.srihayavadhana.in','Type':'A',
            'SetIdentifier':'singapore-secondary',
            'Failover':'SECONDARY',
            'HealthCheckId': singapore_hc,
            'AliasTarget':{'HostedZoneId':'Z1LMS91P8CMLE5',
                           'DNSName': SINGAPORE_ALB, 'EvaluateTargetHealth': True},
        }},
    ]},
)
print('Route 53 failover records created: Mumbai PRIMARY, Singapore SECONDARY.')

Step 4 — DNS Failover Drill

Simulate a regional outage by blocking the Mumbai health check endpoint (or by stopping the Mumbai ALB) and measuring the time for Route 53 to mark the primary health check as unhealthy and start returning the Singapore endpoint in DNS responses. Use the `dig` command or Boto3 to query the DNS record before and after the simulated outage, recording the timestamps to calculate the actual DNS failover time. Compare the measured failover time against the target RTO.

Analogy🏏Cricket

🏏 Think of it like cricket: Naming a reserve venue means nothing until you run a live, timed fire drill — deliberately declaring the main ground unplayable and clocking exactly how long before officials are directing everyone to the reserve. Just as the drill's whole value is the stopwatch, this step simulates a Mumbai outage by blocking its health endpoint or stopping its ALB and measures how long Route 53 takes to mark it unhealthy and start returning Singapore. Just as you note the announced venue before and after to time the switch precisely, you query DNS with dig or Boto3 before and after, recording timestamps to compute the actual failover time. Just as the measured turnaround is judged against the promised readiness window, you compare the observed DNS failover time against the target RTO. The payoff: an untested failover is only a hope — this timed drill produces a real, measured recovery number, exposing whether health-check thresholds and DNS TTL truly meet the RTO before a genuine outage puts them to the test.

python

import boto3, subprocess, time

r53 = boto3.client('route53')

# Record DNS resolution before simulating outage.
def dns_lookup(hostname):
    result = subprocess.run(['dig', '+short', hostname], capture_output=True, text=True)
    return result.stdout.strip()

print('DNS before outage simulation:')
print(f'  api.ipl.srihayavadhana.in -> {dns_lookup("api.ipl.srihayavadhana.in")}')

# Simulate outage: update Mumbai health check to point to a non-responding endpoint.
# In a real drill, stop the Mumbai ALB target group or block the /health endpoint.
# For lab purposes, update the health check config to an invalid endpoint.
mumbai_hc_id = 'YOUR_MUMBAI_HEALTH_CHECK_ID'
r53.update_health_check(
    HealthCheckId=mumbai_hc_id,
    FullyQualifiedDomainName='ipl-scorecard-alb-STOPPED.ap-south-1.elb.amazonaws.com',
    ResourcePath='/health',  # this endpoint will return no response
)
print('Simulated outage: Mumbai health check now targeting stopped endpoint.')

# Poll DNS until it returns Singapore endpoint.
failover_start = time.time()
timeout = 300  # 5 minutes maximum wait
while time.time() - failover_start < timeout:
    current_dns = dns_lookup('api.ipl.srihayavadhana.in')
    elapsed     = int(time.time() - failover_start)
    print(f'  [{elapsed:3d}s] DNS -> {current_dns}')
    if 'ap-southeast-1' in current_dns or 'singapore' in current_dns.lower():
        print(f'\nFAILOVER COMPLETE: DNS switched to Singapore after {elapsed} seconds.')
        print(f'Measured RTO: {elapsed} seconds (target: <120 seconds)')
        break
    time.sleep(15)
else:
    print(f'TIMEOUT: DNS did not switch to Singapore within {timeout} seconds.')

Expected Results

S3 replication: IPL2024FINAL.json appears in the Singapore bucket within 15 minutes with matching MD5 checksum, confirming cross-region replication is active.
Route 53 health check status: Mumbai health check transitions from Healthy to Unhealthy within 90 seconds of the simulated outage (3 checks × 30-second interval).
DNS failover: the dig query returns the Singapore ALB DNS name within 60 to 120 seconds of the Mumbai health check becoming Unhealthy.
Measured RTO: total time from outage simulation to DNS returning Singapore endpoint is under 180 seconds, confirming the warm standby target RTO is achievable.
Recovery: after restoring the Mumbai health check to a valid endpoint, DNS returns to the Mumbai PRIMARY record within 90 seconds of the health check recovering.

Pro Tip

Use S3 Replication Time Control (RTC) and S3 Replication Metrics in combination to meet strict RPO requirements. RTC provides a 15-minute SLA with 99.99% of objects replicated within 15 minutes and publishes CloudWatch metrics (ReplicationLatency and OperationsPendingReplication) that enable monitoring the replication backlog in real time. A CloudWatch alarm on ReplicationLatency exceeding 10 minutes alerts the team to replication delays before they impact the RPO commitment.

Warning: S3 cross-region replication does not replicate objects that existed in the source bucket before replication was enabled. Only objects created or modified after the replication configuration is applied are replicated. For a DR implementation on an existing bucket with historical data, a one-time sync using `aws s3 sync s3://source-bucket s3://destination-bucket` must be run after enabling replication to copy the existing objects to the destination bucket. Verify the sync completion before relying on the destination bucket for DR recovery.