Lab — ALB weighted target groups for blue/green traffic split

This lab implements the blue/green deployment pattern using ALB weighted target groups in ap-south-1. You will deploy two versions of the IPL live-scores microservice into separate target groups, configure weighted forwarding rules on the ALB listener, implement a CloudWatch-based rollback trigger, and execute a full canary-to-cutover deployment sequence. By the end of the lab, you will have a reusable deployment runbook validated against a live AWS environment.

Analogy🏏Cricket

🏏 Think of it like cricket: In Test cricket, the ICC publishes playing conditions — governing over rates, DRS quotas, pitch inspection protocols, and player conduct — that both captains sign before the first session, whether the match is at Lord’s, the MCG, or Eden Gardens. Just as the playing conditions give umpires a single authoritative standard so every ruling references the same document rather than personal judgement, the Well-Architected Framework gives architects a shared evaluation language so every workload is measured against the same six pillars rather than each engineer’s intuition. Just as a team posting a slow over rate incurs penalties regardless of their score, a workload with Security or Reliability gaps carries structural risk regardless of how quickly it shipped. Just as every specialist role — opener, keeper, tail — has defined performance expectations against which selectors evaluate each player, every workload component is evaluated against pillar-specific best-practice questions. This reveals why the framework must precede any advanced architectural decision: a shared, evidence-based standard transforms subjective trade-offs into structured, auditable risk assessments that hold across teams, accounts, and regions.

The lab uses the Boto3 SDK throughout rather than the AWS Console, because production deployments must be automatable, auditable, and repeatable. Every step is an API call that can be inserted into a CI/CD pipeline. The pattern demonstrated here is directly applicable to ECS services, Lambda function versions, and EC2 auto-scaling groups — the ALB weighted target group mechanism is the same regardless of the backend compute type.

Prerequisites for this lab: an AWS account with IAM permissions for ELB, EC2, CloudWatch, and Auto Scaling; a VPC with at least two public subnets in different AZs; and two sets of EC2 instances or ECS tasks representing the blue (current stable) and green (new version) deployments. The lab assumes the application exposes a /health endpoint returning HTTP 200 and a /api/scores endpoint returning JSON match data.

Prerequisites

AWS CLI configured with a profile that has elasticloadbalancing:*, ec2:Describe*, cloudwatch:GetMetricStatistics, and cloudwatch:PutMetricAlarm permissions.
VPC with at least two public subnets in ap-south-1a and ap-south-1b, and a security group allowing inbound HTTPS on port 443 from 0.0.0.0/0.
Blue target group already registered with stable version instances responding to /health with HTTP 200.
Green target group registered with new version instances; all targets must pass health checks before beginning the weighted traffic shift.
CloudWatch namespace ipl/deployment already configured and receiving custom metrics from both blue and green instances via the CloudWatch Agent.

Step 1 — Create and Verify Target Groups

Create two target groups in the same VPC: ipl-scores-blue for the current stable version and ipl-scores-green for the new version. Configure both with the same health check settings — path /health, HTTP, port 8080, healthy threshold 2, unhealthy threshold 3, interval 30 seconds — so that their health states are evaluated on equivalent criteria. Verify that all instances in both groups are reporting Healthy before proceeding.

Analogy🏏Cricket

🏏 Think of it like cricket: creating the blue and green target groups with identical health checks is like registering both the current and the new bowler under the exact same fitness test before either is allowed on. Just as ipl-scores-blue holds the stable version and ipl-scores-green the new one, the team lists its proven strike bowler and the debutant side by side on the squad sheet. Just as both groups share the same health-check settings — path /health, port 8080, healthy threshold 2, unhealthy threshold 3, interval 30 seconds — both bowlers must pass the identical fitness screening on the same schedule, so neither gets an easier bar. Just as you verify every instance in both groups is reporting healthy before proceeding, the physio confirms both bowlers are fully fit before the captain considers either for a spell. The payoff: like judging two bowlers on equal fitness criteria so the comparison is fair, matching health checks ensure blue and green are evaluated on the same terms before any traffic shifts.

python

import boto3
elb = boto3.client('elbv2', region_name='ap-south-1')

def create_target_group(name: str, vpc_id: str) -> str:
    tg = elb.create_target_group(
        Name=name, Protocol='HTTP', Port=8080, VpcId=vpc_id,
        HealthCheckProtocol='HTTP', HealthCheckPort='8080',
        HealthCheckPath='/health', HealthCheckIntervalSeconds=30,
        HealthyThresholdCount=2, UnhealthyThresholdCount=3,
        TargetType='instance',
    )['TargetGroups'][0]
    return tg['TargetGroupArn']

VPC_ID = 'vpc-ipl-prod'
blue_tg  = create_target_group('ipl-scores-blue',  VPC_ID)
green_tg = create_target_group('ipl-scores-green', VPC_ID)
print(f'Blue TG:  {blue_tg}')
print(f'Green TG: {green_tg}')

# Verify all targets in both groups are healthy before proceeding.
def check_all_healthy(tg_arn: str) -> bool:
    healths = elb.describe_target_health(TargetGroupArn=tg_arn)['TargetHealthDescriptions']
    states = [t['TargetHealth']['State'] for t in healths]
    print(f'  {tg_arn.split("/")[-3]}: {states}')
    return all(s == 'healthy' for s in states)

assert check_all_healthy(blue_tg),  'Blue targets not all healthy'
assert check_all_healthy(green_tg), 'Green targets not all healthy — do not proceed'
print('Both target groups healthy. Proceeding with traffic shift.')

Step 2 — Configure ALB Listener with Weighted Forwarding

Modify the ALB listener’s default action to use ForwardConfig with weighted target groups. Start with 100% blue and 0% green to establish the baseline. Confirm that the listener is correctly configured by checking the default action type is forward with ForwardConfig rather than the simple forward action. The transition from simple forward to weighted forward is an in-place modification — no new listener or rule is required.

Analogy🏏Cricket

🏏 Think of it like cricket: switching the ALB listener to weighted ForwardConfig is like the captain formally adopting an over-sharing plan while keeping the same bowling end open. Just as you change the listener's default action from a simple forward to ForwardConfig with weighted target groups, the captain moves from 'one bowler bowls every over' to 'a split rota' without setting up a new bowling end. Just as you start at 100% blue and 0% green to establish the baseline, the plan begins with the trusted bowler taking every over so nothing changes for the batsmen yet. Just as the transition is an in-place modification needing no new listener, the captain reworks the existing rota rather than stopping play to build a fresh setup. The payoff: like arming an over-sharing plan that currently still hands every over to the proven bowler, the weighted listener is ready to divert traffic gradually while the baseline stays fully on the version you trust.

python

import boto3
elb = boto3.client('elbv2', region_name='ap-south-1')

LISTENER_ARN = 'arn:aws:elasticloadbalancing:ap-south-1:111111:listener/app/ipl-scorecard-alb/abc/def'

def set_weights(blue_pct: int, green_pct: int) -> None:
    """Shift traffic between blue and green target groups.
    Weights are relative, not absolute percentages, but using 0-100
    integers makes the intent clear for on-call engineers reading the code."""
    elb.modify_listener(
        ListenerArn=LISTENER_ARN,
        DefaultActions=[{
            'Type': 'forward',
            'ForwardConfig': {'TargetGroups': [
                {'TargetGroupArn': blue_tg,  'Weight': blue_pct},
                {'TargetGroupArn': green_tg, 'Weight': green_pct},
            ]},
        }],
    )
    print(f'Traffic weights updated: blue={blue_pct} green={green_pct}')

# Establish baseline: all traffic on blue.
set_weights(100, 0)

# Verify listener configuration reflects the weighted forward action.
listener = elb.describe_listeners(ListenerArns=[LISTENER_ARN])['Listeners'][0]
action = listener['DefaultActions'][0]
print(f'Action type: {action["Type"]}')  # should be 'forward'
print(f'Target groups: {[(t["Weight"]) for t in action["ForwardConfig"]["TargetGroups"]]}')

Step 3 — Deploy Canary at 10%

Shift 10% of traffic to the green target group and observe error rate and latency on both groups for 10 minutes. Use CloudWatch GetMetricStatistics to retrieve HTTPCode_Target_5XX_Count and RequestCount for each target group over the observation window. Calculate the 5XX error rate as 5XX count divided by total request count. The rollback threshold for this exercise is 0.5% — if green’s error rate exceeds 0.5% at any point during the observation window, execute the rollback function immediately.

Analogy🏏Cricket

🏏 Think of it like cricket: shifting 10% of traffic to green and watching for ten minutes is like giving the debutant a single trial over and studying every ball closely. Just as you send 10% to the green target group and observe error rate and latency on both for 10 minutes, the captain hands the newcomer one over while tracking their deliveries against the established bowler at the other end. Just as you use CloudWatch GetMetricStatistics to pull HTTPCode_Target_5XX_Count and RequestCount per group and compute the 5XX rate as errors over total requests, the scorer tallies the newcomer's boundaries conceded against balls bowled to get their economy. Just as a 0.5% error-rate threshold triggers rollback, the captain pulls the debutant the moment they exceed the agreed run rate, handing the over back to the trusted bowler. The payoff: like a trial over that reveals a weakness on limited exposure, the 10% canary surfaces faults in the new version while risking only a fraction of real traffic.

python

import boto3, time
from datetime import datetime, timedelta

elb = boto3.client('elbv2', region_name='ap-south-1')
cw  = boto3.client('cloudwatch', region_name='ap-south-1')

ROLLBACK_THRESHOLD = 0.005  # 0.5% error rate
OBSERVATION_MINS   = 10

def get_error_rate(tg_arn: str, lookback_mins: int = 5) -> float:
    end   = datetime.utcnow()
    start = end - timedelta(minutes=lookback_mins)
    tg_name = tg_arn.split('/')[-3] + '/' + tg_arn.split('/')[-2] + '/' + tg_arn.split('/')[-1]

    def metric(name):
        resp = cw.get_metric_statistics(
            Namespace='AWS/ApplicationELB', MetricName=name,
            Dimensions=[{'Name':'TargetGroup','Value': f'targetgroup/{tg_name}'}],
            StartTime=start, EndTime=end, Period=lookback_mins*60, Statistics=['Sum'],
        )['Datapoints']
        return resp[0]['Sum'] if resp else 0.0

    errors   = metric('HTTPCode_Target_5XX_Count')
    requests = metric('RequestCount')
    return (errors / requests) if requests > 0 else 0.0

# Canary: 10% to green.
set_weights(90, 10)
print(f'Canary deployed. Observing for {OBSERVATION_MINS} minutes...')

start_time = time.time()
while time.time() - start_time < OBSERVATION_MINS * 60:
    green_err = get_error_rate(green_tg)
    print(f'  Green error rate: {green_err:.4%}')
    if green_err > ROLLBACK_THRESHOLD:
        print('ROLLBACK triggered: green error rate exceeded threshold.')
        set_weights(100, 0)  # immediate rollback to blue
        raise SystemExit(1)
    time.sleep(60)

print('Canary observation complete. Green error rate within threshold.')

Step 4 — Full Cutover and Verification

After the canary observation passes, advance to 50/50 for a second 10-minute observation window applying the same rollback logic. If that passes, cut over to 100% green. Immediately after the full cutover, register the old blue instances as draining, wait for in-flight connections to complete (default 300-second deregistration delay), and then deregister them from the blue target group. This step is often skipped in practice, leaving blue instances in the target group as dead weight, which inflates cost and complicates the next deployment cycle.

Analogy🏏Cricket

🏏 Think of it like cricket: advancing to 50/50, then full cutover, then draining the old version is like promoting the debutant to an equal share, then the full workload, while easing the old bowler out gracefully. Just as you move to 50/50 for a second 10-minute window applying the same rollback logic, the captain gives the newcomer half the overs under the same economy check before trusting them fully. Just as you then cut over to 100% green, the debutant takes over the bowling entirely once proven. Just as you register the old blue instances as draining and wait out the 300-second deregistration delay so in-flight connections finish before deregistering, the retiring bowler completes their current over rather than being yanked mid-delivery, letting every ball in progress conclude cleanly. The payoff: like retiring a bowler without abandoning a half-bowled over, the drain-and-deregister sequence completes the cutover with zero dropped requests and the new version fully in charge.

python

import boto3, time
elb = boto3.client('elbv2', region_name='ap-south-1')

# Stage 2: 50/50 observation.
set_weights(50, 50)
print('50/50 stage active. Monitor for 10 minutes...')
time.sleep(600)  # in production, replace with the monitoring loop from Step 3

# Assuming observation passed — full cutover.
set_weights(0, 100)
print('Full cutover to green complete.')

# Retrieve blue instance IDs to deregister.
blue_targets = elb.describe_target_health(
    TargetGroupArn=blue_tg
)['TargetHealthDescriptions']
blue_instance_ids = [t['Target']['Id'] for t in blue_targets]

# Set blue instances to draining (deregistration delay applies).
elb.deregister_targets(
    TargetGroupArn=blue_tg,
    Targets=[{'Id': iid} for iid in blue_instance_ids],
)
print(f'Deregistering {len(blue_instance_ids)} blue instances. Waiting for drain (300s)...')
time.sleep(300)

# Verify blue group is now empty.
final_health = elb.describe_target_health(TargetGroupArn=blue_tg)['TargetHealthDescriptions']
print(f'Blue target group state after drain: {[t["TargetHealth"]["State"] for t in final_health]}')
print('Deployment complete. Green is now the stable version.')

Expected Results

Both target groups show all targets in Healthy state before traffic shift begins.
After set_weights(90, 10), the ALB routes approximately 10% of requests to green targets, observable in CloudWatch RequestCount metrics per target group within 1 minute.
The error rate monitoring loop checks green’s 5XX rate every 60 seconds and automatically calls set_weights(100, 0) if the rate exceeds 0.5%, restoring all traffic to blue within seconds.
After set_weights(0, 100), all RequestCount metrics for the blue target group drop to zero, confirming no traffic is reaching blue instances.
Blue instance deregistration completes within 300 seconds (the default deregistration delay), after which describe_target_health returns an empty list for the blue target group.
The green target group becomes the new blue for the next deployment cycle by re-tagging and updating the BLUE_TG and GREEN_TG constants in the deployment script.

Pro Tip

After completing the full cutover to green, immediately re-tag the green target group as the new blue and create a fresh empty target group as the new green for the next deployment cycle. Swap the BLUE_TG and GREEN_TG constants in your deployment pipeline. This ‘leapfrog’ pattern ensures every deployment always has a clean, empty target group ready to receive the next version, preventing the common mistake of trying to deploy into a target group that still contains draining instances from the previous release.

Warning: Never set a target group weight to zero and assume it receives no traffic immediately. ALB distributes requests proportionally based on weight, but in-flight connections to zero-weight targets are completed before the target group stops receiving new connections. Set the deregistration delay on the target group to a value that matches your application’s maximum request duration. For long-running SSE or WebSocket connections, this may need to be several minutes rather than the default 300 seconds.