CoursesAWS security for DevOps engineersInstance roles and SSM Session Manager

Instance roles and SSM Session Manager

Kill standing SSH keys on bastions.

Intermediate25 min · lesson 11 of 13

Somewhere in most AWS accounts sits a small server whose only job is letting humans reach other servers. The bastion, or jump box. Public address, port 22 open, and a file called authorized_keys holding public keys that people pasted in over four years. Nobody remembers who owns half of them.

That file is the spare office key that got copied. The contractor who finished in 2023 still has one. So does a laptop that went missing in an airport. A key does not expire, does not report where it has been, and carries no name. When something destructive happens at 3am, your logs tell you ec2-user did it, and ec2-user is everybody.

Two AWS features remove the reason that box exists. An IAM instance profile (Identity and Access Management, the service that decides which identity may call which API, application programming interface, the machine-to-machine way software asks AWS to do something) hands a running machine short-lived credentials automatically, so no key file lives on disk. Session Manager, part of AWS Systems Manager (SSM, the fleet management service), gives a human an interactive shell through the AWS API instead of a network port. Nothing inbound, and every session tied to a named identity.

Why The Spare Key Is Worse Than It Looks

Put a public address with port 22 on the internet and scanners find it in minutes. That part is well known. The underrated part is what the key wraps. SSH (Secure Shell, the encrypted remote login protocol) gets an attacker a shell, the machine has an instance role, and the role can call AWS APIs. A stolen key is worth whatever that role can do.

Revoking it is the other half. No API call kills an SSH key across a fleet. You edit a file on every machine that trusts it, which means finding every machine, which means an accurate inventory, which is the thing you never have at 2am.

The Instance Profile Is A Vending Machine Bolted To The Wall

An instance profile is a vending machine bolted inside a locked room, dispensing day passes. Nothing guards the button. Anyone already in the room presses it and walks out with a pass good for a few hours. The room is the EC2 instance (Elastic Compute Cloud, a rented virtual machine). The button is one HTTP request to an address that only answers from inside.

Start with the trust policy, the short document naming which kind of thing may wear this role. For an instance, that thing is the EC2 service itself.

ec2-trust.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "ec2.amazonaws.com" },
"Action": "sts:AssumeRole"
}
]
}
terminal
aws iam create-role --role-name ec2-ssm \
--assume-role-policy-document file://ec2-trust.json
output
{
"Role": {
"Path": "/",
"RoleName": "ec2-ssm",
"RoleId": "AROAEXAMPLEID123456",
"Arn": "arn:aws:iam::111122223333:role/ec2-ssm",
"CreateDate": "2026-07-27T08:41:02+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "ec2.amazonaws.com" },
"Action": "sts:AssumeRole"
}
]
}
}
}

Attach the managed policy AWS publishes for agent traffic, then wrap the role in an instance profile and hand it to the machine. The console does that wrapping invisibly. The CLI (command line interface) does not, and forgetting it is the most common reason a fresh instance never appears in Session Manager.

terminal
aws iam attach-role-policy --role-name ec2-ssm \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam create-instance-profile --instance-profile-name ec2-ssm
aws iam add-role-to-instance-profile \
--instance-profile-name ec2-ssm --role-name ec2-ssm
aws ec2 associate-iam-instance-profile \
--instance-id i-0123456789abcdef0 \
--iam-instance-profile Name=ec2-ssm
output
{
"IamInstanceProfileAssociation": {
"AssociationId": "iip-assoc-0abc123def4567890",
"InstanceId": "i-0123456789abcdef0",
"IamInstanceProfile": {
"Arn": "arn:aws:iam::111122223333:instance-profile/ec2-ssm",
"Id": "AIPAEXAMPLEID123456"
},
"State": "associating"
}
}

Give the role a moment to reach the instance. Now the vending machine. From a shell on the box, the credentials are one request away.

terminal
TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-ssm
output
{
"Code" : "Success",
"LastUpdated" : "2026-07-27T09:14:22Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAQ4XMPLE7KJ2VN3RD",
"SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Token" : "IQoJb3JpZ2luX2VjEJr...truncated...",
"Expiration" : "2026-07-27T15:31:07Z"
}

Read the prefix. ASIA marks a temporary session issued by STS (Security Token Service, the part of AWS that prints time-limited badges), and the instance refreshes it before it lapses. Compare AKIA, the prefix on a long-lived access key that keeps working until a human deletes it, which is why leaked long-lived keys remain one of the most common first footholds in a cloud account. The same argument applies one level up: give an ECS task (Elastic Container Service) its own role and a Kubernetes pod its own identity rather than borrowing whatever the host is holding.

The catch lives in the analogy. Nothing guards the button. Your web application, the monitoring agent, a cron job, and any code an attacker manages to run all reach the same address and get the same credentials. Whatever you attach to that role is the ceiling of an application compromise on that box. A role holding s3:GetObject on one bucket makes a bug a bad afternoon. A role holding iam:PassRole and ec2:RunInstances makes it an account takeover.

IMDSv2 Makes The Button Hard To Press By Accident

That address has a name: IMDS, the instance metadata service, a local-only endpoint at 169.254.169.254 that tells a machine about itself and hands out its role credentials. Version 1 answers a plain GET, which matters because of SSRF (server-side request forgery, a bug where you trick an application into fetching a URL of your choosing). An application that fetches image URLs on behalf of users will cheerfully fetch the credentials path and print the answer back to the attacker. The 2019 Capital One breach followed that route, and the damage was set by how much the instance role could reach once the attacker held it.

Version 2 changes the shape of the conversation. You send a PUT first to obtain a token, then send that token back as a header on every read. Most SSRF primitives can only issue GET requests and cannot set custom headers, so they hit a wall. Separately, IMDS refuses any request carrying an X-Forwarded-For header, which shuts down an open reverse proxy relaying a request in from outside. That header rule applies to both versions, so it is not a reason to stay on version 1.

Do not flip the switch blind. EC2 publishes a per-instance CloudWatch metric called MetadataNoToken counting version 1 calls, so you can find who still depends on the old behaviour before you break them.

terminal
aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 --metric-name MetadataNoToken \
--dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
--start-time 2026-07-20T00:00:00Z --end-time 2026-07-27T00:00:00Z \
--period 86400 --statistics Sum
output
{
"Label": "MetadataNoToken",
"Datapoints": [
{ "Timestamp": "2026-07-25T00:00:00+00:00", "Sum": 0.0, "Unit": "None" },
{ "Timestamp": "2026-07-26T00:00:00+00:00", "Sum": 0.0, "Unit": "None" }
]
}

A flat zero across a full week means nothing on that instance still speaks version 1. Empty days simply had no calls at all, so read the whole window rather than the last point. Old SDK versions and hand-written curl calls in startup scripts are the usual stragglers. Once the metric stays flat, require tokens.

terminal
aws ec2 modify-instance-metadata-options \
--instance-id i-0123456789abcdef0 \
--http-tokens required \
--http-endpoint enabled \
--http-put-response-hop-limit 1
output
{
"InstanceId": "i-0123456789abcdef0",
"InstanceMetadataOptions": {
"State": "pending",
"HttpTokens": "required",
"HttpPutResponseHopLimit": 1,
"HttpEndpoint": "enabled",
"HttpProtocolIpv6": "disabled",
"InstanceMetadataTags": "disabled"
}
}

State moves from pending to applied within seconds, and no reboot is involved. Prove it landed from inside the instance by reading the status code rather than the body.

terminal
curl -s -o /dev/null -w '%{http_code}\n' \
http://169.254.169.254/latest/meta-data/iam/security-credentials/
output
401

HttpPutResponseHopLimit is the setting people get wrong, and the wrong mental model is the reason. It is not a general firewall on metadata. It is the time-to-live written into the IP packet carrying the token back from that first PUT, so it limits how many network hops the *token* may travel. At 1, the token dies before it leaves the instance, so a container sitting behind the Docker bridge can never complete the handshake. That is exactly what you want on a plain server and exactly what breaks a container host, because bridge networking costs a hop. Note the trap this creates: while --http-tokens optional is still in force, that same container reads metadata perfectly well over version 1, because GET responses are not hop limited. The breakage appears the day you require tokens, which is usually a different day from the one where you set the hop limit. Raising it to 2 everywhere is the tempting fix, and its price is that a compromised container can borrow the node's role.

A fat node role turns one bad pod into a fleet problem
On Kubernetes the worker node role is shared by every pod scheduled there, so one vulnerable container inherits everything the node can do, and if that role carries iam:PassRole you have rebuilt the classic privilege escalation path. Give pods their own identity with IRSA (IAM Roles for Service Accounts, which trades a Kubernetes service account token for an AWS role session) or EKS Pod Identity, keep the node role down to the handful of permissions the kubelet and the network plugin genuinely need, and block pod traffic to 169.254.169.254 at the network layer so nobody can fall back to the host's credentials.

How A Shell Reaches A Box With No Open Ports

The trick is direction. Nothing dials in. The SSM Agent on the instance opens an outbound connection to AWS on port 443 and holds it open, the way a phone stays connected for the length of a call. When you ask for a session, AWS pushes the request down that existing connection. The security group can have zero inbound rules and the shell still works.

How a shell arrives without an open port
1You run start-session
IAM checks ssm:StartSession, the target and the tags
2SSM control plane
is the instance registered and pinging
3Agent's outbound 443
the instance dialled out first and held the line
4Shell opens on the box
as ssm-user, not as you
5Data channel both ways
keystrokes over TLS, nothing on port 22
6Two records land
CloudTrail says who, session logs say what

Check registration before anything else. One command answers three questions at once: is the agent alive, did the role arrive, is there a network path.

terminal
aws ssm describe-instance-information \
--filters Key=InstanceIds,Values=i-0123456789abcdef0
output
{
"InstanceInformationList": [
{
"InstanceId": "i-0123456789abcdef0",
"PingStatus": "Online",
"LastPingDateTime": "2026-07-27T09:12:44.318000+00:00",
"AgentVersion": "3.3.1142.0",
"IsLatestVersion": true,
"PlatformType": "Linux",
"PlatformName": "Amazon Linux",
"PlatformVersion": "2023",
"ResourceType": "EC2Instance",
"IPAddress": "10.20.3.117"
}
]
}

An empty list means one of three things, and in practice it is always one of three: the agent is not running, the role never actually reached the instance, or the subnet has no route to the SSM endpoints. Starting a session on an unregistered instance gives the same terse message in all three cases, which is why you check registration first.

terminal
aws ssm start-session --target i-0123456789abcdef0
output
An error occurred (TargetNotConnected) when calling the StartSession
operation: i-0123456789abcdef0 is not connected.

For a private subnet with no NAT gateway (network address translation, the device that lets private machines reach the internet), the route is interface VPC endpoints (Virtual Private Cloud, your own isolated network inside AWS): ssm for the control plane and ssmmessages for the session data channel. Add ec2messages only if you still run agents older than 3.3, which routed Run Command traffic through it. Turn on private DNS for each endpoint, and let the endpoint security group accept 443 from your instances. If you turn on session logging, S3 traffic can ride a gateway endpoint, which costs nothing, while CloudWatch Logs (logs) and KMS (kms) need interface endpoints of their own.

Interface endpoints bill roughly a cent per hour each, per Availability Zone, plus data processing, so three endpoints across three zones in every VPC is around two hundred dollars a year per VPC before traffic, and finance will ask. The honest answer is that it usually replaces NAT gateway charges you were paying anyway, and it keeps this traffic off the public internet entirely.

terminal
aws ssm start-session --target i-0123456789abcdef0
output
Starting session with SessionId: alice-0f3a9c1b2d4e5f678
sh-5.2$ id
uid=1001(ssm-user) gid=1001(ssm-user) groups=1001(ssm-user)

If the CLI answers SessionManagerPlugin is not found, that is a missing binary on your laptop rather than a permissions problem. The aws command shells out to a plugin you install once.

Your session is passwordless root by default
Look at the id output above. Sessions land as ssm-user, an account the agent creates on first use, and on Linux the agent also writes /etc/sudoers.d/ssm-agent-users containing ssm-user ALL=(ALL) NOPASSWD:ALL. On Windows it puts ssm-user in the Administrators group. Anyone who can start a session is therefore administrator on that machine without typing a password. If that is not what you intended, remove that sudoers file in your image build, or turn on runAsEnabled in the session preferences so each operator lands as their own named account under normal sudo rules. Expect a few runbooks to break the day you change it, and change it anyway.

Deciding Who Gets Which Shell

ssm:StartSession on Resource: "*" is a master key to every machine in the account. Scope it with a condition on instance tags, so an on-call engineer from the payments team opens shells on payments machines and nothing else.

operator-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ShellsOnMyTeamsInstancesOnly",
"Effect": "Allow",
"Action": "ssm:StartSession",
"Resource": "arn:aws:ec2:eu-west-1:111122223333:instance/*",
"Condition": {
"StringEquals": { "ssm:resourceTag/Team": "payments" }
}
},
{
"Sid": "AndTheShellDocumentItself",
"Effect": "Allow",
"Action": "ssm:StartSession",
"Resource": "arn:aws:ssm:eu-west-1:111122223333:document/SSM-SessionManagerRunShell"
},
{
"Sid": "EndOnlyYourOwnSessions",
"Effect": "Allow",
"Action": ["ssm:TerminateSession", "ssm:ResumeSession"],
"Resource": "arn:aws:ssm:eu-west-1:111122223333:session/${aws:username}-*"
}
]
}

Three details there are worth slowing down for. The instance and the shell document are separate resources and you need both, which catches out people whose policy looks correct and still returns access denied. ssm:resourceTag/Team is read from the target at the moment of the call, so an untagged instance is unreachable by anyone using this policy, which doubles as a quiet compliance signal. The terminate statement uses ${aws:username}, the variable AWS uses in its own example, and that variable only resolves for IAM users. If your engineers arrive through IAM Identity Center or any assumed role, it never resolves, the ARN matches nothing, and people cannot close their own sessions. Test that statement with the identity your team actually signs in as.

There is a hole in tag-based access control, which is why this needs a second policy. Whoever can write tags can grant themselves a shell. An engineer holding ec2:CreateTags retags a production database host as Team=payments and walks in. Deny writes to the tag keys you make decisions with, and remember that tagging at launch goes through the same ec2:CreateTags permission, so this covers RunInstances --tag-specifications too.

deny-tag-promotion.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "NoSelfServiceTagPromotion",
"Effect": "Deny",
"Action": ["ec2:CreateTags", "ec2:DeleteTags"],
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"ForAnyValue:StringEquals": { "aws:TagKeys": ["Team"] }
}
}
]
}

A Deny beats every Allow anywhere in the evaluation, which is what makes this worth writing down rather than trusting everyone's least-privilege intentions. Put it in a permission boundary or a service control policy and the people who provision instances keep their normal tagging powers for every key except the one that opens doors.

Recording What Happened Inside The Shell

CloudTrail records the door opening, not what happened in the room. A StartSession event gives you the identity, the source address, the time and the target.

terminal
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=StartSession \
--max-results 1 --query 'Events[0].CloudTrailEvent' --output text \
| jq '{who: .userIdentity.arn, when: .eventTime,
from: .sourceIPAddress, target: .requestParameters.target,
sessionId: .responseElements.sessionId}'
output
{
"who": "arn:aws:sts::111122223333:assumed-role/oncall-engineer/alice",
"when": "2026-07-27T09:20:11Z",
"from": "203.0.113.44",
"target": "i-0123456789abcdef0",
"sessionId": "alice-0f3a9c1b2d4e5f678"
}

The full event carries two more fields, tokenValue and streamUrl, both replaced with HIDDEN_DUE_TO_SECURITY_REASONS because either one would let the reader join the session. lookup-events also only reaches back ninety days, so anything older has to come from the trail's own bucket. That is better attribution than a shared key ever gives you, and notice what is missing: not one keystroke. For the commands themselves you turn on session logging, configured in a Session-type document with the reserved name SSM-SessionManagerRunShell.

session-prefs.json
{
"schemaVersion": "1.0",
"description": "Session Manager preferences",
"sessionType": "Standard_Stream",
"inputs": {
"s3BucketName": "acme-ssm-session-logs",
"s3KeyPrefix": "sessions/",
"s3EncryptionEnabled": true,
"cloudWatchLogGroupName": "/aws/ssm/sessions",
"cloudWatchEncryptionEnabled": true,
"cloudWatchStreamingEnabled": true,
"kmsKeyId": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"runAsEnabled": true,
"runAsDefaultUser": "opsuser",
"idleSessionTimeout": "20",
"maxSessionDuration": "60"
}
}
terminal
aws ssm create-document --name SSM-SessionManagerRunShell \
--document-type Session --document-format JSON \
--content file://session-prefs.json
output
{
"DocumentDescription": {
"Name": "SSM-SessionManagerRunShell",
"CreatedDate": "2026-07-27T09:33:18.402000+00:00",
"Status": "Creating",
"DocumentType": "Session",
"SchemaVersion": "1.0",
"DocumentVersion": "1",
"Owner": "111122223333"
}
}

You create that document once per Region. Later changes go through aws ssm update-document --name SSM-SessionManagerRunShell --document-version '$LATEST' --content file://session-prefs.json, and a second create-document returns DocumentAlreadyExists. Four of those keys deserve a sentence each. idleSessionTimeout accepts 1 to 60 minutes and closes shells somebody walked away from. maxSessionDuration accepts 1 to 1440 minutes and caps any single session, which finally kills the terminal window that has been open since March. runAsEnabled with a runAsDefaultUser means sessions land as that operating system account instead of ssm-user, and you can override it per person by tagging their IAM user or role with the key SSMSessionRunAs and their own username as the value. Leaving runAsEnabled on with no default user and no tag does not fall back to anything, it fails the session, so set one or the other before you ship it.

kmsKeyId encrypts the session stream itself between your terminal and the instance, which is separate from the two EncryptionEnabled flags governing the stored copies. Both ends need access to that key: the instance role and the identity starting the session. Two more quiet failures live here. cloudWatchEncryptionEnabled: true refuses to start sessions unless the log group is already encrypted with a KMS key, and s3EncryptionEnabled: true expects the bucket to have default encryption turned on. Set them to false only if you have decided the transcript needs no protection at rest, which is rarely the honest answer.

Here is the part that fails without telling you. AmazonSSMManagedInstanceCore grants no permission to write those logs. The instance role needs its own statement covering the bucket, the log group and the key, or you end up with a configuration that looks finished and a bucket that stays empty.

ssm-session-logging.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "WriteSessionTranscriptsToS3",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::acme-ssm-session-logs/*"
},
{
"Sid": "ReadBucketEncryptionSettings",
"Effect": "Allow",
"Action": "s3:GetEncryptionConfiguration",
"Resource": "arn:aws:s3:::acme-ssm-session-logs"
},
{
"Sid": "StreamSessionsToCloudWatch",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:eu-west-1:111122223333:log-group:/aws/ssm/sessions:*"
},
{
"Sid": "UseTheSessionKey",
"Effect": "Allow",
"Action": ["kms:GenerateDataKey", "kms:Decrypt"],
"Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}
]
}

s3:GetEncryptionConfiguration is the one everybody leaves out. With s3EncryptionEnabled on, the agent checks the bucket's encryption settings before it uploads, and without that permission the check fails and the transcript never lands. The KMS key policy has to allow the role as well, because a key is guarded from both sides: the identity policy above and the key's own resource policy.

Verify it the boring way. Turn logging on, open one session, run one command, close it, then go and look for the object in the bucket and the stream in the log group. The failure mode here is silence rather than an error, so a test you did not run is a control you do not have.

Then decide who may read the transcripts. Session logs hold whatever an engineer typed, which sooner or later includes a token pasted into an environment variable or a customer record printed to the screen. Keep the bucket in a separate log archive account, set retention long enough to survive an investigation that starts months late, and treat read access as its own privilege with its own approval.

Proving The Door Is Shut

Two checks turn this from an intention into a control. First, that the machines you migrated really do have nothing listening.

terminal
aws ec2 describe-security-groups --group-ids sg-0a1b2c3d4e5f60718 \
--query 'SecurityGroups[].IpPermissions[]'
output
[]

An empty array means zero inbound rules, and the session you opened a minute ago proves Session Manager does not care. Second, sweep the Region for anything still offering SSH to the whole internet.

terminal
aws ec2 describe-security-groups \
--filters Name=ip-permission.from-port,Values=22 \
Name=ip-permission.cidr,Values=0.0.0.0/0 \
--query 'SecurityGroups[].[GroupId,GroupName]' --output text
output
sg-0d41f8a9b2c3e4d5f legacy-bastion-sg
sg-07c2ba5e9d18f3a64 jenkins-agents-old

Be honest about what that filter does and does not catch. It matches rules whose start port is exactly 22, so a lazy rule opening 0 to 65535 slips straight past it. It also matches the two conditions independently, meaning a group with SSH open to your office range and 443 open to the world satisfies both filters and shows up as a false positive. Open every hit and read the actual rules. AWS Config rules and Security Hub controls evaluate port ranges properly, so treat the CLI sweep as a smoke test rather than the audit. The same query style finds instances still accepting version 1 metadata calls.

terminal
aws ec2 describe-instances \
--filters Name=metadata-options.http-tokens,Values=optional \
--query 'Reservations[].Instances[].[InstanceId,MetadataOptions.HttpPutResponseHopLimit]' \
--output text
output
i-04b7c9d2e5f1a3c88 2
i-0f19a2b3c4d5e6f70 1

Both sweeps belong on a schedule, because new instances keep launching from stale images and from Terraform modules nobody has updated in two years. Stop the drift at the source with an SCP (service control policy, an account-wide ceiling set in AWS Organizations that no administrator inside the account can edit away) refusing any launch that does not require tokens.

scp-require-imdsv2.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "NoLaunchWithoutIMDSv2",
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"StringNotEquals": { "ec2:MetadataHttpTokens": "required" }
}
}
]
}

StringNotEquals also fires when the key is absent from the request, which is the behaviour you want here: a caller who says nothing about metadata options is denied rather than waved through. Pair it with the Region-wide default so anything created by hand, or by a tool you have never heard of, starts out correct instead of waiting for your next sweep.

terminal
aws ec2 modify-instance-metadata-defaults \
--http-tokens required --http-put-response-hop-limit 2 \
--region eu-west-1
output
{
"Return": true
}

A hop limit of 2 in the account default is a deliberate compromise for Regions that run containers, since 1 would break every bridged workload the moment tokens became mandatory. On Regions with no container hosts, set it to 1 and keep the tighter blast radius. These defaults apply per Region and only to instances launched from now on, so the existing fleet still needs the sweep. If you build your own images, aws ec2 register-image --imds-support v2.0 bakes the requirement into the AMI so anything booted from it starts with tokens required.

Access Is Only Half The Machine

Session Manager fixes how people get in. It says nothing about the state of what they get into, and an unpatched host with a perfect SSM path is still an unpatched host. Patch Manager applies baselines on a schedule and reports compliance, Inventory tells you what is actually installed, and both ride the agent you already deployed for shells.

Encrypt EBS volumes (Elastic Block Store, the disks attached to instances) with a customer managed KMS key (Key Management Service, where AWS keeps encryption keys) rather than the AWS managed default. The difference shows during an incident: your own key records every use in CloudTrail, and its key policy is yours to edit, so you can cut off a snapshot that turned up shared with an account nobody recognises.

After that, stop keeping pets. Rebuild from a golden AMI (Amazon Machine Image, the template an instance boots from) on a cadence, and roll fleets with an Auto Scaling instance refresh instead of logging in to fix things by hand. A machine replaced every fortnight cannot accumulate ten years of keys in authorized_keys, because that file arrives fresh from the image every time.

The Honest Trade-off

You have moved the risk rather than deleted it, and you should be able to say out loud where it went. The old question was who holds a key on the bastion, and answering it meant grepping files across machines you might not know existed. The new question is who holds ssm:StartSession, which IAM answers in seconds and a policy can narrow to a tag. The catch is that with a generous instance role and ssm-user keeping passwordless sudo, that single permission means administrator on the fleet plus everything the role can reach in AWS.

The second cost is a new dependency. When the agent breaks, or someone deletes an endpoint, or a subnet quietly loses its route, nobody gets in at all, and the pressure to open port 22 temporarily arrives inside the hour. Decide the answer before that day. Either there is no SSH path whatsoever, which is the cleanest position and genuinely achievable, or there is a written break-glass procedure: who approves it, a key issued for hours rather than months, an alarm that fires the moment it is used, and a named person responsible for closing it again. An emergency procedure nobody wrote down turns into a permanent exception.

Quick check
01Session Manager gives you a shell on an instance whose security group has no inbound rules at all. How is that possible?
Incorrect — Nothing touches your security groups. You can watch the rules stay empty while a session is open.
Correct — The instance dialled out first, so no inbound rule is ever needed.
Incorrect — Security groups apply to inbound traffic no matter where it comes from, including AWS ranges.
Incorrect — A new security group allows no inbound traffic at all, and Session Manager needs none.
02An EC2 host runs containers on the default Docker bridge with --http-tokens optional --http-put-response-hop-limit 1. The containers read instance role credentials without trouble. You then set --http-tokens required and they break immediately. Why?
Incorrect — The token requirement applies to every caller equally, and the agent gets no special treatment.
Correct — Only the PUT response is hop limited, which is why version 1 GETs worked fine right up until tokens became required.
Incorrect — A container can send any HTTP verb it likes. The verb is not what fails here.
Incorrect — The endpoint stays enabled, and a container on host networking would still reach it at hop limit 1.
03A new instance in a private subnet never appears in describe-instance-information, and start-session returns TargetNotConnected. The agent process is running and you attached AmazonSSMManagedInstanceCore to the role ec2-ssm. What do you check next?
Incorrect — Session Manager never needs an inbound rule, so this adds exposure and fixes nothing.
Incorrect — That managed policy is enough for registration, and widening it hands every process on the box account-wide power.
Correct — Creating a role is not the same as attaching one, and a private subnet needs a route to those endpoints.
Incorrect — Systems Manager is regional. The instance registers in its own Region and nothing is picked automatically.

Try this

Run aws iam create-instance-profile --instance-profile-name ec2-ssm on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a fat node role turns one bad pod into a fleet problem. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related