VPC security groups and private access

Default deny, least ports, no public sprawl.

A public IP address (Internet Protocol address, the number that identifies your machine on the internet) is not a secret. Sweeping the whole address space takes minutes with tools anyone can download, and researchers who park a fresh server with an open admin port online tend to see the first uninvited connection before their coffee goes cold. Nobody has to find you. You get found by default, and the only question worth asking is what the visitor meets when they arrive.

IAM (Identity and Access Management, the AWS service that decides which API calls a caller may make) has nothing to say about that visitor. IAM guards the control plane, the application programming interface you use to create, change and destroy AWS resources. A TCP connection (Transmission Control Protocol, the ordinary way two machines open a two-way data stream) landing on port 5432 of your PostgreSQL database never passes through IAM at all. The database checks a password, or it does not. RDS (Relational Database Service, AWS's managed database hosting) can hand that login check to IAM if you turn on database authentication, and even then IAM has no vote on whether the packet reaches the port. That is how an account full of careful least-privilege policies still ends up mining cryptocurrency for a stranger. Nobody authenticated to AWS. They reached a cache or a container API that shipped with no password on it.

A security group is a guest list, and the surprising part is where the list is stapled. Not to the building, not to the floor, but to each individual network card. Every EC2 instance (Elastic Compute Cloud, a rented virtual machine), every load balancer node and every managed database in your VPC (Virtual Private Cloud, your own fenced-off slice of AWS networking) has its traffic filtered by the groups attached to its own ENI (elastic network interface, the virtual network card AWS hands the resource). Two instances a metre apart in the same rack still have to satisfy each other's lists before they trade a single packet.

The second thing to hold onto is that this guest list keeps notes. Security groups are stateful, meaning AWS remembers every connection it allowed in. Permit inbound HTTPS (Hypertext Transfer Protocol Secure, encrypted web traffic) on port 443 and the server's replies leave again on their own, without you writing one outbound rule. A doorman who remembered only one direction would make you describe both halves of every conversation, and you would spend your career debugging return traffic. Hold that thought, because network ACLs (access control lists, the filter that sits on the subnet boundary instead of the network card) work exactly that way, and it is their defining pain.

What a Security Group Actually Is

Three properties do most of the work. A security group holds allow rules only, so there is no deny rule to write and none to go hunting for; the absence of a matching allow is the deny. Rules are unordered, because with nothing to contradict them there is nothing to resolve: every rule gets checked, and one match is enough to let the packet through. A group you create from scratch starts with zero inbound rules and a single outbound rule permitting everything to 0.0.0.0/0 (CIDR notation, the 10.0.0.0/16 style shorthand for a range of addresses, where 0.0.0.0/0 means every address there is). Shut going in, open going out.

The default security group that every VPC is born with does not follow that shape, and it is the one nobody opens.

aws ec2 describe-security-groups \
  --filters Name=group-name,Values=default Name=vpc-id,Values=vpc-0f2a91c4d8e7b3a05 \
  --query 'SecurityGroups[0].{Id:GroupId,Inbound:IpPermissions,Outbound:IpPermissionsEgress}'

{
    "Id": "sg-0b17d4e9a3c62f118",
    "Inbound": [
        {
            "IpProtocol": "-1",
            "IpRanges": [],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "UserIdGroupPairs": [
                {
                    "GroupId": "sg-0b17d4e9a3c62f118",
                    "UserId": "111122223333"
                }
            ]
        }
    ],
    "Outbound": [
        {
            "IpProtocol": "-1",
            "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ],
            "Ipv6Ranges": [],
            "PrefixListIds": [],
            "UserIdGroupPairs": []
        }
    ]
}

Read the inbound block twice. An IpProtocol of -1 means every protocol on every port, and the source is not an address range at all but the group's own ID. Anything wearing the default group reaches anything else wearing it, on any port, with no further check. Wherever resources picked up that group because nobody specified one, you have a flat network with a security label taped to the side of it. You cannot delete the default group, so empty its rules and give every workload a purpose-built group of its own.

Quotas shape your design more than people expect. One group holds 60 inbound and 60 outbound rules by default, and IPv4 rules (Internet Protocol version 4, the familiar four-number addresses) and IPv6 rules (version 6, the long hexadecimal ones) are counted separately, so a dual-stack group carries 60 of each. One network card takes 5 groups by default and 16 at the outside, and the two limits multiply: rules per group times groups per interface may not exceed 1,000. Teams who paste every office address range into every group hit that ceiling, merge groups to buy back room, and end up with one change that quietly touches forty unrelated hosts.

Find What the Internet Can Already Reach

Start every account review the same way. Ask AWS which groups name the whole internet as a source.

aws ec2 describe-security-groups \
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 \
  --query 'SecurityGroups[].[GroupId,GroupName,VpcId]' --output text

sg-0a1b2c3d4e5f60718	prod-alb-public	vpc-0f2a91c4d8e7b3a05
sg-04c9e18b7a2d5f3c6	legacy-jump-host	vpc-0f2a91c4d8e7b3a05
sg-09fbe27d1c8a4630b	eks-node-shared	vpc-0f2a91c4d8e7b3a05

That names the groups but not the ports, and the port is the whole question. A load balancer group open on 443 is doing its job. A jump host group open on 22 is a different conversation. describe-security-group-rules gives you one line per rule, together with the rule ID you need in order to remove it.

aws ec2 describe-security-group-rules \
  --filters Name=group-id,Values=sg-04c9e18b7a2d5f3c6 \
  --query 'SecurityGroupRules[?IsEgress==`false`].[SecurityGroupRuleId,IpProtocol,FromPort,ToPort,CidrIpv4,CidrIpv6,Description]' \
  --output text

sgr-0d61f8a4b93c25e70	tcp	22	22	0.0.0.0/0	None	temp access for vendor, remove after cutover
sgr-07b3c9e15f8d420a6	tcp	22	22	None	::/0	None
sgr-02e4a7c60b1d9f835	tcp	443	443	0.0.0.0/0	None	None

Two lines here deserve your attention. The first carries a description that dates itself, the archaeology of every real account: somebody opened SSH (Secure Shell, the encrypted remote-login protocol) for a vendor during a migration, and the migration finished without the rule finishing. The second line is worse, because the previous command never showed it. The ip-permission.cidr filter matches IPv4 and only IPv4. That ::/0 entry is the IPv6 twin of 0.0.0.0/0, a separate rule with separate accounting, and if the subnet has an IPv6 range assigned it is carrying live traffic.

Removing a rule by its ID is far safer than restating the permission and hoping your JSON (JavaScript Object Notation, the text format the CLI speaks) matches what is already there byte for byte.

aws ec2 revoke-security-group-ingress \
  --group-id sg-04c9e18b7a2d5f3c6 \
  --security-group-rule-ids sgr-0d61f8a4b93c25e70 sgr-07b3c9e15f8d420a6

{
    "Return": true
}

Chain the Groups Instead of Pasting Address Ranges

The highest-value habit in AWS networking is naming another security group as the source of a rule instead of an address range. A rule that reads "port 5432 from sg-0c5a92f7e31b48d6a" means "from anything wearing the app group", and membership is the identity. Instances launched by an autoscaling group are covered the moment they appear. Instances that go away lose their access when the network card detaches. Nobody maintains a list of single addresses that goes stale the first time a host is replaced.

# the load balancer is the only thing the internet may speak to
aws ec2 authorize-security-group-ingress --group-id sg-0a1b2c3d4e5f60718 \
  --ip-permissions 'IpProtocol=tcp,FromPort=443,ToPort=443,IpRanges=[{CidrIp=0.0.0.0/0,Description="public HTTPS"}]'
 
# the app accepts 8080 from the load balancer group and nothing else
aws ec2 authorize-security-group-ingress --group-id sg-0c5a92f7e31b48d6a \
  --ip-permissions 'IpProtocol=tcp,FromPort=8080,ToPort=8080,UserIdGroupPairs=[{GroupId=sg-0a1b2c3d4e5f60718,Description="from public ALB"}]'
 
# the database accepts 5432 from the app group and nothing else
aws ec2 authorize-security-group-ingress --group-id sg-03d8f1e4a97b6c250 \
  --ip-permissions 'IpProtocol=tcp,FromPort=5432,ToPort=5432,UserIdGroupPairs=[{GroupId=sg-0c5a92f7e31b48d6a,Description="from app tier"}]'

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0f7c31a9e6b845d20",
            "GroupId": "sg-03d8f1e4a97b6c250",
            "GroupOwnerId": "111122223333",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 5432,
            "ToPort": 5432,
            "ReferencedGroupInfo": {
                "GroupId": "sg-0c5a92f7e31b48d6a",
                "UserId": "111122223333"
            },
            "Description": "from app tier",
            "Tags": []
        }
    ]
}

Each of those three calls answers in the same shape; the block above is the reply from the last one, the database rule. Compare it with the version most people write first, which allows 5432 from 10.0.0.0/16. Both look private on a diagram. Only one is least privilege. The VPC range covers the build agent, the monitoring box, the forgotten test instance from 2022, and whatever an attacker lands on after phishing a developer. With the referenced-group version, an intruder holding an unrelated host in the same subnet gets a timeout on 5432, because that host is not wearing the app group. That gap is often the difference between an incident and a breach.

Two ports deserve a standing rule of their own. Port 22 for SSH and port 3389 for RDP (Remote Desktop Protocol, the Windows equivalent of a remote login) should never name 0.0.0.0/0 or ::/0 as a source, and on a modern account they should carry no inbound rule at all. AWS Systems Manager Session Manager, which a later lesson covers, hands you an audited interactive shell through the AWS API with zero inbound ports open. Egress deserves a second look too: that default allow-everything outbound rule is convenient, and it is also the road a compromised host takes to reach whatever server is giving it orders. Where the data matters, pin outbound traffic to the endpoints and prefix lists the workload genuinely needs.

A Network Load Balancer can carry security groups only if you attached them when you created it. Build one without any and there is no way to add them later; you rebuild the load balancer and move DNS (Domain Name System, the internet's name-to-address directory). Application Load Balancers have always had them. Group boundaries surprise people on EKS (Elastic Kubernetes Service, AWS's managed Kubernetes) as well: with the default VPC CNI setup (Container Network Interface, the plugin that gives pods their addresses) every pod on a node shares that node's security group, so a rule you wrote for one workload opens the port for every pod on the machine. Per-pod groups need ENABLE_POD_ENI set on the aws-node DaemonSet, a SecurityGroupPolicy resource, and Nitro-based instance types.

Network ACLs, and Why They Are the Blunt Instrument

A NACL (network access control list, a firewall on the subnet boundary rather than on a network card) is the other filter in the path, and nearly everything about it mirrors a security group. It is stateless, so a reply counts as a brand new packet that has to match a rule of its own. Its rules are numbered and read lowest first, with the first match winning. It has real deny rules, which security groups lack entirely. The default NACL that comes with your VPC allows everything both ways, while a NACL you create yourself denies everything until you write rules, and that asymmetry has ruined many afternoons.

Statelessness is where the cost shows up. Allow inbound 443 and the reply leaves from port 443 toward whatever high-numbered port the client picked, so you need an outbound allow covering that range. Different clients pick from different ranges: most Linux kernels use 32768 to 61000, Windows Server 2008 and later use 49152 to 65535, and NAT gateways, Lambda functions and Elastic Load Balancing all use 1024 to 65535. So you end up allowing 1024 to 65535 outbound, which is most of the port space, which is most of the reason NACLs deliver less than people hope.

There is one thing they do that security groups cannot: block a named source. During an incident, when a single address is hammering you, a numbered deny at the subnet edge is the fastest tool you have.

aws ec2 create-network-acl-entry \
  --network-acl-id acl-0e1b73c95f2a68d40 \
  --rule-number 10 --protocol -1 --rule-action deny \
  --cidr-block 198.51.100.24/32 --ingress
 
# that command prints nothing on success, so read the ACL back
aws ec2 describe-network-acls --network-acl-ids acl-0e1b73c95f2a68d40 \
  --query 'NetworkAcls[0].Entries[?Egress==`false`].[RuleNumber,RuleAction,CidrBlock,Protocol]' \
  --output text

10	deny	198.51.100.24/32	-1
100	allow	0.0.0.0/0	-1
32767	deny	0.0.0.0/0	-1

Rule 10 is read before rule 100, so that host is dropped at the subnet edge before any security group is consulted. Rule 32767 is the implicit final deny that closes every ACL. Check the limits before you get ideas: 20 rules per direction by default, counted separately for IPv4 and IPv6, raisable to 40, and AWS warns that network performance suffers as you climb. One more thing to know before you trust it: a NACL never sees traffic between two instances in the same subnet, because that traffic never crosses the subnet boundary. If your east-west controls live in NACLs, they are not running.

Take the Traffic Off the Public Internet Entirely

The strongest version of this control is having nothing left to filter. An instance in a private subnet, meaning a subnet whose route table has no route to an internet gateway, cannot be reached from outside no matter how sloppy its security group later becomes. That is a structural control rather than a rule, and structural controls keep working on the Friday evening when everyone is tired.

The catch is that your workload still needs S3 (Simple Storage Service, AWS's object storage), ECR (Elastic Container Registry, where your container images live), Secrets Manager and CloudWatch, and all of those answer on public endpoints. A NAT gateway (Network Address Translation, which lets private hosts reach the internet through one shared public address) works and also hands malware a tidy path out. VPC endpoints solve it properly. Gateway endpoints cover exactly two services, S3 and DynamoDB (AWS's key-value database), cost nothing, and amount to a route table entry pointing at a managed prefix list. Interface endpoints, the PrivateLink kind, place a real network card with a private address inside your subnet for almost any AWS service, bill per hour for each subnet plus a charge per gigabyte, and carry a security group of their own.

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0f2a91c4d8e7b3a05 \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.eu-west-1.secretsmanager \
  --subnet-ids subnet-0a3f7c9e1d5b28460 subnet-06e4b2d8f7a91c35d \
  --security-group-ids sg-0e93b7a5f14c8206d \
  --private-dns-enabled \
  --query 'VpcEndpoint.{Id:VpcEndpointId,State:State,PrivateDns:PrivateDnsEnabled}'

{
    "Id": "vpce-0d47a1b93e6f52c08",
    "State": "pending",
    "PrivateDns": true
}

Once the endpoint exists you can make it the only door. An aws:SourceVpce condition on a bucket policy turns "this data is private" from a promise into an enforced rule, and credentials stolen from a developer laptop stop working the moment they are used from anywhere else. Requests that arrive outside a VPC endpoint carry no aws:SourceVpce key at all, and a StringNotEquals test against a missing key evaluates to true, so the deny catches them. Name a break-glass role in the same condition or you will lock yourself out of your own bucket.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyAccessOutsideOurEndpoint",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::app-artifacts-prod",
      "arn:aws:s3:::app-artifacts-prod/*"
    ],
    "Condition": {
      "StringNotEquals": {
        "aws:SourceVpce": "vpce-0d47a1b93e6f52c08",
        "aws:PrincipalArn": "arn:aws:iam::111122223333:role/break-glass-s3-admin"
      }
    }
  }]
}

Be honest about what this costs you. Console access to the bucket dies, because the console reaches S3 from AWS's own network rather than through your endpoint. So does any CI runner (continuous integration, the machines that build and test your code) living outside the VPC, any cross-account backup job, and any replication rule. An explicit deny in a resource policy beats every allow anywhere else, including an administrator's. Endpoint policies are a second authorization layer in their own right, so give them least privilege the way you would a bucket policy, and never roll this out account-wide the day before a long weekend.

Prove It Instead of Guessing

In an account with a few hundred groups, reasoning out loud about whether a path is open is a party trick you will eventually get wrong. Reachability Analyzer answers the question by walking the path through your real configuration, without sending a packet, for ten cents a run. It reads configuration and configuration only, so it cannot tell you whether a process is listening or whether the host's own firewall drops the packet.

PATH_ID=$(aws ec2 create-network-insights-path \
  --source i-0b9f3a7c15d842e60 \
  --destination i-04e7c8a2f19d63b05 \
  --protocol tcp --destination-port 5432 \
  --query 'NetworkInsightsPath.NetworkInsightsPathId' --output text)
 
ANALYSIS_ID=$(aws ec2 start-network-insights-analysis \
  --network-insights-path-id "$PATH_ID" \
  --query 'NetworkInsightsAnalysis.NetworkInsightsAnalysisId' --output text)
 
# the run takes a few seconds; repeat this until Status leaves "running"
aws ec2 describe-network-insights-analyses \
  --network-insights-analysis-ids "$ANALYSIS_ID" \
  --query 'NetworkInsightsAnalyses[0].{Status:Status,Reachable:NetworkPathFound,Why:Explanations[].ExplanationCode}'

{
    "Status": "succeeded",
    "Reachable": false,
    "Why": [
        "ENI_SG_RULES_MISMATCH"
    ]
}

ENI_SG_RULES_MISMATCH means a security group along the path carries no rule permitting that destination port. Not a routing problem, not an ACL problem, not DNS. That one string is worth more than an hour of packet captures. Run the check after every change to a sensitive path, then run it backwards as a control test: analyse the path from a public subnet to your database and confirm the answer comes back false. A control you have never watched fail is a control you have never tested.

For what actually happened, rather than what could happen, you want VPC Flow Logs, which write one line per network flow with source, destination, ports, byte counts and an ACCEPT or REJECT verdict.

aws ec2 create-flow-logs --resource-type VPC \
  --resource-ids vpc-0f2a91c4d8e7b3a05 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::sec-flowlogs-111122223333/prod/ \
  --max-aggregation-interval 60

{
    "ClientToken": "kR2mQ8x1vTn5b7Jd0pYcAw4zLs6HgE3sNf9Uq1Wm2Xo=",
    "FlowLogIds": [
        "fl-0a2c847bd39e15f6"
    ],
    "Unsuccessful": []
}

Learn the blind spots before you lean on the logs. Flow logs record IP traffic, so ARP (Address Resolution Protocol, how machines find each other's hardware addresses on a local link) never shows up at all. Neither does DHCP (Dynamic Host Configuration Protocol, the service that hands out addresses at boot), nor queries to the Amazon-provided DNS resolver, nor anything to or from 169.254.169.254, the instance metadata address. That last omission is the expensive one. An SSRF bug (server-side request forgery, where an attacker tricks your own application into fetching a URL of their choosing) that pulls role credentials out of instance metadata leaves no line in the flow log whatsoever. You catch it in CloudTrail instead, by spotting the instance's role in use from an address that is not the instance, which is what the GuardDuty findings UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS and .InsideAWS look for. Requiring IMDSv2 (Instance Metadata Service version 2, the session-token version) with a hop limit of 1 shuts most of that door in the first place. One more limit: an ACCEPT line tells you the security group let the packet through at the network card, and a subnet ACL further out can still have dropped it, while inbound packets an ACL kills never reach the card to be logged.

Try This

In a lab account, measure the gap between how many groups exist and how many are doing any work at all.

aws ec2 describe-security-groups --query 'length(SecurityGroups)'
 
aws ec2 describe-network-interfaces \
  --query 'NetworkInterfaces[].Groups[].GroupId' --output text \
  | tr '\t' '\n' | sort -u | wc -l

42
9

Forty-two groups, nine of them attached to a network card. The orphans are noise that slows every audit and buries the rules that matter. Now take the open-to-the-world list from earlier, run the IPv6 pass beside it, and give every rule one of three endings: delete it, swap the address range for a referenced security group, or tag the group with who approved the exception and the date it expires. Then move the whole arrangement into your infrastructure code with a policy check that fails the build on 0.0.0.0/0 or ::/0 over an admin port, because a console edit labelled temporary comes with no expiry date attached.

Next: KMS, where encryption at rest stops being a checkbox and turns into a decision about who holds the key.

Takeaway

The trap worth remembering here: the IPv6 rule your search missed. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related