Security Groups, NACLs and VPC Connectivity

AWS Cloud & DevOps Engineering · Lesson 7 · AWS Cloud Foundations

Lesson Overview

Monday’s progress board loads because three network decisions lined up: the load balancer can reach the progress API, the progress API can reach its database, and nothing else can. When those decisions are wrong, the board is blank, the on-call page is loud, and the first hypothesis is always “the app is down.” More often the app is fine and a return path, a subnet NACL, or a security group that still allows SSH from 0.0.0.0/0 is the real story.

Amazon VPC Architecture gave you subnets, route tables, and gateways. This lesson attaches the filters that sit on the path: security groups as stateful allow-lists on elastic network interfaces, network ACLs as optional subnet-level filters, and the connectivity patterns that let accounts and VPCs talk without opening the public internet. You will leave with a connectivity matrix you can defend in review, a preference for Session Manager over bastion hosts where it fits, and a clear tease of peering and Transit Gateway for multi-VPC work that Module 5 will finish.

The TechLearningPro teaching stack stays fixed: a public edge, private application subnets for the progress API, private data subnets for the store, and a Monday reliability window that cannot depend on an engineer’s laptop being the only path into the VPC. When the matrix is wrong, you will also see false confidence: a curl from an instance with a permissive group “proves” production while the load balancer health check uses a different source. This lesson trains you to test the path the user takes, not the path that is convenient from your shell.

Learning Objectives

By the end of this lesson you will be able to:

1. Explain why security groups are stateful and how that changes what you write for return traffic.
2. Explain why network ACLs are stateless and when an explicit deny is useful.
3. Draw a connectivity matrix for edge, app, data, and management paths for the progress API.
4. Choose security-group references over wide CIDR allows for tier-to-tier traffic.
5. Decide when a bastion host is justified and when AWS Systems Manager Session Manager is the better default.
6. Describe how a VPC endpoint for Systems Manager avoids public SSH while keeping private subnets private.
7. Distinguish security-group “timeouts” from NACL “hard denies” when diagnosing Monday board failures.
8. State what VPC peering solves, what it does not, and when Transit Gateway becomes the hub.
9. Apply least privilege to administrative access: no permanent 0.0.0.0/0 on port 22 in production paths.
10. Run a sandbox lab that proves allow and deny flows, with a budget alert and full cleanup.

Prerequisites

• Networking Fundamentals for AWS for CIDR, ports, and the difference between reachability and permission.
• Amazon VPC Architecture for public and private subnets, NAT, and route tables.
• AWS Identity and Access Management for instance profiles and the idea that network allow is not identity allow.
• AWS CLI, SDKs and Infrastructure Access for verifying flows from a controlled identity.
• A sandbox account with a budget alert before any lasting resources. Cleanup is part of the design, not an afterthought.

Detailed Core Concepts

Two filters, two jobs

A security group is a virtual firewall attached to an elastic network interface (ENI). It evaluates inbound and outbound rules for that ENI. Rules are allow only. There is no explicit deny in a security group. If no rule matches, the traffic is denied. Security groups are stateful: if you allow inbound on port 443, the return packets for that connection are allowed automatically without a matching outbound rule for the ephemeral ports of that flow. You still need outbound rules for new connections the instance initiates, such as the progress API opening a TCP session to the database.

A network ACL (NACL) is a subnet-level filter. It evaluates traffic entering or leaving the subnet. NACLs are stateless: allowing inbound on 443 does not automatically allow the outbound response. You must allow the return path, typically by permitting ephemeral ports (commonly the range used by clients, often documented as 1024–65535 for many lab diagrams, with production teams tightening to what their stack actually uses). NACLs support allow and deny rules evaluated in number order. The first matching rule wins. That is why a low-numbered deny can stop a broad allow that looks correct if you only read the allow lines.

Default NACLs allow all inbound and outbound. Custom NACLs start deny-all until you add allows. Most well-designed VPCs for TechLearningPro’s shape lean on security groups for day-to-day least privilege and use NACLs sparingly: guardrails at subnet boundaries, quarantine subnets, or explicit denies for known bad ranges when policy requires them. Do not duplicate every security-group rule as a NACL “for defense in depth” without a written reason. Two places to forget a return path is how Monday becomes a three-hour incident.

Stateful versus stateless in one Monday story

A learner hits the board. The browser talks to the load balancer. The load balancer opens a connection to a progress API instance on port 8080. The API opens a connection to the database on 5432. Three different security-group conversations happen.

If the app security group allows inbound 8080 from the load balancer’s security group, the response from app to load balancer is covered by state. If the app must call the database, the app initiates; the database security group needs inbound 5432 from the app security group, and the app needs outbound that permits that new connection. People often open inbound on the database and forget outbound on the app after they tighten “default all outbound” in a hardening pass. The symptom is a timeout, not a clear “NACL denied” message.

If someone placed a custom NACL on the data subnet that allows inbound 5432 but forgot ephemeral outbound, the database accepts the SYN and the response is dropped. The client sees a hang. Security group logs are quiet. The NACL is the filter that forgot the return path. Stateful and stateless are not trivia; they are which layer you inspect first.

Rule targeting: CIDR versus security-group ID

You can allow a CIDR (10.0.10.0/24) or a security-group ID (sg-…). Prefer security-group references between tiers inside the VPC. When Auto Scaling replaces progress API instances, their private IPs change; the security group membership does not need a matrix rewrite. CIDR allows are right for known external networks, on-premises ranges via hybrid connectivity, or carefully scoped management subnets. 0.0.0.0/0 on administrative ports is a finding, not a convenience.

Outbound defaults that allow all traffic simplify labs and hide mistakes. Production designs for TechLearningPro often keep broader outbound early, then tighten to destinations the workload needs: database ports, HTTPS to specific prefixes or VPC endpoints, and package repositories through approved paths. Tighten with a change window and a rollback. Do not tighten outbound for the first time during a Monday incident.

Bastion hosts versus Session Manager

A bastion (jump host) is an instance in a public or semi-public management path that you SSH or RDP into, then hop to private instances. It concentrates risk: patching, key distribution, and a security group that must allow administrative ingress from somewhere. Narrowing that “somewhere” to a corporate VPN CIDR is better than the open internet; it is still a host to operate.

AWS Systems Manager Session Manager starts a shell session to an instance through the Systems Manager service, typically without inbound SSH. The instance needs the SSM Agent (present on many Amazon Linux and Windows AMIs), an instance profile with permissions to communicate with Systems Manager, and network reachability to Systems Manager endpoints—often via VPC interface endpoints so private subnets never need a public IP or NAT solely for admin access. Session logging can land in CloudWatch Logs or S3 for audit. Module 2’s AWS Systems Manager and Fleet Operations deepens fleet ops; this lesson only needs the connectivity and security-group consequence: you can leave port 22 closed from the internet and still operate.

Choose a bastion when policy, tooling, or a third-party workflow truly requires SSH, or when you are bridging environments that cannot run the agent. Prefer Session Manager for TechLearningPro’s private app and data tiers when the agent and endpoints are in place. A bastion and open SSH from 0.0.0.0/0 is not defense in depth; it is two mistakes.

Peering and Transit Gateway (foundations tease)

VPC peering connects two VPCs so private IPs can route between them, subject to security groups and NACLs still applying. Peering is not transitive: if A peers with B and B peers with C, A does not automatically reach C. Overlapping CIDRs block peering. Peering is fine for a small number of VPCs with a clear mesh you can draw on one page.

AWS Transit Gateway (TGW) is a hub that attaches many VPCs (and later VPN or Direct Connect) so you are not maintaining an exploding peer mesh. Routing tables on the transit gateway decide which attachments talk. Module 5’s AWS Transit Gateway and Network Architecture and hybrid lessons own the design depth. For foundations, remember: security groups still filter at the ENI after the packet arrives; TGW and peering only solve reachability. Reachability without least-privilege security groups is a larger blast radius, not a completed design.

Connectivity matrix as a design artifact

Before you click Create, write the matrix:

Every row is a test case. Every missing row is an outage waiting for Monday. Every extra row is attack surface.

Identity and network are different planes

Opening a security group does not grant IAM permission, and an AdministratorAccess IAM role does not open a closed security group. TechLearningPro’s progress API instance profile may call s3:PutObject for lesson assets while the instance security group still blocks the path to the VPC endpoint or NAT that would carry that call. Operators confuse “AccessDenied” in the application log with a network timeout, and the other way around. When Monday fails, ask which plane failed first: DNS resolution, TCP connect, TLS handshake, HTTP status, or AWS API error code. Network lessons fix connect and timeout classes. IAM lessons fix AccessDenied. This lesson keeps the planes separate on purpose so lesson 3 and lesson 7 do not blur into a single vague “permissions” ticket.

Security groups also do not replace authentication on the progress API. Allowing the load balancer into port 8080 means the balancer may open a TCP connection. It does not mean every caller is a learner with a valid session. Application authentication, authorization to a course, and rate limits remain application concerns. The network filter reduces who can knock; it does not implement the product’s access control list.

Ephemeral ports without mythology

Client and server roles decide who picks an ephemeral source port. When a browser connects to the load balancer on 443, the browser (or the NAT in front of it) chooses a high source port. When the load balancer connects to the progress API, the load balancer chooses a high source port and the API listens on the fixed app port. When the API connects to PostgreSQL on 5432, the API chooses a high source port. Custom NACLs that only allow “the service port” in both directions without high ports for the response path break the conversation. Security groups avoid this class of mistake for allowed flows because of state. That is a primary reason enterprise VPC designs lean on security groups for application policy and keep NACLs boring.

Document the ephemeral range you intend if you must use custom NACLs, and test with a real cross-subnet connection, not only with packets that never leave the subnet. Same-subnet traffic may never exercise the NACL behavior you think you configured for “the data tier.”

Multiple security groups on one ENI

Attaching several security groups to one ENI unions their allow rules. That is useful for composing “baseline OS management” and “progress API application” policies without a single mega-group. It is dangerous when a legacy “temporary debug” group remains attached and reopens SSH or a wide outbound path. Inventory attachments during review: sg-baseline, sg-app, and nothing else. Automation should prevent humans from attaching sg-open-debug outside an expiry tag and an alarm.

Managed prefix lists and shared rules

AWS-managed prefix lists and customer-managed prefix lists let you reference a named set of CIDRs in security group rules—for example, to allow outbound HTTPS only toward an approved set of destinations, or to allow inbound from a corporate VPN range maintained in one place. Prefer prefix lists over copy-pasted CIDR blocks across dozens of groups when your organization already maintains them. Do not invent a fake prefix list ID in IaC samples; create or reference a real one in your account, or describe the pattern without a pretend ID.

Connection tracking limits and flood awareness

Stateful tracking has scale characteristics. Extreme connection rates, unfinished handshakes, or misbehaving clients can pressure tracking. The operational answer is rarely “disable security groups.” It is to place appropriate edge protection (later lessons on load balancing, WAF, and Shield), to health-check and scale correctly, and to avoid exposing origin instances directly to the internet. For foundations, know that “stateful” is not “infinite,” and that sudden Monday traffic should hit an edge designed for it, not a single instance security group facing the world.

Default deny and the myth of the outbound allow

New security groups in current VPC experience typically allow all outbound traffic and allow no inbound traffic until you add inbound rules. That default outbound convenience is why labs work on the first try and why production hardenings surprise people on the second try. If your organization requires restricted outbound, treat it as a project: list destinations (database, cache, VPC endpoints, artifact repositories, telemetry), implement, test Session Manager and deploys, then enforce. TechLearningPro’s reliability story includes deploys that still pull configuration; outbound least privilege that breaks configuration pull breaks Monday as surely as an inbound hole.

Architecture and Design Diagrams

Stateful allow between tiers, with Session Manager for ops:

mermaidCopy
flowchart TB
  subgraph public [Public subnets]
    ALB["Load balancer SG-lb"]
  end
  subgraph app [Private app subnets]
    API["Progress API SG-app"]
  end
  subgraph data [Private data subnets]
    DB["Database SG-data"]
  end
  subgraph mgmt [Management path]
    SSM["Systems Manager + VPC endpoints"]
  end
  Client["Learners"] -->|"443"| ALB
  ALB -->|"app port; SG ref"| API
  API -->|"db port; SG ref"| DB
  Ops["Operators"] -->|"Session Manager; no inbound SSH"| SSM
  SSM --> API
  SSM --> DB

Where peering and TGW fit later—reachability hubs, filters remain local:

mermaidCopy
flowchart LR
  VPC_A["App VPC"] --- Peer["Peering pair"]
  Peer --- VPC_B["Shared-services VPC"]
  VPC_A --- TGW["Transit Gateway hub"]
  VPC_C["Data VPC"] --- TGW
  VPC_D["Tooling VPC"] --- TGW

Packet path when diagnosing “cannot connect”:

textCopy
Client → Edge SG → (optional subnet NACL) → Route → App ENI SG
                 ↘ return: stateful SG OK; NACL needs ephemeral allow

Technical Explanation

How a security group decision is made

For inbound traffic to an ENI, AWS evaluates the security groups associated with that ENI. Multiple security groups on one ENI are effectively a union of their allow rules. The packet’s source must match a rule: source CIDR or source security group, protocol, and port range. If any allow matches, the inbound path for that filter is permitted. Stateful tracking then permits the return for that connection.

Outbound works the same way for new connections initiated from the ENI. Hardening teams that set outbound to “nothing” and then forget HTTPS to the VPC endpoint for SSM discover that Session Manager “randomly” fails while ICMP from a bastion still works. Separate the failure modes: IAM role missing, agent down, endpoint missing, security group on the endpoint ENIs, and route table without the prefix list routes for gateway endpoints.

How a NACL decision is made

Inbound to a subnet: evaluate inbound rules in ascending rule number. First match wins. Outbound from a subnet: separate list, same idea. Because the filter is stateless, a connection is two evaluations. Ephemeral ports matter for responses. Rule numbers should leave gaps (100, 110, 120) so you can insert without renumbering under pressure. A deny for a specific CIDR at rule 50 and an allow all at 100 is a deliberate quarantine pattern; an allow all at 50 and a deny at 100 never fires for that traffic.

NACLs do not care about security-group IDs as sources in the same way; they work with CIDR ranges. That alone makes them a poor place to encode Auto Scaling–friendly tier rules. Keep tier rules in security groups.

Health checks and asymmetric mental models

Load balancer health checks are connections from the load balancer nodes to your targets. The target security group must allow the health-check port from the load balancer security group (or the documented health-check ranges for classic patterns you may still encounter in older material). If you allow application clients but not health checks, instances look unhealthy while curl from a peer works. Auto Scaling then replaces healthy-enough hosts. Lesson 8 will connect this to group health; here, put the health-check row in the matrix.

VPC endpoints and private API access

Gateway endpoints for S3 and DynamoDB are route-table entries, not ENIs. Interface endpoints are ENIs in your subnets with their own security groups. Those endpoint security groups must allow inbound 443 from the callers that need them. Forgetting the endpoint security group is a classic “NAT works, endpoint does not” ticket. For Session Manager you typically need interface endpoints for ssm, ssmmessages, and ec2messages (verify current endpoint names in AWS documentation for your Region and agent version). Budget them in the lab; delete them in cleanup.

Flow logs as evidence, not decoration

VPC Flow Logs record accept and reject metadata for ENIs or subnets. They do not show application payloads. Use them to see whether a reject happened and which direction. Combine with security group intent and NACL rule numbers. Flow Logs have a cost; turn them on for the lab VPC or for a single subnet while learning, then clean up. Production keeps them where the threat model and budget agree.

Ordered troubleshooting for “progress API unreachable”

1. Confirm the target is running and listening on the expected port (Session Manager, not public SSH).
2. Confirm the load balancer or client is using the correct port and target group.
3. Confirm security group inbound on the target includes the true source (SG reference or CIDR).
4. Confirm outbound on the caller for new connections if the caller initiates.
5. Confirm NACLs on both subnets allow request and response.
6. Confirm route tables: private subnet to NAT or endpoint as designed; no missing peer route.
7. Confirm Network Firewall or third-party appliances if your account uses them (out of scope for the minimal lab, in scope for enterprise review).
8. Only then deep-dive application logs.

Skipping to application logs first is how teams waste the morning. Skipping to “open everything” is how teams waste the year.

Security group changes and risk windows

Security group edits apply to new connection evaluations as AWS documents for your context; do not assume you can “briefly open 0.0.0.0/0” without risk. Temporary opens become permanent through ticket fatigue. Prefer change through infrastructure as code with review, or through a break-glass group with expiry automation if your org has it. Record who approved a temporary rule in the incident timeline when Monday forces an exception—and schedule the removal before the postmortem ends.

Peering DNS and security group references across peers

Depending on peering options and DNS settings, name resolution and security-group referencing patterns across peers need verification against current AWS documentation for your setup. The safe foundations statement: do not assume that an sg-… reference from another VPC works the way it does inside one VPC without checking. Many designs allow peer CIDRs explicitly for cross-VPC rules. Module 5 will refine hub patterns; foundations refuse silent assumptions.

Cost dimensions for connectivity labs

Billable items you might create while learning this lesson include EC2 instance hours, optional Elastic IP charges if allocated and unused carefully, interface VPC endpoint hourly charges, NAT Gateway hourly and data processing charges if you stand one up, data transfer, and Flow Logs storage. Gateway endpoints for S3/DynamoDB do not have the same hourly ENI pattern as interface endpoints—verify the current price card. The lesson rule is unchanged: budget alert first, smallest instance types, shortest lifetime, delete endpoints and NAT when the exercise ends. A forgotten NAT Gateway teaches a more expensive lesson than the security group matrix.

Practical Implementation or Design Walkthrough

Problem

TechLearningPro needs a sandbox VPC that mirrors production tiers enough to prove connectivity: a public subnet with a tiny web instance or load-balancer substitute, a private app instance representing the progress API, and a private data instance representing the database listener. You will build a connectivity matrix, implement security groups (and an optional custom NACL on the data subnet), prove allow and deny paths, prefer Session Manager if your AMI and IAM allow it, and tear everything down.

Before you create anything

1. Confirm you are in a sandbox account and Region you intend to use.
2. Create or verify an AWS Budgets alert (or equivalent billing alarm your org uses) so unexpected hours and endpoints cannot silently accumulate.
3. Tag every resource with Project=tlp-lesson07 and Owner=<you> for cost allocation and cleanup greps.
4. Write the matrix on paper or in the repo before opening the console.

Minimum working path

1. Reuse or create a VPC with public and private subnets from lesson 6 patterns. Keep CIDRs non-overlapping and small.
2. Create sg-lb (or sg-edge), sg-app, and sg-data.
3. Allow 443 or 80 from a controlled CIDR (your IP /32, not the world) into the edge. Prefer /32 for labs.
4. Allow the app port from sg-edge into sg-app.
5. Allow the database port from sg-app into sg-data.
6. Launch two Amazon Linux instances (or one edge and one app if you simulate the DB with a listener on the data instance). Place app and data in private subnets. Attach instance profiles for SSM if you will use Session Manager.
7. From the edge, confirm the app port works. From the app, confirm the data port works.
8. Remove or alter a rule to prove a deny path: app can no longer reach data; capture the symptom (timeout).
9. Optional: attach a custom NACL to the data subnet that allows the DB port inbound and ephemeral outbound; then break ephemeral outbound and observe the hang; restore it.
10. Prefer connecting with Session Manager. If you must use SSH for the lab, restrict to your /32 and delete the key material from uncontrolled locations after.

Hardening and observability

• Add VPC Flow Logs to S3 or CloudWatch Logs for the lab VPC with a short retention; delete the log group or bucket objects in cleanup.
• Document IAM permissions used for SSM: instance role and your user/role permissions separately.
• Ensure no security group leaves the lab with 0.0.0.0/0 on port 22.

Evidence to capture

• Screenshot or CLI output of successful edge-to-app and app-to-data tests.
• Screenshot or CLI output of the intentional deny failure.
• Description of how you connected for administration (Session Manager preferred).
• List of security group rules in force at success time (redact account IDs if sharing externally).
• Budget alert identifier and cleanup confirmation timestamp.

Mapping to TechLearningPro production

Sandbox ports may be 80/8080/5432. Production may terminate TLS at the load balancer and use different app ports. The matrix pattern does not change: edge, app, data, health, management. Replace lab CIDRs with production CIDRs only through review. Never copy a lab rule that allows your home /32 into a shared account and leave it.

Cleanup (required)

Terminate instances, delete ENIs if stuck, delete VPC endpoints, delete custom NACLs associations (re-attach default), delete security groups in dependency order, delete Flow Log subscriptions and storage if created only for the lab, and remove unused key pairs. Confirm the budget dashboard shows no lingering hourly resources. A lab without cleanup is a cost incident with educational branding.

Illustrative CLI shape (verify flags against current AWS CLI documentation; do not invent resource IDs):

bashCopy
# After tagging consistently, list candidates before delete
aws ec2 describe-instances \
  --filters "Name=tag:Project,Values=tlp-lesson07" \
  --query "Reservations[].Instances[].InstanceId"

# Prefer console or your IaC destroy for ordered teardown in early labs

Enterprise Real-World Scenarios

TechLearningPro: the board is up, the API is “down”

Learners see timeouts on progress writes. The ALB target group shows unhealthy targets. An engineer opens SSH from a wide CIDR “just for today,” finds the process listening, and closes the ticket. The next day health checks fail again. Root cause: sg-app allowed human SSH and browser tests from a bastion, but not the load balancer security group on the health-check port. The matrix never had a health-check row. Fix the security group reference; revoke the temporary SSH; move operators to Session Manager; add the health-check row to the architecture decision record.

Shared services VPC without a hub plan

A team peers every application VPC to a shared-services VPC for CI runners and a package cache. The tenth application VPC makes the diagram unreadable. Overlapping CIDRs appear in an acquired environment and peering fails. The short-term patch is NAT to public package mirrors, which breaks the private-only story. The medium-term design is Transit Gateway with deliberate route tables, owned in Module 5. The foundations lesson’s job is to refuse another peer “for now” without writing the CIDR and blast-radius cost of that choice.

NACL “hardening” that broke ephemeral traffic

A compliance checklist demanded custom NACLs on every subnet. An engineer mirrored inbound application ports and forgot outbound ephemeral ranges on the data subnet. Deployments on Tuesday worked from an instance in the same subnet (never crossing the NACL boundary the way they thought). Cross-subnet calls from app to data failed intermittently under load. The fix was not “open everything”; it was either return to default NACL allow-all and rely on security groups, or complete the stateful equivalent with correct ephemeral allows and a test that crosses subnets.

Banking-style quarantine subnet

A financial services partner of TechLearningPro isolates compromised or unpatched instances by moving them to a quarantine subnet whose NACL denies nearly all outbound except to a forensics collector CIDR, and whose security group allows only Session Manager through a tightly controlled path. The application subnets never share that NACL. The point of the story is not theatrics; it is that explicit deny and sparse allow are for containment, while day-to-day least privilege for healthy progress API hosts remains security-group based. If every subnet is “quarantine-hard,” you have built fragility, not safety.

Ecommerce flash sale and the wide-open data tier

An ecommerce team allows the database security group to accept 5432 from the entire VPC CIDR because “many microservices need it.” A compromised low-value service scrapes personal data. The postmortem blames encryption at rest—which was fine—and ignores network blast radius. Corrective action: per-service security groups, SG references only from services that own a connection string, and rejection of VPC-wide database allows in review. TechLearningPro’s learner PII deserves the same refusal.

Best Practices

• Design with a connectivity matrix before creating security groups.
• Prefer security-group references between application tiers over brittle IP lists.
• Keep administrative ingress off the public internet; prefer Session Manager and VPC endpoints.
• Treat NACLs as sparse guardrails, not a second copy of every security-group rule.
• Leave gaps in NACL rule numbers; document the first-match-wins behavior in the runbook.
• Put load balancer health checks in the matrix and in the security group from day one.
• Tag lab and prod differently; never “borrow” prod CIDR allows in a sandbox.
• Use Flow Logs when proving a deny, then retain only what operations will actually read.
• Remember peering is non-transitive; do not promise transitive routing you do not have.
• Plan Transit Gateway when the peer mesh no longer fits on one diagram.
• Separate IAM deny from network deny when troubleshooting; fix the layer that failed.
• Require budget alerts and cleanup evidence for every network lab.
• Deny by absence in security groups; use explicit NACL denies only with a named threat or quarantine.
• Review outbound rules when you harden; new connections need explicit allows.
• Keep data-tier security groups free of client CIDRs; only app tiers should talk to data.

Common Mistakes and Anti-Patterns

Opening port 22 to 0.0.0.0/0 on private instances “because NAT.” Private IP does not mean unreachable from the internet if you also attach a public IP or misuse a load balancer. Close inbound SSH from the world; use Session Manager or a tightly scoped path.

Expecting security groups to deny explicitly. They cannot. If you need an explicit deny, you are in NACL or perimeter service territory, or you should remove the allow you regret.

Forgetting ephemeral ports on custom NACLs. Symptom: SYN works, conversation dies, timeouts everywhere. Fix the return path or revert to default NACL.

Encoding Auto Scaling tiers with CIDR allows. IPs churn; tickets churn with them. Use security-group IDs.

Peering everything to everything. Operational debt and CIDR conflicts. Introduce a hub when the mesh hurts.

Assuming VPC peering replaces security groups. It does not. Filters still apply; you widened reachability.

Tightening outbound to deny-all during an incident. You will break package pulls, SSM, and telemetry at once. Change outbound in a planned window with a rollback.

One security group for all tiers. The data tier inherits edge exposure. Split sg-edge, sg-app, sg-data.

Bastion in front of wide-open app security groups. The bastion becomes theater. Harden the destination groups.

Skipping deny tests in the lab. An allow-only lab teaches confidence, not correctness. Always break a rule on purpose and record the symptom.

Leaving interface endpoints running after class. Quiet hourly cost. Cleanup is part of the rubric.

Troubleshooting and Design Review

Timeouts from app to database, security groups look right. Check outbound on the app, inbound on the data, route tables, NACLs on both subnets, and whether you are testing the correct ENI (multiple interfaces). Use Flow Logs to see rejects.

Session Manager cannot connect. Verify agent status, instance profile, endpoint connectivity and endpoint security groups, and your IAM permissions. Do not “fix” by opening SSH to the world.

Works from instance A, fails from the load balancer. Health check or listener security group mismatch. Compare source of the failing connection.

Intermittent failures after NACL change. Rule order and ephemeral gaps; also asymmetric routes if you are mixing peering and local routes incorrectly.

Peering active but traffic fails. Security groups referencing the peer CIDR or peer SG patterns, route tables for the peer CIDR, and NACL allows for the peer ranges.

Design review questions:

• Is every row in the matrix implemented and tested, including deny cases?
• Can operators reach instances without inbound SSH from the internet?
• Are tier rules SG-referenced?
• Is any custom NACL justified in writing, with ephemeral return paths tested?
• What is the plan when a third VPC appears—more peers or a hub?
• Was the budget alert set, and was cleanup evidenced?

Interview Questions and Answers

Q1. What does it mean that security groups are stateful?

If an inbound (or outbound) connection is allowed, the return packets for that connection are allowed automatically. You do not write a matching reverse rule for that flow’s ephemeral ports. You still need rules for new connections in the other direction.

Q2. What does it mean that NACLs are stateless?

Each direction is evaluated separately. Allowing inbound traffic does not allow the outbound response. You must allow the return path explicitly, which is why ephemeral port ranges appear in NACL designs.

Q3. Why prefer security-group references over CIDR for app-to-database?

Instance IPs change with scaling and replacement. The security group membership remains the stable identity of the tier. CIDR rules become stale and overly broad.

Q4. When would you use a custom NACL?

Sparse guardrails: quarantine subnets, explicit deny of known bad prefixes, or compliance patterns that require subnet-level controls. Not as a full duplicate of security-group rules without a reason.

Q5. Bastion or Session Manager for private progress API hosts?

Default to Session Manager with instance profile, agent, and VPC endpoints, keeping inbound SSH closed. Use a bastion when you truly need SSH-based workflows and can lock its ingress to a trusted path.

Q6. Does VPC peering make security groups unnecessary?

No. Peering provides routing reachability. Security groups and NACLs still filter. Peering can increase blast radius if groups are loose.

Q7. Why is peering non-transitive?

A connection between A–B and B–C does not create A–C. Each pair needs its own path, or you introduce a hub such as Transit Gateway.

Q8. A connection times out. How do you decide SG vs NACL?

Check whether SYN is accepted and whether return traffic can flow. Review Flow Logs accepts/rejects. Validate SG allows and statefulness assumptions, then NACL rule order and ephemeral ports. Also verify routes.

Q9. What security group mistake breaks load balancer health checks?

Allowing user traffic patterns but not the load balancer’s health-check traffic to the target port from the load balancer security group.

Q10. How do interface VPC endpoints affect security groups?

Endpoint ENIs have security groups that must allow inbound 443 from clients. Caller outbound rules must allow the path. Missing either side breaks private connectivity to AWS APIs.

Q11. What belongs in a connectivity matrix that often gets omitted?

Management access, health checks, outbound to dependencies (database, caches, AWS APIs), and the explicit deny or absence cases you will test.

Q12. How does least privilege apply here?

No wider source than needed, no extra ports, no shared all-tier groups, no permanent world-open admin ports, and IAM still required beyond network allow.

Q13. Why might same-subnet tests hide a NACL mistake?

Traffic that never crosses the subnet boundary may not exercise the NACL rules you believe protect “the data tier.” Always validate with a client in the app subnet and a listener in the data subnet.

Q14. What is the risk of a union of many security groups on one ENI?

Any attached group’s allows apply. A forgotten debug group reopens ports you thought you closed. Inventory attachments and manage composition deliberately.

Q15. How do you explain Transit Gateway vs peering in one interview minute?

Peering connects pairs and does not transit. Transit Gateway is a hub for many attachments with centralized routing. Both still require security groups; neither is a substitute for least privilege. Choose peering for simple pairs; plan a hub when the mesh grows.

Q16. What evidence would you bring to a design review for this lesson?

A connectivity matrix, deny-path test results, confirmation that admin access does not rely on world-open SSH, a note on NACL strategy, and cleanup plus budget alert proof for any lab that informed the design.

Hands-On Exercises

Exercise: Connectivity matrix with prove-deny

Problem statement. In a sandbox account, implement edge → app → data allows with security-group references. Prove two failure paths (broken SG rule and, optionally, broken NACL ephemeral return). Prefer Session Manager for shell access. Set a budget alert first. Clean up all billable resources.

Tasks.

1. Write the matrix including health checks and management.
2. Implement security groups; avoid 0.0.0.0/0 on port 22.
3. Demonstrate a successful app-to-data connection.
4. Demonstrate a timeout or failure after removing the data inbound allow; capture evidence.
5. Optional: custom NACL misconfiguration and fix.
6. Document SSM versus bastion choice in five sentences.
7. Destroy all lab resources and confirm no leftover endpoints or instances.

Expected deliverables. Matrix, screenshots or CLI evidence of allow and deny, cleanup checklist signed off, budget alert proof.

Evaluation rubric.

Challenge

Add a second VPC representing shared services. Document whether you would peer or wait for Transit Gateway. Do not build a permanent peer mesh in a shared sandbox without CIDR planning and cleanup ownership. Write a short architecture decision record.

Challenge ADR prompt (worked example shape)

Title: Prefer Session Manager over bastion for progress API hosts.

Context: Private app and data subnets; Monday on-call needs shell access for rare live diagnosis; compliance asks for session audit.

Decision: Session Manager with instance profiles, required VPC interface endpoints in the app and data subnets or a shared endpoints subnet, session logging to a locked S3 bucket, inbound SSH removed from instance security groups.

Consequences: Endpoint hourly cost appears on the bill and must be owned; AMI must include a supported agent; break-glass SSH, if required by policy, uses a separate time-limited path with /32 or VPN CIDR and an expiry ticket—not 0.0.0.0/0.

Alternatives considered: Always-on bastion; VPN-only SSH to instances; third-party PAM. Bastion remains acceptable when SSH is mandated and endpoints cannot be deployed, provided ingress is not world-open.

Lesson Summary

Security groups are the primary, stateful, allow-only filters on ENIs. NACLs are optional, stateless subnet filters with explicit denies and first-match rule numbers. TechLearningPro’s Monday board depends on a matrix that includes edge, app, data, health checks, and management—not only the happy-path application port. Prefer Session Manager and VPC endpoints over world-open SSH. Peering and Transit Gateway change reachability between VPCs; they do not replace least-privilege filters. Lab work without a budget alert and cleanup is incomplete.

Compute Services: EC2, AMIs and Auto Scaling puts workloads on the instances these filters protect, including launch templates and health-aware scaling that assume the security groups you just designed.

Further Reading

• Security groups for your VPC — stateful behavior and rule referencing.
• Network ACLs — ephemeral ports and rule evaluation order.
• AWS Systems Manager Session Manager — private administration patterns.
• VPC endpoints — gateway and interface endpoints.
• VPC peering — non-transitive connectivity limits.
• Transit Gateway — hub connectivity for many VPCs (deep dive later in the course).
• VPC Flow Logs — accept/reject evidence for troubleshooting.