Incident Response : Best Practices for Addressing Compromised Assets

Imagine you’re faced with a compromised AWS setup, like having two Availability Zones (AZs), one VPC, and two instances running in an autoscaling group connected to a load balancer, all protected by a WebApp security group. If one of these instances is compromised, here’s what to do:

1. Capture Instance Metadata: Start by collecting all relevant instance metadata. This information will be crucial for analysis and remediation.
2. Enable Termination Protection: Protect the compromised instance from accidental termination to preserve evidence and aid in forensic investigation.
3. Isolate the Instance: Replace the instance’s security group with one that restricts outbound traffic, effectively quarantining the compromised instance from external communication.
4. Detach from Auto-Scaling Group: Remove the compromised instance from any auto-scaling groups to prevent it from affecting the scaling processes and to allow for individual handling.
5. De-register from ELB: Ensure the compromised instance is removed from any load balancer to end its involvement in network traffic.
6. Snapshot Attached EBS Volumes: Take snapshots of any attached EBS volumes for further analysis. These snapshots will provide insights into the root cause of the compromise.
7. Forensic Analysis: Create a forensics EC2 instance and attach the snapshot volumes to conduct a detailed investigation into the compromised instance’s data and configurations.
8. Tagging for Tracking: Tag the compromised EC2 instance with an investigation ticket number or relevant identifier to track its status and facilitate communication within the team.

For investigations, there are two approaches:

• Offline Investigation: Shut down the compromised instance and examine the snapshot volumes attached to the forensics EC2 instance.
• Online Investigation: Capture memory or network traffic while the compromised instance is still active to gather real-time insights into its activities and potential data leakage.

To automate these processes, leverage AWS Lambda for orchestration and SSM Run Command for tasks like memory capture.

Now, let’s turn our attention to other types of AWS resources that may be vulnerable to compromise, such as S3 buckets.

1. Identify Compromised S3 Bucket: Begin by leveraging AWS GuardDuty to detect any suspicious activity within your S3 buckets. GuardDuty provides continuous monitoring and threat detection capabilities to help identify potential security issues.
2. Investigate Malicious Activity: Utilize CloudTrail logs or Amazon Detective to trace the source of the malicious activity. This investigation helps in understanding the nature of the breach and the actions taken by the unauthorized user.
3. Assess Authorization: Determine whether the source of the malicious activity had legitimate authorization to perform the API calls. This assessment helps in identifying potential security misconfigurations or compromised credentials.
4. Enhance S3 Bucket Security: Once the nature of the breach is understood, take proactive steps to reinforce the security of your S3 buckets. Recommended security measures include:
   • Blocking Public Access: Ensure that public access to your S3 buckets is blocked to prevent unauthorized access.
   • Implementing Strong Bucket Policies: Develop robust bucket policies that define access controls and permissions, restricting access to only authorized users and applications.
   • Implementing IAM Policies: Configure IAM policies to enforce least privilege access, ensuring that only necessary permissions are granted to users and applications.
   • Utilizing VPC Endpoints: Create VPC endpoints for Amazon S3 to enable private connectivity and prevent data exfiltration over the public internet.
   • Using S3 Pre-signed URLs: Implement S3 pre-signed URLs for secure and temporary access to objects, limiting exposure to unauthorized users.
   • Migrating to S3 Access Points: Transition from using deprecated methods to newer features like S3 access points, which offer more granular access controls and simplify bucket access management.

Let’s delve into what to do if AWS credentials, IAM roles, or accounts have been compromised.

Compromised IAM User:

1. Rotate Exposed Credentials: Promptly rotate any credentials that have been compromised to prevent unauthorized access.
2. Invalidate Exposed Credentials: Attach an explicit deny policy to the affected IAM user, with an STS date condition, to invalidate any previously issued credentials before the current date.
3. Check CloudTrail Logs: Review CloudTrail logs for any signs of unauthorized activity, such as unusual access attempts or modifications to resources.
4. Review Resources: Thoroughly inspect your resources for any unauthorized changes or deletions that may have occurred due to the compromised credentials.
5. Verify Account Information: Ensure that your account details, including contact information and billing details, have not been tampered with or altered.

Compromised IAM Role:

1. Identify Compromised Role: Determine the compromised IAM role, especially if you have enabled Active Directory integration or utilize role assumption mechanisms.
2. Invalidate Tokens: Attach an explicit deny policy to deny any actions for tokens issued before the current date, effectively invalidating any previously issued tokens.
3. Revoke Access in Linked Systems: If the compromised role is linked to external systems like Active Directory or Single Sign-On (SSO), revoke access to prevent further unauthorized usage.
4. Check CloudTrail Logs: Analyze CloudTrail logs for any unauthorized activity associated with the compromised role.
5. Review Resources and Account Information: Conduct a comprehensive review of your resources for any unauthorized changes and verify the integrity of your account information.

Compromised AWS Account:

1. Delete and Rotate Access Keys: Immediately delete and rotate all access keys associated with the compromised AWS account to prevent unauthorized access.
2. Revoke Unauthorized IAM User Credentials: Revoke any unauthorized IAM user credentials and promptly rotate the passwords of existing IAM users.
3. Delete and Rotate EC2 Key Pairs: Eliminate and rotate all EC2 key pairs to prevent unauthorized SSH access to your instances.
4. Review CloudTrail Logs and Resources: Review CloudTrail logs for any signs of unauthorized activity across your account and review all resources for any unexpected deletions or modifications.
5. Verify Account Information: Ensure that your account details, including contact information and billing details, have not been tampered with or altered.