What Is Configuration Drift in AWS Infrastructure?

Configuration drift can occur in any environment where there is a mismatch between the intended configuration and the actual state of resources, whether managed through IaC tools or shell scripts. For example, in a setup using Terraform, a developer may define a security group to restrict SSH access to specific IP addresses, and the configuration is stored in the state file. Later, a team member may log into the AWS Management Console and change the access port for troubleshooting, creating a drift between the intended and actual states. Similarly, drift can happen in environments managed through scripts or console if changes are made directly through a cloud provider’s console or CLI.

How Does Drift Occur?

Now that we know what drift is, let’s take a look at how it can impact your environment, whether in AWS, other cloud providers like Azure and Google Cloud, or within Kubernetes clusters.

Manual Changes

Sometimes, quick changes are made directly in the cloud console or CLI to troubleshoot or resolve an issue. For example, a DevOps Engineer might open ports like 22 (SSH) or 3389 (RDP) in a security group to test application connectivity. If these ports are not closed after the issue is resolved, they leave the resource exposed to potential security threats. Open ports like these can act as entry points for unauthorized users, allowing them to access your instance and compromise sensitive data or disrupt operations. 

Resource Lifecycle Events

Resource lifecycle events can also lead to drift. For example, if an EC2 instance is terminated and then recreated using a default Amazon Machine Image (AMI), any custom configurations, such as application-specific logging or monitoring agents for performance metrics, will not be automatically restored unless explicitly redefined. This results in a lack of visibility into the application's behavior and system performance, making it harder to track performance metrics or detect anomalies. Without proper monitoring in place, issues such as resource overutilization, application crashes, or unhandled errors may go unnoticed, potentially impacting service availability or user experience.

How Drift Affects Infrastructure

Drifts have a serious impact on cloud environments by introducing unintended and often unnoticed changes to resource configurations. These changes deviate from the desired state and can affect security and resources by keeping track of every resource, particularly in complex environments like AWS, which involve interconnected systems and diverse services like EC2, S3, and Lambda.

Security Risks

Configuration drift in network settings, such as security groups or VPC configurations, can lead to serious security risks. For instance, a security group might mistakenly allow unrestricted inbound traffic on critical ports like SSH (port 22), exposing EC2 instances to brute-force attacks or unauthorized access. Such misconfigurations can lead to issues like exposing database credentials, leaving storage buckets open to public access, or allowing unauthorized users to modify infrastructure configurations.

Similarly, drift in IAM roles can also create vulnerabilities. For example, a role meant to have limited permissions, such as read-only access to S3 buckets, might unintentionally gain broader permissions, like the ability to modify or delete bucket contents. These seemingly small errors can go unnoticed but have the potential to escalate into major security breaches if exploited.

Improper Resource Management

Drift often results in resources being mismanaged or underutilized, leading to unnecessary expenses and operational challenges. For example, if auto-scaling policies are unintentionally altered, they may provision more EC2 instances than required, significantly increasing costs without improving performance. Similarly, idle resources like unused Elastic IPs, orphaned EBS volumes, or incorrectly configured Elastic Load Balancers can remain active and continue to incur charges, even though they serve no purpose. These inefficiencies can quickly add up, particularly in cloud environments like AWS, where resource usage is directly tied to billing.

Compliance Violation

Configuration drift can lead to failures in meeting regulatory standards defined by the organization, creating serious compliance rules defined by AWS for each type of resource. A common scenario involves the accidental disabling of encryption on S3 buckets storing personally identifiable information. For organizations governed by GDPR, HIPAA, or similar regulations, this misconfiguration could result in legal penalties and reputational damage.

For example, an unintended change might disable encryption for an S3 bucket or a database, leaving important data exposed in plain text. Similarly, audit logs, which are essential for tracking system activity, might be altered, disabled, or misconfigured due to drift. This could involve turning off logging for specific resources or redirecting logs to an inaccessible location, preventing organizations from monitoring activity effectively. Without accurate and complete logging, it becomes challenging to detect and respond to security incidents, such as unauthorized access, data breaches, or malicious activities. Moreover, such misconfigurations can compromise compliance with regulations that require data encryption and traceability, further increasing the risk to the organization.

AWS Config Recorder

AWS Config Recorder is an AWS service that tracks and records configuration changes to supported AWS resources, providing visibility into your cloud environment. It works alongside AWS Config to help detect drifts, support compliance, and enforce governance. The recorder captures changes made through the AWS Management Console, CLI, SDKs, or APIs, ensuring all modifications are logged for supported resource types.

AWS Config Recorder continuously creates snapshots of resource configurations, including details like properties, dependencies, and metadata. These snapshots allow organizations to monitor changes over time, such as updates to security group rules, IAM policies, or EC2 instance configurations. For example, if an S3 bucket's access policy changes from private to public, the recorder logs details like the user, time, and specific changes.

The recorder supports audits and compliance efforts by providing a historical timeline of configuration states. It integrates with Amazon SNS to send real-time alerts for non-compliant changes, helping administrators act quickly.

AWS Config Recorder supports centralized monitoring, enabling consistent governance across multi-account and multi-region setups. For example, it can flag differences in encryption settings between RDS instances in different regions to ensure uniformity.

By maintaining a detailed log of configuration changes, AWS Config Recorder helps organizations enhance security, compliance, and operational efficiency. 

Now, let's take a look at some steps to set up your AWS Config Recorder to simplify your resource configuration monitoring.

How to Set Up AWS Config Recorder

Setting up the AWS Config Recorder involves three key steps: creating a delivery channel, enabling the recorder, and defining resource types. Let’s take a look at these steps for better clarity. 

Before moving further, we need to make sure that we have an S3 bucket to store configuration snapshots and an Amazon SNS topic set up for notifications.

Create a Delivery Channel

The delivery channel specifies where AWS Config stores the configuration snapshots and sends notifications.

In the given code --delivery-channel-name sets the name of the delivery channel as Test-delivery-channels.--s3-bucket-name sets the S3 bucket where AWS Config stores configuration snapshots ( test-config-bucket in this code). --sns-topic-arn sets the ARN of the SNS topic to send notifications about changes. Also, ensure the S3 bucket policy allows AWS Config to write to it.

Create a Configuration Recorder:

After setting up the delivery channel, we will create an AWS configuration recorder with the following command

In the above code --role-arn is the IAM role that grants AWS Config permissions to record configurations. --recording-group specifies the resources you want to monitor where allSupported tracks all supported resource types. We can also set recorders for specific resources, such as our AWS EC2 instance.

Enable the Recorder:

The recorder must be enabled to capture resource configurations and changes. You can specify the resource types you want to monitor.

Here, --configuration-recorder-name sets the name of the recorder as test-recorder. This recorder uses the delivery channel we created earlier. Once the recorder is enabled, ensure that it is running and properly recording resource configurations using aws configservice describe-configuration-recorders.

How Drift Alerts Maintain AWS Configs

Drift alerts are notifications generated when AWS Config detects unauthorized or unintended changes in resource configurations. These alerts are pivotal in maintaining consistency and compliance.

What Are Drift Alerts?

Drift alerts are automated notifications sent when AWS Config detects that a resource has deviated from its intended configuration state. These deviations, known as drifts, can occur due to manual changes, incomplete automation rollouts, or errors in configuration scripts.

AWS Config tracks and evaluates changes against predefined compliance rules or desired configurations. When a mismatch is found, it generates a drift alert, providing details about the affected resource, the specific drift, and the violated rule. For example, an S3 bucket becomes publicly accessible, an IAM role gains excessive permissions, and resource tags are modified or removed.

Monitoring Drift Alerts in AWS Config Recorder

Monitoring drift alerts involves configuring notifications and analyzing drift details. AWS Config Recorder, combined with AWS Config and SNS services, provides a solution for monitoring drift within your cloud environment. Let’s break down the steps and get a clearer understanding of this process:

Enabling Drift Detection

To monitor drift, first, enable AWS Config and set it to cover the necessary resources like EC2 instances, security groups, IAM roles, and S3 buckets. Start by selecting the resources you want AWS Config to keep an eye on.

Once AWS Config is running, it continuously checks your resources against these rules. If a resource's configuration deviates from its intended state, AWS Config flags it as non-compliant and logs the event. This monitoring system is important for detecting configuration changes early, helping prevent potential security or operational issues.

Setting Up Notifications

Monitoring drift alerts requires effective notification mechanisms to inform teams about non-compliant resources. AWS Config integrates seamlessly with Amazon Simple Notification Service (SNS) to deliver these alerts. By creating an SNS topic, you can configure AWS Config to publish notifications whenever a rule violation occurs.

For example, you might configure an SNS topic specifically for the security team to receive alerts about unauthorized changes to IAM policies. Alternatively, a separate topic can be set up for the operations team to track changes in EC2 instance configurations. SNS supports multiple protocols for delivering alerts, such as email, SMS, or even HTTP endpoints, allowing notifications to be tailored to the needs of different stakeholders.

Integrating with Monitoring Tools

To enhance alert management, AWS Config integrates with AWS CloudWatch, enabling a centralized view of drift alerts alongside other operational metrics. By creating CloudWatch metrics and alarms for AWS Config rule evaluations, you can track the frequency of non-compliant events.

For example, a CloudWatch alert can be set up to trigger a Lambda function that restores a resource to its compliant state whenever a drift is detected within the infrastructure. Alternatively, these metrics can feed into third-party monitoring tools like Datadog or PagerDuty, giving teams the flexibility to manage alerts using the tools they already use for managing their infrastructure.

By using AWS Config’s integration with CloudWatch, organizations can build a monitoring strategy that ensures timely detection, escalation, and resolution of drift-related issues, all while maintaining compliance and operational stability.

Responding to Drift Alerts in AWS Config Recorder

Responding to drift alerts effectively is important in maintaining the integrity, security, and compliance of your cloud infrastructure. It involves verifying and analyzing the drift, applying remediation strategies, and implementing measures to prevent future occurrences. Here, we will take a look into each of these aspects in detail.

Immediate Response to Drift Alerts

When a drift alert is triggered, the first step is to verify the alert and assess its impact. Start by reviewing the specific resource flagged by AWS Config and identifying the rule or predefined configuration it has violated. This involves checking the AWS Config dashboard or querying the AWS CLI to retrieve details of the drift, including the resource type, time of change, and the specific configuration deviation.

For example, if AWS Config flags an EC2 instance’s security group with overly permissive rules, assess how this change impacts your application’s security. Analyze the change's scope to identify whether dependent resources are indirectly affected, such as dependent services or applications relying on the EC2 instance. Such checks help you find issues early so that you can fix them first.

Automated Remediation

Automating remediation workflows not only accelerates the resolution of drift alerts but also reduces the likelihood of misconfiguration during the process. AWS Systems Manager is a powerful tool for creating runbooks; automated workflows that execute predefined steps to address drift.

For example, you can configure a Systems Manager Automation Document to restore security group rules to their compliant state whenever drift is detected. By combining this with CloudWatch alarms triggered by AWS Config drift alerts, the remediation process becomes simpler and quicker.

Manual Remediation

In some cases, manual remediation is necessary, especially for complex configurations where automated fixes might not fully address the issue. The AWS Config dashboard provides a user-friendly interface for identifying and resolving drifted resources.

To address a drifted resource, start by reviewing its configuration timeline in AWS Config to understand when and how the drift occurred. For example, if a drift alert flags an RDS instance missing encryption, you can use the timeline to identify the source of the change. 

Preventing Future Drift

Consistently review and update your baseline configurations to match approved states. This involves regularly auditing your AWS Config rules and configurations to make sure that they reflect current security and operational requirements as well. As your application architecture scales, make sure AWS Config rules are updated to meet new standards. For example, if your encryption requirements change due to new compliance regulations, update the rules to enforce these new policies immediately. 

Additionally, use tools like CloudFormation drift detection to keep your IaC templates in sync with deployed resources. This step is important to make sure that there are no differences between your defined templates and actual cloud configurations.

Hands-On Examples for Drift Monitoring and Responding to Alerts

So far, we've talked about why it's important to enable AWS Config, create rules, resolve drifts, and keep things up-to-date. Now, let's see how to actually do this using Terraform configurations. These configurations will help us set up AWS Config, create custom rules, and fix those drifts. Let's go through each step in detail:

Enabling AWS Config Recorder

First, we need to enable AWS Config to track changes for EC2 instances. This involves setting up the necessary resources like an IAM role, an S3 bucket for the delivery channel, and the config recorder. Here's how you can do it with Terraform:

Now, we’ll start the recorder using

To verify we can check if our recorder is running or not using aws configservice describe-configuration-recorder-status which shares the real-time json format of our recorder.

Creating a Custom Config Rule

Creating a Custom Config Rule lets you define compliance requirements specific to your environment. For example, making sure that no S3 bucket is publicly accessible is a common requirement for most of the organization. AWS Config provides managed rules like S3_BUCKET_PUBLIC_READ_PROHIBITED, which continuously evaluate bucket policies for compliance. If any bucket is misconfigured and allows public access, the rule flags it immediately.

To create such a rule, start by defining an IAM role for an AWS Lambda function. This role, called LambdaExecutionRole, allows the function to access AWS services securely. The role uses the sts:AssumeRole action, granting Lambda the necessary trust relationship. Here’s how we can define this role in Terraform:

Next, we assign a policy named “LamdaPolicy” to the role, granting it permission to read S3 bucket policies and access logs.

This policy makes sure that the Lambda function can evaluate the compliance of S3 buckets effectively. After this, we create an S3 bucket named secure-test-bucket using 

After applying the resources we can check their compliance status using AWS CLI code as aws configservice get-compliance-details-by-config-rule --config-rule-name s3-bucket-public-read-prohibited

Resolving a Drift Alert

When a drift is detected, AWS Config generates alerts that prompt action. For example, if an EC2 security group is modified to allow unrestricted SSH access, the alert notifies administrators of the deviation. To start with, let's first look at the security group and config rules using

Now, to simulate a drift, we add a rule that allows unrestricted SSH access

Verifying drift detection aws configservice get-compliance-details-by-config-rule --config-rule-name ec2-restricted-ssh

Remediation involves updating the security group to restore the intended configuration by

Automating Drift Responses

AWS Systems Manager and AWS Lambda can create workflows to automatically fix drifts. For example, AWS Systems Manager can detect a drift in a CloudFormation stack and automatically reapply the original template to restore the intended state. Similarly, AWS Lambda functions can respond to specific events, such as reversing an unauthorized change to a security group, ensuring a quick and consistent resolution without any intervention from your end.

To automate the resolution of drifts, we begin by creating an IAM role for the Lambda function using Terraform. Here's how we can define this role:

Next, we create a Lambda function to generate a drift detection event in AWS config.

Here is the code for checking and reverting the security group drift by ensuring that the SSH access is limited to 203.0.113.0/24

To add the lambda function in our terraform configuration we add

to invoke on rule evaluation.

Now if we trigger a drift by allowing open SSH access by changing 203.0.113.0/24 to 0.0.0.0/0. AWS Config will detect this drift and the lambda function will automatically revoke open SSH access by replacing it with the restricted IP range 203.0.113.0/24

To verify if the security group ID is updated aws ec2 describe-security-groups --group-id sg-004c3e7fb478392ed

Now, managing drift can be quite hectic. From enabling AWS Config, creating rules, monitoring for drifts, and resolving issues manually or through automation, each step requires meticulous attention to detail. Here comes Firefly to make things easier.

Firefly helps you prevent drift by automatically detecting drifts and misconfigurations, making it easier to keep your cloud environment consistent. With Firefly, you can monitor drifts, view change history, and roll back to previous settings if needed.

Monitor Drift in Firefly's Dashboard

Firefly's centralized dashboard offers a clear and comprehensive view of your infrastructure, enabling you to monitor drift effectively. The dashboard provides detailed insights into your cloud resources, such as the percentage of unmanaged assets, instances of drift, and the status of various IaC stacks. The visual representation simplifies identifying where drift has occurred and ensures your infrastructure is in line with your IaC.

You can quickly pinpoint issues like drift, unmanaged resources, or potential cost savings and take swift action to address them. This dashboard not only shows the current state but also tracks changes over time, aiding in maintaining a consistent and secure cloud environment.

By clicking on a drifted data source, you can see exactly what has changed compared to your IaC. This provides a detailed breakdown of the drift, highlighting differences in properties, tags, and other key configurations. You can either codify these changes back into your IaC or revert the resource to match the original configuration, ensuring that your infrastructure remains consistent and secure.

Codify Your Drift

Firefly enables you to automatically generate codified versions of your infrastructure, including unmanaged resources. As shown in the image below, Firefly presents a detailed view of the infrastructure's code, highlighting key attributes such as instance type, storage settings, and security groups. This codified view makes it simple to incorporate any unmanaged resources back into your IaC setup.

You can directly export this codified configuration, create pull requests, or integrate it into your existing IaC tools like Terraform, Pulumi, or Ansible. This feature ensures that your infrastructure is always aligned with your desired state, reducing the risk of drift and making it easier to manage and scale your cloud environment.

Firefly simplifies cloud management by helping you detect and correct drift. With features like monitoring, detailed codification, and instant alerts, you can keep your infrastructure aligned with your IaC. It makes it easy to manage your cloud environment, preventing costly mistakes using a single platform. 

FAQs

Which AWS service allows you to monitor configuration changes for all AWS resources?

AWS Config monitors resource configurations so that you can evaluate the recorded configurations against the desired secure configurations. 

How often does AWS config update?

There are two frequencies at which AWS Config can deliver configuration items: continuous and periodic. Continuous recording records and delivers configuration changes whenever a change occurs.

Which tool is used for monitoring in AWS?

With AWS CloudTrail, you can monitor your AWS deployments in the cloud by getting a history of AWS API calls for your account, including API calls made via the AWS Management Console, the AWS SDKs, the command line tools, and higher-level AWS services.

What role does tagging play in managing configuration drift?

Consistent resource tagging is critical in identifying and managing configuration drift. Tags help categorize and track resources, making it easier to detect unauthorized changes. For instance, tags like Environment: Production or Owner: Team A allow teams to filter resources quickly and compare their current state against the desired configuration defined in governance policies.

What is GuardDuty AWS?

Amazon GuardDuty is a threat detection service that continuously monitors your AWS accounts and workloads for malicious activity and delivers detailed security findings for visibility and remediation.