Installing and configuring CloudWatch Agent

Connecting the EC2 Instance via EC2 Instance Connect

• Launch an EC2 instance with the following config:
  • Name: <ec2-name>
  • AMI: Ubuntu
  • Instance type: t2.micro
  • Key pair (create one)
    • Name: <key-pair-name>
    • Type: RSA
    • Format: .pem
  • Network settings
    • Allow SSH traffic from My IP
• Create an IAM role
• Ensure the Instance state is running
• Attach the IAM role to the instance: actions -> security -> modify IAM role, click Update IAM role

Installing the CloudWatch Agent

• Download the CloudWatch Agent Package (ssh into your instance and run the command): sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
• Install the CloudWatch Agent: sudo dpkg -i ./amazon-cloudwatch-agent.deb

Configuring the CloudWatch Agent to Send Custom Metrics to CloudWatch

• Edit the agents config file: sudo vi /opt/aws/amazon-cloudwatch-agent/bin/config.json
• Paste the follow:

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
  },
  "metrics": {
    "append_dimensions": {
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    },
    "metrics_collected": {
      "disk": {
        "measurement": ["used_percent"],
        "metrics_collection_interval": 60,
        "resources": ["/"]
      },
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      }
    }
  }
}

• Save and close: :wq!
• Stop the CloudWatch agent: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
• Start the CloudWatch agent: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
• Validate agent is active (start the agent, check the logs and review the console for server log data and metrics)
  • sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
• Confirm if the custom metrics are being sent to CloudWatch: cloudwatch -> metrics -> all metrics -> CWAgent

Creating a CLoudWatch Alarm

Create an Amazon EC2 Instance

• Name: <ec2-name>
• AMI: Ubuntu
• Instance type: t2.micro
• Key pair (create one)
  • Name: <key-pair-name>
  • Type: RSA
  • Format: .pem
• Network settings: Allow SSH from My IP

Create a CloudWatch Alarm

• Navigate to CloudWatch and click Create alarm under All Alarms
• Select metric: ec2 -> per-instance metrics -> CPUUtilisation metric
• Conditions: static, greater/equal, than 50, 1 out 5
• click Next
• In EC2 action: in alarm, reboot this instance
• Add name and description
• Click Next
• Click Create alarm

Stress Testing your CloudWatch Alarm

• Simulate high CPU load called “stress” by SSHing into the EC2 instance and running these commands
  • sudo apt update -y
  • sudo apt install stress -y
  • stress --cpu 50 --timeout 5m
• Check the alarm in EC2 instances console

Create a CloudWatch Dashboard

Create an EC2 Instance

• Name: <ec2-name>
• Type: t2.micro
• AMI: Ubuntu
• Key pair: do not create one
• Click Create instance
• Click on Actions (next to launch instances) and then select security and modify IAM role
• Click update IAM role

Create a CloudWatch Dashboard

• Go to Instances within EC2 and select the monitoring tab
• Click Add to dashboard and then click Create new in the dialog box
• Give it a title and then click Add to dashboard
• Go to CloudWatch via the AWS Management Console
• Click Create new dashboard, give it a name
• Add a widget with the following config
  • Data type: metrics
  • Widget type: line
  • Click Next
• Configure the metric you want to add
  • Select the AWS Namespace, eg EC2 and click per-instance metrics

Introduction to CloudWatch Logs Insights

Creating Sample CloudWatch Logs Data

• Navigate to the Lambda console and chose Author from scratch
• Function name: <lambda-function-name>
• Runtime: Python 3.12
• Execution role: chose an existing one or create a new one
• Click Create function
• Paste the following into the code editor

import boto3
import time
import random
logs_client = boto3.client('logs')
def lambda_handler(event, context):
    log_group_name = 'TestLogGroup'
    log_stream_name = 'TestLogStream'
    # Create log group if it doesn't exist
    try:
        logs_client.create_log_group(logGroupName=log_group_name)
    except logs_client.exceptions.ResourceAlreadyExistsException:
        pass  # Log group already exists
    # Create log stream if it doesn't exist
    try:
        logs_client.create_log_stream(logGroupName=log_group_name, logStreamName=log_stream_name)
    except logs_client.exceptions.ResourceAlreadyExistsException:
        pass  # Log stream already exists
    # Sample data for log entries
    http_methods = ['GET', 'POST', 'PUT', 'DELETE']
    request_urls = [
        '/home',
        '/api/user',
        '/login',
        '/products',
        '/checkout',
        '/cart',
        '/search?q=aws',
        '/api/order/123',
        '/api/product/567'
    ]
    user_agents = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15',
        'Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15A5341f Safari/604.1',
        'Mozilla/5.0 (Linux; Android 10; SM-G973F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.81 Mobile Safari/537.36'
    ]
    status_codes = [200, 201, 400, 401, 403, 404, 500]
    # Create log events
    log_events = []
    for i in range(20):
        log_event = {
            'timestamp': int(time.time() * 1000),
            'message': (
                f"{random.choice(http_methods)} "
                f"{random.choice(request_urls)} "
                f"{random.choice(status_codes)} "
                f"{random.choice(user_agents)}"
            )
        }
        log_events.append(log_event)
    # Put log events
    logs_client.put_log_events(
        logGroupName=log_group_name,
        logStreamName=log_stream_name,
        logEvents=log_events
    )
    return {
        'statusCode': 200,
        'body': 'Successfully created log events.'
    }

• Click Deploy
• Click Test and Create a new event
• click Test again

Verify the Log Group and Log Stream in CloudWatch Logs

• Navigate to CloudWatch and then click on Log groups
• Find the TestLogGry=oup and within it you should find TestLogStream
  • You can re-trigger the Lambda function to get more logs

Query Logs using CloudWatch Logs Insights

• In CloudWatch, go to Logs Insights
• Select your Log Group
• Run basic query
• Copy and paste the following query

fields @timestamp, @message
| parse @message /(?<http_method>\w+) (?<request_url>[^ ]+) (?<status_code>\d+) (?<user_agent>.+)/
| sort @timestamp desc
| limit 20

• Filter logs by status code

fields @timestamp, @message
| parse @message /(?<http_method>\w+) (?<request_url>[^ ]+) (?<status_code>\d+) (?<user_agent>.+)/
| filter status_code = 403
| sort @timestamp desc

• Counter requests by HTTP method

fields @timestamp, @message
| parse @message /(?<http_method>\w+) (?<request_url>[^ ]+) (?<status_code>\d+) (?<user_agent>.+)/
| filter http_method in ["GET", "PUT"]
| stats count(*) as request_count by http_method
| sort request_count desc

Setting Up CloudWatch Agent to Monitor Server Logs

Create an EC2 Instance

• Name: <ec2-name>
• AMI: Amazon Linux
• Type: t2.micro
• Key pair (proceed without one)
• Network settings (click Create security group)
  • Auto assign public IP: select enable
  • Firewall: tick Create security group
    • Ensure Allow SSH traffic from is set to anywhere
• Click Launch Instance
• Click on Actions (next to launch instances)
• Security > Modify IAM role and select the available IAM role

## Install CloudWatch Agent

• Select the instance created and click on Connect
• Run: sudo yum install amazon-cloudwatch-agent
• Type y and enter

Configure CloudWatch Agent

• Create a CloudWatch Agent configuration file: sudo vi /opt/aws/amazon-cloudwatch-agent/bin/cloudwatch-config.json
• Paste the following code in:

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/ping.log",
            "log_group_name": "PingLogGroup",
            "log_stream_name": "{instance_id}/ping.log",
            "timestamp_format": "%b %d %H:%M:%S"
          }
        ]
      }
    }
  }
}

• Save the file via :wq!
• Apply the config and start/stop the CloudWatch agent: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
  • sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/cloudwatch-config.json

Try to Ping Google to Generate Logs

• Save the output to /var/log/ping.log: ping google.com | tee -a /ping.log
• Verify logs: CloudWatch > Logs > Log Groups (find the group named PingLogGroup)

Capture Network Traffic Information with VPC Flow Logs to CloudWatch Logs

Launch an EC2 Instance

• Name: <ec2-name>
• AMI: t2.micro
• Key pair (create a new one)
  • Name: <key-pair-name>
  • Type: RSA
  • Format: .pem
• Network settings (click create security group)
  • Auto assign public IP: enabled
  • Firewall: tick Create security group
  • Ensure Allow SSH traffic from is checked and set to My IP
• Connect to instance: actions > connect
• Go to networking > change source/destination check
• Tick stop checkbox and save

Create a CloudWatch Log Group

• Navigate to CloudWatch and locate log groups under logs on the left
• Click Create log group
• Enter a name and description
• Click Create log group

Create a VPC Flow Log

• Navigate to VPC and select the VPC where the EC2 instance is running
• Click Actions > Create flow log
• Configurations of flow log:
  • Name: <flow-log-name>
  • Filter: all
  • Max aggregation interval: 1 min
  • Destination: send to CloudWatch Logs
  • Log group name: chose log group created previously
  • IAM role: select one or create one
  • Log record format: select AWS default format
  • Click Create flow log

Test the Flow Logs

• SSH into your EC2 instance
• Generate network traffic by pinging a website, eg ping google.com

Capture network traffic information with VPC flow logs to Amazon S3 Bucket

Set Up an Amazon S3 Bucket

• Bucket name: <unique-name>
• Click Create Bucket and take note of the ARN (Amazon Resource Name)

## Launch an EC2 Instance

• Name: <ec2-name>
• AMI: Amazon Linux
• Type: t2.micro
• Key pair (create a new one)
  • Name: <key-pair-name>
  • Type: RSA
  • Format: .pem
• Network settings (click create security group)
  • Auto assign public IP: enable
  • Firewall: tick create security group
  • Allow SSH traffic from is checked with My IP
• Click Launch Instance
• Wait for the instance state to be running
• Actions > Connect
• Networking > Change source/destination check
• Check the stop checkbox and click save

## Create a VPC Flow Log for S3

• Navigate to VPC
• Click on Actions and then select Create flow log
  • Name: <flow-log-to-s3-name>
  • Filter: All, Maximum aggregation interval: 1 min
  • Destination: select send to an s3 bucket
  • S3 bucket ARN: enter arn
• Click Create Flow Log

Generate Network Traffic and Verify Flow Logs

• ping google.com
• Review logs in S3