Prerequisites:

• A valid AWS Account with access to EC2 (VPC), S3, Athena, and Glue services.
• VPC Flow Logs must be enabled for the target VPC and delivered to an S3 bucket.
• An IAM User or Role with the required permissions (detailed below).
• A dedicated S3 bucket for storing VPC Flow Logs and Athena query outputs.
• Athena and Glue must be enabled in the same AWS region as the VPC Flow Logs.

How to Enable VPC Flow Logs (Required Configuration)

1. Sign in to the AWS Management Console.
2. Navigate to VPC → Your VPCs.
3. Select the target VPC for which monitoring needs to be enabled.
4. Open the Flow Logs tab and click Create flow log.
5. For Resource type, ensure VPC is selected.
6. Select the required Traffic type:
   • Accept – Records accepted traffic only
   • Reject – Records rejected traffic only
   • All – Records both accepted and rejected traffic (recommended)
7. For Destination, select Send to an Amazon S3 bucket.
   
   

   Do not select CloudWatch Logs.
8. Specify the ARN of the S3 bucket where VPC Flow Logs should be delivered.
9. Choose or create an IAM role with permissions to write Flow Logs to the S3 bucket.
10. Under Log format:
    • Ensure the log file format is plain text.
    • Use the following custom log format exactly as shown (mandatory):

            ${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} \
            ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
                

11. Ensure log delivery is partitioned by time.
    • Logs are expected to be partitioned every 24 hours (default AWS behavior).
    • Each partition corresponds to a unique year/month/day directory.
    s3://<bucket-name>/AWSLogs/<account-id>/vpcflowlogs/<region>/<year>/<month>/<day>/
12. Click Create flow log to complete the configuration.

Required AWS IAM Permissions and Policies

The following IAM policy provides the minimum required permissions to:

1. Discover VPC resources
   
2. Read flow log data
   
3. Query traffic metrics using Athena
   

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeFlowLogs"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<VPC_FLOW_LOG_BUCKET>",
        "arn:aws:s3:::<VPC_FLOW_LOG_BUCKET>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabase",
        "glue:CreateDatabase",
        "glue:GetTable",
        "glue:CreateTable",
        "glue:GetPartitions"
      ],
      "Resource": "*"
    }
  ]
}

Why are these permissions required?

• EC2 permissions – Discover VPCs, subnets, ENIs, and validate Flow Log configuration.
• S3 permissions – Verify Flow Log delivery and read log files for analysis.
• Glue permissions – Create and manage databases, tables, and partitions for Athena.
• Athena permissions – Execute SQL queries and retrieve traffic analytics.

Database, Table, and Storage Structure

Data Retention and Query Execution Details