Introduction

Hi, I'm Jungmin Choi from the DevOps team at WhaTap Labs. Today, many organizations are moving from on-premises to cloud environments. There are many different cloud vendors helping organizations move to the cloud.

Among them, AWS (Amazon Web Service) is a leader in the cloud market with an overwhelming share. According to a survey conducted by the Korea Fair Trade Commission, AWS ranks first in the domestic cloud market with a market share of about 70 percent.

AWS is the largest cloud vendor that WhaTap Labs also uses. As a DevOps engineer, one of the most inconvenient things about using AWS was that there was no place to collect logs from various resources in one place. This inconvenience led us to develop AWS Log, which allows you to collect logs from AWS resources in WhaTap.

In this article, I will introduce the components and principles of AWS Log provided by WhaTap, especially how to collect logs in AWS environment.

AWS Log Architecture

The overall structure of AWS Log is as follows.

Most logs in AWS environments are collected in AWS CloudWatch LogGroup or S3.

A Whatap Forwarder is installed in your environment to collect logs that accumulate in AWS CloudWatch LogGroup or S3 and send them to the WhaTap environment.

The WhaTap Ingestion Server processes the data so that the logs can be monitored in Whatap and displayed to the user.

Users can easily check the logs of AWS resources in the same environment as the log monitoring previously provided by WhaTap

Whatap Forwarder

Whatap Forwarder is installed in your AWS environment via CloudFormation provided by WhaTap with AWS Lambda.

Collect

(1) AWS CloudWatch LogGroup

Whatap Forwarder uses a Lambda subscription filter in the AWS CloudWatch LogGroup to collect logs.

Set up a CloudWatch LogGroup Lambda subscription filter on the Whatap-Forwarder.

The json data (base64 encoded) containing the logs stored in the CloudWatch LogGroup will be sent to the Whatap-Forwarder.

Whatap-Forwarder runs, decodes, and sends the collected logs to the WhaTap environment.

(2) AWS S3

Whatap Forwarder collects logs using Bucket Notification of AWS S3 buckets and AWS Python SDK (GetObject).

• Set up Bucket Notification on the AWS S3 Bucket where AWS Resources are accumulated.
• When an Object(log) is created in the Bucket, json data containing the bucket and key of that Object in S3 is sent to the Whatap-Forwarder.

• • The user enters AWS IAM ROLE (AWS account and trust relationship) on the screen and requests the required action.
  • Move the request information from WhaTap Front to WhaTap Backend and send the request to AWS APIGATEWAY.
  • AWS APIGATEWAY runs the AWS Lambda and passes the information received to the user.
  • The Lambda in your AWS account receives the AWS IAM ROLE entered by the user, and performs tasks such as setting up a lookup.
  Wrapping up In this article, I have introduced the log collection principle and configuration principle of AWS Log, a new service of WhaTap. As a Devops engineer, it is really meaningful to develop a feature that is necessary for operations and productize it to help many Devops engineers, operators, and developers. If you want to collect various logs from your AWS environment at once, I recommend using WhaTap AWS Log.
• [AWS Log Monitoring Guide]