와탭랩스 블로그 오픈 이벤트 😃
자세히 보기
Tech
2023-06-26
Creating Your Own Simple URL Monitoring

Introduction

We launched WhaTap URL Monitoring in April 2019, and we know that many of our customers have been using it since it became free on May 1, 2022. In this article, I will show you how to build your own URL monitoring service and compare it to WhaTap URL Monitoring.

What is URL monitoring?

URL monitoring is a monitoring service that is intended to determine whether access to a website is normal from the end user's perspective. In other words, it is a service in which a monitoring agent performs access to a site, collects access results, stores them, and sends notifications.

Why I built URL monitoring

There are three main reasons why I implemented my own URL monitoring service, even though the company has one, there are many other URL monitoring services, and there is open source.

The curiosity of an engineer

I think being an engineer is a profession of endless curiosity. I knew in my head how to collect data, but when I implemented it, I wondered if it would work.

Comparison with WhaTap URL Monitoring

We wanted to know how our service compares to building and using your own.

Multiple types of notifications

WhaTap URL Monitoring offers a variety of notification modules, including SMS, EMAIL, Slack, Telegram, and JANDI, but I wanted to customize the notifications to my liking.

1. How to Deploy URL Monitoring

Architecture

main

Get the content from url_agent.json for sending data to Slack and InfluxDB.

  • Send notifications and store data if empty X

Get a list of URLs from url.json, along with the maximum connection attempt time and notification conditions. URL Monitoring sends a Request to the URLs it got. It receives the Response and sends the StatusCode and ResponseTime to InfluxDB. It sends a message to Slack when the notification conditions are met.

2. Logic

Create a Goroutine that reads a list of URLs from url.json and goes through a loop, sending requests to the URLs, sending notifications and storing data.

main

Send a request to the URL with the TimeOut setting, and check how long it takes.

(default Timeout 5s)

main

 Create a GoRoutine to get the StatusCode and put the data into Influxdb, and a Goroutine to check the notification conditions and send it to Slack.

(Ignore the task if it is not set in url_agent.json)

main

How to install and use

Source
  • Navigate to the appropriate repository.

[GitHub-cjm2921401]

Download
  • Navigate to the build folder and download the zip file for your operating system.
main
Setting
  • Unzip the downloaded zip file and set up the json file.

url_agent.json

main

If you have InfluxDB and Slack App set up, enter those values.

If they are empty, the InfluxDB or Slack module will not work.

url.json

main
  • name : Alias
  • url : Url you want to call (https://www.naver.com) If https:// or http:// does not exist, create url with http://를 appended
  • timeout : Maximum connection attempt time
    • max, default : 5s
  • alert
    • 4xx : Notify when there is an error in the 400s
    • 5xx : Notified when there is an error in the 500s
    • unknown : Notified when an error occurs other than 1xx~5xx

Excute

  • Execute mayf_urlagent or mayf_urlagent.exe

Results

Output

  • Collected once per minute and sent to InfluxDB
main

Can view data in the InfluxDB UI

(Unknown → Represented as StatusCode 999) 

main

Can view notifications in Slack 

main

Comparison to WhaTap URL Monitoring

Compared to WhaTap URL Monitoring, it has the following characteristics.

Custom URL Monitoring

WhaTap URL Monitoring

Degree of Freedom

High (Can do more with code changes)

Medium (Cannot segment notification conditions)

Manageability

High (Direct verification in case of agent failure)

Low (Verification by WhaTap in case of agent failure)

Cost

Accrues infrastructure cost

Free

Closing thoughts

In this content, I tried to build a URL monitoring service, which is a monitoring service for determining whether a website is being accessed properly or not. Although it seems simple to build a URL monitoring service, it was more complicated than I thought, and I admire the company's decision to open such a service for free. I highly recommend WhaTap URL monitoring service to those who are still using open source or actual URL monitoring services.

 

와탭 모니터링을 무료로 체험해보세요!