We launched WhaTap URL Monitoring in April 2019, and we know that many of our customers have been using it since it became free on May 1, 2022. In this article, I will show you how to build your own URL monitoring service and compare it to WhaTap URL Monitoring.
URL monitoring is a monitoring service that is intended to determine whether access to a website is normal from the end user's perspective. In other words, it is a service in which a monitoring agent performs access to a site, collects access results, stores them, and sends notifications.
There are three main reasons why I implemented my own URL monitoring service, even though the company has one, there are many other URL monitoring services, and there is open source.
The curiosity of an engineer
I think being an engineer is a profession of endless curiosity. I knew in my head how to collect data, but when I implemented it, I wondered if it would work.
Comparison with WhaTap URL Monitoring
We wanted to know how our service compares to building and using your own.
Multiple types of notifications
WhaTap URL Monitoring offers a variety of notification modules, including SMS, EMAIL, Slack, Telegram, and JANDI, but I wanted to customize the notifications to my liking.
Architecture
Get the content from url_agent.json for sending data to Slack and InfluxDB.
Get a list of URLs from url.json, along with the maximum connection attempt time and notification conditions. URL Monitoring sends a Request to the URLs it got. It receives the Response and sends the StatusCode and ResponseTime to InfluxDB. It sends a message to Slack when the notification conditions are met.
Create a Goroutine that reads a list of URLs from url.json and goes through a loop, sending requests to the URLs, sending notifications and storing data.
Send a request to the URL with the TimeOut setting, and check how long it takes.
(default Timeout 5s)
Create a GoRoutine to get the StatusCode and put the data into Influxdb, and a Goroutine to check the notification conditions and send it to Slack.
(Ignore the task if it is not set in url_agent.json)
url_agent.json
If you have InfluxDB and Slack App set up, enter those values.
If they are empty, the InfluxDB or Slack module will not work.
url.json
Excute
Results
Output
Can view data in the InfluxDB UI
(Unknown → Represented as StatusCode 999)
Can view notifications in Slack
Comparison to WhaTap URL Monitoring
Compared to WhaTap URL Monitoring, it has the following characteristics.
Custom URL Monitoring
WhaTap URL Monitoring
Degree of Freedom
High (Can do more with code changes)
Medium (Cannot segment notification conditions)
Manageability
High (Direct verification in case of agent failure)
Low (Verification by WhaTap in case of agent failure)
Cost
Accrues infrastructure cost
Free
Closing thoughts
In this content, I tried to build a URL monitoring service, which is a monitoring service for determining whether a website is being accessed properly or not. Although it seems simple to build a URL monitoring service, it was more complicated than I thought, and I admire the company's decision to open such a service for free. I highly recommend WhaTap URL monitoring service to those who are still using open source or actual URL monitoring services.