In the case of the webhook function provided by Watap Notifications, it's not just about receiving notifications from other messengers

It can also be used for automation in conjunction with various programs.

screenplay

Company A's servers are servers with zfs configured on disk.

Recently, it was confirmed that a frequent phenomenon in these servers is a phenomenon where delays occur on the server several times a day due to a sudden surge in I/O wait indicators. The person in charge of the server confirmed that this phenomenon is caused by zed, which is one of the modules of zfs, and that the server returns normally when the zed process is restarted. After checking the details, the person in charge found what kind of server it was every time an I/O wait occurred, then connected to and restarted the server, and began to worry about whether automation rather than repetitive work was possible.

Solution

I thought about how to make automation easier in a scenario situation, and after receiving a webhook from Jenkins, the Jenkins server executed a command on the server where the notification occurred.

Set reminders

First, set up an alert by specifying a threshold in the metric notification. For this example, we'll use the cpu_iowait indicator in the ServerBase category.

Next, set up a 3rd party plugin to receive notifications. Notification reception is set to prevent receiving other notifications through the receipt tag, and the notification reception settings to deliver webhooks to Jenkins are set to the webhook json type.



{

“pcode” :1234,

“projectName” :"Sample project”,

“time” :1577840400000,

“oid” :1234567890,

“oname” :"was-01" //optional,

“title” :"Alert Title”,

“message” :"Sample warning message” //Optional,

“level” :"Warning”,

“status” :"on”,

“metricName” :"cpu” //Optional,

“metricValue” :"99.99"//Optional,

“metricThreshold” :"90" //Optional”

}



In the case of webhook json, the format above is sent to the address set. While I was working, I had a problem.

“How do I forward the notification to the server where it occurred?”

Looking at the values sent to the webhook, I found that there is no separator to tell which server it is. Event messages can be used at this time.

Metrics notification settings allow you to include metric data tag information in notification messages. Using this, host_ip, which is one of the tags in the metric indicator, can be sent as a message by setting $ {host_ip} as follows. When set in this way, the ip will be sent as the “message” key value of the webhook json set above.

Event tags are attached to notifications set later and registered webhooks to prevent notifications from being sent to other monitoring notifications.

Jenkins settings

Connect to Jenkins and create a job. After that, Jenkins uses the webhook trigger to perform the following settings.

When set as above, the message received via webhook can be used as MESSAGE, which is a variable that can be used in Jenkins.

Using the set variables, the restart script can be simply written as follows:

ssh $message sudo systemctl restart zed

The preparations for automation are now complete.

Webhook address registration

If you look at Jenkins's webhook trigger description, after specifying the token value http://JENKINS_URL/generic-webhook-trigger/invoke It says that if you set a webhook with, you will receive a value.

As shown in the picture above, I specified ZED_RESTART, and in the case of a Jenkins address jenkins.example.com I'll assume so. The invocation jenkins webhook url at this time is as follows:

https://jenkins.example.com/generic-webhook-trigger/invoke?token=ZED_RESTART

Enter this address into the third-party plugin's Webhook JSON.

After registration, as shown in the picture below, when the receiving tag is registered, and when the receiving notification is set to off, only the I/O wait notification will be received and delivered to Jenkins.

corollary

In the case of webhooks, it can be linked with various programs.

If used with a service that uses a webhook body value, there are endless functions that can be used, such as automatically registering an issue in the issue tracker when a notification occurs, and automating operations as in the above scenario.