Use Prometheus+Grafana to monitor the Internet backbone connectivity quality

Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.Prometheus is primarily based on a PULL model, in which the Prometheus server has a list of targets it should scrape metrics from. The pull protocol is HTTP based and simply put, the target returns a list of "<metric> <value>". 


Grafana is the leading graph and dashboard builder for visualizing time series infrastructure and application metric and it supports Prometheus as the data source, in our project here we leverage Grafana as the monitoring dashboard.


Exporters is the plugin Prometheus uses to collect the data for different components, and Blackbox_exporter is one of them, it supports to collect HTTP, HTTPS (via the http prober), DNS, TCP socket and ICMP monitoring data and it is a Prometheus official supported exporter. The link below lists all of the exports Prometheus currently supports.


https://prometheus.io/docs/instrumenting/exporters/

 

In our case, we will use blackbox_exporter to collect the ICMP ping data and send to the Prometheus engine, then use Grafana to visualize the collected data.

1.Install Grafana

a) Run the command below to install Grafana first.

cd /optwget https://grafanarel.s3.amazonaws.com/builds/grafana-latest-1.x86_64.rpmyum install grafana-latest-1.x86_64.rpm

b) The key files are located at the path as below.

File CategoryFile PathGrafana Binary file/usr/sbin/grafana-serverStartup Script /etc/init.d/grafana-serverEnvironment Parameter /etc/sysconfig/grafana-serverConfiguration File /etc/grafana/grafana.iniGrafana Log /var/log/grafana/grafana.log

c) Grafana is by default listening on port 3000 and the default user account and password is admin

Service Name:       grafana-server

Default Port:          3000

User Account:       admin

Password:             admin

d) Run the command below to start Grafana and enable auto-run.

systemctl daemon-reloadsystemctl start grafana-serversystemctl status grafana-serversystemctl enable grafana-server.service

2. Install Prometheus

a) We use the stable version 1.7.1 in our system, launch the command below for installation & start in the background.

cd /optwget https://github.com/prometheus/prometheus/releases/download/v1.7.1/prometheus-1.7.1.linux-amd64.tar.gztar xvfz prometheus-*.tar.gzcd prometheus-*./Prometheus &

b) Prometheus has its own HTTP service and can be accessed via TCP port 9090.

3.Install Blackbox_exporter

a) Install Golang as Blackbox exporter is written in GO

cd /optwget https://www.golangtc.com/static/go/1.7.6/go1.7.6.linux-amd64.tar.gztar -C /usr/local -xzf go1.7.6.linux-amd64.tar.gzexport GOROOT=/usr/local/goexport GOPATH=/usr/binexport PATH=$PATH:$GOROOT/bin

b) Run "go version" to make sure Golang is installed successfully.

    

       [root@ctum2-vmo-214117022 blackbox_exporter]# go version

       go version go1.7.6 linux/amd64

c) Get the source code for blackbox_exporter and build it.

yum install gitcd /usr/binmkdir srccd srcmkdir github.comcd github.commkdir prometheuscd /usr/bin/src/github.com/prometheusgit clone https://github.com/prometheus/blackbox_exporter.gitcd blackbox_exporter/make

d) When you see the output below, it means the blackbox_exporter  is installed successfully.

[root@ctum2-vmo-214117022 blackbox_exporter]# make >> checking code style >> building binaries  >   blackbox_exporter >> running tests ok      github.com/prometheus/blackbox_exporter 2.210s

e) Run blackbox_exporter by following the command below

cd /usr/bin/src/github.com/prometheus/blackbox_exporter./blackbox_exporter &

4. Add Prometheus Job

Navigate to the Prometheus folder and configure prometheus.yml as below.

  – in the static_configs part, each targets configure one monitoring point, group specifies for the ISP provider and state specifies the city where the monitoring point is located. You can change the configs as per your own.

  – at the end of the file "replacement: 127.0.0.1:9115 " means the data comes from local blackbox_exporter.

# my global configglobal:  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.  # scrape_timeout is set to the global default (10s).  # Attach these labels to any time series or alerts when communicating with  # external systems (federation, remote storage, Alertmanager).  external_labels:      monitor: 'codelab-monitor'# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.rule_files:  # - "first.rules"  # - "second.rules"# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.scrape_configs:  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.  - job_name: 'prometheus'    # metrics_path defaults to '/metrics'    # scheme defaults to 'http'.    static_configs:      - targets: ['localhost:9090']    metrics_path: /probe    params:      module: [icmp]  #ping    static_configs:      - targets: ['219.150.32.132']        labels:          group: 'CT'          state: '天津'      - targets: ['219.148.204.66']        labels:          group: 'CT'          state: '辽宁'      - targets: ['222.173.113.3']        labels:          group: 'CT'          state: '山东'    relabel_configs:      - source_labels: [__address__]        regex: (.*)(:80)?        target_label: __param_target        replacement: ${1}      - source_labels: [__param_target]        regex: (.*)        target_label: ping        replacement: ${1}      - source_labels: []        regex: .*        target_label: __address__        replacement: 127.0.0.1:9115  # Blackbox exporter.                                         

 Browsing from the Prometheus portal, by filter by "probe_duration_seconds{group="CT"}" and click on the Execute button, we can see the date collected to Prometheus.

5. Configure Grafana Data Source

a) From the Grafana home page, click on "Add data source"

b) Config the name as prometheus , type as Prometheus, url as http://localhost:9090, access as proxy because our Prometheus is running on the local host.

c) Click the Add button, when it pops up the success information then click on "Save & Test" to continue.

6. Add the ChinaMap Panel Plugin

Grafana only has WorldMap Panel Plugin to illustrate the received date from Promethus but the province and city name are all in English, I have developed our own ChinaMap plugin based on Baidu Echarts to suit in our scenarios. 

The source code is hosted on the github site : https://github.com/xianl/Grafana_Plugin_ChinaMap 

a) Deploy ChinaMap Panel Plugin to Grafana and restart the Grafana service.

cd /var/lib/grafana/plugins/git clone https://github.com/xianl/Grafana_Plugin_ChinaMap.gitsystemctl restart grafana-server

b) Refresh the web page , in the Installed Panels part, you will see ChinaMap as installed.

c) Create a new dashboard and insert a new ChinaMap panel. Click on the Panel Title to start the Edit menu.

d) In the General tab, set the Height to a high value like 800px.

e) In the Metrics tab, set Query as "probe_duration_seconds{group="CT"}*1000" , Legend format as {{state}} , Panel data source as prometheus.

f) Switch to the ChinaMap tab to change some other configuration as what you want. After the configuration is saved, you will see the map similar as below with the data  from different provinces visualized on it.