Outils pour utilisateurs

Outils du site


admin:monitoring:start

Installation et configuration du monitoring

Ici on parle de l'installation du serveur maître et de l'IHM web. Pour l'ajout du monitoring sur une machine, consulter cette page.

Prérequis

  • enregistrement DNS,
  • reverse proxy + TLS sur le bastion,
  • serveur httpd fonctionnel en local.

Installation du serveur maître

Doc officielle : sur le site web.

Icinga2 est packagé et le paquet debian aide sur qq étapes. Pour l'installation :

apt install icinga2 vim-icinga2 monitoring-plugins

Icinga doit tourner :

systemctl status icinga2

Installation du greffon vim :

apt install vim-addon-manager
vim-addons install icinga2

Mise en place de la base de données :

apt install mariadb-server mariadb-client
apt install icinga2-ido-mysql

Les réponses que l'installeur peut attendre : oui à la préconfiguration puis vide au mot de passe. Si besoin les details d'accès à la BDD (user,password,database) seront dans /etc/icinga2/features-{en,dis}abled/ido-mysql.conf

Activation des fonctions utiles :

icinga2 feature enable ido-mysql
icinga2 feature enable command

Redémarrage pour la prise en compte :

systemctl restart icinga2

Installation de l'IHM web

Doc officielle : sur le site web.

L'interface web d'Icinga2 est fort séparée du cœur d'exécution (qui sera commun aux maîtres et esclaves). On parle d'Icingaweb2. Celui-ci échange avec Icinga2 via la base de données et via une possibilité de commande locale. On installera également icingacli qui aide sur la gestion des modules d'Icingaweb2.

Icingaweb2 est packagé et le paquet debian aide sur qq étapes. Pour l'installation :

  
apt install icingacli icingaweb2 icingaweb2-module-monitoring php-mysql php7.0-intl
usermod -a -G icingaweb2 www-data
  

Configuration nginx

Pour rappel on suppose que le reverse proxy fait déjà le job. Sinon, la documentation est ici.

On commence par partir d'une conf de base :

/etc/nginx/sites-available/icinga.chapril.org
server {
    listen 80;
 
    access_log /var/log/nginx/icinga.chapril.org.access_log;
    error_log /var/log/nginx/icinga.chapril.org.error_log;
 
    server_name icinga.chapril.org;
 

Puis on enrichi la conf à l'aide de icingacli :

icingacli setup config webserver nginx --document-root /usr/share/icingaweb2/public >> /etc/nginx/sites-available/icingaweb2.conf

Il faut ensuite fermer l'accolade finale, reformatter, et re-définir fastcgi_pass au profit de

/etc/nginx/sites-available/icingaweb2.conf
fastcgi_pass unix:/run/php/php7.0-fpm.sock

Il faut aussi définir dans /etc/php/7.0/fpm/php.ini

/etc/php/7.0/fpm/php.ini
  date.timezone = Europe/Paris

Ensuite on active le site

ln -s /etc/nginx/sites-available/icinga.chapril.org /etc/nginx/sites-enabled/icinga.chapril.org
nginx -t
systemctl reload nginx

Finalisation de l'installation

Le setup d'icinga va exiger un jeton d'authentification, qu'on génère ainsi :

icingacli setup token create
=> The newly generated setup token is: deadbeef

Il suffit ensuite d'ouvrir https://icinga.chapril.org/icingaweb2/setup dans un butineur, et de placer le jeton deadbeef…

Si vous avez oubliez le jeton :
icingacli setup token show

Ensuite laissez vous faire. Il faudra configurer l'accès à la base de données d'Icinga2 pour lire l'état du monitoring, et configurer une base propre à Icingaweb2 pour son fonctionnement propre.

Si besoin, création d'une base icingaweb2 :

=(^-^)=root@admin:~# mariadb
MariaDB [(none)]> CREATE DATABASE icingaweb2;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icingaweb2.* TO 'icingaweb2'@'localhost' IDENTIFIED BY 'icingaweb2';
Query OK, 0 rows affected (0.01 sec)

Après tout ça, votre Icingaweb2 devrait être fonctionnel. Si le monitoring n'apparaît pas dans l'IHM web, il faut peut être activer le module :

icingacli module enable monitoring

Configuration du serveur maître

Doc officielle : sur le site web.
Remarque : globalement on n'utilisera que des fqdn locaux au cluster, dont on est sûr qu'il ne résolvent que sur des ip privées. Ici le nom de la machine est admin.cluster.chapril.org.

Le node wizard d'Icinga2 prend ici tout en mains :

=(^-^)=root@admin:/etc/icinga2# icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.



Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specify the common name (CN) [admin.cluster.chapril.org]: 
Checking for existing certificates for common name 'admin.cluster.chapril.org'...
Certificates not yet generated. Running 'api setup' now.
information/cli: Generating new CA.
information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
information/cli: Generating new CSR in '/etc/icinga2/pki/admin.cluster.chapril.org.csr'.
information/base: Writing private key to '/etc/icinga2/pki/admin.cluster.chapril.org.key'.
information/base: Writing certificate signing request to '/etc/icinga2/pki/admin.cluster.chapril.org.csr'.
information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/admin.cluster.chapril.org.crt'.
information/pki: Writing certificate to file '/etc/icinga2/pki/admin.cluster.chapril.org.crt'.
information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
Generating master configuration for Icinga 2.
information/cli: Adding new ApiUser 'root' in '/etc/icinga2/conf.d/api-users.conf'.
information/cli: Enabling the 'api' feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
Please specify the API bind host/port (optional):
Bind Host []: 
Bind Port []: 
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

Il faut ensuite passer à la configuration à proprement parler.

Pour donner un aperçu :

  • Le fichier /etc/icinga2/zones.conf défini les zones. Chaque machine du cluster doit s'y trouver.
  • Les configurations génériques sont dans une zone globale et situés ici : /etc/icinga2/zones.d/global-templates/. A priori ce qui vous intéressera le plus sera :
    • zones.d/global-templates/groups.conf
    • zones.d/global-templates/services/*
  • les configurations propres aux machines sont dans zones.d/master :
    tree /etc/icinga2/zones.d/master/
        /etc/icinga2/zones.d/master/
        ├── cluster
        │   ├── admin.conf
        │   ├── bastion.conf
        │   ├── bling.conf
        │   ├── dns.conf
        │   ├── mail.conf
        │   ├── pouet.conf
        │   └── sympa.conf
        ├── coon.conf
        ├── galanga
        │   └── icinga2.conf
        └── maine.conf

Installation d'un bot IRC

On va utiliser un l'API d'icinga2. Pour activer l'API icinga2 :

# icinga2 feature enable api

Pour installer un wrapper IRC pour python :

# apt install python3-irc

Installation du bot

Le code du bot est le suivant, à copier dans /usr/local/bin/chatonic.py et à donner les droits d'exécution :

/usr/local/bin/chatonic.py
#! /usr/bin/env python3
# 
# Simple IRC Bot statusing icinga2 via its API.
#
# François Poulain <fpoulain@metrodore.fr>
#
#
# Based on example program using irc.bot.
# from
# Joel Rosdahl <joel@rosdahl.net>
# https://github.com/jaraco/irc/raw/master/scripts/testbot.py
 
"""A simple IRC Bot statusing icinga2.
 
It is based on TestBot example bot that uses the SingleServerIRCBot class from
irc.bot.  The bot enters a channel and listens for commands in private messages
and channel traffic.  Commands in channel messages are given by prefixing the
text by the bot name followed by a colon.  Periodically, the bot read Icinga2
status and report all new KO services.
 
The known commands are:
    """
commands = {
        "ack" : {
            "help": "Acknowledge a given service.",
            },
        "recheck" : {
            "help": "Recheck a given service or all services.",
            "synonyms" : [r"refresh"],
            },
        "list" : {
            "help": "List all KO services.",
            },
        "leave" : {
            "help": "Disconnect the bot.  The bot will try to reconnect after 300 seconds.",
            },
        "mute" : {
            "help": "Mute the bot (no more status report).  The bot will unmute after 1 hour or after receiving any command.",
            "synonyms" : [r"fuck", r"chut", r"couché", r"sieste", r"t(a|o)\s*g(ueu|o)le"],
            },
        "die" : {
            "help": "Let the bot cease to exist.",
            },
        "help" : {
            "help": "Print help.",
            },
        }
 
settings = {
        "ircsrv":  "irc.freenode.net",
        "ircchan": "#april-chapril",
        "ircport": 6667,
        "ircnick": "chatonic",
 
        # see /etc/icinga2/conf.d/api-users.conf
        "icinga2user" : "root",
        "icinga2pass" : "bling",
        "icinga2ca"   : "/etc/icinga2/pki/ca.crt",
        "icinga2fqdn" : "admin.cluster.chapril.org",
        "icinga2port" : 5665,
        }
 
import irc.bot
import irc.strings
import re
import requests, json
 
class Icinga2ServiceManager:
    notifications= []
 
    def build_request_url (self, uri, params={}):
        # Since icinga2 wants « URL-encoded strings » but requests
        # seems to makes « application/x-www-form-urlencoded »,
        # so we munge params and shortcut urlencode.
        # See https://www.icinga.com/docs/icinga2/latest/doc/12-icinga2-api/#parameters
        url = 'https://{}:{}{}'.format(settings["icinga2fqdn"], settings["icinga2port"], uri)
        return requests.Request('GET', url, params=params).prepare().url.replace('+', '%20')
 
    def fetch_notifications (self):
        headers = {
                'Accept': 'application/json',
                'X-HTTP-Method-Override': 'GET'
                }
        data = {
                "attrs": [ "last_check_result" ],
                "filter": "service.state!=ServiceOK",
                }
        try:
            r = requests.post(self.build_request_url ("/v1/objects/services"),
                    headers=headers,
                    auth=(settings["icinga2user"], settings["icinga2pass"]),
                    data=json.dumps(data),
                    verify=settings["icinga2ca"])
            if (r.status_code == 200):
                new_notifications = [n for n in r.json()['results'] if n != None]
                news= [n for n in new_notifications  if n['name'] not in [nn['name'] for nn in self.notifications]]
                lost= [n for n in self.notifications if n['name'] not in [nn['name'] for nn in new_notifications ]]
                self.notifications = new_notifications
                return (lost, news)
        except:
            self.send("Unable to fetch from Icinga2")
            return (False,False)
 
    def ack_service (self, srv, comment, nick):
        # weird but needed:
        if comment == '':
            comment = ' '
            # /weird
        headers = {
                'Accept': 'application/json',
                'X-HTTP-Method-Override': 'POST'
                }
        data = {
                "author": nick,
                "comment": comment,
                }
        params = {
                "type"   : "Service",
                "filter" : ('service.__name=="{}"'.format(srv) if srv != None else "service.state!=ServiceOK"),
                }
        try:
            r = requests.post(self.build_request_url ("/v1/actions/acknowledge-problem", params=params),
                    headers=headers,
                    auth=(settings["icinga2user"], settings["icinga2pass"]),
                    data=json.dumps(data),
                    verify=settings["icinga2ca"])
            if r.status_code == 200:
                for a in r.json()['results']:
                    if a["code"] == 200.0:
                        self.send (a["status"])
                if srv != None and not r.json()['results']:
                    self.send("No result for service name « {} »".format(srv))
            else:
                self.send("{} for service name « {} »".format(r.text, srv))
        except:
            self.send("Unable to post to Icinga2")
 
    def recheck_service (self, srv):
        headers = {
                'Accept': 'application/json',
                'X-HTTP-Method-Override': 'POST'
                }
        params = {
                "type"   : "Service",
                "filter" : ('service.__name=="{}"'.format(srv) if srv != None else "service.state!=ServiceOK"),
                }
        try:
            r = requests.post(self.build_request_url ("/v1/actions/reschedule-check", params=params),
                    headers=headers,
                    auth=(settings["icinga2user"], settings["icinga2pass"]),
                    verify=settings["icinga2ca"])
            if r.status_code == 200:
                for a in r.json()['results']:
                    if a["code"] == 200.0:
                        self.send (a["status"])
                if srv != None and not r.json()['results']:
                    self.send("No result for service name « {} »".format(srv))
            else:
                self.send("{} for service name « {} »".format(r.text, srv))
        except:
            self.send("Unable to post to Icinga2")
 
class ChatTonic(Icinga2ServiceManager, irc.bot.SingleServerIRCBot):
    args= ''
    muted= False
    nick_suffix= ''
 
    def __init__(self, channel, nickname, server, port=6667):
        irc.bot.SingleServerIRCBot.__init__(self, [(server, port)], nickname, nickname, reconnection_interval=300)
        self.nick= nickname
        self.channel = channel
        self.connection.execute_every(30, self.refresh_notifications)
        self.refresh_notifications ()
 
    def suffix_nick (self, suffix):
        self.nick_suffix= suffix
        if self.connection.is_connected():
            self.connection.nick ('{}{}'.format(self.nick, suffix))
 
    def unmute (self):
        self.muted= False
        if self.notifications:
            self.suffix_nick ('[{}]'.format(len(self.notifications)))
        else:
            self.suffix_nick ('')
 
    def send (self, msg):
        if not self.muted and self.connection.is_connected():
            for line in msg.split('\n'):
                self.connection.privmsg(self.channel, line)
 
    def refresh_notifications (self):
        lost, news = self.fetch_notifications()
        if lost == False and news == False:
            return
        if self.notifications:
            self.suffix_nick ('[{}]'.format(len(self.notifications)))
        else:
            self.suffix_nick ('')
        for srv in lost:
            if srv != None:
                self.send("{} is OK".format(srv['name']))
        for srv in news:
            try:
                self.send("{}: => {}".format(srv['name'], srv['attrs']['last_check_result']['output']))
            except:
                self.send("{}: => No check result.".format(srv['name']))
 
    def on_nicknameinuse(self, c, e):
        c.nick( + "_")
 
    def on_welcome(self, c, e):
        c.join(self.channel)
 
    def on_privmsg(self, c, e):
        self.do_command(e, e.arguments[0])
 
    def on_pubmsg(self, c, e):
        if e.arguments[0].startswith ('!'):
            self.do_command(e, e.arguments[0][1:])
            return
        a = e.arguments[0].split(":", 1)
        if len(a) > 1 and a[0].lower() == self.connection.get_nickname().lower():
            self.do_command(e, a[1].strip())
        return
 
    def do_cmd (self, s):
        self.args= None
        l = s.split(' ', 1)
        cmd= l[0]
        if len(l)>1:
            self.args= l[1].strip()
 
        for k,v in commands.items():
            if cmd == k and hasattr(self, "do_"+cmd):
                return getattr(self, "do_"+cmd)
            if "synonyms" in v:
                for s in v["synonyms"]:
                    if re.match(s, cmd) and hasattr(self, "do_"+k):
                        return getattr(self, "do_"+k)
        return False
 
    def do_help (self, c, e):
        for k,v in commands.items():
            self.send("{} -- {}".format(k, v["help"]))
            if "synonyms" in v:
                self.send("   synonyms: {}".format(", ".join(v["synonyms"])))
 
    def do_die (self, c, e):
        print('Ok master, I die. Aaargh...')
        self.die ()
 
    def do_leave (self, c, e):
        self.disconnect()
 
    def do_list (self, c, e):
        for srv in self.notifications:
            try:
                self.send("{}: => {}".format(srv['name'], srv['attrs']['last_check_result']['output']))
            except:
                self.send("{}: => No check result.".format(srv['name']))
 
    def do_ack (self, c, e):
        if self.args == None:
            self.send(e.source.nick + ": usage: !ack <service|all> [: comment]")
            return
        l = self.args.split(':', 1)
        srv, comment= l[0].strip(), ''
        if len(l)>1:
            comment= l[1].strip()
        if srv == 'all':
            srv= None
        self.ack_service(srv, comment, e.source.nick)
 
    def do_recheck (self, c, e):
        if self.args == 'all':
            self.args == None
        self.recheck_service(self.args)
        self.connection.execute_delayed(1, self.refresh_notifications)
 
    def do_mute (self, c, e):
        self.muted= True
        self.suffix_nick ('[zZz]')
        self.connection.execute_delayed(3600, self.refresh_notifications)
 
 
    def do_command(self, e, cmd):
        nick = e.source.nick
        c = self.connection
 
        if self.do_cmd (cmd):
            self.unmute ()
            (self.do_cmd (cmd)) (c, e)
        else:
            self.send(nick + ": Not understood: " + cmd)
 
def main():
    bot = ChatTonic(settings['ircchan'], settings['ircnick'], settings['ircsrv'], settings['ircport'])
    bot.start()
 
if __name__ == "__main__":
    main()

Intégration au système et démarrage

/lib/systemd/system/chatonic.service
[Unit]
Description=Chatonic IRC/Icinga2 bot
 
[Service]
ExecStart=/usr/local/bin/chatonic.py
Restart=always
RestartSec=60min
 
[Install]
WantedBy=multi-user.target
Alias=chatonic.service
# systemctl enable chatonic.service
Created symlink /etc/systemd/system/chatonic.service → /lib/systemd/system/chatonic.service.
Created symlink /etc/systemd/system/multi-user.target.wants/chatonic.service → /lib/systemd/system/chatonic.service.
# systemctl start chatonic.service
admin/monitoring/start.txt · Dernière modification: 2018/05/05 10:03 par qgibeaux