ClementinaXXI admines

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Performance Monitoring ======


===== Intro a componentes ======

La [[https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=monitoring-using-performance-tool | performance monitoring tool]] colecciona métricas de GPFS y provee información de performance del sistema. 
Está habilitada por default e incluye //Collectors, Sensors // y // Proxies//.

==== Collector ====

Un collector soporta hasta 150 sensor nodes. Con un collector debería ser suficiente para nuestro sistema. Se pueden armar esquemas con más de un collector (multi-collector federation) por razones de escala y de tolerancia a fallas.


Podemos utilizar mmgt01 como Node Collector en el cluster de storage y vlmgt02 en el cluster cliente, ya que es buena práctica mantener servicios extra (como monitoreo) fuera de los quorum nodes. 

==== Sensors ====

El componente que recopila datos de performance de un nodo

==== Proxy ====

 Se corre un proxy por cada protocolo para recopilar métricas de ese protocolo.

===== IBM Storage Scale bridge for Grafana =====

Queremos incorporar [[https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=data-using-storage-scale-performance-monitoring-bridge-grafana | IBM Storage Scale bridge for Grafana]] para poder ver métricas de performance de GPFS en Grafana.

Para eso necesitamos configurar la performance monitoring tool de GPFS.

==== Resolución de dependencias ====

Necesitamos los paquetes
gpfs.gss.pmsensors
gpfs.gss.pmscollectors

Los encontramos únicamente en el storage node.

<code>[root@sdmgt01 ~]# dnf search gpfs.gss.pmcollector

=========================== Name Exactly Matched: gpfs.gss.pmcollector =====================
gpfs.gss.pmcollector.x86_64 : ZIMonCollector - an in-memory database for collecting and storing performance metrics.</code>

Descargamos los rpm con dnf.

<code>
[root@sdmgt01 ~]# dnf download gpfs.gss.pmcollector gpfs.gss.pmsensors
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register.

Last metadata expiration check: 17:29:09 ago on Thu 30 Jul 2026 06:16:49 PM -03.
(1/2): gpfs.gss.pmcollector-5.2.3-3.el9.x86_64.rpm                                                                              58 MB/s | 7.7 MB     00:00
(2/2): gpfs.gss.pmsensors-5.2.3-3.el9.x86_64.rpm                                                                                55 MB/s | 8.1 MB     00:00
</code>

Instalamos los paquetes en vlmgt02

<code>
[root@vlmgt02 gpfs_perf_testing]# dnf localinstall /data/admin/gpfs_perf_testing/gpfs.gss.pmcollector-5.2.3-3.el9.x86_64.rpm /data/admin/gpfs_perf_testing/gpfs.gss.pmsensors-5.2.3-3.el9.x86_64.rpm --disablerepo=Lenovo-Lenovo-OpenHPC-local,lenovo-hpc,lico-dep-local-library,lico-dep-local-standalone,lico-release
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register.

Local install repository for AppStream                                                                                          76 kB/s | 3.2 kB     00:00
Local install repository for BaseOS                                                                                             54 kB/s | 2.7 kB     00:00
nvidia-driver-local-rhel9-590.48.01                                                                                            3.4 MB/s | 3.5 kB     00:00
Dependencies resolved.
===============================================================================================================================================================
 Package                                      Architecture                   Version                                Repository                            Size
===============================================================================================================================================================
Installing:
 gpfs.gss.pmcollector                         x86_64                         5.2.3-3.el9                            @commandline                         7.7 M
 gpfs.gss.pmsensors                           x86_64                         5.2.3-3.el9                            @commandline                         8.1 M

Transaction Summary
===============================================================================================================================================================
Install  2 Packages

Total size: 16 M
Installed size: 65 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64                                                                                                 1/1
  Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                                                                               1/1
  Preparing        :                                                                                                                                       1/1
  Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64                                                                                                 1/2
  Installing       : gpfs.gss.pmsensors-5.2.3-3.el9.x86_64                                                                                                 1/2
  Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64                                                                                                 1/2
Created symlink /etc/systemd/system/multi-user.target.wants/pmsensors.service → /usr/lib/systemd/system/pmsensors.service.

  Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                                                                               2/2
  Installing       : gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                                                                               2/2
  Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                                                                               2/2
Created symlink /etc/systemd/system/multi-user.target.wants/pmcollector.service → /usr/lib/systemd/system/pmcollector.service.

  Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64                                                                                                 2/2
  Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                                                                               2/2
/usr/lib/tmpfiles.d/pmcollector.conf:1: Line references path below legacy directory /var/run/, updating /var/run/perfmon → /run/perfmon; please update the tmpfiles.d/ drop-in file accordingly.
/usr/lib/tmpfiles.d/pmsensors.conf:1: Line references path below legacy directory /var/run/, updating /var/run/perfmon → /run/perfmon; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                                                                               1/2
  Verifying        : gpfs.gss.pmsensors-5.2.3-3.el9.x86_64                                                                                                 2/2
Installed products updated.

Installed:
  gpfs.gss.pmcollector-5.2.3-3.el9.x86_64                                         gpfs.gss.pmsensors-5.2.3-3.el9.x86_64

Complete!
</code>

Definimos la config de la perf monitoring tool y designamos vlmgt02 como collector

<code>
[root@vlmgt02 gpfs_perf_testing]# /usr/lpp/mmfs/bin/mmperfmon config generate --collectors vlmgt02-ib0
mmperfmon: Node vlmgt02-ib0 is not a perfmon node.
mmperfmon: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@vlmgt02 gpfs_perf_testing]# /usr/lpp/mmfs/bin/mmchnode --perfmon -N vlmgt02-ib0
Fri Jul 31 01:54:56 PM -03 2026: mmchnode: Processing node vlmgt02-ib0
mmchnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
  </code>

Habilitamos los servicios.

<code>
systemctl enable --now pmcollector pmsensors
systemctl status pmcollector
systemctl status pmsensors</code>

==== CherryPy ====

En el nodo collector tiene que estar instalado el paquete de python CherryPy.

<code>[root@vlmgt02 gpfs_perf_testing]# python -m venv env
[root@vlmgt02 gpfs_perf_testing]# source env/bin/activate
(env) [root@vlmgt02 gpfs_perf_testing]# pip install cherrypy</code>

==== Generación de API Keys ====

Leer archivo ubicado en ''/data/admin/gpfs_perf_testing/apikey_config.txt''

==== Set Up de prueba ====

[[https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/releases | Clonar la release 
]]

Necesitamos python3.11.

Para una prueba de concepto voy a usar micromamba, pero vamos a tener que containerizar con podman. 

<code>
curl -Ls https://micro.mamba.pm/install.sh | bash
source ~/.bashrc
micromamba create -n grafana_bridge python=3.11 -c conda-forge -y
micromamba run -n grafana_bridge pip install CherryPy
micromamba run -n grafana_bridge pip install requests
micromamba run -n grafana_bridge python zimonGrafanaIntf.py
</code>

No está armada la config de perf monitoring

<code>[root@vlmgt02 source]# micromamba run -n grafana_bridge python zimonGrafanaIntf.py
2026-07-31 14:50 - MainThread                               - INFO     -  *** IBM Storage Scale bridge for Grafana - Version: 8.1.3 ***
2026-07-31 14:50 - MainThread                               - ERROR    - QueryHandler: getTopology returns no data.
2026-07-31 14:50 - MainThread                               - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 1 in 60s (MAX_ATTEMPTS_COUNT:3)
2026-07-31 14:51 - MainThread                               - ERROR    - QueryHandler: getTopology returns no data.
2026-07-31 14:51 - MainThread                               - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 2 in 60s (MAX_ATTEMPTS_COUNT:3)
2026-07-31 14:52 - MainThread                               - ERROR    - QueryHandler: getTopology returns no data.
2026-07-31 14:52 - MainThread                               - WARNING  - No Metadata results received from the pmcollector. Start retry attempt 3 in 60s (MAX_ATTEMPTS_COUNT:3)
%s Server internal error occurred. Reason: Empty results received
2026-07-31 14:53 - MainThread                               - ERROR    - Metadata could not be retrieved. Check log file for more details, quitting
</code>

==== Containerizando el bridge ====

Utilizamos podman en mmgt02

<code>systemd-run --pty --wait -p User=podman -p PAMName=login /bin/bash</code>


Actualmente estamos trabajando con la [[https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/releases/tag/v9.1.0
 | release 9.1.0]] del bridge.

Tiene un bug que debemos parchear para poder ver correctamente las métricas que son counters.

Editar el archivo source/prometheus.py

Cambiar la sección que figura como:

<code>
345-        if self.raw_data or "counter" in self.TOPO.getSensorMetricTypes(sensor).values():
346:            attrs.update({'nsamples': period, 'rawData': True, 'skipNullValues': True})
347-            self.logger.debug(MSG['SensorForceRawData'].format(sensor))
</code>

Para que skipNullValues sea False

<code>
345-        if self.raw_data or "counter" in self.TOPO.getSensorMetricTypes(sensor).values():
346:            attrs.update({'nsamples': period, 'rawData': True, 'skipNullValues': False})
347-            self.logger.debug(MSG['SensorForceRawData'].format(sensor))
</code>

Esto generaba que se inyecte la flag -s en la REST API de GPFS, que no era soportada. Generaba un log:

<code>400: unknown option 's',</code>

<code>
[podman@vlmgt02 ~]$ cd ibm-spectrum-scale-bridge-for-grafana-9.1.0
</code>

Buildeamos la imagen

<code>podman build --format=docker -t bridge_image:latest .</code>

Generamos unit file en /opt/containers/.config/containers/systemd/monitoring/gpfs-bridge.container

El contenido del unit file es:

<code>
[Unit]
Description=GPFS Grafana Bridge
After=pmcollector.service

[Container]
ContainerName=gpfs_bridge
Image=localhost/bridge_image:latest
Volume=/opt/IBM/zimon/ZIMonSensors.cfg:/opt/IBM/zimon/ZIMonSensors.cfg:ro
Volume=/opt/containers/monitoring/gpfs_bridge/config.ini:/opt/IBM/bridge/config.ini:ro
Volume=/opt/IBM/zimon/defaults:/opt/IBM/zimon/defaults:ro

Network=host

[Install]
WantedBy=default.target
</code>

Asegurarse que los permisos de /opt/IBM/zimon sean lo suficientemente permisivos para que el usuario podman pueda leer sus contenidos. Tuvimos que editar esto.

Luego se puede lanzar o detener el container con systemd.

<code>
podman $  systemctl --user daemon-reload
podman $  systemctl --user start gpfs-bridge
podman $  podman logs gpfs_bridge
</code>