====== Performance Monitoring ======
===== Intro a componentes ======
La [[https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=monitoring-using-performance-tool | performance monitoring tool]] colecciona métricas de GPFS y provee información de performance del sistema.
Está habilitada por default e incluye //Collectors, Sensors // y // Proxies//.
==== Collector ====
Un collector soporta hasta 150 sensor nodes. Con un collector debería ser suficiente para nuestro sistema. Se pueden armar esquemas con más de un collector (multi-collector federation) por razones de escala y de tolerancia a fallas.
Podemos utilizar mmgt01 como Node Collector en el cluster de storage y vlmgt02 en el cluster cliente, ya que es buena práctica mantener servicios extra (como monitoreo) fuera de los quorum nodes.
==== Sensors ====
El componente que recopila datos de performance de un nodo
==== Proxy ====
Se corre un proxy por cada protocolo para recopilar métricas de ese protocolo.
===== IBM Storage Scale bridge for Grafana =====
Queremos incorporar [[https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=data-using-storage-scale-performance-monitoring-bridge-grafana | IBM Storage Scale bridge for Grafana]] para poder ver métricas de performance de GPFS en Grafana.
Para eso necesitamos configurar la performance monitoring tool de GPFS.
==== Resolución de dependencias ====
Necesitamos los paquetes
gpfs.gss.pmsensors
gpfs.gss.pmscollectors
Los encontramos únicamente en el storage node.
[root@sdmgt01 ~]# dnf search gpfs.gss.pmcollector
=========================== Name Exactly Matched: gpfs.gss.pmcollector =====================
gpfs.gss.pmcollector.x86_64 : ZIMonCollector - an in-memory database for collecting and storing performance metrics.
Descargamos los rpm con dnf.
[root@sdmgt01 ~]# dnf download gpfs.gss.pmcollector gpfs.gss.pmsensors
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register.
Last metadata expiration check: 17:29:09 ago on Thu 30 Jul 2026 06:16:49 PM -03.
(1/2): gpfs.gss.pmcollector-5.2.3-3.el9.x86_64.rpm 58 MB/s | 7.7 MB 00:00
(2/2): gpfs.gss.pmsensors-5.2.3-3.el9.x86_64.rpm 55 MB/s | 8.1 MB 00:00
Instalamos los paquetes en vlmgt02
[root@vlmgt02 gpfs_perf_testing]# dnf localinstall /data/admin/gpfs_perf_testing/gpfs.gss.pmcollector-5.2.3-3.el9.x86_64.rpm /data/admin/gpfs_perf_testing/gpfs.gss.pmsensors-5.2.3-3.el9.x86_64.rpm --disablerepo=Lenovo-Lenovo-OpenHPC-local,lenovo-hpc,lico-dep-local-library,lico-dep-local-standalone,lico-release
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register.
Local install repository for AppStream 76 kB/s | 3.2 kB 00:00
Local install repository for BaseOS 54 kB/s | 2.7 kB 00:00
nvidia-driver-local-rhel9-590.48.01 3.4 MB/s | 3.5 kB 00:00
Dependencies resolved.
===============================================================================================================================================================
Package Architecture Version Repository Size
===============================================================================================================================================================
Installing:
gpfs.gss.pmcollector x86_64 5.2.3-3.el9 @commandline 7.7 M
gpfs.gss.pmsensors x86_64 5.2.3-3.el9 @commandline 8.1 M
Transaction Summary
===============================================================================================================================================================
Install 2 Packages
Total size: 16 M
Installed size: 65 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/1
Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 1/1
Preparing : 1/1
Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/2
Installing : gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/2
Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/2
Created symlink /etc/systemd/system/multi-user.target.wants/pmsensors.service → /usr/lib/systemd/system/pmsensors.service.
Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2
Installing : gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2
Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2
Created symlink /etc/systemd/system/multi-user.target.wants/pmcollector.service → /usr/lib/systemd/system/pmcollector.service.
Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 2/2
Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2
/usr/lib/tmpfiles.d/pmcollector.conf:1: Line references path below legacy directory /var/run/, updating /var/run/perfmon → /run/perfmon; please update the tmpfiles.d/ drop-in file accordingly.
/usr/lib/tmpfiles.d/pmsensors.conf:1: Line references path below legacy directory /var/run/, updating /var/run/perfmon → /run/perfmon; please update the tmpfiles.d/ drop-in file accordingly.
Verifying : gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 1/2
Verifying : gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 2/2
Installed products updated.
Installed:
gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 gpfs.gss.pmsensors-5.2.3-3.el9.x86_64
Complete!
Definimos la config de la perf monitoring tool y designamos vlmgt02 como collector
[root@vlmgt02 gpfs_perf_testing]# /usr/lpp/mmfs/bin/mmperfmon config generate --collectors vlmgt02-ib0
mmperfmon: Node vlmgt02-ib0 is not a perfmon node.
mmperfmon: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[root@vlmgt02 gpfs_perf_testing]# /usr/lpp/mmfs/bin/mmchnode --perfmon -N vlmgt02-ib0
Fri Jul 31 01:54:56 PM -03 2026: mmchnode: Processing node vlmgt02-ib0
mmchnode: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
Habilitamos los servicios.
systemctl enable --now pmcollector pmsensors
systemctl status pmcollector
systemctl status pmsensors
==== CherryPy ====
En el nodo collector tiene que estar instalado el paquete de python CherryPy.
[root@vlmgt02 gpfs_perf_testing]# python -m venv env
[root@vlmgt02 gpfs_perf_testing]# source env/bin/activate
(env) [root@vlmgt02 gpfs_perf_testing]# pip install cherrypy
==== Generación de API Keys ====
Leer archivo ubicado en ''/data/admin/gpfs_perf_testing/apikey_config.txt''
==== Set Up de prueba ====
[[https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/releases | Clonar la release
]]
Necesitamos python3.11.
Para una prueba de concepto voy a usar micromamba, pero vamos a tener que containerizar con podman.
curl -Ls https://micro.mamba.pm/install.sh | bash
source ~/.bashrc
micromamba create -n grafana_bridge python=3.11 -c conda-forge -y
micromamba run -n grafana_bridge pip install CherryPy
micromamba run -n grafana_bridge pip install requests
micromamba run -n grafana_bridge python zimonGrafanaIntf.py
No está armada la config de perf monitoring
[root@vlmgt02 source]# micromamba run -n grafana_bridge python zimonGrafanaIntf.py
2026-07-31 14:50 - MainThread - INFO - *** IBM Storage Scale bridge for Grafana - Version: 8.1.3 ***
2026-07-31 14:50 - MainThread - ERROR - QueryHandler: getTopology returns no data.
2026-07-31 14:50 - MainThread - WARNING - No Metadata results received from the pmcollector. Start retry attempt 1 in 60s (MAX_ATTEMPTS_COUNT:3)
2026-07-31 14:51 - MainThread - ERROR - QueryHandler: getTopology returns no data.
2026-07-31 14:51 - MainThread - WARNING - No Metadata results received from the pmcollector. Start retry attempt 2 in 60s (MAX_ATTEMPTS_COUNT:3)
2026-07-31 14:52 - MainThread - ERROR - QueryHandler: getTopology returns no data.
2026-07-31 14:52 - MainThread - WARNING - No Metadata results received from the pmcollector. Start retry attempt 3 in 60s (MAX_ATTEMPTS_COUNT:3)
%s Server internal error occurred. Reason: Empty results received
2026-07-31 14:53 - MainThread - ERROR - Metadata could not be retrieved. Check log file for more details, quitting
==== Containerizando el bridge ====
Utilizamos podman en mmgt02
systemd-run --pty --wait -p User=podman -p PAMName=login /bin/bash
Actualmente estamos trabajando con la [[https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/releases/tag/v9.1.0
| release 9.1.0]] del bridge.
Tiene un bug que debemos parchear para poder ver correctamente las métricas que son counters.
Editar el archivo source/prometheus.py
Cambiar la sección que figura como:
345- if self.raw_data or "counter" in self.TOPO.getSensorMetricTypes(sensor).values():
346: attrs.update({'nsamples': period, 'rawData': True, 'skipNullValues': True})
347- self.logger.debug(MSG['SensorForceRawData'].format(sensor))
Para que skipNullValues sea False
345- if self.raw_data or "counter" in self.TOPO.getSensorMetricTypes(sensor).values():
346: attrs.update({'nsamples': period, 'rawData': True, 'skipNullValues': False})
347- self.logger.debug(MSG['SensorForceRawData'].format(sensor))
Esto generaba que se inyecte la flag -s en la REST API de GPFS, que no era soportada. Generaba un log:
400: unknown option 's',
[podman@vlmgt02 ~]$ cd ibm-spectrum-scale-bridge-for-grafana-9.1.0
Buildeamos la imagen
podman build --format=docker -t bridge_image:latest .
Generamos unit file en /opt/containers/.config/containers/systemd/monitoring/gpfs-bridge.container
El contenido del unit file es:
[Unit]
Description=GPFS Grafana Bridge
After=pmcollector.service
[Container]
ContainerName=gpfs_bridge
Image=localhost/bridge_image:latest
Volume=/opt/IBM/zimon/ZIMonSensors.cfg:/opt/IBM/zimon/ZIMonSensors.cfg:ro
Volume=/opt/containers/monitoring/gpfs_bridge/config.ini:/opt/IBM/bridge/config.ini:ro
Volume=/opt/IBM/zimon/defaults:/opt/IBM/zimon/defaults:ro
Network=host
[Install]
WantedBy=default.target
Asegurarse que los permisos de /opt/IBM/zimon sean lo suficientemente permisivos para que el usuario podman pueda leer sus contenidos. Tuvimos que editar esto.
Luego se puede lanzar o detener el container con systemd.
podman $ systemctl --user daemon-reload
podman $ systemctl --user start gpfs-bridge
podman $ podman logs gpfs_bridge