La performance monitoring tool colecciona métricas de GPFS y provee información de performance del sistema. Está habilitada por default e incluye Collectors, Sensors y Proxies.
Un collector soporta hasta 150 sensor nodes. Con un collector debería ser suficiente para nuestro sistema. Se pueden armar esquemas con más de un collector (multi-collector federation) por razones de escala y de tolerancia a fallas.
Podemos utilizar mmgt01 como Node Collector en el cluster de storage y vlmgt02 en el cluster cliente, ya que es buena práctica mantener servicios extra (como monitoreo) fuera de los quorum nodes.
El componente que recopila datos de performance de un nodo
Se corre un proxy por cada protocolo para recopilar métricas de ese protocolo.
Queremos incorporar IBM Storage Scale bridge for Grafana para poder ver métricas de performance de GPFS en Grafana.
Para eso necesitamos configurar la performance monitoring tool de GPFS.
Necesitamos los paquetes gpfs.gss.pmsensors gpfs.gss.pmscollectors
Los encontramos únicamente en el storage node.
[root@sdmgt01 ~]# dnf search gpfs.gss.pmcollector =========================== Name Exactly Matched: gpfs.gss.pmcollector ===================== gpfs.gss.pmcollector.x86_64 : ZIMonCollector - an in-memory database for collecting and storing performance metrics.
Descargamos los rpm con dnf.
[root@sdmgt01 ~]# dnf download gpfs.gss.pmcollector gpfs.gss.pmsensors Updating Subscription Management repositories. Unable to read consumer identity This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register. Last metadata expiration check: 17:29:09 ago on Thu 30 Jul 2026 06:16:49 PM -03. (1/2): gpfs.gss.pmcollector-5.2.3-3.el9.x86_64.rpm 58 MB/s | 7.7 MB 00:00 (2/2): gpfs.gss.pmsensors-5.2.3-3.el9.x86_64.rpm 55 MB/s | 8.1 MB 00:00
Instalamos los paquetes en vlmgt02
[root@vlmgt02 gpfs_perf_testing]# dnf localinstall /data/admin/gpfs_perf_testing/gpfs.gss.pmcollector-5.2.3-3.el9.x86_64.rpm /data/admin/gpfs_perf_testing/gpfs.gss.pmsensors-5.2.3-3.el9.x86_64.rpm --disablerepo=Lenovo-Lenovo-OpenHPC-local,lenovo-hpc,lico-dep-local-library,lico-dep-local-standalone,lico-release Updating Subscription Management repositories. Unable to read consumer identity This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register. Local install repository for AppStream 76 kB/s | 3.2 kB 00:00 Local install repository for BaseOS 54 kB/s | 2.7 kB 00:00 nvidia-driver-local-rhel9-590.48.01 3.4 MB/s | 3.5 kB 00:00 Dependencies resolved. =============================================================================================================================================================== Package Architecture Version Repository Size =============================================================================================================================================================== Installing: gpfs.gss.pmcollector x86_64 5.2.3-3.el9 @commandline 7.7 M gpfs.gss.pmsensors x86_64 5.2.3-3.el9 @commandline 8.1 M Transaction Summary =============================================================================================================================================================== Install 2 Packages Total size: 16 M Installed size: 65 M Is this ok [y/N]: y Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/1 Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 1/1 Preparing : 1/1 Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/2 Installing : gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/2 Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 1/2 Created symlink /etc/systemd/system/multi-user.target.wants/pmsensors.service → /usr/lib/systemd/system/pmsensors.service. Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2 Installing : gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2 Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2 Created symlink /etc/systemd/system/multi-user.target.wants/pmcollector.service → /usr/lib/systemd/system/pmcollector.service. Running scriptlet: gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 2/2 Running scriptlet: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 2/2 /usr/lib/tmpfiles.d/pmcollector.conf:1: Line references path below legacy directory /var/run/, updating /var/run/perfmon → /run/perfmon; please update the tmpfiles.d/ drop-in file accordingly. /usr/lib/tmpfiles.d/pmsensors.conf:1: Line references path below legacy directory /var/run/, updating /var/run/perfmon → /run/perfmon; please update the tmpfiles.d/ drop-in file accordingly. Verifying : gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 1/2 Verifying : gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 2/2 Installed products updated. Installed: gpfs.gss.pmcollector-5.2.3-3.el9.x86_64 gpfs.gss.pmsensors-5.2.3-3.el9.x86_64 Complete!
Definimos la config de la perf monitoring tool y designamos vlmgt02 como collector
[root@vlmgt02 gpfs_perf_testing]# /usr/lpp/mmfs/bin/mmperfmon config generate --collectors vlmgt02-ib0 mmperfmon: Node vlmgt02-ib0 is not a perfmon node. mmperfmon: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. [root@vlmgt02 gpfs_perf_testing]# /usr/lpp/mmfs/bin/mmchnode --perfmon -N vlmgt02-ib0 Fri Jul 31 01:54:56 PM -03 2026: mmchnode: Processing node vlmgt02-ib0 mmchnode: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
Habilitamos los servicios.
systemctl enable --now pmcollector pmsensors systemctl status pmcollector systemctl status pmsensors
En el nodo collector tiene que estar instalado el paquete de python CherryPy.
[root@vlmgt02 gpfs_perf_testing]# python -m venv env [root@vlmgt02 gpfs_perf_testing]# source env/bin/activate (env) [root@vlmgt02 gpfs_perf_testing]# pip install cherrypy
Leer archivo ubicado en /data/admin/gpfs_perf_testing/apikey_config.txt
Necesitamos python3.11.
Para una prueba de concepto voy a usar micromamba, pero vamos a tener que containerizar con podman.
curl -Ls https://micro.mamba.pm/install.sh | bash source ~/.bashrc micromamba create -n grafana_bridge python=3.11 -c conda-forge -y micromamba run -n grafana_bridge pip install CherryPy micromamba run -n grafana_bridge pip install requests micromamba run -n grafana_bridge python zimonGrafanaIntf.py
No está armada la config de perf monitoring
[root@vlmgt02 source]# micromamba run -n grafana_bridge python zimonGrafanaIntf.py 2026-07-31 14:50 - MainThread - INFO - *** IBM Storage Scale bridge for Grafana - Version: 8.1.3 *** 2026-07-31 14:50 - MainThread - ERROR - QueryHandler: getTopology returns no data. 2026-07-31 14:50 - MainThread - WARNING - No Metadata results received from the pmcollector. Start retry attempt 1 in 60s (MAX_ATTEMPTS_COUNT:3) 2026-07-31 14:51 - MainThread - ERROR - QueryHandler: getTopology returns no data. 2026-07-31 14:51 - MainThread - WARNING - No Metadata results received from the pmcollector. Start retry attempt 2 in 60s (MAX_ATTEMPTS_COUNT:3) 2026-07-31 14:52 - MainThread - ERROR - QueryHandler: getTopology returns no data. 2026-07-31 14:52 - MainThread - WARNING - No Metadata results received from the pmcollector. Start retry attempt 3 in 60s (MAX_ATTEMPTS_COUNT:3) %s Server internal error occurred. Reason: Empty results received 2026-07-31 14:53 - MainThread - ERROR - Metadata could not be retrieved. Check log file for more details, quitting
Utilizamos podman en mmgt02
systemd-run --pty --wait -p User=podman -p PAMName=login /bin/bash
Actualmente estamos trabajando con la release 9.1.0 del bridge.
Tiene un bug que debemos parchear para poder ver correctamente las métricas que son counters.
Editar el archivo source/prometheus.py
Cambiar la sección que figura como:
345- if self.raw_data or "counter" in self.TOPO.getSensorMetricTypes(sensor).values():
346: attrs.update({'nsamples': period, 'rawData': True, 'skipNullValues': True})
347- self.logger.debug(MSG['SensorForceRawData'].format(sensor))
Para que skipNullValues sea False
345- if self.raw_data or "counter" in self.TOPO.getSensorMetricTypes(sensor).values():
346: attrs.update({'nsamples': period, 'rawData': True, 'skipNullValues': False})
347- self.logger.debug(MSG['SensorForceRawData'].format(sensor))
Esto generaba que se inyecte la flag -s en la REST API de GPFS, que no era soportada. Generaba un log:
400: unknown option 's',
[podman@vlmgt02 ~]$ cd ibm-spectrum-scale-bridge-for-grafana-9.1.0
Buildeamos la imagen
podman build --format=docker -t bridge_image:latest .
Generamos unit file en /opt/containers/.config/containers/systemd/monitoring/gpfs-bridge.container
El contenido del unit file es:
[Unit] Description=GPFS Grafana Bridge After=pmcollector.service [Container] ContainerName=gpfs_bridge Image=localhost/bridge_image:latest Volume=/opt/IBM/zimon/ZIMonSensors.cfg:/opt/IBM/zimon/ZIMonSensors.cfg:ro Volume=/opt/containers/monitoring/gpfs_bridge/config.ini:/opt/IBM/bridge/config.ini:ro Volume=/opt/IBM/zimon/defaults:/opt/IBM/zimon/defaults:ro Network=host [Install] WantedBy=default.target
Asegurarse que los permisos de /opt/IBM/zimon sean lo suficientemente permisivos para que el usuario podman pueda leer sus contenidos. Tuvimos que editar esto.
Luego se puede lanzar o detener el container con systemd.
podman $ systemctl --user daemon-reload podman $ systemctl --user start gpfs-bridge podman $ podman logs gpfs_bridge