gpfs
This is an old revision of the document!
Table of Contents
GPFS
Estructura del cluster GPFS
Un cluster GPFS consiste en varios elementos:
- Cluster manager: 172.27.253.31 (sdmgt01-ib0). Realiza el monitoreo de los disk leases, detecta fallas, elige el nodo de file system manager, determina la continuidad del servicio, maneja los UIDs, etc.
- Quorum nodes: sdmgt01-ib0 y sdmgt02-ib0. Nodos seleccionables para Cluster Manager.
- File system manager. Hace las reparaciones, agregado de discos, redimensionamiento de espacio, maneja las quotas (reserva bloques en el fs para quienes están escribiendo, etc.). Hay un quota manager para administrar esto.
- Metanode: uno por archivo/directorio abierto, mantiene la integridad de la metadata. La metadata se almacena en el inodo correspondiente.
- AFM gateway node: Cada “cache fileset” se vale de un servidor que se designa como gateway. Cada nodoe en el cache cluster puede funcionar como gateway.
La info del cluster manager se obtiene de correr:
$ mmlsmgr file system manager node ---------------- ------------------ data_fs 172.27.253.32 (sdmgt02-ib0) home_fs 172.27.253.32 (sdmgt02-ib0) Cluster manager node: 172.27.253.31 (sdmgt01-ib0)
La data se almacena directamente en el inodo si no hay niveles de indirección, y se almacena en bloques indirectos como en UNIX en el caso contrario.
Directorios compartidos
Los discos de GPFS son /home y /data: /home: ~46TB de scratch, almacenamiento rápido
/data: ~1PB de almacenamiento frío
Las quotas son
mmlsquota
Block Limits | File Limits
Filesystem type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks
data_fs USR 3663216 0 0 0 none | 36899 0 0 0 none
home_fs USR 20519320 0 0 0 none | 151819 0 0 0 none
/data/admin: directorio restringido a users en el grupo qadmin
/data/shared: directorio abierto a todos, pero solo escritura por el grupo qadmin
/data/contrib: directorio abierto a todos, de escritura por todos pero permisos de borrado solo para el owner/sudo (sticky bit, chmod 1777)
Rebuild de los módulos del kernel
Cada vez que se instala un kernel nuevo, hace falta correr mmbuildgpl:
[root@mmgt02 ~]# mmbuildgpl
--------------------------------------------------------
mmbuildgpl: Building GPL (5.1.8.2) module begins at Thu Jun 26 15:17:13 -03 2025.
--------------------------------------------------------
Verifying Kernel Header...
kernel version = 41800477 (418000477075001, 4.18.0-477.75.1.el8_8.x86_64, 4.18.0-477.75.1)
module include dir = /lib/modules/4.18.0-477.75.1.el8_8.x86_64/build/include
module build dir = /lib/modules/4.18.0-477.75.1.el8_8.x86_64/build
kernel source dir = /usr/src/linux-4.18.0-477.75.1.el8_8.x86_64/include
Found valid kernel header file under /usr/src/kernels/4.18.0-477.75.1.el8_8.x86_64/include
Getting Kernel Cipher mode...
Will use skcipher routines
Verifying Compiler...
make is present at /bin/make
cpp is present at /bin/cpp
gcc is present at /bin/gcc
g++ is present at /bin/g++
ld is present at /bin/ld
Verifying libelf devel package...
Verifying elfutils-libelf-devel is installed ...
Command: /bin/rpm -q elfutils-libelf-devel
The required package elfutils-libelf-devel is installed
Verifying Additional System Headers...
Verifying kernel-headers is installed ...
Command: /bin/rpm -q kernel-headers
The required package kernel-headers is installed
make World ...
make InstallImages ...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Thu Jun 26 15:17:56 -03 2025.
--------------------------------------------------------
Binarios
Los binarios de GPFS se encuentran en /usr/lpp/mmfs/bin/, con los siguientes comandos:
auditConsumerManager mmcesmmcchexport mmchqos mmimgrestore mmremotefs tsadddisk tskmipkm audit_watch.snap.py mmcesmmccrexport mmclidecode mmimportfs mmrepquota tsaddpdisk tslinkfileset auth.snap.py mmcesmmcdelexport mmclone mmimportnvmeof mmrestoreconfig tsaddrmap tslsattr basicsnap.py mmcesmonitor mmcloudfuncs mmjob mmrestorefs tsapolicy tslsdisk bscpwrap mmcesmonitord mmcloudgateway mmksh mmrestrictedctl tsauth tslsfileset bsshscpwrap mmcesnetaddaddress mmcmi mmktupgrade mmrestripefile tsbackup tslsfs bsshwrap mmcesnetchaddress mmcollectdebugdata mmlinkfileset mmrestripefs tsbackup33 tslsfsckx callhome.snap.py mmcesnetdeladdress mmcommon mmlockf mmrpldisk tsbuhelper tslsmgr ccrrestore mmcesnetdown mmcrcluster mmlogmsg mmrsh tscachectl tslspdisk ces.snap.py mmcesnetlsaddress mmcrfileset mmlogsort mmsdrcli tschattr tslspolicy cleanupkeystonedb mmcesnetmvaddress mmcrfs mmlsattr mmsdrfsdef tschcarrier tslspool clstrwatchcallbacks mmcesnetup mmcrfsc mmlscallback mmsdrfsdef.Linux tschdisk tslspool34 clstrwatchconsumermanager mmcesnetworkmonitor mmcrnodeclass mmlscluster mmsdrquery tscheckquota tslsqos cnss.snap.py mmcesnfschcfg mmcrnsd mmlsconfig mmsdrquerydef tschfileset tslsquota commonsnap.py mmcesnfschexport mmcrsnapshot mmlsdisk mmsdrrestore tschfs tslsrecgroup copyutil mmcesnfscrexport mmdefedquota mmlsfileset mmsdrserv tschmgr tslsrecoverygroupevents core.snap.py mmcesnfsdelexport mmdefquotaoff mmlsfs mmsetquota tschpdisk tslssnapshot daemonize mmcesnfsfuncs mmdefquotaon mmlsfsckx mmsetrcmd tschpolicy tslsvdisk ece.callhome.py mmcesnfsloadexport mmdefragfs mmlslicense mmshutdown tschpool tsmergeshadow ess_hal.snap.py mmcesnfslscfg mmdelacl mmlsmgr mmsmb tschqos tsnetverify fab3.snap.py mmcesnfslsexport mmdelcallback mmlsmount mmsmbfuncs tschrecgroup tsnsdaccess fcm.snap.py mmcesnfsmonitor mmdeldisk mmlsnode mmsmbman tschstoragepool tsnsddiscover gds.snap.py mmcesnode mmdelfileset mmlsnodeclass mmsmbsupportedoptions tsclone tsnsdrediscover gpfsrunlevel mmcesobjaccess mmdelfs mmlsnsd mmsmbvalidoptions tsclwatchrules tspcache gpfs.snap mmcesobjaccesslinkfset mmdelnode mmlsperfdata mmsnapdir tscmd tspcacheevict gui.callhome.py mmcesobjcfgman mmdelnodeclass mmlspolicy mmsnmpagentd tscp tspcachescan gui.snap.py mmcesobjchcfg mmdelnsd mmlspool mmsnmpagentd10 tscreatedir tspcachesnapdiff hadoop.snap.py mmcesobjchstoragepolicy mmdelsnapshot mmlsqos mmsnmpagentd30 tscrfileset tspcacheu lxtrace mmcesobjcrbase mmdevdiscover mmlsquota mmstartpolicy tscrfs tspcacheutil lxtrace-4.18.0-372.32.1.el8_6.x86_64 mmcesobjcrring mmdf mmlssnapshot mmstartup tscrrecgroup tspmon lxtrace-4.18.0-477.75.1.el8_8.x86_64 mmcesobjcrstoragepolicy mmdiag mmmigratefs mmstatdcallout tscrsnapshot tsprclear mmaddcallback mmcesobjdepstoragepolicy mmdsh mmmount mmsync tscrvdisk tspreparedisk mmadddisk mmcesobjfuncs mmdsm mmmsgqueue mmsysmonc tsctl tsprinquiry mmaddnode mmcesobjlscfg mmdspmsg mmnetverify mmsysmoncontrol tsdbfs tsprpreemptabort mmadquery mmcesobjlsstoragepolicy mmdump mmnfs mmsysmon.py tsdbmemleak tsprquery mmadquery_py2 mmcesobjmr mmdumpfilocks mmnfscleanup mmtrace tsdbmemleakparse tsprreadkeys mmafmcmd mmcesobjpolicyfuncs mmdumpkthreads mmnfsdown mmtracectl tsdbvdisk tsprreadres mmafmconfig mmcesobjs3 mmdumpperfdata mmnfsfuncs mmumount tsdefedquota tsprregister mmafmcosaccess mmcesop mmdynamicmap mmnfsgrace mmunlinkfileset tsdefquotaoff tsprreserve mmafmcosconfig mmcespnfsip mmeditacl mmnfsinit mmunmount tsdefquotaon tsprshowfence mmafmcosctl mmcesrecovernode mmedquota mmnfsinstall mmuserauth tsdefragfs tsputacl mmafmcoskeys mmcesservice mmexectsmcmd mmnfsmonitor mmvaryonvg tsdelacl tsqosnice mmafmctl mmcessmbchconfig mmexecvfile mmnfsnodeback mmwatch tsdeldisk tsquotaoff mmafmhomeconfig mmcessmbchexport mmexpelnode mmnfspreunmount mmxargs tsdelfileset tsquotaon mmafmlocal mmcessmbcrexport mmexportfs mmnfsquorumloss mmxcp tsdelfs tsrecgroupserver mmapplypolicy mmcessmbdelexport mmfileid mmnfsrecovernode mmyfields tsdelpdisk tsreclaimspace mmaudit mmcessmbexportacl mmfixcfg mmnfstakeover msgqueueDaemonManager tsdelrecgroup tsremount mmauditmount mmcessmbfuncs mmfs mmnfsuninstall msgqueueservicesmonitormanager tsdelsnapshot tsrepquota mmauditpreunmount mmcessmblsconfig mmfsadm mmnfsup network.callhome.py tsdelvdisk tsressnapshot mmauth mmcessmblsexport mmfsck mmnsddiscover network.snap.py tsdf tsrestorefileset mmautoload mmcesuserauthckservice mmfsckx mmobj nfsdebug.snap.py tsdiag tsrestripefile mmbackup mmcesuserauthcrservice mmfsctl mmpercentdecode nfspod.snap.py tsdsh tsrestripefs mmbackupconfig mmcesuserauthlsservice mmfsd mmpercentencode nfs.snap.py tseditacl tsrpldisk mmblock mmcesuserauthrmservice mmfsenv mmperfmon noobaa.snap.py tsedquota tssbrff mmbuildgpl mmchattr mmfsfuncs mmpmon object.callhome.py tsexpelnode tssbrindex mmcachectl mmchcluster mmfsfuncs.Linux mmprodname object.snap.py tsextattr tssetquota mmcallhome mmchconfig mmfsmount mmprotocoltrace objmetric_count.py tsfileid tssnapdir mmcallhremote mmchdisk mmgensymtable mmpsnap perfmon.callhome.py tsfindinode tsstarthelper mmccr mmcheckquota mmgetacl mmputacl perfmon_helper.py tsfindname tsstatus mmccrmonitor mmchecksubsys mmgetconfdata mmqos perfmon.snap.py tsfsck tsunlinkfileset mmccrnv mmchfileset mmgetifconf mmqosnice perf.snap.py tsfsckx tsunmount mmces mmchfs mmgetkafkaspecinfo mmquotaoff pymmcesnfsfuncs.py tsfsctl tsusercmd mmcesauthfuncs mmchlicense mmgetstate mmquotaon pymmcesnfslsexport.py tsgetacl tsxattr mmcesblockfuncs mmchmgr mmglobfuncs mmrcmd runmmfs tsgetscsiinfo tsyselect mmcesfuncs mmchnode mmglobfuncs.Linux mmrcp scpwrap tsgnrgethospdata wftopichk mmceshdfsfuncs mmchnodeclass mmgskkm mmreclaimspace smb.snap.py tsgnrvdiskfgscrub wftopics mmcesloadmonitor mmchnsd mmhealth mmrefresh sshscpwrap tsgskkm xcputil.sh mmceslog mmchpolicy mmhelp-apolicy mmremote sshwrap tsimgrestore mmcesminfuncs mmchpool mmimgbackup mmremotecluster sysmon.snap.py tskafkaauth
Documentación (GPFS v5.18)
gpfs.1752166191.txt.gz · Last modified: by joaquintorres