HPC.NRW PC.NRW is driving innovation in research and HPC infrastructure

The North Rhine-Westphalian Competence Network for High Performance Computing (HPC.NRW) has achieved significant progress through the efforts of its dedicated working groups, advancing both the technological development and user-friendliness of high-performance computing (HPC) centers in North Rhine-Westphalia. The focus has been on innovative solutions, energy-efficient operation and the strengthening of scientific excellence within the region. The following provides an overview of key developments and milestones reached.

Interactive Notebooks in HPC: Combining Efficiency and Innovation

The introduction of JupyterHub instances at various HPC sites in NRW enables intuitive and efficient use of high-performance computing resources. A deliverable supports administrators with implementation and long-term management of these systems.

Efficient Data Management for Optimal Use

A site-specific overview of research data management tools helps users find suitable tools for their needs. Additionally, a regular seminar series on research data management topics has been introduced to promote professional exchange and facilitate the effective use of these tools.

Focus on Energy Efficiency and Sustainability

To promote environmentally friendly HPC operations, practical guides have been developed. These include recommendations for energy-efficient procurement and operation of server rooms, taking legal requirements into account. The goal is to reduce energy consumption without compromising system performance.

Statewide Tier-3 System: hpcBasisCluster.nrw

A jointly operated Tier-3 system (hpcBasisCluster.nrw) is currently being established at the University of Cologne. This system is designed to provide universities without their own HPC infrastructure with access to powerful computing resources. It complements existing local capacities and offers an alternative for institutions planning to phase out their own Tier-3 systems. By linking the system to a higher-level Tier-2 infrastructure, seamless integration into the HPC supply pyramid of HPC.NRW is ensured.

Optimization of Job and Resource Efficiency

A deliverable for optimizing job efficiency helps researchers use computing resources more effectively and reduce the energy consumption of their HPC jobs. Additionally, a guide for administrators was developed for monitoring and managing a job monitoring system, enabling better resource utilization.

Precise Reporting of HPC Usage

A framework for automated collection of usage metrics has been introduced. It enables the creation of detailed reports, providing a comprehensive overview of HPC infrastructure utilization and forming a basis for targeted optimizations.

Best Practices for Artificial Intelligence

Best practice guides have been developed for integrating AI frameworks like TensorFlow and PyTorch into high-performance computing centers. These deliverables assist researchers and administrators with the efficient use and implementation of machine learning tools and promote innovative applications in research. Additionally, a weekly AI consultation hour has been established in cooperation with GCS and NHR.

Secure Data Migration and System Transformation

To ensure optimal use of available HPC resources, a new procedure based on Nextcloud supports both vertical and horizontal migration of HPC jobs. This approach enables secure and efficient data transfer between HPC systems. A migration deliverable outlines the steps required for a smooth execution of system transitions.

Support through HPC Consulting

An deliverable on HPC consulting and a competence directory provide orientation and support for researchers and administrators. Already completed consultations, such as the optimization of a Fortran MPI CFD code, demonstrate the effectiveness of this support and contribute to improved HPC usage in NRW.

Strengthening HPC Competence through Training

A broad training offering on topics such as MATLAB, C++, and Julia as well as a “Train-the-Trainer” program in cooperation with The Carpentries and the Digital Research Academy promote participants' HPC skills. These measures ensure knowledge transfer and strengthen the research community in NRW.

Focus on Security and Prevention

The establishment of a communication channel for the exchange of information on security risks and countermeasures improves the protection of HPC infrastructures. The dialogue on best practices and security policies helps safeguard systems against potential threats.

Central Software Solutions for all of NRW

Thanks to successful state license agreements – including for TotalView, Linaro and Vampir – access to essential development and analysis tools for HPC applications has been simplified across the state. Additionally, premium support contracts for Intel and the workload manager SLURM were concluded. These measures relieve individual sites financially and organizationally, while improving the availability and quality of central software solutions throughout the network.

Greater Visibility and User-oriented Communication

The HPC.NRW website was restructured and redesigned with a clear focus on user needs. Additionally, an open mailing list was established, which now regularly informs around 140 subscribers about current developments, events, and opportunities – an important step toward increasing the network’s visibility.

Shaping the future through progress

In the 2023/24 funding phase HPC.NRW set important impulses to sustainably strengthen the HPC landscape in North Rhine-Westphalia. The focus on innovation, sustainability, and user-friendliness provides a solid foundation for successfully meeting future challenges in research and development.