CECAM Flagship Workshop: Electronic Structure Software Development: Advancing the Modular Paradigm
Held at CECAM-HQ, EPFL, Lausanne, February 19-24, 2024
This workshop continues the successful trajectory of the CECAM Electronic Structure Library initiative, aimed at supporting and connecting developers of shared libraries and software tools in support of electronic structure simulations across our entire community. As always, CECAM’s professionalism and remarkable organization, as well as their truly welcoming hospitality, deserve the highest praise.
Background
Shared computational libraries that provide key functionality are now firmly established parts of the electronic structure software ecosystem. As electronic structure methods and codes diversify and mature, the development of libraries strengthens collaborations and avoids reimplementing the same methods in the context of a different, monolithic code. Over the years, a modular paradigm has emerged in which central pieces can be shared and reused between different projects.
Many efforts carried out so far have focused on making this paradigm successful and sustainable, by building robust software components with stable interfaces, offering the best possible performance on a broad range of hardware architectures, and sharing development practices between developers with sometimes very different backgrounds. Among the most important ongoing challenges are (1) training and educating new developers to take advantage of existing developments where that is appropriate and (2) continuously evolving library software to be useful, current, and usable by end users in often complex environments like HPC systems.
The scientific reach of electronic structure theory continues to expand rapidly, including by generalisation of high-throughput calculations, creation of complex workflows, and rapid growth of data-driven methods and machine learning. All these developments now attract participants beyond academia. On the computational side, the involvement of industrial partners is growing steadily, bringing feedback and insights from engineers. Globally, several key collaborative efforts develop shared software, including the MaX and NOMAD Centers of Excellence (CoE), the UK ExCALIBUR exascale project, the Swiss THEOS and MARVEL projects, the U.S. based Molecular Sciences Software Institute, DoE’s Exascale initiative, or DoE funded centers such as MiCCOM. Connections with other communities, like quantum chemistry, are getting stronger because of shared needs, as illustrated by the TREX CoE. The Swiss Scientific Computing Center (CSCS), the U.K.’s Daresbury Laboratory, and many other individual institutions likewise act as lighthouses for shared developments within the broad electronic structure software ecosystem.
The CECAM Electronic Structure Library (ESL) initiative (https://esl.cecam.org) is a key venue that connects developers from electronic structure codes across the community (contributors represent, e.g., SIESTA, BigDFT, DFTB+, Quantum Espresso, Octopus, FHI-aims, and several others). It provides a space for coordination between developers, new library developments and enhancements of existing libraries, as well as interaction with the broader ecosystem of library developments for electronic structure theory. Especially through its workshops, the ESL also acts as a venue that brings together developers across the many broader centers and collaborative efforts (see above) providing a common ground to exchange information and coordinate developments.
Summary
Major outcomes:
- Unifying practices and tools to build software. We have agreed on using CMake in all ESL modules and created a repository to share good CMake recipes: https://gitlab.com/ElectronicStructureLibrary/esl-cmake-recipes.
- On top of this, we have agreed on further promoting the use of Spack and EasyBuild.
- Create a new section in the programme for early career colleagues to expose their research.
- Identified potential new partners for reviving a planned COST action, meetings are planned in the coming weeks.
- Increased participation from the US, with the help of a generous sponsorship from MOLSSI.
- A healthy intake of newcomers, around half of the 38 participants.
- Increased awareness of our Zulip channel to bring together the community
- Participants from other communities that made use of electronic structure libraries, e.g. Machine Learning and experimental materials
This workshop has identified a series of problems that can and should be addressed rapidly:
- Making use cases more accessible to newcomers. For this, we have started to upgrade the ESL Demonstrator.
- Improving the visibility of the Continuous Integration infrastructure. It now has its own dedicated repository: https://gitlab.com/ElectronicStructureLibrary/esl-ci.
- Sustain a collaborative chat channel (Zulip) where people can exchange ideas, and request feedback
Discussions about the future of the ESL
Challenges
It is more and more difficult to keep momentum within the community and important sustainability issues are now emerging, e.g. LibXC is used by around 80 different codes but is maintained by a single person who has to deal with complex technical problems. Another prominent issue is employment churn, which limits commitments to 3 years or less. The “publish or perish” race, with a retraction rate of articles doubling every 2 years, is also adding an unnecessary pressure on young researchers, depriving the community from a strongly needed generational renewal. As a result, the spreading of code reuse and good practices is quite slower than the production of sharable and reusable software modules.
Comparison with the climate research community
One ESL contributor is now experiencing first-hand how software activities are organised in the climate research community, through a position in the Access-NRI infrastructure (https://www.access-nri.org.au/). There, software development is inherently collaborative, since all models require more than one code and have highly complex interactions. Most of the research involves both academic and non-academic institutions, which has had a great impact on the professionalisation of all software activities. Hiring Research Software Engineers (RSE) is part of the core development strategy. Access-NRI has been active since 2022 and is the result of many years of lobbying. It is currently employing 25 RSE and will expand to 35 RSE by 2025. Its activities are evaluated every 5 years to renew its funding. Such an organisation with a long-term view lets RSE serve the whole community all the time, instead of being limited to one research group / one software project as it is the case within the electronic structure community, leading to a much higher impact of the software development outcomes on society. Even if it is quite challenging, the paradigm shift it represents deserves at least to be considered and discussed.
Possible actions to increase momentum
One possible way to improve communication and ensure constant progress within the community is to set up an exchange platform for good practices. The following questions are open: How to define a list of experts? Would it be a regular call, e.g. monthly, or an on-demand call when a critical mass of interest is reached? Can this help us save time when addressing software-related issues?
How can we discuss more openly? One way is to promote the ESL discussion space on Zulip (https://esl-cecam.zulipchat.com/join/uvatdjuxwzytg5sgwv3ojxsq/). There, discussions can be organised in a way that will make them easy to find even after a few months of inactivity. To make it easier for people to join, the invitation link now appears prominently on the first page of the ESL website (https://esl.cecam.org/).
Another way to improve communication and visibility could be to structure our community around topics and guide new developers through the different kinds of software available: reference implementations vs. fully optimised software, and/or research/experimental software vs. production-grade software. How can the idea of an “electronic-structure ecosystem” be promoted and made more understandable?
An important aspect affecting momentum is to find ways to alleviate the current contributors to the ESL, in particular by identifying what should not be worried about. It is also essential to remember that researchers are generally not software experts and do need training regarding software engineering aspects.
There is a general consensus that the outcomes of ESL efforts should be promoted significantly more than they currently are. More outreach could be achieved by a more prominent involvement of ESL contributors in online communities like Stack Exchange. A well-defined protocol for announcements would help defining an “ESL style”.
All this being said, several new developments and connections showcased at the workshop are very encouraging and generated praise – for example, the US DOE funded infrastructure around NWChemEx, the report on Julia software for DFT development, or the experiences of the Psi4 project, to name just a few. Clearly, our software ecosystem is moving. Connecting the different, rapidly evolving developments in our community continues to be a major function of the ESL workshops and of the ESL.
Community Needs
The ESL initiative is, in fact, centered around a key community need in the experience of the organizers, i.e., a venue to communicate and (where possible) create bridges between the diverse software projects and developments surrounding electronic structure theory.
Since the inception of the ESL, it has sustained a substantial number of consistent participants from the DFT community. However, these same participants are all agreeing that the community needs to do more to bring together developments, training, and knowledge sharing. This requires close collaboration with entities with expertise, such as HPC centers.
Since GPUs has become mainstream, and we are likely to see more specialized architectures in the future. It is becoming increasingly more difficult for non-research software engineers to tackle the difficulties on achieving performant and accurate software on said architectures. The community needs a continuous communication channel with field experts, such as HPC centers, compiler vendors and hardware designers.
Societal Needs
The Electronic Structure Library and related collaborative efforts are building a world-wide community and network within the electronic structure theory community. By bringing together developers, researchers, and practitioners from around the world, the initiative fosters a sense of community and shared purpose, facilitating collaboration, knowledge exchange, and professional networking opportunities. This can enhance the visibility and reputation of individuals and institutions involved in the initiative, leading to increased opportunities for collaboration and recognition within the scientific community. Societal benefits are emerging in 4 different areas:
- Accelerating scientific progress: By fostering collaboration and sharing among developers from various academic, industrial, and research institutions, the ESL can foster advancements in areas such as materials science, chemistry, and condensed matter physics, which can ultimately benefit society through the development of new technologies and innovations.
- Democratising the access to cutting-edge software: By sharing and reusing cutting-edge computational tools and libraries developed by experts in the field, as well as good programming practices, the ESL aims at make the access to advanced electronic structure methods easier, enabling researchers from diverse backgrounds, including those in academia, industry, and smaller research institutions, to utilize state-of-the-art computational techniques for their research projects.
- Broadening education and training opportunities: By facilitating knowledge exchange and collaboration among developers with different backgrounds and levels of expertise, the ESL can help cultivate a new generation of computational scientists skilled in advanced electronic structure methods. This can contribute to the growth of the scientific workforce and empower individuals to pursue careers in STEM fields.
- Increasing the impact of publicly-funded research: By defending the modular paradigm and avoiding repetitive efforts and errors, the ESL intends to help researchers and scientific software developers make a better use of their time and resources, which in the long run will improve the impact and returns of the related investments.
Detailed Program, Speakers and Participants, Event Website, Abstracts and Further Information