A young Steve Jobs once called computers ‘bicycles for the mind’—he was referring to the dramatic decrease in the energetic cost of transportation that could be obtained with the bicycle, which breaks all scaling laws for how efficiently an animal can achieve motion. The creativity of new approaches to old materials science problems facilitated by the dramatic uptake of machine learning (ML) is a testament to this notion. The uptake of ML in our subject has been facilitated by exceptional efforts to provide open datasets, e.g. Novel Materials Discovery (NOMAD)1, Materials Project2, Joint Automated Repository for Various Integrated Simulations (JARVIS)3, Automatic FLOW for Materials Discovery (AFLOW)4, Open Quantum Materials Database (OQMD)5 and many others, as well as the extremely high quality of openly available software packages such as scikit-learn6, PyTorch7, (Just After eXecution) JAX8, Quantum Espresso9, and Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS)10. These resources, along with continued extraordinary developments in hardware, are helping a wide-range of researchers integrate ML into their work.

The dramatic rise in ML application started in the early 2010s. The ImageNet moment in 2012, when deep learning methods (AlexNet11), accelerated by GPUs and trained on a large, labelled dataset first outperformed traditional image classification methods is often identified as a key event. Since then, there has been a proliferation of ML methods applied to all aspects of society and research. In the sciences, the initial report of the performance of AlphaFold12 in the CASP14 contest captured the wider scientific imagination and made many researchers experts in protein folding overnight. While computational materials science has not had a moment as dramatic as AlphaFold, the field was a relatively early adopter of ML, and the prevalence of data-driven studies has steadily grown. In an ad hoc experiment (that would make a proper statistician shudder), we ranked the titles of articles from this journal from some recent years, to see how many papers appeared to be largely ML-focused. As early as 2017 we counted 7 out of 42 papers titles that had an ML flavour (~16%), in 2019 this was 28 out of 113 (~25%), this year so far that number is 49/112 (~42%). Performing this analysis it also becomes apparent just how diverse and creative the applications of ML in materials science have been. Figure 1 shows a slightly more systematic look at the rise in popularity of machine learning in materials science (and vice versa).

Fig. 1: Publication trends for machine learning in materials science.
figure 1

The bar chart shows the percentage of materials science papers including machine learning (green) and the percentage of machine learning papers including materials science (orange). The line plot shows the total number of publications for each of ‘materials science’ and ‘machine learning’ as obtained from the Web of Science. The underlying data and the notebook to generate the plot are available online (https://github.com/ML-Materials-Standards/ml-materials-checklist/tree/master/editorial-data).

Full size image

In another famous aphorism credited by Francis Crick to Linus Pauling ‘The best way to have a good idea is to have lots of ideas’. However, Pauling’s quote has an important second clause ‘[…] and throw away the bad ones’. While the ease of adoption of ML techniques is broadly a welcome development, we need to be aware that these are highly-sophisticated, nuanced methods, and without careful practice one can often obtain results that seem exciting, but are actually not as useful as they appear. To take a materials science example, it is becoming common to find predictive models for materials properties that have limited domains due to their training data, and where their applicability to broader data and associated problems is highly uncertain. In a recent article published in npj Computational Materials Omee et al. demonstrate how some of the most performant modern graph neural networks suffer significant prediction drops when moved outside of this training distribution13, highlighting the need for rigorous validation in any model application.

At npj Computational Materials Science we are proud to have been at the forefront of ML adoption in materials science and we are committed to continuing to provide a widely read and respected platform for the latest and best developments in materials informatics. A key aspect of this commitment is ensuring the highest standards of scientific rigour in the material published in the journal. As new methods and technologies are rapidly adopted by large numbers of practitioners there is a well-known tendency for inflated expectations and unsubstantiated hype leading eventually to a ‘trough of despair’ period where disillusionment with the broken promises abounds. Luckily in scientific publishing, we have a strong safeguard against unchecked expectation inflation and hype; the peer review process. While many argue that peer review is not perfect (and it surely isn’t), it provides an opportunity for ideas to be strenuously tested by experts with an interest in preserving the integrity of a particular research field.

However, no matter how expert peer reviewers are, it is impossible to keep abreast of all the latest developments in a field. When we are working on an inherently multidisciplinary topic like ML in materials science, this challenge becomes even greater. It is increasingly difficult to identify individuals who are qualified to comment on all aspects of the latest research papers. In recognition of this challenge and considering our commitment to continuing to publish the best ML for materials science research, our editorial team has developed a checklist for reviewers (and authors) that can be used to guide assessments of papers with significant ML content. The guidelines that we developed are designed to be as general as possible and therefore we expect them to apply to a wide-range of datasets and ML methods. The list is available here: https://github.com/ML-Materials-Standards/ml-materials-checklist.

Our checklist is heavily indebted to existing resources14,15,16, and is aligned with general efforts in the community towards ensuring best practice17,18, we have adapted the guidelines to reflect what we see as the needs and uses of the computational materials science community. Some of the items on the checklist are certainly required, such as clear descriptions of models, data and training procedures. We expressly do not want this checklist to be seen as a barrier to publication and we do not see it as a prescription for what a paper must adhere to for publication. For example, we strongly recommend that datasets used in studies should be made open and available, however, if authors have valid reasons such as anonymity of subjects or safety concerns around sharing data then this will be considered. By making honest best efforts to follow these guidelines we hope that this will improve the quality and impact of a study rather than present a nuisance. We also do not think that these standards should only apply to ML-based studies, but striving for openness and reproducibility should be central to all computational materials science.

Finally, we recognise that this field moves fast, new technologies arrive with great frequency. As an example, the development of automated experimentation in materials synthesis and characterisation presents new challenges for reproducibility and benchmarking. These technologies have the potential to drastically change the landscape, so we don’t want to see this checklist become a dead letter. Therefore, we are treating the checklist as a living document, to be regularly assessed and updated. For this to be a truly useful and relevant resource we rely heavily on the input from our readership. In that regard, we always welcome your input and opinions. To facilitate this we are hosting the checklist as a GitHub repository, where you can comment and make suggestions on the document and we will regularly review and update the contents (https://github.com/ML-Materials-Standards/ml-materials-checklist). The leap forward facilitated by ML has been achieved because of the highest standards of openness and reproducibility and we intend to replicate that spirit in the research that we publish.