Nanoscale Horizons View Article Online REVIEW View Journal | View Issue Machine learning in electron microscopy for This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. advanced nanocharacterization: current Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Cite this: Nanoscale Horiz., 2022, 7, 1427 developments, available tools and future outlook a Marc Botifoll, Ivan Pinto-Hugueta and Jordi Arbiol *ab In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for Received 10th August 2022, material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the Accepted 12th September 2022 offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines DOI: 10.1039/d2nh00377e with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to rsc.li/nanoscale-horizons narrow down the incoming future of electron microscopy, its challenges and outlook. 1. Introduction fields, ranging from the less obvious finances to the funda- ments of robotics. It is ubiquitous in a huge variety of scientific Machine Learning (ML) has been a core partner for the scientific fields, providing both tools for automating processes and and technical breakthroughs of the last decade in multiple knowledge-revealing algorithms exploitable without a deep computer science background. Among many others, Electron a Catalan Institute of Nanoscience and Nanotechnology (ICN2), CSIC and BIST, Microscopy (EM) is a valuable example of how ML is and will be Campus UAB, Bellaterra, 08193 Barcelona, Catalonia, Spain. E-mail: arbiol@icrea.cat providing a solid framework for the upcoming advances. This b ICREA, Pg. Lluı́s Companys 23, 08010 Barcelona, Catalonia, Spain review intends to compile and highlight the most important and Marc Botifoll graduated in Nano- Ivan Pinto-Huguet graduated science and Nanotechnology at with a double Major in Physics Universitat Autònoma de Barcelona and Chemistry in 2020 and got (UAB) ranking first in 2018 his Major in Mathematics in promotion. He was one of the top 2021. In 2022, he got the Master students in the Master’s program of of Multidisciplinary Research in Multidisciplinary Research in Experimental Sciences (MMRES) Experimental Sciences (MMRES) at at BIST-UPF. In 2019 he joined BIST-Universitat Pompeu Frabra the ICN2 Advanced Electron (UPF) in 2019. In 2019 he joined Nanoscopy Group (GAe N) as the ICN2 PhD Programme within undergrad student and enrolled the Advanced Electron Nanoscopy for the ICN2 PhD Programme in Marc Botifoll Group (GAe N). He joined GAe N Ivan Pinto-Huguet 2022. His research is dealing with at ICN2 already in 2017 and since the advanced (S)TEM related then, his research is dealing with the advanced (S)TEM related study of study of nanostructures and developing AI methods based on ML/ nanostructures and developing AI methods based on ML/DL for DL for automating its data analysis. automating its data analysis. He is author of 5 publications. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1427 View Article Online Review Nanoscale Horizons recent advances in Transmission Electron Microscopy (TEM) in transferred from one to another to enrich both tackled eco- which ML has had a key role in the scientific discussion. systems, therefore revealing an accurate scope of the direction Additionally, we discuss the future perspectives of ML in EM ML applied to EM is going to follow in the next few years. and how the cross-fertilization with other fields can expand the experimental setups to a yet unexplored domain. From astro- physics, where imaging distant galaxies or searching for black 2. Electron microscopy advances with matter generates terabytes per second, to high-energy physics, machine learning where the particle colliders can hide particle interactions in a noisy background. From cryo-TEM to the broad window of 2.1. Most important imaging developments: from 2D images This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. optical microscopy techniques, to even geological sciences, to 4D-STEM Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. robotics, epidemiology dynamics or finances. For already many 2.1.1 Automation of the electron microscope vs. auto- years, these fields have in common that they have fed from mated data processing. Given an electron microscopy setup, machine learning to solve manifold scientific and technical with no hardware update, there are two main strategies towards challenges. Therefore, the potential of EM to mirror the pro- acquiring the best possible EM data, which fortunately are not gress done in these fields to solve its own future challenges is mutually exclusive: data postprocessing, and fine-tuning of the real, straightforward, and worth a review. This reading is mainly acquisition conditions. On the one hand, data cleaning or intended for the broad EM and materials science communities, denoising is of vital importance to pursue reproducible and although general microscopy or even data analysis communities statistically meaningful quantitative analysis on STEM data.1–10 may also find useful and mind-broadening approaches to their Therefore, using ML, which was already shining in denoising in hurdles. Next section details the progress of ML applied to other scientific and technical fields was the natural decision. imaging techniques ranging from traditional parallel-beam Here, the unsupervised approaches were the first approxima- TEM or Scanning TEM (STEM) to more recent breakthroughs tions that succeeded in this direction. Unsupervised ML has such as 4D-STEM. TEM spectroscopies and their ML-based been used to unveil structure and patterns in data and classify applications are discussed in this section as well. After this first them according to complex criteria. Unsupervised approaches part, the section that follows is intended as a pragmatic guide were the first to be applied and to succeed thanks to a versatility for the community newcomers to provide them with useful awarded by requiring no training and only relying on the data (mostly open-source) tools and state-of-the-art computational to which they are applied.11–13 Therefore, the first literature frameworks developed to apply ML to experimental datasets in a pieces applying unsupervised routines to EM data were mainly general and less case-dependant basis. Finally, the last part of used to denoise spectral data. The pioneer works of P. Trebbia this review correlates EM with other fields by exploring experi- and N. Bonnet represented the starting point for dealing with ments and data analysis routines that could easily be Electron Energy Loss Spectroscopy (EELS) with multivariate statistics and currently are a standard first step in any post- processing routine. These ideas flattened the path towards more exotic denoising routines such as unsupervised Gaussian Jordi Arbiol graduated in Physics Processing (GP) for cleaner strain mapping.11,12,14–17 at Universitat de Barcelona in Nevertheless, just recently more complex supervised methods 1997, where also obtained his are emerging and showing their potential to push the current PhD in 2001. Since 2015 he is methods forward.18–20 Generally, supervised ML, specially Deep ICREA Prof. at Institut Català de Learning (DL), aims to mimic human-level operation by finding Nanociència i Nanotecnologia complex regressions that adapt to the data. Instead, unsupervised (ICN2). He was President of the ML is the preferred choice when an exploratory approach based Spanish Microscopy Society on unveiling the internal data structure is needed, although it is (SME) (2017–2021) and Vice- still useful for application engineering to provide a more versatile President (2013–2017). Since but typically less general performance than supervised solutions. 2019, he is Member of the Execu- The pros and cons of each approach are more detailed in the next tive Board of the International section and extensively throughout the review. In fact, the Jordi Arbiol Federation of Societies for Micro- denoising ideas tackled by unsupervised approaches are being scopy (IFSM). He is Scientific explored from the supervised perspective and beyond. C. Lee Supervisor of the Joint Electron Microscopy Center at ALBA Syn- et al. could increase the Signal-to-Noise Ratio (SNR) in an annular chrotron (JEMCA) and Founder member of e-DREAM (R. Ciancio, R. dark-field STEM image of a single-atom defect and measure the E. Dunin-Borkowski, E. Snoeck, M. Kociak, R. Holmestad, strain field around it by training a Fully Convolutional Neural J. Verbeeck, A. I. Kirkland, G. Kothleitner and J. Arbiol, e- Network (FCNN) on different defect types.21 Then, the initially DREAM: the European Distributed Research Infrastructure for hidden-by-noise but theoretically predicted strain field could only Advanced Electron Microscopy, Microsc. Microanal., 2022, 28, be measured after this key postprocessing step. This was because 2900–2902). He is author of 426 scientific publications, with more FCNN are primarily designed to find shared patterns in than 25 700 citations and a h-index of 90 (GoS). image sets. 1428 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review The successful proofs of concept made additional denoising analysis of beam-sensitive materials by training a sparse coding alternatives arise, including a denoising–noising Generative model on simulations.32–34 Adversarial Network (GAN) for the active denoising of STEM On the other hand, computer vision routines for automating data (Fig. 2 GAN)22–24 or case-independent denoising models the electron microscope alignments have already been reported that successfully outperformed classical restoration filters for in order to fine-tune the (S)TEM acquisition process. The first both TEM and STEM.23,25–31 Interestingly J. Vincent et al. approaches, mainly lead by microscope manufacturers, were studied the latent features learned by the DL model to unveil based on automating the column (fine) alignments through. For the nature of the trained denoising dependencies to shine light instance, solutions for the automatic aberration correction, or the on what is typically left as a black box. They showed that the aberration-free converge angle maximisation are available both This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. FCNN learns to adapt its filtering strategies depending on the through manufacturers and in the literature (Fig. 1a).35–37 In fact, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. structural properties of every particular region in the recent results exploit these provided tools to automate the image.26–28 Importantly, ML-based noise reduction methods also detection of features of interest in the TEM and their online reached niche EM experimental approaches such as electron classification by Few-Shot Learning (FSL).38 FSL is a sub-area of holography. This latest example highlights the significance that supervised ML, where new data is classified, despite models the ML-based methodology may eventually have in the field and are trained with only a few training samples. Meaningfully, how broad it might be. Electron holography has also benefited K. Roccapriore et al. could automate the dynamic STEM explora- from ML-based noise reduction methods. For instance, ML has tion and EELS acquisition, through training a deep kernel cap- been used to obtain the accurate phase estimation at low SNR, able of actively distinguishing physically-significant regions.39 which flattened the path towards the low-dose holography However, acquisition and alignment automation are not limited Fig. 1 The two main trends of Machine Learning (ML) for Electron Microscopy (EM): (a) representing the automation of the microscope tuning and data acquisition, and (b) showing the potential of ML in advanced data analysis. (a) Research work from JEOL devoted to the automatic measurement and correction of aberrations through Ronchigram ML analysis. Each column is linked to a different defocus. The top and bottom rows display simulated Ronchigrams with aberrations determined manually (top) or by a Convolutional Neural Network (CNN) (bottom), while the mid row corresponds to the experimentally equivalent ones.37 R. Sagawa et al., Microsc. Microanal., 2021, 27, 814–816, reproduced with permission. (b) Diagram of a CNN trained to count the atoms of an atomic column in gold nanoparticles. The CNN admits both single images and focal series as inputs to classify every atomic column by setting, in the depicted case, a probability (PX) of containing from 1 (P0) to 6 (P5) atoms.66 Reproduced with permission of J. Madsen et al., Adv. Theory Simulat., 2018, 1, 1–12 Copyright 2018 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1429 View Article Online Review Nanoscale Horizons to experiment optimisation, and was key to open unprecedented The next step in complexity comprises the defect detection experimental setups impossible otherwise. In that sense, in 3D systems. For this, the available research is sparser and E. Rotunno et al. trained a CNN to align an orbital angular mostly presenting a theoretical scope. The proofs of concept are momentum sorter in the context of beam shaping experiments. in relatively simple systems such as zinc blende GaAs, and for The authors demonstrated a high accuracy and computational seeing its future steps it will certainly be of interest to resemble speed in estimating the aberrations induced by the sorter. Such a the progress lastly achieved in SPM with more complex 3D performance opened the possibility to implement this methodol- systems, just as expected with the self-driven experiments.68–70 ogy to every optical system and aberration corrector in the As a result, we see an outstanding opportunity for pushing the electron microscope for real-time self-alignment.40–44 Parallelly, state-of-the-art defect detection to a broader spectrum of 3D This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. optical microscopy on, for example, the real-time estimation of systems. To achieve this, CNN combined with unsupervised Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. the wavefront aberration, and Scanning Probe Microscopies clustering or anomaly detection methods seem to be the way to (SPM) on thoroughly autonomous experimental setups, are set- go. We envision that this general model will also include the ting the basis for the automation of more complex microscopy identification of planar defects and dislocations, both in an setups.43–51 These meaningful results, though, constitute nothing atomistic nature, and in a more macroscopic or industrially- but baby steps towards an eventual and highly anticipated fully oriented basis.71–73 automated TEM, which is where ML is called to play a pilar role 2.1.2 Exploratory and knowledge-revealing routines. ML is in skyrocketing its advances in the next decade.52,53 To achieve exceling in the exploration of local descriptors and knowledge this, the industry-science cross-fertilisation will be mandatory to extraction. Most of the available work relies on supervised deep open the microscopes up for implementing the research com- neural networks or in unsupervised complex routines for munity advances in a user-friendly manner. Moreover, it is our dimensionality reduction or classification. Nonetheless, we community’s duty to validate open and universal data formats to anticipate that the unsupervised generative models will be push their support by the companies.54,55 the key players in pushing the limits of what data science However, the current bottleneck of the workflow lies in the can do for EM. This should drastically reduce the typically time- analysis of the acquired data, not in the acquisition itself. consuming and tedious generation of training sets, and Therefore, most of the current efforts are pushing towards remove the excessive tailoring and fine-tuning associated with the automated analysis and ML-based knowledge extraction. traditional unsupervised methods. This latter idea will be For instance, CNNs have widely emerged as excellent tools for tackled further in Section 4. In fact, it is an expected heritage the identification, classification, and quantification of defects from other scientific fields that opens new possibilities for in EM data. In this way, there is an abundant and successful materials science and EM.74 For the sake of simplicity, we enough literature to constitute a strong enough basis to con- divide the following exploratory studies in either unsupervised glomerate a starting unitary and general Artificial Intelligence or supervised approaches. (AI) model that would replace the human intervention in this specific characterisation task. The first studies faced the pre- 2.1.2.1 Unsupervised exploratory routines. As mentioned liminary complexity of 2D materials in STEM, which allowed earlier, the path of unsupervised ML in EM started with the the direct correlation between image and structure.56–58 multivariate analysis of EELS spectra, mainly aimed for noise M. Ziatdinov et al. trained an encoder–decoder FCNN to detect reduction.11,12,16 Since the very first nearest neighbour algo- the atomic coordinates of Si-doped graphene. Importantly, the rithm that aimed to find the shortest path to interconnect data trained model proved its ability to adapt to another 2D system, points, unsupervised learning has rapidly grown in popularity Mo1 xWxSe2, showing a generalised performance. Moreover, in manifold ways and fields. Its strongest virtue is the absence the authors converted the outputted atomic coordinates into of training process, making its easy access and implementation graphs (i.e., atoms as the graph nodes, with the chemical bonds its key to success. Although the main drawback of unsupervised being the links) for the automated classification of the Si routines is their lack of robustness when aiming for general dopants (namely, point defects) based on their bonding with models, they can provide an excellent platform for exploring neighbouring Si and C atoms (Fig. 2 CNN).59,60 This idea has specific imaged systems. extensively been reproduced in further research in similar 2D Unsupervised ML on imaging techniques followed the logical systems by taking advantage of the monotonic (with the atomic path started by the multivariate analysis of spectra. Principal weight and thickness) dark-field STEM signal.21,61–64 On the Component Analysis (PCA) constitutes the most straightforward other hand, tackling harder to interpret signal, J. Madsen et al. dimensionality reduction and data decomposition available, trained a similar FCNN architecture65 on High-Resolution TEM allowing both data cleaning and classification. Nevertheless, it (HRTEM) simulated micrographs, including focal series and can also be key in the core of more intricate routines. PCA can therefore phase information, to identify the atomic coordinates map the crystal phases and defects, such as twins or phase of graphene and count atoms in gold nanoparticles (Fig. 1b).66 boundaries, in encoded dichalcogenide (i.e., MoSe2 and WS2) This core idea extended the analysis of HRTEM defects micrographs with rotational-invariance (i.e., independently of data further to surface contaminants on more complex gra- their in-(image) plane rotation).56,75 The main drawback of PCA phene configurations or to quantitative atom counting in Au is its pure mathematical nature and the inability to directly nanoparticles.2,4,66,67 correlate the obtained results with a physical interpretation. 1430 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Fig. 2 A selection of the key Machine Learning (ML) methods most recurrently employed in Electron Microscopy (EM) imaging and spectroscopy. The first layer contains the first ML techniques that arose in EM, unsupervised methods, consisting of Gaussian Processing14 (GP), Principal Component Analysis and Non-negative Matrix Factorisation100 (PCA/NMF) (reprinted from F. Uesugi et al., Ultramicroscopy, 2021, 221, 113168 Copyright 2021 with permission from Elsevier), fuzzy c-clustering78 (C-cluster), Vertex Component Analysis101 (VCA) (M. Jacob et al., Microsc. Microanal., 2019, 25(suppl. 2), 156–157, reproduced with permission), among others. The evolution in complexity leads to the second layer of supervised techniques, with MultiLayer Perceptron102 (MLP), and Convolutional Neural Networks59 (CNN) (adapted with permission from M. Ziatdinov et al., ACS Nano, 2017, 11(12), 12742– 12752 Copyright 2017 American Chemical Society) and Recurrent NN103 (RNN). The third layer stands for Compressed Sensing104 (CS) (reproduced with permission of K. Kelley et al., Small, 2020, 2002878 Copyright 2020 Willey-VCH GmbH), a group of alternative algorithms with powerful compatibility with ML. Finally, the last layer outputs the future of ML in EM in the use of generative models like Variational AutoEncoders (VAE) and Generative Adversarial Networks22 (GAN), and more complex learning paradigms such as Reinforcement Learning (RL). This is why alternative physically-constrained methods arose. For instance, R. Kannan et al. tested NMF in spectral data For instance, ensuring every component is thoroughly positive generated by multimodal STEM and X-ray Diffraction (XRD).58 in its domain with Non-negative Matrix Factorisation (NMF). The authors generated a hyperspectral dataset from the sliding This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1431 View Article Online Review Nanoscale Horizons window Fast Fourier Transform (FFT) of a single atomically dataset representing the statistical variability of the problem resolved STEM image. Then, they mapped the crystallographic data. The key idea is that after engineering a model (i.e. the phase by assigning a phase to each meaningful NMF component neural network) capable of recognising the statistical variability extending the previous PCA approach further.58,73,76,77 Similarly, and descriptors of the data (i.e. training process), any data lying B. Martineau et al. also applied NMF, combining it with fuzzy within these statistical limits could be automatically and c-means clustering (i.e., assigning each datum a probability of robustly analysed. Thus, in the same way a linear regression being in each cluster), to overcome the effects of sample bending would adapt to data following a linear trend, neural networks on the diffraction patterns obtained in precession mode while generate complex non-linear multi-dimensional regressions scanning twinned GaAs nanowires (Fig. 2 C-cluster).78 that can adapt, in theory, to any data structure. However, the This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. The outstanding balance between the ease of implementation main drawback relies on the resource-consuming model set-up, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. and the remarkable results obtained by PCA or NMF raised the which essentially consists of generating and getting the train- interest to explore higher-complexity unsupervised routines.63,79–81 ing data ready, and tailoring the model architecture to the Variational AutoEncoders (VAEs) are emerging as a powerful problem. Nevertheless, recent results elucidated that the auto- dimensionality reduction tool able to extract physically meaningful matic AI-based architecture and hyperparameter fine-tuning information (Fig. 2 VAE). Although they are catalogued as unsu- (i.e. the supervised model properties) is currently possible for pervised processes, they require the manual specification of the EM problems and datasets.92–95 features of interest of the data for that study. For example, in a Currently, the main trend of DL in the field has been the crystal phase classifier, the mapping of the phases throughout the finding of atomic positions and their correlation with mathema- micrograph should be manually given a priori as input. Then, by tical graphs, as stated before, for defect identification59–61,66,67,70 specifying the features of interest in the micrographs, the data is and quantification,96 or image denoising.21,26–28 Nevertheless, compressed and decompressed in an encoder–decoder architec- the extraction of further properties with physical meaning can ture that generates a few latent variables that ‘‘simply’’ explain the be envisioned from this idea, such as atom counting or variability of the original image.82–84 These compressed latent quantitative TEM in a broader sense.3,4,66 Interestingly, the variables can further go through complementary clustering or atomic positions identification started with FCNN on 2D sys- refinement routines to eventually be correlated with physically tems and model 3D systems. However, FCNN extended their meaningful features, such as local crystallography, defects, or reach to higher complexity systems, therefore exhibiting their subtler sample-dependent identifiers.85–87 Relevantly, S. Kalinin generalisation capabilities. For example, M. Ziatdinov et al. et al. used rotationally invariant VAEs (rVAEs) to explore the trained a FCNN to detect the atomic positions of a La-doped evolution of Si on graphene under the electron beam. To achieve BiFeO3 system to extract local descriptors of the lattice such as so, the authors encoded the variability of this model system in just the polarisation.97 Importantly, local descriptors coming from three latent variables: rotations, and x and y translations. The supervised networks can be post-processed by unsupervised latest proved the robustness of VAEs to track time-resolved data, means to group them in physically equivalent categories. outperforming traditional unmixing methods by capturing the Indeed, the authors compressed the supervised output with rotation information in just a single latent variable and clarify- PCA and grouped it by k-means clustering. As a result, they ing the origin of the remaining variability in other independent could map back the distribution of the lattice distortions in the variables.88 As indicated by their name, rVAEs are phenomenal original image.97,98 Multimodal approaches, as the former tools to evaluate features that are susceptible to suffer in-plane example, mostly introduced in atomic column finding and rotations, which may be particularly advantageous for ferro- phase mapping, are of huge importance nowadays to go electric, ferromagnetic or generally polar materials. In that through the intrinsic limitations of standalone models. These sense, S. Kalinin et al. applied rVAEs to correlate the rotation methodologies are common for dealing with data outputted latent variable with the orientation of ferroic variants, directly from neural networks, as their formats are susceptible to being locating the unit cell deformations through the sample, inde- further simplified by classical unsupervised routines or by more pendently of the structural and chemical variability and under complex algorithms such as VAEs.85,89,99 In fact, most of the non-ideal imaging conditions.89 DL-based studies available in the literature for atomic column- positioning routines interact with similar data (roughly, atomic 2.1.2.2 Supervised exploratory routines. Mimicking the intri- columns ‘‘always’’ appear as rounded dots in micrographs). Thus, cate neural structure of the human brain, and even building the first step towards a universal model could be a shared new neuromorphic hardware architectures, is the latest revolu- transfer learning-based starting point, which would save time tionary idea in computer science, together with quantum and resources in updating and tailoring it for any specific computing, to overcome the computing capabilities of classical or individual need. And for that, the attention paid to these algorithmics. Since the very first perceptron proposed by multimodal approaches would perfectly fit in the inter- W. Pitts and W. McCulloch, to the explosion of DL, neural communication of the specific models towards a more general networks and other supervised algorithms have systematically and beneficial model. outperformed classical classification and regression tasks.90,91 The next complexity stage for supervised algorithms was Supervised routines, and paradigmatically DL, are based on shifting to the reciprocal space analysis. Fortunately, ML methods a training process that requires the preparation of a large are sensitive to geometry to easily detect symmetry constraints. 1432 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review This makes them excellent options for dealing with crystals generating a supervised model that fits into the target informa- and their reciprocal space description. In order to simplify tion. However, the solutions to these tailored research pro- the electron diffraction data and reduce the computational blems are meant to converge into broader unified models with pressure, the 2D diffractograms can be turned into 1D spectra. a customisable or modular nature gathering all the developed In that sense, as J. Aguiar et al. proposed, a CNN can be trained features at once. In fact, the latest EM hardware evolutions and on electron diffraction data or equivalently, FFTs obtained their software assistance were meaningfully based on automa- on atomic resolution STEM micrographs, to identify the space tion. It was fundamental for cryo-TEM and its autonomous group of a given unit cell.105 The key idea relies on the azimuthal particle search and analysis, and also played a role in delivering integration of diffraction patterns (or FFTs) to generate a line intuitive parallelisation capabilities to the newest microscopes This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. profile containing the whole information in a simplified encoding. and focused ion beam machines. Therefore, it is important to Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. The authors used a huge dataset with 571.340 crystals to reach a emphasise the effort led by the computational EM community classification confidence of 95% in regular SNR scenarios, and of towards the open software and its easy accessibility and the 70% in noisier data. Similarly, and by directly tackling 2D centralisation. This idea should make the next incoming hard- diffraction data, R. Vasudevan et al. worked on a CNN capable of ware and software evolution of EM and materials science be determining the Bravais lattice of both experimental and simu- deeply based on ML. Initiatives such as sharing public codes, lated Scanning Tunnelling Microscopy (STM) and STEM images. libraries and repositories, or ‘‘papers with code’’ (i.e., Jupyter Indeed, the authors mapped this Bravais symmetry distribution notebooks, Google Colab notebooks) to straightforwardly repro- within a time-resolved set of images of electron beam-damaged duce the data analysis shown in the publications, foresee a new WS2.61,106,107 Interestingly, these constitute preliminary signs high- format for the papers that journals will need to embrace from lighting the interest to apply supervised methods, and generally now.89,97,105 Interestingly, these paradigmatic tendencies con- ML, to evolving systems or in situ setups. This idea is reviewed in firm that this new incoming evolution is already here. detail a few paragraphs below. 2.1.3 Processing of high dimensionality data by machine The high versatility of supervised methods confirmed their learning. In the following, we evaluate how ML is inducing ease to handle manifold experimental data productively. None- welcomed changes in the way in which higher dimensionality theless, the ML practitioners in the community attempted EM data is processed. The new generation of pixelated detec- the ML implementation in microscopy simulations, too. tors is powering the acquisition of large data volumes, already Simulations are key in EM to facilitate the interpretation of close to the domains of big data. As a result, data handling micrographs taken under complex conditions, and also allow strategies relative to that field, such as processing the data in for virtual experiments on the characterised structures. In this fragments or chunks, need to be incorporated for the EM direction, R. Pennington et al. combined Density Functional community. Numerous techniques that fit into the high dimen- Theory (DFT) calculations and a neural network-based optimi- sionality data tag, like Lorentz microscopy or Differential Phase sation algorithm both to improve TEM simulations and to Contrast (DPC) STEM, have benefited from ML to improve the retrieve properties along the beam propagation dimension retrieval of the phase information.113 Nevertheless, in the next (properties like ferroelectric polarisation domains and strain). paragraphs we will mainly focus on the management of huge data In this latter case, this methodology was tested with simulated streams generated by ptychography and 4D-STEM nanodiffraction. Convergent-Beam Electron Diffraction (CBED) patterns, although Ptychography is a promising technique based on the math- it should be comparable to the available imaging and spectro- ematical reconstruction of the electron phase information (i.e. scopic modes.103,108 Indeed, literature extensively showed that the inverse problem) based on the acquired experimental features learnt by DL may extrapolate to data which a priori had signal. It has recently proven its high value achieving the no relation with the training data. Interestingly, the way DL current spatial resolution record in EM, 16 pm, only limited models learn these complex routines is still mostly unknown to by lattice vibrations.114–118 Whilst it is still an early technique, date in a general basis. The generated weights of the networks are the community has identified the need to enhance the compu- task-specific, which hampers distilling general and common tations assisting the reconstructions. For that, ML applied to patterns. J. Horwath et al. carefully checked the learned convolu- ptychography is currently relying on optimising the phase tional filters of a U-Net architecture intended to segment TEM retrieval routines.119 For instance, M. Schloz et al. combined images of rounded gold nanoparticles.109 They learnt that the the multislice formalism (interestingly, implemented with a kernels filters could be easily engineered by combinations of multilayer perceptron, which is a fully-connected neural network) traditional filters (i.e., Laplacian, Gaussian, etc.), emphasising the with gradient descent regularised optimisation to perform the importance of tailoring the model architecture for a task.26,109–112 reconstruction. The multislice formalism accounting for multiple At this point, it is important not only to clarify the nature of DL scattering improved the resolution, and the regularisation applied to (materials) science, from a computer science perspec- allowed to diminish the oversampling requirements and still tive, but also to extract individual filters repeatedly observed in reconstruct the experimental data under noisy conditions.120 networks sharing objectives and apply them to simplify specific After improving the reconstruction mechanisms and relaxing data treatment problems. its constraints, the ML practitioner’s aim is to make ptycho- As reviewed, in EM the typical research workflow involving graphy an end-to-end process. In this way, M. Cherukara et al. supervised ML consists of tackling a specific problem and trained a FCNN (named PtychoNN) to retrieve both the This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1433 View Article Online Review Nanoscale Horizons amplitude and phase from diffraction data. The authors claimed more general and goes beyond ptychography. Nanobeam diffrac- to achieve a fast, sub-sampling tolerant and computationally- tion patterns of very wide fields of view can be obtained by friendly method to solve the inverse problem in real time, in a means of 4D-STEM nanodiffraction (generally just 4D-STEM for single step (Fig. 3a and b).121 The computational speed of the simplicity). In order to collect the 4D-STEM 2D diffraction FCNN opened real-time ptychography, which could be poten- patterns, pixelated detectors are a cornerstone that also received tially used to apply ptychography in dose-sensitive and thick the attention of ML practitioners.124–128 In fact, so far, we have samples. Consequently, ML pushed even more the relaxation of reviewed how close the ML software development can be with the the required electron dose with advanced methods. DL, specifi- progress of the actual hardware receiving the signals. Despite cally deep Reinforcement Learning (RL), was used for this that, it is not common in the bibliography to directly face the This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. purpose. RL learns by rewarding the behaviours or patterns we hardware, or as in the following case, the capabilities of the Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. are interested in, while penalising the rest. For that, RL was before mentioned detectors. Interestingly enough, G. Correa studied to generate real-time adaptive scanning paths that and D. Muller exceptionally pushed the performance of these benefited the relaxation of the overdetermination constraints detectors by training a CNN on Monte Carlo electron beam towards low dose experiments (Fig. 2 RL). For that, a Recurrent trajectories. The idea was the prediction of experimental beam Neural Network (RNN) rewarded the most physically meaningful paths and detector hitting spots to achieve sub-pixel super- diffraction patterns (i.e., highest phase dynamic range) and resolution.129 Again, similar research is currently scarce in the automatically engineered the most optimal scanning path for the literature, but certainly illustrates what we were referring to as reconstruction.122,123 This example of ptychography expanding the next coupled software–hardware evolution in EM. its target materials to beam-sensitive and thick devices is a It is currently much more common to find ML on 4D-STEM paradigmatic victory of ML. It accentuates that even in refined focusing on facilitating the extraction of physical properties of techniques capable of leading to super resolution, ML can still huge datasets. To do so, traditional algorithms would inevitably play a role improving the classical approach. Nevertheless, the take ages. A common approach, as happens with regular acquisition of diffraction patterns to generate a 4D dataset is atomically-resolved (S)TEM data, is to employ unsupervised Fig. 3 Machine Learning (ML) applied to Electron Microscopy (EM) high-dimensionality data analysis. (a) An encoder–double decoder convolutional neural network dubbed PtychoNN designed for the ptychography reconstruction of the amplitude and phase information out of single diffraction patterns. PtychoNN allowed (b) to reconstruct the wave information with higher fidelity than traditional iterative phase retrieval algorithms, and up to 300 times faster.121 Reprinted from M. J. Cherukara et al., Appl. Phys. Lett., 2020, 117, with the permission of AIP Publishing. (c) Comparison of the accuracy of the Electron Tomography (ET) reconstruction of an iron oxide nanoparticle with a concavity between the SIRT algorithm and a compressed sensing (CS)-based one. The CS algorithm reproducibly estimated the concavity size independently of the number of acquired projections while the SIRT approximation was strongly affected by that.148 Reprinted with permission from Z. Saghi et al., Nano Lett., 2011, 11, 4666–4673. Copyright 2011 American Chemical Society. (d) ET reconstruction by, from left to right, the SIRT algorithm with 151 acquired projections, and a Neural Network (NN), SIRT and WBP algorithms with 10 projections, and their orthoslices in the three bottom rows. The NN overwhelmingly improved the 10 projections SIRT and WBP reconstruction, but also the reference 151-projections SIRT. The NN orthoslices clarified its superior performance highlighting the potential for low-dose ET.154 Reprinted from E. Bladt et al., Ultramicroscopy, 2015, 158, 81–88, Copyright (2015), with permission from Elsevier. 1434 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review methods to give a physical origin to a mathematical component, reconstruct additional challenging biological needle-shaped such as assigning each NMF/PCA component a characteristic tomograms by means of CS (Fig. 3c).146,148 Besides, the promis- diffraction pattern. Its recent success unveils unsupervised ing achievements of CS in the field opened the tendency of ML techniques as a standard for the processing of high-dimensional to emulate these algorithms and make their implementation datasets, resembling the multimodal crystal phase mapping we more agile. This played a substantial role after involving DL in described for regular STEM.130–135 Indeed, the mapping of crystal the equation.149–153 For instance, E. Bladt et al. went beyond CS phases and their relative rotations was tested with success in with the implementation of a multilayer perceptron capable of complex oxide systems (e.g., Ti0.87O2 vs. Ti2O3) and in dichalco- automatically reconstructing the sample upon a sparse tilt genide multilayers, such as MoS2 bilayers.100,136 series, without the user-defined prior knowledge of the sample This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Supervised ML has also been proven beneficial for 4D-STEM, typically required in traditional reconstruction algorithms Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. as regarded in ptychography. Nonetheless, its mere considera- (Fig. 3d).154–156 Interestingly, as it happens with ML mimicking tion can become thoroughly more time-consuming than in DFT and multislice, ML copying CS enforces the future of ML regular (S)TEM, given the higher dimensionality of the former overwriting classical algorithms and making them worth a try and the consequent higher-complexity network-architectures when contemplating a scientific problem. required.137 Explicitly, the high dimensionality engineering of Beyond tomography, CS burst in with the development of training data would nowadays become the main implementation non-rectangular scanning paths in STEM, which promise faster barrier. Therefore, unsupervised methods would constitute the and lower dose STEM experiments.157,158 The idea lies in the best starting point for newcomers diving into the smart proces- measurement of (random) sparse pixels and the mathematical sing of 4D datasets. Nevertheless, the preparation of useful 4D inference of the unsampled signal. This approach was success- training sets is still possible as showed by the 3D-CNN trained on ful for practical scenarios such as undersampled lattice distor- simulated diffraction patterns arising from a LaAlO3–SrTiO3 tions like point defects, in both STEM imaging and EELS.159 heterostructure.138 In this case, the capability of the resulting However, as indicated, the velocity boost is not the main model was to distil whether the interface presented atomically attraction. Its application for beam-sensitive nanostructures sharp steps or a chemically diffuse nature, a task often compli- as a way to ensure not surpassing the maximum allowed dose cated even for trained microscopists. The necessary tools for was also corroborated experimentally. X. Li et al. could perform reproducing this and the previous cases, as well as to deploy real-time CS-based reconstructions on beam-sensitive materials custom models on 4D data are described in detail in Section 3. (i.e., graphene) with nonrectangular spiral and Lissajous 2.1.4 Compressed sensing and machine learning for tomo- scans.160 Interestingly, the scanning path optimisations were graphy and in situ. In the previous paragraphs we have pointed parallelly extended in SPM following this same logic.104,160,161 It out that general statistical parameter estimation and DL is not surprising, then, that as happened with CS-tomography, methods allowed for atom counting and local structure DL also emulated the goodness of CS in STEM path optimisation. determination.3,4,9,29,66,139,140 These methods may result in an For that, J. Ede and R. Beanland modelled a GAN to automatically eventual 3D model representative of the initially unknown reconstruct nonrectangular scan paths. Based on this, J. Ede nanostructure. Even though these tools can become powerful additionally created a sample-aware adaptive scan path learned enough to solve certain case studies, the reproducible extrac- by reinforcing a RNN that scores the reconstruction of the GAN tion of reliable 3D information compulsorily demands Electron from the previous study (Fig. 2 RNN).162,163 This research piece Tomography (ET). In order to relax the time-consuming and constitutes one of the most sophisticated examples of ML applied high-dose experimental setups that this technique usually to EM currently available. Nonetheless, it unveils that the exper- demands, Compressive Sensing (CS) has recently become a tise in a wide ML toolset is key for bringing the applications into keystone for the recent results in the field.141,142 In just a few a next level. Indeed, it is the shown stacking of complementary words, CS allows to reconstruct a randomly-sampled sparse ML tools (referred previously as multimodal ML) what would lead signal beyond the Nyquist limit by means of a mathematical to general ML models for EM. In favour of this, in Section 3 we algorithm that contemplates the sampling path. Although review the currently available ML toolset for easily transferring strictly speaking CS does not fit inside the ML tag, its shared this knowledge into helpful algorithms, in a practical and easy- objective and similar methodology make it worthwhile being going way. considered in this review. CS has also showed potential for compressing the data As mentioned earlier, the research on CS applied to tomo- volume associated to time series in in situ experiments.164 As graphy mostly tries to improve the current tomogram recon- introduced before, this shines light on the important role ML struction schemes. Virtually every experimental setup would can play when dealing with in situ experimental setups, topic benefit from having its total dose reduced. For that, CS pointed that will conclude this first section of the review.165–168 In the towards a dose-reduced but also real-time and shape- previous paragraphs we have already reviewed some examples independent tomography.143–147 This idea provided extra sensitivity in which the effect of the beam, mainly on 2D materials, was to morphological features difficult to resolve, such as surface carefully considered. Specifically, the potential reproducer of rugosity and porosity, also allowing the tomography of challenging these experiments would find value in the evaluation of shapes. For example, Z. Saghi et al. could both quantitatively changing defects or phase distributions.26,29,61,88,109 For evaluate the concavities of iron oxide nanoparticles and to instance, T. Patra et al. studied MoS2 in different time scales This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1435 View Article Online Review Nanoscale Horizons by combining ML-enhanced dynamic simulations of defects At the same time, the computing efficiency of ML and the with time-resolved HRTEM. They found that the long-time scale advent of fast detectors could open the real-time correlation of displacement of the defects could lead to rapid (up to pico- in situ experiments with parallel simulations that automatically seconds) 2H to 1T phase transitions.62 Remarkably, ML record and explain the evolution of materials in a limitless unlocked the required time resolution to capture this ultrafast scenario of knowledge extraction. phase transitions invisible to classical in situ data analysis. As a final comment for this first section devoted to imaging, The current investigations involving ML on in situ experiments we cannot forget about those technical fields in which ML has are only on preliminary stages, fundamentally facing relatively extensively helped to push the limits of the achievable. In fact, simple systems such as nanoparticles, and their evolution in size as this review is mainly intended for the materials science This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. and morphology. For these primary applications, the methodol- community, we have not covered ML on cryo-TEM, in which the Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. ogy to deploy is not advanced either, which makes it even more problems solved are mainly bio-related. However, it is important accessible to newcomer ML practitioners. Some of these works to keep it in mind when trying to apply ML to materials science rely on combining traditional computer vision routines (i.e., (or to any other field), as similar routines might have already thresholding and edge detection) with unsupervised clustering been developed, for instance, within the mentioned cryo-TEM to draw the contours and central positions of the particles.169,170 community.177–192 The cross-fertilization with not only other In this context, the clustering could be understood as a sample- microscopy techniques, but other scientific and technical dis- aware thresholding that better adapts to the changing conditions ciplines is of capital importance and is deeply discussed in the of time series or dynamic stimuli. In that sense, Y. Qian et al. fourth section of this review. developed an unsupervised segmentation routine that is repre- sentative of what can be obtained by taking advantage of rela- 2.2. Most important advances in spectroscopy tively simple clustering routines. The authors pooled the k-means The previous section highlights the latest ML developments in clustering (intensity evaluation) with the edge detection (gradient the EM community to extract information from images. Even evaluation) to extract complete statistics from videos of silica though we have seen strategies for dealing with high- nanoparticles.171,172 Interestingly, the authors achieved a robust dimensionality data such as 4D-STEM related techniques, most method that was tested on more complex situations such as the of the processes tackle 2D signals (mainly images). In fact, this environmental TEM study of the formation of Fe nanoparticles is because computer vision has evolved fast in fields such as after the dewetting of a Fe thin film.173 autonomous driving or macroscopic pattern recognition, in The previous example remarked that evolving systems may which the way to communicate the information to the compu- find in simple algorithms the best ally to capture their ter is through digital imaging. Therefore, the processing of dynamics without the need to deploy complex codes. However, more complex data, such as the generated by Electron Spectro- abrupt intensity changes in the micrographs coming from, for scopy (ES), still remains a challenge even more arduous than instance, thickness variations, or compositional changes can the previously reviewed EM image analysis.193 This section make unsupervised routines miss. If this case is identified, it shines light on whether the advances on spectroscopy can would probably require a supervised approach specifically compare to the imaging ones, and in which situations the trained for these observed image variations. This may allow employed methods can share convenience. In the process, we to push the conditions of the tolerated in situ experiments and pinpoint suggestions to push the development of these data still get robustness, at the cost model-design resources. For types, trying to imitate the ascending trend observed in EM image example, supervised routines based on the CNN U-Net archi- processing. Indeed, opposed to ML for imaging, the reader may tecture successfully segmented and tracked the statistics of find, both in this review and in an autonomous literature nanoparticles. As U-Net was the first notorious DL model for research, a substantially smaller number of ML-related research dealing with scientific images (i.e. segmentation of cells), most works devoted to ES. By now, ES has mainly received a much of the currently available CNN for microscopy are fine tunings simpler ML modelling, the architectures of which head the of U-Net.65 Then, before the time-consuming neural network following subsections. architecture design, it is always worth to try first with U-Net as a 2.2.1 Unsupervised machine learning. Experimental ES reliable supervised proof of concept. In fact, meaningful results setups typically lead to 1D or 3D datasets. Whilst a spectrum were obtained just with U-Net both in liquid and gas phases, in 1D can hold unmeasurably valuable information, to take and under temperature changes, surpassing the former tradi- advantage of the high spatial resolution of the TEM, we will tional and unsupervised approaches.174–176 In any case, it is normally end up with 3D datasets. In fact, the main reason for clear that ML has much more to say in in situ EM, and it will choosing EM and ES should be exploiting its archetypical high surely explode as soon as the in situ machinery becomes a much spatial resolution, which will systematically result in the micro- wider standard within the community. For instance, we envi- scopist dealing with high-dimensionality data. Nevertheless, sion the use of CS to reduce the required frame rate, similarly to the generation of synthetic high-dimensionality data for train- CS-tomography, further allowing the chemical and physical ing supervised models capable of dealing with these data tracking of beam-sensitive materials. Furthermore, we envisage structures is of immense complexity in ES. This is because it the use of VAEs to encode time-dependent meaningful descrip- would require either gathering numerous similar experiments, tors for its direct correlation with actual materials dynamics. or simulating them. Fortunately, the simulation routines to 1436 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review generate reliable spectroscopic data are manifold in literature. comparison between experiments and simulations.209–211 However, Unfortunately, all these time-consuming routines would demand none of these classical methods has been able to improve the SNR too many resources for generating a representative enough as straightforwardly as the nowadays standard post-acquisition training set.194–198 Moreover, its generated data would probably operations based on Singular Value Decomposition (SVD). engineer a too case-sensitive model not capable of generalising As described previously, the first contact of ML with ES was and justifying the time and resources spent on generating this precisely in the denoising of spectra via unsupervised unmixing training data. Therefore, by now, the most popular routines to methods. As a result, the examples in the literature highlighting deal with spectroscopic data are the unsupervised ones. As the benefits, limits, associated handicaps, and mostly the direct happened with imaging, ES identified the limitations of tradi- routinary application of (mostly) PCA to ES data is really This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. tional postprocessing and evolved by relying on ML. Representa- manifold.16,17,212–224 The idea behind unmixing is the simplifica- Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. tively, either improving the effective energy resolution or tion of the spectral features and its description as a function of increasing the SNR, postprocessing has always been of outmost key physical properties such as material type or spatial distribu- importance in spectroscopy.199–203 For instance, these improve- tion. Most of the unsupervised unmixing methods are variations ments have been used to emulate the effects of monochromators or inspirations of SVD, which is currently an incredibly well- without the costs of the hardware upgrade, or to spectrally documented and accessible method that is worth relying distinguish electronic changes at the interface of a semiconduc- on.225–227 Its use decomposes the data into components that tor heterostructure allowing to map a direct-to-indirect band-gap can be weighted in importance and reconstructed to dismiss any transition induced by a high epitaxial strain (Fig. 4d).204 In noise or inconsequential information (Fig. 4a). For example, imaging, for example, the denoising based on the non-rigid SVD and its alternative methods can be used to clean atomic averaged cross-correlation of single atomic columns was used resolution EDX and EELS maps without significative distortions for both Energy-Dispersive X-ray spectroscopy (EDX) and of the spectral fine structure.228,229 Importantly, the chosen word EELS,1,205–208 in addition to the denoising based on the ‘‘significative’’ is indeed significative, as special care must be Fig. 4 Unsupervised unmixing methods for hyperspectral analysis. (a) Principal Component Analysis (PCA) of a BN Electron Energy Loss Spectroscopy (EELS) Spectrum Image (SI) and the first six components, from top to bottom, with higher statistical significance. The presence of peaks and features of interest diminishes as going down in significance, where background or noise is the main feature. PCA does not physically constrain the components, as regarded by the negative losses.13 Reprinted from M. Bosman et al., Ultramicroscopy, 2006, 106, 1024–1032 Copyright (2006), with permission from Elsevier. (b) Independent Components Analysis (ICA) of an Energy Dispersive X-Ray (EDX) SI of Fe-based core–shell nanoparticles with the three main independent components. The independence of the components mapped the cores (IC#1), the shells (IC#0), and the C background (IC#2).249 Reproduced with permission of D. Rossouw et al., Part. Part. Syst. Charact., 2016, Copyright 2016 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim. (c) Vertex Component Analysis (VCA) of an EELS SI of BN/BOx nanoparticles, displaying the eight more important components. As the components were from the original SI, VCA was capable of mapping physically meaningful variations for each component such as particle edges vs. particle centres.250 Reprinted from N. Dobigeon and N. Brun, Ultramicroscopy, 2012, 120, 25–34 Copyright (2012), with permission from Elsevier. (d) Band-gap map of a ZnSe–ZnTe core–shell nanowire separating the areas where a direct or indirect band type was found. PCA was used on the low-loss SI to isolate the interfacial pixels between core and shell where the strain was accumulated, as visible in the Middle-Angle Annular Dark-Field (MAADF) micrograph.204 This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1437 View Article Online Review Nanoscale Horizons taken when unmixing signals and choosing the components to Following this direction, D. Rossouw et al. developed a compo- reconstruct. For that, P. Potapov and A. Lubk published a guide on sitional quantification routine based on dual-PCA + ICA and how to automatically choose the meaningful PCA components.230 dual-EDX + EELS. To prove it, the authors tested their method It was based on an initial smoothing of the data followed by the on FePt@Fe3O4 core–shell nanoparticles. Interestingly, they evaluation of the anisotropy of the generated scree plot and its solved the quantification of the spatially overlapping core and components. Finally, only those components not exhibiting the shell phases with the complementary spectral unmixing, and the characteristic isotropy of noise are selected for the reconstruction. mapping of both light and heavy atoms with the complementary Related to this, the introduced statistical bias as a function of the spectroscopy (Fig. 4b).249 truncated portion of the scree plot, total pixels and total energy The mixed use of PCA/ICA allowed to directly distinguish This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. channels was also extensively studied, facilitating the correct phases which would require an exhaustive and tedious least- Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. adoption of the method.225,226 squares fitting to be reproduced. However, the ICA assumption We previously introduced that despite all the advantages of of independence between spectral peaks might cause artefacts PCA, its main limitation is the lack of physical interpretability when dealing with similar phases (e.g. sharing elements or for its resulting components. To overcome so, alternatives oxidation states). This is where VCA can make a difference, as it physically constraining PCA became popular and demonstrated succeeds at identifying the individual spectral profiles (i.e. extended capabilities. In fact, the addition of physical con- combination of peaks) of distinguishable phases. VCA should straints to ML is the logic evolution to follow in the coming then be used when the expected components already appear in years, both in the unsupervised and supervised domains. isolated hyperspectral pixels representing a physically/chemi- Among others, as introduced in the imaging section, we can cally independent structure. This idea was validated in a find NMF, which forces the components to be positive, mimick- comparative study carried out by, N. Dobigeon and N. Brun, ing in this case the measured energies in ES; Independent who confronted PCA and ICA, least-square fitting and VCA. By Component Analysis (ICA), which only extracts statistically requiring gentler statistical assumptions, the authors claimed independent features assuming, for instance, that the spectral the superior interpretability of VCA versus both PCA/ICA for the peaks for each element are not correlated; while contrarily, finer analysis of high-complexity sample configurations Gaussian Mixture Modelling (GMM) assumes that all the (Fig. 4c).250 From this analysis we may get spatial information sources are governed by Gaussian distributions and separates from the mapping of spectral components that are extracted by the signals into a finite combination of these distributions; or just looking at the energy domain. Therefore, spatial correla- Vertex Component Analysis (VCA), which assumes the presence of tions, if present, may be ignored by just searching for simila- pure spectral signals, for instance, coming from pure materials, rities between spectra and not caring where the probe was. and evaluates their distribution through the hyperspectral Alternatively, and in order to further account for spatial corre- dataset.231,232 lations within the SI, S. Kalinin et al. proposed an unmixing Again, the literature referring unmixing methods for ES is method based on spatial Gaussian kernels. These kernels were bast. Specially from articles comparing, for a single application based on traditionally unmixed components that reduced the or for a similar purpose, some of these methods and confront- complexity of the energy domain and democratised the com- ing them towards finding the best performer.233–244 In a general putational needs of the GP. The process can be understood as basis, NMF was the preferred way to proceed when the major the convolution of the unmixed spectral features through the goal lied beyond denoising or data inspection. That is because spatial dimension, leading into higher-fidelity reconstructions and it can admit not only the direct physical significance of the mapping.251 Interestingly, the authors demonstrate the ease to extracted components, but to add additional constraints with tune and adapt the kernels to a combination of unmixed compo- ease. In this way, M. Shiga et al. broadened NMF by adding extra nents or to engineer them based on the physical characteristics we physical constraints that better imitate the statistical nature of want to find from the hyperspectral dataset. As a result, this both EDX and EELS. The constraints were the automatic prior- methodology was proposed as the preferred way to go if we handle based component relevance and soft orthogonality determination, highly spatially-correlated data (e.g., core–shell nanoparticles, although these constraints can be tailored to more complex setups nanoparticle clusters, compositional gradients,. . .). such as simultaneously acquired multimodal spectroscopy (multi- The main unsupervised routines used for ES, as commen- detector imaging, EELS, EDX, cathodoluminescence,. . .).245,246 ted, rely on unmixing methods. The decomposition in statis- Nevertheless, alternative unmixing methods do not necessarily tical components can be oriented to spectra denoising or need to substitute PCA or SVD, but to complement them. This is cleaning, for unlocking a smoother post-processing, or even the case for ICA. After the pioneer work of N. Bonnet and D. to map and quantify crystal phases and their stoichiometries. Nuzillard on applying ICA to simulated EELS, ICA has been mainly However, more intricate processes have been devoted to solve used to complement PCA. In fact, N. Bonnet and D. Nuzillard unattended matters such as spectral classification or the claimed that ICA was a complementary technique to the standard analysis of finer features such as the Energy Loss Near Edge unmixing methods, not a substitute, given that EELS does not Structure (ELNES). For this purpose, unsupervised clustering fulfil the condition of statistical independence.247 This comple- methods may represent an strategy worth relying on.252–256 The mentarity can make use of PCA to avoid overfitting and next ICA idea behind spectral clustering is to automatically detect fine can be used to spatially map crystal phases or elements.248 spectral changes and group them accordingly. As it is an 1438 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review unsupervised method, it would require the manual tuning of a available online in the software solution dubbed WhatEELS, the hyperparameter referring (somehow) to the number of final use of which can be complemented with many other practical groups we want to have (for instance, the total number of tools described in detail in the third section of this review.260 In different crystal phases in a Spectrum Image (SI)). For example, fact, clustering and unmixing are not exclusive for core losses and we can create a cluster for each spectra representing the can also be applied to low-loss EELS, for instance, to map different ELNES oxygen K edges. In fact, this kind of reasoning plasmons or even in the future to map with high accuracy the can act as the building block for elaborating models further. electronic properties of nanomaterials such as the topology of the Representatively, S. Kiyohara et al. generated a decision tree band structure.204,261 (supervised learning) that dealt with the clusters representing Independently on the spectral range in which it was applied, This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. the ELNES oxygen K edges.257,258 As its name points, a decision the complexity of the ML reviewed so far for ES was standard. It Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. tree is a supervised model capable of acting differently on its is a good signal that in most of the cases this basic stage it is reference data depending on the way these reference data are just enough. For the rest, more sophisticated routines are structured. Interestingly, in this case, the reference data are arising and will appear as the methodology reaches broader directly the clusters representing the ELNES information. From audiences. For that, a fruitful approach would be mimicking the that, the authors managed to, on the one hand, unveil the progress done in EM imaging, where developing ML-based oxidation nature of oxides given their spectral description, and solutions might be more intuitive. In that sense, S. Kalinin on the other hand, predict the spectra of a known oxide-based et al. brought the idea of autoencoders from imaging to nanostructure. spectroscopy.82–85,88 In a paradigmatic case, they deployed 1D The idea behind clustering is not far from unmixing, as we convolutional autoencoders to unsupervisedly represent each identify common patterns from spectra. Indeed, we can combine spectral pixel into a set of only two latent variables. Afterwards, both to improve the identification of materials, either by applying they grouped these outputted variables by GMM to distil the PCA before clustering, or by applying it to segment each indivi- spectrally distinguishable regions of an heterogenous array of dual cluster.259 In addition, this approach can be combined with nanoparticles, in this case, composed of fluorine and tin-doped non-linear least squares fitting for a deeper ELNES analysis that indium oxide (Fig. 5c and d).262–264 Importantly, as happened on not only classifies the data, but unveils finer ELNES features such imaging, this research work and related265 might draw the future as valence, oxidation and coordination states. These options are of spectral analysis, and point it towards the implementation of Fig. 5 (a–d) Representations of S. Kalinin et al.’s work,262 comparing two learning paradigms to unveil physically-distinguishable Electron Energy-Loss Spectroscopy (EELS) components. The authors suggested: (a) a supervised approach based on a multilayer perceptron capable of (b) extrapolating the partially manually labelled pixels of (b.a and b) two, (b.e and f) four and (b.i and j) six physically-different classes. On the other hand, (c) the authors proposed an unsupervised approach based on an autoencoder. (d) The autoencoder reduced the EELS dataset into two latent variables, which after being Gaussian clustered in (d.a) two, (d.b) four and (d.c) six classes, produced comparable maps to the supervised approach (b and d, reproduced with permission of S. Kalinin et al., Adv. Opt. Mater., 2021, 9, 2001808 Copyright 2021 Wiley-VCH GmbH). (e) L. Roest et al.’s work,296 represents the training of a physically-aware multilayer perceptron capable of (f) modelling a zero-loss peak and its uncertainty under different exposure times and beam energies. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1439 View Article Online Review Nanoscale Horizons more complex autoencoder-based routines, soft supervision, and Anyhow, these approximations can still be too computation- the use of GAN architectures. Furthermore, this important ally demanding for generating training data for 3D and 4D CNN. last example from S. Kalinin et al. highlighted how different Promisingly, ML has already offered alternatives to EM simula- unsupervised (i.e., the commented autoencoder) and semi- tion, such as the potential propagation through slices via the supervised (discussed below) approaches can lead to the same forward propagation of a trained neural network.120 In this case, spectral analysis. For that, we discuss this revealing complemen- each slice in the multislice approach can be modelled with a layer tary approach further in the next section about supervised ES. of the neural network, as if the propagated weights were the 2.2.2 Supervised machine learning and generation of train- actual propagated potential. This ML-based acceleration, ing sets. The previous section hopefully clarified the reasons together with the most efficient alternatives suggested could be This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. why unsupervised ML would hitherto win in popularity versus enough to generate enough data for 1D analyses. Unfortunately, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. the supervised in ES data analysis. The main limitation of spatial correlations in SI would be majorly lost in this way, supervised ML for ES lies in the arduous generation of training although the community already found ways to circumvent data. In consequence, the increase in accessibility of supervised that.251 Indeed, there is already research using 1D convolutions solutions for ES must go hand in hand with a more straight- for dealing with SI, but 3D convolutions have solely tackled forward and resource-friendly generation of representative ptychography so far.138,262,290,291 Importantly, the unsupervised synthetic labelled data. When we refer to labelled data in ES, example from S. Kalinin et al. would perfectly fill the gap of the we mean having either full spectra or individual energy simulated 1D data to emulate the spatial correlation of SI.251 channels linked to one or more quantities or properties. This Therefore, this would dramatically reduce the costs of data could be, for instance, having a set of spectra and each of them engineering and model training while mirroring the expected having a tag indicating which elements they contain. Of course, advantages of 3D CNN (i.e. direct spatial correlations). Moreover, this task requires a manual labelling done by a trained micro- ML could still make the simulation scene evolve to reach scopist, or the simulation of spectra based on a known compo- simulation efficiencies that could unlock 3D hyperspectral data- sition. Employing talent to label data should never be an option sets. For example, taking advantage of new ML paradigms such in these cases, and existing solutions such as mechanical Turks as graph neural networks (i.e., supervised models for dealing with are neither elegant nor efficient for long-term perspectives. mathematical graphs). One idea is deconvolving the electron Current EELS databases foresaw this issue, but still do not probe with the electrostatic potential encoded in trained graph provide answer to the labelling handicap, whose case-specific neural networks, associating the atoms with the nodes and the nature complicates the whole supervised picture further.266,267 bonding with the edges of these graphs. This inspiration from Nevertheless, the literature is full of multiple levels of theory, the multislice neural network may not be immediate, that is why which can facilitate the generation of trustworthy simulated more practical approaches resembling the discussed generative spectra.197,202,268 In fact, the key for relying on simulated data is models are called to play a key role in the next years. Eventually, its ability to parallelly generate the labels (thus, paired data) the desired paired data would be rapidly generated by these considering the application of the eventual supervised model. For trained but tuneable generative models. A more complete picture that, balancing the accuracy and efficiency of these simulations is about simulation and generation of data is found in Section 3 a key step for planning this endeavour. The most time-effective about the practical ML deployment for EM. approximations are based on classical electrodynamics and the As reviewed, the ease of application and reliability of unsu- dielectric properties of the materials, whose accuracy can be pervised methods in ES has hitherto left supervised routines improved by adding quantum corrections.269–271 The same can to a second plane. Nevertheless, novel supervised ideas have be applied to Monte Carlo simulations.272–275 More accurate been introduced to face more challenging cases, or even to have quantum calculations can be semi-empirical by the use of their performance compared with equivalent unsupervised experimental scattering cross-sections, calculated by ab initio methods.235,258 At this point, it is very interesting to go back methods, or even estimated by ML.222,276–280 High-fidelity simu- to S. Kalinin et al.’s work on autoencoders for categorising low- lations require multiple scattering (more efficient), with the FEFF loss EELS.262 A few paragraphs above we present an autoenco- code,281 or ab initio computations (more accurate) of the sample der plus GMM approach capable of mapping spectrally distinct potential in the multislice approximation,194–196,198,209,210,282,283 or regions in an SI (Fig. 5c and d). In the same manuscript, the even by Bloch wave formulations in thin specimens.284–286 Inter- authors presented a multilayer perceptron obtaining compar- estingly, these time-consuming approaches can reduce their cost able results to the autoencoder. The idea behind this super- by modelling the traditional pixelwise simulation of the probe vised model was the partial labelling of some pixels (spectra) of displacement with scattering matrices.287 Although these solutions the SI and to base its training on the labelled pixels to infer the were conceived with EELS in mind, the generation of EDX non-labelled ones. This labelling was done by assigning a class simulated spectra is equivalent but tailoring the potential for this (element type and even chemical surrounding) to a fraction of interaction, modelling the bremsstrahlung background, and con- the pixels belonging to this class. The authors evaluated the templating the specific detector geometry.271,288 DTSA-II from NIST percentage of pixels that needed to be manually labelled for the is an open tool that can be used to simulate EDX spectra, which correct supervised classification of the remaining unlabelled also allows the analysis and processing of experimental spectra.289 pixels. Surprisingly, they found that the well-chosen labelling These tools are extended and detailed in Table 1. of only a 0.31% of representative pixels led to an output 1440 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review Table 1 Main available software solutions for simulating electron microscopy images and spectra. The availability of a partner or complementary programmatic software can facilitate the generation of (labelled) datasets suitable for machine learning Software solution Level of theory Programmatic interface Accessibility Ref. Electron abTEM Multislice, ab initio DFT bonding infor- Yes, Python-based Open 343–345 microscopy mation + PRISM approximation (imaging) Bloch wave simulations Bloch wave formulation No Open 330 and 331 Cerius2, Molecular Multislice, molecular dynamics No Paid 346 Simulations Inc. cudaTEM - clTEM Multislice, independent atomic potential Yes, command line Open 347 and 348 Dr. Probe Multislice, independent atomic potential Yes, command line Open 349 This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Electron direct methods Multislice, independent atomic potential No Open 350 (EDM) + kinematic scattering Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Multis Multislice, independent atomic potential No Open 330 Multivariate polynomial fit Multislice, independent atomic potential Yes, Python-based Open 337 Prismatic Multislice, independent atomic potential Yes, Python (PyPrismatic), Open 351–353 + PRISM approximation C++ QSTEM Multislice, independent atomic potential Yes, Python (PyQSTEM) Open 354 and 355 scikit-ued Multislice, independent atomic potential Yes, Python-based Open 356 + Kinematic scattering STEM_CELL Multislice, independent atomic potential No Open 357 and 358 + linear image approximation Tempas Multislice, independent atomic potential Yes, digital micrograph-like Paid 359 + kinematic scattering + Bloch wave scripting formulation Electron Bloch wave simulations Bloch wave formulation No Open 284–286 spectroscopy DTSA-II (NIST) EDX (characteristic and Bremsstrahlung) Yes, command line Open 289 - Monte Carlo + j(rz) + XPP Electrodynamics Classical relativistic (faster) - quantum Yes, command line Upon 269–271 (more accurate) request FEFF Ab initio, projector-augmented wave Yes, command line Paid 281 LEEPS (and variations) Monte Carlo + classical relativistic cross Yes, command line Upon 272–275 sections (faster) - quantum (more request accurate) PENELOPE Monte Carlo + classical relativistic cross Yes, Python (pyPENELOPE) Open 360 and 361 sections (faster) - quantum (more accurate) Prismatic PRISM approximation Yes, Python (PyPrismatic), Open 351–353 C++ Semi-empirical Experimental cross-sections or dielectric Yes, command line Upon 222 and 276–280 calculations functions request WIEN2k Ab initio, projector-augmented wave Yes, command line Paid 194–196, 198, 209, 210, 282 and 283 comparable to the unsupervised autoencoder, validating the logic, we want to directly make the ML aware of the science it outstanding generalisation capabilities of DL (Fig. 5a and b). would eventually represent. This is known as physics-aware ML Remarkably, this approach facilitated the labour of training a and is currently scarcer than standalone supervised ML. How- supervised model for ES. Any microscopist wondering whether ever, it is called to be the next breakthrough in scientific ML. to invest resources on engineering a supervised model might Interestingly, there are already some examples that are worth to just label a key fraction of the data to see how well an eventual comment and guide the reader on what can be achieved by model may generalise. It is excellent, then, to gradually put following this path. L. Roest et al. tried to answer the repeatedly resources on model design and avoid massive resource invest- assessed universal modelling of the Zero-Loss Peak (ZLP) with a ments unless it is mandatory. neural network trained on Monte Carlo simulations.294,295 They This juicy research piece is not the only supervised approxi- generated a physics-aware model by adapting the regression mation in ES. FCNN and support vector machines were also model describing the ZLP (and its uncertainty) with the physical applied for automatic mapping and elemental identification, conditions of the acquisition, namely the exposure time and the for ELNES characterisation, for instrument-independent spectral beam energy (Fig. 5e and f).296 This means that the input data calibration, etc.290,292,293 Nonetheless, these applications essen- the model received was the ZLP curve with the exposure time tially resembled the reviewed unsupervised methods and their and beam energy, making these variables constitute individual achievements. To be fair, the scenarios were more challenging, dimensions within the parametric space of the model. Typically, but the main workflow remained essentially unchanged. Indeed, the science awareness arises from directly inputting the physical the extra energy dimension that spectroscopy introduced, conditions, although similar effects can be achieved with unlocks the in most cases missed opportunity to actively phy- physically aware labels. For instance, one could use different sically constrain the ML used. As happened with NMF versus unit cells, jointly simulate spectra and some physical properties, PCA, where we force the maths to behave under some physical and train a model where the input is the spectra and the label is This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1441 View Article Online Review Nanoscale Horizons the simulated physical property. This is what S. Kiyohara et al. variance, we should reduce the energy sampling).297,298 For did with a MultiLayer Perceptron (MLP) on core-loss spectra instance, full STEM-EELS SI were fully reconstructed from only (i.e., each neuron represented each energy channel) for the acquiring the 18% of the total spectral pixels.159 Importantly, extraction of up to six properties (Fig. 2 MLP).102 The training the chosen basis of sparsity may strongly depend on the image was performed on 1171 spectra, simulated by DFT from 188 features, but choosing it wisely may definitely boost the fidelity, different silicon oxides, and the labels were also simulated by dose reduction, noise resistance and efficiency of CS-based DFT. Therefore, they eventually engineered a ML-based DFT EELS sparse imaging.299 Based on this, ML could be applied simulator from experimental EELS spectra. From the input as a complement to automatically choose the appropriate basis spectra, they were capable of extracting the physical properties given a training on a fast preliminary imaging step. In any case, This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. they chose to generate the labels: properties such as average bond the overall same ML strategies presented for CS-inspired ima- Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. length and angle, Voronoi volume, bond overlap population, ging could be extended to ES with minimal adaptation given Mulliken charge and excitation energy. the extra energy axis, also taking advantage of mixed imaging- Until now, the physics-aware approaches are based on spectroscopy (High-Angle Annular Dark-Field (HAADF) + EDX) simulations as the best way to directly link curves or images CS-enabled multimodal approaches.162,163,300 with a specific physical property. However, its implementation On the other hand, EDX, EELS and even Energy-Filtered on experimental datasets would start with the reproducible and TEM (EFTEM) tomography skyrocketed its progress in the last systematic acquisition of the metadata accompanying the two decades.301–307 Both qualitatively and quantitatively.308–311 experiment. In microscopy, the acquisition conditions and Even though its major advances in reconstruction algorithmics microscope parameters would be an already vast starting point. were not CS/ML-based, classic data processing established a The works of L. Roest et al. and S. Kiyohara et al. shined light on starting point to which apply these new methodologies for the promising future of physics-aware ML in EM/ES.102,296 refinement (majorly for EDX and EELS). As happened in imaging, Indeed, these are two of the few physics-guided ML models the use of CS-based reconstructions overcame the effects of we can currently find among the EM literature. The importance traditional reconstructions in managing the missing wedge. Even of adding scientific constraints to ML not only does it turn in more given the additional energy channel and the requirement of more general models, but can remarkably reduce the required DualEELS to avoid multiple scattering artefacts in ST.312 training volume (similarly to rVAEs).88 For instance, S. Kiyohara Interestingly, tomography may also benefit from CS-enabled et al.’s work could be extended by training high-complexity multimodal approximations, namely STEM-HAADF and EDX DL models on full databases of spectra if a systematic spectra- with total generalized variation. The combination of comple- property correlation was done. In addition, related to L. Roest mentary signals and their simultaneous regularisation (i.e. et al., a plural scattering remover could be trained with a signal balancing) might be helpful for research cases where dataset of dual EELS SI and then applied to correct only the the separated techniques cannot reach the desired level of core-loss spectral region of interest. Similarly, the energy reso- detail. For example, R. Huber et al. explored this method for lution post-acquisition could be effectively improved by train- the 3D reconstruction of nanostructures with both sharp edges ing a neural network on sets of the same spectra but acquired and gradual compositional changes.313 Despite the benefits of with different resolutions. Moreover, L. Roest et al.’s work multimodal approaches, it can often be a time-consuming or highlighted the importance of the cross-fertilization with other dose-demanding methodology, even with the support of CS scientific and technical fields, as the ML methodology the algorithms. Then, it may be even more interesting the use of authors followed was inspired by equivalent solutions in parti- unmixing methods as a pre-reconstruction step, following the cle physics. This cross-fertilization is further discussed in the leading work of L. Yedra et al.314 PCA, ICA and VCA can be used to final section of the review. prepare the data for the posterior CS reconstruction. This initial 2.2.3 Compressed sensing and spectroscopic tomography. unmixing step allows to reduce the reconstruction computational As happened with imaging, CS is strongly linked to Spectral cost and its complexity by relaxing the prior assumptions, both Tomography (ST). ML has also tried to mimic the CS function- for EDX and EELS, and under challenging sample configurations alities in both reconstructing sparse SI and ST datasets. In fact, (Fig. 2 VCA).101,315,316 Nevertheless, the ST processing is not given the higher complexity of adding the energy axis into an limited to the pre-reconstruction and reconstruction steps. In already complex tomography reconstruction, CS and advanced fact, meaningful post-processing and cleaning routines might be data processing routines are more justified. However, let us applied after a reconstruction that has already met the benefits of first evaluate the use of CS for the relaxation of the acquisition the reviewed strategies. In the same way as DL was a successful conditions of traditional SI. Indeed, CS must be regarded as a tool used to denoise micrographs, it was also used to denoise complementary data processing method to the ML reviewed in EDX tomography reconstructions, following the original develop- the previous two subsections, rather than a substitute. The ment in X-ray computed tomography.317 To do so, A. Skorikov strategies for sparsity are based on reducing either the spatial et al. trained a U-Net architecture on a set of 850 noisy EDX maps or the energy sampling. The choice lies in identifying which is obtained on nanoparticles.318 The DL denoising of the 2D maps the most important dimension for the considered experiment outperformed traditional denoising methods and proved to be (if there are abundant spectral features, we should just limit the cumulative for 3D reconstructions of both simulated and experi- spatial sampling, whereas if the beam scans through a lot of mental tilt series of metallic nanoparticles. As a result, EDX 1442 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review tomography was rewarded with a much gentler trade-off between and its orthogonality (correlations between variables), the task, dose management and SNR. and even the computational resources available. In the previous Ultimately, this section reviewed the manifold ML techni- sections we have reviewed the major pros and cons of each ques already used in ES. Nonetheless, its preliminary stage is approach, and the scenarios in which they better shine.319 For stressed if compared with the variety presented in the imaging example, in case we opt to follow a supervised approach to section. Therefore, the opportunities for applying the already challenge the data preparation step, as previously emphasised, developed methodologies to EDX or EELS are huge, but not we should carefully plan the labelling step and not just rely on only to these two techniques, but to EFTEM or even cathodo- human workforce for doing this repetitive and tedious task. luminescence (CL). We were astonished to find absolutely no Therefore, as the first approximation to follow, we should self- This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. bibliographical match with ML for EFTEM or CL. In fact, the engineer the dataset by using model-based methods or simula- Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. experimental similarities between EFTEM and, especially EELS, tions to allow the automatic generation of the labels. In addi- result in a theoretically straightforward implementation of its tion, special attention should be initially paid to the data methods for energy-filtered imaging (despite its lately decreas- format, and the current trends towards open and universal ing popularity), or differently, the use of CS for adding sparsity ML-intended data formats, such as the Universal Spectroscopy in the temporal axis of analytical tomography. Nonetheless, one and Imaging Data (USID), based on hierarchical data formats of the most important ideas this section showed us is the like HDF5 or NumPy’s native file type.320–323 These formats are potential of the inter-collaboration between different fields, as designed to intuitively store paired data and any metadata that shown in the multidisciplinary work of L. Roest et al., in this could be used either to physically constrain our models or to be case towards physics-aware modelling.296 Indeed, we cannot used as labels. stress enough the importance of investing towards deeper Data can be generated by means of very complex simulations physically-aware models aimed to simplify the nowadays too but also through extremely simple ones. For instance, it is often unaffordable human and computational costs of devel- possible to simulate an atom image by calculating its electric oping complex ML models. potential by DFT and then deconvolving an electron probe to generate a micrograph. However, it is also possible to generate a circular feature (like a 2D Gaussian curve) and add some 3. Tools to deploy machine learning in Poisson noise to have a simpler and cheaper (in computational electron microscopy terms) representation of the same atom.97 Even further, although a bit circular, a properly trained generative model The resources for applying ML to research are nowadays should be capable of outputting tones of synthetic atoms with countless. For this reason, the present section is intended to the accuracy used for its training.324–326 Of course, the utility of be pragmatic. In the following we will discuss on the different the latter would lie in having the generative model ready, resources concerning ML the reader interested in EM and ES may thanks to someone else’s previous efforts. In any case, the have access to. This includes software packages and code snip- physical accuracy between these three approaches is incompar- pets which relax not only the required knowledge in coding, but able, but the rudimentary model might be enough if it can in all the steps of the workflow. These steps consist of the represent the variability of the data of interest. At the end, ML (training) data preparation, the renting of high-performance can be seen as just an esoteric regression representing the computing time to run the models, the availability of already variability of a dataset. If higher-complexity simulations are trained models or either the design of optimised architectures for required, the resources available are manifold. Importantly, the a given problem, and even the final benchmarking. Therefore, label generation must always accompany the simulation, either those EM users willing to start applying ML methodologies to with the same software or with a complementary custom script. their research but also those more advanced willing to optimise In fact, this parallel automatic labelling is key to minimise the their case-specific bottlenecks or barriers, may find useful bias of a human labelling. For example, the pixelwise labelling resources in the following paragraphs. In fact, we will show that of the compositional distribution of a qualitative EELS map will it is possible to develop a ML-based application for EM data find pixelwise discrepancies if two different microscopists label analysis without an advanced knowledge on ML and not even them. Specially at the interface, the contrast and intensity of a with a big funding/infrastructure investment. pixel might be interpreted differently by two different expert microscopists. However, if a model was built and this interface 3.1 Preparation of the training set was perfectly defined by a mathematical curve or function, the On one hand, we should closely evaluate the problem to solve pixelwise labelling would be completely objective. Another and classify it. The possible categories can be data analysis or example is the labelling of the central position of the atom data acquisition automation, knowledge extraction, modelling and we have previously simulated to create a model for atom experiment optimisation, among other possibilities. Afterwards we finding. If it is an experimental STEM atom, the scan distor- should decide whether to go for an unsupervised, supervised or a tions may generate an odd intensity distribution. On the other semi-supervised approach. The decision will be mainly dependent hand, for a simulated atom, the central position is just the pixel on the ease to generate labelled data (e.g., data accompanied with closer to the spatial coordinate where we placed the atom in our the ground truth of the expected classification), the data structure model. Fortunately, this is independent on how we paint the This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1443 View Article Online Review Nanoscale Horizons intensities within this atom. In other words, we are free to an even better starting transferable knowledge given the proximity generate as much variability as we want. of the variance of the data. The most interesting EM databases to The following simulation resources are precise but time- initially use or work with are depicted in Table 2, Databases for consuming. As a result, in these cases, the optimisation of the electron microscopy images and spectra, where both unlabelled simulated volume is a priority. For instance, a handful of unit and labelled datasets are available free of charge. Complementa- cells might be simulated and artificially spatially translated, rily, these micrograph databases could also be linked with crystal rather than simulating a large area of the same repeated unit structures (e.g., encoded .cif files as the labels) for a rapid and cell. For imaging, the multislice approximation is the most automated crystallographic phase identification. This idea is adopted and adapted solution: from the first formulation by represented in Fig. 6, where the development of this hypothetical This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. J. Cowley and A. Moodie,327 and E. Kirkland’s first adaptation to application is schematically represented, requiring no specific Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. STEM,268,328,329 there are many solutions currently available, as coding ability nor extraordinary computing resources. The avail- summarised in Table 1, Electron microscopy (imaging). Most of able crystallography databases to complete similar tasks are huge these multislice solutions and Bloch wave codes perform and are updated regularly. A summary of the available crystallo- similarly in terms of computing time and accuracy, thus the graphy databases can also be found in Table 2. These databases final choice mainly depend on our own interface implementa- thoroughly store the structural crystallographic information of tion preferences.330,331 Nevertheless, the Python-based codes manifold crystal phases. Interestingly, its format is suitable for might be more suitable for customising the parallel labelling encoding physically meaningful information in our models, such step. For this purpose, existing Python packages can thoroughly as cell parameters, space groups, chemical bonding and valence facilitate the communication with atomistic models and crystal- states, among others. lographic analysis.332–336 Interestingly, STEM_CELL and Pris- If we cannot either label via modelling or thoroughly rely on matic offer simplified versions of the multislice algorithm with existing databases, we have no other option than labelling the higher speed but reducing the precision (linear imaging, and data manually. After carefully planning the necessary training combined Bloch wave-multislice, respectively). We can even find volume, one solution is the trivial manual labelling done by the a multivariate polynomial fit trained on multislice simulations to researcher. However, it is not recommended, as the researchers’ provide a comparable accuracy in an up to six orders of magni- time is such a valuable resource that should not be spent in this tude faster algorithm than a CPU-parallelized multislice.337 In repetitive task with no intrinsic nor transferable value. Alterna- addition, fast neural network-based computations mimicking tively, one may opt for a company specialised in preparing ab initio calculations could be implemented in a more sophisti- datasets for ML as another valid purchasable resource, as can cated approach for computing the specimen potential with existing be reagents, human talent or computing time. These companies solutions.338–342 The same applies for the ES simulation (Table 1, excel in optimally augmenting and labelling the data of interest. Electron spectroscopy). The solutions and their particularities are Nonetheless, scientific labelling may require a trained eye. As a presented in the previous section devoted to ML for ES, for which result, the labelling instructions should be clarified to reduce the labelling process would also demand a parallel script. biases as much as possible, with a regular supervision of the In case the application of interest cannot benefit from results. Companies offer both tools for an optimised and intuitive model-based labelling, we could instead take advantage of the manual labelling, and the labelling service itself: Amazon’s already existing images or crystal databases. The goal will be mechanical Turk,387 appen,388 TrainingSet.AI,389 Superb AI,390 finding the datasets in which the data structure resembles as iMerit,391 Clickworker,392 ImageLabeller in MATLAB,393 Sama,394 much as possible our targeted data. Regarding images, this or LabelMe Annotation tool,395 among others. means having (or forcing) the same number of information channels (typically single, grayscale), number of pixels repre- 3.2 Model design senting edges (learnt convolutional filters may not detect edges As a matter of fact, the data generation step should always be or features represented by a different number of pixels), similar accompanied with a decision on the model architecture. The histograms and contrast/brightness descriptions, total number description and purposes of the available unsupervised algo- of pixels, the image calibration or resolution (if accounting for a rithms, models, supervised networks and architectures is out of physics-aware model), or the materials and morphologies illu- the scope of this review article. There is plenty of information strated. In these cases, the starting point can be the transfer about it in the literature, and we summon the attentive reader learning of models trained on habitual and general computer to the review by J. Ede, which described in detail the most vision databases (Table 2, General digital images). The idea is to widely used architecture types in EM.92 Generally speaking, exploit transfer learning with the datasets that resemble our data CNN will be the preferred option for the EM data that can easily the most and provide a better starting point in the model training be structured as a vector, matrix or tensor (line spectra, images, than just initialising the weights randomly or with the Xavier SI. . .). Anyhow, assuming the architecture choice is appropri- method.362 In fact, despite the disparity between mainstream ate, we can search for its code implementation, and preferably digital imaging and electron micrographs, this procedure reported open source. One of the key intrinsic concerns of the Python advantageous results when training an ImageNet-initialised VGG- language and specifically of the AI community, is the will 16 architecture used for the classification of HRTEM images of towards open science. Gladly, this is translated into huge carbon nanomaterials.95 However, EM databases may constitute sources of open code for implementing any imaginable ML 1444 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review Table 2 Main available databases for training supervised models on general digital images, exploitable in transfer learning, and specific databases containing electron microscopy images and spectra, and crystallographic information, for applications such as those depicted in Fig. 6 Data amount Database (images) Content Data type Accessibility Ref. General digital ImageNet 414.000.000 Multiple categories of digital images (animals, Labelled Open 363 images vehicles, objects. . .) and 364 MNIST 70.000 Handwritten digits Labelled Open 365 and 366 This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. LabelMe 44.000 City landscapes showing people, vehicles, Multi-labelled Open 367 objects. . . and Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 368 ESP 4350.000 All kinds of digital images and increasing Multi-labelled Upon request 369 Lotus Hill 4636.000 Multiple categories of digital images (animals, Multi-labelled Upon request 369 vehicles, objects. . .) Caltech101 49.000 Multiple categories of digital images (animals, Labelled Open 370 vehicles, objects. . .) Caltech 256 430.000 Multiple categories of digital images (animals, Labelled Open 371 vehicles, objects. . .) MSRC kinect gesture 4719.000 Human movements and gestures Labelled Open 371 dataset PASCAL 419.000 Multiple categories of digital images (animals, Multi-labelled Open 372 vehicles, objects, gestures. . .) Electron TEM ImageNet 14.000 – Simulated atomically-resolved STEM Multi-labelled Open 373 microscopy – Multiple materials and orientations images and EPFL EM dataset 1.065 Cell biology, experimental Labelled Open 374 spectra Freiburg University 637 Cell biology, experimental Labelled Open 375 ImageJ Multiple Cell biology, experimental Labelled and Open 376 sources 430 TB unlabelled in images CNR – IOM SEM 22.000 – Experimental images Labelled Open 377 dataset – Low-dimensionality nanostructures – MEMS Zelinsky Institute of 1.000 Simulated images of nanoparticles Labelled Open 378 Organic Chemistry SEM dataset DeCost – Holm SEM 2.048 Simulated images of nanoparticles Labelled Open 379 dataset Warwick EM dataset 135.375 – 19769 experimental STEM Unlabelled Open 380 – 17 266 experimental TEM – 98 340 simulated exit wavefunctions EELS DB 4200 spectra Experimental EELS + XRD Unlabelled, but Open 266 with metadata EELS.info Sample spectra Experimental EELS Unlabelled, but Open 381 for every element with metadata Crystallography Crystallography open 4490.000 crystal Complete crystallography of inorganic, organic, Unlabelled, but Open 382 databases database structures minerals and metal–organic compounds with metadata and – Both experimental and theoretical studies 383 Inorganic crystal 60.000 crystal Complete crystallography of inorganic Unlabelled, but Paid 384 structure database, structures structures with metadata and ICSD – Both experimental and theoretical studies 385 NIST crystal database – 220.000 crystal Complete crystallography of inorganic, organic, Unlabelled, but Paid 386 structures minerals and metal–organic compounds with metadata – 81.000 electron – Both experimental and theoretical studies diffractograms model. Fortunately, the EM community has followed this weights initialised on EM/materials data (for transfer learning). democratisation trend and has already provided open tailored On the other hand, the Materials Simulation Toolkit for ML solutions. Interestingly, these solutions come along with tutor- (MAST-ML) provides a broader platform to boost materials- ials and intuitive guides both for newcomers searching for related research with ML.410,411 DeepImageJ is its equivalent for basic implementations and for experts wanting to carefully fine biosciences, although the network architectures and weights tune well-established models. For example, Pycroscopy (and its might be exploited universally.412 In a similar way, and despite sub packages like AICrystallographer,404 AtomAI,405,406 GPax,407 it was originally intended for particle tracking in optical micro- pyTEMlib408) offers a centralised platform of ML/DL pretrained scopy, DeepTrack 2.0 offers a pythonic solution to easily call models and raw architectures, among other analytical ML/DL models.413,414 This is why this might be the preferred functionalities.320,405,409 It provides a straightforward way to choice for optical microscopy or biosciences applications. In call and train models like U-Net or VAEs, as well as having their this case, it also displays a user-friendly graphical interface, This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1445 View Article Online Review Nanoscale Horizons This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Fig. 6 Example of the hypothetical development of a Deep Convolutional Neural Network (DCNN) model to correlate Annular Dark-Field Scanning Transmission Electron Microscopy (ADF-STEM) images with the crystal phase and zone axis displayed. The Machine Learning (ML) workflow is thoroughly based on open software and resources, and requires almost zero ML-specific knowledge and funding/infrastructure. The ADF-STEM micrographs are taken from the TEM ImageNet open database,373 with 14.364 images of multiple crystal structures. The images metadata correlates them with a class defined by the crystal phase and the zone axis (labels). The complete crystallographic information of the crystal phases is extracted from the free Crystallography Open Database.382,383 Next, a VGG396-like architecture is used from an open-source397,398 repository like GitHub,399 almost eliminating the coding. The training can be done either in a PC or in a cloud service such as Microsoft Azure,400 Google Cloud401/Colab,402 IBM Cloud,403 among others. Eventually, the model can be tested and used, allowing the user to identify the crystal phases and orientations of different micrographs. with which newcomers will thank the interpretability of the image analysis, VGG,396–398 AlexNet423,424 or YOLO9000,425 deployable tools. Additionally, CDeep3M and ZeroCostDL4Mic among many others.426–431 The use of one or another network work similarly, while offering the added possibility to run the will depend on the final application: we would use U-Net if we code directly on cloud computers via notebooks.415,416 The want a pixelwise classification of our image, but VGG-like comprehensible tutorial format of these notebooks makes architectures for classifying the whole image into a category. the implementation of the provided methods even more Alternatively, in case the research project demands a ML model direct.417 In addition, for those looking for CS tools, they can not yet programmed nor shared, it could be coded from scratch. also be easily implemented with open codes like CSET418 and Obviously, this approach demands a higher understanding of RTSSTEM,160 tailored for EM and its tomography. coding and of the ML scheme. Nevertheless, the Python The previous software packages are the formal answer to the packages indicated in Table 3 offer intuitive programmatic community’s need for straightforward and centralised ML/DL interfaces to easily implement from the simplest unsupervised tools. However, if a given publication is inspiring us a specific model to the most complex neural network. The most interest- architecture, one possibility is to directly dig into the article ing or exploitable features of each are indicated in Table 3, material. In this way, most of the papers developing a certain although in a general basis, the beginner user would find a ML model contribute further with the code (and even datasets) better ally in high-level APIs such as Keras, while the advanced towards reproducibility and for it to be applied in complementary user looking for further fine-tuning will benefit from low-level research. Seldom do the authors protect their codes with private APIs like TensorFlow. intellectual property. The proof is that most of the articles reviewed in the imaging and spectroscopy sections provide a 3.3 Training and testing GitHub or Zenodo link to freely download the represented Once the (training, validation and test) data and the model are code.399,419 Indeed, these repositories, among others like ready, the question is where to train the model. Generally, in OSDN,420 Bitbucket421 or GitLab,422 have become powerful search DL the training volume is of about the 90–95% of the total data, engines to intermediary-free get open software and code snip- while the test and validation set equally share the remaining pets. Representatively, searching for ‘‘U-Net’’ in GitHub results in 5–10%. This assumes that our data volume is large (i.e. 10 000 27 732 resources (Search date: July 2022) that could be used images or more). Nonetheless, these numbers are nothing but a instead of coding U-Net from scratch. Similar results are obtained guide towards best practises that can be modified depending by searching for the most well-known network architectures in on the total data volume. For instance, if our data volume is 1446 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review Table 3 Packages devoted to machine learning application, both in an unsupervised and a supervised way, and their main exploitation feature. The main environment for applying them is Python, although some of them also present APIs in other languages Package Used for Programming interface/language/APIs Key feature Accessibility Ref. Caffe/Caffe2 Supervised Python, C++ DL performance and cross-platform Open 432 and deployment - features migrated 433 to PyTorch Ctypes Un- and supervised C/C++ - Python Use C/C++ libraries in Python Open 434 Cython Un- and supervised C/C++ - Python Use C/C++ libraries in Python Open 435 Dask Un- and supervised Python Scalability, parallelisation Open 436 DL Toolkits Supervised Wolfram Mathematica Multifunction and interpretability Paid 437 This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Gensim Un- and supervised Python For natural language processing Open 438 GNU Octave Un- and supervised Octave Flexibility Open 439 Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Hadoop Un- and supervised Apache Scalability, big data and parallel Open 440 computing Keras Supervised Python Easy implementation of DL Open 441 Mahout Un- and supervised Apache Scalability Open 442 MATLAB Un- and supervised MathWorks Flexibility Paid 443 Matplotlib Un- and supervised Python Visualization and plotting Open 444 MXNet Supervised Apache (integration with Python, Scalability and interpretability Open 445 Java, C++, R, Scala, Julia, Clojure, Perl) Numba Un- and supervised Python Code parallelisation Open 446 Numpy Un- and supervised Python Flexibility Open 321 OpenCV Un- and supervised Windows, Android, Linux, Computer vision and image Open 447 and MacOS, FreeBSD, OpenBSD – processing 448 C/C++, Python, Java, Android Pandas Un- and supervised Python Tabular data processing Open 449 Pattern Un- and supervised Python Web mining Open 450 Pillow, Python Un- and supervised Python General image processing Open 451 Imaging Library PyTorch Supervised Python, C++, Java DL flexibility and rapid prototyping Open 452 scikit-image Unsupervised Python ML image processing Open 453 scikit-learn Unsupervised Python Flexible ML built-in tools for data analysis Open 454 SciPy Unsupervised Python Flexible general built-in tools for data science Open 455 Shogun Unsupervised Python, JavaScript, C/C++, R, Multi-language support, ML data science Open 456 Ruby, Octave, Java, Scala MLlib Un- and supervised Apache Spark (integration with Fast general ML Open 457 Python, Java, Scala and R) TensorFlow Un- and supervised Python, Java, C++, Swift Highest flexibility, low-level DL Open 458 Theano Un- and supervised Python DL flexibility and rapid prototyping Open 459 scarce because it is difficult to generate it, we will decrease the insight and suggestions to optimise the training and hyper- training percentage in favour of test and validation. At the end, parameter tuning of our models (Weights & Biases,466 and these percentages could even be considered as hyperpara- PerceptiLabs467). In fact, these solutions are not only intended meters to tune and will eventually be defined to maximise the for ML newcomers, as anyone looking for the effects entailed by overall performance. Practically, if the model and training the hyperparameter tuning of their models will benefit from volume is reasonable (i.e., around 106–107 trainable parameters these visualisation tools. and from 10 000 to 15 000 512  512 images), the training time After training the model, the ML workflow is almost com- should mostly be a few hours in a regular PC with a GPU. Larger plete. The only step remaining would be the testing or bench- models, and even the generation of larger data volumes would marking of the resulting trained model. At this point is when require a larger computing infrastructure. For that, the ideal the dataset division into test and validation sets play a role. The case scenario would be a partner institution placing computing test set will be used to iterate into the hyperparameter tuning systems at the researcher’s disposal. In case this requirement is till the test metric is maximised. Only then, the validation set is not met, then online and cloud computing services would evaluated to output a final metric on the performance of the instead be suggested. The following are the most established model on this dataset. This split into test and validation is not services for renting high-performance computing time, which mandatory but recommended. With this, we reduce the bias of most of them provide specific solutions for ML projects: the final metric by avoiding the tuning of the model with Alibaba Cloud,460 Amazon Web Services,461 Microsoft Azure,400 the information that must serve the final metric. Apart from Deepnote,462 Genesis Cloud,463 Google Cloud401 and Colab,402 testing with our own data, it is recommended to evaluate the IBM Cloud,403 OVHcloud,464 and Paperspace CORE.465 Whilst performance of the model with a standardised benchmarking accessing powerful computing solutions is key to speed up the dataset. Each community can formalise a dataset intended for a training-tuning cycle, the huge hyperspace susceptible to be given purpose and make it the benchmark to follow when modified can be overwhelming to face. To cope with that, we creating similar applications. For example, EM for cell biology can nowadays find intuitive and powerful interfaces providing received formal segmentation benchmark standards with a This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1447 View Article Online Review Nanoscale Horizons platform dubbed EM-stellar.326 Nevertheless, the feature com- 4. Inspirations from other fields for plexity of the materials science EM data, along with the early future developments stage in which ML/DL is currently in the field, turns into the absence of formal benchmarking tools. As reviewed, the The previous sections described manifold EM applications and applications achievable by ML methods are manifold, and each how the numerous available ML methodologies pushed would require a specific benchmark routine. As a result, and towards thrilling advances in the field. Nevertheless, the until task-specific (or even more general task-agnostic) bench- reviewed potential of ML is just the tip of the iceberg of mark tools are standardised in EM, the best practice can be the everything ML has to offer. The best way to realise this fact is strict evaluation of a representative test/validation set, by by the examination of the significance of ML in other fields This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. the most suitable metric given the aim of the model. Even- where it is not a complementary resource but a keystone. As Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. tually, the model should be tested with the experimental data promised, this section is intended to evaluate the ML-based for which it was originally aimed. Complementarily, existing case studies in these other fields and the way they could be ML or general data processing tools can be useful to crosscheck beneficial to EM in order to solver materials science questions the performance of our trained model. This can be both a in the near future. Importantly, it is easy to repeatedly consider substitute for absent benchmarks and a validation tool for the following results as robust starting points for tailoring successfully benchmarked models. For instance, if our model specific experiments to the materials/EM community. Or the is devoted to crystallographic analysis or phase identification, other way around, the previously reviewed developments could we can use the powerful CrysTBox for a quick validation.468 If our be extrapolated to these fields to strengthen them even more. In model is for 4D-STEM data processing, we may additionally use fact, both the electron microscopists and materials science Py4DSTEM469 or Pyxem470–472 for corroboration. In case our devel- researchers, but also domain experts from the specified fields oped model tries to automate a particular step of the TEM data may benefit from the ideas discussed below. The ideas and acquisition, maybe combining it with the ANIMATED-TEM toolbox findings discussed throughout the section have been schema- allows us to reach the next level of overall automation.473 On the tised as a guide in Fig. 8. other hand, if our application is intended for image or spectra analysis, it may benefit from double-checking the results manually 4.1 Microscopies and imaging in one of the many data processing software available, among 4.1.1 Cryo-EM. The Nobel prize-worth technique for the which we highlight the broad capabilities and open philosophy of structural reconstruction of proteins, viruses and other macro- gempa,474 LiberTEM475 and HyperSpy.476,477 More specifically, if molecules demands working with huge amounts of data. It is the goal of the trained model is general STEM strain analysis, we fundamentally based on the electron (diffraction) tomography can crosscheck it with Fourier-space phase analysis and with or single particle reconstruction from images and/or diffracto- real-space atomic position determination, or even with Moiré grams acquired from hundreds of crystals containing the fringe imaging or the simulation of lattice elasticity molecule of interest.491,492 An important step of the workflow relaxation.7,8,478 For instance, among the real-space analysis is the search for the single-crystalline particles containing the we find traditional and well-stablished methods such as target biomolecule, from where the images are going to be Atomap,99,479 ranger,1,10 qHAADF,480,481 iMtools,482 StatSTEM2, acquired. The process of selecting the particles of interest can or oxygen octahedra picker,483 but also ML/DL-based methods be laborious and bottleneck the whole workflow. As a result, (e.g., atomic column finding), which can be even more indica- tons of efforts have been allocated to its automation. In fact, tive of the state of the art for comparison purposes.6,15,59,97 this automation constituted the angular stone in extending the The tools and resources reviewed in this section are meant reach of cryo-microscopy, consequently settling it down. This is to make the electron microscopists’ life easier. In fact, ML because the obtention of multiple equivalent projections from routines can be fully developed thanks to them, requiring little different particles allows to match them together and to resources to make brand new models come true, as depicted in improve the resolution. Multiple ML methods such as support Fig. 6. Nevertheless, these are the tools that are currently being vector machines, logistic regressions, K-means clustering, clas- used or have been developed or adapted mostly by the EM sification trees, or deep CNN attempted to automate this community. Therefore, there is plenty of room at the bottom for process.183,192,493–496 Interestingly, deep neural networks were introducing new models and architectures yet unexplored in successfully used to automatically guide the microscope in EM. For instance, the need for rapid and trustworthy dynamical procedurally finding the carbon holes, single particles inside, simulations has catapulted the popularity of graph neural and eventually acquiring the different image projections for networks.484–488 Interestingly, the structure of the EM data their final projection matching.178,189 It is great that this was could perfectly fit these flexible networks, allowing to point achieved by directly applying the wide-spread YOLO network towards dimensionality-independent models. Even (deep) RL architecture, reinforcing the message we wanted to spread out (already reviewed for a couple of applications107,122,163), key in the previous section about the ease to deploy ML in micro- in robotics and in videogames design, could be the major scopy despite the scientific background. Interestingly, this breakthrough for the complete TEM automation.489,490 These approach could be directly exported to general TEM to automate and many other ML developments not yet implemented in EM the region of interest finding for any material or nanostructure are reviewed with further attention in the next section. (as with ANIMATED-TEM473). 1448 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review Moreover, autoencoders were also used to detect those single particles orientation and help with the optimisation of the 3D reconstruction step, similarly to S. Kalinin et al.’s work.84,88,497 In fact, ML and DL was used to improve the reconstruction of the tilt series. Remarkably, this ML-based improvement unlocked new fidelity landmarks allowing to decipher the secondary structure of proteins.180,498,499 To achieve it, D. Si et al. used support vector machines, while R. Li et al. used a CNN instead, obtaining comparable results.191,500 As happened with region-finding auto- This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. mation, these approaches can be used to improve the resolution Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. of materials ET, and to make it gentler, as observed with CS.142,146,148 In fact, CS, which was extensively used for materials science, is just testimonial in cryo-EM. Therefore, CS may be a way to improve the biomolecule reconstruction further, even with ML algorithms mimicking CS, by relaxing the total dose and data necessary to complete these reconstructions. 4.1.2 Optical microscopy. A document even larger and denser than the present contribution could be created by reviewing ML in optical microscopy and all the derived techni- ques that fit in this field. Most of the recent advances combin- ing some sort of AI with optical microscopy aim for image segmentation. That is, separating distinctive regions in micro- graphs such as different cell types, steels grain boundaries, or the planar extent of 2D materials (Fig. 7). This distinction will be mainly represented by differences in the intensities of the micrograph. This means that the main parameter to exploit for the segmentation is the value of every pixel, although using the pixel coordinates within the image and neighbouring pixel correlations will be recommended, too. Specifically for biosciences, contouring and drawing the cells area is of paramount importance for multiple studies such as the cell division and the cell cycle, or cell dynamics and the interactions with substrates. Doing it in an automated way is therefore a requirement to obtain representative results over large populations of cells.501 One of the main challenges is that cellular and subcellular features are complex and diverse in morphologies and textures. This problem would hamper the Fig. 7 Segmentation of optical microscopy images of cells, 2D materials implementation of a robust unsupervised solution. As a result, and steels. The cells were imaged with fluorescence microscopy and supervised (mostly CNN) approaches are excelling in this job in segmented using a neural network trained on synthetic 3D models of contrast, fluorescence, and super-resolution microscopy, cells simulating the imaging conditions.511 The optical microscopy of stacks of 2D materials like MoS2 were unsupervisedly segmented and among others.502–510 For that, the idea we have widely reviewed classified to map the number of stacked layers.513 The same applies to on simulating data that resembles the variability of our target optical images of steels and its different phases. This case is a combination micrographs is also valid here. For instance, A. Sekh et al. of the previous two: the authors segmented the images with a trained followed the idea of physics-based labelling by simulating convolutional neural network and classified the steel phases with an fluorescence microscopy images out of the simulated 3D geo- unsupervised clustering algorithm.514 metries of organelles (Fig. 7).511 The authors proved the viabi- lity to train a subcellular DL segmenting tool on simulated data with a workflow that faithfully resembles most of the super- synthesised when optimising the growth of 2D materials, the vised routines reviewed in atomic resolution STEM. At the end, automation of which is welcomed. In these cases, the complexity it is important to remember that the original U-Net was of the imaged features perishes in comparison to those imaged in designed for cell segmentation!65,512 biosciences. Therefore, these simpler features allow unsupervised The reach of ML-based segmentation in optical microscopy routines to succeed. For example, the number of layers of stacked is broadened when applied to materials science. For instance, graphene or dichalcogenide sheets and the micrograins of steel in metallurgy it is a common practise to image steels and locate can be automatically counted and mapped with clustering-based and quantify the spatial extension of their phases and grains. segmentation (Fig. 7).513 In addition, postprocessing can be Furthermore, it is common to extract statistics on the total area applied to the segmented data if an end-to-end application is This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1449 View Article Online Review Nanoscale Horizons 4.1.3 Medical imaging and diagnosis. Radiomics is the study of medical images and their features to uncover shared patterns within related diseases.517 Its main objectives are the risk assessment, the diagnosis and the prognosis (i.e. preven- tion) of (virtually every) imageable pathologies.518,519 Among others, the imaging techniques include computed tomography scan, positron emission tomography or Magnetic Resonance Imaging (MRI), to address multiple diseases such as cancer, neurodegenerative diseases, or more recently lung damage This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. caused by Covid-19.520–523 From the reviewed methods, we Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. know that ML excels in extracting features from data. In addition, it turns out that one of the key points of ML-based radiomics is the generation of meaningful features out of the data to feed richer ML models. These features are descriptors derived from the intensity distribution of the image, as dis- cussed later. Actually, a great strategy to improve the prognosis of these ML classifiers is the enlargement of the feature space. Long story short, to get more meaningful information (features) out of the images. For instance, C. Chen et al. and H. Kniep Fig. 8 The cross-fertilisation between the listed disciplines and electron et al., artificially enlarged the feature space of MRI to up to 43 microscopy for materials science can entail the next breakthroughs in the and 1423 image features, respectively, to distinguish between field. This can happen both by the direct collaboration between specialists metastatic and non-malignant brain tumours.524–526 Both the and by just the inspirational nature of the newest literature. In a potentially feature enlargement and its classification could be done by ML, bidirectional scientific flow, the scheme considers the most important and both offered promising results for predictive medicine. knowledge transfer that might arise from the reviewed fields towards electron microscopy. The studies augmenting the feature space are abundant, but all share the emphasis on capturing the images texture.527–529 By texture the researchers refer to different ways to deal with sought. An example of this is the semi-supervised approach the pixels intensity to convert it into potential features.524,530,531 followed by H. Kim et al., who mixed a CNN-based segmentation For instance, the intensity histogram and the segmentation of and simple linear iterative clustering to firstly, identify, and after- the images and its resulting shapes could be considered as wards classify the microstructure of experimental optical images of additional straightforward features. Moreover, not as direct steel (up to fields of view of about 1 mm) (Fig. 7).514–516 This local descriptors such as the grey-level co-occurrence matrix example highlighted that, as mentioned, the automation makes or the neighbourhood grey-level dependence matrix proved the statistics extraction and the handling of large regions of helpful.525 Indeed, the powerful idea of adding meaningful interest realistic. Nevertheless, its application to EM and materials extra features to the EM data is a synergy that it is worth science might not be so straightforward. The idea of segmentation exploring. This is because most of the explored articles just is a bit more abstract in EM, as the features presented strongly relied on the intensity as ‘‘the feature’’, and only occasionally was depend on the magnification, and there are multiple magnifica- it accompanied by additional channels of information. Being tion intervals providing meaningful information. Thus, the tech- careful about the statistical significance of the added features to niques to be used for segmenting the atoms in an atomically avoid overfitting, this approach could potentially benefit espe- resolved micrograph will differ from the segmentation of a large cially unsupervised routines in EM. This approach could start field of view where no atoms appear. Consequently, the reviewed with the simple stacking of, for example, multiple edge detection segmentation approaches for optical microscopy could be bene- filters as fake channels of the original image. Furthermore, we ficial for EM experimental setups exploring the morphology of could even add multiple DL segmentation routines or higher- nanostructures and nanodevices, as in the case of SEM and low complexity image processing as the added features. Interestingly, magnification (S)TEM. This is because these would be the EM within an interactive exploratory workflow, additional features situations that better mimic the scenarios handled by optical could be progressively added till the model is found to overfit, microscopy, with immense fields of view being analysed. Indeed, and then to progressively constrain these features towards a more both industrial processes and other tedious counting steps would robust eventual unsupervised model. take valuable advantage from it. Importantly, the sparsity of the 4.1.4 Scanning probe microscopy. The previous sections EM features would likely demand a supervised model to attain showed that certain developments involving ML in STEM were robustness. Hypothetically, the first step towards this time-saving closely inspired by previous works on SPM, specially STM, and model could be transferring the learned weights from the optical vice versa. In fact, some ground-breaking data analysis techniques microscopy models reviewed here. Unfortunately, the additional were general enough to allow a positive proof of concept in both generation of the necessary meaningful ground truths to complete STEM and STM simultaneously.59,81 This healthy cross-feeding the training remains an open challenge, by now. could meet new horizons with the implementation of the 1450 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review following trends in SPM automation.532,533 We could hitherto equivalent to force–distance curves would allow the immediate find little research done in segmentation and SPM data analysis exploitation of these already built-in model validation tools. As involving ML (thus it could heavily benefit from the routines a result, EM experiments correlating time and beam-damaged described for optical microscopy and EM, respectively), whilst the area, voltage and domain orientation, or gas pressure and ML-based approaches for SPM automation are manifold.534–537 growth ratio, would instantly have their analysis automated This is justified given that the main bottleneck in SPM could be when matched with the workflow provided by atomic force considered the actual data acquisition and the preparation of the microscopy and its accessory techniques. experimental setup. In summary, the whole SPM automation is based on optimising the tip-sample interaction. Moreover, that 4.2 Big data and physics This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. interaction and the resulting data must be evaluated to accept it 4.2.1 Astronomy. ML in astronomy discovery is a pilar in Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. or not as an appropriate source of data. For this purpose, practically all its subfields, and becomes a source of hope for automatically assessing the tip quality for its conditioning the next decades advances in the field.550,551 Astronomers, becomes a key first step. In fact, M. Rashidi and R. Wolkow, accustomed to dealing with huge datasets of up to many and S. Wang et al. trained neural networks to automatically terabytes, embraced data mining as the only viable route identify tip artifacts in imaging and spectroscopic STM, against increasing data volumes.552,553 Paradigmatically, only respectively.538,539 Thanks to that, and beyond tip characterisa- a fast literature search involving the ‘‘ML’’ and ‘‘astronomy’’ tion, the next step could be the ML-fulfilled conditioning of the keywords is required to reach many research articles involving tip based on this automatic assessment. This can be done either the advanced use of 3D CNN.554,555 The fact that convolutional by directly fabricating the tip, as exemplified in B. Li et al.’s work, filters of higher than two dimensions appeared just once (for or by functionalising it with, for instance, carbon monoxide ptychography) in the reviewed EM bibliography, emphasises molecules to enable molecular imaging, as in B. Alldritt et al.’s the power this synergy may have in, for instance, dealing with work.540,541 Most of these works rely on the partial data acquisi- 4D STEM and hyperspectra. tion for extracting the minimum necessary information quickly, The detection of a priori hidden features or patterns in data closely resembling what reported for CS in STEM and ET.542 This is of outmost relevance in the characterisation of the outer idea could be extended further to the automation of the electron space.554–557 Finding new galaxies, black holes, dark matter and microscope and in high-throughput sample preparation layouts even dark energy is driving astronomers mad, and the ML tools based on automated focused ion beam.543,544 Interestingly, the are helpful on this crusade.558–560 For example, neural networks tip evaluation could be turned into the analysis of artifacts in the and decision trees proved useful for detecting black holes in acquisition of EM, and its decomposition into a probability globular clusters, dark matter in strong-lensing systems, and distribution stating the likelihood of these artifacts coming from even for gravitational wave sensing.561–563 These examples a malfunction of the tip, some dirty aperture, or maybe just the could fit into the category of anomaly detection, which has aberrations of the lenses. not been formally explored in detail in EM yet. The concept of A similar idea is recurrent in Atomic Force Microscopy anomaly detection is quite self-explanatory, being key in multi- (AFM), although most of its ML studies are devoted to optimise ple scientific, technical or business fields, as finding and the extremely time-consuming analysis of force–distances understanding outliers is inevitably a source of knowledge. curves.545,546 These curves arise when testing the proteins Besides, as ML excels in finding patterns or regressions in superstructure in tension/elasticity experiments, with the bind- data, just by complementarity so does it to find data that ing/unbinding of proteins or macromolecules that interact with deviates from those found regressions. Other astronomy appli- each other, after the indentation of the tip into cellular and cations that also shine light on the potential of ML anomaly subcellular structures, or into organic molecules and inorganic detection are the spotting of extragalactic transient astronomic materials. The typical workflow is then to fit these curves into a events (e.g., supernovae), the refinement of supernovae simulation possible model that explains the (bio)physics behind the inter- through anomaly detection and correction, or even the general action. Of course, this may end up in a time-consuming searching for any anomaly in any dataset in the so-called iterative process guessing the most close-fitting assumption. ‘‘Systematic serendipity’’, among others.564–567 Moreover, an Nevertheless, the advances in ML allow to automate its analysis especially interesting field of application is the detection of by fitting the data with trained models. In fact, this process was exoplanets or near-Earth objects watch (e.g., asteroids and automated by training neural networks both to fit the curves comets), and the premature detection of those having orbits and to generate test models, therefore accelerating the model- with potential impact hazard to the Earth.568–571 Not surpris- experiment correlation.547–549 ingly, CNN excelled in the detection of both exoplanets and Interestingly, the tackled experimental setups of atomic asteroids, again confirming their versatility.572–575 force microscopy share the fact that they induce some degree Importantly, these ideas closely bonded to anomaly detection of modification to the sample, either by digging a topological would fit in the EM framework with ease. Some would say that hole in a wafer or by stretching the ternary structure of a some of the applications reviewed would fit in the anomaly protein. Therefore, these time or effect-resolved analyses would detection category, but as said, the truth is that the concept fit properly in the framework of in situ EM experiments. The hardly ever appeared formalised in EM studies. At least not conversion of the acquired in situ datasets into a data format until P. Cho et al. developed an anomaly detection method This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1451 View Article Online Review Nanoscale Horizons combining PCA and CNN for detecting point defects in atom- particles in traces of jets (e.g., quarks vs. gluons) or to directly ically resolved micrographs.68 Interestingly, the authors could discriminate actual data from background.585,586 A particularly generate heatmaps highlighting the image regions in which the interesting example representing this trend came from presence of defects (i.e., anomalies) was more likely. As men- P. Komiske et al.’s work, in the framework of physics-aware tioned, so far, we have reviewed a few EM studies that operated models.585 The authors used a CNN in which the input was a similarly, but none performed as maturely as the surveyed multichannel image and the output the subtle features and examples in astronomy. It is not about being picky about the patterns (e.g. particles) to detect. The novelty lied in that the wording given to every application, but to have the anomaly first channel of the input image represented the image itself, detection routines in mind as tailored solutions for problems but every additional one represented a different measured This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. like the point defects example. For instance, the astronomy physical property. In this way, the authors made sure the model Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. example holding the comparative and cooperative training would treat differently two images if obtained under different between anomaly-free and anomaly-full datasets would not even experimental conditions, consequently making the model be thinkable unless the anomaly detection concept was deeply aware of the experimental setup employed. The idea introduced implanted in the researcher mindset.567 Indeed, the defects here is of paramount importance for ML-based EM and will be detection idea could be opened up further to interpret any developed a few subsections ahead. Also of great importance is lattice distortion or strain: a model trained on a perfect regular the use of generative models in the field. In fact, we could find atomic lattice (anomaly-free), complemented with a model multiple studies exploiting the goodness of GANs, specially for trained on a distorted lattice labelling the strain and the defects simulating particle showers, as reviewed above.587–589 The (anomaly-full). On the other hand, in an EELS edge identifi- advances and examples in EM using GANs are not scarce, but cation model, the anomaly-free set could be represented by a as mentioned, they have only just scratched the surface of what smooth background, with the anomalies being the core-loss is possible. Again, and in a general basis, the EM community edges themselves. Or even subtler with the edges acting as the should slowly push towards embracing generative models over main signal and the anomalies being the edge shifts or near- supervised ones, even if it may seem intimidating. This is why, edge variations, in a finer chemical analyser. the imitation of the progress done in the previous jet-simulator 4.2.2 High-energy physics GANs studies may serve us to simplify an otherwise titanic task. 4.2.2.1 Classical methodologies. The idea of doing a final fourth section comparing the state of the art of ML in EM with 4.2.2.2 Quantum machine learning. The outstanding level of its state in other fields has its roots in astronomy, but also in the ML practitioners in high-energy physics allowed the field to high-energy physics. Since its origins, the uninterrupted mas- reach the next level of complexity: quantum ML (QML). While the sive data flow coming from particle colliders demanded special quantum computing researchers strenuously battle to profile the data processing tools, as in astronomy. Again, ML embraced problems and algorithms that can benefit from quantum this labour fitting well for necessities such as the adoption of mechanics, the research on particle physics has already applied generative models to provide a real-time answer to simulations some solutions based on this new paradigm.590 In any case, the on the detector response, to accelerate the exploration of end-to-end QML is currently a chimera, although intermediate parameters based on the matrix element method, or to allow steps such as algebra, linear differential equations solving, or the analysis of the rawest data, such as raw detector hits, pattern recognition have been implemented.591,592 Excitingly, the in end-to-end models.576–580 These cases constitute narrow literature referring to quantum advantage in this field is vast, field-specific problems with a hard to stablish equivalence to although we want to specifically highlight the next piece of work EM. However, the aspect that receives most of the attention, given its potential (direct) applicability to EM. S. Chen et al. and that is also easier to link, is the analysis, identification, proposed a quantum CNN to classify multiple high-energy phy- discrimination and classification of the collisions themselves sics events.593 In the model, the convolution operations were and the resulting particle jets and showers.581,582 performed by a variational quantum circuit, which is a set of Generally, the main goal is finding rare events deviating quantum states and parameters that can be optimised iteratively from a standard model background (i.e. the main theoretical and classically. In this case, the network consisted of only two model describing the elemental particles and their interac- quantum convolutional kernels, but already learned and tested tions). Importantly, most of these works are centralised in the faster than the classical analogue (Fig. 9a).594 Therefore, we Large Hadron Collider (LHC), which the foremost hub to be cannot help seeing this exact same quantum kernel learning inspired by.583,584 Either by a centralised acquisition and the patterns from EM images, as the workflow is exactly the same processing in the same LHC, or by acquiring in LHC and as the multiple times reviewed along this contribution. processing in collaborating centres, or vice versa, hardly ever Interestingly enough, two of the main hot topics in the is the LHC not involved in one way or another. In fact, it current quantum computing/quantum ML research could be conducted the main advances in ML for finding these rare of direct concern for EM. These are quantum Fourier transforms events in data. The huge amount of available data and the ease and quantum image processing.595–597 This increasing interest to accurately simulate it made that mainly supervised models is based on the belief of the eventual supremacy of quantum were used since the beginning. These were typically based on computing for these applications.594,598–600 As a result, it is easy both CNN and RNN to distinguish between different types of to get overexcited with the perspectives QML would promise in 1452 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Fig. 9 Future perspectives of advanced Machine Learning (ML) for Electron Microscopy (EM) data analysis and automation. (a) Quantum Convolutional Neural Networks (QCNN) in high-energy physics already provided quantum advantage versus classical CNN even with just two convolution kernels. However, there are no research examples of quantum ML in EM yet.594 Reprinted by permission from Springer Nature, I. Cong et al., Nat. Phys., 2019, 15, 1273–1278, Copyright (2019). (b) Schematic of a Reinforcement Learning (RL) CNN designed to autonomously play videogames. The convolutional layers allowed to get information from the videogame frames and to translate that into actions (joystick + buttons) that would maximise the score in a given game. Instead of a joystick, we can think of the microscope control panel in a RL approach that seeks to maximise the image quality.642 Reprinted by permission from Springer Nature, V. Mnih et al., Nature, 2015, 518, 529–533 Copyright (2015). (c) RL used to train an autonomous surgical robot. The RL trained the robot to take the elements on the table one upon the other as in a surgical intervention. The algorithm would reward putting the objects together and penalise any other action that would not end up in this result.627 We can imagine a similar robot that takes a lamella or a grid to a holder and afterwards inserts it into the electron microscope. Furthermore, if we concatenate this autonomous holder loader/inserter with the joystick-like approach, we obtain the fully automated microscope! these two routinary processes in EM. Nevertheless, as reviewed on classically simulated atoms but adding a quantum circuit through the entire text, the path classical ML must follow is still forcing quantum mechanical interactions between them. This huge, and QML should only be thought as a complementary and idea lies within the science-aware experiment planning, treated exploratory tool, by now. For instance, it would make sense to in more detail below. explore the advantages of QML for well-established applications 4.2.3 Earth sciences. Earth sciences comprise a wide variety such as DL-based atomic-column finding. Even to just prepare of disciplines like meteorology and climate sciences, minerology, the equivalent algorithms for the highly-anticipated establish- seismology, and geophysics, among many others. These disci- ment of quantum computers, or for testing them in quantum plines have in common a special attention to forecasting and simulators or tensor networks, just like the quantum computing causal analysis. As a result, these domain scientists embraced ML practitioners do. On the other hand, exploring the new horizons as a logic response for the predictive modelling of different of the ML functionalities in EM, directly stepping through phenomena.601 Representatively, in climate sciences, unveiling quantum algorithms might be detrimental given the additional the cause-consequence relations of extreme weather events implementation barriers. through data analysis is a major research line.602–605 Indeed, Quantum algorithms are thought to transform the modelling the literature is full of ML-based examples (i.e., climate infor- of physical systems in which quantum physics have a primordial matics) predicting extreme climate events for preventive role. As nanostructures are typically ruled by quantum laws, the meteorology.606–610 For instance, deep CNN are used to detect access to these machines should enhance its simulation fidelity. tropical cyclones, atmospheric rivers and weather fronts, and In the case of EM, as commented, the required level of theory to decision-trees are used for the parametrisation of moist convec- build large datasets for ML applications is not too strict. However, tion in predictive climate modelling.611,612 The fact these models using quantum circuits to force a physical constrain in our ML are fed with images from satellites make the paralleling with models would be amazing. It would allow our models to encode electron micrographs direct. Equivalently, in geosciences, ML is certain complex behaviours that otherwise would be extremely employed for predicting land movement, and the magnitude and difficult to learn. For instance, an image dataset on classical epicentre of earthquakes and their causes.613–615 Particularly, the atomic columns could be immediately converted into an ab initio preventive forecast of land tremors based on the seismologic dataset with the following idea: we could train a neural network history of a given region of the planet is widely studied by the This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1453 View Article Online Review Nanoscale Horizons community.616–620 Interestingly, support vector machines could ML was used in more critical situations such as the training of predict the distribution of earthquakes in Indonesia after being surgical robots both to precisely respond to the inputs of surgeons, trained (i.e., with time, location, magnitude and depth) with and to perform key interventions in the absence of human super- 30 years of its seismic history.621 Similar results were obtained visors (Fig. 9c).626,627 More closely related to materials science, also using support vector machines in the short-term prediction robotics already proved the autonomous ML-guided discovery of of low-magnitude earthquakes in Cyprus.622 Support vector materials and nanostructures.628,629 Therefore, the technology that machines have a similar behaviour than regular unsupervised should unlock the autonomous TEM is already here, ‘‘only’’ clustering, except that they are trainable. Indeed, when seeking requiring the community’s effort for guiding it to the longing goal. for a clustering-like behaviour but labelled data is available (i.e., Interestingly, one of the challenges in the nowadays robotics This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. the label would represent the cluster), the best way to proceed is the better identification and representation of the elements Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. would probably be the training of support vector machines. For present in the robots environment.630 Equivalently, we have example, by grouping seismic data into whether it ended up or reviewed many examples of identification of these features in not in an earthquake (label), as in the previous examples. EM micrographs and spectra, mainly by CNN. However, EM Overall, the image processing and CNN usage of most of could take extra advantage of a trend in robotics called perceptual these examples only exhibited the first degrees of complexity if learning, which aims to optimise the feedback with the compared to EM. This is why, (imaging-based) Earth sciences environment.631 Bioinspired, perceptual learning modifies the would leverage the previously reviewed image processing techni- nature of signals before reaching the traditional learning algo- ques. The trivial example is on crystallography, which is key for rithms to enhance the interpretability. As if electron micrographs geology. Oppositely, as evaluated, Earth sciences excel in forecast- were abstracted and encoded (e.g., through simple processes like ing tasks. As a result, EM could take advantage of similar predictive downsampling, filtering, (e.g., Fourier transforming) before the decision-tree algorithmics to work, for instance, in conjunction main processing through the deep convolutional filters of a CNN. with CS. Thus, they could fill missing data gaps through predic- Particularly, this idea is becoming key in the development of tions, after being trained on pairs of partial-complete datasets. mobile robots.632,633 The TEM cannot be considered a mobile These data gaps could be either time-resolved for an efficient and robot, although the constantly changing interaction with the low-dose in situ setup, 3D-correlated to overcome the missing environment (i.e., changing magnification and sample position) wedges in tomography, or for partial scans in STEM, as hap- is comparable and could surely benefit from perceptual pre- pened with CS. Another application of interest would include processing. Even the holder handling and loading could be the prediction of events such as spontaneous crystallisation or directly tackled with the mobile robot perspective! phases transitions during in situ experiments. Another learning paradigm that is being and will be a primary breakthrough for robotics is RL.489,634 Surprisingly, 4.3 Artificial human-like systems: the path towards its applicability to robotics is mainly motivated by its reach and automation? research in videogames. From the well-known Deep Blue that As discussed, the ultimate goal of AI in EM is the total automation defeated the chess world champion Garry Kasparov, and Deep- of the electron microscope. In other words, we dream of a TEM to Mind’s AlphaGo that beat the Go world champion Lee Sedol, to autonomously behaving as if it was operated by a human expert. recent algorithms capable to beat much more complex game With this major goal in mind, the close examination of technolo- mechanics, the AI in videogames has constantly evolved since its gical fields devoted to automating human actions can be highly simple origins. It was capable of remarkable milestones even if it beneficial. Obviously, the major methodology breakthroughs are was not actually based on AI algorithms: Deep Blue relied on a linked to the fundamental research on pure computer science and traditional brute-force decision tree while AlphaGo was based on AI. Unfortunately, though, these are occasionally difficult to Monte Carlo tree search. In summary, these algorithms are based abstract to practical cases. Nevertheless, there are some fields on the so-called behaviour trees, an evolution of predefined states closer to the application development the strategies of which reacting to certain stimuli, following a heuristic logic. It was might be even more useful for the practical purposes to which only recently when videogames incorporated AI to generate our community is devoted: robotics and videogames. These dis- human-behaving game engines, but also to self-challenge these ciplines rapidly come to mind when thinking on complex auto- same games.635 RL, together with evolutionary algorithms and mated behaviours and continuous interactions with human automated ML burst into the field to answer the need of beings, similarly to voice-assistant services. For EM, the main adaptability.636–639 Meaning that a supervised model will perform automation challenge is in the reproduction of domain-specific great to a given data but cannot be adapted to new data unless it knowledge rather than in the development of complex methodol- is trained with it. These new paradigms act as if they were ogies. At the end, the microscope automation would just demand continuously trained with the new data they may end up facing. the tuning of the lenses, which is translated as modifying a value The main idea behind these new learning paradigms is the in a software. In a second stage of complexity, the most exotic iterative maximisation of a score or fitness metric that encodes automation step would be sample mounting and holder insertion/ whether the model performed well or not in a given task. Thanks extraction. Indeed, robotics and videogames present manifold to this, powerful RL algorithms such as deep Q-Network or automation routines oriented to multiple processes of comparable Go-Explore surpassed the human performance in multiple video- or even higher complexity in their mechanics.623–625 For instance, games (Fig. 9b).640–642 Interestingly, the mechanics of some of 1454 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review these games implied hundreds of actions that required optimisa- Moreover, if adequately planned, the response of the models tion depending on the in-game situation. Astonishingly, some against the constrained physical parameters should be more learned action trees (i.e., the decisions of the RL algorithm) could general and easier to achieve and interpret. go beyond the preestablished rules of the game in favour of the The first way to explore physics constraints and also the score maximisation.643 Yes, the AI learned to cheat in these typical way to introduce them in our models is by guiding the games! In terms of implementation, importantly, the main algorithms as steps with a physically logical sequence.647 This companies developing these algorithms, such as OpenAI is the workflow of the 99% of works in any scientific field and is or DeepMind, provide RL solutions for free to academia that just processing the data differently based on the science ruling constitute optimal starting points for introducing newcomers to the phenomena. Indeed, the concept we want to communicate is This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. these new learning paradigms.644–646 the natural inclusion of this logic in end-to-end ML models that Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Going back to the microscope automation, the correlation overwrite the manual steps. A great starting point is to use with the proposed research is direct, and the cross-fertilisation, models the architecture of which explicitly carry a physical welcome. As indicated, a robotic electron microscope would constraint. In this case, approaches like the previously reviewed autonomously change the power of the electromagnetic lenses rVAEs are capable of extracting some latent variables with to improve the image quality. In this case, the image quality physical constraints like rotations or translations as output.84,87 would represent the score whilst the modifiable lenses are the Furthermore, in the formal science-guided approach, we a priori set interactable parameters. Therefore, the problem is reduced to a physical constraint forcing the ML model to have this predefined the optimal chose of the score metric based on the data to condition or consideration as part of its nature.648,649 A straightfor- acquire. Defocus and higher-order aberrations, artifacts, and ward implementation can just be the addition of accessory channels the different regions of interest in our nanodevices should be to EM images, each of which accounts for an experimental para- encoded in this metric, assessing the quality of the data and the meter of the acquisition. For instance, adding the defocus in a focal complete scanning of the nanostructure. More precisely, this series for its automated reconstruction, or adding a channel per problem would lie in the sparse-reward framework of RL, in independent aberration in a digital post-acquisition aberration which a positive reward might depend on multiple consecutive corrector. A clear example already reviewed doing this was the work and correct actions. Importantly, the identification of the of L. Roest et al. for modelling the ZLP in different experimental features defining the score metric was mostly automated by the conditions.296 Another option to physically constrain models is the reviewed examples.35–38,40–44,473 Then, the goal would be to modification of the architecture of a priori unphysical models to reduce this sparsity or make it less meaningful. One option could capture these restrictions. Although it is not strictly EM, the follow- be linking each tuneable parameter with a score modification, ing inspirational example in the framework of X-ray coherent which would imply a hard encoding of the score. In addition, this diffraction imaging represents this idea. Moreover, it would be score or metric could be further guided as well by recorded equivalent to EM as it provides a 3D reciprocal reconstruction microscopy sessions capturing the variation of the tuneable similar to electron diffraction tomography. In this way, H. Chan parameters and the resulting transient image quality. Alterna- et al. built a 3D autoencoder with a single encoder-double decoder tively, and mimicking the research on videogames, multiagent architecture trained to invert 3D X-ray diffraction patterns into real training could enhance the creativity of the algorithms in mod- space information of nanostructures.650 The innovation lied in the ifying the interactable parameters towards a top score.642,643 multiple-decoder architecture that allowed to split the training and Thus, training competing agents on the same problem to force the output in different physically distinctive information: in the unexpected behaviour trees. This could be translated into many present case, the predicted shape of the nanostructure (retrieved microscopy sub-routines or entire microscopes parallelly compet- amplitude) and its strain through (retrieved phase). The authors ing with each other to maximise the sparse score based on the named this architecture AutoPhaseNN and proved its inverse image quality. Excitingly, this could potentially mean just leaving reconstruction to be about one hundred times faster than other the microscopes being trained alone (i.e., trial and error to phase retrieval methods.651 This work is an equivalent approach to maximise the score) for a week to retrieve a fully operative the reported PtychoNN for EM ptychography reconstructions.121 autonomous entity afterwards. Even more excitingly, this could Nonetheless, the main inherent drawback of doubling the lead to innovative ways to operate the microscopes that we, as outputs is the consequent doubling of the required labels for trained humans, could have never thought before, in the same training. Therefore, this would almost deny the possibility to way a reinforced AI can break the game designers’ rules. train this kind of models with experimental data. Oppositely, when using synthetic simulated data, the obtention of extra 4.4 Other fields of research labels will not cost additional computing resources, making it Beyond the discussed learning paradigms like RL, automated the preferred choice (i.e., in simulations, we get simultaneously ML, or evolutionary algorithms, complementary constraints the phase and the amplitude, both useable as double labels). In can also add extra value to the more traditional supervised conclusion, this application further highlighted the potential of and unsupervised approaches. We have briefly stated through physics-aware modelling to enhance the research in more this review the importance of adding knowledge constraints to general reverse engineering EM scenarios.652–655 Unfortunately, ML algorithms. Interestingly, this can simplify our model in similar instances in EM are not yet common, certainly display- terms of necessary training volume and architecture complexity. ing a knowledge gap yelling to be filled. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1455 View Article Online Review Nanoscale Horizons Quite related, the physics-aware architecture design might cognitive biases, or the creation of optimised portfolios given an be intuitively represented by the recent advances in graph investor’s profile, count as ML-based daily-life operations.665–667 neural networks.656 As mentioned, the properties of EM data- The common pattern in both fields lies behind the assignation sets should allow a direct transformation of their formats into and optimisation of the variables, typically numerous and with graph representations. The easiest example would apply for an intricate data structures. For EM, this expertise might help in image in which each pixel represents a node and connects with experiment planning and decision-making refinement in autono- as many edges as neighbouring pixels. There is an absence of mous EM, through a set of predefined goals and sample features EM bibliography referring to the potential benefits of graph constituting the variables space. Additionally, the optimisation of networks for image processing, although more general compu- financial and trading problems has undergone through mean- This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. ter vision applications have already benefited from them.657,658 ingful improvements with quantum computing, and more pre- Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Indeed, multiple physically constraining strategies can be cisely with QML.668–670 Therefore, as introduced by the research in followed at once to improve the generalisation capabilities of high-energy physics leading to the consequent possibilities of QML the final model. Indeed, a fatal error would be to limit the in EM, quantum finances strike to reiterate the likely future of EM thinkable possibilities of the reviewed tools to just the reviewed with QML. cases and the resembling. Interestingly, general digital imaging benefited from combining a 3D CNN with graph neural networks and the distance channel (i.e. physical constrain) of RGB-D data to 5. Conclusions and final remarks automatically segment images.659 The addition of the D channel (in-depth distance) allowed to, by graph representations, encode Through this document, we have reviewed in detail the and refine geometric information otherwise impossible to extract advances that ML has introduced to the broad field of EM. By only with intensity-based channels. Conveniently, this simple unavoidably sliding to the domains of materials science, the constraining of digital images consolidated the previous discus- main electron-based imaging and spectroscopy techniques sion on the benefits that even a simple physics guiding may have. have revealed important scenarios in which ML is shown to Nonetheless, when graph-like systems were allowed to be essential to decipher new physical phenomena, to model evolve in time under a certain physical constraint, graph net- nanostructures or for advanced micrograph and multidimen- works showed the most outstanding and promising results. For sional data processing, among many others. From traditional example, when they were used to simulate dynamical systems parallel-beam TEM and STEM imaging to the possibilities of 4D for providing trustworthy digital twins of complex physical STEM, and from spectrum profiles to fully 3D compositional behaviours like mass-springs, rigid bodies (e.g., to be applied maps, the variability of unsupervised, supervised, semi- in a videogames physics engine, closing the previous sections supervised, generative and RL, or even CS, have extensively loop), molecules, or to encode finite element simulations, showed their goodness. Nevertheless, we have also shown and among others.485,486,488 Again, the previous examples would discussed the current ML challenges, still open due to the early better fit into pure AI research, but the versatility these graph stage of the ML journey into the EM community. neural networks have shown are worth the exploration. As This early stage and the recent motivating discoveries will briefly pointed through the document, the properties of graph likely turn into an avalanche of (electron) microscopists want- networks could be applied to EM, for instance, to follow the ing to become ML practitioners. Consequently, we have tried to evolution of features changing along an in situ stimuli. Either provide an easy step-by-step guide for microscopists and mate- by the pixelwise evolution of node-per-pixel graphs, or by the rials scientists to start with ML despite their initial level. Our identification of features and its posterior encoding into con- scope was not the review of the algorithms and algebra behind nected nodes. We would like to warn the reader, not to forget ML, as there are endless resources covering this, but to present that complementarily, this already scientific and technical a sorted set of tools that may make the (new) practitioners’ life encoding could be further assisted by adding extra information much easier. At least in the first stages, applications, or proto- channels to the micrographs with the mentioned stimuli: time types. Relatedly, the guide aimed to schematically educate frame of a sequence, voltage, gas pressure, beam intensity, the about the possibilities of ML in EM applied to materials discussed distance-to-camera, etc.659 science. Ideally, its application should be accompanied by the To conclude this section, we want to briefly go through a vast deepest possible understanding of the mathematical models and distant field of knowledge specially experienced in the behind to avoid falling into the comfortable but dangerous void parallel optimisation of huge multivariate systems: marketing of the black boxes. and finances. Both marketing and finances automated most of Finally, the exploration of the ML literature around other their routinary tasks by using ML.660–662 On one hand, market- scientific fields, revealed an enormous source of refined meth- ing seeks to turn into a variable every action a customer may do, odologies and successful implementations of AI into science. first, before deciding to buy a product, and then to actually buy From neighbouring microscopies to high-energy physics and it. As a result, monitoring and statistically linking these vari- astronomy, and even from apparently unrelated fields such as ables for all the potential customers would represent a titanic videogames or finances. From all of them we have been able to task in which ML has already provided insight.663,664 On the extract valuable insight to draw the future in which the EM/ other hand, the automation of small stock transactions avoiding materials science points to. Interestingly enough, while surfing 1456 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review this literature some of the unanswered challenges presented for RNN Recurrent neural network EM started receiving a preliminary treatment. As we frequently rVAEs Rotationally invariant Variational AutoEncoders wrote, the slight adaptation and fine tuning of certain reviewed SEM Scanning electron microscopy methodologies would instantly mean solving or improving a SI Spectrum image lack in EM. It could be a data-format transformation, or SNR Signal-to-noise ratio just a direct exchange between physical properties, but rarely SPM Scanning probe microscopies a logistically prohibitive process. Moreover, the reviewed tech- ST Spectral tomography nical and scientific fields outlined the learning paradigms that STEM Scanning transmission electron microscopy must rule the progress of the next decade, with RL and STM Scanning tunnelling microscopy This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. generative modelling being the most promising ones. In addi- SVD Singular value decomposition Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. tion, special care and attention must be paid to QML as a TEM Transmission electron microscopy future insurance, but specially to physically-aware AI as a VAEs Variational autoencoders present guarantee. In conclusion, we wanted to foster the habit VCA Vertex component analysis of browsing through other fields to seek for new ideas to XRD X-ray diffraction implement in the readers’ own community. The idea of inter- ZLP Zero-loss peak communicating scientific fields is rich and global and adds value to one of the most beautiful aspects of ML being the amazing community devoted to open science and knowledge Conflicts of interest transfer. There are no conflicts to declare. Abbreviations Acknowledgements AFM Atomic force microscopy The authors acknowledge funding from Generalitat de Catalu- AI Artificial intelligence nya 2017 SGR 327. ICN2 is supported by the Severo Ochoa CL Cathodoluminescence programme from the Spanish Ministry of Economy (MINECO) CBED Convergent-beam electron diffraction (grant no. SEV-2017-0706) and is funded by the CERCA Pro- CNN Convolutional neural network gramme/Generalitat de Catalunya. The authors thank support CS Compressive sensing from the project NANOGEN (PID2020-116093RB-C43), funded DFT Density functional theory by MCIN/AEI/10.13039/501100011033/and by ‘‘ERDF A way of DL Deep learning making Europe’’, by the ‘‘European Union’’. Part of the present DPC Differential phase contrast work has been performed within the framework of the Uni- EDX Energy dispersive X-ray spectroscopy versitat Autònoma de Barcelona Materials Science PhD pro- EELS Electron energy loss spectroscopy gramme. We acknowledge support from CSIC Interdisciplinary EFTEM Energy-filtered transmission electron microscopy Thematic Platform (PTI+) on Quantum Technologies (PT- ELNES Energy loss near edge structure QTEP+). M. B. acknowledges funding from SUR Generalitat de EM Electron microscopy Catalunya and the EU Social Fund; project ref. 2020 FI 00103. ES Electron spectroscopy This study was supported by MCIN with the funding form ET Electron tomography European Union NextGenerationEU (PRTR-C17-I1) and Gener- FCNN Fully convolutional neural network alitat de Catalunya. This study was supported by EU HORIZON FFT Fast Fourier transform INFRA TECH 2022 project IMPRESS (Ref.: 101094299). FSL Few-shot learning GAN Generative adversarial network GMM Gaussian mixture modelling References GP Gaussian processing HAADF High-angle annular dark-field 1 L. Jones, et al., Smart Align—a new tool for robust non- HRTEM High-resolution transmission electron microscopy rigid registration of scanning microscope data, Adv. Struct. ICA Independent component analysis Chem. Imaging, 2015, 1, 8. LHC Large hadron collider 2 A. De Backer, K. H. W. van den Bos, W. Van den Broek, ML Machine learning J. Sijbers and S. Van Aert, StatSTEM: An efficient approach MLP Multilayer perceptron for accurate and precise model-based quantification of MRI Magnetic resonance imaging atomic resolution electron microscopy images, Ultramicro- NMF Non-negative matrix factorisation scopy, 2016, 171, 104–116. PCA Principal component analysis 3 A. De Backer, S. Van Aert, P. D. Nellist and L. Jones, QML Quantum machine learning Procedure for 3D atomic resolution reconstructions using RL Reinforcement learning atom-counting and a Bayesian genetic algorithm, arXiv, This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1457 View Article Online Review Nanoscale Horizons 2021, preprint, arXiv.2105.05562, DOI: 10.48550/arXiv. 20 M. Shen, et al., Multi defect detection and analysis of 2105.05562. electron microscopy images with deep learning, Comput. 4 A. De Backer, G. T. Martinez, A. Rosenauer and S. Van Aert, Mater. Sci., 2021, 199, 110576. Atom counting in HAADF STEM using a statistical model- 21 C. H. Lee, et al., Deep learning enabled strain mapping of based approach: Methodology, possibilities, and inherent single-atom defects in two-dimensional transition metal limitations, Ultramicroscopy, 2013, 134, 23–33. dichalcogenides with sub-picometer precision, Nano Lett., 5 M. Vatanparast, et al., Strategy for reliable strain measure- 2020, 20, 3369–3377. ment in InAs/GaAs materials from high-resolution Z-contrast 22 F. Wang, T. R. Henninen, D. Keller and R. Erni, Noise2A- STEM images, J. Phys. Conf. Ser., 2017, 902, 012021. tom: unsupervised denoising for scanning transmission This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 6 J. M. Zuo, et al., Lattice and strain analysis of atomic electron microscopy images, Appl. Microsc., 2020, 50, 23. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. resolution Z-contrast images based on template matching, 23 J. M. Ede and R. Beanland, Improving electron micrograph Ultramicroscopy, 2014, 136, 50–60. signal-to-noise with an atrous convolutional encoder- 7 M. J. Hÿtch, J.-L. Putaux and J.-M. Pénisson, Measurement decoder, Ultramicroscopy, 2019, 202, 18–25. of the displacement field of dislocations to 0.03 Å by 24 T. M. Quan, et al., Removing imaging artifacts in electron electron microscopy, Nature, 2003, 423, 270–273. microscopy using an asymmetrically cyclic adversarial net- 8 M. J. Hÿtch, E. Snoeck and R. Kilaas, Quantitative measure- work without paired training data, in Proc. – 2019 Int. Conf. ment of displacement and strain fields from HREM micro- Comput. Vis. Work. ICCVW 2019, 2019, pp. 3804–3813, DOI: graphs, Ultramicroscopy, 1998, 74, 131–146. 10.1109/ICCVW.2019.00473. 9 S. Bals, S. Van Aert, G. Van Tendeloo and D. Ávila-Brande, 25 J. P. Buban and S.-Y. Choi, Auto-encoders for Noise Statistical estimation of atomic positions from exit wave Reduction in Scanning Transmission Electron Microscopy, reconstruction with a precision in the picometer range, Microsc. Microanal., 2017, 23, 130–131. Phys. Rev. Lett., 2006, 96, 096106. 26 J. L. Vincent, et al., Developing and Evaluating Deep Neural 10 L. Jones and P. D. Nellist, Identifying and correcting Network-Based Denoising for Nanoparticle TEM Images scan noise and drift in the scanning transmission electron with Ultra-Low Signal-to-Noise, Microsc. Microanal., 2021, microscope, Microsc. Microanal., 2013, 19, 1050–1060. 27, 1431–1447. 11 N. Bonnet, Multivariate statistical methods for the analysis 27 S. Mohan, Adaptive Denoising via GainTuning, Adv. Neural of microscope image series: Applications in materials Inf. Process. Syst., 2021, 34, 23727–23740. science, J. Microsc., 1998, 190, 2–18. 28 S. Mohan, Deep Denoising For Scientific Discovery: A Case 12 N. Bonnet, E. Simova, S. Lebonvallet and H. Kaplan, Study In Electron Microscopy, IEEE Trans. Comput. Imaging, New applications of multivariate statistical analysis in 2022, 8, 585–597. spectroscopy and microscopy, Ultramicroscopy, 1992, 40, 29 R. Manzorro, et al., A Deep Learning Approach to 1–11. Retrieving 3D Structure Information from High Resolution 13 M. Bosman, M. Watanabe, D. T. L. Alexander and Time-Resolved TEM Images, Microsc. Microanal., 2021, 27, V. J. Keast, Mapping chemical and bonding information 464–465. using multivariate analysis of electron energy-loss spectrum 30 J. Lee, Y. Lee, J. Kim and Z. Lee, Contrast transfer function- images, Ultramicroscopy, 2006, 106, 1024–1032. based exit-wave reconstruction and denoising of atomic- 14 K. Aso, K. Shigematsu, T. Yamamoto and S. Matsumura, resolution transmission electron microscopy images of Detection of picometer-order atomic displacements in graphene and cu single atom substitutions by deep learn- drift-compensated HAADF-STEM images of gold nanorods, ing framework, Nanomaterials, 2020, 10, 1977. Microscopy, 2016, 65, 391–399. 31 A. Suveer, A. Gupta, G. Kylberg and I. M. Sintorn, Super- 15 K. Aso, J. Maebe, T. Yamamoto and S. Matsumura, Lattice resolution reconstruction of transmission electron micro- Tetragonality and Local Strain Depending on Shape of Gold scopy images using deep learning, Proc. Int. Symp. Biomed. Nanoparticles, Microsc. Microanal., 2019, 25, 2122–2123. Imaging, 2019, 548–551. 16 P. Trebbia, EELS elemental mapping with unconventional 32 S. Anada, Y. Nomura, T. Hirayama and K. Yamamoto, methods I. Theoretical basis: image analysis with multi- Simulation-Trained Sparse Coding for High-Precision variate statistics and entropy concepts, Ultramicroscopy, Phase Imaging in Low-Dose Electron Holography, Microsc. 1990, 34, 165–178. Microanal., 2020, 26, 429–438. 17 N. Bonnet, N. Brun and C. Colliex, Extracting information 33 S. Anada, Y. Nomura, T. Hirayama and K. Yamamoto, from sequences of spatially resolved EELS spectra using Sparse coding and dictionary learning for electron holo- multivariate statistical analysis, Ultramicroscopy, 1999, 77, gram denoising, Ultramicroscopy, 2019, 206, 112818. 97–112. 34 Y. Midoh and K. Nakamae, Accuracy improvement of phase 18 C. Tian, et al., Deep learning on image denoising: An estimation in electron holography using noise reduction overview, Neural Networks, 2020, 131, 251–275. methods, Microscopy, 2020, 69, 123–131. 19 S. V. Kalinin, et al., Lab on a beam-Big data and artificial 35 N. Schnitzer, S. H. Sung and R. Hovden, Maximal Resolu- intelligence in scanning transmission electron micro- tion from the Ronchigram: Human vs. Deep Learning, scopy, MRS Bull., 2019, 44, 565–575. Microsc. Microanal., 2019, 25, 160–161. 1458 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 36 C. Zhang, et al., Aberration Corrector Tuning with 52 O. Dyck, S. Jesse and S. V. Kalinin, A self-driving micro- Machine-Learning-Based Emittance Measurements and scope and the Atomic Forge, MRS Bull., 2019, 44, 669–670. Bayesian Optimization, Microsc. Microanal., 2021, 27, 810–812. 53 S. R. Spurgeon, et al., Towards data-driven next-generation 37 R. Sagawa, F. Uematsu, K. Aibara, T. Nakamichi and transmission electron microscopy, Nat. Mater., 2021, 20, S. Morishita, Aberration Measurement and Correction in 274–279. Scanning Transmission Electron Microscopy using Machine 54 GitHub – PyJEM/PyJEM. Learning, Microsc. Microanal., 2021, 27, 814–816. 55 Gatan Microscopy Suite (GMS), 2021. 38 M. Olszta, et al., An Automated Scanning Transmission 56 J. Dan, et al., A hierarchical active-learning framework for Electron Microscope Guided by Sparse Data Analytics, classifying structural motifs in atomic resolution microscopy, This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Microsc. Microanal., 2022, 1611–1621. arXiv, 2020, preprint, arXiv:2005.11488, DOI: 10.48550/ Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 39 K. Roccapriore, S. V. Kalinin and M. Ziatdinov, Physics arXiv.2005.11488. discovery in nanoplasmonic systems via autonomous experi- 57 H. I. Rasool, C. Ophus and A. Zettl, Atomic Defects in Two ments in Scanning Transmission Electron Microscopy, arXiv, Dimensional Materials, Adv. Mater., 2015, 27, 5771–5777. 2021, preprint, arXiv:2108.03290, DOI: 10.48550/arXiv.2108. 58 R. Kannan, et al., Deep data analysis via physically con- 03290. strained linear unmixing: universal framework, domain 40 E. Rotunno, et al., Alignment of electron optical beam examples, and a community-wide platform, Adv. Struct. shaping elements using a convolutional neural network, Chem. Imaging, 2018, 4, 6. Ultramicroscopy, 2021, 228, 113338. 59 M. Ziatdinov, et al., Deep Learning of Atomically Resolved 41 E. Rotunno, et al., Convolutional neural network as a tool Scanning Transmission Electron Microscopy Images: for automatic alignment of electron optical beam shaping Chemical Identification and Tracking Local Transforma- devices, Microsc. Microanal., 2021, 27, 822–824. tions, ACS Nano, 2017, 11, 12742–12752. 42 E. F. Rauch, et al., Automated nanocrystal orientation and 60 M. Ziatdinov, O. Dyck, S. Jesse and S. V. Kalinin, Deep phase mapping in the transmission electron microscope Learning for Atomically Resolved Imaging, Microsc. Micro- on the basis of precession electron diffraction, Z. Kristallogr., anal., 2018, 24, 60–61. 2010, 225, 103–109. 61 A. Maksov, et al., Deep learning analysis of defect and 43 Y. Jin, et al., Machine learning guided rapid focusing with phase evolution during electron beam-induced transfor- sensor-less aberration corrections, Opt. Express, 2018, mations in WS2, npj Comput. Mater., 2019, 5, 12. 26, 30162. 62 T. K. Patra, et al., Defect dynamics in 2-D MoS2 probed by 44 Y. Xu, et al., An improved method of measuring wavefront using machine learning, atomistic simulations, and high- aberration based on image with machine learning in free resolution microscopy, ACS Nano, 2018, 12, 8006–8016. space optical communication, Sensors, 2019, 19, 3665. 63 R. K. Vasudevan, et al., Investigating phase transitions 45 B. P. Cumming and M. Gu, Direct determination of aberra- from local crystallographic analysis based on statistical tion functions in microscopy by an artificial neural network, learning of atomic environments in 2D MoS2-ReS2, Appl. Opt. Express, 2020, 28, 14511. Phys. Rev., 2021, 8, 011409. 46 B. Huang, Z. Li and J. Li, An artificial intelligence atomic 64 Z. Maxim, S. Jesse, B. G. Sumpter, S. V. Kalinin and force microscope enabled by machine learning, Nanoscale, O. Dyck, Tracking atomic structure evolution during direc- 2018, 10, 21320–21326. ted electron beam induced Si-atom motion in graphene via 47 S. V. Kalinin, B. G. Sumpter and R. K. Archibald, Big-deep- deep machine learning, Nanotechnology, 2021, 32, 035703. smart data in imaging for guiding materials design, Nat. 65 O. Ronneberger, P. Fischer and T. Brox, U-Net: Convolu- Mater., 2015, 14, 973–980. tional Networks for Biomedical Image Segmentation. Med. 48 S. V. Kalinin, et al., Automated and Autonomous Experi- Image Comput. Comput. Interv. – MICCAI 2015. MICCAI ments in Electron and Scanning Probe Microscopy, ACS 2015, Lect. Notes Comput. Sci., 2015, 9351, 1–8. Nano, 2021, 15, 12604–12627. 66 J. Madsen, et al., A Deep Learning Approach to Identify 49 R. K. Vasudevan, et al., Autonomous Experiments in Local Structures in Atomic-Resolution Transmission Elec- Scanning Probe Microscopy and Spectroscopy: Choosing tron Microscopy Images, Adv. Theory Simulat., 2018, 1, Where to Explore Polarization Dynamics in Ferroelectrics, 1–12. ACS Nano, 2021, 15, 11253–11262. 67 R. Sadre, C. Ophus, A. Butko and G. H. Weber, Deep 50 A. Ghosh, B. G. Sumpter, O. Dyck, S. V. Kalinin and Learning Segmentation of Complex Features in Atomic- M. Ziatdinov, Ensemble learning-iterative training machine Resolution Phase-Contrast Transmission Electron Micro- learning for uncertainty quantification and automated scopy Images, Microsc. Microanal., 2021, 27, 804–814. experiment in atom-resolved microscopy, npj Comput. 68 P. Cho, A. Wood, K. Mahalingam and K. Eyink, Defect detec- Mater., 2021, 7, 100. tion in atomic resolution transmission electron microscopy 51 C. Ophus, H. I. Rasool, M. Linck, A. Zettl and J. Ciston, images using machine learning, Mathematics, 2021, 9, 1209. Automatic software correction of residual aberrations in 69 W. Lin, et al., Local crystallography analysis for atomically reconstructed HRTEM exit waves of crystalline samples, resolved scanning tunneling microscopy images, Nanotech- Adv. Struct. Chem. Imaging, 2016, 2, 15. nology, 2013, 24, 415707. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1459 View Article Online Review Nanoscale Horizons 70 M. Ziatdinov, U. Fuchs, J. H. G. Owen, J. N. Randall and Learning Discovery of Chemical and Physical Transforma- S. V. Kalinin, Robust multi-scale multi-feature deep learning tion Pathways from Imaging Data, arXiv, 2020, preprint, for atomic and defect identification in Scanning Tunneling arXiv:2010.09196, DOI: 10.48550/arXiv.2010.09196. Microscopy on H-Si(100) 2x1 surface, arXiv, 2020, preprint, 86 L. Vlcek, et al., Learning from Imperfections: Predicting arXiv:2002.04716, DOI: 10.48550/arXiv.2002.04716. Structure and Thermodynamics from Atomic Imaging of 71 G. Roberts, et al., DefectNet – A Deep Convolutional Neural Fluctuations, ACS Nano, 2019, 13, 718–727. Network for Semantic Segmentation of Crystallographic 87 M. P. Oxley, et al., Probing atomic-scale symmetry breaking Defects in Advanced Microscopy Images, Microsc. Micro- by rotationally invariant machine learning of multidimen- anal., 2019, 25, 164–165. sional electron scattering, npj Comput. Mater., 2021, 7, 65. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 72 C. Kunka, A. Shanker, E. Y. Chen, S. R. Kalidindi and 88 S. V. Kalinin, O. Dyck, S. Jesse and M. Ziatdinov, Exploring Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. R. Dingreville, Decoding defect statistics from diffractograms order parameters and dynamic processes in disordered via machine learning, npj Comput. Mater., 2021, 7, 67. systems via variational autoencoders, Sci. Adv., 2021, 73 J. Dan, X. Zhao and S. J. Pennycook, A machine perspective 7, eabd5084. of atomic defects in scanning transmission electron micro- 89 S. V. Kalinin, C. T. Nelson, M. Valleti, J. J. P. Peters, W. Dong, scopy, InfoMat, 2019, 1, 359–375. R. Beanland, X. Zhang, I. Takeuchi and M. Ziatdinov, 74 J. Yi, Z. Yuan and J. Peng, Adversarial-Prediction Guided Unsupervised learning of ferroic variants from atomically Multi-Task Adaptation for Semantic Segmentation of Elec- resolved STEM images, arXiv, 2021, preprint, arXiv:2101.06892, tron Microscopy Images, Proc. – Int. Symp. Biomed. Imaging, DOI: 10.48550/arXiv.2101.06892. 2020, 1205–1208. 90 W. S. McCulloch and W. Pitts, A logical calculus of the 75 Q. Li, et al., Quantification of flexoelectricity in PbTiO3/ ideas immanent in nervous activity, Bull. Math. Biol., 1990, SrTiO3 superlattice polar vortices using machine learning 52, 99–115. and phase-field modeling, Nat. Commun., 2017, 8, 1468. 91 Y. Lecun, Y. Bengio and G. Hinton, Deep learning, Nature, 76 D. Daniel and L. H. Sebastian, Learning the parts of objects 2015, 521, 436–444. by non-negative matrix factorization, Nature, 1999, 401, 92 J. M. Ede, Deep learning in electron microscopy, Mach. 788–791. Learn. Sci. Technol., 2021, 2, 011004. 77 B. R. Jany, A. Janas and F. Krok, Automatic microscopic 93 A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena- image analysis by moving window local Fourier Transform Martinez and J. Garcia-Rodriguez, A Review on Deep and Machine Learning, Micron, 2020, 130, 102800. Learning Techniques Applied to Semantic Segmentation, 78 B. H. Martineau, D. N. Johnstone, A. T. J. Van Helvoort, arXiv, 2017, preprint, arXiv:1704.06857, DOI: 10.48550/ P. A. Midgley and A. S. Eggeman, Unsupervised machine arXiv.1704.06857. learning applied to scanning precession electron diffraction 94 R. M. Patton, et al., 167-PFlops deep learning for electron data, Adv. Struct. Chem. Imaging, 2019, 5, 3, DOI: 10.1186/ microscopy: From learning physics to atomic manipula- s40679-019-0063-3. tion, in Proc. - Int. Conf. High Perform. Comput. Networking, 79 E. Winter, M. N-FINDR: an algorithm for fast spectral Storage, Anal. SC 2018, 2019, pp. 638–648, DOI: 10.1109/ endmember determination in hyperspectral data, Int. SC.2018.00053. Geosci. Remote Sens. Symp., 1999, 3753, 266–275. 95 Q. Luo, E. A. Holm and C. Wang, A transfer learning 80 M. Ziatdinov, et al., Causal analysis of competing atomistic approach for improved classification of carbon nanoma- mechanisms in ferroelectric materials from high- terials from TEM images, Nanoscale Adv., 2021, 3, 206–213. resolution scanning transmission electron microscopy 96 S. Yang, et al., Deep Learning-Assisted Quantification of data, npj Comput. Mater., 2020, 6, 127. Atomic Dopants and Defects in 2D Materials, Adv. Sci., 81 R. K. Vasudevan, M. Ziatdinov, S. Jesse and S. V. Kalinin, 2021, 2101099. Phases and Interfaces from Real Space Atomically Resolved 97 M. Ziatdinov, C. Nelson, R. K. Vasudevan, D. Y. Chen and Data: Physics-Based Deep Data Image Analysis, Nano Lett., S. V. Kalinin, Building ferroelectric from the bottom up: 2016, 16, 5574–5581. The machine learning analysis of the atomic-scale ferro- 82 P. Baldi, Autoencoders, Unsupervised Learning, and Deep electric distortions, Appl. Phys. Lett., 2019, 115, 052902. Architectures, ICML Unsupervised Transf. Learn., 2012, 98 M. Ziatdinov, A. Maksov and S. V. Kalinin, Learning sur- 37–49, DOI: 10.1561/2200000006. face molecular structures via machine vision, npj Comput. 83 W. H. Lopez Pinaya, S. Vieira, R. Garcia-Dias and A. Mechelli, Mater., 2017, 3, 31. Autoencoders, Mach. Learn. Methods Appl. Brain Disord., 99 M. Nord, P. E. Vullum, I. MacLaren, T. Tybell and 2019, 193–208, DOI: 10.1016/B978-0-12-815739-8.00011-0. R. Holmestad, Atomap: a new software tool for the automated 84 M. A. Ziatdinov and S. V. Kalinin, Robust Feature Disen- analysis of atomic resolution images using two-dimensional tanglement in Imaging Data via Joint Invariant Variational Gaussian fitting, Adv. Struct. Chem. Imaging, 2017, 3, 9. Autoencoders: from Cards to Atoms, arXiv, 2021, preprint, 100 F. Uesugi, et al., Non-negative matrix factorization for arXiv:2104.10180, DOI: 10.48550/arXiv.2104.10180. mining big data obtained using four-dimensional scanning 85 S. V. Kalinin, O. Dyck, A. Ghosh, Y. Liu, R. Proksch, transmission electron microscopy, Ultramicroscopy, 2021, B. G. Sumpter and M. Ziatdinov, Unsupervised Machine 221, 113168. 1460 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 101 M. Jacob, et al., Statistical Machine Learning and Com- 116 S. Ghosh, Y. S. G. Nashed, O. Cossairt and A. Katsaggelos, pressed Sensing Approaches for Analytical Electron Tomo- ADP: Automatic differentiation ptychography, in IEEE Int. graphy – Application to Phase Change Materials, Microsc. Conf. Comput. Photogr. ICCP 2018, 2018, pp. 1–10, DOI: Microanal., 2019, 25, 156–157. 10.1109/ICCPHOT.2018.8368470. 102 S. Kiyohara, M. Tsubaki, K. Liao and T. Mizoguchi, Quan- 117 W. Hoppe, Trace structure analysis, ptychography, phase titative estimation of properties from core- loss spectrum tomography, Ultramicroscopy, 1982, 10, 187–198. via neural network, J. Phys. Mater., 2019, 2, 024003. 118 B. A. R. Lupini, M. P. Oxley and S. V. Kalinin, Pushing the 103 R. S. Pennington, C. Coll, S. Estradé, F. Peiró and C. T. limits of electron ptychography, Science, 2018, 362, 399–400. Koch, Neural-network-based depth-resolved multiscale 119 M. Cao, et al., Machine Learning for Phase Retrieval from This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. structural optimization using density functional theory 4D-STEM Data, Microsc. Microanal., 2020, 26, 2020–2021. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. and electron diffraction data, Phys. Rev. B, 2018, 97, 024112. 120 M. Schloz, et al., Overcoming information reduced data 104 K. P. Kelley, et al., Fast Scanning Probe Microscopy via and experimentally uncertain parameters in ptychography Machine Learning: Non-Rectangular Scans with Com- with regularized optimization, Opt. Express, 2020, 28, 28306. pressed Sensing and Gaussian Process Optimization, 121 M. J. Cherukara, et al., AI-enabled high-resolution scan- Small, 2020, 16, 2002878. ning coherent diffraction imaging, Appl. Phys. Lett., 2020, 105 J. A. Aguiar, M. L. Gong, R. R. Unocic, T. Tasdizen and B. D. 117, 044103. Miller, Decoding crystallography from high-resolution 122 M. Schloz, J. Müller, T. Pekin, W. Van den Broek and C. Koch, electron imaging and diffraction datasets with deep Adaptive Scanning in Ptychography through Deep Reinforce- learning, Sci. Adv., 2019, 5, aaw1949. ment Learning, Microsc. Microanal., 2021, 27, 818–821. 106 R. K. Vasudevan, et al., Mapping mesoscopic phase evolu- 123 X. Huang, et al., Optimization of overlap uniformness for tion during E-beam induced transformations via deep ptychography, Opt. Express, 2014, 22, 12634. learning of atomically resolved images, npj Comput. Mater., 124 C. Ophus, Four-Dimensional Scanning Transmission 2018, 4, 30. Electron Microscopy (4D-STEM): From Scanning Nano- 107 R. K. Vasudevan, A. Ghosh, M. Ziatdinov and S. V. Kalinin, diffraction to Ptychography and Beyond, Microsc. Micro- Exploring Electron Beam Induced Atomic Assembly anal., 2019, 563–582, DOI: 10.1017/S1431927619000497. via Reinforcement Learning in a Molecular Dynamics 125 G. W. Paterson, et al., Fast Pixelated Detectors in Scanning Environment, Nanotechnology, 2021, 33, 115301. Transmission Electron Microscopy. Part II: Post-Acquisition 108 R. S. Pennington, W. Van den Broek and C. T. Koch, Third- Data Processing, Visualization, and Structural Characteriza- dimension information retrieval from a single convergent- tion, Microsc. Microanal., 2020, 26, 944–963. beam transmission electron diffraction pattern using an 126 M. Nord, et al., Fast Pixelated Detectors in Scanning Transmis- artificial neural network, Phys. Rev. B: Condens. Matter Mater. sion Electron Microscopy. Part I: Data Acquisition, Live Proces- Phys., 2014, 89, 205409. sing, and Storage, Microsc. Microanal., 2020, 26, 653–666. 109 J. P. Horwath, Understanding important features of 127 M. Nord, et al., Developing Rapid and Advanced Visualisation deep learning models for segmentation of high-resolution of Magnetic Structures Using 2-D Pixelated STEM Detectors, transmission electron microscopy images, npj Comput. Microsc. Microanal., 2016, 22, 530–531. Mater., 2020, 6, 1–9, DOI: 10.1038/s41524-020-00363-x. 128 M. Nord, et al., Strain Anisotropy and Magnetic Domains 110 G. Benton, M. Finzi, P. Izmailov and A. G. Wilson, Learning in Embedded Nanomagnets, Small, 2019, 15, 1904738. invariances in neural networks from training data, Adv. 129 G. Correa and D. Muller, Machine Learning for Sub-pixel Neural Inf. Process. Syst., 2020, 33, 17605–17616. Super-resolution in Direct Electron Detectors, Microsc. 111 M. Ziatdinov, A. Maksov and S. V. Kalinin, Deep data Microanal., 2020, 26, 1932–1934. analytics in structural and functional imaging of nanoscale 130 C. Shi, M. Cao, D. Muller and Y. Han, Rapid and Semi- materials. Springer Series in Materials Science, Springer Automated Analysis of 4D-STEM data via Unsupervised International Publishing, 2018, vol. 280. Learning, Microsc. Microanal., 2021, 27, 58–59. 112 M. Finzi, S. Stanton, P. Izmailov and A. G. Wilson, Generalizing 131 P. Cueva, E. Padget and D. A. Muller, A Natural Basis for convolutional neural networks for equivariance to lie groups Unsupervised Machine Learning on Scanning Diffraction on arbitrary continuous data, in 37th Int. Conf. Mach. Learn. Data, Microsc. Microanal., 2018, 24, 490–491. ICML 2020 PartF16814, 2020, pp. 3146–3157. 132 F. I. Allen, et al., Fast Grain Mapping with Sub-Nanometer 113 T. Zhou, M. Cherukara and C. Phatak, Differential program- Resolution Using 4D-STEM with Grain Classification by ming enabled functional imaging with Lorentz transmission Principal Component Analysis and Non-Negative Matrix electron microscopy, npj Comput. Mater., 2021, 7, 141. Factorization, Microsc. Microanal., 2021, 27, 794–803. 114 Z. Chen, et al., Electron ptychography achieves atomic- 133 X. Li, et al., Manifold learning of four-dimensional scanning resolution limits set by lattice vibrations, Science, 2021, transmission electron microscopy, npj Comput. Mater., 2019, 372, 826–831. 5, 5. 115 S. Kandel, et al., Using automatic differentiation as a 134 X. Li, et al., Unsupervised Machine Learning to Distill general framework for ptychographic reconstruction, Opt. Structural-Property Insights from 4D-STEM, Microsc. Micro- Express, 2019, 27, 18653–18672. anal., 2021, 25, 2016–2017. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1461 View Article Online Review Nanoscale Horizons 135 C. Zhang, R. Han, A. R. Zhang and P. M. Voyles, Denoising 150 J. M. Muñoz-Ocaña, et al., Optimization of STEM-HAADF atomic resolution 4D scanning transmission electron Electron Tomography Reconstructions by Parameter Selection microscopy data with tensor singular value decomposition, in Compressed Sensing Total Variation Minimization-Based Ultramicroscopy, 2020, 219, 113123. Algorithms, Part. Part. Syst. Charact., 2020, 37, 2000070. 136 A. Nalin Mehta, et al., Unravelling stacking order in 151 A. Rakowski, J. Merham, L. Li, P. Baldi and J. Patterson, epitaxial bilayer MX2 using 4D-STEM with unsupervised Learning Frame Interpolation for Tilt Series Tomography, learning, Nanotechnology, 2020, 31, 445702. Microsc. Microanal., 2020, 928–930, DOI: 10.1017/ 137 C. Zhang, J. Feng, L. R. DaCosta and P. M. Voyles, Atomic S1431927620016360. resolution convergent beam electron diffraction analysis 152 Y. Zhao, et al., Five-second STEM dislocation tomography This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. using convolutional neural networks, Ultramicroscopy, for 300 nm thick specimen assisted by deep-learning- Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 2020, 210, 112921. based noise filtering, Sci. Rep., 2021, 11, 20720. 138 M. P. Oxley, et al., Deep learning of interface structures 153 A. A. Hendriksen, D. M. Pelt and K. J. Batenburg, Noise2- from simulated 4D STEM data: cation intermixing vs. Inverse: Self-supervised deep convolutional denoising for roughening, Mach. Learn.: Sci. Technol., 2020, 1, 04LT01. linear inverse problems in imaging, IEEE Transactions on 139 S. Van Aert, A. J. Den Dekker, A. Van Den Bos, D. Van Dyck and Computational Imaging, 2020, 6, 1320–1335. J. H. Chen, Maximum likelihood estimation of structure para- 154 E. Bladt, D. M. Pelt, S. Bals and K. J. Batenburg, Electron meters from high resolution electron microscopy images. Part tomography based on highly limited data using a neural II: A practical example, Ultramicroscopy, 2005, 104, 107–125. network reconstruction technique, Ultramicroscopy, 2015, 140 L. C. Gontard, R. Schierholz, S. Yu, J. Cintas and R. E. 158, 81–88. Dunin-Borkowski, Photogrammetry of the three-dimensional 155 D. M. Pelt and K. J. Batenburg, Fast Tomographic Recon- shape and texture of a nanoscale particle using scanning struction from Limited Data Using Artificial Neural electron microscopy and free software, Ultramicroscopy, 2016, Networks, IEEE Trans. Image Process, 2013, 22, 5238–5251. 169, 80–88. 156 Q. Yang, et al., Low-Dose CT Image Denoising Using a 141 J. M. Thomas, R. Leary, P. A. Midgley and D. J. Holland, Generative Adversarial Network With Wasserstein Distance A new approach to the investigation of nanoparticles: and Perceptual Loss, IEEE Trans. Med. Imaging, 2018, 37, Electron tomography with compressed sensing, J. Colloid 1348–1357. Interface Sci., 2013, 392, 7–14. 157 A. Stevens, H. Yang, L. Carin, I. Arslan and N. D. Browning, 142 R. Leary, Z. Saghi, P. A. Midgley and D. J. Holland, Com- The potential for bayesian compressive sensing to signifi- pressed sensing electron tomography, Ultramicroscopy, cantly reduce electron dose in high-resolution STEM images, 2013, 131, 70–91. Microscopy, 2014, 63, 41–51. 143 L. Staniewicz and P. A. Midgley, Machine learning as a tool 158 N. Browning, R. Klie, A. Barker, A. Stevens and C. Buurma, for classifying electron tomographic reconstructions, Adv. The Potential Benefits of Compressed Sensing and Struct. Chem. Imaging, 2015, 1, 9. Machine Learning for Advanced Imaging and Spectroscopy 144 A. Béché, B. Goris, B. Freitag and J. Verbeeck, Development in the Electron Microscope, Microsc. Microanal., 2020, 26, of a fast electromagnetic beam blanker for compressed 2458–2460. sensing in scanning transmission electron microscopy, 159 D. Mucke-Herzberg, et al., Practical Implementation of Appl. Phys. Lett., 2016, 108, 093103. Compressive Sensing for High Resolution STEM, Microsc. 145 L. Kovarik, A. Stevens, A. Liyu and N. D. Browning, Imple- Microanal., 2016, 22, 558–559. menting an accurate and rapid sparse sampling approach 160 X. Li, O. Dyck, S. V. Kalinin and S. Jesse, Compressed for low-dose atomic resolution STEM imaging, Appl. Phys. sensing of Scanning Transmission Electron Microscopy Lett., 2016, 109, 164102. (STEM) with nonrectangular scans, Microsc. Microanal., 146 Z. Saghi, et al., Compressed sensing electron tomography 2018, 24, 623–633. of needle-shaped biological specimens - Potential for 161 X. Li, O. Dyck, S. V. Kalinin and S. Jesse, Compressive Sensing improved reconstruction fidelity with reduced dose, Ultra- on Diverse STEM Scans: Real-time Feedback, Low-dose and microscopy, 2016, 160, 230–238. Dynamic Range, Microsc. Microanal., 2019, 25, 1688–1689. 147 B. Goris, W. Van den Broek, K. J. Batenburg, H. Heidari 162 J. M. Ede and R. Beanland, Partial Scanning Transmission Mezerji and S. Bals, Electron tomography based on a total Electron Microscopy with Deep Learning, Sci. Rep., 2020, variation minimization reconstruction technique, Ultra- 10, 8332. microscopy, 2012, 113, 120–130. 163 J. M. Ede, Adaptive partial scanning transmission electron 148 Z. Saghi, et al., Three-dimensional morphology of iron microscopy with reinforcement learning, Mach. Learn. Sci. oxide nanoparticles with reactive concave surfaces. A com- Technol., 2021, 2, 045011. pressed sensing-electron tomography (CS-ET) approach, 164 S. Zheng, C. Wang, X. Yuan and H. L. Xin, Super-compression Nano Lett., 2011, 11, 4666–4673. of large electron microscopy time series by deep compressive 149 M. López-Haro, et al., A Macroscopically Relevant sensing learning, Patterns, 2021, 2, 100292. 3D-Metrology Approach for Nanocatalysis Research, Part. 165 M. D. Sangid, Coupling in situ experiments and modeling – Part. Syst. Charact., 2018, 35, 1700343. Opportunities for data fusion, machine learning, and 1462 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review discovery of emergent behavior, Curr. Opin. Solid State and Health Informatics, 2020, 1–9, DOI: 10.1145/ Mater. Sci., 2020, 24, 100797. 3388440.3414711. 166 H. Zheng, Y. S. Meng and Y. Zhu, Frontiers of in situ 181 T. Bendory, A. Bartesaghi and A. Singer, Single-Particle Cryo- electron microscopy, MRS Bull., 2015, 40, 12–18. Electron Microscopy: Mathematical Theory, Computational 167 Y. Luo, N. Zaluzec, M. Cherukara, X. Wu and S. Chen, Real- Challenges, and Opportunities, IEEE Signal Process. Mag., Time Image Registration via A Deep Leaning Approach for 2020, 58–76. Correlative X-ray and Electron Microscopy, Microsc. Micro- 182 D. Si, et al., Artificial intelligence advances for de novo anal., 2021, 27, 302–304. molecular structure modeling in cryo-electron microscopy, 168 K. Higgins, et al., Exploration of Electrochemical Reactions Wiley Interdiscip. Rev.: Comput. Mol. Sci., 2021, 12, e1542, This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. at Organic–Inorganic Halide Perovskite Interfaces via DOI: 10.1002/wcms.1542. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Machine Learning in In Situ Time-of-Flight Secondary Ion 183 N. Kumar, et al., CryoDiscovery TM: A Machine Learning Mass Spectrometry, Adv. Funct. Mater., 2020, 30, 2001995. Platform for Automated Cryo-electron Microscopy Particle 169 X. Wang, et al., AutoDetect-mNP: An Unsupervised Classification, Microsc. Microanal., 2020, 26, 2308–2310. Machine Learning Algorithm for Automated Analysis of 184 T. Slater, et al., Automating 3D Imaging of Inorganic Transmission Electron Microscope Images of Metal Nano- Nanoparticles, Microsc. Microanal., 2021, 27, 2864–2866. particles, JACS Au, 2021, 1, 316–327. 185 F. L. Kyrilis, J. Belapure and P. L. Kastritis, Detecting 170 N. M. Schneider, J. H. Park, M. M. Norton, F. M. Ross and Protein Communities in Native Cell Extracts by Machine H. H. Bau, Automated analysis of evolving interfaces dur- Learning: A Structural Biologist’s Perspective, Front. Mol. ing in situ electron microscopy, Adv. Struct. Chem. Imaging, Biosci., 2021, 8, 660542. 2016, 2, 2. 186 J. Merham, A. Rakowski and J. Patterson, Particle Picking 171 Y. Qian, J. Z. Huang, X. Li and Y. Ding, Robust nano- in Cryo-TEM Images Using Machine Learning, Microsc. particles detection from noisy background by fusing com- Microanal., 2020, 26, 2102–2103. plementary image information, IEEE Trans. Image Process., 187 M. Weber, et al., Automatic identification of crossovers in 2016, 25, 5713–5726. cryo-EM images of murine amyloid protein A fibrils with 172 Y. Qian, J. Z. Huang and Y. Ding, Identifying multi-stage machine learning, J. Microsc., 2020, 277, 12–22. nanocrystal growth using in situ TEM video data, IISE 188 A. S. Morgunov, K. L. Saar, M. Vendruscolo and T. P. J. Trans., 2017, 49, 532–543. Knowles, New Frontiers for Machine Learning in Protein 173 A. A. Ezzat and M. Bedewy, Machine learning for revealing Science, J. Mol. Biol., 2021, 433, 167232. spatial dependence among nanoparticles: Understanding 189 K. Yonekura, S. Maki-Yonekura, H. Naitow, T. Hamaguchi catalyst film dewetting via Gibbs point process models, and K. Takaba, Machine learning-based real-time object J. Phys. Chem. C, 2020, 124, 27479–27494. locator/evaluator for cryo-EM data collection, Commun. 174 L. Yao, Z. Ou, B. Luo, C. Xu and Q. Chen, Machine Biol., 2021, 4, 1044. Learning to Reveal Nanoparticle Dynamics from Liquid- 190 E. Alnabati and D. Kihara, Advances in structure modeling Phase TEM Videos, ACS Cent. Sci., 2020, 6, 1421–1430. methods for cryo-electron microscopy maps, Molecules, 175 K. Faraz, T. Grenier, C. Ducottet and T. Epicier, A Machine 2019, 25, 82. Learning pipeline to track the dynamics of a population of 191 D. Si, S. Ji and K. Nasr, Al & He, J. A machine learning nanoparticles during in situ Environmental Transmission approach for the identification of protein secondary struc- Electron Microscopy in gases, Microsc. Microanal., 2021, ture elements from electron cryo-microscopy density 27, 2236–2237. maps, Biopolymers, 2012, 97, 698–708. 176 X. Li, et al., Statistical learning of governing equations of 192 R. Langlois, J. Pallesen and J. Frank, Reference-free particle dynamics from in-situ electron microscopy imaging data, selection enhanced with semi-supervised machine learn- Mater. Des., 2020, 195, 108973. ing for cryo-electron microscopy, J. Struct. Biol., 2011, 175, 177 Y. Zhu, Q. Ouyang and Y. Mao, A deep convolutional 353–361. neural network approach to single-particle recognition in 193 J. Chen, Advanced Electron Microscopy of Nanophased cryo-electron microscopy, BMC Bioinf., 2017, 18, 348. Synthetic Polymers and Soft Complexes for Energy and 178 R. Sanchez-Garcia, J. Segura, D. Maluenda, J. M. Carazo Medicine Applications, Nanomaterials, 2021, 11, 2405. and C. O. S. Sorzano, Deep Consensus, a deep learning- 194 M. P. Prange, M. P. Oxley, S. J. Pennycook and S. T. based approach for particle pruning in cryo-electron Pantelides, Simulation of STEM-EELS Including Diffrac- microscopy, IUCrJ, 2018, 5, 854–865. tion and Solid-State Effects I: Mixed Dynamic Form Factor 179 T. Hey, K. Butler, S. Jackson and J. Thiyagalingam, beyond the Dipole Approximation, Microsc. Microanal., Machine learning and big scientific data, Philos. Trans. R. 2021, 17, 808–809. Soc., A, 2020, 378, 20190054. 195 M. P. Oxley, et al., Simulation of Probe Position-Dependent 180 L. Chen, R. Jebril and K. Al Nasr, Segmentation-based Electron Energy-Loss Fine Structure, Microsc. Microanal., Feature Extraction for Cryo-Electron Microscopy at 2014, 20, 784–797. Medium Resolution, Proceedings of the 11th ACM Inter- 196 A. Eljarrat, X. Sastre, F. Peiró and S. Estradé, Density national Conference on Bioinformatics, Computational Biology Functional Theory Modeling of Low-Loss Electron This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1463 View Article Online Review Nanoscale Horizons Energy-Loss Spectroscopy in Wurtzite III-Nitride Ternary 214 P. Y. Tang, et al., Boosting Photoelectrochemical Water Alloys, Microsc. Microanal., 2016, 22, 706–716. Oxidation of Hematite in Acidic Electrolytes by Surface 197 F. J. G. De Abajo, Optical excitations in electron micro- State Modification, Adv. Energy Mater., 2019, 9, 1901836. scopy, Rev. Mod. Phys., 2010, 82, 209–275. 215 M. De La Mata, et al., The Role of Polarity in Nonplanar 198 S. Chen, et al., Electron energy loss spectroscopy and Semiconductor Nanostructures, Nano Lett., 2019, 19, ab initio investigation of iron oxide nanomaterials grown 3396–3408. by a hydrothermal process, 2009, 1–10, DOI: 10.1103/ 216 R. Zamani and J. Arbiol, Understanding semiconductor PhysRevB.79.104103. nanostructures via advanced electron microscopy and 199 A. Gloter, A. Douiri, M. Tencé and C. Colliex, Improving spectroscopy, Nanotechnology, 2019, 30, 262001. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. energy resolution of EELS spectra: an alternative to the 217 N. Mevenkamp, et al., Multi-modal and multi-scale non-local Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. monochromator solution, Ultramicroscopy, 2003, 96, means method to analyze spectroscopic datasets, Ultramicro- 385–400. scopy, 2019, 112877, DOI: 10.1016/j.ultramic.2019.112877. 200 O. L. Krivanek, et al., Progress in ultrahigh energy resolu- 218 S. Jesse and S. V. Kalinin, Principal component and spatial tion EELS, Ultramicroscopy, 2019, 203, 60–67. correlation analysis of spectroscopic-imaging data in scan- 201 P. Cueva, R. Hovden, J. A. Mundy, H. L. Xin and D. A. ning probe microscopy, Nanotechnology, 2009, 085714. Muller, New Approaches to Data Processing for Atomic 219 K. Roccapriore, Z. Gai, B. Sumpter, M. Yoon and S. V. Resolution EELS, Microsc. Microanal., 2012, 18, 970–971. Kalinin, Spectral Classification of Structurally Organized 202 R. F. Egerton, Electron-Loss Spectroscopy in the Electron Adatom Configurations, Microsc. Microanal., 2021, 26, Microscope, Springer Science & Business Media, 2011. 2988–2989. 203 K. Kimoto and Y. Matsui, Software techniques for EELS to 220 H. Chang and D. Yeung, Robust locally linear embedding, realize about 0. 3 eV energy resolution using 300 kV Pattern Recognit., 2006, 39, 1053–1065. FEG-TEM, J. Microsc., 2002, 208, 224–228. 221 B. J. Rodriguez, et al., Dynamic and Spectroscopic Modes 204 S. Martı́-Sánchez, et al., Sub-nanometer mapping of strain- and Multivariate Data Analysis in Piezoresponse Force induced band structure variations in planar nanowire Microscopy, Scanning probe microscopy of functional materi- core–shell heterostructures, Nat. Commun., 2022, 13, 4089. als, Springer, New York, NY, 2010, pp. 491–528, ISBN: 205 L. Jones, R. B. S. Lozano-perez, K. Baba-kishi and P. D. 201010.1007/978-1-4419-7167-8. Nellist, Improving the SNR of Atomic Resolution STEM 222 A. A. Varambhia, L. Jones and A. London, Determining EELS & EDX Mapping while Reducing Beam-damage by EDS and EELS partial cross-sections from multiple calibra- using Non-rigid Spectrum-image Averaging, Microsc. Micro- tion standards to accurately quantify bi-metallic nano- anal., 2021, 21, 1215–1216. particles using STEM, Micron, 2018, 113, 69–82. 206 J. S. Jeong and K. A. Mkhoyan, Improving Signal-to-Noise 223 M. C. Spadaro, et al., Rotated domains in selective area Ratio in Scanning Transmission Electron Microscopy epitaxy grown Zn3P2: formation mechanism and function- Energy-Dispersive X-Ray (STEM-EDX) Spectrum Images ality, Nanoscale, 2021, 13, 18441–18450. Using Single-Atomic-Column Cross-Correlation Averaging, 224 A. Genç, et al., Tuning the Plasmonic Response up: Hollow Microsc. Microanal., 2016, 22, 536–543. Cuboid Metal Nanostructures, ACS Photonics, 2016, 3, 207 L. Jones, et al., Managing dose-, damage- and data-rates in 770–779. multi-frame spectrum-imaging, Microscopy, 2018, 98–113, 225 J. Spiegelberg and J. Rusz, Can we use PCA to detect small DOI: 10.1093/jmicro/dfx125. signals in noisy data?, Ultramicroscopy, 2017, 172, 40–46. 208 K. Sader, et al., Smart acquisition EELS, Ultramicroscopy, 226 S. Lichtert and J. Verbeeck, Statistical consequences of 2010, 110, 998–1003. applying a PCA noise filter on EELS spectrum images, 209 P. Torruella, et al., Assessing Oxygen Vacancies in Bismuth Ultramicroscopy, 2013, 125, 35–42. Oxide Through EELS Measurements and DFT Simulations 227 P. Potapov, Why Principal Component Analysis of STEM Assessing Oxygen Vacancies in Bismuth Oxide through spectrum-images results in ‘‘ abstract’’, uninterpretable EELS Measurements and DFT Simulations, J. Phys. Chem. C, loadings?, Ultramicroscopy, 2016, 160, 197–212. 2017, 121, 24809–24815, DOI: 10.1021/acs.jpcc.7b06310. 228 P. Potapov, P. Longo and E. Okunishi, Enhancement of 210 I. Arslan, S. Ogut, P. D. Nellist and N. D. Browning, noisy EDX HRSTEM spectrum-images by combination of Comparison of simulation methods for electronic struc- filtering and PCA, Micron, 2017, 96, 29–37. ture calculations with experimental electron energy-loss 229 P. Cueva, R. Hovden, J. A. Mundy, H. L. Xin and spectra, Micron, 2003, 34, 255–260. D. A. Muller, Data Processing for Atomic Resolution Elec- 211 G. Kothleitner, et al., Quantitative Elemental Mapping at Atomic tron Energy Loss Spectroscopy, Microsc. Microanal., 2012, Resolution Using X-Ray, Spectroscopy, 2014, 112, 085501. 667–675. 212 A. Genç, et al., Hollow metal nanostructures for enhanced 230 P. Potapov and A. Lubk, Optimal principal component plasmonics: Synthesis, local plasmonic properties and analysis of STEM XEDS spectrum images, Adv. Struct. applications, Nanophotonics, 2017, 6, 193–213. Chem. Imaging, 2019, 5, 4. 213 D. Jirovec, et al., A singlet-triplet hole spin qubit in planar 231 A. Tharwat, Independent component analysis: An intro- Ge, Nat. Mater., 2021, 20, 1106–1112. duction, Appl. Comput. Informat., 2018, 17, 222–249. 1464 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 232 J. M. P. Nascimento and J. M. B. Dias, Vertex component 248 F. de la Peña, et al., Mapping titanium and tin oxide analysis: A fast algorithm to unmix hyperspectral data, phases using EELS: An application of independent com- IEEE Trans. Geosci. Remote Sens., 2005, 43, 898–910. ponent analysis, Ultramicroscopy, 2011, 111, 169–176. 233 T. Blum, et al., Machine Learning Method Reveals Hidden 249 D. Rossouw, B. R. Knappett, A. E. H. Wheatley and P. A. Strong Metal-Support Interaction in Microscopy Datasets, Midgley, A New Method for Determining the Composition Small Methods, 2021, 5, 2100035. of Core – Shell Nanoparticles via Dual-EDX + EELS Spec- 234 T. Blum, J. Graves, M. Zachman, R. Kannan and X. Pan, trum Imaging, Part. Part. Syst. Charact., 2016, 33, 749–755. Machine Learning for Challenging EELS and EDS Spectral 250 N. Dobigeon and N. Brun, Spectral mixture analysis of Decomposition, Microsc. Microanal., 2021, 25, 180–181. EELS spectrum-images, Ultramicroscopy, 2012, 120, 25–34. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 235 Y. Suzuki, H. Hino, M. Kotsugi and K. Ono, Automated 251 S. V. Kalinin, A. R. Lupini, R. K. Vasudevan and M. Ziatdinov, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. estimation of materials parameter from X-ray absorption Gaussian process analysis of electron energy loss spectroscopy and electron energy-loss spectra with similarity measures, npj data: multivariate reconstruction and kernel control, npj Com- Comput. Mater., 2019, 1–7, DOI: 10.1038/s41524-019-0176-1. put. Mater., 2021, 7, 154, DOI: 10.1038/s41524-021-00611-8. 236 P. Crozier, J. Vincent, K. Venkatraman, Y. Wang and 252 M. Chang, R. Cai, C. Chen and S. Lo, Development of S. Yang, Probing Properties of Nanomaterials with Clustering Algorithm Applied for the EELS Analysis of Advanced Electron Energy-Loss Spectroscopy, Microsc. Advanced Devices, Microsc. Microanal., 2020, 26, 2112–2114. Microanal., 2021, 27, 872–874. 253 N. Creange, et al., Propagation of priors for more accurate 237 H. Chen, F. Nabiei, J. Badro, D. T. L. Alexander and and efficient spectroscopic functional fits and their appli- C. Hébert, STEM EDS/EELS for Phase Analysis of Deep- cation to ferroelectric hysteresis, Mach. Learn. Sci. Technol., Mantle Rock Assemblages Supported by, Mach. Learn., 2021, 2, 045002. 2021, 25, 2474–2475. 254 G. Yang, et al., Distilling nanoscale heterogeneity of amor- 238 A. Heimbrook, K. Higgins, S. V. Kalinin and M. Ahmadi, phous silicon using tip-enhanced Raman spectroscopy (TERS) Exploring the physics of cesium lead halide perovskite via multiresolution manifold learning, Nat. Commun., 2021, quantum dots via Bayesian inference of the photolumines- 12, 578. cence spectra in automated experiment, Nanophotonics, 255 M. Pfannm, et al., Visualizing a Homogeneous Blend in 2021, 10, 1977–1989. Bulk Heterojunction Polymer Solar Cells by Analytical 239 R. D. Leapman and C. R. Swyt, Separation of Overlapping Electron Microscopy, Nano Lett., 2011, 11, 3099–3107. Core Edges in Electron Energy Loss Spectra by Multiple- 256 L. Lajaunie, et al., Fast Automated Phase Differentiation in Least-Squares Fitting, Ultramicroscopy, 1988, 26, 393–404. Industrial Stainless Steel by Combining Low-Loss EELS 240 K. P. Kelley, et al., Tensor factorization for elucidating Experiments with Machine Learning-based Algorithms, mechanisms of piezoresponse relaxation via dynamic Microsc. Microanal., 2021, 27, 34–36. Piezoresponse Force Spectroscopy, npj Comput. Mater., 257 S. Kiyohara, T. Miyata, K. Tsuda and T. Mizoguchi, Data- 2020, 6, 113, DOI: 10.1038/s41524-020-00384-6. driven approach for the prediction and interpretation of 241 K. M. Roccapriore, et al., Revealing the Chemical Bonding core-electron loss spectroscopy, Sci. Rep., 2018, 8, 13548, in Adatom Arrays via Machine Learning of Hyperspectral, DOI: 10.1038/s41598-018-30994-6. ACS Nano, 2021, 15, 11806–11816. 258 T. Mizoguchi and S. Kiyohara, Machine learning approaches 242 R. Hovden, P. Cueva, J. A. Mundy and D. A. Muller, The for ELNES/XANES, Microscopy, 2020, 69, 92–109. Open-Source Cornell Spectrum Imager, Microsc. Today, 259 P. Torruella, et al., Clustering analysis strategies for electron 2013, 21, 40–44. energy loss spectroscopy (EELS), Ultramicroscopy, 2018, 185, 243 M. Ziatdinov, D. Kim, S. Neumayer and S. V. Kalinin, 42–48. Imaging mechanism for hyperspectral scanning probe 260 J. Blanco-Portals, et al., WhatEELS. A python-based inter- microscopy via Gaussian process modelling, npj Comput. active software solution for ELNES analysis combining Mater., 2020, 6, 21, DOI: 10.1038/s41524-020-0289-6. clustering and NLLS, Ultramicroscopy, 2022, 232, 113403. 244 L. Yedra, et al., Oxide Wizard: An EELS Application to 261 J. Ryu, et al., Dimensionality reduction and unsupervised Characterize the White Lines of Transition Metal Edges, clustering for EELS-SI, Ultramicroscopy, 2021, 231, 113314. Microsc. Microanal., 2014, 20, 698–705. 262 S. V. Kalinin, et al., Separating Physically Distinct Mechan- 245 M. Shiga, et al., Sparse modeling of EELS and EDX spectral isms in Complex Infrared Plasmonic Nanostructures via imaging data by nonnegative matrix factorization, Ultra- Machine Learning Enhanced Electron Energy Loss microscopy, 2016, 170, 43–59. Spectroscopy, Adv. Opt. Mater., 2021, 9, 2001808. 246 S. Muto and M. Shiga, Application of machine learning 263 J. Hachtel, N. Borodinov, K. Roccapriore, S. H. Cho and techniques to electron microscopic/spectroscopic image P. Banerjee, Beyond NMF: Advanced Signal Processing data analysis, Microscopy, 2020, 69, 110–122. and Machine Learning Methodologies for Hyperspectral 247 N. Bonnet and D. Nuzillard, Independent component Analysis in EELS, Microsc. Microanal., 2021, 27, 322–324. analysis: A new possibility for analysing series of 264 M. Oxley, M. Ziatdinov and S. Kalinin, Denoising STEM electron energy loss spectra, Ultramicroscopy, 2005, 102, Electron Energy Loss Spectra using Convolutional Auto- 327–337. encoders, Microsc. Microanal., 2021, 27, 1180–1182. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1465 View Article Online Review Nanoscale Horizons 265 C. M. Pate, J. L. Hart and M. L. Taheri, RapidEELS: structure, Phys. Rev. B: Condens. Matter Mater. Phys., 1998, machine learning for denoising and classification in rapid 58, 7565–7576. acquisition electron energy loss spectroscopy, Sci. Rep., 282 M. P. Oxley, et al., Simulation of STEM-EELS Including 2021, 11, 19515. Diffraction and Solid-State Effects II: Adding the Experi- 266 P. Ewels, T. Sikora, V. Serin, C. P. Ewels and L. Lajaunie, A ment, Microsc. Microanal., 2011, 17, 810–811. Complete Overhaul of the Electron Energy-Loss Spectro- 283 M. P. Prange, M. P. Oxley, M. Varela, S. J. Pennycook and scopy and X-Ray Absorption Spectroscopy Database: eelsdb. S. T. Pantelides, Simulation of Spatially Resolved Electron eu, Microsc. Microanal., 2017, 717–724, DOI: 10.1017/ Energy Loss Near-Edge Structure for Scanning Transmission S1431927616000179. Electron Microscopy, Phys. Rev. Lett., 2012, 109, 246101. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 267 H. L. Xin, et al., One Million EEL Spectra Acquisition with 284 L. J. Allen, S. D. Findlay, M. P. Oxley and C. J. Rossouw, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Aberration-Corrected STEM: 2-D Chemical Investigation of Lattice-resolution contrast from a focused coherent elec- a Statistically Significant Ensemble of Nanocatalysts, tron probe. Part I, Ultramicroscopy, 2003, 96, 47–63. Microsc. Microanal., 2010, 16, 2009–2010. 285 S. D. Findlay, L. J. Allen, M. P. Oxley and C. J. Rossouw, 268 E. J. Kirkland, Advanced Computing in Electron Microscopy. Lattice-resolution contrast from a focused coherent electron Advanced Computing in Electron Microscopy, 1998, DOI: probe. Part II, Ultramicroscopy, 2003, 96, 65–81. 10.1007/978-1-4757-4406-4. 286 T. Morimura and M. Hasaka, Bloch-wave-based STEM 269 F. J. Garcı́a de Abajo and A. Howie, Retarded field calculation image simulation with layer-by-layer representation, Ultra- of electron energy loss in inhomogeneous dielectrics, Phys. microscopy, 2009, 109, 1203–1209. Rev. B: Condens. Matter Mater. Phys., 2002, 65, 115418. 287 H. G. Brown, S. D. Findlay, L. J. Allen, J. Ciston and 270 L. Kiewidt and M. Karamehmedovi, The Generalized Multi- C. Ophus, Rapid Simulation of Elemental Maps in Core- pole Technique for the Simulation of Low-Loss Electron Energy Loss Electron Energy Loss Spectroscopy, Microsc. Microanal., Loss Spectroscopy, Springer, Cham, 2018, pp. 147–167, DOI: 2019, 25, 574–575. 10.1007/978-3-319-74890-0. 288 F. Eggert, EDX-spectra simulation in electron probe micro- 271 M. P. Oxley and S. J. Pennycook, Image simulation for analysis. Optimization of excitation conditions and detection electron energy loss spectroscopy, Micron, 2008, 39, 676–684. limits, Microchim. Acta, 2006, 155, 129–136. 272 A. Mitsutake, Y. Sugita and Y. Okamoto, Replica-exchange 289 NIST, DTSA-II Microscopium, 2021, available at: https:// multicanonical and multicanonical replica-exchange cstl.nist.gov/div837/837.02/epq/dtsa2/. Monte Carlo simulations of peptides. I. Formulation and 290 M. Chatzidakis and G. A. Botton, Towards calibration-invariant benchmark test, J. Chem. Phys., 2003, 118, 6664–6675. spectroscopy using deep learning, Sci. Rep., 2019, 9, 2126. 273 A. Desalvo, R. Rosa, A. Armigliato and A. Parisini, Analysis 291 A. Scheinker and R. Pokharel, Adaptive 3D convolutional of light elements in superposed layers by Monte Carlo neural network-based reconstruction method for 3D coherent simulation of EELS spectra, Mikrochim. Acta, 1994, 114–115, diffraction imaging, J. Appl. Phys., 2020, 128, 184901. 267–275. 292 A. Y. Borisevich, S. V. Kalinin, A. R. Lupini, S. Jesse and 274 M. Attarian Shandiz, F. Salvat and R. Gauvin, Fine Struc- H. J. Chang, Using Neural Network Algorithms for Compo- ture of Core Loss Excitations in EELS by Monte Carlo sitional Mapping in STEM EELS, Microsc. Microanal., 2009, Simulation, Microsc. Microanal., 2013, 19, 366–367. 15, 50–51. 275 M. Attarian Shandiz, F. Salvat and R. Gauvin, Detectability 293 D. del Pozo-Bueno, F. Peiró and S. Estradé, Support vector Limits in EELS by Monte Carlo Simulations, Microsc. machine for EELS oxidation state determination, Ultramicro- Microanal., 2012, 18, 998–999. scopy, 2021, 221, 113190. 276 J. Ã. Verbeeck and S. Aert, Van. Model based quantification 294 P. J. Thomas and P. A. Midgley, Image-spectroscopy – I. of EELS spectra, Ultramicroscopy, 2004, 101, 207–224. The advantages of increased spectral information for 277 J. Ã. Verbeeck, S. Van Aert and G. Bertoni, Model-based compositional EFTEM analysis, Ultramicroscopy, 2001, 88, quantification of EELS spectra: Including the fine structure, 179–186. Ultramicroscopy, 2006, 106, 976–980. 295 P. J. Thomas and P. A. Midgley, Image-spectroscopy – II. 278 X. Liu, et al., Machine learning approach for the prediction The removal of plural scattering from extended energy- of electron inelastic mean free paths, Phys. Rev. Mater., filtered series by Fourier deconvolution, Ultramicroscopy, 2021, 033802. 2001, 88, 187–194. 279 E. Quattrocchi, et al., The deep-DRT: A deep neural net- 296 L. I. Roest, S. E. van Heijst, L. Maduro, J. Rojo and S. Conesa- work approach to deconvolve the distribution of relaxation Boj, Charting the low-loss region in electron energy loss times from multidimensional electrochemical impedance spectroscopy with machine learning, Ultramicroscopy, 2021, spectroscopy data, Electrochim. Acta, 2021, 392, 139010. 222, 113202. 280 M. S. Moreno, K. Jorissen and J. J. Rehr, Practical aspects of 297 A. Stevens, et al., Compressive STEM-EELS, Microsc. Micro- electron energy-loss spectroscopy (EELS) calculations anal., 2016, 22, 560–561. using FEFF8, Micron, 2007, 38, 1–11. 298 E. Monier, et al., Fast reconstruction of atomic-scale STEM- 281 A. Ankudinov and B. Ravel, Real-space multiple-scattering EELS images from sparse sampling, Ultramicroscopy, 2020, calculation and interpretation of x-ray-absorption near-edge 215, 112993, DOI: 10.1016/j.ultramic.2020.112993. 1466 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 299 S. M. Collins, et al., Scan Strategies for Electron Energy Loss 318 A. Skorikov, W. Heyvaert, W. Albecht, D. Pelt and S. Bals, Spectroscopy at Optical and Vibrational Energies in Perylene Deep learning-based denoising for improved dose effi- Diimide Nanobelts, Microsc. Microanal., 2019, 25, 1738–1739. ciency in EDX tomography of nanoparticles, Nanoscale, 300 J. Schwartz, et al., Recovering Chemistry at Atomic Resolu- 2021, 13, 12242–12249. tion using Multi-Modal Spectroscopy, Microsc. Microanal., 319 B. C. Love, Comparing supervised and unsupervised cate- 2021, 27, 1226–1228. gory learning, Psychon. Bull. Rev., 2002, 9, 829–835. 301 S. M. Collins and P. A. Midgley, Progress and opportunities 320 S. Somnath, USID and Pycroscopy – Open frameworks for in EELS and EDS tomography, Ultramicroscopy, 2017, 180, storing and analyzing spectroscopic and imaging data, 133–141. Microsc. Microanal., 2019, 25, 220–221. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 302 R. K. Leary and P. A. Midgley, Analytical electron tomo- 321 C. R. Harris, et al., Array programming with NumPy, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. graphy, MRS Bulletin, 2016, 41, 531–536. Nature, 2020, 585, 357–362. 303 Z. Zhong, et al., A bimodal tomographic reconstruction 322 pyUSID, available at: https://pycroscopy.github.io/pyUSID/ technique combining EDS-STEM and HAADF-STEM, Ultra- about.html, (accessed: 28th December 2021). microscopy, 2017, 174, 35–45. 323 The HDF Group. The HDF5s Library & File Format, avail- 304 Z. Zhong, W. J. Palenstijn, N. R. Viganò and K. J. able at: https://www.hdfgroup.org/solutions/hdf5/ (accessed: Batenburg, Numerical methods for low-dose EDS tomo- 28th December 2021). graphy, Ultramicroscopy, 2018, 194, 133–142. 324 G. H. Weber, C. Ophus and L. Ramakrishnan, Automated 305 M. Weyland and P. A. Midgley, Extending Energy-Filtered Labeling of Electron Microscopy Images Using Deep Learn- Transmission Electron Microscopy (EFTEM) into Three ing, in Proc. MLHPC 2018 Mach. Learn. HPC Environ. Held Dimensions Using Electron Tomography, Microsc. Micro- conjunction with SC 2018 Int. Conf. High Perform. Comput. anal., 2003, 9, 542–555. Networking, Storage Anal., 2019, pp. 26–36, DOI: 10.1109/ 306 P. A. Midgley and M. Weyland, 3D electron microscopy in MLHPC.2018.8638633. the physical sciences: the development of Z-contrast and 325 R. K. Vasudevan, et al., Materials science in the artificial EFTEM tomography, Ultramicroscopy, 2003, 96, 413–431. intelligence age: High-throughput library generation, 307 B. Goris, et al., Discrete spectroscopic electron tomogra- machine learning, and a pathway from correlations to the phy: using prior knowledge of reference spectra during the underpinning physics, MRS Commun., 2019, 9, 821–838. reconstruction, EMC Proc., 2016, 8, 976–977. 326 A. Khadangi, T. Boudier and V. Rajagopal, EM-stellar: 308 M. Pfannmöller, et al., Quantitative Tomography of benchmarking deep learning for electron microscopy Organic Photovoltaic Blends at the Nanoscale, Nano Lett., image segmentation, Bioinformatics, 2021, 37, 97–106. 2015, 15, 6634–6642. 327 J. M. Cowley and A. F. Moodie, The scattering of electrons 309 D. Zanaga, et al., A New Method for Quantitative XEDS by atoms and crystals. I. A new theoretical approach, Acta Tomography of Complex Heteronanostructures, Part. Part. Crystallogr., 1957, 10, 609–619. Syst. Charact., 2016, 33, 396–403. 328 E. J. Kirkland, R. F. Loane and J. Silcox, Simulation of 310 B. Goris, et al., Towards Quantitative EDX Results in 3 annular dark field stem images using a modified multi- Dimensions, Microsc. Microanal., 2014, 20, 766–767. slice method, Ultramicroscopy, 1987, 23, 77–96. 311 S. Bals, et al., Spectral Electron Tomography as a Quantitative 329 D. A. Muller, B. Edwards, E. J. Kirkland E and J. Silcox, Technique to Investigate Functional Nanomaterials, Microsc. Simulation of thermal diffuse scattering including a Microanal., 2021, 22, 274–275. detailed phonon dispersion curve, Ultramicroscopy, 2001, 312 A. Alafeef, et al., Linear chemically sensitive electron 86, 371–380. tomography using DualEELS and dictionary-based com- 330 J. C. H. Spence and J. M. Zuo, Electron Microdiffraction. pressed sensing, Ultramicroscopy, 2016, 170, 96–106. Electron Microdiffraction, Springer US, 1992, DOI: 10.1007/ 313 R. Huber, G. Haberfehlner, M. Holler, G. Kothleitner and 978-1-4899-2353-0. K. Bredies, Total generalized variation regularization for 331 C. Koch and J. M. Zuo, Comparison of Multislice Computer multi-modal electron tomography, Nanoscale, 2019, 11, Programs for Electron Scattering Simulations and The 5617–5632. Bloch Wave Method, Microsc. Microanal., 2000, 6, 126–127. 314 L. Yedra, et al., EEL spectroscopic tomography: Towards a 332 A. Hjorth Larsen, et al., The atomic simulation environ- new dimension in nanomaterials analysis, Ultramicro- ment - A Python library for working with atoms, J. Phys.: scopy, 2012, 122, 12–18. Condens. Matter, 2017, 29, 273002. 315 Z. Saghi, et al., Improved Data Analysis and Reconstruction 333 J. Richard, T. John, N. Manuel, L. Sean and M. Ian, Methods for STEM-EDX Tomography, Microsc. Microanal., MDAnalysis: A Python Package for the Rapid Analysis of 2016, 22, 284–285. Molecular Dynamics Simulations, Proc. 15th Python Sci. 316 P. Torruella, et al., 3D visualization of iron oxidation state Conf., 2019, vol. 11. in FeO/Fe3O4 core–shell nanocubes from electron energy 334 mendeleev, available at: https://mendeleev.readthedocs.io/ loss tomography, Nano Lett., 2016, 16, 5068–5073. en/stable/ (accessed: 27th December 2021). 317 X. Yang, et al., Low-dose x-ray tomography through a deep 335 crystals PyPI, available at: https://pypi.org/project/crystals/ convolutional neural network, Sci. Rep., 2018, 8, 2575. (accessed: 27th December 2021). This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1467 View Article Online Review Nanoscale Horizons 336 J. B. Greisman, K. M. Dalton and D. R. Hekstra, Recipro- 355 jacobjma, PyQSTEM: A Python interface to the electron micro- calspaceship: a Python library for crystallographic data scopy simulation program QSTEM, available at: https://github. analysis, J. Appl. Cryst., 2021, 54, 1521–1529, ISSN: 1600- com/jacobjma/PyQSTEM (accessed: 27th December 2021). 5767. 356 L. P. René de Cotret, M. R. Otto, M. J. Stern and B. J. Siwick, 337 A. H. Combs, et al., Fast approximate STEM image simula- An open-source software ecosystem for the interactive tions from a machine learning model, Adv. Struct. Chem. exploration of ultrafast electron scattering data, Adv. Imaging, 2019, 5, 2. Struct. Chem. Imaging, 2018, 4, 11. 338 J. S. Smith, O. Isayev and A. E. Roitberg, ANI-1: an exten- 357 V. Grillo and F. Rossi, STEM_CELL: A software tool for sible neural network potential with DFT accuracy at force electron microscopy. Part 2 analysis of crystalline materi- This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. field computational cost, Chem. Sci., 2017, 8, 3192–3203. als, Ultramicroscopy, 2013, 125, 112–129. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 339 A. V. Sinitskiy and V. S. Pande, Deep Neural Network 358 V. Grillo and E. Rotunno, STEM_CELL: A software tool for Computes Electron Densities and Energies of a Large Set electron microscopy: Part I-simulations, Ultramicroscopy, of Organic Molecules Faster than Density Functional The- 2013, 125, 97–111. ory (DFT), arXiv, 2018, preprint, arXiv:1809.02723, DOI: 359 Berkeley CA USA, Total Resolution LLC | HRTEM Software 10.48550/arXiv.1809.02723. Provider | TEMPAS, available at: https://www.totalresolu 340 N. Artrith and A. Urban, An implementation of artificial tion.com/ (accessed: 27th December 2021). neural-network potentials for atomistic materials simula- 360 F. Salvat, J. Fernández-Vera and J. Sempau, PENELOPE-2018: A tions: Performance for TiO2, Comput. Mater. Sci., 2016, Code System for Monte Carlo Simulation of Electron and 114, 135–150. Photon Transport, in Work. Proceedings, Barcelona, Spain, 2019. 341 J. Behler, First Principles Neural Network Potentials for 361 pyPENELOPE, available at: https://pypenelope.sourceforge. Reactive Simulations of Large Molecular and Condensed net/index.html (accessed: 4th August 2022). Systems, Angew. Chem., Int. Ed., 2017, 56, 12828–12840. 362 X. Glorot and Y. Bengio, Understanding the difficulty of 342 N. Artrith, T. Morawietz and J. Behler, High-dimensional training deep feedforward neural networks, J. Mach. Learn. neural-network potentials for multicomponent systems: Res., 2010, 9, 249–256. Applications to zinc oxide, Phys. Rev. B: Condens. Matter 363 L. Fei-Fei, J. Deng and K. Li, ImageNet: Constructing a Mater. Phys., 2011, 83, 153101. large-scale image database, J. Vis., 2010, 9, 1037. 343 J. Madsen and T. Susi, abTEM: ab Initio Transmission 364 Y. You, Z. Zhang, C. J. Hsieh, J. Demmel and K. Keutzer, Electron Microscopy Image Simulation, Microsc. Microanal., ImageNet training in minutes, Proceedings of the 47th 2020, 26, 448–450. International Conference on Parallel Processing, 2018, 1, 344 J. Madsen, T. J. Pennycook and T. Susi, Ab Initio Description DOI: 10.1145/3225058.3225069. of Bonding for Transmission Electron Microscopy, Ultra- 365 Y. LeCun, C. C. and C. B. MNIST handwritten digit microscopy, 2021, 113253, DOI: 10.1016/j.ultramic.2021.113253. database, available at: https://yann.lecun.com/exdb/mnist/ 345 T. Susi, et al., Efficient first principles simulation of (accessed: 23rd December 2021). electron scattering factors for transmission electron micro- 366 Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, Gradient- scopy, Ultramicroscopy, 2019, 197, 16–22. Based Learning Applied to Document Recognition, 346 cerius2, available at: https://www-jmg.ch.cam.ac.uk/cil/ Proceedings of the IEEE, 1998, 86, 2278–2324. SGTL/cerius2.html (accessed: 22nd December 2021). 367 LabelMe.Dataset, available at: https://labelme.csail.mit. 347 J. J. P. Peters, clTEM | GPU accelerated multislice. edu/Release3.0/browserTools/php/dataset.php (accessed: 348 ningustc, cudaEM, available at: https://github.com/nin 23rd December 2021). gustc/cudaEM (accessed: 22nd December 2021). 368 B. C. Russell, A. Torralba, K. P. Murphy and W. T. Freeman, 349 J. Barthel, Dr Probe: A software for high-resolution STEM LabelMe: A database and web-based tool for image anno- image simulation, Ultramicroscopy, 2018, 193, 1–11. tation, Int. J. Comput. Vis., 2008, 77, 157–173. 350 R. Kilaas, L. D. Marks and C. S. Own, EDM 1.0: Electron 369 L. Von Ahn and L. Dabbish, Labeling images with a direct methods, Ultramicroscopy, 2005, 102, 233–237. computer game, in Conf. Hum. Factors Comput. Syst. - Proc., 351 C. Ophus, A fast image simulation algorithm for scanning 2004, vol. 6, pp. 319–326. transmission electron microscopy, Adv. Struct. Chem. Imaging, 370 Caltech101, available at: https://www.vision.caltech.edu/Ima 2017, 3, 13. ge_Datasets/Caltech101/ (accessed: 23rd December 2021). 352 A. Pryor, C. Ophus and J. Miao, A streaming multi-GPU 371 Microsoft, Kinect Gesture Data Set from Official Microsoft implementation of image simulation algorithms for Download Center, available at: https://www.microsoft. scanning transmission electron microscopy, Adv. Struct. com/en-us/download/details.aspx?id=52283&from=https% Chem. Imaging, 2017, 3, 15. 3A%2F%2Fresearch.microsoft.com%2Fen-us%2Fum%2 353 L. Rangel DaCosta, et al., Prismatic 2.0 – Simulation software Fcambridge%2Fprojects%2Fmsrc12%2F (accessed: 23rd for scanning and high resolution transmission electron December 2021). microscopy (STEM and HRTEM), Micron, 2021, 151, 103141. 372 The PASCAL Visual Object Classes Homepage, available 354 C. Koch, QSTEM: Quantitative TEM/STEM Simulations — at: https://host.robots.ox.ac.uk/pascal/VOC/ (accessed: 23rd Strukturforschung/Elektronenmikroskopie. December 2021). 1468 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 373 R. Lin, R. Zhang, C. Wang, X. Q. Yang and H. L. Xin, 392 AI Training Data and other Data Management Services, TEMImageNet training library and AtomSegNet deep-learning available at: https://www.clickworker.com/ (accessed: 28th models for high-precision atom segmentation, localization, December 2021). denoising, and deblurring of atomic-resolution images, 393 MathWorks, Label images for computer vision applica- Sci. Rep., 2021, 11, 5386. tions – MATLAB, available at: https://es.mathworks.com/ 374 A. Lucchi, Y. Li and P. Fua, Learning for structured prediction help/vision/ref/imagelabeler-app.html (accessed: 28th using approximate subgradient descent with working sets, in December 2021). Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 394 Sama – Make Training Data Your Competitive Advantage, 2013, pp. 1987–1994, DOI: 10.1109/CVPR.2013.259. available at: https://www.sama.com/ (accessed: 22nd This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 375 V. Morath, et al., Semi-automatic determination of cell December 2021). Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. surface areas used in systems biology, Front. Biosci., 2013, 395 LabelMe, The Open annotation tool, available at: https:// 5, 533–545. labelme.csail.mit.edu/Release3.0/index.php (accessed: 23rd 376 ImageJ: Public data sets, available at: https://imagej.net/ December 2021). plugins/public-data-sets (accessed: 27th December 2021). 396 K. Simonyan and A. Zisserman, Very deep convolutional 377 R. Aversa, M. H. Modarres, S. Cozzini, R. Ciancio and networks for large-scale image recognition, in 3rd Int. Conf. A. Chiusole, The first annotated set of scanning electron Learn. Represent. ICLR 2015 – Conf. Track Proc., 2015, pp. 1–14. microscopy images for nanoscience, Sci. Data, 2018, 5, 180172. 397 GitHub – machrisaa/tensorflow-vgg: VGG19 and VGG16 on 378 D. A. Boiko, E. O. Pentsak, V. A. Cherepano and V. P. Tensorflow, available at: https://github.com/machrisaa/ Ananiko, Electron microscopy dataset for the recognition tensorflow-vgg (accessed: 24th January 2022). of nanoscale ordering effects and location of nano- 398 GitHub – rcmalli/keras-vggface: VGGFace implementation particles, Sci. Data, 2020, 7, 101. with Keras Framework, available at: https://github.com/ 379 B. L. Decost and E. A. Holm, A large dataset of synthetic rcmalli/keras-vggface (accessed: 24th January 2022). SEM images of powder materials and their ground truth 399 GitHub, GitHub: Where the world builds software, available 3D structures, Data Br., 2016, 9, 727–731. at: https://github.com/ (accessed: 29th December 2021). 380 J. M. Ede, Warwick Electron Microscopy Datasets, Machine 400 Cloud Computing Services | Microsoft Azure, available at: Learning: Science and Technology, 2020, 1, 045003. https://azure.microsoft.com/en-us/ (accessed: 21st December 381 Gatan, EELS.info, available at: https://eels.info/ (accessed: 2021). 28th December 2021). 401 Cloud computing services – Google Cloud, available at: 382 S. Gražulis, et al., Crystallography Open Database (COD): https://cloud.google.com/ (accessed: 21st December 2021). An open-access collection of crystal structures and plat- 402 Google, GoogleColab – Colaboratory, available at: https:// form for world-wide collaboration, Nucleic Acids Res., 2012, colab.research.google.com/ (accessed: 21st December 40, 420–427. 2021). 383 S. Graulis, et al., Crystallography Open Database - An open- 403 IBM Cloud | IBM, available at: https://www.ibm.com/cloud access collection of crystal structures, J. Appl. Crystallogr., (accessed: 21st December 2021). 2009, 42, 726–729. 404 M. Ziatdinov, pycroscopy/AICrystallographer, available at: 384 A. Belkly, M. Helderman, V. L. Karen and P. Ulkch, New https://github.com/pycroscopy/AICrystallographer (accessed: developments in the Inorganic Crystal Structure Database 8th September 2022). (ICSD): Accessibility in support of materials research and 405 M. Ziatdinov, A. Ghosh, T. Wong and S. V. Kalinin, AtomAI: design, Acta Crystallogr. Sect. B: Struct. Sci., 2002, 58, 364–369. A Deep Learning Framework for Analysis of Image and 385 M. Hellenbrandt, The inorganic crystal structure database Spectroscopy Data in (Scanning) Transmission Electron Micro- (ICSD) - Present and future, Crystallogr. Rev., 2004, 10, 17–22. scopy and Beyond, arXiv, 2021, preprint, arXiv:2105.07485, 386 A. D. Mighell and V. L. Karen, NIST crystallographic DOI: 10.48550/arXiv.2105.07485. databases for research and analysis, J. Res. Natl. Inst. Stand. 406 M. Ziatdinov, pycroscopy/atomai: Deep and Machine Technol., 1996, 101, 273–280. Learning for Microscopy, available at: https://github.com/ 387 Amazon Mechanical Turk, available at: https://www.mturk. pycroscopy/atomai (accessed: 8th September 2022). com/ (accessed: 28th December 2021). 407 M. Ziatdinov, ziatdinovmax/gpax: Structured Gaussian Pro- 388 appen, Confidence to Deploy AI with World-Class Training cesses and Deep Kernel Learning, available at: https:// Data, available at: https://appen.com/ (accessed: 28th December github.com/ziatdinovmax/gpax (accessed: 8th September 2021). 2022). 389 TrainingSet.AI, available at: https://trainingset.ai/ (accessed: 408 M. Ziatdinov, pycroscopy/pyTEMlib: TEM data quantifica- 28th December 2021). tion library through a model-based approach, available 390 Superb AI | Fastest training data platform for computer at: https://github.com/pycroscopy/pyTEMlib (accessed: 8th vision, available at: https://www.superb-ai.com/ (accessed: September 2022). 28th December 2021). 409 D. Mukherjee and R. Unocic, STEMTooL: An Open Source 391 Human-labeled AI Training Data | iMerit, available at: Python Toolkit for Analyzing Electron Microscopy Data- https://imerit.net/ (accessed: 28th December 2021). sets, Microsc. Microanal., 2020, 26, 2960–2962. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1469 View Article Online Review Nanoscale Horizons 410 uw-cmg/MAST-ML: MAterials Simulation Toolkit for 428 A. van den Oord, et al., WaveNet: A Generative Model for Machine Learning (MAST-ML), available at: https://github. Raw Audio, arXiv, 2016, preprint, arXiv:1609.03499, DOI: com/uw-cmg/MAST-ML (accessed: 8th September 2022). 10.48550/arXiv.1609.03499. 411 R. Jacobs, et al., The Materials Simulation Toolkit for 429 A. Bansal, X. Chen, B. Russell, A. Gupta and D. Ramanan: Machine learning (MAST-ML): An automated open source Representation of the pixels, by the pixels, and for the toolkit to accelerate data-driven materials research, pixels, arXiv, 2017, preprint, arXiv:1702.06506, DOI: Comput. Mater. Sci., 2020, 176, 109544. 10.48550/arXiv.1702.06506. 412 E. Gómez-de-Mariscal, et al., DeepImageJ: A user-friendly 430 A. Van Den Oord, N. Kalchbrenner and K. Kavukcuoglu, environment to run deep learning models in ImageJ, Nat. Pixel recurrent neural networks, in 33rd Int. Conf. Mach. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Methods, 2021, 18, 1192–1195. Learn. ICML, 2016, vol. 4, pp. 2611–2620. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 413 B. Midtvedt, et al., Quantitative digital microscopy with 431 Y.-W. Chang, et al., Neural Network Training with Highly deep learning, Appl. Phys. Rev., 2021, 8, 011310. Incomplete Datasets, Mach. Learn.: Sci. Technol., 2021, 414 F. Cichos, K. Gustavsson, B. Mehlig and G. Volpe, Machine 3, 035001. learning for active matter, Nat. Mach. Intell., 2020, 2, 432 GitHub – intel/caffe, 2019. 94–103. 433 Caffe2|A New Lightweight, Modular, and Scalable Deep Learn- 415 L. von Chamier, et al., Democratising deep learning for ing Framework, available at: https://caffe2.ai/ (accessed: 4th microscopy with ZeroCostDL4Mic, Nat. Commun., 2021, August 2022). 12, 2276. 434 ctypes — A foreign function library for Python, available at: 416 M. G. Haberl, et al., CDeep3M—Plug-and-Play cloud-based https://docs.python.org/3/library/ctypes.html (accessed: 24th deep learning for image segmentation, Nat. Methods, 2018, December 2021). 15, 677–680. 435 S. Behnel, et al., Cython: The best of both worlds, Comput. 417 E. Bisong, Building Machine Learning and Deep Learning Sci. Eng., 2011, 13, 31–39. Models on Google Cloud Platform, in Build. Mach. Learn. 436 Dask, Parallel computation with blocked algorithms and task Deep Learn. Model. Google Cloud Platf., 2019, DOI: 10.1007/ scheduling, 2015, 130, 136. 978-1-4842-4470-8. 437 S. Wagon, Mathematica in action: Problem solving through 418 H. Banjak, et al., Evaluation of noise and blur effects with visualization and computation. Mathematica in Action: SIRT-FISTA-TV reconstruction algorithm: Application to Problem Solving Through Visualization and Computation, fast environmental transmission electron tomography, Springer New York, 2010, DOI: 10.1007/978-0-387- Ultramicroscopy, 2018, 189, 109–123. 75477-2. 419 Zenodo – Research, Shared, available at: https://zenodo. 438 R. Řehůřek, Gensim: Topic modelling for humans, avail- org/ (accessed: 29th December 2021). able at: https://radimrehurek.com/gensim/ (accessed: 29th 420 Develop and Download Open Source Software – OSDN, December 2021). available at: https://osdn.net/ (accessed: 29th December 439 J. W. Eaton, GNU Octave, Distribution, 2007. 2021). 440 Apache, Hadoop. 421 Bitbucket | The Git solution for professional teams, available 441 A. Gulli and S. Pal, Deep Learning with Keras - Antonio Gulli, at: https://bitbucket.org/product/ (accessed: 29th December Sujit Pal – Google Books, Packt Publishing, 2017. 2021). 442 Apache Mahout. 422 Iterate faster, innovate together|GitLab, available at: 443 MathWorks, MATLAB and Simulink, available at: https://es. https://about.gitlab.com/ (accessed: 29th December 2021). mathworks.com/?s_tid=gn_logo (accessed: 29th December 423 M. Z. Alom, et al., The History Began from AlexNet: A 2021). Comprehensive Survey on Deep Learning Approaches, 444 J. D. Hunter, Matplotlib: A 2D graphics environment, arXiv, 2018, preprint, arXiv:1803.01164, DOI: 10.48550/ Comput. Sci. Eng., 2007, 9, 90–95. arXiv.1803.01164. 445 Apache MXNet | A flexible and efficient library for deep 424 A. Krizhevsky, I. Sutskever and G. E. Hinton, ImageNet learning, available at: https://mxnet.apache.org/versions/1. Classification with Deep Convolutional Neural Networks, 9.0/ (accessed: 24th December 2021). NIPS, 2012, 145–151, DOI: 10.1145/3383972.3383975. 446 S. K. Lam, A. Pitrou and S. Seibert, Numba, 2015, pp. 1–6, 425 J. Redmon and A. Farhadi, YOLO9000: Better, faster, DOI: 10.1145/2833157.2833162. stronger, in Proc. – 30th IEEE Conf. Comput. Vis. Pattern 447 G. Bradski, tools, A. K.-D. D., journal of software & 2000, Recognition, CVPR 2017, 2017, pp. 6517–6525. undefined. OpenCV. roswiki.autolabor.com.cn. 426 E. Shelhamer, J. Long and T. Darrell, Fully Convolutional 448 G. Bradski and A. Kaehler, Learning OpenCV: Computer Networks for Semantic Segmentation, IEEE Trans. Pattern vision with the OpenCV library, 2008. Anal. Mach. Intell., 2017, 39, 640–651. 449 J. Reback, et al., Pandas, 2021, DOI: 10.5281/ZENODO.5774815. 427 S. Ren, K. He, R. Girshick and J. Sun, Faster R-CNN: 450 T. De Smedt and W. Daelemans, Pattern for python, Towards Real-Time Object Detection with Region Proposal J. Mach. Learn. Res., 2012, 13, 2063–2067. Networks, IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, 451 Pillow – Pillow (PIL Fork). 1137–1149. 452 GitHub – pytorch/pytorch. 1470 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 453 S. Van Der Walt, et al., Scikit-image: Image processing in 474 Gempa – ER-C. python, PeerJ, 2014, 2, e453. 475 A. Clausen, et al., LiberTEM: Software platform for scalable 454 F. Pedregosa, et al., Scikit-learn: Machine Learning in multidimensional data processing in transmission elec- Python, J. Mach. Learn. Res., 2011, 12, 2825–2830. tron microscopy, J. Open Source Softw., 2020, 5, 2006. 455 P. Virtanen, et al., SciPy 1.0: fundamental algorithms for 476 F. De La Peña, et al., hyperspy/hyperspy: Release v1.6.4. zndo, scientific computing in Python, Nat. Methods, 2020, 17, 2021, DOI: 10.5281/ZENODO.592838. 261–272. 477 F. de la Pena, et al., Electron Microscopy (Big and Small) 456 S. Sonnenburg, et al., shogun-toolbox/shogun: Shogun 6.1.0, Data Analysis With the Open Source Software Package 2017, DOI: 10.5281/ZENODO.1067840. HyperSpy, Microsc. Microanal., 2017, 23, 214–215. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 457 databricks/spark-deep-learning, Deep Learning Pipelines 478 A. B. Naden, K. J. O’Shea and D. A. MacLaren, Evaluation of Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. for Apache Spark. crystallographic strain, rotation and defects in functional 458 M. Abadi, et al., TensorFlow: A System for Large-Scale oxides by the moiré effect in scanning transmission elec- Machine Learning, in Proc. 12th USENIX Symp. Oper. Syst. tron microscopy, Nanotechnology, 2018, 29, 165704. Des. Implement., 2016, vol. 101, pp. 582–598. 479 M. Nord, atomap/atomap GitLab, available at: https://gitlab. 459 J. Bergstra, et al., Theano: Deep Learning on GPUs with com/atomap/atomap (accessed: 8th September 2022). Python, J. Mach. Learn. Res., 2011, 1, 1–48. 480 P. L. Galindo, et al., The Peak Pairs algorithm for strain 460 Elastic GPU Service: Powerful Computing Capabilities for mapping from HRTEM images, Ultramicroscopy, 2007, 107, Deep Learning – Alibaba Cloud, available at: https://www. 1186–1193. alibabacloud.com/es/product/gpu (accessed: 21st December 481 S. I. Molina, et al., Column-by-column compositional 2021). mapping by Z-contrast imaging, Ultramicroscopy, 2009, 461 AWS | Cloud Computing, available at: https://aws.amazon. 109, 172–176. com/ (accessed: 24th December 2021). 482 iMtools – ER-C. 462 Deepnote – Data science notebook for teams. Available at: 483 Y. Wang, U. Salzberger, W. Sigle, Y. Eren Suyolcu and https://deepnote.com/ (accessed: 28th December 2021). P. A. van Aken, Oxygen octahedra picker: A software tool to 463 The Acceleration Cloud | Genesis Cloud, available at: extract quantitative information from STEM images, Ultra- https://www.genesiscloud.com/ (accessed: 22nd December microscopy, 2016, 168, 46–52. 2021). 484 Y. Li, O. Vinyals, C. Dyer, R. Pascanu and P. Battaglia, 464 OVHcloud, available at: https://www.ovhcloud.com/en/ Learning Deep Generative Models of Graphs, arXiv, 2018, (accessed: 22nd December 2021). preprint, arXiv:1803.03324, DOI: 10.48550/arXiv.1803.03324. 465 CORE, available at: https://www.paperspace.com/core 485 T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez and P. W. Battaglia, (accessed: 21st December 2021). Learning Mesh-Based Simulation with Graph Networks, arXiv, 466 Weights & Biases – Developer tools for ML, available at: 2020, preprint, arXiv:2010.03409, DOI: 10.48550/arXiv.2010. https://wandb.ai/site (accessed: 13th January 2022). 03409. 467 PerceptiLabs, available at: https://www.perceptilabs.com/ 486 P. W. Battaglia, et al., Relational inductive biases, deep (accessed: 13th January 2022). learning, and graph networks, arXiv, 2018, preprint, 468 M. Klinger and A. Jäger, Crystallographic Tool Box (CrysT- arXiv:1806.01261, DOI: 10.48550/arXiv.1806.01261. Box): Automated tools for transmission electron microsco- 487 A. Sanchez-Gonzalez, et al., Learning to Simulate Complex pists and crystallographers, J. Appl. Crystallogr., 2015, 48, Physics with Graph Networks, International Conference on 2012–2018. Machine Learning, PMLR, 2020, pp. 8459–8468. 469 B. H. Savitzky, et al., Py4DSTEM: A Software Package for 488 P. Battaglia, R. Pascanu, M. Lai, D. Rezende and Four-Dimensional Scanning Transmission Electron Micro- K. Kavukcuoglu, Interaction networks for learning about scopy Data Analysis, Microsc. Microanal., 2021, 27, 712–743. objects, relations and physics, Adv. Neural Inf. Process. 470 D. N. Johnstone, et al., pyxem, 2021, DOI: 10.5281/ Syst., 2016, 4509–4517. ZENODO.5075520. 489 J. Kober, J. A. Bagnell and J. Peters, Reinforcement learning 471 M. Nord, pyxem/pyxem: An open-source Python library for in robotics: A survey, Int. J. Rob. Res., 2013, 32, 1238–1274. multi-dimensional diffraction microscopy, available at: 490 K. Shao, Z. Tang, Y. Zhu, N. Li and D. Zhao, A Survey of Deep https://github.com/pyxem/pyxem/ (accessed: 8th Septem- Reinforcement Learning in Video Games, arXiv, 2019, pre- ber 2022). print, arXiv:1912.10944, DOI: 10.48550/arXiv.1912.10944. 472 N. Cautaerts, et al., Free, flexible and fast: Orientation 491 M. Adrian, J. Dubochet, J. Lepault and A. W. McDowall, mapping using the multi-core and GPU-accelerated tem- Cryo-electron microscopy of viruses, Nature, 1984, 308, plate matching capabilities in the Python-based open 32–36. source 4D-STEM analysis toolbox Pyxem, Ultramicroscopy, 492 P. Schultz, Cryo-electron microscopy of vitrified specimens, 2022, 237, 113517. Q. Rev. Biophys., 1988, 21, 129–228. 473 G. Hermann, et al., ANIMATED-TEM: A toolbox for electron 493 D. Tegunov and P. Cramer, Real-time cryo-electron micro- microscope automation based on image analysis, Mach. scopy data preprocessing with Warp, Nat. Methods, 2019, Vis. Appl., 2012, 23, 691–711. 16, 1146–1152. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1471 View Article Online Review Nanoscale Horizons 494 C. O. S. Sorzano, et al., Automatic particle selection from 508 J. M. Phillip, K. S. Han, W. C. Chen, D. Wirtz and P. H. Wu, electron micrographs using machine learning techniques, A robust unsupervised machine-learning method to J. Struct. Biol., 2009, 167, 252–260. quantify the morphological heterogeneity of cells and 495 R. Langlois, et al., Automated particle picking for low- nuclei, Nat. Protoc., 2021, 16, 754–774. contrast macromolecules in cryo-electron microscopy, 509 I. Arganda-Carreras, et al., Trainable Weka Segmentation: J. Struct. Biol., 2014, 186, 1–7. A machine learning tool for microscopy pixel classifica- 496 A. Krull, T.-O. Buchholz and F. Jug, Noise2Void – Learning tion, Bioinformatics, 2017, 33, 2424–2426. Denoising From Single Noisy Images, Proceedings of the 510 G. Martı́n, et al., ContactJ: Lipid droplets-mitochondria IEEE/CVF Conference on Computer Vision and Pattern Recog- contacts characterization through fluorescence micro- This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. nition, 2019, pp. 2129–2137. scopy and image analysis, F1000 Res., 2021, 1–8. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 497 Y. S. G. Nashed, et al., CryoPoseNet: End-to-End Simultaneous 511 A. A. Sekh, et al., Physics-based machine learning for Learning of Single-particle Orientation and 3D Map Recon- subcellular segmentation in living cells, Nat. Mach. Intell., struction from Cryo-electron Microscopy Data, Proceedings of 2021, 3, 1071–1080. the IEEE/CVF International Conference on Computer Vision, 512 C. Mcquin, et al., Cellprofiler 3.0: Next-generation image 2021, pp. 4066–4076, DOI: 10.1109/iccvw54120.2021.00452. processing for biology, PLoS Biol., 2018, 16, e2005970. 498 J.-G. Wu, et al., Machine Learning for Structure Determina- 513 R. M. Sterbentz, K. L. Haley and J. O. Island, Universal tion in Single-Particle Cryo-Electron Microscopy: A Systematic image segmentation for optical identification of 2D materials, Review, IEEE Trans. Neural Networks Learn. Syst., 2021, 1–21, Sci. Rep., 2021, 11, 5808. DOI: 10.1109/TNNLS.2021.3131325. 514 H. Kim, J. Inoue and T. Kasuya, Unsupervised microstructure 499 P. Mostosi, H. Schindelin, P. Kollmannsberger and segmentation by mimicking metallurgists’ approach to A. Thorn, Haruspex: A Neural Network for the Automatic pattern recognition, Sci. Rep., 2020, 10, 17835. Identification of Oligonucleotides and Protein Secondary 515 H. Kim, Y. Arisato and J. Inoue, Unsupervised segmenta- Structure in Cryo-Electron Microscopy Maps, Angew. tion of microstructural images of steel using data mining Chem., Int. Ed., 2020, 59, 14788–14795. methods, Comput. Mater. Sci., 2022, 201, 110855. 500 R. Li, D. Si, T. Zeng, S. Ji and J. He, Deep convolutional 516 S. Masubuchi, et al., Deep-learning-based image segmentation neural networks for detecting secondary structures in integrated with optical microscopy for automatically searching protein density maps from cryo-electron microscopy, in for two-dimensional materials, npj 2D Mater. Appl., 2020, Proc. – 2016 IEEE Int. Conf. Bioinforma. Biomed. BIBM 2016, 4, 3. 2017, pp. 41–46, DOI: 10.1109/BIBM.2016.7822490. 517 V. Kumar, et al., Radiomics: The process and the challenges, 501 T. W. Nattkemper, Automatic segmentation of digital Magn. Reson. Imaging, 2012, 30, 1234–1248. micrographs: A survey, Stud. Health Technol. Informat., 518 G. Langs, et al., Machine learning: from radiomics to 2004, 107, 847–851. discovery and routine, Radiologe, 2018, 58, 1–6. 502 L. Luo, et al., Identification of voids and interlaminar 519 G. Currie, K. E. Hawk, E. Rohren, A. Vial and R. Klein, shear strengths of polymer-matrix composites by optical Machine Learning and Deep Learning in Medical Imaging: microscopy experiment and deep learning methodology, Intelligent Imaging, J. Med. Imaging Radiat. Sci., 2019, Polym. Adv. Technol., 2021, 32, 1853–1865. 1–11, DOI: 10.1016/j.jmir.2019.09.005. 503 C. D. Ly, et al., Full-view in vivo skin and blood vessels 520 S. Leger, et al., A comparative study of machine learning profile segmentation in photoacoustic imaging based on methods for time-To-event survival data for radiomics risk deep learning, Photoacoustics, 2022, 25, 100310. modelling, Sci. Rep., 2017, 7, 13206. 504 X. Chen, et al., Deep learning provides high accuracy in 521 M. Kolossváry, C. N. De Cecco, G. Feuchtner and P. Maurovich- automated chondrocyte viability assessment in articular Horvat, Advanced atherosclerosis imaging by CT: Radiomics, cartilage using nonlinear optical microscopy, Biomed. Opt. machine learning and deep learning, J. Cardiovasc. Comput. Express, 2021, 12, 2759. Tomogr., 2019, 13, 274–280. 505 M. I. Razzak and S. Naz, Microscopic Blood Smear 522 A. Scheinker, S. Gessner, C. Emma and A. L. Edelen, Segmentation and Classification Using Deep Contour Adaptive model tuning studies for non-invasive diagnos- Aware CNN and Extreme Machine Learning, in IEEE Comput. tics and feedback control of plasma wakefield acceleration Soc. Conf. Comput. Vis. Pattern Recognit. Work. 2017-July, at FACET-II, Nucl. Instrum. Methods Phys. Res., Sect. A, 2020, 2017, pp. 801–807. 967, 163902. 506 A. Durand, et al., A machine learning approach for online 523 R. Ito, S. Iwano and S. Naganawa, A review on the use of automated optimization of super-resolution optical micro- artificial intelligence for medical imaging of the lungs of scopy, Nature Commun., 2018, 9, 5247. patients with coronavirus disease 2019, Diagn. Interv. 507 C. Canavesi, A. Cogliati and H. B. Hindman, Unbiased Radiol., 2020, 26, 443–448. corneal tissue analysis using Gabor-domain optical coher- 524 P. Sun, D. Wang, V. C. Mok and L. Shi, Comparison of ence microscopy and machine learning for automatic Feature Selection Methods and Machine Learning Classi- segmentation of corneal endothelial cells, J. Biomed. Opt., fiers for Radiomics Analysis in Glioma Grading, IEEE 2020, 25, 1. Access, 2019, 7, 102010–102020. 1472 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 525 C. Chen, X. Ou, J. Wang, W. Guo and X. Ma, Radiomics- 541 B. Alldritt, et al., Automated Tip Functionalization via Based Machine Learning in Differentiation Between Machine Learning in Scanning Probe Microscopy, Comput. Glioblastoma and Metastatic Brain Tumors, Front. Oncol., Phys. Commun., 2021, 273, 108258. 2019, 9, 806. 542 O. M. Gordon, F. L. Q. Junqueira and P. J. Moriarty, Embedding 526 H. C. Kniep, D. F. Madesta, T. Schneider and U. Hanning, human heuristics in machine-learning-enabled probe micro- Radiomics of Brain MRI: Utility in Prediction of Metastatic scopy, Mach. Learn. Sci. Technol., 2020, 1, 015001. Tumor Type, Radiology, 2019, 290, 479–487. 543 ThermoFischer Scientific, AutoTEM 5 Fully automated 527 Y. W. Park, et al., Radiomics MRI Phenotyping with preparation of high-quality TEM samples with DualBeam, Machine Learning to Predict the Grade of Lower-Grade for any user, 2019. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Gliomas: A Study Focused on Nonenhancing Tumors, 544 TEM Sample Preparation | AutoTEM 5 Software – ES. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Korean J. Radiol., 2019, 20, 1381–1389. 545 M. Ziatdinov, et al., Quantifying the Dynamics of Protein 528 D. Kawahara, X. Tang, C. K. Lee, Y. Nagata and Y. Watanabe, Self-Organization Using Deep Learning Analysis of Atomic Predicting the Local Response of Metastatic Brain Tumor to Force Microscopy Data, Nano Lett., 2021, 21, 158–165. Gamma Knife Radiosurgery by Radiomics With a Machine 546 Y. Liu, et al., General Resolution Enhancement Method in Learning Method, Front. Oncol., 2021, 10, 3003. Atomic Force Microscopy Using Deep Learning, Adv. Theory 529 M. Kocher, M. I. Ruge, N. Galldiks and P. Lohmann, Simul., 2019, 2, 1800137. Applications of radiomics and machine learning for radio- 547 B. Alldritt, et al., Automated structure discovery in atomic therapy of malignant brain tumors, Strahlenther. Onkol., force microscopy, Sci. Adv., 2020, 6, eaay6913. 2020, 196, 856–867. 548 P. Müller, et al., Nanite: Using machine learning to assess 530 C. Parmar, P. Grossmann, J. Bussink, P. Lambin and the quality of atomic force microscopy-enabled nano- H. J. W. L. Aerts, Machine Learning methods for Quanti- indentation data, BMC Bioinf., 2019, 20, 465. tative Radiomic Biomarkers, Sci. Rep., 2015, 5, 13087. 549 M. Checa, R. Millan-Solsona, A. G. Mares, S. Pujals and 531 K. H. Jin, M. T. McCann, E. Froustey and M. Unser, Deep G. Gomila, Fast Label-Free Nanoscale Composition Map- Convolutional Neural Network for Inverse Problems in ping of Eukaryotic Cells Via Scanning Dielectric, Small Imaging, IEEE Trans. Image Process., 2017, 26, 4509–4522. Methods, 2021, 5, 12. 532 A. Krull, P. Hirsch, C. Rother, A. Schiffrin and C. Krull, 550 N. M. Ball and R. J. Brunner, Data mining and machine Artificial-intelligence-driven scanning probe microscopy, learning in astronomy, Int. J. Mod. Phys. D, 2010, 19, 1049–1106. Commun. Phys., 2020, 3, 54. 551 M. Ntampaka, et al., The Role of Machine Learning in 533 O. M. Gordon and P. J. Moriarty, Machine learning at the the Next Decade of Cosmology, arXiv, 2019, preprint, (sub)atomic scale: next generation scanning probe micro- arXiv:1902.10159, DOI: 10.48550/arXiv.1902.10159. scopy, Mach. Learn. Sci. Technol., 2020, 1, 023001. 552 D. Baron, Machine Learning in Astronomy: a practical 534 A. G. Okunev, A. V. Nartova and A. V. Matveev, Recognition of overview, arXiv, 2019, preprint, arXiv:1904.07248, DOI: nanoparticles on scanning probe microscopy images using 10.48550/arXiv.1904.07248. computer vision and deep machine learning, in Sib. 2019 – 553 M. Garofalo, A. Botta and G. Ventre, Astrophysics and Big Int. Multi-Conference Eng. Comput. Inf. Sci. Proc., 2019, Data: Challenges, Methods, and Tools, Proc. Int. Astron. pp. 940–943, DOI: 10.1109/SIBIRCON48586.2019.8958363. Union, 2016, 12, 345–348. 535 L. Burzawa, S. Liu and E. W. Carlson, Classifying 554 A. Mathuriya, et al., CosmoFlow: Using deep learning to surface probe images in strongly correlated electronic learn the universe at scale, in Proc. – Int. Conf. High systems via machine learning, Phys. Rev. Mater., 2019, Perform. Comput. Networking, Storage, Anal. SC 2018, 3, 033805. 2019, pp. 819–829, DOI: 10.1109/SC.2018.00068. 536 K. Choudhary, et al., Density Functional Theory and Deep- 555 S. Ravanbakhsh, et al., Estimating cosmological para- learning to Accelerate Data Analytics in Scanning Tunnel- meters from the dark matter distribution, in 33rd Int. Conf. ing Microscopy, arXiv, 2019, preprint, arXiv:1912.09027, Mach. Learn. ICML 2016, 2016, vol. 5, pp. 3584–3594. DOI: 10.48550/arXiv.1912.09027. 556 R. A. de Oliveira, Y. Li, F. Villaescusa-Navarro, S. Ho and 537 N. Borodinov, et al., Spectral Map Reconstruction Using D. N. Spergel, Fast and Accurate Non-Linear Predictions of Pan-Sharpening Algorithm: Enhancing Chemical Imaging Universes with Deep Learning, arXiv, 2020, preprint, with AFM-IR, Microsc. Microanal., 2019, 25, 1024–1025. arXiv:2012.00240, DOI: 10.48550/arXiv.2012.00240. 538 M. Rashidi and R. A. Wolkow, Autonomous Scanning 557 F. Villaescusa-Navarro, et al., The CAMELS Project: Cos- Probe Microscopy in Situ Tip Conditioning through mology and Astrophysics with Machine-learning Simula- Machine Learning, ACS Nano, 2018, 12, 5185–5189. tions, Astrophys. J., 2021, 915, 71. 539 S. Wang, J. Zhu, R. Blackwell and F. R. Fischer, Automated 558 A. M. Delgado, et al., Modeling the galaxy-halo connection tip conditioning for scanning tunneling spectroscopy, with machine learning, Monthly Notices of the Royal Astro- J. Phys. Chem. A, 2021, 125, 1384–1390. nomical Society, 2022, 515, 2733–2746. 540 B. Li, et al., Fabricating ultra-sharp tungsten STM tips with 559 R. Garnett, S. Ho and J. Schneider, Finding galaxies in the high yield: double-electrolyte etching method and shadows of quasars with Gaussian processes, in 32nd Int. machine learning, SN Appl. Sci., 2020, 2, 1246. Conf. Mach. Learn. ICML 2015, 2015, 2, 1025–1033. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1473 View Article Online Review Nanoscale Horizons 560 T. E. Collett, The population of galaxy-galaxy strong lenses 577 A. Radovic, et al., Machine learning at the energy and intensity in forthcoming optical imaging surveys, Astrophys. J., 2015, frontiers of particle physics, Nature, 2018, 560, 41–48. 811, 20. 578 Y. Zhang, et al., Machine learning in electronic-quantum- 561 A. Askar, A. Askar, M. Pasquato and M. Giersz, Finding matter imaging experiments, Nature, 2019, 570, 484–490. black holes with black boxes – Using machine learning to 579 D. Turvill, L. Barnby, B. Yuan and A. Zahir, A Survey of identify globular clusters with black hole subsystems, Mon. Interpretability of Machine Learning in Accelerator-based Not. R. Astron. Soc., 2019, 485, 5345–5362. High Energy Physics, in Proc. – 2020 IEEE/ACM Int. 562 J. Brehmer, S. Mishra-Sharma, J. Hermans, G. Louppe and Conf. Big Data Comput. Appl. Technol. BDCAT 2020, 2020, K. Cranmer, Mining for Dark Matter Substructure: Infer- pp. 77–86, DOI: 10.1109/BDCAT50828.2020.00025. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. ring Subhalo Population Properties from Strong Lenses 580 A. Andreassen, I. Feige, C. Frye and M. D. Schwartz, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. with Machine Learning, Astrophys. J., 2019, 886, 49. JUNIPR: a framework for unsupervised machine learning 563 P. G. Krastev, Real-time detection of gravitational waves in particle physics, Eur. Phys. J. C, 2019, 79, 102. from binary neutron stars using artificial neural networks, 581 J. Brehmer, F. Kling, I. Espejo and K. Cranmer, MadMiner: Phys. Lett. Sect. B: Nucl. Elem. Part. High-Energy Phys., 2020, Machine Learning-Based Inference for Particle Physics, 803, 135330. Comput. Softw. Big Sci., 2020, 4, 3. 564 V. A. Villar, et al., A Deep-learning Approach for Live 582 L. Del Debbio, S. Forte, J. I. Latorre, A. Piccione and J. Rojo, Anomaly Detection of Extragalactic Transients, Astrophys. Neural network determination of parton distributions: The J., Suppl. Ser., 2021, 255, 24. nonsinglet case, J. High Energy Phys, 2007, 2007, 039. 565 D. Schmidt, B. Messer, M. T. Young and M. Matheson, 583 D. Guest, K. Cranmer and D. Whiteson, Deep learning and Towards the Development of Entropy-Based Anomaly its application to LHC physics, Annu. Rev. Nucl. Part. Sci., Detection in an Astrophysics Simulation, arXiv, 2020, pre- 2018, 68, 161–181. print, arXiv:2009.02430, DOI: 10.48550/arXiv.2009.02430. 584 E. Govorkova, et al., Autoencoders on field-programmable 566 D. Giles and L. Walkowicz, Systematic serendipity: A test of gate arrays for real-time, unsupervised new physics detec- unsupervised machine learning as a method for anomaly tion at 40 MHz at the Large Hadron Collider, Nat. Mach. detection, Mon. Not. R. Astron. Soc., 2019, 484, 834–849. Intell., 2022, 4, 154–161. 567 B. Hoyle, et al., Anomaly detection for machine learning 585 P. T. Komiske, E. M. Metodiev and M. D. Schwartz, Deep redshifts applied to SDSS galaxies, Mon. Not. R. Astron. learning in color: towards automated quark/gluon jet Soc., 2015, 452, 4183–4194. discrimination, J. High Energy Phys., 2017, 110. 568 Asteroid Watch, available at: https://www.jpl.nasa.gov/ 586 S. Egan, W. Fedorko, A. Lister, J. Pearkes and C. Gay, Long asteroid-watch (accessed: 8th February 2022). Short-Term Memory (LSTM) networks with jet constituents 569 M. Jara-Maldonado, V. Alarcon-Aquino, R. Rosas-Romero, for boosted top tagging at the LHC, arXiv, 2017, 3–8. O. Starostenko and J. M. Ramirez-Cortes, Transiting Exo- 587 L. de Oliveira, M. Paganini and B. Nachman, Learning planet Discovery Using Machine Learning Techniques: A Particle Physics by Example: Location-Aware Generative Survey, Earth Sci. Informat., 2020, 13, 573–600. Adversarial Networks for Physics Synthesis, Comput. Softw. 570 P. Márquez-Neila, C. Fisher, R. Sznitman and K. Heng, Big Sci., 2017, 1, 4. Supervised machine learning for analysing spectra of 588 M. Paganini, L. De Oliveira and B. Nachman, Accelerating exoplanetary atmospheres, Nat. Astron., 2018, 2, 719–724. Science with Generative Adversarial Networks: An Applica- 571 N. Schanche, et al., Machine-learning approaches to exoplanet tion to 3D Particle Showers in Multilayer Calorimeters, transit detection and candidate validation in wide-field ground- Phys. Rev. Lett., 2018, 120, 042003. based surveys, Mon. Not. R. Astron. Soc., 2019, 483, 5534–5547. 589 M. Paganini, L. De Oliveira and B. Nachman, CaloGAN: 572 I. Priyadarshini and V. Puri, A convolutional neural Simulating 3D high energy particle showers in multilayer network (CNN) based ensemble model for exoplanet electromagnetic calorimeters with generative adversarial detection, Earth Sci. Informat., 2021, 14, 735–747. networks, Phys. Rev. D, 2018, 97, 014021. 573 P. Chintarungruangchai and I. G. Jiang, Detecting 590 K. K. Sharma, Quantum machine learning and its supre- exoplanet transits through machine-learning techniques macy in high energy physics, Mod. Phys. Lett. A, 2021, with convolutional neural networks, Publ. Astron. Soc. Pac., 36, 2030024. 2019, 131, 64502. 591 W. Guan, et al., Quantum machine learning in high energy 574 D. Cecil and M. Campbell-Brown, The application of con- physics, Mach. Learn. Sci. Technol., 2021, 2, 011003. volutional neural networks to the automation of a meteor 592 A. Blance and M. Spannowsky, Quantum machine learning detection pipeline, Planet. Space Sci., 2020, 186, 104920. for particle physics using a variational quantum classifier, 575 M. Lieu, L. Conversi, B. Altieri and B. Carry, Detecting Solar J. High Energy Phys., 2021, 2021, 212. system objects with convolutional neural networks, Mon. 593 S. Y.-C. Chen, T.-C. Wei, C. Zhang, H. Yu and S. Yoo, Not. R. Astron. Soc., 2019, 485, 5831–5842. Quantum Convolutional Neural Networks for High Energy 576 K. Albertsson, et al., Machine Learning in High Energy Physics Data Analysis, Phys. Rev. Res., 2020, 4, 013231. Physics Community White Paper, J. Phys. Conf. Ser., 2018, 594 I. Cong, S. Choi and M. D. Lukin, Quantum convolutional 1085, 022008. neural networks, Nat. Phys., 2019, 15, 1273–1278. 1474 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 595 G. Beach, C. Lomont and C. Cohen, Quantum image 611 Y. Liu, et al., Application of Deep Convolutional Neural processing (QuIP), in Proc. – Appl. Imag. Pattern Recognit. Networks for Detecting Extreme Weather in Climate Data- Work. 2003-Janua, 2004, pp. 39–44. sets, arXiv, 2016, preprint, arXiv:1605.01156, DOI: 10.48550/ 596 F. Yan, A. M. Iliyasu and P. Q. Le, Quantum image arXiv.1605.01156. processing: A review of advances in its security technolo- 612 P. A. O’Gorman and J. G. Dwyer, Using Machine Learning gies, Int. J. Quantum Inf., 2017, 15, 1730001. to Parameterize Moist Convection: Potential for Modeling 597 Y. S. Weinstein, M. A. Pravia, E. M. Fortunato, S. Lloyd and of Climate, Climate Change, and Extreme Events, J. Adv. D. G. Cory, Implementation of the quantum Fourier trans- Model. Earth Syst., 2018, 10, 2548–2563. form, Phys. Rev. Lett., 2001, 86, 1889–1891. 613 K. J. Bergen, P. A. Johnson, M. V. De Hoop and G. C. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 598 J. J. G. Ripoll, Quantum-inspired algorithms for multi- Beroza, Machine learning for data-driven discovery in solid Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. variate analysis: From interpolation to partial differential Earth geoscience, Science, 2019, 363, eaau0323. equations, Quantum, 2021, 5, 431. 614 H. Maniar, S. Ryali, M. S. Kulkarni and A. Abubakar 599 X. W. Yao, et al., Quantum image processing and its Machine learning methods in Geoscience, in 2018 SEG application to edge detection: Theory and experiment, Int. Expo. Annu. Meet. SEG 2018, 2019, pp. 4638–4642, DOI: Phys. Rev. X, 2017, 7, 031041. 10.1190/segam2018-2997218.1. 600 W. W. Zhang, F. Gao, B. Liu, Q. Y. Wen and H. Chen, A 615 A. Karpatne, I. Ebert-Uphoff, S. Ravela, H. A. Babaie and watermark strategy for quantum images based on quantum V. Kumar, Machine Learning for the Geosciences: Chal- Fourier transform, Quantum Inf. Process., 2013, 12, 793–803. lenges and Opportunities, IEEE Trans. Knowl. Data Eng., 601 G. Camps-Valls, D. Tuia, X. X. Zhu and M. Reichstein, Deep 2019, 31, 1544–1554. Learning for the Earth Sciences: A Comprehensive Approach to 616 B. Rouet-Leduc, et al., Machine Learning Predicts Labora- Remote Sensing, Climate Science and Geosciences, Wiley, tory Earthquakes, Geophys. Res. Lett., 2017, 44, 9276–9282. 2021. 617 K. M. Asim, F. Martı́nez-Álvarez, A. Basit and T. Iqbal, 602 J. H. Faghmous and V. Kumar, A Big Data Guide to Earthquake magnitude prediction in Hindukush region Understanding Climate Change: The Case for Theory- using machine learning techniques, Nat. Hazards, 2017, Guided Data Science, Big Data, 2014, 2, 155–163. 85, 471–486. 603 V. Lakshmanan, E. Gilleland, A. McGovern and M. Tingley, 618 F. Corbi, et al., Machine Learning Can Predict the Timing Machine Learning and Data Mining Approaches to Climate and Size of Analog Earthquakes, Geophys. Res. Lett., 2019, Science. Machine Learning and Data Mining Approaches to 46, 1303–1311. Climate Science, Springer International Publishing, 2015, 619 S. M. Mousavi and G. C. Beroza, A Machine-Learning DOI: 10.1007/978-3-319-17220-0. Approach for Earthquake Magnitude Estimation, Geophys. 604 C. Monteleoni, G. A. Schmidt and S. Mcquade, Climate Res. Lett., 2020, 47, e2019GL085976. Informatics: Accelerating Discovering in Climate Science 620 C. Hulbert, et al., Similarity of fast and slow earthquakes with Machine Learning, Comput. Sci. Eng., 2013, 32–40. illuminated by machine learning, Nat. Geosci., 2019, 12, 605 F. V. Davenport and N. S. Diffenbaugh, Using Machine 69–74. Learning to Analyze Physical Causes of Climate Change: A 621 I. M. Murwantara, P. Yugopuspito and R. Hermawan, Case Study of U.S. Midwest Extreme Precipitation, Comparison of machine learning performance for earth- Geophys. Res. Lett., 2021, 48, e2021GL093787. quake prediction in Indonesia using 30 years historical 606 I. Ebert-Uphoff and Y. Deng, Causal discovery from spatio- data, Telkomnika, 2020, 18, 1331–1342. temporal data with applications to climate science, in Proc. – 622 K. M. Asim, et al., Seismicity analysis and machine learning 2014 13th Int. Conf. Mach. Learn. Appl. ICMLA 2014, 2014, models for short-term low magnitude seismic activity pre- pp. 606–613, DOI: 10.1109/ICMLA.2014.96. dictions in Cyprus, Soil Dyn. Earthq. Eng., 2020, 130, 105932. 607 K. Kashinath, et al., ClimateNet: An expert-labeled open 623 W. Wang and K. Siau, Artificial intelligence, machine dataset and deep learning architecture for enabling high- learning, automation, robotics, future of work and future precision analyses of extreme weather, Geosci. Mod. Dev., of humanity: A review and research agenda, J. Database 2021, 14, 107–124. Manage., 2019, 30, 61–79. 608 J. E. Shortridge, S. D. Guikema and B. F. Zaitchik, Machine 624 M. Bowling, J. Fürnkranz, T. Graepel and R. Musick, Machine learning methods for empirical streamflow simulation: A learning and games, Mach. Learn., 2006, 63, 211–215. comparison of model accuracy, interpretability, and uncer- 625 C. Bauckhage and C. Thurau, Exploiting the Fascination: tainty in seasonal watersheds, Hydrol. Earth Syst. Sci., 2016, Video Games in Machine Learning Research and Educa- 20, 2611–2628. tion, in Proc. Int. Conf. Comput. Game Des. Technol., 2004, 609 X. Ren, et al., A Simplified Climate Change Model and pp. 61–70. Extreme Weather Model Based on a Machine Learning 626 Y. Kassahun, et al., Surgical robotics beyond enhanced Method, Symmetry, 2020, 12, 139. dexterity instrumentation: a survey of machine learning 610 H. Hu and B. M. Ayyub, Machine learning for projecting techniques and their role in intelligent and autonomous extreme precipitation intensity for short durations in a surgical actions, Int. J. Comput. Assist. Radiol. Surg., 2016, changing climate, Geoscience, 2019, 9, 209. 11, 553–568. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1475 View Article Online Review Nanoscale Horizons 627 F. Richter, R. K. Orosco and M. C. Yip, Open-Sourced 644 G. Brockman, et al., OpenAI Gym., arXiv, 2016, preprint, Reinforcement Learning Environments for Surgical Robotics, arXiv:1606.01540, DOI: 10.48550/arXiv.1606.01540. arXiv, 2019, preprint, arXiv:1903.02090, DOI: 10.48550/ 645 OpenAI, OpenAI Gym, available at: https://gym.openai. arXiv.1903.02090. com/ (accessed: 13th January 2022). 628 R. Shimizu, S. Kobayashi, Y. Watanabe, Y. Ando and 646 DeepMind, DeepMind, available at: https://deepmind. T. Hitosugi, Autonomous materials synthesis by machine com/ (accessed: 14th January 2022). learning and robotics, APL Mater., 2020, 8, 2–8. 647 A. Ulvestad, et al., Identifying Defects with Guided Algo- 629 J. P. Correa-Baena, et al., Accelerating Materials Develop- rithms in Bragg Coherent Diffractive Imaging, Sci. Rep., ment via Automation, Machine Learning, and High- 2017, 7, 9920. This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. Performance Computing, Joule, 2018, 2, 1410–1420. 648 J. Brehmer, K. Cranmer, G. Louppe and J. Pavez, Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. 630 C. M. Lin, C. Y. Tsai, Y. C. Lai, S. A. Li and C. C. Wong, Constraining Effective Field Theories with Machine Learning, Visual Object Recognition and Pose Estimation Based on a Phys. Rev. Lett., 2018, 121, 111801. Deep Semantic Segmentation Network, IEEE Sens. J., 2018, 649 J. Brehmer, K. Cranmer, G. Louppe and J. Pavez, A guide 18, 9370–9381. to constraining effective field theories with machine learn- 631 N. Bredeche, Z. Shi and J. D. Zucker, Perceptual learning ing, Phys. Rev. D, 2018, 98, 52004. and abstraction in machine learning: An application to 650 H. Chan, et al., Rapid 3D nanoscale coherent imaging via autonomous robotics, IEEE Trans. Syst. Man Cybern. Part C: physics-aware deep learning, Appl. Phys. Rev., 2021, Appl. Rev., 2006, 36, 172–181. 8, 021407. 632 C. Y. Lee, H. Lee, I. Hwang and B. T. Zhang, Visual 651 Y. Yao, et al., AutoPhaseNN: Unsupervised Physics-aware Perception Framework for an Intelligent Mobile Robot, Deep Learning of 3D Nanoscale Coherent Imaging, npj in 2020 17th Int. Conf. Ubiquitous Robot. UR 2020, 2020, Comput. Mater., 2022, 8, 124. pp. 612–616, DOI: 10.1109/UR49135.2020.9144932. 652 M. Dijkstra and E. Luijten, From predictive modelling to 633 G. Shan, T. Wang, X. Li, Y. Fang and Y. Zhang, A Deep machine learning and reverse engineering of colloidal self- Learning-based Visual Perception Approach for Mobile assembly, Nat. Mater., 2021, 20, 762–773. Robots, in Proc. 2018 Chinese Autom. Congr. CAC 2018, 653 J. Hoffmann, et al., Data-Driven Approach to Encoding and 2019, pp. 825–829, DOI: 10.1109/CAC.2018.8623665. Decoding 3-D Crystal Structures, arXiv, 2019, preprint, 634 A. S. Polydoros and L. Nalpantidis, Survey of Model-Based arXiv:1909.00949, DOI: 10.48550/arXiv.1909.00949. Reinforcement Learning: Applications on Robotics, J. Intell. 654 K. Cranmer, J. Brehmer and G. Louppe, The frontier of Robot. Syst. Theory Appl., 2017, 86, 153–173. simulation-based inference, Proc. Natl. Acad. Sci. U. S. A., 635 J. Togelius, Playing Smart: On Games, Intelligence, and 2020, 117, 30055–30062. Artificial Intelligence, The MIT Press, 2019, vol. 6. 655 D. Liu, Y. Tan, E. Khoram and Z. Yu, Training Deep Neural 636 K. O. Stanley and R. Miikkulainen, Evolving neural networks Networks for the Inverse Design of Nanophotonic Structures, through augmenting topologies, Evol. Comput., 2002, 10, ACS Photonics, 2018, 5, 1365–1369. 99–127. 656 J. Zhou, et al., Graph neural networks: A review of methods 637 J. Drozdal, et al., Trust in AutoML: Exploring Information and applications, AI Open, 2020, 1, 57–81. Needs for Establishing Trust in Automated Machine 657 Y. Shen and X. Wang, Person Re-identification with Deep Learning Systems, Proceedings of the 25th International Similarity-Guided Graph Neural Network, Proc. Eur. Conf. Conference on Intelligent User Interfaces, 2020, pp. 297–307, Comput. Vis., 2018, 236, 1567–1570. DOI: 10.1145/3377325.3377501. 658 D. Lin, J. Lin, L. Zhao, Z. J. Wang and Z. Chen, Multilabel 638 F. Hutter, L. Kotthoff and J. Vanschoren, Automated Aerial Image Classification with a Concept Attention machine learning: Methods, Systems, Challenges, Springer, Graph Neural Network, IEEE Trans. Geosci. Remote Sens., 2019, DOI: 10.1515/9783110629453-084. 2022, 60, 1–12. 639 Q. Yao, et al., Taking Human out of Learning Applications: 659 X. Qi, R. Liao, J. Jia, S. Fidler and R. Urtasun, 3D Graph A Survey on Automated, Mach. Learn., 2018, 104, 148–175. Neural Networks for RGBD Semantic Segmentation, Proc. 640 J. Won, D. Gopinath and J. Hodgins, Control strategies for IEEE Int. Conf. Comput. Vis., 2018, 726–732. physically simulated characters performing two-player 660 V. A. Brei, Machine Learning in Marketing: Overview, competitive sports, ACM Trans. Graph., 2021, 40, 145. Learning Strategies, Applications, and Future Developments, 641 A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley and J. Clune, Found. Trends Mark., 2020, 14, 173–236. Go-Explore: a New Approach for Hard-Exploration Problems, 661 E. A. Gerlein, M. McGinnity, A. Belatreche and S. Coleman, arXiv, 2019, preprint, arXiv:1901.10995, DOI: 10.48550/ Evaluating machine learning classification for financial arXiv.1901.10995. trading: An empirical approach, Expert Syst. Appl., 2016, 642 V. Mnih, et al., Human-level control through deep reinfor- 54, 193–207. cement learning, Nature, 2015, 518, 529–533. 662 B. Krollner, B. Vanstone and G. Finnie, Financial time 643 B. Baker, et al., Emergent Tool Use From Multi-Agent series forecasting with machine learning techniques: A Autocurricula, arXiv, 2020, preprint, arXiv:1909.07528, survey, in Proc. 18th Eur. Symp. Artif. Neural Networks - DOI: 10.48550/arXiv.1909.07528. Comput. Intell. Mach. Learn. ESANN 2010, 2010, pp. 25–30. 1476 | Nanoscale Horiz., 2022, 7, 1427–1477 This journal is © The Royal Society of Chemistry 2022 View Article Online Nanoscale Horizons Review 663 L. Ma and B. Sun, Machine learning and AI in marketing – 667 M. Jaeger, et al., Interpretable Machine Learning for Diver- Connecting computing power to human insights, Int. sified Portolio Construction, J. Fin. Data Sci., 2021, 3, J. Res. Mark., 2020, 37, 481–504. 31–51. 664 A. Miklosik, M. Kuchta, N. Evans and S. Zak, Towards the 668 J. Alcazar, V. Leyton-Ortega and A. Perdomo-Ortiz, Adoption of Machine Learning-Based Analytical Tools in Classical versus quantum models in machine learning: Digital Marketing, IEEE Access, 2019, 7, 85705–85718. Insights from a finance application, Mach. Learn. Sci. 665 Z. Jiang, D. Xu and J. Liang, A Deep Reinforcement Learn- Technol., 2020, 1, 035003. ing Framework for the Financial Portfolio Management 669 R. Orús, S. Mugel and E. Lizaso, Quantum computing Problem, Deep Portf. Manage., 2017, 1–31. for finance: Overview and prospects, Rev. Phys., 2019, This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. 666 F. D. Paiva, R. T. N. Cardoso, G. P. Hanaoka and W. M. 4, 100028. Open Access Article. Published on 14 October 2022. Downloaded on 2/1/2023 10:04:09 AM. Duarte, Decision-making for financial trading: A fusion 670 D. Venturelli and A. Kondratyev, Reverse quantum annealing approach of machine learning and portfolio selection, approach to portfolio optimization problems, Quantum Expert Syst. Appl., 2019, 115, 635–655. Mach. Intell., 2019, 1, 17–30. This journal is © The Royal Society of Chemistry 2022 Nanoscale Horiz., 2022, 7, 1427–1477 | 1477