CECAM Flagship Workshop: From operando electron microscopy images to atomistic models: Machine Learning assisted analysis in the age of big data

The advent of in situ and operando analytical techniques in material science has allowed uncovering the importance of structural dynamics in research fields such as heterogeneous catalysis [1], nucleation and growth of crystals [2] or proteins [3]. This holds, in particular for electron microscopy related operando approaches, true to the slogan: “seeing is believing" [4]. In general, operando electron microscopy experiments cause relatively large data stacks, for which the analysis per se is time consuming and the statistical relevance and quantitative analysis is often missing. Consequently, a correlation of important features with the function is often impossible. In addition, the interaction of the highly energetic electrons with the liquid or gaseous environment forms reactive radicals that can interact with the matter under study and alter its atomistic structure [5]. Thus, a true atomistic description, which can be obtained from high resolution ex situ analysis [6], is not directly possible. However, this knowledge would be required to understand the origin of the function of the investigated material. Consequently, to reduce the harmful electron-environment interaction to a negligible minimum often operando electron microscopy experiments are conducted at lower resolution or with a bad signal-to-noise ratio [4].On the theory side, ML-enabled techniques have recently allowed for unprecedented modelling of complex functional materials systems by molecular dynamics and advanced sampling techniques [7]. Compared to biophysical simulations of e.g. molecular machines, detailed experimental structural data that would allow for in-depth validation of models of interfaces in catalysis or energy-conversion systems is largely missing and urgently required.To overcome the outlined dilemma, machine learning aided computervision approaches have been identified to be of outmost importance in enhancing the interpretative depths of operando electron microscopy experiments [8-10]. In addition, they are able to generate automated atomistic models from huge operando electron microscopy data stacks recorded at low resolution or signal-to-noise ratio. Atomistic simulations can aid in detecting unlikely outliers or act as reqularization to such approaches, building on related ideas that e.g. allowed for NMR structure prediction of non-crystalizable proteins.The applied machine learning approaches rely on neural network algorithm, such as convolution neural network (CNN), fully convolution network (FCN) or U-Net [8], which enable novel insights via automated analysis into defects, structure, morphology and spectral features. Consequently, linking electron microscopy to integrated machine learning and atomistic simulation approaches enhances not only the quantitative and statistical expression and the temporal evolution of the feature under study, but also the generation of realistic atomistic models towards a more accurate simulation of the function of the material under study. For instance, CNN assisted data evaluation has generated a 3D atomistic model of nanoparticles with the precise determination of the thickness [11]. In addition, applying a U-Net algorithm for Au nanoparticles segmentation during an in situ transmission electron microscopy (TEM) experiment has disclosed the curvature dependent edging rate of these particles [12]. To ultimately automatize experiments, recent advances in interpretable machine learning can be of direct impact [13]. These architectures can permit to discover governing equations, which could serve for controlling experimental apparatus [14].

[1] J. Thomas, K. Zamaraev, Top. Catal., 1, 1-8 (1994)
[2] Z. Wang, C. Liu, Q. Chen, Journal of Crystal Growth, 601, 126955 (2023)
[3] E. Sisley, E. Illes-Toth, H. Cooper, TrAC Trends in Analytical Chemistry, 124, 115534 (2020)
[4] S. Chee, T. Lunkenbein, R. Schlögl, B. Roldán Cuenya, Chem. Rev., 123, 13374-13418 (2023)
[5] N. Schneider, M. Norton, B. Mendel, J. Grogan, F. Ross, H. Bau, J. Phys. Chem. C, 118, 22373-22382 (2014)
[6] H. Türk, F. Schmidt, T. Götsch, F. Girgsdies, A. Hammud, D. Ivanov, I. Vinke, L. de Haart, R. Eichel, K. Reuter, R. Schlögl, A. Knop‐Gericke, C. Scheurer, T. Lunkenbein, Adv. Materials. Inter., 8, (2021)
[7] H. Jung, L. Sauerland, S. Stocker, K. Reuter, J. Margraf, npj. Comput. Mater., 9, 114 (2023)
[8] S. Kalinin, D. Mukherjee, K. Roccapriore, B. Blaiszik, A. Ghosh, M. Ziatdinov, A. Al-Najjar, C. Doty, S. Akers, N. Rao, J. Agar, S. Spurgeon, npj. Comput. Mater., 9, 227 (2023)
[9] Z. Cheng, C. Wang, X. Wu, J. Chu, J. Semicond., 43, 081001 (2022)
[10] M. Botifoll, I. Pinto-Huguet, J. Arbiol, Nanoscale Horiz., 7, 1427-1477 (2022)
[11] M. Ragone, V. Yurkiv, B. Song, A. Ramsubramanian, R. Shahbazian-Yassar, F. Mashayek, Computational Materials Science, 180, 109722 (2020)
[12] L. Yao, Z. Ou, B. Luo, C. Xu, Q. Chen, ACS Cent. Sci., 6, 1421-1430 (2020)
[13] S. Brunton, J. Proctor, J. Kutz, Proc. Natl. Acad. Sci. U.S.A., 113, 3932-3937 (2016)
[14] M. Hoffmann, C. Fröhner, F. Noé, The Journal of Chemical Physics, 150, (2019)