George Karniadakis, Brown University
(Approved for Public Release)
We propose to develop generalized neural networks (NNs) that learn nonlinear multi-operators and can solve problems at a higher level of abstraction and 1,000 times faster, hence developing the next generation of neural operators such as DeepOnet, a biologically-inspired NN we introduced recently for real-time forecasting of multiphysics systems. The overarching goal is to develop new algorithms for biologically-inspired neural operators that can be implemented in the next generation of neuromorphic computers. The proposed work is to develop technology for both military and civil applications.
Building on our preliminary theoretical results on the universal approximation properties of DeepOnet, we plan to first extend the theory to multi-operator regression architectures for multi-modal inputs, inspired by dendritic branching of human neurons, and resolve the issue of catastrophic forgetting in order to enable multi-tasking and continual learning. We will develop corresponding approximation and generalization theory, focusing on the curse of-dimensionality in the multi-modal input space and examining system-stability subject to random and adversarial perturbations. In addition to approximation and generalization, we will address optimization related to the critical issue of efficiency and energetics, first studying promising fractional gradient methods but more importantly going beyond back propagation. Specifically, we will introduce alternative methods to the current backpropagation such as local methods and weight mirrors, and explore biologically-plausible models, e.g.,
Hebbian Learning for forward-only weight updates. Spiking Neural Networks (SNNs) have emerged as biologically plausible methods with significantly lower computational cost but the lack of smoothness of the spiking signals has led to fundamental difficulties. We will investigate the bio-inspired spike time-dependent backpropagation (STDB) incremental training although current results show sub-optimal performance compared to artificial neural networks (ANNs). To this end, we propose to investigate a combination of methods such as the ANN-SNN conversion for initialization and STDB for final convergence, as well as new batch-normalization-through-time techniques. We also plan to investigate NeuroEvolution of Augmenting Topologies (NEAT) to learn simultaneously both the ideal topology and weights in SNNs.
On the application front, we will consider real-time forecasting of complex multi-physics systems (e.g., hypersonics and autonomy), and we will combine deep learning with deep reasoning (e.g., a DeepOnet that includes Relation Networks) for interactive design and for predicting human behavior in social dynamics. Developing higher level abstractions for deep learning, addressing the exponentially rising costs of inefficient training, and incorporating reasoning in DeepOnet will lead to a paradigm shift in scientific machine learning and more broadly to AI. These fundamental developments will have an immediate technological impact on autonomy, design, humanrobot interactions, which are critical areas to DoD, but also societal impact, e.g., quantifying human dynamics.
We plan to work with neuroscientists and cognitive scientists, the computer industry, other VBF fellows, and with researchers from ARL, NRL, AFRL, and the DOE labs. The PI will develop new curricula and tutorials for training a new cadre of simulation scientists on “mathematics + machine learning + X”, including under-represented groups. He will organize the 2023 MSML (Mathematical and Scientific Machine Learning) conference at Brown University, which will include tutorials both on theory and practical implementation of machine learning tools.