SU\G(𝔸)/K·U
Sign Up for free Log In

Author Description

Login to generate an author description

Ask a Question About This Mathematician

Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives

2023-01-01
Yahong Yang , Haizhao Yang , Yang Xiang
This paper addresses the problem of nearly optimal Vapnik--Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: … This paper addresses the problem of nearly optimal Vapnik--Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: 1) Establishing a nearly tight approximation result of DNNs in the Sobolev space; 2) Characterizing the generalization error of machine learning methods with loss functions involving function derivatives. This theoretical investigation fills the gap of learning error estimations for a wide range of physics-informed machine learning models and applications including generative models, solving partial differential equations, operator learning, network compression, distillation, regularization, etc.
View PDF Chat

Nearly Optimal Approximation Rates for Deep Super ReLU Networks on Sobolev Spaces

2023-01-01
Yahong Yang , Yue Wu , Haizhao Yang +1 more
This paper introduces deep super ReLU networks (DSRNs) as a method for approximating functions in Sobolev spaces measured by Sobolev norms $W^{m,p}$ for $m\in\mathbb{N}$ with $m\ge 2$ and $1\le p\le … This paper introduces deep super ReLU networks (DSRNs) as a method for approximating functions in Sobolev spaces measured by Sobolev norms $W^{m,p}$ for $m\in\mathbb{N}$ with $m\ge 2$ and $1\le p\le +\infty$. Standard ReLU deep neural networks (ReLU DNNs) cannot achieve this goal. DSRNs consist primarily of ReLU DNNs, and several layers of the square of ReLU added at the end to smooth the networks output. This approach retains the advantages of ReLU DNNs, leading to the straightforward training. The paper also proves the optimality of DSRNs by estimating the VC-dimension of higher-order derivatives of DNNs, and obtains the generalization error in Sobolev spaces via an estimate of the pseudo-dimension of higher-order derivatives of DNNs.
View PDF Chat

Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations

2024-05-22
Wenrui Hao , Xinliang Liu , Yahong Yang
Solving nonlinear partial differential equations (PDEs) with multiple solutions using neural networks has found widespread applications in various fields such as physics, biology, and engineering. However, classical neural network methods … Solving nonlinear partial differential equations (PDEs) with multiple solutions using neural networks has found widespread applications in various fields such as physics, biology, and engineering. However, classical neural network methods for solving nonlinear PDEs, such as Physics-Informed Neural Networks (PINN), Deep Ritz methods, and DeepONet, often encounter challenges when confronted with the presence of multiple solutions inherent in the nonlinear problem. These methods may encounter ill-posedness issues. In this paper, we propose a novel approach called the Newton Informed Neural Operator, which builds upon existing neural network techniques to tackle nonlinearities. Our method combines classical Newton methods, addressing well-posed problems, and efficiently learns multiple solutions in a single learning process while requiring fewer supervised data points compared to existing neural network methods.
View PDF Chat

Near-optimal deep neural network approximation for Korobov functions with respect to <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e79" altimg="si14.svg"><mml:msup><mml:mrow><mml:mi>L</mml:mi></mml:mrow><mml:mrow><mml:mi>p</mml:mi></mml:mrow></mml:msup></mml:math> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e89" altimg="si15.svg"><mml:msup><mml:mrow><mml:mi>H</mml:mi></mml:mrow><mml:mrow><mml:mn>1</mml:mn></mml:mrow></mml…

2024-09-06
Yahong Yang , Yulong Lu

Approximation of Functionals by Neural Network without Curse of Dimensionality

2022-01-01
Yahong Yang , Yang Xiang
In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces. The approximation error of the neural network is … In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces. The approximation error of the neural network is $O(1/\sqrt{m})$ where $m$ is the size of networks, which overcomes the curse of dimensionality. The key idea of the approximation is to define a Barron spectral space of functionals.
View PDF Chat

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

2024-01-31
Yahong Yang , Juncai He
Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. … Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. This paper explores a comparison between deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers, focusing on their optimal generalization error in Sobolev losses. Analytical investigations reveal that the architecture of a neural network can be significantly influenced by various factors, including the number of sample points, parameters within the neural networks, and the regularity of the loss function. Specifically, a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs. We ultimately apply this theory to address partial differential equations using deep Ritz and physics-informed neural network (PINN) methods, guiding the design of neural networks.
View PDF Chat

Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations

2024-05-22
Chuqi Chen , Yahong Yang , Xiang Yang +1 more
Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or the incorporation of empirical data. … Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or the incorporation of empirical data. One advantage of the neural network method for PDEs lies in its automatic differentiation (AD), which necessitates only the sample points themselves, unlike traditional finite difference (FD) approximations that require nearby local points to compute derivatives. In this paper, we quantitatively demonstrate the advantage of AD in training neural networks. The concept of truncated entropy is introduced to characterize the training property. Specifically, through comprehensive experimental and theoretical analyses conducted on random feature models and two-layer neural networks, we discover that the defined truncated entropy serves as a reliable metric for quantifying the residual loss of random feature models and the training speed of neural networks for both AD and FD methods. Our experimental and theoretical analyses demonstrate that, from a training perspective, AD outperforms FD in solving partial differential equations.
View PDF Chat

Convergence from atomistic model to Peierls–Nabarro model for dislocations in bilayer system with complex lattice

2022-01-01
Yahong Yang , Tao Luo , Yang Xiang
In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of … In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of the dislocation solution of the PN model converges to the dislocation solution of the atomistic model with second-order accuracy. The consistency of PN model and the stability of atomistic model are essential in our proof. The main idea of our approach is to use several low-degree polynomials to approximate the energy due to atomistic interactions of different groups of atoms of the complex lattice.
View PDF Chat

Stochastic Continuum Models for High--Entropy Alloys with Short-range Order

2022-01-01
Yahong Yang , Luchan Zhang , Yang Xiang
High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in … High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in the atomic level randomness strongly influences the properties of HEAs. In this paper, we derive stochastic continuum models for HEAs with short-range order from atomistic models. A proper continuum limit is obtained such that the mean and variance of the atomic level randomness together with the short-range order described by a characteristic length are kept in the process from the atomistic interaction model to the continuum equation. The obtained continuum model with short-range order is in the form of an Ornstein--Uhlenbeck (OU) process. This validates the continuum model based on the OU process adopted phenomenologically by Zhang et al. [Acta Mater., 166 (2019), pp. 424--434] for HEAs with short-range order. We derive such stochastic continuum models with short-range order for both elasticity in HEAs without defects and HEAs with dislocations (line defects). The obtained stochastic continuum models are based on the energy formulations, whose variations lead to stochastic partial differential equations.
View PDF Chat

Convergence from Atomistic Model to Peierls-Nabarro Model for Dislocations in Bilayer System with Complex Lattice

2021-01-01
Yahong Yang , T. Luo , Yang Xiang
In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of … In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of the dislocation solution of the PN model converges to the dislocation solution of the atomistic model with second-order accuracy. The consistency of PN model and the stability of atomistic model are essential in our proof. The main idea of our approach is to use several low-degree polynomials to approximate the energy due to atomistic interactions of different groups of atoms of the complex lattice.
View PDF Chat

Approximation of Functionals by Neural Network without Curse of Dimensionality

2022-06-01
Yahong Yang , Yang Xiang
In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces.The approximation error of the neural network is O(1/ … In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces.The approximation error of the neural network is O(1/ √ m) where m is the size of networks, which overcomes the curse of dimensionality.The key idea of the approximation is to define a Barron space of functionals.
View PDF Chat

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks

2023-01-01
Yahong Yang , Qipin Chen , Wenrui Hao
In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates … In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks.
View PDF Chat

Stochastic Continuum Models for High-Entropy Alloys with Short-range Order

2023-10-10
Yahong Yang , Luchan Zhang , Yang Xiang
High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in … High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in the atomic level randomness strongly influences the properties of HEAs. In this paper, we derive stochastic continuum models for HEAs with short-range order from atomistic models. A proper continuum limit is obtained such that the mean and variance of the atomic level randomness together with the short-range order described by a characteristic length are kept in the process from the atomistic interaction model to the continuum equation. The obtained continuum model with short-range order is in the form of an Ornstein–Uhlenbeck (OU) process. This validates the continuum model based on the OU process adopted phenomenologically by Zhang et al. [Acta Mater., 166 (2019), pp. 424–434] for HEAs with short-range order. We derive such stochastic continuum models with short-range order for both (i) the elastic deformation in HEAs without defects and (ii) HEAs with dislocations (line defects). The obtained stochastic continuum models are based on the energy formulations, whose variations lead to stochastic partial differential equations.
View PDF Chat

Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms

2023-01-01
Yahong Yang , Yulong Lu
This paper establishes the nearly optimal rate of approximation for deep neural networks (DNNs) when applied to Korobov functions, effectively overcoming the curse of dimensionality. The approximation results presented in … This paper establishes the nearly optimal rate of approximation for deep neural networks (DNNs) when applied to Korobov functions, effectively overcoming the curse of dimensionality. The approximation results presented in this paper are measured with respect to $L_p$ norms and $H^1$ norms. Our achieved approximation rate demonstrates a remarkable "super-convergence" rate, outperforming traditional methods and any continuous function approximator. These results are non-asymptotic, providing error bounds that consider both the width and depth of the networks simultaneously.

Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers

2024-10-08
Chuqi Chen , Qixuan Zhou , Yahong Yang +2 more
Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods … Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods leverage neural networks as basis functions to approximate PDE solutions. However, training such networks can be challenging, often resulting in limited accuracy. In this paper, we investigate the training dynamics of neural network-based PDE solvers with a focus on the impact of initialization techniques. We assess training difficulty by analyzing the eigenvalue distribution of the kernel and apply the concept of effective rank to quantify this difficulty, where a larger effective rank correlates with faster convergence of the training error. Building upon this, we discover through theoretical analysis and numerical experiments that two initialization techniques, partition of unity (PoU) and variance scaling (VS), enhance the effective rank, thereby accelerating the convergence of training error. Furthermore, comprehensive experiments using popular PDE-solving frameworks, such as PINN, Deep Ritz, and the operator learning framework DeepOnet, confirm that these initialization techniques consistently speed up convergence, in line with our theoretical findings.
View PDF Chat

On pattern formation in the thermodynamically-consistent variational Gray-Scott model

2024-09-06
Wenrui Hao , Chun Liu , Yiwei Wang +1 more
In this paper, we explore pattern formation in a four-species variational Gary-Scott model, which includes all reverse reactions and introduces a virtual species to describe the birth-death process in the … In this paper, we explore pattern formation in a four-species variational Gary-Scott model, which includes all reverse reactions and introduces a virtual species to describe the birth-death process in the classical Gray-Scott model. This modification transforms the classical Gray-Scott model into a thermodynamically consistent closed system. The classical two-species Gray-Scott model can be viewed as a subsystem of the variational Gray-Scott model in the limiting case when the small parameter $\epsilon$, related to the reaction rate of the reverse reactions, approaches zero. We numerically study the physically more complete Gray-Scott model with various $\epsilon$ in one dimension. By decreasing $\epsilon$, we observed that the stationary pattern in the classical Gray-Scott model can be stabilized as the transient state in the variational model for a significantly small $\epsilon$. Additionally, the variational model admits oscillated and traveling-wave-like pattern for small $\epsilon$. The persistent time of these patterns is on the order of $O(\epsilon^{-1})$. We also analyze the stability of two uniform steady states in the variational Gary-Scott model for fixed $\epsilon$. Although both states are stable in a certain sense, the gradient flow type dynamics of the variational model exhibit a selection effect based on the initial conditions, with pattern formation occurring only if the initial condition does not converge to the boundary steady state, which corresponds to the trivial uniform steady state in the classical Gray-Scott model.
View PDF Chat

DeepONet for Solving PDEs: Generalization Analysis in Sobolev Training

2024-10-05
Yahong Yang
In this paper, we investigate the application of operator learning, specifically DeepONet, to solve partial differential equations (PDEs). Unlike function learning methods that require training separate neural networks for each … In this paper, we investigate the application of operator learning, specifically DeepONet, to solve partial differential equations (PDEs). Unlike function learning methods that require training separate neural networks for each PDE, operator learning generalizes across different PDEs without retraining. We focus on the performance of DeepONet in Sobolev training, addressing two key questions: the approximation ability of deep branch and trunk networks, and the generalization error in Sobolev norms. Our findings highlight that deep branch networks offer significant performance benefits, while trunk networks are best kept simple. Moreover, standard sampling methods without adding derivative information in the encoding part are sufficient for minimizing generalization error in Sobolev training, based on generalization analysis. This paper fills a theoretical gap by providing error estimations for a wide range of physics-informed machine learning models and applications.
View PDF Chat

Multiscale Neural Networks for Approximating Green's Functions

2024-10-24
Wenrui Hao , Ruipeng Li , Yuanzhe Xi +2 more
Neural networks (NNs) have been widely used to solve partial differential equations (PDEs) in the applications of physics, biology, and engineering. One effective approach for solving PDEs with a fixed … Neural networks (NNs) have been widely used to solve partial differential equations (PDEs) in the applications of physics, biology, and engineering. One effective approach for solving PDEs with a fixed differential operator is learning Green's functions. However, Green's functions are notoriously difficult to learn due to their poor regularity, which typically requires larger NNs and longer training times. In this paper, we address these challenges by leveraging multiscale NNs to learn Green's functions. Through theoretical analysis using multiscale Barron space methods and experimental validation, we show that the multiscale approach significantly reduces the necessary NN size and accelerates training.
View PDF Chat

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks

2025-01-03
Yahong Yang , Qipin Chen , Wenrui Hao
View PDF Chat

An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks

2025-01-19
Jia‐Qi Luo , Yahong Yang , Yuan Yuan +2 more
This paper introduces Residual-based Smote (RSmote), an innovative local adaptive sampling technique tailored to improve the performance of Physics-Informed Neural Networks (PINNs) through imbalanced learning strategies. Traditional residual-based adaptive sampling … This paper introduces Residual-based Smote (RSmote), an innovative local adaptive sampling technique tailored to improve the performance of Physics-Informed Neural Networks (PINNs) through imbalanced learning strategies. Traditional residual-based adaptive sampling methods, while effective in enhancing PINN accuracy, often struggle with efficiency and high memory consumption, particularly in high-dimensional problems. RSmote addresses these challenges by targeting regions with high residuals and employing oversampling techniques from imbalanced learning to refine the sampling process. Our approach is underpinned by a rigorous theoretical analysis that supports the effectiveness of RSmote in managing computational resources more efficiently. Through extensive evaluations, we benchmark RSmote against the state-of-the-art Residual-based Adaptive Distribution (RAD) method across a variety of dimensions and differential equations. The results demonstrate that RSmote not only achieves or exceeds the accuracy of RAD but also significantly reduces memory usage, making it particularly advantageous in high-dimensional scenarios. These contributions position RSmote as a robust and resource-efficient solution for solving complex partial differential equations, especially when computational constraints are a critical consideration.
View PDF Chat

Learn Sharp Interface Solution by Homotopy Dynamics

2025-02-01
Chuqi Chen , Yahong Yang , Y. Xiang +1 more
This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, … This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination \al{}, showing the superiority of \al{}, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.
View PDF Chat

On pattern formation in the thermodynamically-consistent variational Gray-Scott model

2025-04-01
Wenrui Hao , Chun Liu , Yiwei Wang +1 more
View PDF Chat

Novel CSC-SMES based modular unified power quality controller

2025-06-03
Wenyong Guo , Yun Hong , Zhixiong He +5 more | Physica C Superconductivity

Novel CSC-SMES based modular unified power quality controller

2025-06-03
Wenyong Guo , Yun Hong , Zhixiong He +5 more | Physica C Superconductivity

On pattern formation in the thermodynamically-consistent variational Gray-Scott model

2025-04-01
Wenrui Hao , Chun Liu , Yiwei Wang +1 more
View PDF Chat

Learn Sharp Interface Solution by Homotopy Dynamics

2025-02-01
Chuqi Chen , Yahong Yang , Y. Xiang +1 more
This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, … This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination \al{}, showing the superiority of \al{}, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.
View PDF Chat

An Imbalanced Learning-based Sampling Method for Physics-informed Neural Networks

2025-01-19
Jia‐Qi Luo , Yahong Yang , Yuan Yuan +2 more
This paper introduces Residual-based Smote (RSmote), an innovative local adaptive sampling technique tailored to improve the performance of Physics-Informed Neural Networks (PINNs) through imbalanced learning strategies. Traditional residual-based adaptive sampling … This paper introduces Residual-based Smote (RSmote), an innovative local adaptive sampling technique tailored to improve the performance of Physics-Informed Neural Networks (PINNs) through imbalanced learning strategies. Traditional residual-based adaptive sampling methods, while effective in enhancing PINN accuracy, often struggle with efficiency and high memory consumption, particularly in high-dimensional problems. RSmote addresses these challenges by targeting regions with high residuals and employing oversampling techniques from imbalanced learning to refine the sampling process. Our approach is underpinned by a rigorous theoretical analysis that supports the effectiveness of RSmote in managing computational resources more efficiently. Through extensive evaluations, we benchmark RSmote against the state-of-the-art Residual-based Adaptive Distribution (RAD) method across a variety of dimensions and differential equations. The results demonstrate that RSmote not only achieves or exceeds the accuracy of RAD but also significantly reduces memory usage, making it particularly advantageous in high-dimensional scenarios. These contributions position RSmote as a robust and resource-efficient solution for solving complex partial differential equations, especially when computational constraints are a critical consideration.
View PDF Chat

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks

2025-01-03
Yahong Yang , Qipin Chen , Wenrui Hao
View PDF Chat

Multiscale Neural Networks for Approximating Green's Functions

2024-10-24
Wenrui Hao , Ruipeng Li , Yuanzhe Xi +2 more
Neural networks (NNs) have been widely used to solve partial differential equations (PDEs) in the applications of physics, biology, and engineering. One effective approach for solving PDEs with a fixed … Neural networks (NNs) have been widely used to solve partial differential equations (PDEs) in the applications of physics, biology, and engineering. One effective approach for solving PDEs with a fixed differential operator is learning Green's functions. However, Green's functions are notoriously difficult to learn due to their poor regularity, which typically requires larger NNs and longer training times. In this paper, we address these challenges by leveraging multiscale NNs to learn Green's functions. Through theoretical analysis using multiscale Barron space methods and experimental validation, we show that the multiscale approach significantly reduces the necessary NN size and accelerates training.
View PDF Chat

Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers

2024-10-08
Chuqi Chen , Qixuan Zhou , Yahong Yang +2 more
Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods … Neural network-based methods have emerged as powerful tools for solving partial differential equations (PDEs) in scientific and engineering applications, particularly when handling complex domains or incorporating empirical data. These methods leverage neural networks as basis functions to approximate PDE solutions. However, training such networks can be challenging, often resulting in limited accuracy. In this paper, we investigate the training dynamics of neural network-based PDE solvers with a focus on the impact of initialization techniques. We assess training difficulty by analyzing the eigenvalue distribution of the kernel and apply the concept of effective rank to quantify this difficulty, where a larger effective rank correlates with faster convergence of the training error. Building upon this, we discover through theoretical analysis and numerical experiments that two initialization techniques, partition of unity (PoU) and variance scaling (VS), enhance the effective rank, thereby accelerating the convergence of training error. Furthermore, comprehensive experiments using popular PDE-solving frameworks, such as PINN, Deep Ritz, and the operator learning framework DeepOnet, confirm that these initialization techniques consistently speed up convergence, in line with our theoretical findings.
View PDF Chat

DeepONet for Solving PDEs: Generalization Analysis in Sobolev Training

2024-10-05
Yahong Yang
In this paper, we investigate the application of operator learning, specifically DeepONet, to solve partial differential equations (PDEs). Unlike function learning methods that require training separate neural networks for each … In this paper, we investigate the application of operator learning, specifically DeepONet, to solve partial differential equations (PDEs). Unlike function learning methods that require training separate neural networks for each PDE, operator learning generalizes across different PDEs without retraining. We focus on the performance of DeepONet in Sobolev training, addressing two key questions: the approximation ability of deep branch and trunk networks, and the generalization error in Sobolev norms. Our findings highlight that deep branch networks offer significant performance benefits, while trunk networks are best kept simple. Moreover, standard sampling methods without adding derivative information in the encoding part are sufficient for minimizing generalization error in Sobolev training, based on generalization analysis. This paper fills a theoretical gap by providing error estimations for a wide range of physics-informed machine learning models and applications.
View PDF Chat

Near-optimal deep neural network approximation for Korobov functions with respect to <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e79" altimg="si14.svg"><mml:msup><mml:mrow><mml:mi>L</mml:mi></mml:mrow><mml:mrow><mml:mi>p</mml:mi></mml:mrow></mml:msup></mml:math> and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e89" altimg="si15.svg"><mml:msup><mml:mrow><mml:mi>H</mml:mi></mml:mrow><mml:mrow><mml:mn>1</mml:mn></mml:mrow></mml…

2024-09-06
Yahong Yang , Yulong Lu

On pattern formation in the thermodynamically-consistent variational Gray-Scott model

2024-09-06
Wenrui Hao , Chun Liu , Yiwei Wang +1 more
In this paper, we explore pattern formation in a four-species variational Gary-Scott model, which includes all reverse reactions and introduces a virtual species to describe the birth-death process in the … In this paper, we explore pattern formation in a four-species variational Gary-Scott model, which includes all reverse reactions and introduces a virtual species to describe the birth-death process in the classical Gray-Scott model. This modification transforms the classical Gray-Scott model into a thermodynamically consistent closed system. The classical two-species Gray-Scott model can be viewed as a subsystem of the variational Gray-Scott model in the limiting case when the small parameter $\epsilon$, related to the reaction rate of the reverse reactions, approaches zero. We numerically study the physically more complete Gray-Scott model with various $\epsilon$ in one dimension. By decreasing $\epsilon$, we observed that the stationary pattern in the classical Gray-Scott model can be stabilized as the transient state in the variational model for a significantly small $\epsilon$. Additionally, the variational model admits oscillated and traveling-wave-like pattern for small $\epsilon$. The persistent time of these patterns is on the order of $O(\epsilon^{-1})$. We also analyze the stability of two uniform steady states in the variational Gary-Scott model for fixed $\epsilon$. Although both states are stable in a certain sense, the gradient flow type dynamics of the variational model exhibit a selection effect based on the initial conditions, with pattern formation occurring only if the initial condition does not converge to the boundary steady state, which corresponds to the trivial uniform steady state in the classical Gray-Scott model.
View PDF Chat

Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations

2024-05-22
Chuqi Chen , Yahong Yang , Xiang Yang +1 more
Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or the incorporation of empirical data. … Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or the incorporation of empirical data. One advantage of the neural network method for PDEs lies in its automatic differentiation (AD), which necessitates only the sample points themselves, unlike traditional finite difference (FD) approximations that require nearby local points to compute derivatives. In this paper, we quantitatively demonstrate the advantage of AD in training neural networks. The concept of truncated entropy is introduced to characterize the training property. Specifically, through comprehensive experimental and theoretical analyses conducted on random feature models and two-layer neural networks, we discover that the defined truncated entropy serves as a reliable metric for quantifying the residual loss of random feature models and the training speed of neural networks for both AD and FD methods. Our experimental and theoretical analyses demonstrate that, from a training perspective, AD outperforms FD in solving partial differential equations.
View PDF Chat

Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations

2024-05-22
Wenrui Hao , Xinliang Liu , Yahong Yang
Solving nonlinear partial differential equations (PDEs) with multiple solutions using neural networks has found widespread applications in various fields such as physics, biology, and engineering. However, classical neural network methods … Solving nonlinear partial differential equations (PDEs) with multiple solutions using neural networks has found widespread applications in various fields such as physics, biology, and engineering. However, classical neural network methods for solving nonlinear PDEs, such as Physics-Informed Neural Networks (PINN), Deep Ritz methods, and DeepONet, often encounter challenges when confronted with the presence of multiple solutions inherent in the nonlinear problem. These methods may encounter ill-posedness issues. In this paper, we propose a novel approach called the Newton Informed Neural Operator, which builds upon existing neural network techniques to tackle nonlinearities. Our method combines classical Newton methods, addressing well-posed problems, and efficiently learns multiple solutions in a single learning process while requiring fewer supervised data points compared to existing neural network methods.
View PDF Chat

Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss

2024-01-31
Yahong Yang , Juncai He
Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. … Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question. This paper explores a comparison between deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers, focusing on their optimal generalization error in Sobolev losses. Analytical investigations reveal that the architecture of a neural network can be significantly influenced by various factors, including the number of sample points, parameters within the neural networks, and the regularity of the loss function. Specifically, a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs. We ultimately apply this theory to address partial differential equations using deep Ritz and physics-informed neural network (PINN) methods, guiding the design of neural networks.
View PDF Chat

Stochastic Continuum Models for High-Entropy Alloys with Short-range Order

2023-10-10
Yahong Yang , Luchan Zhang , Yang Xiang
High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in … High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in the atomic level randomness strongly influences the properties of HEAs. In this paper, we derive stochastic continuum models for HEAs with short-range order from atomistic models. A proper continuum limit is obtained such that the mean and variance of the atomic level randomness together with the short-range order described by a characteristic length are kept in the process from the atomistic interaction model to the continuum equation. The obtained continuum model with short-range order is in the form of an Ornstein–Uhlenbeck (OU) process. This validates the continuum model based on the OU process adopted phenomenologically by Zhang et al. [Acta Mater., 166 (2019), pp. 424–434] for HEAs with short-range order. We derive such stochastic continuum models with short-range order for both (i) the elastic deformation in HEAs without defects and (ii) HEAs with dislocations (line defects). The obtained stochastic continuum models are based on the energy formulations, whose variations lead to stochastic partial differential equations.
View PDF Chat

Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives

2023-01-01
Yahong Yang , Haizhao Yang , Yang Xiang
This paper addresses the problem of nearly optimal Vapnik--Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: … This paper addresses the problem of nearly optimal Vapnik--Chervonenkis dimension (VC-dimension) and pseudo-dimension estimations of the derivative functions of deep neural networks (DNNs). Two important applications of these estimations include: 1) Establishing a nearly tight approximation result of DNNs in the Sobolev space; 2) Characterizing the generalization error of machine learning methods with loss functions involving function derivatives. This theoretical investigation fills the gap of learning error estimations for a wide range of physics-informed machine learning models and applications including generative models, solving partial differential equations, operator learning, network compression, distillation, regularization, etc.
View PDF Chat

Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks

2023-01-01
Yahong Yang , Qipin Chen , Wenrui Hao
In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates … In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks.
View PDF Chat

Nearly Optimal Approximation Rates for Deep Super ReLU Networks on Sobolev Spaces

2023-01-01
Yahong Yang , Yue Wu , Haizhao Yang +1 more
This paper introduces deep super ReLU networks (DSRNs) as a method for approximating functions in Sobolev spaces measured by Sobolev norms $W^{m,p}$ for $m\in\mathbb{N}$ with $m\ge 2$ and $1\le p\le … This paper introduces deep super ReLU networks (DSRNs) as a method for approximating functions in Sobolev spaces measured by Sobolev norms $W^{m,p}$ for $m\in\mathbb{N}$ with $m\ge 2$ and $1\le p\le +\infty$. Standard ReLU deep neural networks (ReLU DNNs) cannot achieve this goal. DSRNs consist primarily of ReLU DNNs, and several layers of the square of ReLU added at the end to smooth the networks output. This approach retains the advantages of ReLU DNNs, leading to the straightforward training. The paper also proves the optimality of DSRNs by estimating the VC-dimension of higher-order derivatives of DNNs, and obtains the generalization error in Sobolev spaces via an estimate of the pseudo-dimension of higher-order derivatives of DNNs.
View PDF Chat

Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms

2023-01-01
Yahong Yang , Yulong Lu
This paper establishes the nearly optimal rate of approximation for deep neural networks (DNNs) when applied to Korobov functions, effectively overcoming the curse of dimensionality. The approximation results presented in … This paper establishes the nearly optimal rate of approximation for deep neural networks (DNNs) when applied to Korobov functions, effectively overcoming the curse of dimensionality. The approximation results presented in this paper are measured with respect to $L_p$ norms and $H^1$ norms. Our achieved approximation rate demonstrates a remarkable "super-convergence" rate, outperforming traditional methods and any continuous function approximator. These results are non-asymptotic, providing error bounds that consider both the width and depth of the networks simultaneously.

Approximation of Functionals by Neural Network without Curse of Dimensionality

2022-06-01
Yahong Yang , Yang Xiang
In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces.The approximation error of the neural network is O(1/ … In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces.The approximation error of the neural network is O(1/ √ m) where m is the size of networks, which overcomes the curse of dimensionality.The key idea of the approximation is to define a Barron space of functionals.
View PDF Chat

Convergence from atomistic model to Peierls–Nabarro model for dislocations in bilayer system with complex lattice

2022-01-01
Yahong Yang , Tao Luo , Yang Xiang
In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of … In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of the dislocation solution of the PN model converges to the dislocation solution of the atomistic model with second-order accuracy. The consistency of PN model and the stability of atomistic model are essential in our proof. The main idea of our approach is to use several low-degree polynomials to approximate the energy due to atomistic interactions of different groups of atoms of the complex lattice.
View PDF Chat

Stochastic Continuum Models for High--Entropy Alloys with Short-range Order

2022-01-01
Yahong Yang , Luchan Zhang , Yang Xiang
High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in … High entropy alloys (HEAs) are a class of novel materials that exhibit superb engineering properties. It has been demonstrated by extensive experiments and first principles/atomistic simulations that short-range order in the atomic level randomness strongly influences the properties of HEAs. In this paper, we derive stochastic continuum models for HEAs with short-range order from atomistic models. A proper continuum limit is obtained such that the mean and variance of the atomic level randomness together with the short-range order described by a characteristic length are kept in the process from the atomistic interaction model to the continuum equation. The obtained continuum model with short-range order is in the form of an Ornstein--Uhlenbeck (OU) process. This validates the continuum model based on the OU process adopted phenomenologically by Zhang et al. [Acta Mater., 166 (2019), pp. 424--434] for HEAs with short-range order. We derive such stochastic continuum models with short-range order for both elasticity in HEAs without defects and HEAs with dislocations (line defects). The obtained stochastic continuum models are based on the energy formulations, whose variations lead to stochastic partial differential equations.
View PDF Chat

Approximation of Functionals by Neural Network without Curse of Dimensionality

2022-01-01
Yahong Yang , Yang Xiang
In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces. The approximation error of the neural network is … In this paper, we establish a neural network to approximate functionals, which are maps from infinite dimensional spaces to finite dimensional spaces. The approximation error of the neural network is $O(1/\sqrt{m})$ where $m$ is the size of networks, which overcomes the curse of dimensionality. The key idea of the approximation is to define a Barron spectral space of functionals.
View PDF Chat

Convergence from Atomistic Model to Peierls-Nabarro Model for Dislocations in Bilayer System with Complex Lattice

2021-01-01
Yahong Yang , T. Luo , Yang Xiang
In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of … In this paper, we prove the convergence from the atomistic model to the Peierls--Nabarro (PN) model of two-dimensional bilayer system with complex lattice. We show that the displacement field of the dislocation solution of the PN model converges to the dislocation solution of the atomistic model with second-order accuracy. The consistency of PN model and the stability of atomistic model are essential in our proof. The main idea of our approach is to use several low-degree polynomials to approximate the energy due to atomistic interactions of different groups of atoms of the complex lattice.
View PDF Chat
Coauthor Papers Together
Wenrui Hao 8
Yang Xiang 4
Yang Xiang 4
Chuqi Chen 3
Chun Liu 2
Luchan Zhang 2
Yiwei Wang 2
T. Luo 2
Haizhao Yang 2
Yulong Lu 2
Zhian Jia 1
Wenrui Hao 1
Wenju Sang 1
Yuanzhe Xi 1
Yuan Yuan 1
Jianyu Lan 1
Xinliang Liu 1
Yue Wu 1
Shixin Xú 1
Tao Luo 1
Ruipeng Li 1
Xiang Yang 1
Yun Hong 1
Qipin Chen 1
Xiang Yang 1
Qixuan Zhou 1
Zhixiong He 1
Tianshi Xu 1
Qipin Chen 1
Ronghua Zhao 1
Y. Xiang 1
Jia‐Qi Luo 1
Juncai He 1
Wenyong Guo 1