Progress Report

Author: Songbai Yao

Work Conducted

I have been reading the following literature [1][2][3][4][5][6]. Artificial Neural Networks (ANNs) have the ability to learn and construct relationships among data. This report summarizes my weekly progress on understanding ANN theory and its applications in combustion.

ANNs Background Reading

Artificial Neurons

A typical ANN consists of a single input layer and a single output layer, with any number of hidden layers performing transformations on the input data. The values of neurons in the hidden layer are computed using weights and biases:

\[\begin{bmatrix} a_0^{(1)} \\ a_1^{(1)} \\ \vdots \\ a_n^{(1)} \end{bmatrix} = \begin{bmatrix} \omega_{0,0} & \omega_{0,1} & \ldots & \omega_{0,n} \\ \vdots & \vdots & \vdots & \vdots \\ \omega_{k,0} & \omega_{k,1} & \ldots & \omega_{k,n} \end{bmatrix} \begin{bmatrix} a_0^{(0)} \\ a_1^{(0)} \\ \vdots \\ a_n^{(0)} \end{bmatrix} + \begin{bmatrix} b_0 \\ b_1 \\ \vdots \\ b_n \end{bmatrix} \]

The output of each artificial neuron is computed using a non-linear activation function, such as the sigmoid function:

\[\sigma(x) = \frac{1}{1 + e^{-x}} \]

Cost Function

To find optimal weights and biases, a cost function is defined, typically the mean squared error (MSE):

\[C(\omega, b) = \frac{1}{2n} \sum \|y(x) - a\|^2 \]

Gradient Descent (GD)

Gradient descent is employed to find the global minimum of the cost function by iteratively updating weights and biases:

\[\delta v = \eta \nabla C, \quad v := v - \eta \nabla C \]

Back-propagation (BP)

Back-propagation is a key algorithm in ANNs, involving the following steps:

Input \(x\)
Feedforward: \(y'\)
Output error: \(\delta = (y - y')\)
Backpropagate the error from the output layer to every hidden layer
Gradient descent: use GD or SGD to update weights and biases

Applications of ANNs in Combustion

Ge et al.'s Paper [4]

Objective: Ge et al. used ANNs to calculate thermodynamic quantities in liquid rocket engine problems. They employed a Deep Residual Learning Network, developed by He et al. [5]. The main steps include:

Use data from test cases to train ANNs (dataset \(\Phi_{1}\)) to model the relation between (\(h\),\(p\)) and (\(T\), \(\rho\), \(c_p\), \(a_s\), \(\phi\), \(\kappa\), \(\mu\)).
Use (\(h\),\(p\)) from \(\Phi_{2}\) as input to obtain output data (\(T\),\(\rho\), \(c_p\), \(a_s\), \(\phi\), \(\kappa\), \(\mu\)) using the trained ANNs.
Compare the output data with those from \(\Phi_{2}\).
Use (\(h\),\(p\)) from \(\Phi_{2}\) as input again, but this time obtain output data by solving the EoS equation.
Compare the speed of ANNs and solving EoS equation. See Fig. ~[6].

Figure 1: Comparison of test data and ANN predictions for \(\rho\) and \(p\)

Figure 2: Average computation time for real-gas EoS model (CPU) and ResNet inference (CPU and GPU) source

Gamahara and Hattori's Paper [2]

Objective: Gamahara and Hattori used a traditional ANN with multiple hidden layers to find a new subgrid-scale (SGS) stress model in LES. The trained ANNs were then used as the new SGS model. The steps include:

A DNS dataset, which includes ALL data.
Extract grid-scale (GS) data from DNS by filtering.
Use filtered GS data as input and SGS data as output to train ANNs.
Use the trained ANNs as the SGS model in LES. They do not use, e.g., Smagorinsky model. Instead, they used the trained ANNs to provide SGS data needed. See Fig.~[7].
Try different ANN models, e.g., ANNs with different hidden layers or with different input and output neurons.

Figure 3: Comparison of mean flow obtained by LES with Smagorinsky model and ANN model. Left, \(R_{e_t}=180\); right, \(R_{e_t}=400\)

Aims for Next Week

Continue reading: ANNs theory and contemporary ANN methods.
Residual neural networks.
Reading: Rigopoulos's papers.

References:
[1] Goodfellow et al. (2016)
[2] Gamahara and Hattori (2017)
[3] Sanderson (2017) [YouTube]
[4] Ge et al. (2018)
[5] He et al. (2016)
[6] Nielsen (2015) [Neural Networks and Deep Learning]
[7] Chollet (2017) [Deep Learning with Python]