Nonlinear System Identification Using
Maximum Likelihood Estimation

Siny Paul; Bindu Elias

Nonlinear System Identification Using Maximum Likelihood Estimation

Siny Paul, Bindu Elias
Associate Professor, Department of EEE, Mar Athanasius College of Engineering, Kothamangalam, Kerala, India

Related article at Pubmed, Scholar Google

Visit for more related articles at International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering

Abstract

Different algorithms can be used to train the Neural Network Model for Nonlinear system identification. Here the ‘Maximum Likelihood Estimation’ is implemented for modeling nonlinear systems and the performance is evaluated. Maximum likelihood is a well-established procedure for statistical estimation. In this procedure first formulate a log likelihood function and then optimize it with respect to the parameter vector of the probabilistic model under consideration. Four nonlinear systems are used to validate the performance of the model. Results show that Neural Network with the algorithm of Maximum Likelihood Estimation is a good tool for system identification, when the inputs are not well defined.

Keywords

Neural Network, Nonlinear system, Mean square error, Modeling.

INTRODUCTION

This paper concentrates on modeling problem which arise when we can identify a certain quantity as a definite measurable output or effect but the causes are not well defined. This is called time series modeling, where inputs or causes are numerous and not quite known in addition to often being unobservable. This type modeling is also called stochastic modeling. In system identification we are concerned with the determination of the system models from records of system operation. The problem can be represented diagrammatically as below.

where u(t) is the known input vector of dimension ‘m’

z(t) is the output vector of dimension ‘p’

w(t) is the input disturbance vector

n(t) is the observation noise vector

v(t) is the measured output vector of dimension ‘p’

Thus the problem of system identification is the determination of the system model from records of u(t) and y(t).

An artificial neural network is a powerful tool for many complex applications such as function approximation, optimization, nonlinear system identification and pattern recognition. This is because of its attributes like massive parallelism, adaptability, robustness and the inherent capability to handle nonlinear system. It can extract information from heavy noisy corrupted signals.

System identification can be either state space model or input-output model [1].

INPUT-OUTPUT MODEL

An I/O model can be expressed as y(t) = g(φ (t,θ )) + e(t) , where, θ is the vector containing adjustable parameters which in the case of neural network are known as weights, g is the function realized by neural network and φ is the regression vector. Depends on the choice of regression vector different model structures emerge.

Using the same regressors as for the linear models, corresponding families of nonlinear models were obtained which are named NARX, NARMAX, etc. Different model structures in each model family can be obtained by making a different assumption about noise.

Where y(t) is the output, u(t),the input and e(t) is the error. For the implementation of the above system, Feed forward neural network can be used.

NARX Model is well suited for Input-Output modeling of stochastic nonlinear systems [3] So in the proposed work, NARX model is chosen as the system model; in which the model structure is a Multi Layer Feed Forward Neural Network(MLFFN) as shown in Fig. 2 For all the models (using different algorithms).

MAXIMUM LIKELIHOOD ESTIMATION.

The term “maximum likelihood estimate” with the desired asymptotic properties usually refers to a root of the likelihood equation that globally maximizes the likelihood function [3]. In other words the ML estimate xML is that value of the parameter vector x for which the conditional probability density function P(z/x) is maximum [4].The maximum likelihood estimate xML of the target parameters x is the mode of the conditional probability density function(likelihood function):

The log likelihood function

A. System Modeling Using Gauss-Newton Method.

RESULTS AND DISCUSSIONS

The same four nonlinear systems are modeled using feed forward neural network. The NARX model with 14 inputs is used similar to the earlier cases. The performance analysis is done by plotting the mean square error in each case.

From the above results it is seen that the model is giving good performance for all the four nonlinear systems which proves that NARX Model is well suited for Input-Output modeling of stochastic nonlinear systems

CONCLUSION

In this paper, a comprehensive analysis for nonlinear system identification is done and its performance is compared by implementation of the same in a Neural Network NARX model using MATLAB programs. The adaptive feature revealed by feed forward and recurrent neural network as well as their ability to model nonlinear time varying process, provides a surplus value to the model based predictive control. When applied correctly, a neural or adaptive system may considerably outperform other methods. This is an attempt to provide guideline to the practitioners to choose the suitable method for their specific problem in the field of system identification especially in the stochastic modeling of nonlinear systems. It is proved that MLE can be reformulated as a minimization problem; The results show good performance of the models and it is proven that MLE is good for nonlinear system identification. Four different nonlinear systems are used to check the consistency of the performance of algorithm. The performance of MLE is good in terms of mean square error.

Tables at a glance

Table 1

Figures at a glance


Figure 1	Figure 2	Figure 3	Figure 4	Figure 5

Figure 6	Figure 7	Figure 8	Figure 9	Figure 10

References

N.K Sinha and B Kuszta, Modeling and Identification of Dynamic systems, Vol.1, New York,VanNostrand Reinhold Company, 1983.

Simon Haykin, Neural Networks a comprehensive Foundation, 2nd ed.,India, Pearson Education, 1999.

A.V. Balakrishnan, Kalman Filtering Theory, New York, Optimization Software Inc. Publications Division, ,1992.

Ben James, Brian D.O, Anderson, and Robert . C. Williamson “Conditional Mean and Maximum Likelihood approaches to Multi harmonic frequency estimation.”, IEEE Transactions on Signal Processing, Vol 42, No.6, pp 1366-1375, June 1994.

Roy L Streit and Tod E Luginbubl, “Maximum Likelihood Training of Probabilistic Neural Networks”, IEEE Transactions on Neural Networks, Vol.5, No.5, pp 764-783, September 1994.

Tsung-Nan Lin, C.Lee Giles and Sun-Yaun Kung, ”A Delay Damage Model Selection Algorithm for NARX Neural Networks”, IEEE Transactions on Signal Processing Vol.45,No.11, pp 2719-2730,November 1997.

Wen Yu and XiaoouLi”Some New Results on System Identification with Dynamic Neural Network”, IEEE Transactions on Neural Network”, Vol 12, No.2, pp 412-417, March 2001.

X.M. Ren, A.B. Rad, P.T. Chan and WaiLan Lo, “Identification and control of continuous time Nonlinear Systems via Dynamic Neural Networks” IEEE Transactions on Industrial Electronics”, Vol 50,No.3, pp 478 – 486, June 2003.

Songwu Lu and Tamer Basar , “ Robust Nonlinear System Identification using Neural Network Models”, IEEE Transactions on Neural Networks, Vol.9, No.3, pp 407 –429, May 1998.

Mohamed Ibnkabla, “Statistical Analysis of Neural Network Modeling and Identification of Nonlinear Systems with Memory” ,IEEE Transactions on Signal Processing, Vol. 50, No.6, pp 1508-1517, June 2002.

Nader Sadegh, “A Perceptron Network for Functional Identification and Control of Nonlinear Systems”, IEEE Transactions on Neural Networks, Vol.4, No.6, pp 982-989, November 1993.

Octavian Stan and Edward Kamen, “ A localized Least Squares Algorithm for Training Feed forward Neural Networks”, IEEE Transactions on Neural networks, Vol.11, No.2, pp 487-495, march 2000.

Si Zhao Qin, Hong Te Su and Thomas J McAvoy, “Comparison of four Neural Network Learning Methods for Dynamic System Identification”, IEEE Transactions on Neural Networks , Vol. 3, No.1, pp 122-130, January 1992.

G.V. Puskorius and L.A Feldkamp, “Neuro Control of nonlinear dynamical systems with Kalman Filter trained recurrent networks” IEEE Transactions on neural networks, Vol 5, No.2, pp 279-297,March 1994.

K.S. Narendra and K Parthasarathy, “Identification and control of Dynamical systems using neural networks”, IEEE Transactions on Neural Networks, Vol 1, No.2, pp 4-27, March 1990

Shuhi Li, “Comparative Analysis of back propagation and Extended Kalman filter in Pattern and Batch forms for training Neural Networks”, IEEE Transactions on Neural Network, Vol 2, No.4, pp144-149, June 2001