Robust Neural Network Training Algorithms and Artificial Intelligence
Robust Training Algorithms of Recurrent Neural Network (RNN) and Neural Network (NN) with Real Word Applications
While increasing learning rate may help to improve performance of the RNN, it can result in unstable training in terms of weight divergence. Therefore, an optimal trade-off between RNN training speed and weight convergence is desired. Different robust training algorithms of RNN are developed based on a novel RNN hybrid training concept. It switches the training patterns between standard real time on-line Back Propagation (BP) and RTRL according to the derived convergence and stability conditions. Further extension of the new theoretical result to spiking neuron activities are under investigation.
1. Robust recurrent kernel learning incorporating neural network structural change based on variable length estimation error, which is a kind of policy selection strategies and similar to the basic concept used in Google’s famous AlphaGo, in certain extents. We use only automatic selection of the target function based on a single kernel neural network (be more theoretical sound and easy to use) and the later employs a separated trained deep learning neural network called value network (be more practical orientated and a lot of computing resources, used 1202 CPUs with 176 GPUs in total). How to integrate the two approaches would be an interesting direction in the future.
2. Robust training algorithm of recurrent Jordan networks
3. Robust training algorithms of recurrent Elman networks
4. Robust Support Vector Machine Based Computerized Auto-scoring System (patented commercial project with CISCO police)
Robust Information Clustering
The ultimate target of Robust Information Clustering is not to look for or approximate a true probability distribution of input data set (similarly, we are also not looking for “true” clusters or cluster centers), which is proved to be an ill-posed problem (Vapnik, 1998), but to determine an optimal number of effective clusters by eliminating the unreliable data points (outliers) based on the Chernoff bound with asymptotic expression (especially for the nonlinear robust information clustering algorithm). It also turns out that outliers are in fact data points that fail to achieve capacity. Therefore, any data point could become an outlier as the cluster number is increased in cluster annealing procedure. Furthermore, by replacing the Euclidean distance with other dissimilarity measures, it is possible to extend the new algorithm into the kernel and nonlinear clustering algorithms for linearly non-separable patterns (see the latest research reports).
1. Robust Information Clustering (basic algorithm with loose bound)
2. Nonlinear Robust Information-theoretic Clustering (improved algorithm with tighter bound)
3. MRI image segmentation and classification (joint research with IMH)
4. Intelligent eye tracking system for disabled people (joint research with TTH)
Sumit Bam Shrestha and Qing Song, “Robustness to Training Disturbances in SpikeProp Learning”, IEEE Transactions on Neural Networks and Learning Systems, Digital Preview Page (99): 1-14, July, 2017.
Sumit Bam Shrestha and Qing Song, “Robust spike-train learning in spike-event based weight update”, Neural Networks, Volume 96, December 2017, Pages 33-46.
Qing Song, X. Zhao, H. J. Fan and D.W. Wang, “Robust Recurrent Kernel Online Learning", IEEE Transactions on Neural Networks and Learning Systems, Vol. 28, No. 5, May, 2017.
Sumit Bam Shrestha, Qing Song, “Robust learning in SpikeProp”, Neural Networks, Volume 86, February 2017, Pages 54-68.
Xulei Yang, Qing Song and Yi Su, “Automatic segmentation of left ventricle cavity from short-axis cardiac magnetic resonance images”, Medical & Biological Engineering & Computing, Feb. 2016.
Sumit Bam Shrestha, Qing Song, “Adaptive learning rate of SpikeProp based on weight convergence analysis”, Neural Networks, 03/2015.
Haijin Fan, Qing Song, Zhao Xu, “An information theoretic sparse kernel algorithm for online learning”, Expert Systems with Applications 07/2014; 41(9):4349–4359.
Haijin Fan, Qing Song, Sumit Bam Shrestha, “Online Learning with Kernel Regularized Least Mean Square Algorithms”, Knowledge-Based Systems 03/2014.
Haijin Fan, Qing Song, “A linear recurrent kernel online learning algorithm with sparse updates”, Neural networks, 11/2013.
Zhimin Wang, Qing Song, Y.C. Soh, Sim Kang, “An adaptive spatial information-theoretic fuzzy clustering algorithm for image segmentation”, Computer Vision and Image Understanding 10/2013; 117(10):1412-1420.
Qing Song, “Robust initialization of a Jordan Network with Recurrent Constrained Learning”, Special issue of White Box Nonlinear Prediction Models, IEEE Transactions on Neural Networks, Vol. 22, Vol. 12, 2011, pp. 2460-2473.
Z.M. Wang, Qing Song, Y.C. Soh, and Sim Kang, “A Robust Curve Clustering based on a Multivariate Model”, IEEE Transactions on Neural Networks, Vol. 21, No. 12, 2010, pp.1976-1984.
Qing Song, X.L. Yang and Y.C. Soh, “An Information-theoretic Fuzzy C-Spherical Shells Clustering Algorithm”, Fuzzy Sets and Systems, Vol.161, 2010, pp.1755-1773.
Z.M. Wang, Y.C. Soh, Qing Song and Sim Kang, “Adaptive Spatial Information-theoretic Clustering for Image Segmentation”, Pattern Recognition, Vol. 42., 2009, pp. 2029-2044.
C.Y. Guo and Qing Song, “Real-time control of variable air volume system based on a robust neural network associated PI controller”, IEEE Transactions on Control Systems Technology, Vol.17, No. 3, 2009, pp.600-607.
X. L. Yang, Qing Song, Y.L. Wu “A Novel Pruning Approach for Robust Data Clustering”, Neural Computing & Applications, Vol. 18, No.7, pp.759-768, 2009.
Qing Song, Y.L.Wu and Y.C. Soh, “Robust Adaptive Gradient Descent Training Algorithm for Recurrent Neural Networks in Discrete Time Domain", IEEE Transactions on Neural Networks, Vol. 19, No.11, 2008, pp.1841-1853.
Qing Song, J.Spalls, Y.C.Soh and J.Ni, "Robust Neural Network Tracking Controller Using Simultaneous Perturbation Stochastic Approximation”, IEEE Transactions on Neural Networks, Vol.19, No.5, 2008, pp.817-835.
Y.L. Wu, Qing Song, and S. Liu, "A Normalized Adaptive Training of Recurrent Neural Networks with Augmented Error Gradient" , IEEE Transactions on Neural Networks, Vol.19, No.2, 2008, pp.351-356.
A. Z. Cao and Qing Song, “Robust Information Clustering for Automatic Breast Mass Detection in Digitized Mammograms”, Computer Vision and Image Understanding, Vol. 109, No.1, 2008, pp.87-96.
Xulei Yang, Qing Song, "A weighted support vector machine for data classification", International Journal of Pattern Recognition and Artificial Intelligence, 21 (5), 2007, pp.961-976.
Jie Ni and Qing Song, “Pruning Based Robust Backpropagation Training Algorithm for RBF Network Tracking Controller”, Journal of Intelligent and Robotics Systems, Vol. 48, No.3, 2007, pp.375-396.
Y.L. Wu and Qing Song, “Robust Recurrent Neural Control of Biped Robot”, Journal of Intelligent and Robotics Systems, No. 2, Vol. 49, 2007, pp.151-169.
Xulei Yang and Qing Song, “A Robust Deterministic Annealing Algorithm for Data Clustering”, Journal of Data & Knowledge Engineering, 62 (1), 2007, pp.84-100.
Chengyi Guo, Qing Song, and W.J. Cai, “Neural Network Assisted Cascade Control System for Air Handling Unit”, IEEE Transactions On Industrial Electronic,Vol. 54, No.1, 2007, pp.620-628.
J. Ni and Qing Song. ”Dynamic Pruning Algorithm for Multilayer Perceptron Based Neural Control Systems”, Journal of Neurocomputing, Vol. 69, Issues 16-18, 2006.
Xulei Yang and Qing Song, Kernel-based deterministic annealing algorithm for data clustering, IEE Proceedings: Vision, Image and Signal Processing, Vol. 153, No.5, 2006, pp. 557-568.
Xulei Yang, Qing Song, A.Z. Cao, " A weighted deterministic annealing algorithm for data clustering", International Journal of Computational Intelligent Research, Vol.2, No.1, 2006, pp.81-85.
Xulei Yang, Qing Song and A. Z. Cao, “A New Cluster Validity for Data Clustering”, Neural Processing Letter, 23 (3), 2006, pp. 325-344.
W.J.Hu and Qing.Song “An Accelerated Training Algorithm for Robust Support Vector Machine”, IEEE Transactions on Circuits and Systems II, Vol. 51, No.5, 2004.
Qing. Song, W. J. Hu, “Robust Neural Controller for VAV System”, IEE Proceedings Part D - Control Theory and Applications, Vol. 150, No. 2, 2003.
Qing Song, L. Yin, Y. C. Soh, “Robust Identification of Nonlinear Plant using Neural Networks”, Asian Journal of Control, Vol. 3, No. 2, 2001, Special Issue on Advances in Neural and Fuzzy Controllers.
Qing Song, L. Yin, Y. C. Soh, “Robust Adaptive Dead Zone Technology for Fault Diagnosis and Control of Robot Manipulators using Neural Networks”, Journal of Intelligent and Robotics Systems, the Netherlands, 2001.
Qing Song, Lin Yin, “Robust Adaptive Fault Accommodation for Robot System using RBF Neural Network”, International Journal of Systems Science, UK, Vol. 32, No. 2, 2001.
Qing Song, “Design of Robust Neural Tracking Controller”, Journal of Intelligent and Robotics Systems, the Netherlands, Vol. 20, 2000.
D. J. Hou, Qing Song, “Computerised Auto-Scoring System Based upon Feature Extraction and Neural Network Technologies”, Journal of Intelligent and Robotics Systems, the Netherlands, Vol. 29, 2000, Special Issue of Neural Network and Image Processing.
Qing Song, J. Xiao, Y. C. Soh, “Robust Back-Propagation Algorithm for Multi-Layered Neural Tracking Controller”, IEEE Transactions on Neural Networks, USA, Vol. 10, No. 5, 1999, pp. 1133-1141.
Qing Song, “Implementation of Two Dimensional Systolic Algorithms for Multilayered Neural Networks”, Journal of Systems Architecture, Italy, Vol. 48, pp.1209-1218, 1999.
Qing Song, L. Yin, Y. C. Soh, “Shifting of the Center of Radial Basis Function Neural Network in the Presence of Disturbance”, Neural and Parallel Computation, USA, Vol. 7, No. 1, pp. 121-131, 1999.
Qing Song, “Robust Training Algorithm of Multi-Layered Neural Network for Identification of Nonlinear Dynamic Systems”, IEE Proceedings Part D - Control Theory and Applications, UK, Vol. 145, No. 1, 1998.
Qing Song, M. J. Grimble, “Design of a Multivariable Neural Controller and its Application to Gas Turbine”, ASME Journal of Dynamic Systems, Measurement and Control, USA, Vol. 119, No. 3, 1997.
Qing Song, J. Xiao, “On the Convergence Performance of Multi-Layered NN Tracking Controller”, Neural and Parallel Computation, USA, Vol. 5, No. 3, 1997.
Qing Song, “Training of a Single Perceptron Neuron using Robust Projection Algorithm”, Neural and Parallel Computation, USA, Vol. 4, No. 2, pp. 193-203, 1996.
Qing Song, E. K. Teoh, D. P. Mital, “Multilayered Neural Network Implementation on Transputer Systolic Array”, Journal of Microprocessing and Microprogramming, the Netherlands, Vol. 41, , pp. 289-299, 1995.
Qing Song, M. R. Katebi, M. J. Grimble, “Robust Multivariable Implicit Adaptive Control”, IMA Journal of Mathematical Control and Information, UK, Vol. 10, pp. 49-70, 1993.
Qing Song, J. Wilkie, M. J. Grimble, “Robust Controller for Gas Turbines Based Upon LQG/LTR Design with Self-Tuning Features”, ASME Journal of Dynamic Systems, Measurement and Control, USA, Vol. 115, No. 3, pp. 569-571, 1993.
W.F. Xie (Canada)
G. Yin (USA)
J.Z. Xiao (USA)
L. Yin (Singapore)
W.J. Hu (USA)
A.Z. Cao (USA)
J. Ni (Singapore)
C.Y. Guo (Singapore)
Y.L. Wu (Singapore)
X.L. Yang (Singapore)
Z.M. Wang (Singapore)
Z. Xu (Singapore)
H.J. Fan (Singapore)
Sumit Bam Shrestha (Singapore)
Lecture notes of postgraduate (PhD) course.
Lecture notes of undergraduate course.
Lecture notes of undergraduate course.
Lecture notes of MSc course.
Lecture notes of MSc course.
Last revised: Date Jan, 2016.