Welcome to Dr. Qing Song’s Home Page

Robust Neural Network Training Algorithms and Artificial Intelligence  

 

Description: Description: Description: Description: Description: Description: Description: \\web\home\eqsong\eqsong.jpg

Robust Training with World Leading Research:

A unique research direction - starting from fundamental theory to real world applications.

 

Biographical Information

Dr. Qing SONG (IEEEM00116020) received the B.S. and the M.S. degrees from Harbin Shipbuilding Engineering Institute and Dalian Maritime University, China in 1982 and 1986, respectively. He obtained the PhD degree from the Industrial Control Centre at Strathclyde University, U.K in 1992. He was an assistant professor at the Dalian Maritime University from 1986 to 1987. He worked as a research engineer at the System and Control Pte Ltd, U.K., before joined the School of Electrical and Electronic Engineering, Nanyang Technological University as a lecturer in 1992. Dr. Song is currently an associate professor and an active industrial consultant at the school of EEE, NTU. He is the principal investigator of a few computational intelligence related research programs targeted for practical applications. He also held visiting research positions at different universities, like The Johns Hopkins University, U.S.A. and  Tsinghua University, P.R.C. etc.

 

Back to top

Research for Future

 

Robust Training Algorithms of Recurrent Neural Network (RNN) and Neural Network (NN) with Real Word Applications

 

While increasing learning rate may help to improve performance of the RNN, it can result in unstable training in terms of weight divergence. Therefore, an optimal trade-off between RNN training speed and weight convergence is desired. Different robust training algorithms of RNN are developed based on a novel RNN hybrid training concept. It switches the training patterns between standard real time on-line Back Propagation (BP) and RTRL according to the derived convergence and stability conditions. Further extension of the new theoretical result to spiking neuron activities are under investigation.

1.     Robust recurrent kernel learning incorporating neural network structural change based on variable length estimation error, which is a kind of policy selection strategies and similar to the basic concept used in Google’s famous AlphaGo, in certain extents. We use only automatic selection of the target function based on a single kernel neural network (be more theoretical sound and easy to use) and the later employs a separated trained deep learning neural network called value network (be more practical orientated and a lot of computing resources, used 1202 CPUs with 176 GPUs in total). How to integrate the two approaches would be an interesting direction in the future. 

2.     Robust training algorithm of recurrent Jordan networks

3.     Robust training algorithms of recurrent Elman networks

4.     Robust Support Vector Machine Based Computerized Auto-scoring System (patented commercial project with CISCO police)

 

Robust Information Clustering

 

The ultimate target of Robust Information Clustering is not to look for or approximate a true probability distribution of input data set (similarly, we are also not looking for true clusters or cluster centers), which is proved to be an ill-posed problem (Vapnik, 1998), but to determine an optimal number of effective clusters by eliminating the unreliable data points (outliers) based on the Chernoff bound with asymptotic expression (especially for the nonlinear robust information clustering algorithm). It also turns out that outliers are in fact data points that fail to achieve capacity. Therefore, any data point could become an outlier as the cluster number is increased in cluster annealing procedure. Furthermore, by replacing the Euclidean distance with other dissimilarity measures, it is possible to extend the new algorithm into the kernel and nonlinear clustering algorithms for linearly non-separable patterns (see the latest research reports).

1.     Robust Information Clustering (basic algorithm with loose bound)

2.     Nonlinear Robust Information-theoretic Clustering (improved algorithm with tighter bound)

3.     MRI image segmentation and classification (joint research with IMH)

4.     Intelligent eye tracking system for disabled people (joint research with TTH)

 

Selected Important Referred Research Journal Publications

 

Sumit Bam Shrestha and Qing Song, “Robustness to Training Disturbances in SpikeProp Learning”, IEEE Transactions on Neural Networks and Learning Systems, Digital Preview Page (99): 1-14, July, 2017.

Sumit Bam Shrestha and Qing Song, “Robust spike-train learning in spike-event based weight update”, Neural Networks, Volume 96, December 2017, Pages 33-46.

Qing Song, X. Zhao, H. J. Fan and D.W. Wang, “Robust Recurrent Kernel Online Learning", IEEE Transactions on Neural Networks and Learning Systems, Vol. 28, No. 5, May, 2017.

(Digital: Qing Song, X. Zhao, H. J. Fan and D.W. Wang, “Robust Recurrent Kernel Online Learning", IEEE Transactions on Neural Networks and Learning Systems, 10.1109/TNNLS.2016.2518223.)

Sumit Bam Shrestha, Qing Song, “Robust learning in SpikeProp”, Neural Networks, Volume 86, February 2017, Pages 54-68.

Xulei Yang, Qing Song and Yi Su, “Automatic segmentation of left ventricle cavity from short-axis cardiac magnetic resonance images”, Medical & Biological Engineering & Computing, Feb. 2016.

Sumit Bam Shrestha, Qing Song, “Adaptive learning rate of SpikeProp based on weight convergence analysis”, Neural Networks, 03/2015.

Haijin Fan, Qing Song, Zhao Xu, “An information theoretic sparse kernel algorithm for online learning”, Expert Systems with Applications 07/2014; 41(9):4349–4359.

Haijin Fan, Qing Song, Sumit Bam Shrestha, “Online Learning with Kernel Regularized Least Mean Square Algorithms”, Knowledge-Based Systems 03/2014.

Haijin Fan, Qing Song, “A linear recurrent kernel online learning algorithm with sparse updates”, Neural networks, 11/2013.

Zhimin Wang, Qing Song, Y.C. Soh, Sim Kang, “An adaptive spatial information-theoretic fuzzy clustering algorithm for image segmentation”, Computer Vision and Image Understanding 10/2013; 117(10):1412-1420.

Qing Song, “Robust initialization of a Jordan Network with Recurrent Constrained Learning”, Special issue of White Box Nonlinear Prediction Models, IEEE Transactions on Neural Networks, Vol. 22, Vol. 12, 2011, pp. 2460-2473.

Z.M. Wang, Qing Song, Y.C. Soh, and Sim Kang, “A Robust Curve Clustering based on a Multivariate Model”, IEEE Transactions on Neural Networks, Vol. 21, No. 12, 2010, pp.1976-1984.

Qing Song, On the Weight Convergence of Elman Networks, IEEE Transactions on Neural Networks, Vol. 21, No. 3,  2010, pp.463-480.

Qing Song, X.L. Yang and Y.C. Soh, “An Information-theoretic Fuzzy C-Spherical Shells Clustering Algorithm”, Fuzzy Sets and Systems, Vol.161, 2010, pp.1755-1773.

Z.M. Wang, Y.C. Soh, Qing Song and Sim Kang, “Adaptive Spatial Information-theoretic Clustering for Image Segmentation”, Pattern Recognition, Vol. 42., 2009, pp. 2029-2044.

C.Y. Guo and Qing Song, “Real-time control of variable air volume system based on a robust neural network associated PI controller”, IEEE Transactions on Control Systems Technology, Vol.17, No. 3, 2009, pp.600-607.

X. L. Yang, Qing Song, Y.L. Wu “A Novel Pruning Approach for Robust Data Clustering, Neural Computing & Applications, Vol. 18,  No.7,  pp.759-768, 2009. 

Qing Song, Y.L.Wu and Y.C. Soh, Robust Adaptive Gradient Descent Training Algorithm for Recurrent Neural Networks in Discrete Time Domain", IEEE Transactions on Neural Networks, Vol. 19, No.11, 2008, pp.1841-1853.

Qing Song, J.Spalls, Y.C.Soh and J.Ni, "Robust Neural Network Tracking Controller Using Simultaneous Perturbation Stochastic Approximation, IEEE Transactions on Neural Networks, Vol.19, No.5, 2008, pp.817-835.

Y.L. Wu, Qing Song, and S. Liu,  "A Normalized Adaptive Training of Recurrent Neural Networks with Augmented Error Gradient" , IEEE Transactions on Neural Networks, Vol.19, No.2, 2008, pp.351-356.

A. Z. Cao and Qing Song, “Robust Information Clustering for Automatic Breast Mass Detection in Digitized Mammograms”, Computer Vision and Image Understanding, Vol. 109, No.1, 2008,  pp.87-96.

Xulei Yang, Qing Song, "A weighted support vector machine for data classification", International Journal of Pattern Recognition and Artificial Intelligence, 21 (5), 2007, pp.961-976.

Jie Ni and Qing Song, “Pruning Based Robust Backpropagation Training Algorithm for RBF Network Tracking Controller”, Journal of Intelligent and Robotics Systems, Vol. 48, No.3, 2007, pp.375-396.

Y.L. Wu and Qing Song, “Robust Recurrent Neural Control of Biped Robot”, Journal of Intelligent and Robotics Systems, No. 2, Vol. 49, 2007, pp.151-169.

Xulei Yang and Qing Song, “A Robust Deterministic Annealing Algorithm for Data Clustering”, Journal of Data & Knowledge Engineering, 62 (1), 2007, pp.84-100.

Chengyi Guo, Qing Song, and W.J. Cai, “Neural Network Assisted Cascade Control System for Air Handling Unit”, IEEE Transactions On Industrial Electronic,Vol. 54, No.1, 2007, pp.620-628. 

J. Ni and Qing Song. ”Dynamic Pruning Algorithm for Multilayer Perceptron Based Neural Control Systems”, Journal of Neurocomputing, Vol. 69, Issues 16-18, 2006.

Xulei Yang and Qing Song, Kernel-based deterministic annealing algorithm for data clustering, IEE Proceedings: Vision, Image and Signal Processing, Vol. 153, No.5, 2006, pp. 557-568.

Xulei Yang, Qing Song, A.Z. Cao, " A weighted deterministic annealing algorithm for data clustering", International Journal of Computational Intelligent Research, Vol.2,  No.1, 2006, pp.81-85.

Xulei Yang, Qing Song and A. Z. Cao, “A New Cluster Validity for Data Clustering”, Neural Processing Letter, 23 (3), 2006, pp. 325-344.

Qing Song, “A Robust Information Clustering Algorithm”, Neural Computation, MIT, Vol.17, No.12, 2005, pp.2672-2698.

W.J.Hu and Qing.Song “An Accelerated Training Algorithm for Robust Support Vector Machine”, IEEE Transactions on Circuits and Systems II, Vol. 51, No.5, 2004.

Qing. Song, W. J. Hu, “Robust Neural Controller for VAV System”, IEE Proceedings Part D - Control Theory and Applications, Vol. 150, No. 2, 2003.

Qing. Song, W. J. Hu, W. F. Xie, “Robust Support Vector Machine for Bullet Hole Image Classification”, IEEE Transactions on Systems, Man and Cybernetics, Part C, Vol. 32, 2002.

Qing Song, L. Yin, Y. C. Soh, “Robust Identification of Nonlinear Plant using Neural Networks”, Asian Journal of Control, Vol. 3, No. 2, 2001, Special Issue on Advances in Neural and Fuzzy Controllers.

Qing Song, L. Yin, Y. C. Soh, “Robust Adaptive Dead Zone Technology for Fault Diagnosis and Control of Robot Manipulators using Neural Networks”, Journal of Intelligent and Robotics Systems, the Netherlands, 2001.

Qing Song, Lin Yin, “Robust Adaptive Fault Accommodation for Robot System using RBF Neural Network”, International Journal of Systems Science, UK, Vol. 32, No. 2, 2001.

Qing Song, “Design of Robust Neural Tracking Controller”, Journal of Intelligent and Robotics Systems, the Netherlands, Vol. 20, 2000.

D. J. Hou, Qing Song, “Computerised Auto-Scoring System Based upon Feature Extraction and Neural Network Technologies”, Journal of Intelligent and Robotics Systems, the Netherlands, Vol. 29, 2000, Special Issue of Neural Network and Image Processing.

Qing Song, J. Xiao, Y. C. Soh, “Robust Back-Propagation Algorithm for Multi-Layered Neural Tracking Controller”, IEEE Transactions on Neural Networks, USA, Vol. 10, No. 5, 1999,  pp. 1133-1141.

Qing Song, “Implementation of Two Dimensional Systolic Algorithms for Multilayered Neural Networks”, Journal of Systems Architecture, Italy, Vol. 48, pp.1209-1218, 1999.

Qing Song, L. Yin, Y. C. Soh, “Shifting of the Center of Radial Basis Function Neural Network in the Presence of Disturbance”, Neural and Parallel Computation, USA, Vol. 7, No. 1, pp. 121-131, 1999.

Qing Song, “Robust Training Algorithm of Multi-Layered Neural Network for Identification of Nonlinear Dynamic Systems”, IEE Proceedings Part D - Control Theory and Applications, UK, Vol. 145, No. 1, 1998.

Qing Song, M. J. Grimble, “Design of a Multivariable Neural Controller and its Application to Gas Turbine”, ASME Journal of Dynamic Systems, Measurement and Control, USA, Vol. 119, No. 3, 1997.

Qing Song, J. Xiao, “On the Convergence Performance of Multi-Layered NN Tracking Controller”, Neural and Parallel Computation, USA, Vol. 5, No. 3, 1997.

Qing Song, “Training of a Single Perceptron Neuron using Robust Projection Algorithm”, Neural and Parallel Computation, USA, Vol. 4, No. 2, pp. 193-203, 1996.

Qing Song, E. K. Teoh, D. P. Mital, “Multilayered Neural Network Implementation on Transputer Systolic Array”, Journal of Microprocessing and Microprogramming, the Netherlands, Vol. 41, , pp. 289-299, 1995.

Qing Song, M. R. Katebi, M. J. Grimble, “Robust Multivariable Implicit Adaptive Control”, IMA Journal of Mathematical Control and Information, UK, Vol. 10, pp. 49-70, 1993.

Qing Song, J. Wilkie, M. J. Grimble, “Robust Controller for Gas Turbines Based Upon LQG/LTR Design with Self-Tuning Features”, ASME Journal of Dynamic Systems, Measurement and Control, USA, Vol. 115, No. 3, pp. 569-571, 1993.

 

Back to top

Current and past post-graduate research fellows/students (current location)

 

W.F. Xie (Canada)

G. Yin (USA)

J.Z. Xiao (USA)

L. Yin (Singapore)

W.J. Hu (USA)

A.Z. Cao (USA)

J. Ni (Singapore)

C.Y. Guo (Singapore)

Y.L. Wu (Singapore)

X.L. Yang (Singapore)

Z.M. Wang (Singapore)

Z. Xu (Singapore)

H.J. Fan (Singapore)

Sumit Bam Shrestha (Singapore)

 

Teaching for Generations to Come

EE7201 Computational Method for Engineering

Lecture notes of postgraduate (PhD) course.

EE4285 Computational Intelligent Systems

Lecture notes of undergraduate course.

EE4205 Biomedical Signal Processing

Lecture notes of undergraduate course.

M6225 Medical Informatics

Lecture notes of MSc course.

EE6224 Neural Networks & Fuzzy Systems

Lecture notes of MSc course.

 

Back to top

 

Contact Information

E-mail address

eqsong@ntu.edu.sg

Web address

www.ntu.edu.sg/home/eqsong

Office phone

(65)-67904530

 

Back to top

 

Hobbies

Tennis

Ski 

Golf

Bird

 

Back to top

 

Last revised: Date Jan, 2016.