Electrical and Electronics Engineering PhD Thesis Defense by Altynbek Isabekov

KOÇ UNIVERSITY

GRADUATE SCHOOL OF SCIENCES & ENGINEERING

ELECTRICAL AND ELECTRONICS ENGINEERING

PhD THESIS DEFENSE BY ALTYNBEK ISABEKOV

Title: On the Importance of the Hidden Bias and Hidden Entropy in Representational Efficiency of the Gaussian-Bipolar Restricted Boltzmann Machines

Speaker: Altynbek Isabekov

Time: March 8, 2018, 9:30 AM

Place: ENG 208

Koç University

Rumeli Feneri Yolu

Sariyer, Istanbul

Thesis Committee Members:

Assoc. Prof. Dr. Engin Erzin (Advisor, Koç University)

Prof. Dr. Alper Erdoğan (Koç University)

Prof. Dr. Yücel Yemez (Koç University)

Prof. Dr. Murat Saraçlar (Boğaziçi University)

Assoc. Prof. Dr. Burak Acar (Boğaziçi University)

Abstract:

With development of machine learning and deep learning fields, the importance of unsupervised learning algorithms also increases. One of these algorithms is Gaussian-Bernoulli Restricted Boltzmann Machines (GBLRBMs), which are capable of modelling real-valued data. Moreover, GBLRBMs are used to pretrain weights in artificial neural networks, which improves performance of these networks. In this work, we analyze the role of hidden bias in representational efficiency of the Gaussian-Bipolar Restricted Boltzmann Machines (GBPRBMs), which are similar to the widely used Gaussian-Bernoulli RBMs. Our experiments show that hidden bias plays an important role in shaping of the probability density function of the visible units. Correspondingly, we define hidden entropy and propose it as a measure of representational efficiency of the model. By using this measure, we investigate the effect of hidden bias on the hidden entropy and provide a full analysis of the hidden entropy as function of the hidden bias for small models with up to three hidden units. We also provide an insight into understanding of the representational efficiency of the larger scale models. Furthermore, we introduce Normalized Empirical Entropy as an approximation of hidden entropy that can be computed for large models. Experiments on the MNIST and synthetic data show that this approximation can serve as measure of representational efficiency and gives an insight on minimum number of hidden units required to represent the data.