Discriminative Dimensionality Reduction using Deep Neural Networks

Bahaadini Beigy Zarandi, Sara

doi:https://doi.org/10.21985/n2-tr2h-n980

Work

Discriminative Dimensionality Reduction using Deep Neural Networks

Public

Download PDF

Deep neural networks have shown impressive performance for many applications. In this dissertation, leveraging the capabilities of neural networks for modeling the non-linearity exists in the data, we propose several models that can project data into a low dimensional, discriminative, and smooth manifold. The suggested models can transfer knowledge from the domain of known classes to a new domain where the classes are unknown. A clustering algorithm is further applied to the new domain to find potentially new classes from the pool of unlabeled data. The research problem and data for this doctoral thesis comes from the Gravity Spy project which is a side project of Advanced Laser Interferometer Gravitational-wave Observatory (LIGO). LIGO project is an effort for detecting cosmic gravitational waves using huge detectors. However non-cosmic, non-Gaussian disturbances known as ``glitches'', show up in gravitational-wave data of LIGO. This is undesirable as it makes problems for the gravitational wave detection process. Gravity Spy aids in the characterization of glitches by combining human intuition and pattern recognition with the power of computers to process large amounts of data. The primary purpose of glitches identification is to understand their characteristics and origin, which facilitates their removal from the data or the detector entirely. In this dissertation, we first present the process of making dataset from the pool of glitches and then our machine learning models for the characterization of the glitches into morphological families (referred to classes). At first, we identify initial glitch classes based on morphology and structure and make a dataset. This dataset, named Gravity Spy dataset, is used to train the machine learning algorithm and we have released it for public use to help other researchers. Then we develop an automatic machine learning classifier for glitches. Having more than one duration for each glitch example, we consider such extra duration as novel views and develop multi-view deep neural networks for glitch classification. Furthermore, we propose an ultimate classifier framework that combines the advantage of all other base classifiers including deep multi-view and traditional machine learning algorithm. The proposed model achieves better results compared to the baselines. After making the dataset and developing the initial machine learning tools, we focus on developing deep neural network based dimensionality reduction models. First, we propose a fully supervised model that learns a non-linear mapping function using deep neural networks. The proposed model, named the Deep Discriminative Embedding for Clustering (DIRECT), can transfer the knowledge from the space of morphologically known glitch classes to the new space. The performance of the clustering algorithm employed on the new feature space is better than the baseline features. To further improve the performance of DIRECT and make it even more robust and transferable to a new domain, we develop a semi-supervised deep model that consists of two main parts: i) auto-encoder based component that can learn the general salient structures from unlabeled data, and ii) a discriminative component which uses the labeled data and encourages a discriminative feature space. The two components are tied together to allow an end-to-end training of the two components. Our experimental results on the task of clustering show that the proposed model outperforms the fully supervised model and also the other baselines. Although adding the auto-encoder based loss function to the discriminative part, made it more powerful and also applicable for the new domain (new classes that have never seen during training), it is a ``straightforward" way of using unlabeled data. We improve the discriminative dimensionality reduction model by adding a virtual adversarial training (VAT) loss to the model. The motivation for this work is to have a dimensionality reduction model that can provide a smooth manifold. The sensitivity of the model's output to small perturbations in the inputs is not desirable for machine learning models. Making the model robust to such perturbations can be considered as regularizing the model. We use the regularization effect of VAT to further improve the discriminative dimensionality reduction model. The experimental results show that the VAT based model outperforms all the baselines by a significant margin.

Creator