Add to Chrome
Add to Firefox
Add to EdgeBashima Islam
Classification of Infant Sleep/Wake States: Cross-Attention among Large Scale Pretrained Transformer Networks using Audio, ECG, and IMU Data
Jun 27, 2023
Infant sleep is critical to brain and behavioral development. Prior studies on infant sleep/wake classification have been largely limited to reliance on expensive and burdensome polysomnography (PSG) tests in the laboratory or wearable devices that collect single-modality data. To facilitate data collection and accuracy of detection, we aimed to advance this field of study by using a multi-modal wearable device, LittleBeats (LB), to collect audio, electrocardiogram (ECG), and inertial measurement unit (IMU) data among a cohort of 28 infants. We employed a 3-branch (audio/ECG/IMU) large scale transformer-based neural network (NN) to demonstrate the potential of such multi-modal data. We pretrained each branch independently with its respective modality, then finetuned the model by fusing the pretrained transformer layers with cross-attention. We show that multi-modal data significantly improves sleep/wake classification (accuracy = 0.880), compared with use of a single modality (accuracy = 0.732). Our approach to multi-modal mid-level fusion may be adaptable to a diverse range of architectures and tasks, expanding future directions of infant behavioral research.
RecNet: Early Attention Guided Feature Recovery
Feb 18, 2023
Uncertainty in sensors results in corrupted input streams and hinders the performance of Deep Neural Networks (DNN), which focus on deducing information from data. However, for sensors with multiple input streams, the relevant information among the streams correlates and hence contains mutual information. This paper utilizes this opportunity to recover the perturbed information due to corrupted input streams. We propose RecNet, which estimates the information entropy at every element of the input feature to the network and interpolates the missing information in the input feature matrix. Finally, using the estimated information entropy and interpolated data, we introduce a novel guided replacement procedure to recover the complete information that is the input to the downstream DNN task. We evaluate the proposed algorithm on a sound event detection and localization application where audio streams from the microphone array are corrupted. We have recovered the performance drop due to the corrupted input stream and reduced the localization error with non-corrupted input streams.
Zygarde: Time-Sensitive On-Device Deep Intelligence on Intermittently-Powered Systems
May 05, 2019
In this paper, we propose a time-, energy-, and accuracy-aware scheduling algorithm for intermittently powered systems that execute compressed deep learning tasks that are suitable for MCUs and are powered solely by harvested energy. The sporadic nature of harvested energy, resource constraints of the embedded platform, and the computational demand of deep neural networks (even though compressed) pose a unique and challenging real-time scheduling problem for which no solutions have been proposed in the literature. We empirically study the problem and model the energy harvesting pattern as well as the trade-off between the accuracy and execution of a deep neural network. We develop an imprecise computing-based scheduling algorithm that improves the schedulability of deep learning tasks on intermittently powered systems. We also utilize the dependency of the computational need of data samples for deep learning models and propose early termination of deep neural networks. We further propose a semi-supervised machine learning model that exploits the deep features and contributes in determining the imprecise partition of a task. We implement our proposed algorithms on two different datasets and real-life scenarios and show that it increases the accuracy by 9.45% - 3.19%, decreases the execution time by 14\% and successfully schedules 33%-12% more tasks.
Intermittent Learning: On-Device Machine Learning on Intermittently Powered System
Apr 21, 2019
In this paper, we introduce the concept of intermittent learning, which enables energy harvested computing platforms to execute certain classes of machine learning tasks. We identify unique challenges to intermittent learning relating to the data and application semantics of machine learning tasks. To address these challenges, we devise an algorithm that determines a sequence of actions to achieve the desired learning objective under tight energy constraints. We further increase the energy efficiency of the system by proposing three heuristics that help an intermittent learner decide whether to learn or discard training examples at run-time. In order to provide a probabilistic bound on the completion of a learning task, we perform an energy event-based analysis that helps us analyze intermittent learning systems where the uncertainty lies in both energy and training example generation. We implement and evaluate three intermittent learning applications that learn the air quality, human presence, and vibration using solar, RF, and kinetic energy harvesters, respectively. We demonstrate that the proposed framework improves the energy efficiency of a learner by up to 100% and cuts down the number of learning examples by up to 50% when compared to state-of-the-art intermittent computing systems without our framework.