On Norm Preservation of Residual Neural Networks (ResNets)

Augmenting neural networks with skip connections, as introduced in the so-called ResNet architecture, surprised the community by enabling the training of networks of more than 1,000 layers with significant performance gains. This paper deciphers ResNet by analyzing the effect of skip connections, and puts forward new theoretical results on the advantages of identity skip connections in neural networks. We prove that the skip connections in the residual blocks facilitate preserving the norm of the gradient, and lead to stable back-propagation, which is desirable from optimization perspective. We also show that, perhaps surprisingly, as more residual blocks are stacked, the norm-preservation of the network is enhanced. Our theoretical arguments are supported by extensive empirical evidence.

Can we push for extra norm-preservation? We answer this question by proposing an efficient method to regularize the singular values of the convolution operator and making the ResNet’s transition layers extra norm-preserving.

On Norm Preservation of ResNets Project Page

Learning with Less Labeled Data and Data Selection

Thanks to recent advances in computing, deep learning based systems, which employ very large numbers of inputs, have been developed in the last decade. However, processing/labeling/communication of a large number of input data has remained challenging. Therefore, novel machine learning algorithms that make the best use of a significantly less amount of data are of great interest. We have introduced two novel data selection algorithms with many applications in different problems such as active leaning, database summarization, video summarization,  training Generative Adversarial Networks (GANs) with less data, …. For more information on our algorithms, Iterative Projection and Matching (IPM) (CVPR’19) and Spectrum Pursuit (SP) (CVPR’20) please see below:

IPM Project Page

SP Project Page

Radio Environment Map (REM)

The main objective of this project is to devise an efficient framework for achieving RF spectrum awareness through generating a reliable and dynamic radio environment map, also referred to as radio (spectrum) cartography. We are tackling this problem using tensor data analysis tools and compressive sensing.

Tensor Data Analysis

TBA

Spectrum Sensing in Cognitive Radio Networks

TBD

Adaptive Compressive Sensing

TBD

Related Publications: Compressed Sensing and Compressed Sensing Applications

Compressive Sensing: Design and Applications

The emerging field of compressive sensing (CS) has overturned the traditional concept of sensing and sampling, established by Nyquist sampling theorem, for signals that have a sparse representation over some proper basis. We are studying new CS sampling and recovery schemes with low complexity and improved performance. Our CS framework offers the flexibility to utilize problem-specific signal and systems properties such as non-uniform sparsity or time-correlation of signals. We are also exploring new exciting applications for CS.

Related Publications: Compressed Sensing and Compressed Sensing Applications

Cross-layer Protocol Designs in Sensor Networks Utilizing Compressive Sensing (CS)

Since sparsity is a feature presented in many physical phenomena, CS can be utilized in different applications in wireless sensor networks (WSNs). Our approach is to our proposed CS techniques (such as non-uniform CS and time-correlated CS ) into underlying networking protocols (routing, MAC,…) for cross-layer optimization to minimize the overall cost of sensing. Such cross-layer designs not only help to reduce the number of required samples, but they also facilitate reducing high RF transmission cost associated with communication among nodes. We have studied efficient data dissemination, distributed data storage, and efficient data acquisition in WSNs.

Cross-layer Protocol Designs in Sensor Networks Utilizing Compressive Sensing (CS)

Since sparsity is a feature presented in many physical phenomena, CS can be utilized in different applications in wireless sensor networks (WSNs). Our approach is to our proposed CS techniques (such as non-uniform CS and time-correlated CS ) into underlying networking protocols (routing, MAC,…) for cross-layer optimization to minimize the overall cost of sensing. Such cross-layer designs not only help to reduce the number of required samples, but they also facilitate reducing high RF transmission cost associated with communication among nodes. We have studied efficient data dissemination, distributed data storage, and efficient data acquisition in WSNs.

Unequal Error Protection (UEP) Rateless coding: Design and Applications

Rateless codes are a new class of codes that have been invented recently. Rateless codes on lossy channels do not assume any knowledge about the channel, unlike the traditional codes. This feature makes them very interesting in the applications that the channel loss is unknown, time-varying, or nonuniform. In the original work on rateless codes, equal error protection (EEP) of all data was considered. EEP would be sufficient in the applications such as multicasting bulk data. However, in several applications, a portion of data may need more protection than the rest of data. For example, in an MPEG stream, I-frames need more protection that P-frames.

In some other applications, a portion of data may need to be recovered prior to the other parts. An example would be on-demand media streaming, in which the stream should be reconstructed in sequence. Such applications raise a need for having codes with unequal error protection (UEP) or unequal recovery time (URT) property. We developed , for the first time, rateless codes that can provide UEP and URT. We analyzed the proposed codes under both iterative decoding and maximum-likelihood decoding. Results are very promising and show the applicability of UEP-rateless codes in many important applications, such as transferring data frames or video/audio-on-demand streaming.

Related Publications

UEP Rateless Coding for Video Sptreaming

Rateless codes with unequal error protection (UEP) property can be employed to increase the video transmission efficiency.

First, we employed the UEP rateless codes in video-on-demand (VOD) systems. In VOD systems, required bandwidth is decreased to a great extend by sharing a single video stream among several users watching the same video. We proposed to employ unequal error protection (UEP) rateless codes in VOD systems and have show this scheme decreases the initial waiting time considerably compared to the case where equal error protection (EEP) codes are employed.

Second, we studied the application of UEP rateless codes for efficient MPEG video transmission. MPEG movie has different frame types namely, I, P, and B, which have different levels of importance. UEP rateless codes can be efficiently used to increased the MPEG video transmission efficiency.

Distributed Rateless Codes with UEP Property

When multiple sources of data need to transmit their rateless coded symbols through a single relay to a common destination, a distributed rateless code can be employed to encode the source symbols instead of several separated conventional rateless codes to increase the transmission efficiency and flexibility.

In this research, we have proposed DU-rateless codes, which are distributed rateless codes that can provide UEP for sources with different data lengths (see right figure). We have designed degree distributions for DU-rateless codes using genetic-algorithms. The degree distributions are available HERE for ρ={0.3,0.5,1}.

Provide desired 1<η and ρ={0.3,0.5,1} to the Matlab function enclosed to find the appropriate optimum DU-rateless code.

> [dd1 dd2 p1 p2 p3] = best_DUrateless_finder( [desired η], [desired ρ] )

Example in Matlab environment for ρ=1 and η=10:

>> [dd1 dd2 p1 p2 p3] = best_DUrateless_finder(10,1)

Rateless Codes with Optimum Intermediate Performance

We have designed several rateless codes similar to LT codes with optimum intermediate recovery rate. You can download the databases for optimum degree distribution of 4 different lengths (Asymptotic, k=10,000, k=1000, k=100) in Matlab MAT file format. The first three columns are the packet error rate of each degree distribution at received overheads of 50%, 75% and 100%, respectively. The rest of each column shows the respective degree distribution from degree one and up. All files are available in one compressed file that you can download HERE. The Matlab function which returns the best solution out of many optimum degree distributions according to three weights assigned to each objective function is also included in the compressed file.

Please load the MAT files before running the function with command “load x.mat”. Replace “x” with desired results database file name. Then provide desired weights to the function for find optimum degree distribution.

Example in matlab environment: we want to find the best degree distribution for rateless codes of length 10k in intermediate range with equal weights of one. The input weights are for packet error rates at 50%, 75% and 100%, respectively

>> load costs_final_k10000

degree_distr = best_degree_finder(1,1,1,costs_final)

Minimum-Energy Broadcast in Wireless Sensor Networks Using Rateless Coding

Efficient network-wide broadcasting is an important issue in wireless networks that attracted a lot of attention. In this research, we considered the case that a large amount of packets have to be broadcasted in a multihop wireless network with our main concerns being reliability and energy-efficiency. We proposed a two-phase broadcasting scheme referred as Collaborative Rateless Broadcast (CRBcast). Our two-phase protocol is based on Probabilistic Broadcasting (PBcast) and an application layer rateless coding. At the first phase, the rateless-encoded packets are broadcasted based on PBcast, in which each node probabilistically relays every new received packet. The second recovery phase, which is based on simple collaborations of nodes, ensures that all nodes can recover original data. We showed that CRBcast can provide both reliability and energy efficiency. Simulation results indicate that CRBcast saves at least 72% and 60% energy in comparison with flooding and PBcast, respectively. We implemented CRBcast protocol in a testbed including MICAz motes with TinyOS distributed software operating system. The result showed similar improvement in the energy efficiency while providing reliability. We are currently working on another scheme for reliable and energy-efficient one-to-all broadcasting in multihop wireless networks, where each link is modeled as a packet erasure channel. In this scheme, referred to as Fractional Transmission Scheme (FTS), rateless coding enables each node to send a fraction of the total encoded packets.