Residual Networks
Residual Neural Networks or ResNet for short, are a form of artificial Neural network. ResNet network design can be predominantly used in Super Resolution Models due to the availability of the SRResNet architecture.
EDSR (Enhanced Deep Residual Networks for Single Image Super-Resolution)
EDSR can handle specific super-resolution scales. It improves the performance of SR for single-scale architectures. Its architecture is based on SRResNet architecture, but it has no Batch Normalization layers because it normalizes the input, which results in limiting the range of the network, and the removal of BN results in an improvement in the accuracy of the model. The BN layers also consume 40% of the memory available. So, its removal results in memory reduction and makes the network training better. They make use of residual blocks as shown in the diagram below:
MDSR (Multi-scale Deep Super-Resolution system)
MDSR is an extension of the EDSR. It reconstructs various scales of high-resolution images in a single model. It has multiple input and output modules that give corresponding resolution outputs at 2x, 3x, and 4x. A larger kernel is used here as the pre-processing layers, which makes the network simple, while still attaining a high receptive field. The common shared residual blocks at the end of scale-specific pre-processing modules for all resolutions. After the upsampling, the depth of MDSR will reach 5 times as compared to single-scale EDSR. It can give comparable results to scale-specific EDSR combined model with lesser parameters.
Other Network Designs
Apart from Residual Networks, these are some other Network Designs that can be used in designing SR models:
- Recursive Network
- Dense Connection Network
- Group Convolution Network
- Local Multi-path Network
However, Residual Network is preferred because of the availability of residual blocks.
Generative Models (GAN)
Generative models (GAN) optimize the quality to produce images that are pleasant to the human eye because humans don’t distinguish images by pixel difference. The networks optimize the pixel difference between expected and output HR images.
Some commonly used GAN architectures are:
- SRGAN
- ESRGAN
SRGAN
Same to GAN, SRGAN has also Generator and Discriminator. This framework supports 4x upscaling factors. It uses a perceptual loss function which is a weighted sum of an adversarial loss and a content loss. The adversarial loss pushes the solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original images.
The generator network comprises the residual blocks. They make use of skip connections for easier training. The discriminator network discriminates the read HR image and the obtained output HR image.
Python OpenCV – Super resolution with deep learning
Super-resolution (SR) implies the conversion of an image from a lower resolution (LR) to images with a higher resolution (HR). It makes wide use of augmentation. It forms the basis of most computer vision and image processing models. However, with the advancements in deep learning technologies, deep learning-based super resolutions have gained the utmost importance. Almost all the deep learning models would make great use of super-resolution. Since Super Resolution mainly uses augmentations of data points, it is also called hallucination of the data points.
SR plays an important role in image improvement and restoration. The SR process is carried out as follows- First, a low-resolution image is taken as the input. Next, the image is upscaled and the resolution of the images are increased to a higher resolution and given as an output.
Need for Deep learning based Super Resolution
The traditional Super Resolution Model that does not make use of Deep learning lacks fine details. They fail to remove various defects and compression facts in the systems. All of these problems can be very efficiently addressed by using a deep learning-based SR model to get an image of a higher resolution keeping all the details intact.
Some commonly used conventional SR models are
- Structured illumination microscopy (or SIM)
- Stochastic optical reconstruction microscopy (STORM)
- Photo-activated localization microscopy (PALM)
- Stimulated emission depletion (STED)