Literature Survey on Super Resolution and its Challenges
Murali Krishna AtmakuriAsst. Prof,Dept. of ECE
RVR&JCCE,[email protected] Kumar KattaAsst. Prof,Dept. of ECE
RVR&JCCE,[email protected] Prasad
professor&Head, Dept. of ECE
GEC,[email protected] In the recent trends, image processing field became an interesting area for researchers due to vast advancements over the few decades. Now-a-days one of the most recent and important trend of image processing is Super Resolution. Super resolution utilizes the features of reconstruction; reconstruction is a kind of producing high spatial image from one or more low resolution images. Super resolution combines the non-redundant information of low-resolution images to develop high-resolution images. This article envisages the recent advances in super-resolution techniques and provides its advantages and disadvantages. This article also explains the challenges of super resolution and the scope further research studies.
Index Terms- Image Resolution, Super Resolution, Interpolation, Wavelet Transform, Learning, Reconstruction.
Vision is one of the most valuable of five senses, so images not only play the important role but also used to make decisions based on human perception. In order to improve human perception, it is important to use high resolution images 1-6. High resolution images are used in various kinds of applications, some of the examples are as follows: Military and Civilian 7-10. The recent trends in image and video sensing have been intensified by the expectations of the user on visual quality of captured-data 11, 12. High quality visual captured-data can be obtained with the help of high resolution cameras. The limitation of using high resolution cameras are: Expensive, Need high power, Need high memory size and limited band width even though the high resolution cameras used, sometimes it is not possible to obtain high resolution image 13. To overcome above limitations, super resolution would be an effective solution. In many of the digital image processing applications, high resolution images are used. High image resolution gives more details about the image 14-18. Super resolution became an interesting area for researcher since more the resolution gives more data about image 19-22. Resolution is generally determined by pixel density 23.
Figure 1 depicts the generic image acquisition process, where diverse factors affect the image quality like: Over the air (OTA), Charge-couple device (CCD), Pre-processors and Environment. Optical Blur is caused by non-symmetric design of the lens.
The apparatus before or back to the optic center of the lens lead to image distortions or sometimes
optical blur. Motion blurs are may be due to rapid movement (or) long exposure of camera. Another undesirable quality of image information is noise. Noise is an undesirable quantity which manipulates the actual information of image.
The better way of obtaining high resolution (HR) image is by placing HR sensors; the installation of HR sensors is not practically a perfect solution and is expensive. The placing of HR sensors leads to high power consumption and makes it bulky and complex. The following is an example of HR imaging system. A modest example of it is imaging system in satellites or imaging systems for medical applications, where we can’t use a HR sensor. Therefore in order to roll over these limitations, post processing is needed to make a better resolved image that holds more information. One of the better alternatives for this is super resolution. Super resolution is the process of obtaining HR image with the help of multiple low resolution (LR) images. In the recent trends such methodology is more active research area and is called super resolution or Resolution Enhancement 21.
IMAGE sUPER RESOLUTION
Super resolution was obtained either by using multiple low resolution images as input and generating a high detailed having a single super resolved image as deliverable or improving the details in a single low resolved image and generating a high resolved image for analysis. In SR from multiple LR images, it is a construction of HR image from several LR images, thereby increasing the high frequency components. The basic idea is combining non-repetitive information contained by multiple LR images.
Fig. 1. Generic image acquisition system
Restoration for noise and blur removal
Registration or Motion estimation
Interpolation onto a high resolution grid
Input: LR IMAGES Output: SR IMAGES
PAGE * MERGEFORMAT 2
Fig. 2. Basic super resolution reconstruction stages
The major advantages of SR approach is that, a HR image can be obtained even with the existing LR imaging with lower cost and less power consumption.
Generally, a super resolution imaging comprises of the following simple processing sequence: (1) Registration, (2) Interpolation and (3) De-blurring or noise elimination.
Image registration is the method of over-lapping number of images of the similar scene which has been taken from various angles by the sensors. In registration 2 or more images are aligned geometrically to obtain the information through image fusion or change observation.
Interpolation: It is the process of estimating the intermediate pixels between the pixel values. When any image is converted from LR to HR, intermediate gaps are introduced and these values have to be estimated and filled with interpolation process.
The process of interpolation inhibits various artifacts the resultant image will be blurred or noisy. Through divergent filters and methodologies, noise will be eliminated and finally a super resolution image is obtained.
APPROACHES TO SUPER RESOLUTION
Super resolution methodologies are of various types such as: (1) Frequency domain approach and (2) Spatial domain approach.
Frequency domain approach
Transform the LR image into frequency domain by applying Discrete Fourier Transform (DFT) and combine them according to the relationship between the aliased DFT coefficients of the observed LR images and that of the unknown high-resolution image. The combined data are then transformed back to the spatial domain where the new image could have a higher resolution than that of the input images 3. The principle of frequency domain approach is as follows: i) what is the shift property of the Fourier transform, ii) The aliasing between the continuous Fourier transform (CFT) of an original HR image and the discrete Fourier transform (DFT) of combined LR images, iii) the presumption that an original HR image is band limited.
Frequency domain approach has some features to enhance the details by extrapolating the high frequency information presented in LR images, and it provides lesser computational complexity. But the problem is that it is incapable of handling the practical applications.
Spatial Domain Approach
The frequency domain approach has several confines like: i) It limits the inter-frame motion to be translational; also it is very hard in frequency domain to use the prior knowledge. As the main problem is ill-posed image in SR, prior knowledge is required to overcome this. The main benefit of spatial domain is the support for unbind motion between frames and prior knowledge availability for solving the problems. Some of the methods are interpolation, iterative back projection and projection onto convex.
Interpolation is the process of transferring image from one resolution to another without losing image quality. Image interpolation can be used in several image processing applicatios for image zooming, enhancement of image, resizing and many more. Most common interpolation techniques are nearest neighbor, bilinear and cubic convolution. An image is a two dimensional signal represented as brightness vs spatial coordinates. An analog image can be transformed into digital domain by sampling and quantization process. The basic element of an image is a pixel. When we increase the resolution of image from low to high, it is called up-sampling or up-scaling while reverse is called down sampling or down scaling.
Interpolation is of three types: (i) Bi-linear Interpolation: Bi-linear interpolated point is filled with weighted average of it’s four closest pixel’s. Bi-linear interpolation is recommended for continuous data like elevation and raw slope values. (ii) Bi-cubic Interpolation: Bi-cubic interpolation is recommended for smoothing continuous data, but this incurs a processing performance overhead and (iii) Nearest Neighbor Interpolation: In this method, nearest value is copied for interpolation and this technique has less computational complexity. Nearest neighbor interpolation is recommended for categorical data such as land use classification.
3.2.2. Iterative Back Projection (IBP)
In IBP approach, HR image is estimated by back projecting the difference between the simulated LR image and captured LR on interpolated image. This iterative process of SR does iterations until the minimization of the cost function is achieved.
3.2.3. Classical Multi-Image Super Resolution
In the classical multi-image SR, a set of LR images of the same scene is taken. If enough LR images were available then the equation is determined and a SR image is reconstructed. The assumption here is that the two or more LR images should contain distinguishable features. This approach would give poor results, if distinguishable features in LR images are less.
3.2.4. Example Based Super Resolution
In Example- Based approach, the same rule is applied. This approach is useful when only single LR image is available. In this approach, the image has small patches that redundantly reappear, both within the scale as well as across the scale. Each LR patch in an image is replaced by its corresponding HR patch to generate the SR image. Here assumption is that, the image should have enough HR patches for the correspondence LR patches.
3.2.5. Learning Based Super Resolution
It is a concept of machine learning, where the machine is trained to classify LR images and its corresponding HR patches. In this approach, both LR and HR patches are divided into different classes. Hence, the number of comparisons will be decreased, as it has to compare LR with only HR patches. For an edge-area of the LR image, routine example-based image SR algorithm can be preferred to implement the local and fine SR. For the flat regions of the low-resolution, only interpolation algorithm can be applied for super-resolution. The performance of learning based super-resolution depends on HR patches retrieved from the training data of an input LR patch.
Table 1. Comparison among various super resolution approaches 21.
Categorization Description Disadvantages
Interpolation Based Different interpolation techniques can be used Over-smooth jagged artifacts
Reconstruction Based Reconstruction constraint and image prior Ringing artifacts, imposing additional prior
Learning Based Learning high frequency details from the trained data. High frequency artifacts.
CHALLENGING ISSUES OF SUPER RESOLUTION
In practice developing super resolution image, there are many challenging issues. Some of the challenging issues are mentioned below:
Image registration is a common problem because of ill-posed images. Image registration becomes more and more difficult when the input LR image is having very high aliasing effects. The registration error increases with decrease in the resolution of input images. This registration error affects the quality of an image resolution more than that of interpolation 21.
Real time application is always requires good efficiency. As there are large numbers of unknowns in reconstructing super resolution images, computational complexity of matrix increases.
Super resolution techniques are defense-less to motion errors, inaccurate blur models, noise, moving objects, motion blur etc. These effects are not easy to estimate which are not acceptable in many applications.
SCOPE OF FURTHER WORK
In general, some methods of image enhancement will be used in the sharp edges area such as over-shoots and under- shoots. The significance of image resolution algorithms is that to enhance the image information. The main loss in image resolution enhancement by using interpolation technique is that changing of its high frequency components i.e. edges which are due to the smoothing caused by interpolation. To decompose an input image into several subband images, one level DWT will be used in image processing. The higher frequency components of the input image will be divided into three high frequency subbands , named as LH, HL, HH.
This paper provides literature review about various techniques used to achieve super resolution image with the help of single image or multiple low resolution images. In this paper, interpolation based, reconstruction based and learning based techniques for super resolution are studied.
The future scope of the super resolution is to develop new methods by extending or integrating the existing methods to address their challenges. Finally images with super resolution can provide more details as compared to low resolution images with low cost.
M. S. Alam, J. G. Bognar, R. C. Hardie, and B. J. Yasuda. Infrared im-age registration and high-resolution reconstruction using multiple translationally shifted aliased video frames. IEEE Transactions on Instrumentation and Measurement, 49(5):915-923, 2000.
S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167-1183, 2002.
O. Bowen and C. S. Bouganis. Real-time image super resolution using an FPGA. In International Conference on Field Programmable Logic and Applications, pages 89-94, 2008.
A. Buades, B. Coll, and J. M. More. A non-local algorithm for image denoising. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 60-65, 2005.
J. Chung, E. Haber, and J. Nagy. Numerical methods for coupled super-resolution. Inverse Problems, 22(4):1261-1272, 2006.
D. Datsenko and M. Elad. Example-based single document image super-resolution: a global MAP approach with outlier rejection. Multidimen-sional System and Signal Processing, 18(2-3):103-121, 2007.
D. L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289-1306, 2006.
D. L. Donoho. For most large underdetermined systems of linear equations, the minimal `1-norm near-solution approximates the spars-est near-solution. Communications on Pure and Applied Mathematics, 59(7):907-934, 2006.
X. Jia F. Li and D. Fraser. Universal HMT based super resolution for remote sensing images. In IEEE International Conference on Image Processing, pages 333-336, 2008.
S. Farsiu, D. Robinson, M. Elad, and P. Milanfar. Fast and robust multi-frame super-resolution. IEEE Transaction on Image Processing, 13(10):1327-1344, 2004.
P. H. Hennings-Yeomans, S. Baker, and B. V. K. V. Kumar. Simul-taneous super-resolution and feature extraction for recognition of low-resolution faces. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1-8, 2008.
K. Jia and S. Gong. Generalized face super-resolution. IEEE Transactions on Image Processing, 17(6):873-886, 2008.
A. Levin, Y. Weiss, F. Durand, and W. Freeman. Understanding and evaluating blind deconvolution algorithms. In Proceedings of IEEE Com-puter Society Conference on Computer Vision and Pattern Recognition, pages 1964-1971, 2009.
Frank Lin, Clinton B. Fookes, VinodChandran, and SridhaSridharan. Investigation into optical low super-resolution for surveillance applica-tions. In The Austrilian Pattern Recognition Society Worshop on Digital Image Computing, 2005.
K. Malczewski and R. Stasinski. Toeplitz-based iterative image fusion scheme for MRI. In IEEE International Conference on Image Processing, pages 341-344, 2008.
B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao. A computa-tionally efficient super-resolution algorithm for video processing using partition filters. IEEE Transactions on Circuits and Systems for Video Technology, 17(5):621-634, 2007.
N. Nguyen and P. Milanfar. An efficient wavelet-based algorithm for image super-resolution. In Proceedings of International Conference on Image Processing, volume 2, pages 351-354, 2000.
N. Nguyen, P. Milanfar, and G. H. Golub. A computationally e±cient image superresolution algorithm. IEEE Transactions on Image Process-ing, 10(5):573-583, 2001.
M. Protter and M. Elad. Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 18(8):1899-1904, 2009.
P. Hardeep, B. Prashant, Swadas, M. Joshi, A Survey on Techniques and Challenges in Image Super Resolution Reconstruction. International Journal of Computer Science and Mobile Computing, 2013.
T. sai and Huang. A survey on super-resolution imaging. Springer, 2010.
R. SudheerBabu, K. E. S. Murthy. A Survey on the Methods of Super-Resolution Image Reconstruction. International Journal of Computer Applications. 15(2), 2011.
A. J. Shah, B. S. Gupta. Image Super Resolution – A Survey. IEEE, 2012.