International Journal of Innovative Research in Computer and Communication Engineering
MenuISSN ONLINE(2320-9801) PRINT (2320-9798)
ISSN ONLINE(2320-9801) PRINT (2320-9798)
Khushbu A. Panchal1, Dr.Mehfuza Holia 2
|
| Related article at Pubmed, Scholar Google |
Visit for more related articles at International Journal of Innovative Research in Computer and Communication Engineering
The goal of video completion is to reconstruct the missing part or holes which are created by damage or removal of any selected objects from video. The key issue in video completion is to keep the spatial temporal information. A lot of researchers have worked in the area of video inpainting. Most of the researchers try to maintain either spatial consistency or temporal continuity between the frames. But none of them try to maintain both of them in the same technique with a good quality. In The main goal of inpainting is removed unwanted object from video and fill those region with the background. Inpainting is used for restoration of older films, object removal in digital photographs and for privacy protection. It is also useful to red-eye correction, compression etc. Many approaches of inpainting have been proposed. This proposed work presents a brief survey of different inpainting algorithms. In this paper provide an analysis of different techniques used for Inpainting.
Keywords |
| Image Inpainting, Texture Synthesis, Partial Differential Equation (PDE), Patch, Occlusion. |
INTRODUCTION |
| Video Inpainting deals with the field of image processing that aims to remove objects or restore missing or tainted regions present in a video sequence by utilizing spatial and temporal information from neighboring scenes [2]. The main objective is to generate an inpainted area that is merged seamlessly into the video so that visual coherence is maintained throughout and no distortion in the affected area is observable to the human eye when the video is played as a sequence. Video inpainting is to be considered as a most challenging problem due to spatial and temporal constraint, camera and object motion, color cues, occlusion detection and lighting variation of video. Main applications of video inpainting [2] are in video modification for privacy protection, video stabilization, and video error concealment in video transmission, multimedia editing and visualization. |
| Image Inpainting verses Video inpainting: It is important to distinguish video and image inpainting. The main difference between these approaches is in video inpainting, the region can be much larger with the main focus being the filling in of two dimensional repeating patterns that have some associated textures. In contrast, image inpainting focus on the removal and completion of unwanted or damaged regions in single images using surrounded information. It is not possible to extend methods put forward for image inpainting to the field of video inpainting, because in image inpainting techniques only consider spatial information, and completely neglect the significant temporal information present in video sequences. Many researchers have worked in the area of video inpainting. All available techniques try to ensure either spatial consistency or temporal continuity between the frames. But none of them try to ensure both of them in the same technique with a good quality. |
II. RELATED WORK |
| In the last decade many video Inpainting techniques have been developed for large variety of applications. Partial Differential Equation (PDE) based technique works in iterative manner. According to this algorithm lines which are arriving at the border of the region should be smoothly inpainted, from the outside of the border to the inner region. It Produce good results if missed regions are small one and target region is non-textured, but take long time if target region is large. Some blurring effect is presented in the resultant video sequence. Texture Synthesis based inpainting algorithm is used to fill the damaged or missed regions using similar neighborhoods of the missed pixels. From an initial seed new image should be synthesized. This algorithm is mainly used to inpaint small region and can-not handle natural scenes effectively. |
| The exemplar based approach is an important class of inpainting algorithms. Basically it consists of two basic steps: in the first step priority is given to all the patches and in the second step best matching patch is selected. The main advantage to use this algorithm is that it handles large holes effectively. But only used for simple texture and structure region. Enough samples are required to synthesis the target region. Hybrid inpainting is combination of both texture synthesis and PDE based Inpainting. This technique handle large holes and preserves both structure and texture effectively, but computation time of this algorithm is more for large holes. Semi-automatic image inpainting requires user assistance. This technique follows a two-step process. In the first step a user manually specifies important missing information in the hole by sketching object boundaries from the known to the unknown region and then a patch based texture synthesis is used to generate the texture [1]. |
III. INPAINTING ALGORITHMS |
| There are several approaches for video inpainting which can be categorized as follows: |
| 1. PDE based Inpainting. |
| 2. Texture Synthesis based Inpainting. |
| 3. Exemplar based Inpainting. |
| 4. Semi-automatic and Fast Inpainting |
| 5. Hybrid Inpainting. |
| A. PDE Based Inpainting |
| In the last decade many video Inpainting techniques have been developed for large variety of applications. Partial Differential Equation (PDE) based technique works in iterative manner. According to this algorithm lines which are arriving at the border of the region should be smoothly inpainted, from the outside of the border to the inner region. It Produce good results if missed regions are small one and target region is non-textured, but take long time if target region is large. Some blurring effect is presented in the resultant video sequence. |
| M. Bertalm´?o, A. L. Bertozzi, G. Sapiro [3] proposed fluid dynamic based inpainting. The algorithm automatically transports information into the inpainting region. The inpainting is done in such a way that lines arriving at the regions boundaries are completed inside. The only user interaction required by the algorithm is to mark the regions to be inpainted. At the very practical level, this algorithm not dealt with textures, and the parameters are set manually. These algorithms were focused on maintaining the structure of the Inpainting area. And hence these algorithms produce blurred resulting image. Another disadvantage of these algorithms is that the large textured regions are not well reproduced. |
| B. Texture Synthesis based Inpainting |
| Texture Synthesis based inpainting algorithm is used to fill the damaged or missed regions using similar neighborhoods of the missed pixels. From an initial seed new image should be synthesized. This algorithm is mainly used to inpaint small region and can-not handle natural scenes effectively. Wei-Qi Yan, Mohan S. Kankanhalli [5] proposed algorithm in which every circle is marked with an alphabetic character indicating a pixel in the video frame. Suppose only one pixel needs to be filled, the average color of the surrounding pixels is the new filled color. If there are many concentric rings, all pixels are filled by the average color starting from the outermost ring to the innermost ring. The remaining unfilled pixels, if any, are found by raster-scanning and are filled by the average color of the neighboring unmarked R pixels. In next step, the region is completely filled. This algorithm provides simplicity but cannot handle logos with a large area. Chunbo Zhu, Xiaoyan Sun, Feng Wu, and HouqiangLi[12] proposed texture synthesis and edge-based inpainting. This paper proposes a video coding scheme, in which textural and structural regions are selectively removed in the encoder, and restored in the decoder by spatio-temporal texture synthesis and edge-based inpainting. This approach has been implemented into conventional video coding scheme such as H.264 and has shown remarkable bit rate saving and similar visual quality compared with H.264. |
| C. Exemplar based Inpainting |
| The exemplar based approach is an important class of inpainting algorithms. Basically it consists of two basic steps: in the first step priority is given to all the patches and in the second step best matching patch is selected. The main advantage to use this algorithm is that it handles large holes effectively. But only used for simple texture and structure region. Enough samples are required to synthesis the target region. Generally, an exemplar-based Inpainting algorithm includes [1] the following four main steps: |
| 1. Initializing the Target Region, in which the initial missing areas are extracted and represented with appropriate data structures. |
| 2. Computing Filling Priorities, in this a predefined priority function is used to compute the filling order for all unfilled pixels p∈δΩ in the beginning of each filling iteration. |
| 3. Searching Example and Compositing, in which the most similar example is searched from the source region Φ to compose the given patch, Ψ (of size N × N pixels) that centered on the given pixel p. |
| 4. Updating Image Information, in which the boundary δΩ of the target region Ω and the required information for computing filling priorities are updated. |
| Yuki Umeda and Kaoru Arakawa [17] proposed exemplar based inpainting with directional median filter. Exemplar based method restores the scratch area with the image pixels in a similar undamaged image area around the location of the scratch in both the spatial and temporal domain. A similar image area in the spatial and temporal domain is searched around the location of scratches by a block matching algorithm. Such inpainting method is effective, if similar image areas exist around the scratch area and if the similarity is high enough. When a scratch is detected, this method first applies the spatio-temporal exemplar-based inpainting. Then, if the matching error of the most similar area is small enough, the result of the exemplar-based inpainting is adopted to restore the scratch area. Otherwise, directional median filtering is applied. Video inpainting of occluding and occluded object is proposed Kedar A. Patwardhan,Guillermo Sapiro and Marcelo Bertalmio [14]. Two important cases are considered. The first case is concerned with the removal of non-stationary objects that occlude stationary background. They use a priority based spatio-temporal synthesis scheme for inpainting the stationary background. The next case involves filling-in moving objects when they are partially occluded. For this, they propose a priority scheme to first inpaint the occluded moving objects and then fill-in the remaining area with stationary background using the method proposed for the first case. N.Neelima, M.Arulvan [16] proposed Object Removal by Region Based Filling Inpainting. The actual color values are computed using exemplar-based synthesis. Block based sampling process is used to achieve computational efficiency. The technique can deal with propagating both linear structure and two-dimensional texture into the target region. |
| Aijuan Xia, Yan Gui, Li Yao, Lizhuang Ma, Xiao Lin [22] have presented Exemplar-Based Object Removal in video using GMM. This paper presents an exemplar-based video inpainting mechanism that restores the area of the removal object, and this mechanism can be further employed to extract the background of videos. The region to be inpainted in video is still in background and moving in foreground. This method consists of a simple preprocessing stage and video inpainting step. The preprocessing stage consists in constructing Gaussian Mixture Model (GMM) for both background and foreground separately, then make use of GMMs to distinguish background and foreground of the entire video. That saves the time for calculating the optical flow mosaics as many video inpainting algorithms do in the preprocessing step. As for video inpainting, firstly fill the gap as much as possible by copying information from other frames pixel by pixel, and then inpaint the remaining holes in the background by extending the exemplar-based image inpainting algorithm. Experimental results demonstrate that this method for object removal in video is feasible and effective. |
| Mounira Ebdelli, Olivier Le Meur, Christine Guillemot [23] proposed Loss concealment based on video inpainting for robust video communication. This paper presents an error concealment algorithm combining the advantages of motion-compensated spatio-temporal interpolation and examplar-based video inpainting. The algorithm first estimates the motion information of erroneous/lost blocks in a video sequence using a state-of-the-art method called BMFI (Bilinear Motion Field Interpolation). Then, the video inpainting algorithm estimate the texture of each lost blocks as a linear combination of the most similar blocks in a motion-compensated window. Experiments on several videos show more accurate and visually pleasing results. In terms of PSNR (Peak of Signal-to-Noise Ratio), the average gain is about 2dB compared to state-of-art methods. |
| D. Hybrid inpainting |
| Hybrid inpainting is combination of both texture synthesis and PDE based Inpainting. This technique handle large holes and preserves both structure and texture effectively, but computation time of this algorithm is more for large holes. This technique uses a two-step approach: the first stage is structure completion followed by texture synthesis. In the structure completion stage, segmentation is performed based on the insouciant geometry, color and texture information on the input and then the partitioning boundaries are extrapolated to generate a complete segmentation for the input using tensor voting. Next step consists of synthesizing texture and color information in each segment, again using tensor voting [1]. K. Sangeetha, Dr. P. Sengottuvelan, E. Balamurugan [15] proposed combined structure and texture image inpainting algorithm for natural scene image completion. They present a hybrid method for completion of images of natural scenery, where the removal of a foreground object creates a hole in the image. The basic idea is to decompose the original image into a structure and a texture image. Reconstruction of each image is performed separately. The missing information in the structure component is reconstructed using a structure inpainting algorithm, while the texture component is repaired by an improved exemplar based texture synthesis techniques. Taking advantage of both the structure inpainting methods and texture synthesis techniques, they designed an effective image reconstruction method. A comparison with some existing methods on different natural images shows the merits of this approach in providing high quality inpainted images. |
| E. Semi-automatic and Fast Inpainting |
| Semi-automatic image inpainting requires user assistance. This technique follows a two-step process. In the first step a user manually specifies important missing information in the hole by sketching object boundaries from the known to the unknown region and then a patch based texture synthesis is used to generate the texture [1]. If only a single curve is present, simple dynamic programming can be used to derive the optimal answer. For multiple objects, the optimization is great deal more difficult and proposes approximated the answer by using belief propagation. |
| F. Other Tequeniques |
| Tao Ding,Mario Sznaier, Octavia Camps [11] presented rank minimization approach of video inpainting. This method is consists of following steps [11]: (i) find a set of descriptors that encapsulate the information necessary to reconstruct a frame, (ii) find an optimal estimate of the value of these descriptors for the missing/corrupted frames, and (iii) use the estimated values to reconstruct the frames. A generative definition of descriptors is given by introducing Local Linear Embedding (LLE) to map image space to a low dimensional appearance representation space, proposed as our descriptor space. Correspondingly, inverse mapping is accomplished by Radial Basis Function (RBF). Obtained result shows that the optimal descriptor estimates can be efficiently obtained by minimizing the rank of a matrix directly constructed from the available data, called as Rank Minimization Interpolation (RMI), leading to a simple, computationally attractive, dynamic inpainting algorithm that optimizes the use of spatio/temporal information. This method can handle non–periodic target motions in the presence of severe occlusion, non-stationary backgrounds, and both camera rotations and translations. |
| Timothy K. Shih and Rong-Chi Chang [6] cover a new approach, which divides a Chinese painting into several layers. Each layer is inpainted separately. A layer fusion mechanism then finds the optimal inpaint among layers, which are restored layer-by layer. They apply the algorithm on Chinese and western drawing. The result shows a high PSNR value as well as a high user satisfaction. The exemplar-based synthesis approach used both textural and structural information and recovers user selected area which covers 19% of the image. The multi-resolution and multi-layer inpainting method purpose is to develop an effective algorithm to restore damaged areas on still image, with high PSNR values |
| Amanna Ghanbari, Mohsen Soryani [24] proposed Contour-Based Video Inpainting. This paper proposes an algorithm for video inpainting when an object is totally damaged. In this framework the background and the moving object are separated from each other. By using these separated moving objects, a large mosaic image is constructed. Then a patch-based method with the help of a contour-based method and large patches fills the holes. By settling the objects on their places in each frame, the inpainted foreground is acquired. Missing regions of the stationary background are filled separately. Superimposing the inpainted foreground and the stationary background produces the inpainted video. |
| IV. CONCLUSION |
| In this paper a variety of Inpainting techniques such as texture synthesis based Inpainting, PDE based Inpainting, Exemplar based Inpainting, Hybrid Inpainting and semi-automatic and fast Inpainting techniques are studied. For each technique a detailed explanation of the techniques can be given which are used for filling the missing region building use of image. From this survey, a number of shortcomings and limitations were highlighted in each and every technique. It is experiential that the PDE based Inpainting algorithms cannot fill the large missing region and it cannot renovate the texture pattern. The analysis proved that the exemplar based image Inpainting will create better results for Inpainting the huge missing region also that these algorithms can inpaint both the formation and textured image efficiently. But it will work well only if the missing region contains only simple structure and texture. The Contourlet transform is a recent developed technique which can be more accurate and it overcomes all the problem of the other existing techniques. Advance study includes growth of efficient algorithm to reduce computational cost and to decrease the time required for Inpainting |
ACKNOWLEDGMENT |
| I am very grateful and would like to thank my guide and teacher Dr. Mehfuza Holia for her advice and continued support without her it would not have been possible for me to complete this paper. I would like to thank all my friends, colleague and classmates for all the thoughtful and mind stimulating discussions we had, which prompted us to think beyond the obvious. |
References |
|