Assessment of Multi-Temporal Image Fusion for Remote Sensing Application

Image fusion and subsequent scene analysis are important for studying Earth surface conditions from remotely sensed imagery. The fusion of the same scene using satellite data taken with different sensors or acquisition times is known as multi-sensor or multi-temporal fusion, respectively. The purpose of this study is to investigate the effects of misalignments the multi-sensor, multitemporal fusion process when a pan-sharpened scene is produced from low spatial resolution multispectral (MS) images and a high spatial resolution panchromatic (PAN) image. It is found that the component substitution (CS) fusion method provides better performance than the multi-resolution analysis (MRA) scheme. Quantitative analysis shows that the CS-based method gives a better result in terms of spatial quality (sharpness), whereas the MRA-based method yields better spectral quality, i.e., better color fidelity to the original MS images. Keywords—multi-sensor; multi-temporal fusion; component substitution; multi-resolution analysis


I. INTRODUCTION
Following the rapid advancements of new and greatly improved remote sensing (RS) sensor systems, various kinds of remote sensing data were acquired and applied in many interdisciplinary Earth observational applications. Remote sensing images are characterized by their spectral, spatial, radiometric, and temporal resolutions. Spectral resolution refers to the bandwidth and the sampling rate over which the sensor gathers information about the scene. A low spatial resolution multispectral (MS) and high spatial resolution panchromatic (PAN) imaging sensors are the systems usually used for earth change observation, each one having its own specific advantage. Most of the operating earth observation very high resolution (VHR) imagery (WorldView, QuicBird, GeoEye, and Orbview, etc.), it was very useful issues various RS problems such as image sharpening, land classification, change detection, and object identification, visualization purposes, etc. [1]. Besides that, PAN sensors provide an image in the visible band, which is characterized by high spatial information content well suited for intermediate scale mapping applications and urban analysis [2].
Nowadays, image fusion is also known as pan-sharpening, resolution merge, image integration, or multi-sensor data fusion. It is a useful tool in integrating a high-resolution PAN image with a low-resolution MS image to produce a highresolution multispectral image and a better understanding of the observed earth surface. In image fusion observed scene analysis and RS application, the observed scene information fusion can be available in the following cases [3]; data recorded by different sensors (multi-sensor image fusion), data recorded by the same sensor scanning the same scene at different dates (multi-temporal image fusion), data recorded by the same sensor operating in different spectral bands (multi-frequency image fusion), data recorded by the same sensor at different polarizations (multi-polarization image fusion), and data recorded by the same sensor located on platforms flying at different heights (multi-resolution image fusion). A Multi-sensor, multi-temporal, multi-resolution and multi-parameter image data from operational Earth observation satellites are available and therefore possibly give a complete view of observed objects [4] Recently, some researcher has addressed multi-temporal data fusion problem. Less publication focus on the multi-temporal image fusion and application has reported [3]- [9]. The exploitation of the multi-temporal information fusion is fundamental for change detection purpose but can be also valuable in image classification to improve the classification accuracy. And also study of fusing multiple images provided by heterogeneous image sensor has been proposed in many literatures with different methods, context, and purposes [10].
One of the major problems and limitations of existing image fusion techniques is the degradation in spectral quality (color distortion), which causes visual changes as well as spectral differences between the original and fused images. Another common problem is dependent of fusing quality on different wavelength sensitivity, and/or the preference of the operator, since the different operator or data type may lead to different fusion qualities. The effectiveness of different fusion techniques is examined to reduce color distortion in this study. Concerning various methods developed for image fusion for remote sensing, it is desirable to give a general assessment of the quality of the fused images from the viewpoint of practical use. Since the qualitative (visual) evaluation may contain subjective factor, possibly influence by personal preference, quantitative approaches are often required to prove the correctness of the visual evaluation In this paper, we use fusion algorithm to assess the capability of fusion using multi-resolution analysis (MRA) and component substitution (CS) algorithm to observe of fused image for better quality in object identification. Also, we apply the following statistical approaches for standardizing and automating the evaluation process of the fused images. First, a qualitative assessment is made with the conventional visual inspection. Then, we examine the spectral quality between the original MS and the fused images.

II. MATERIALS AND METHODS
The study site for this work is located in over the downtown of San Francisco, California (US) with geographical coordinates 122°23'1.08"W, 37°42'38.81"N. San Francisco is located on the West Coast of the United States at the tip of the San Francisco Peninsula and includes significant stretches of the Pacific Ocean and San Francisco Bay within its boundaries (Fig. 2).  Table 1. Significant spectral distortion in the fusion product image can occur due mainly to the wavelength extension of the new satellite PAN sensors. A major reason for the significant spectral distortion in image fusion is the wavelength extension of the new satellite PAN sensors. In image fusion techniques, it is important to include sensor spectral response information for the following reasons [11]: first, in order to preserve physical meaning of merged spectral bands, the sensor spectral response for each band has to be taken into account and second, image fusion techniques try to recover the image obtained by an ideal virtual sensor with the same spectral sensitivity of the MS and spatial resolution of the PAN sensor when during the fusion process. The spectral band's response curves of these two sensors are summarized in and Fig. 3. Table 2 shows the wavelength range and spatial resolution of different PAN sensors. A task for analyzing the image are divided into (1) Image correction and registration: The most important prerequisite for accurate data fusion is a precise geometric correction. In the image, fusion needs commons control point on both the input images since different images of the same area used together. The common geometric correction is an image to image registration. Registration can be done by various methods. One of the methods is an image to image registration, as seen in Fig. 4.
An image to image registration is translation and rotation alignment process by which two images of like geometry on the same geographic area [12]. In registration processing, the most accurate way is to register the input images separately by establishing a geometric relationship between the image and the ground using rigorous photogrammetric methods [13]. In this study, the first order polynomial transformation method is used for registration refinement of multispectral images by taking the WV multispectral image as the reference. A common ground control points (GCPs) on both images are selected carefully such that they produce an RMS error smaller than 0.5 pixels with respect to the pixel size of the fused WV image. The accuracy of image registration is usually conveyed in terms of root-mean-square (RMS) error, (2) Re-sampling: Next, we apply re-sampling, in which each data point (pixel) in the high-resolution base map is assigned a value based on the MS image pixels. In order to achieve a good fusion result, low spatial, spectral images should be resampled. At present, nearest neighbour resampling is often adopted which has some effects on the precision of new image.  (Fig. 3). The pixel size of WV PAN (0.5 m) is greater than that for QB PAN (0.6 m). Thus, in order to minimize the spectral difference, WV MS, QB MS and QB PAN imaging are used, after being re-sampled at 0.5 m. To analyse the effect of different spatial resolution ratio images, re-sampling of the two images was done next as different resolution ratio to create a various set of images for fusion. The various resolution obtained due to such a re-sampling techniques are shown in Table 3 when the effective set of input image was generated for fusion using CS and MRA techniques, (3) Spatial resolution ratio: The spatial resolution ratio improvement can make a sharper image while preserving the spectral information contained in the fused images. Since PAN images usually have a better spatial resolution than the multispectral images of the same sensor, while the MS images provide spectral properties of the objects. In image fusion, spatial resolution ratio plays an important role in image fusion. The spatial resolution ratio can be defined as the ratio between the spatial resolution of the high-resolution PAN image and that of the lowresolution MS image [23]. The both of spatial resolution ratio of PAN to MS image was given in Table 3, (4) Band selection processing by the OIF: Band selection is a key step of fusion techniques. For this purpose, values of optimum index factor (OIF) are useful for designating the most favourable band combination according to their information [14]. Selecting of three bands for a color composite RGB (Red-Green-Blue) image is a tedious and time-consuming business since many combinations have to be tried when working with ratio images. The OIF technique simplifies that selection by the quantitative evolution of the scene statistics and avoids the time consuming of the visual analysis process of large numbers of potential R-G-B combinations. Since usually only three bands can be used for color composite (R-G-B), the spectral bands should be selected so as to project the MS information as efficiently as possible onto the final color image by calculating the OIF. Generally, a larger standard deviation of an image infers that it involves more information. Thus, the OIF is defined [15] as where σ i is the standard deviation of each of the three selected bands and r j is the correlation coefficients (CCs) between any pair formed by these bands (Table 4). From the original WV and QB, a total of 56 and 4 bands color combinations are produced and analyzed using the optimum index factor (OIF). The highest value of average OIF has been obtained for the band combination 3-5-7 and 2-3-4, both for WV and QB. Multi-resolution analysis (MRA) is an approach based on Fast Fourier Transform (FFT)-enhanced intensity-huesaturation (IHS) transformation. It has been developed specifically for image merging that preserves spectral characteristics [6], [16]. The procedure of this method can be expressed as follows [16]- [17] (see Fig. 5). The re-sampled multispectral images are transformed from the RGB to IHS color space to obtain the intensity (I), hue (H), and saturation (S) components, and low-pass filtering (LP) is applied to the intensity component. A Component substitution (CS) make use of a filtering process to extract the high-frequency details from the PAN image fall in the category of component substitution methods. The principle idea is to add the details of the PAN image into the MS images making use of some transformation. The spatial resolution of the MS image is enhanced by merging the high-resolution PAN image with the low spatial resolution MS bands. The main steps of the methods are the following: • A GS transformation is performed by employing the simulated PAN as the first band. • After swapping the first GS band with the highresolution PAN image • The inverse GS transform is applied to form the pansharpened spectral bands.  In the CS methods, so far many researchers have addressed the problem of multi-resolution and multitemporal image fusion for RS applications, proposing different methods such as Gram-Schmidt (GS), Intensity-Hue-Saturation (IHS) based fusion methods, Brovey transform based fusion and PCA based fusion.
The quality of fused image was assessed by means of some quality measures that can predict perceived image quality automatically. In this work, two types of quality assessment approaches are taken, namely qualitative (visual analysis) and quantitative methods. The visual analysis is based on the visual comparison of the color between the original MS and the fused image and that of the spatial details between the original PI and the fused image. These methods inevitably involve subjective factors and personal preference that can influence the result of the evaluation. While visual inspection has limitation due to human judgment, a quantitative approach based on the evaluation of "distortion" that is brought about by the noise added to the fused image is more desirable for mathematical modeling [18]. In this work, a validation method is proposed based on a quality criterion, namely, the relative average spectral error (RASE) [19]- [20] and relative dimensionless global error (ERGAS) [21]- [22] parameter. It is based on the root mean squared error (RMSE) [6] to estimation and chosen as a robustness criterion. This statistical parameter is often used for evaluation of fusion techniques. These parameters are defined as follows: • Relative Average Spectral Error (RASE) is used to estimate the global spectral quality of the fused images Where M is the mean radiance of the n spectral bands (Bi) of the original MS bands; RMSE is the root mean square error computed as • Relative Dimensionless Global Error in Synthesis (ERGAS) was proposed by Wald as a multi-modal index to characterize the quality of the process in terms of the normalized average error of each band of processed image. Increasing in ERGAS index may be the result of degradation in images due to the fusion process. ERGAS index for the fusion is expressed as follow 2 / 1 1 2 These formulae can be used for comparing errors obtained from different methods, different cases and different sensors. Where dh/dl is the ratio between the pixel sizes of the PAN and MS images (e.g., 1/4 for QB and WV data), and μ(i) is the mean of the i th band. Since ERGAS is a measure of distortion, its value must be as small as possible.
For analysed information from multi-temporal assessment, the following combinations are employed here:   In the visual (quantitative) analysis, it is seen that CS fusion yields relatively sharp images for both PAN  Other results show somewhat blurred results due to temporal changes. For MRA fusion, most of the color features for all combination has shown the preserve color with the original image. The visual analysis reveals that the quality both of WV-PAN,WV-MS, and WV-PAN, QB-MS are very good and quite comparable. However, the CS-based method gives a better result in terms of spatial quality (sharpness), whereas the MRA-based method yields better spectral quality, i.e., better color fidelity to the original MS images. Table 5

IV. CONCLUSIONS
We have investigated the assessment of multi-temporal fusion by multi-resolution analysis (MRA) and component substitution (CS) algorithms. In both quantitative and qualitative results, it has been found that the CS based method leads to better spatial quality (sharpness), whereas the MRA based method better spectral quality (fidelity to the original color). In the future research, the methodology presented in this paper can be extended to include the multitemporal fusion of optical and synthetic aperture radar (SAR) images from satellite remote sensing.