Colour Fusion in Face Authentication System Based on Visible and Near Infrared Images

— In this paper, face authentication using images taken in visible and near-infrared spectra (NIR) is studied. Visible images are in RGB colour space and near-infrared images are in gray levels colour space. First, the performance of system in each of the primary colour spaces of visible and near-infrared spectrum is evaluated that the verification process is based on the Normalised Correlation measure within the LDA feature space. In order to utilize the information of colour images, the scores associated to an adaptively selected subset of the colour based classifiers are then fused in the decision level. The selection process is based on a sequential search technique called the”plus L and take away R” algorithm. The sum rule and svm rule is used for fusing the related scores. Our extensive experimental studies using the HFB face database demonstrate that using the proposed method, the performance of the system considerably improves as compared to the individual Visible-based or NIR-based face verification systems.

Face verification is among the most interesting research areas in the recent years. A worthful improvement has been made in this research topic over the last decade [1]. Although a diversity of approaches and tools has been proposed for face verification, the authentication systems usually have a desirable performance in controlled environments. Their performance often is highly degraded in uncontrolled environments. Several factors are affecting the performance of the face verification systems including variations in face position, facial expression, hairstyle, face occlusion and environmental lighting. Among them, the problem with environmental lighting is perhaps one of the most important issues which have to be solved [2]. Most current face verification systems are based on face images taken in the visible spectrum (0.3-0.7µm). Therefore, the system is affected by the illumination changes, even for Cooperative user applications indoors.
Many different methods have been proposed for solving the illumination variations problem. An important approach is face verification using different imaging modalities, especially infrared (IR) imaging sensors. So near-infrared images (0.7-0.9 µm) and thermal-infrared images(2.4-14 µm) in face verification system have been widely used.
Imaging in near-infrared spectrum, has many advantages compared to visible and thermal infrared spectra. Unlike the thermal infrared spectrum, some of radiations in nearinfrared spectrum are reflected by object, so near-infrared lighting sources can be used.
One of the most important benefits of near-infrared lighting, being invisible so environmental visible light is eliminated in this spectrum, Also near-infrared radiations easily passed through the glasses, but the thermal radiations are nearly returned by glasses so information around the eyes can be lost [3].
Advances in face recognition technology by fusing the visible and infrared spectra have been reported in [4]. In this paper, it is shown that fusing the information arises from the visible and thermal IR images can significantly improve the performance of a face verification system in uncontrolled illumination environments. In fact, visible and infrared sensors catch the complementary information of reflectance and radiation from a face.
Recently, in a number of studies, it has been demonstrated that colour information can improve the performance of the face recognition and verification systems. A brief history of different methods of involving colour features in face verification systems can be found in [5] where a systematic evaluation of signal, feature and decision level fusion of data derived from a multi-colour channel face image has been carried out. The authors focused on face verification using the Normalised Correlation and Gradient Direction metrics in Linear Discriminate Analysis (LDA) spaces associated with the respective R, G, B colour channels. The associated results demonstrated that the most beneficial fusion methods are the decision level and feature level fusion but the decision level fusion was computationally the simplest. In [6], considering the RGB colour space as the primary space, 15 different colour spaces suggested in the literature have been adopted, giving rise in total to 44 different colour channels. Then, in a decision level fusion framework, the colour expert fusion problem has been formulated as a feature selection problem. Different trained and un-trained fusion rules have been used for combining the selected classifiers. Very good results have been reported using the proposed methods.
Considering the above mentioned argument the main idea behind the current study is fusing the visible and NIR information. In proposed face verification system, the visible and NIR colour spaces are selected and after that the selected scores are fused. The best selected colour space(s) depending on application. The colour space(s) are selected using a sequential search approach similar to the"Plus L and Take away R" algorithm. It is demonstrated that, surprisingly good results are obtained using the proposed method.
The paper is organised as follows: In the next section visible and NIR imaging are briefly reviewed. In Section 3, different colour spaces adopted in different machine vision applications are reviewed. The face verification process is discussed in Section 4. The proposed method of colour space selection and fusion are described in Section 5. The experimental set up is detailed in Section 6. Section 7 presents the results of the experiments. Finally, in Section 8 the paper is drawn to conclusion.

A. Imaging in Visible Spectrum
According to Figure 1, the radiation of visible light (0.3-0.7 µm) which reflected from the object, are collected by visible camera and the image is made. Therefore, this type of imaging is based on reflected light from the object, so images are sensitive to environmental lighting conditions. As previously mentioned, it is the main weakness of imaging in this spectrum.

B. Imaging in Near-Infrared Spectrum
Reflected infrared radiation (0.7-0.9 µm) is not visible for human eye, but by special CCD and CMOS sensors can be measured. Camera that captures the near-infrared images, radiates non-visible near-infrared light (NIR) and collects reflected light from the object and images are made based on this reflected light so the environmental lighting is eliminated in this spectrum. For further explanation can be referred to [7].

III. COLOUR SPACES
On computers, it is more common to describe colour as a mixture of three primary colours: Red, Green and Blue. However, it has been demonstrated that in different applications using different colour spaces could be beneficial and complementary information exist in different subspaces.
We can classify the other colour spaces into two main categories: Linear and Nonlinear transformation of the R, G, and B values. Gray level space (I) is one of linear colour space and it is depended to R, G, and B as below: More details about some of the most important colour spaces can be found in [8].

IV. FACE VERIFICATION PROCESS
The face verification process consists of three main stages: face image acquisition, feature extraction, and finally decision making. The first stage involves sensing and image preprocessing the result of which is a geometrically registered and photometrically normalised face image. In this study the raw cameras channel outputs for both NIR and visible were used.
In the second stage of the face verification process the face image data is projected into a feature space. The final stage of the face verification process involves matching and decision making. Basically, the features extracted for a face image to be verified, x, are compared with a stored template, that was acquired on enrolment, µ i . In this study we adopted the Normalised Correlation (NC) measure in the Linear Discriminate Analysis (LDA) feature space for decision making [5]. The score, s, output by the matching process is then compared to a threshold in order to decide whether the claim is genuine or impostor. If this final stage of processing is applied to different colour spaces separately, we end up with a number of scores, s k = s(x k ), k = 1, 2, . . . ,N which then have to be fused to obtain the final decision. The adopted fusion method is studied in the next section.

V. COLOUR SPACE SELECTION
One of the most exciting research directions in the field of pattern recognition and computer vision is classifier fusion. Multiple expert fusions aim to make use of many different designs to improve the classification performance. The approach we adopted for selecting the best colour space(s) is similar in principal to the sequential feature selection methods in pattern recognition [9]. In this study, the"plus L and take away R" algorithm has been applied for selecting an optimum subset of the colour spaces. The sum rule and suport vector machine (SVM) also has been used in order to combine the scores of the selected colour based classifiers.
In the sum rule, the average of scores is calculated and used for evaluating system error. In fusion method using support vector machines, first support vector machines is trained in training stage and then in the test stage, by using the test dataset, the performance of system by using these trained vector machines are determined. For further explanation can be referred to [10].

VI. EXPERIMENTAL DESIGN
The aim of the experiments is to show that by fusing the NIR and visible sensory data used by component Experts, the performance of the multiple classifier system considerably improves.
We use the HFB face database [11] for this purpose. The HFB database contains 400 pairs of visible and NIR face images of 100 individuals, with 4 images per person from Chinese ethnic groups.
The visible images are in RGB and the NIR images are in Gray level colour space. These images are taken under normal lighting condition and People who wear eyeglasses are asked to take pictures with and without eyeglasses. Size of these images is originally 640×480.   The training set is used to construct client models and the test set is selected to simulate realistic authentication tests where impostor's identity is unknown to the system. In this study, by using the evaluation set the threshold has been determined based on the Equal Error Rate criterion, i.e. by the operating point where the false rejection rate (FRR) is equal to the false acceptance rate (FAR) in evaluation stage.
False acceptance is the case where an impostor, claiming the identity of a client, is accepted. False rejection is the case where a client, claiming his true identity, is rejected. The performance measures of a verification system are the False Acceptance Rate and the False Rejection Rate.
For reducing computation costs the experiments were performed with a relatively low resolution face images, namely 64×64. The results reported in this article have been obtained by applying a geometric face registration based on manually annotated eyes positions. Histogram equalisation was used to normalise the registered face photometrically. Exemplary Face images which used are shown in

VII. EXPERIMENTAL RESULTS
First, the performance of the verification system considering the each subspaces of visible images as well the Gray level space for both visible and NIR images were evaluated. Table 2 contains the relevant results. These results can be considered as the baseline results to compare to the future results. The results demonstrate that the best subspace among visible and NIR is "B" and its total error value is 5.11. Moreover, the results of gray level subspace for both visible and NIR images are worth than B subspace. So as we expected, by using colour information the performance of face verification system can improve.
In the next step, the adopted search method, Plus 'L' and Take away 'R' algorithm was used for selecting a subset of colour spaces among existing 5 subspaces. In the search algorithm, L = 2 and R = 1 were considered. The scores related to each colour space were appropriately normalised before fusion. The normalised scores were then combined using the both sum and SVM rule. First, fusion by using sum rule was done. Figures 4 and 5 show the error rates during the selection process (different number of colour spaces) in evaluation and test stage respectively.    Results reported in table 3 confirm the effectiveness of the adopted fusion strategy. Using the proposed method, the total error is reduced to 1.28 and 2.97 by using sum and SVM rule respectively, but Between fusion rules, using Sum rule lead to better performance.
These results also demonstrate that although some of the selected colour spaces lead to very poor performances individually, they can be sources of complementary information which can be useful via fusion process. As well the selected colour subspaces for both sum and SVM rule are combination of NIR and Visible subspaces. So it shows that the information from different spectrum can complete each other.

VIII. CONCLUSION
We addressed the problem of fusing colour information in NIR and visible images for face authentication. In a face verification system which is based on the normalized correlation measure within the LDA face space, a sequential search approach similar to the"plus L, and take away R" algorithm was applied in order to find an optimum subset of the colour spaces. Using the proposed method, the performance of the verification system considerably improves as compared to each colour spaces of visible and NIR spectrum individually. The proposed NIR and visible colour fusion scheme also consistently outperforms the best colour space in different conditions. Also using sum rule lead to best performance.