Quantization Selection of Colour Histogram Bins to Categorize the Colour Appearance of Landscape Paintings for Image Retrieval

— In the world of today, most images are digitized and kept in digital libraries for better organization and management. With the growth of information and communication technology, collection holders such as museums or cultural institutions have been increasingly interested in making their collections available anytime and anywhere for any Image Retrieval (IR) activities such as browsing and searching. In a colour image retrieval application, images retrieved by users are accomplished according to their specifications on what they want or acquire, which could be based upon so many concepts. We suggest an approach to categorize the colour appearances of whole scene landscape painting images based on human colour perception. The colour features in the image are represented using a colour histogram. We then find the suitable quantization bins that can be used to generate optimum colour histograms for all categories of colour appearances, which is selected based on the Harmonic Mean of the precision and recall, also known as F-Score percentage higher saturated value. Colour appearance attributes in the CIELab colour model ( L -Lightness, a and b are colour-opponent dimension) are used to generate colour appearance feature vectors namely the saturation metric, lightness metric and multicoloured metric. For the categorizations, we use the Nearest Neighbour (NN) method to detect the classes by using the predefined colour appearance descriptor measures and the pre-set thresholds. The experimental results show that the quantization of CIELab colour model into 11 uniformly bins for each component had achieved the optimum result for all colour appearances categories.


I. INTRODUCTION
Recently, designing a search image mechanism based on user requirements has become an important and critical challenge [1]- [4]. Image Retrieval (IR) from a digital library or a database can be done using text description query or image query depending on how the applications of those retrieval systems are. Images in a database have to be indexed for IR activities such as searching and browsing purposes. Two methods commonly used to index the image are first, by manual annotation or text description and second, by using image visual content. Indexing images manually using text descriptions has shown a lack of the user's satisfaction due to human subjectivity and creativity. In addition, this method also can be time-consuming and inconsistent. To overcome those limitations, indexing method using image visual content such as colours, shape, and texture are preferred as it can be done automatically to each image in the digital libraries and provide a better set of relevant results [5]. Visual content or visual features of an image are characteristics that give meaning to an image. Colour feature is always said to be the easiest and strongest feature captured by human eyes [6]. Colour feature extracted from an image contains colour information. This information can be captured and its distribution can be represented. Examples of colour representation are colour histogram, colour correlogram, colour moments, and colour coherence vector [7]. It has been observed that the colour histogram representation can suit many content-based image retrieval applications, image classification, measuring the similarity between images and used effectively for image indexing as well [8]- [10].
In the process of retrieving colour images, each image's colour feature needs to be analyzed from its colour histogram. The colour histogram depends on the appropriate colour space model selection and colour quantization bins implementation. Colour quantization is a process of reducing the number of colours to use in an image. The task of colour quantization is to select and assign a limited set of colours and form a colour histogram. Quantization bins uniformly or non-uniformly must be selected after a meticulous study. Higher quantization bins enable the capturing of more information of colours in a colour histogram. Often in popular belief, the high colour information extracted from an image would give the impression that the search will be better. However, high quantization bins do not necessarily lead to an optimal precision of search. Retrieval performance can be saturated and can lead to a less efficient search when the number of bins is increased beyond some values [11]- [13]. Gavrielides et al. [14] examined a quantization method called Fibonacci lattices for a, b plane sampling which has been introduced by Mojsilovic & Soljanin [15] using CIELab colour space. They showed that Fibonacci Lattice sampling gave an increasing performance result from 10 to 43 samplings but maintained or saturated when the sampling increase to 101. In their work, the dataset images used for the experiment were fingerprinting images that have small range of colours.
Park and Park [16] have studied that, quantization as low as 5x5x5 together with their weighting vector, was fair enough to show discriminations in the characteristics of images. In another research work by Ciobanu et al. [17], an automatic method for computing the bin boundaries for L, a and b axes was proposed to generate optimal colour feature vectors (colour histogram). They proposed a technique to reduce the number of bins on certain colour space quantization. In their work, they detected the highest and the lowest L, a, b values that existed in the dataset sample images and decided the quantization intervals for the L, a and b axes. The distribution of pixels in each interval was computed lightly the same. They used iris segmented images in the experiments. This quantization method generated nonuniform bins for a small range of L, a and b boundaries values and the results were good for classification of iris segmented images. [18] suggested a way recognize the colour of the vehicle contained in the image. The colour feature of a vehicle is represented by a colour histogram. After generating the histograms, template matching was used to decide the vehicle colour. In their experiments using HSI colour space, the partition of H, S, and I into 8, 4, 4, respectively, achieved the highest success rate.
Most research studies on techniques of colour histogram, colour quantization and colour bins that were used for the image retrieval application based on true colour-based similarity. Lack of attention has been given to intensitybased similarity of colours. Sometimes, there is a need to seek an object in an image based on the intensity of the colour. This means that people require only specific-looking images, or search for intensity-based colour in the images. An evident example is the question of image search queries such as Images of a girl wearing bright coloured clothing or Images of pastel-coloured flower pots. The intensity of the colour is one of the important features that can be combined with other features e.g. shape or texture in helping to search images from the database based on the above-mentioned queries. Furthermore, in the world of design such as interior designing or fashion designing, searching images based on the intensity of colour can contribute tremendously to find images that match certain specific design environment. For instance, the colour appearance of a relatively dark room may require bright or pastel paintings to counter-act the dim atmosphere by bringing a pop of colour and life. Through the upgraded search, irrelevant images can be avoided or filtered especially by designers who would require a specific form of colours. With the appearance of colour descriptors for image drawing, the system can provide a list of suggestions for the category of images necessary.
In this paper, we focus on the usage of colour histograms to detect the whole scene colour appearance of landscape paintings which are based on the similarity of colour intensity. In addition, most quantization techniques used to generate colour histograms in the previous research studies were used specifically for IR applications which handle the image dataset with a small number of colours in their content. For example, fingerprinting and iris segmented images which contain colours lying in a narrow range within a colour model. Therefore, the objectives of this study are first, to investigate quantization bin techniques for generating a colour histogram of landscape paintings. For this objective, we have been studying some literature reviews or findings on other researches related to quantization bins, colour histogram, colour features and colour appearance attributes in CIELab: Saturation, Lightness, and Chroma. And secondly, to conduct experiments to determine suitable quantization bins for generating an optimum colour histogram that can effectively categorize all colour appearance of landscape paintings into six selected colour appearances.
In this section, we introduce the problems related to the study of colour appearance categorization of landscape paintings. The remaining part of this paper is organized as follows. Section II discuss the method to construct feature vectors from the colour histogram of a given landscape painting. The colour appearance of a painting is categorized based on a predefined colour appearance descriptor developed using colour appearance feature vectors namely saturation metric, lightness metric and multicoloured metric. Experimental results are shown in Section III and we conclude this paper in Section IV.

A. Colour Appearances Categorization of Landscape Paintings
Landscape paintings are the depiction in art that refers to works of art involving feature views of nature, including seascapes, cityscapes, and waterscapes. Landscape paintings are fully paved with colours all-round. Word descriptions are used by humans to describe the colour appearance of a landscape painting's scene such as bright, vivid, dull etc. These perceptual colour descriptions depend on certain criteria such as the experience of the viewer, the conditions of the colour or other colours around it. Humans describe colours based on the colour component (true colours such as red, blue, green etc.) and the intensity component (amount of lightness and saturation) [19]. Gigarama et al. [20] studied on how Japanese and Sinhala native viewers describe differences in most commonly used colour tones such as bright, vivid, strong, dark, pale and dark in their own native languages. In our previous work, we have done a small scale psychophysical experiment on human colour perception or how human perceived the colours of images. In that research, participants have to categorize landscape paintings images into six colour appearances: bright, pastel, dull, pale, dark and multicoloured. A set of ground-truth images for each category of colour appearance has been determined. In this study, the experiments were done using these ground-truth images.
In this paper, we use colour histogram as a feature vector. Our focus is to find one suitable quantization bins for all appearance that can be used to represent colour feature despite the difference in colour appearance as illustrated in Table 1. For instance, in Table 1, two example images of Bright category show different colour content but both were categorized in the same category based on the intensity of their colour content. A suitable colour histogram could be optimum and it is beneficial for the efficient storage and cost of processing. The desire to identify the colour appearance of landscape paintings led to an attempt at finding a suitable selection of quantization bins. If it is too small, there will be a possibility of losing details about the color content in an image which will affect the categorization process. As we have noticed, landscape paintings may consist of a wide range variety of colours in a colour model. The colour appearances of a colour histogram are categorized based on colour appearance feature vectors metric. These metrics are formulated to index images and these metric are used individually or together in determining the category of colour appearances. Categorization of landscape paintings based on human colour perception can help in browsing images using keywords. Thus, this study also pays attention in narrowing the gap between the human concepts and lowlevel concepts in terms of the colour appearance of an image scene. The goal is to retrieve a set of images that match the colour appearance description queried by humans using keywords.

B. Colour Histogram
In representing human colour perception, we use CIELab colour model as it is a perceptually uniform colour space which means that the same ΔC at two different points in the colour space makes the equal perceivable colour difference [21]. Before the colour histogram for an image can be computed in CIELab, the image which was in RGB colour model need to be converted into CIELab values. RGB colour model is frequently used as the base colour space for most applications. This model requires no transformation to display information on the screen or display devices. Conversion of RGB image pixel values to the CIELab can be achieved in two processes. First, the RGB values have to be converted to CIEXYZ values and then the values will be converted into CIELab. The conversion formula from RGB to CIEXYZ and from CIEXYZ to CIELab is defined as below: (1) For this study, observer angle of 2° was used in the formula to match the standard observer of CIE 1931 and the illuminant used was according to CIE standard illuminant D65 = 6500K. Further detail explanation about the conversion formula from RGB to CIELab can be found in [21]. Let each component L (Lightness), a and b (opponent colours red-green and blue-yellow respectively) divided into dimensions or ranges, DL, Da

CH[L][a][b] = Compute NoPixels[L][a][b]
C. Colour Appearance Feature Vectors Colour appearance feature vectors are vectors that were used for indexing images in the databases. We used colour appearance feature vectors namely saturation metric, lightness metric and multicoloured metric as formulated in [22]. These metrics have been calculated using colour attributes of CIELab colour model: Lightness (L), Chroma (C) and Saturation (S). In [22], in general, overall saturation was measured by calculating the purity of chroma in each bin relative to its condition of lightness. This can be calculated by firstly finding the local saturation for each by dividing the local chroma by its lightness and normalizing the value. Secondly, find the area of each bin. Thirdly, calculate overall saturation by finding the summation of local saturation. The lightness of a whole-scene image was measured by examining the luminous intensity of the colours of the image content. Two calculations for the lightness metric to reflect both low and high illuminance were developed. All the values in the bins of the histogram were analyzed and accumulated according to lightness intensity levels. A multicoloured metric was developed by analyzing all information regarding its hues and chroma level. Four levels of chroma, six unique hues, and their angles were used in the calculation method. Further explanation of colour appearance feature vectors development can be found in [22]. All formulated metrics values were normalized in the range from 0 : (minimum) to 1: (maximum).

D. Landscape Painting's Scene Categorization
Each image is associated with these colour appearance metric values in the database. For the categorization process, the saturation, lightness, and multicoloured metrics are used individually or combined. By using predefined colour appearance descriptor measures, images are categorized into their relevant colour appearance categories using Nearest Neighbour (NN) method. NN is a proximity search for finding closest or similar point in which the points are separated into several separate classes. Despite its simplicity, NN has been successful in a large number of classification problems such as in [23]- [26]. During a query, these metric values will be compared so the process of retrieving or browsing of related images could be performed.

III. RESULTS AND DISCUSSION
A total of sixty-two ground-truth images were used during the experiments. These ground-truth images have been identified from a psychophysical experiment survey which was conducted in the previous work. The ground-truth images were identified for each colour appearance category: Bright, Pastel, Dull, Pale and Dark which have 10 groundtruth images each and 12 ground-truth images for the Multicoloured category. The colour histograms for each image were obtained by generating various quantization of L, a, b components uniformly starting from 5, 7, 9, 11, 13, 15, 17 and 19. We ended up at 19 because [11] in their research studied the quantization bins that give optimal results in HVS, RGB and XYZ colour model. From their experiments, the quantization ranging from 8 to 15 would give optimal results. However, for this study, we increase / reduce one or two range for a better perspective in results comparison. Furthermore, we applied to the different application.
For each image, its colour histogram at every quantization level was used to calculate a collection of colour appearances feature vectors metric; saturation metric, lightness metric and multicoloured metric. Based on the predefined colour appearances descriptors measures, the performance of colour histogram for each quantization levels in IR were measured using Precision & Recall method. Precision explains on how relevant the retrieved images are. Recall, on the other hand explains whether the system retrieves many of the truly relevant images [27]. The precision and recall are defined as below:  Table 2 and Table 3 show the result of precision and recall percentage respectively for every colour appearance category in various quantization bins. The average precision and recall have also been calculated for each category of colour appearance. Average precision shows that the retrieval is better for Bright, Dark and Multicoloured categories. Compared to Pastel, Dull and Pale, these categories have lower retrieval percentages. For recall percentage in Table 3, the percentage for every category are good for Pastel, Dull and Multicoloured and even better for Dark, Bright and Pale categories. Both precision and recall show that the system can discriminate Multicoloured images starting from quantization 11.
The precision percentage for some categories such as Pastel, Dull and Pale are slightly lower as can be seen in Table 2. Although these percentages are below significant value, that is, less than 0.6 according to performance measures of Precision and Recall graph performance [28]- [29], we still uphold with these results because this paper is not focusing mainly on the IR performance evaluation but the main objective is to determine the optimum quantization bins for colour histogram generation for all category of colour appearances. The low recall percentages obtained from the experiments indicate that some predefined colour appearances measure descriptors with their thresholds value used in this experiments work in a fair and adequate manner. This is expected because the descriptors have been developed without a thorough survey of human judgment. Nevertheless, the descriptors and thresholds could be enhanced (this will be discussed in our future work) to give higher precision or recall values which can give a better IR performance. For the purpose of this research, the precision and recall percentage enable us to analyse the performance of colour histogram and helping in selecting the optimum quantization bins.
Based on the precision and recall percentage for categories at different quantization bins as shown in Table 2 and Table 3, we can make such statements for example at quantization bins 9 : 83% of the images retrieved by the system were bright images, and out of the whole total of bright images in the database, 91% were retrieved by the system. Another example: 48 % of the images retrieved by the system were Dark images, and out of the whole total of Dark images in the database, all of them or 100% of Dark images were retrieved successfully by the system. In this case, although the average precision percentage is low, all the Dark images were well retrieved but were together with 52% of images that did not belong to the Dark category.
Both precision and recall measure are useful and needed to evaluate the accuracy of image retrieval with relevance to the query and database images and to explain the effectiveness of IR. However these two measurements cannot be considered as complete accuracy for the effective IR [30]. In this work, we have to find one measurement that can verify which quantization bins is optimum for all categories of images. Hence, for that purpose, precision and recall measurements for each quantization can be combined to give a single value for further analysis. The combination computation used here is called F-Score or Harmonic Mean of the precision and recall as explained in [30].
The formula to compute F-Score measurement for each quantization can be defined as: (9) The precision and recall values used in this formula are the average precision and recall of all colour appearance categories calculated at each quantization. The F-Score was then computed using these average precision and recall values as shown in Table 4. Based on these results, we plotted a graph between the quantization bins and the F-Score percentage as illustrated in Fig. 1. From the graph, we can see that the percentage of F-Score increases from quantization 5 and saturated at quantization 11. The graph shows that quantization 11 is the optimum and could be used to generate a colour histogram for all colour appearance categorization.   We then applied the same technique used by [16] in his work to reduce some unnecessary bins by identifying the lowest and the highest colour component values that exist in the dataset images. Thus, truncated these unnecessary bins which located outside the lowest and the highest values could make the histogram file smaller in size. We scanned the lowest and the highest Lab values from 62 ground-truth images used in this work as well as 500 more images of landscape paintings which were randomly downloaded from various websites. The purpose of scanning more landscape painting images was to get the lowest and highest values as true as possible for this research work. Table 5 shows the lowest and highest values were found to be different for both groups. However, we identified the lowest and the highest values of all and those values were from the ground-truth images. The selected highest and the lowest values as shown in Table 5 have enabled us to detect the unused bins for L, a and b ranges and this unused bin were truncated. As a result, the old quantization of 11, 11, 11 for L, a and b component have been reconstructed into 11, 9, 10 after the truncation of the bins outside the ranges. The colour histogram for each image has been regenerated again according to the new truncated quantization of 11, 9, 10.    Table 6 shows the comparison between the sizes of colour histogram files before and after truncation. The average size of the colour histogram files for quantization of 11, 11, 11 has reduced from 4.76 Kb to 4.21 Kb or 11.55% after the truncated of bins outside the range to make quantization of 11,9,10. The number of processed bins reduced from 1331 bins to 990 bins or 25.62%. In a huge collection of images, these reductions by 11.55% and 25.62% of file size and processed bins respectively could be significant. The reduction can save space and also make the processing time faster.

IV. CONCLUSIONS
As a conclusion, in this work, we have successfully identified the suitable quantization bins that can be used for generating colour histograms for six colour appearance landscape painting scene categorization. These colour appearance categories are Bright, Pastel, Dull, Pale, Dark and Multicoloured. Based on our experiment results using the predefined colour appearance descriptor measures, the quantization of 11, 11, 11 is identified as the optimum quantization for all categories of colour appearances studies in this paper. The result was based on the F-Score percentages which were computed from the average percentage values of Precision and Recall of all colour appearance categories. The F-Score percentage increases from quantization bins of 5 and saturated at quantization bins 11. To further make the quantization bins optimum which, as a result, could reduce processing time and create a smaller size of the colour histogram file, we truncated all bins component outside the highest and lowest values of L, a and b. Thus, the final optimum quantization of L, a and b for this research is quantization bins of 11, 9, 10. In the near future, our next investigation will be focusing on developing and enhancing the colour appearance descriptors using the outcome of this paper, to increase the performance of IR applications. A good descriptor will be able to retrieve a lot of relevant images before retrieving any non-relevant images and its precision percentage will stay high as the recall percentage increases. Therefore, the involvement of human judgement investigation on the colour appearance of images is significantly important for colour appearance descriptors development. This can help in narrowing the gap between the human concepts and lowlevel concepts in terms of the colour appearance of an image scene. High relevancy between human text descriptions and image content-based indexing can help users to get what image they want while browsing or searching for images based on colour appearances. This could be done by doing more surveys to gather all possible details and characteristics

Harmonic Mean of Precision and Recall (%)
Quantization bins of the colour appearance of landscape painting scene from human colour perception.