Independent Component Analysis of Image Data

Sridhar Pentapati        Natacha Gueorguieva
pentapati@postbox.csi.cuny.edu
natachag@postbox.csi.cuny.edu
Department of Computer Science
College of Staten Island/CUNY

Abstract
The objective of independent component analysis (ICA) is to represent a set of multidimensional measure-ment vectors in a basis where the components are statistically independent or as independent as possible. This redundancy reduction between components relates ICA to many other theoretical and practical areas of science that are concerned with information, such as information theory, compression, sensory coding and pattern recognition [1].
There are several goals in the experiments to apply ICA to image data as:
ü Verify/access the applicability of the linear ICA model to image data in order to consider the limita-tions of the model.
ü Access the applicability of the algorithms to the problem at hand.
ü Obtain basic knowledge about the characteristics of the independent components of the data.
ü Consider the applicability of the results to various image processing tasks, notably feature extraction and compression.
ü Interpret the result to access the connections between ICA and other areas of research and the impli-cations of these similarities and differences.
A set of new algorithms for ICA has been developed during the 1990' and some of them have been applied to image data, but no larger investigation has been performed to determine the properties and limitations of these methods. We present the applicability of ICA to analyze statistical dependencies in images as well as the connections of these algorithms with closely connected areas of information science.

1. Introduction

1.1. Principal Component Analysis
Principal Component Analysis (PCA) involves a mathematical procedure that transforms a number of (possibly) correlated variables into a (smaller) number of uncorrelated vari-ables called principal components. The first principal component accounts for as much of the variability in the data as possible, and each succeeding component accounts for as much of the remaining variability as possible.
Traditionally, PCA is performed on a square symmetric matrix of type SSCP (pure sums of squares and cross products), covariance (scaled sums of squares and cross products), or correlation (sums of squares and cross products from standardized data). The analysis results for objects of type SSCP and covariance do not differ. A correlation object has to be used if the variances of individual variates differ much, or the units of measurement of the individual variates differ. The result of a PCA on such objects will be a new object of type PCA.
Objectives of PCA are
· To discover or to reduce the dimensionality of the data set.
· To identify new meaningful underlying variables.

1.2. Independent Component Analysis
ICA is a statistical and computational technique for revealing hidden factors that underlie sets of random variables, measurements, or signals.
ICA defines a generative model for the observed multivariate data, which is typically given as a large database of samples. In the model, the data variables are assumed to be linear or nonlinear mixtures of some unknown latent variables, and the mixing system is also unknown. The latent variables are assumed nongaussian and mutually independent, and they are called the independent components of the observed data. These independent components, also called sources or factors, can be found by ICA [2, 3].
ICA is a statistical technique for decomposing a complex dataset into independent sub-parts.
The most important applications of ICA are
· The cocktail-party problem
· Separation of Artifacts in MEG Data
· Finding Hidden Factors in Financial Data
· Reducing Noise in Natural Images
· Telecommunications

2. Mathematical Background
2.1. Principal Component Analysis
The mathematical technique used in PCA is called eigen analysis: we solve for the eigen values and eigenvectors of a square symmetric matrix with sums of squares and cross products. The eigenvector associated with the largest eigen value has the same direction as the first principal component. The eigenvector associated with the second largest eigen value determines the direction of the second principal component. The sum of the eigen values equals the trace of the square matrix and the maximum number of eigenvectors equals the number of rows (or columns) of this matrix.
2.2. Independent Component Analysis
Linear algebra, sometimes disguised as matrix theory, considers sets and functions that preserve linear structure. In practice this includes a very wide portion of mathematics! Thus linear algebra includes axiomatic treatments, computational matters, algebraic structures, and even parts of geometry; moreover, it provides tools used for analyzing dif-ferential equations, statistical processes, and even physical phenomena. The understand-ing of vector space and matrix theory helps in the mathematical analysis of the ICA.
The data analyzed by ICA could originate from many different kinds of application fields, including digital images and document databases, as well as economic indicators and psychometric measurements. In many cases, the measurements are given as a set of parallel signals or time series; the term blind source separation is used to characterize this problem. Typical examples are mixtures of simultaneous speech signals that have been picked up by several microphones, brain waves recorded by multiple sensors, interfering radio signals arriving at a mobile phone, or parallel time series obtained from some in-dustrial process [4].

3. Example
We demonstrate ICA for solving the Blind Source Separation (BSS) problem [5].
We are given two linear mixtures of two source signals, which we know to be independ-ent of each other, i.e. observing the value of one signal does not give any information about the value of the other. The BSS problem is then to determine the source signals given only the mixtures. Putting this into mathematical notation, we model the problem by
x = As
where s is a two-dimensional random vector containing the independent source signals, A is the two-by-two mixing matrix, and x contains the observed (mixed) signals.
This first plot (below) shows the signal mixtures in Fig.1 and the corresponding joint density plot on the right in Fig.2. That is, at a given time instant, the value of the top sig-nal is the first component of x, and the value of the bottom signal is the corresponding second component. The plot on the right is then simply constructed by plotting each such point x. The marginal densities are also shown at the edge of the plot.
A first step in many ICA algorithms is to whiten (sphere) the data. This means that we remove any correlations in the data, i.e. the signals are forced to be uncorrelated. Again putting the words in mathematical terms, we seek a linear transformation V such that when y = Vx we now have E{yy'} = I. This is easily accomplished by setting V = C-1/2, where C = E{xx'} is the correlation matrix of the data, since then we have E{yy'} = E{Vxx'V'} = C-1/2CC-1/2 = I.



Fig.1 Source Signals                     Fig.2 Density

Fig.3 and Fig. 4 below show whitened signals and density after removing the correlation. After sphering, the separated signals can be found by an orthogonal transformation of the whitened signals y (this is simply a rotation of the joint density). The appropriate rotation is sought by maximizing the non-normality of the marginal densities (shown on the edges of the density plot). As stated by the central limit theorem, a linear mixture of independ-ent random variables is necessarily more Gaussian than the original variables. This im-plies that in ICA we must restrict ourselves to at most one Gaussian source signal.
There are many algorithms for performing ICA, but the most efficient to date is the Fas-tICA (fixed-point) algorithm, which was developed by HUT (Helsinki University Of Technology, Finland). Fig. 5 shows the result after one step of the FastICA algorithm.
It presents the description of the different steps that are followed in the experiment using the FastICA algorithm.
The definitions of independence and convergence in this case are based on the following considerations, the source signals (components of s) in this example were a sinusoid and impulsive noise, as can be seen in the left part of the Fig.5 above. The right plot shows the joint density which can be seen to be the product of the marginal densities, i.e. p(s)= p(s1)p(s2).

Fig.3 Source Signals Fig.4 Joint Density
The definitions of independence and convergence in this case are based on the following considerations, the source signals (components of s) in this example were a sinusoid and impulsive noise, as can be seen in the left part of the fig.5 above. The right plot shows the joint density which can be seen to be the product of the marginal densities, i.e. p(s)= p(s1)p(s2).

4. Conclusions
ICA can be seen as an extension to principal component analysis and factor analysis. ICA is a much more powerful technique, however, capable of finding the underlying factors or sources when these classic methods fail completely.
ICA is a very general-purpose statistical technique in which observed random data are linearly transformed into components that are maximally independent from each other, and simultaneously have "interesting" distributions. ICA can be formulated as the esti-mation of a latent variable model. The intuitive notion of maximum nongaussianity can be used to derive different objective functions whose optimization enables the estimation of the ICA model. Alternatively, one may use more classical notions like maximum like-lihood estimation or minimization of mutual information to estimate ICA; somewhat sur-prisingly, these approaches are (approximately) equivalent. The FastICA algorithm gives a computationally very efficient method performing the actual estimation. Applications of ICA can be found in many different areas such as audio processing, biomedical signal processing, image processing, telecommunications, and econometrics.

Fig.5 Signals and joint densities
References
[1] A. D. Back and A. S. Weigend, "A first application of independent component analysis to extracting structure from stock returns", Int. J. on Neural Systems, 8(4):473-484, 1998.
[2] The FastICA MATLAB package, http://www.cis.hut.fi/projects/ica/fastica/
[3] A. Hyvärinen, "Survey on independent component analysis", Neural Computing Surveys, 2:94-128, 1999.
[4] K. Kiviluoto and E. Oja, "Independent component analysis for parallel financial time series", In Proc. ICONIP'98, volume 2, pages 895-898, Tokyo, Japan, 1998.
[5] Ristaniemi and J. Joutsensalo, "On the performance of blind source separation in CDMA downlink", In Proc. Int. Workshop on Independent Component Analysis and Signal Separation (ICA'99), pages 437-441, Aussois, France, 1999.