LDA classifier

1．1 The main research content of the topic

（1） Introduce LDA The principle and algorithm flow of classifier , And skillfully use LDA The algorithm performs the classification operation of two types of samples ;

（2） Application LDA The algorithm performs classification operations on two types of samples ：

:{(4,2),(2,4),(2,3),(3,6),(4,4)}

:{(9,10),(6,8),(9,5),(8,7),(10,8)}

1．2 Working basis or experimental conditions of topic research

（1） Manual deduction

The algorithm flow is realized through manual deduction , And calculate the classification results ;

（2） Software environment

adopt MATLAB The software implements the algorithm flow to verify the manual deduction results , And draw the classification result graph .

1．3 design idea

Suppose two types of samples ：{(,),(,),…,(,)},:{(,),(,),…,(,)}, among i= ,j= .

（1） Calculate the first kind of sample mean vector   And the second kind of sample mean   And the total sample mean m;

（2） Calculate the divergence matrix , Including intraclass divergence matrix Sw And the inter class divergence matrix Sb;

（3） Calculation   All eigenvectors and corresponding eigenvalues of ;

（4） Select the eigenvector corresponding to the maximum eigenvalue W;

（5） Dimension reduction of samples , Get the corresponding dimension reduction sample Y, among Y=X·W.

1．4 flow chart

chart 1 LDA algorithm flow chart

1．5 Main program code ( Notes are required )

Definition FisherLDA function code ：

function [W] = FisherLDA(w1,w2)

%W Is the eigenvector corresponding to the maximum eigenvalue

%w1 The first kind of sample

%w2 The second type of sample

% First step ： Calculate the sample mean vector

m1=mean(w1);% The first kind of sample mean

m2=mean(w2);% The second kind of sample mean

m=mean([w1;w2]);% Total sample mean

% The second step ： Calculate the intra class Sandu matrix Sw

n1=size(w1,1);% Calculate the number of samples of the first type

n2=size(w2,1);% Calculate the number of samples in the second category

  % Find the divergence matrix of the first kind of samples s1

s1=0;

for i=1:n1

    s1=s1+(w1(i,:)-m1)'*(w1(i,:)-m1);

end

  % Find the divergence matrix of the second kind of samples s2

s2=0;

for i=1:n2

    s2=s2+(w2(i,:)-m2)'*(w2(i,:)-m2);

end

Sw=(n1*s1+n2*s2)/(n1+n2);

% The third step ： Compute the divergence matrix between classes Sb

Sb=(n1*(m-m1)'*(m-m1)+n2*(m-m2)'*(m-m2))/(n1+n2);

% Step four ： Find the maximum eigenvalue and eigenvector

%[V,D]=eig(inv(Sw)*Sb);% Eigenvector V, The eigenvalue D

A = repmat(0.1,[1,size(Sw,1)]);%repmat For creating all elements, the value is 0.1 Of 1 That's ok size(Sw,1) Columns of the matrix

B = diag(A);% Generate diagonal matrix

[V,D]=eig(inv(Sw+B)*Sb);% Find all eigenvalues of the matrix to form a diagonal matrix D, And find the eigenvector of the matrix V The column vector

[a,b]=max(max(D));%a Is the maximum value of the column ,b Is the row number of the column maximum

W=V(:,b);% Eigenvector corresponding to the maximum eigenvalue W

end

The main function LDA Code ：

% Give two types of samples

cls1_data=[4 2;2 4;2 3;3 6;4 4];

cls2_data=[9 10;6 8;9 5;8 7;10 8];

% The point of two-dimensional coordinates before the projection of the sample

plot(cls1_data(:,1),cls1_data(:,2),'.r');

hold on;

plot(cls2_data(:,1),cls2_data(:,2),'*b');

hold on;

% By calling FisherLDA Function to find the eigenvector corresponding to the maximum eigenvalue

W=FisherLDA(cls1_data,cls2_data);

fprintf('the value of W is [%4.2f;%4.2f] ',W);

% Draw the sample projection line

new1=cls1_data*W;

new2=cls2_data*W;

k=W(2)/W(1);

plot([0,12],[0,12*k],'-k');

axis([0 12 0 12]);

hold on;

% Draw the point where the sample is projected onto a straight line

for i=1:5

    temp=cls1_data(i,:);

    newx=(temp(1)+k*temp(2))/(k*k+1);

    newy=k*newx;

    plot(newx,newy,'*r');

end

for i=1:5

    temp=cls2_data(i,:);

    newx=(temp(1)+k*temp(2))/(k*k+1);

    newy=k*newx;

    plot(newx,newy,'ob');

end

1．6 Operation results and Analysis

Pass diagram 2 The best vector of W The running result chart shows MATLAB The result of simulation is basically consistent with the result of manual deduction , It not only verifies the correctness of manual deduction , Also verified. LDA Algorithm principle and flow of ; Pass diagram 3 It can be seen from the operation result diagram of dimension reduction classification that the dimension of coordinate points in the two-dimensional plane is reduced to a one-dimensional straight line , It can be seen from the one-dimensional straight line that the within class distance of each kind of sample point is the smallest , The distance between two classes of samples is the largest , It is further verified that LDA The principle and flow of the algorithm .

chart 2 Best vector W Operation result diagram

chart 3 Dimension reduction classification operation result diagram