ArtificialNeural Networks: Kohonen self-organizing maps (SOM).

ArtificialNeural Networks: Kohonen self-organizing maps (SOM).

Author : Yifan Wang

Supervisors: Dr Irina Biktasheva, Dr GrantMalcolm

 

Abstract

Thisproject is about turning theory into practice. Since Kohonen self-organizingmaps are widely used in Computer Science for pattern recognition, featuredetection, and clustering problems. Hence the theory could be applied tosolve  clustering letters and lettersrecognition relevant problem which is trying to use Kohonen self-organizingmaps to cluster letters from different fonts.

Sodeveloping a java-based software is considered of making progress on theproblem. And the software is actually a simulation of Kohonen self-organizingmaps.

Theresult shows trying to cluster letters from different fonts could work in mostcases. And this project has displayed the process of turning theory namelyKohonen self-organizing maps into practice, which is trying to cluster lettersin different fonts. More over the process also dispose the possibility that theproject could be extended in pattern recognition area.

 

 

1.Introduction

1.1Problem Statement

  The problem addressed by this project isabout turning the studying theory into practice in order to solve letterclustering problem. In more detail, the problem is how to make the projectimplement a software to clustering letters in different fonts based on KohonenSelf-organizing Maps which is a artificial neuron network. 

1.2Aims and Objectives

Theaim is about what the project hopes to do as its overall intention. So at theend, the project will at least solve the problem stated above . And this aimcould be decomposed to sub-aims which are an establishment of the principlemodel of Kohonen self-organizing maps that could solve the problem, theimplementation of the software based on the principle model and a lot ofpractices in the software . After that the project is aim to make the learningsystem be displayed as well. In addition, the project is also going to proofthe possibility of the unsupervised learning system.

Objective1

Andthe objectives are made through few specific steps in order to achieve the aimthat is to solve the declared problem. To achieve the sub-aim that is to builda principle model, firstly i analysis the nature of Kohonen self-organizing maps.Then a simple template could be easily established by writing them down thepaper with a good understanding of the operating principle of Kohonenself-organizing maps. Finally i could construct a complete and robust Kohonenself-organizing neuron network by extending the simple template after it hasbeen testified for the correctness and the robustness. 

Inorder to satisfy the aim that is to simulate the neuron network by a software,i produce a list of pseudo code that coded from the principle model at first.Then i create several java methods to simulate the mechanism of Kohonenself-organizing maps in the way that Kohonen self-organizing maps do which alsodescribed in the pseudo code list. After all a java based software with aconvenient GUI is been made by adding the GUI component from java.swing.

It isnecessary to do massive data practices to the system of software for thesub-aim of practices in the software and further step aims.

Objective2

  Few steps are supposed to be gone through toachieve the aim that displays the learning system as a feature of the software.At first, i will analysis the massive data test collected from last aimachievement to figure out which set of data is suitable for the display oflearning system. Secondly it is necessary to test the data to fit the rule thatmeans one letter of the set of data(a class formed by all fonts in a type ofletter such as B) is failed to be recognized in the initial state of thesoftware. Then there would be a record of the whole process as a evidence forthe aim and contribute to the next aim the proof of unsupervised learningsystem.

Objective3

  This aim could be accomplish by finish of 26letters display as the same way used in aim-2. 

1.3Challenges

1.3.1Lack of constraints

 Building a software is not a subject thatfollows laws and formulas like law of conservation of mass and plenty chemicalformulas used in chemical factories. It clearly states that software seems haveunlimited possibilities for its function. Even the problem that it is going tosolve is specified and limited. There is still many ways to make a software. Sobased on the limited constraints of software, bearing the difficulty ofchoosing a proper way to implement the functions and its corresponding risks isinevitable.

  In the project, the problem declare thatsoftware could cluster letters from different fonts. Therefore building afunction that turns the letters into software recognizable input is necessary.Clearly copy and paste letter seems like it is the simplest way to achieve thefunction. But the way to decompose the letter into inputs that let KohonenSOM(Appendix 1)easy to use is a serious issue. Moreover this is only the aspectof difficulty but does not mean it is impossible to execute for that function.For the aspect of risk, choosing an inappropriate way could bring a lottroubles that make software is highly restricted in some area. And thosetroubles are sometimes unpredictable. Therefore it dramatically increases therisk when choosing a way to accomplish the function.

1.3.2Visualization of the software

  Visualization of a specification of thesoftware is much harder than other disciplines. Since programs are invisible infact and results shown by the program entail user having necessary technicalknowledge. Visualization always become a big problem in this case.

  Especially it is worse ,since visualizationis just introduced briefly during the study of kinds of modules in computerscience. From my 3 years study, the core of courses focus on programminglanguage and computer science theory study. Visualization only be introduced inyear1 about how to craft a user friendly GUI. In this situation, i may have noidea where to start making the graphic of the software. Hence visualization ofthe software is important but challenge-able to me.

1.4Solution produced

  The solution produced is a combination ofJava software and a specified Kohonen SOM neuron network. The Kohonen SOMneuron network follows the rule from Kohonen SOM like normalization, update theweights of connection and stop conditions. And importantly it could recognizeinput patterns that contain strings of binary numbers as the Kohonen SOM alwaysdo. Since the strings of binary numbers could represent letters by decomposethem into an array of digital numbers. As a consequence, the Kohonen SOM neuronnetwork has the ability to cluster letters as input and generate the output toan output unit which represent a letter type. So clearly the Kohonen SOM neuronnetwork has the same function defined in the problem statement of this project.

  After finish of neuron network, list ofpseudo codes would be easily produced. And considering it ahead of the softwaredevelopment is really a wise behavior. Then java has been chosen to be theprogramming language for implementation of the software. Since java is widelyused ,convenient and crossing plant-forms. Then the pseudo code leads to buildthe java file. Then a simulation system would be coded to do the same things asoccurred in Kohonen SOM neuron network by creating methods for normalization,stop conditions etc. And a java based GUI would be in charge of user input asthe function dedicated in problem statement for clustering letters and displaythe outcome of clustering. Last but not least some assistance tool or documentalso will be involved in to make sure the software could execute more smoothly.

1.5Evaluation of solutions

  From my perspective of views, the solution isquite effective. Through the massive practice, the software did its job in mostcases and GUI is user-easy control and visual clear. But there is also acritical limit on user’s behaviors during the use of the software. And learningsystem is able to loose the restraint by its way.

2.Background

2.1Background of the problem

  Sincethe problem is about turning theory of Kohoen SOM into runnable codes. HenceIntroduction of Kohonen SOM’s background is really essential. As declared inthe project statement, Kohonen SOM is widely in use of pattern recognition ,feature detection and clustering system. And it also represents theunsupervised learning neuron network which also contain Hebbian’s rule. And theidea came from Hebbian’s rule also contribute a lot in the build of Kohonen SOMneuron network. Therefor it is necessary to introduce Hebb’s rule ahead.

Hebb’s rule(1949)could be illustrate as a neuron-science postulate:”When an axon of cell A is near enough to excite a cell B andrepeatedly and persistently takes part in firing it, some growth process ormetabolic change takes place in one or both cells,such that A’s efficiency as one of the cells firing B,is increased.”.[1] And the formulation of Hebb’s rulecould be the increase of weight of connection in next time stamp: w^t+1=w^t +∆w^t ;∆w^t= C*a* X^t  where w is weight of connection ,w^t+1  is the weight of next instant,w^tis the instant weight,C is positive coefficient that represent learning rate,ais the value of instant input, X^t is instant output . From theformula weight could be changed only if both input value and output value arenot equal to 0 at same time. This shows the correlation nature of Habbian’s rule. In order to get ideas from Habb’s rule, Habb’s rule innormalization should be introduced as well.

Hebb’s rule inNormalization rule is based on Hebb’s rule apparently.But the sum of weight is kept as a constant value. This could cause thestrength of synapse tends to sharpen a neuron’spredisposition and make its firing get better correlated with a cluster ofstimulus patter, stated by Biktasheva.[2]

Kohonen rule is also based on Hebb’s rule in normalization. The benefit of normalization brought fromHebb’s rule in normalization is to avoid the bug that weight of connectionscould increase forever if normalization is not applied. And It makes differenceto Hebb’s rule by change weight of connections through competitive learningthat would make sure only one set of weight of connections are updated eachrun. They are also different in the way of update of weight of connection. Theformulation of Kohonen SOM is w^t+1= w^t +∆w^t ;∆w^t= C * ( a-w^t). Compare to the formulation of Hebb’s rule, the Kohonen SOM ’s weight ofconnections change towards the value of inputs. In other words, it tends tobecome similar pattern as input do. And this give an ability to Kohonen SOMneuron network that make it capable to solve problems of feature detection,clustering etc.

2.2 Existing solutions

 Unfortunately i did not find any solutions before i had finished mysoftware, or it could be a really big help for my implementation of the software.Because i just found that in one’s report. A simple learning rule forrecognition of hand writing letter draw-ed in computer mouse by usingartificial neural network. He used Kohonen SOM for pattern classification whichapplies unsupervised learning algorithm, stated by Agarwal[3].

From browsing his simple butvaluable report , a diagram above shows the results that are quite amazing interms of percentage of characters being successfully recognized. Compared withmy software, his work is seems more practical than my software. Since handwriting is much more convenient than painting in the aspect of forming aletter. But turning a hand writing letter into a letter formed by cells is notreally difficult. One advantage could be discovered in his diagram is that itssystem is quite robust even in the case that input conditions are reallyunusual . Compared with my system, his software could ask for less restraint tousers. But there is still some good points same as my system, the process ofits learning system never ends until the time software is shut down. That meansthe software is keep learning. In addition, the success rate of his softwareseems really high ,but in my system that rate is also possible after trainedmassive data.

2.3 Reading and research done tounderstand existing approaches

 Apart form Hebb’s rule , Hebb’s rule in normalization and Kohonen SOMintroduced in the background of the problem section, there are still a lotthings required to carry out the project. Lucky from the guidance of project, agood understanding of the project and corresponding skills could be acquiredcomp305 lecture notes which explain the concept of artificial neuron networkfrom easy to difficult. It also follows the sequence that includeMcCulloch-Pitts Neuron, Hebb’s rule, Perceptron rule, Multilayer Perceptronrule, Normalised Hebb’s rule and Kohonen SOM. With learning the rules and allkinds of neurons and Kohonen SOM, a comprehensive view and understanding couldbe effectively used to help the establishment of the project. Because hebb’srule and its normalization are quite relevant to Kohonen SOM. AndMcCulloch-Pitts is introduced as a simple model which is easy for Beginner tocomprehend. The model contains the simulation for logic gates and most terminologyused in latter neuron networks’ study. Moreover, Perceptron rule is declared byRosenblatt(1958) as a supervised learning system that could make neuralnetworks change with experience containing a whole process of settinginput,getting output and calculation of error from output and target output.[4]Perceptron rule is also called error-correction rule. From the rule , weightsof connection in neural network change only if erroneous feedback from thatneuron is found. The formulation is  w^t+1=w^t +∆w^t ;e = t - X ∆w^t= C *a * e. Where e is the error and t is the target output. Hence Perceptron neuronnetwork use the error feedback to adjust weights of connections. And itrepresent the supervised learning , since target output is always available.Catching the nature of supervised learning would be a big help forunderstanding what unsupervised learning really is .

 After reading the materials of module comp305 from its teaching websiteand practicing during its tutorials , one should be capable to make the projectdesign fluently and have a clear clue to produce a solution for the problem.

2.4 Project requirements

2.4.1 Functional requirements

 The project is going to satisfy a function clustering letters fromdifferent fonts. A software with input system will be available to user. Andusers could achieve the function during the use of the software.

2.4.2 Design requirements

 A Kohonen SOM based neuron network would be established to cluster 26uppercase letters in different fonts at first by the knowledge gained fromKohonen SOM related materials. And pseudo code will be generated from thenetwork model. Finally using the pseudo code to program for the network model’ssimulation system and searching java GUI related materials to make userinterface.

2.4.3 Project phase

Once the pseudo code generatedduring the design requirements, the design phase is officially ended. The restof its requirement is working for functional requirements. And after usersuccessfully achieve the function in software. The realization phase ends.

2.4.4 Project schedule

 Appendix [2].

2.4.5 Test plans

 The test plans have already been mentioned in the aims and objectives ofthe project. Plans would be made to proof the existence of learning system andits robustness.

2.4.6 Completeness  

Theproject is regarded as complete and successful once the test plan verify theproject aims and met each documented requirement. And also it is supposed to befinished before the deadline.

3.Design

3.1 Summary of Proposal

3.1.1 Background, aims andobjectives of the project

Appendix[3].

3.1.2 Necessary changes to theoriginal specification ,based on new information and understanding

Complement of the specificationfor proposed software :

A java applet ,which turns thetheory of Kohonen self-organizing maps into codes to achieve the functionclustering letters in different fonts, is used to the proposed system that is asimulation of the neuron network recognition system. More specifically lettersare decomposed to Kohonen SOM tractable version(java object), and input andoutput units of Kohonen SOM could be settled as the representation of lettersas well. And for more details of the simulation system of Kohonen SOM will bestated in 2.1.5 Pseudo-code which stated the key methods. In this comparison,it is based on the old recognition of the difference between my software andother people’s which is not the one i just mentioned in 2.2. And it was madeduring the stage of design. The reason i still keep this is that it is alsogood examples to compare the solution that solved by the neuron network withsupervised learning. And the statement of the modification may stop here. Forthe difference between existing neural-network simulation software and thisproject software, the former is dealing letters recognition in the way that hastarget letters and recognize the letter by a set of training patterns whichonly correspond to some predefined letters. That means its job is abouttraining which help the input letters to realize the specific letter. And inthis case each new input letter found has to be trained to link thecorresponding pattern again. However simulation of Kohonen SOM does not havetarget letters, that means it can recognize letters which even designer has notseen before when the learning process of similar input patterns has beenestablished(massive data practice). In this case new input letter is notnecessary to be learned again and existing neuron network can recognize it.That is so called the nature of all unsupervised learning(teacher less) of theartificial neuron network.

 

And a user interactivefunctionality will be involved in project as a web interface which wouldbriefly do the introduction to software users and support functionality ofdownloading the java software. Since this design is no longer fit in thecurrent software. This one should be totally ignored and the user interactivefunction turns to be a text file contains full user guidance. And this filewould be put in the folder with software. So user do not have to check thewebsite and use a more convenient way to browse the instruction of thesoftware. And considering difference of java running environment betweenwindows and other system, the downloading function is removed as well.

3.1.3 The summary of the researchand analysis carried out so far :

 The reason for using Kohonen SOM neuron network is that after clusteringletters from different fonts, the letter fonts are supposed to be allocated inone specific character. And competitive learning rule applied in Kohonen SOMcould easily achieve the function. Therefore it is suited to detect salientfeatures ,that could be revealed on ensembles of input patterns, and doclassification.

3.2 Design

3.2.1  system design

3.2.1.1 Use-case diagrams

Old diagram is no long fit thedesign. It is displayed in Appendix[4].

The new diagram has been showedbelow. The changes to this diagram make it more mature. Since the software isalready built up .And the understanding of the relationship between the userand project maker is much deeper than before. The reason why i change to thisdiagram is to correct the wrong information(web page) the previous use-casediagram given.

                                                                                                           

 

3.2.1.2 An interaction chart (UMLsequence diagram)

Old interaction chart is availablein Appendix[5]

This section also has beenmodified caused by the same reason stated in section 3.2.1.1.

 

3.2.1.3 The objects to be used inthe system

 

The core object involved in thesystem is a neuron network based on Kohonen SOM. 

It could turn user’s input data,namely the grid below with dark color on several cells to make the grid lookslike a letter A, into Kohonen SOM recognizable input units which are strings ofbinary numbers. So the input pattern unit  index i is in therange 1-49. And input value of units number 4 10 11 12 .....43 49 are supposedto be 1, and rest of units are 0.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

 

And java based Kohonen neuronnetwork simulation system is doing whole calculations as a background program.

So the software interface is theonly platform for communications between the user and the system. User’s interaction like drawing letters , checking outcomes etc couldonly be achieved through the software interface.

The object of the website isremoved.

 

3.2.1.4 Attributes and methods ofobjects UML

      User

 

 

 

   Kohonen SOM simulation in java

inputVector:integer array

weightedInput:double

LearningRate:final integer

Weight:double

getLearningRule(Kohonen SOM)

isInputFieldReturn():Boolean

Learning():void

getInputVector():void

GetLearningResult():void

ReturnResult():integer array

 

 

       Software interface

UserInputButton:object

InputField:object

outputField:object

UserInputButton.start():void

UserInputButton.finish():void

InputField.clean(): void

InputField.return():integer array

isLearningResultReady():Boolean

outputField.display():void

outputField.clean():void

 

 

 

 

 

 

 

 

 

 

         Kohonen SOM

inputVector:integer array

weightedInput:double

LearningRate:final integer

Weight:double

WeightMatrix:double array

setLearningRule()

 

 

 

 

 

 

 

 

 

 

 The table of the website is removed from thisdiagram.

3.2.1.5 Pseudo-code for the keymethods

The old pseudo code is in Appendix[6]. And the new pseudo code is finished in the condition that severalsuccessful tests on the Kohonen SOM neuron network has been done. Therefore itis more robust and precise.

0. set training sets

1. set random initial values forthe weight of connections w

2. for each input pattern

3. calculate the norm of thevector

4. normalize the weights ofconnections

5. check if max|w new - w old|<= 0.01 & then stop and mark the final winner

Else continue

6. assume instant input values aby the input units

7. compute the instant states ofoutput units

S= sum of w*a

8. decide the winner S =max(S1,S2,S3....Sn) only if S_i in the competition is unmarked

9. calculate the instant update ofthe weights of connections

Change of weight = C( a - w)

10. update the weights ofconnections weight vector

11. go to step3

 

 

3.2.1.6 Interface design

Since the web is no longer participatein the project. The interface for the web page is removed and it is declared inthe Appendix[7]

Software Interface design

The graph showed below is how doesthe software interface looks like.

Firstly user could draw the letterin a font by pressing button pen to fill the cell in the grid with dark color.And undo the painted cells by pressing the eraser button and click on the cellthat is wanted to return back its original color. Once drawing is finished.User has to type a name for the font in the text field nearby and click thebutton Apply and name this font.to submit the letter for recognition.

Then the clustered letter with itsfont name will be allocated to corresponding Letter Bar in Clustered Letterstable. User could click on each letter bar to check letter clustering resultand review it by select it and press the button ‘Display selected font’ . 

 

 

 

 

3.2.2  evaluation design

 

3.2.2.1 criteria will be used toevaluate whether the system is successful

 

1. Programs run without errors andthe intervention.

2. Programs run in the way asintended.

3. Programs reach its targetoutputs.

4. If rule 1 is not satisfied.Define which part is working and which is not.

5. Time for program reach the goalis supposed to be rational.

6. Only users attribute input tothe program.

7. Outcome should keep its longterm impact. Close program would not cause loss of data or any results obtainedfrom last run of software.

 

3.2.2.2 how to assess thesecriteria

Observation for criteria 1 2 3 5

Observation and review program for4 6

The observation is based onmassive data practices.

 

3.2.2.3 who will be involved inthe evaluation

Program designer

Non technical user and technicaluser

Project participants

 

3.2.2.4 what testing will becarried out

Unit testing : test each componentof project independently. (same reason website is no longer here)softwareinterface test, Kohonen SOM test, Kohonen SOM simulation system test.

System testing : tests theintegral software. A circle contain testing for successful entry of project ,testing for successful entry of software interface,testing for successful entryof user input,testing for successful entry of review letters database andtesting for correctness of results.

Post test : collect data fromsoftware participant after the intervention.

Pre-test : collect data fromsoftware participant before the intervention.

Software performance testing :focus on different fonts as input see how software clusters them and how doesresults change by different font letters.

 

3.2.2.5 what kind of conclusion doyou expect from evaluation.

Has the project reached the targetgoals as required?

Have the programs run as intended?

Is the software working fine? Ifnot which part goes wrong?

What is the efficiency of thesoftware?

Can only user attribute to changeto software?

Does all data used or generatedduring run of programs keep the long term effect?

 

 

3.2.3Things for a projectinvolving the empirical investigation of some hypothesis

 

3.2.3.1 A statement of thehypotheses to be tested

This project is going to use theKohonen self-organized map to solve the problem. In another word, Kohonen SOMcould go through a learning process to figure out letters in different fontsand cluster them into corresponding letter type.

 

3.2.3.2 A description of the testdata to be used

The data that will be involved inthe test are letters in different fonts,that could be collected from internet,Microsoft Word,any possible users and the project designer. In the case thatdifference of sizes of fonts could affect the precision of software’srecognition process, all fonts would be re-sized to fit the painting grid.

 

3.2.3.3 An experiment design, theexperiments to be performed, any control to be used

Before turns every thing into realcoding,some theory experiments are supposed to be done. Experiment for thedesigned Kohonen self-organized neuron network should compete the practicesabout turn input vector to desired output to see outcome could be obtainedtheoretically.

 

 

3.2.3.4 A description of how theresults will be analyzed, including any statistical techniques that will beused

Once result is available, it issupposed to be allocated into corresponding letter bar. Use common knowledge toanalysis the correctness of instant result.like human make mistakes, so thisartificial neuron network based software is allowed to realize letters in wrongdirections. But statistical technique like make a record about mission successrate is helpful for the evaluation of this project. An example would be addedin the design for more details, a table should be formed as building the firstrow with objects Letter painted, Number of fonts given, Number of fontsrecognized and Rate of recognition in percentage. Then level of the recognitionis available in all type of characters and overall level is easy to becalculated as a important performance evaluation for the software.

3.2.4 New algorithm

3.2.4.1 The approach used in theproject

 The algorithm used in this project is the competitive rule brought fromKohonen SOM neuron network. The output units compete each other to decide awinner by calculate and compare the weighted input namely instant states.Finally winner’s weights of connection are changed to get similar value asinput units have. And 26 template letters representing  the upper-case characters from A to Z aretrained by this algorithm. Therefore only 26 output units are generated duringthe training process and the feature of update weights of connection ofneighbors from Kohonen SOM has not been used in the learning process. After thetraining, the learning process just start. But unfortunately the learningoutcome only occurs on the existing the weight of connections from 26 outputunits. This is actually a really annoying restraint on my system. Since no matterthe outcome of learning do to the template letter, it can only recognize fewletters that similar to him. And if the difference between it and the newletter font is beyond its capability, the recognition will definitely fail. Andin this case,massive data practice has to be made to fit the new font. Hencethis drawback really restricted my system and made it looks really notintelligent and flexible. And the new algorithm based on the approach usedwould over come this easily. 

3.2.4.2 Description of the newalgorithm

 The new algorithm tends to make the system more robust and tolerantunder the change of conditions and varieties of inputs. It is based on the usedalgorithm and extends the algorithm into next level. Therefore there is nodifference between 2 algorithms when the system is training the 26 templateletters. After that ,the learning process begins with mix of the trainingprocess and learning.

Experimental analysis

Once a new font is clustered inthe system, let us make an assumption that it is successfully clustered tocharacter B. For now there is only one output unit in the neuron network thatcan recognize B namely output unit based on template letter. Then the algorithmcompare the winner’s weighted input with total weighted input of the outputunit that recognize the new font. If the difference between weighted input isbig enough then update the weights of connection from output units that belongto character B. And in this case weights of connection from template letter Bare updated, since it is the only output unit that can recognize B so far.Moreover a new output unit is established by training the new font and addingit to the neuron network. If the difference is small enough, the update wouldstill happen to the output units belonging to B. But the new output unit maynot be created. By the assumption, the system allows the new output unit B2 becreated. And the weights of connection from B2 are trained by the same way asold’ algorithm did and using the new font as input pattern. At the end of theassumption, a new font of B is put into the system and recognized by B2. And itis believed that the weighted input of B2 is similar to the one calculated fromcurrent winner. Consequently there is no new output unit generated. And updateare implemented to the output units that belongs to character B from high tolow in term of the weighted input with decreasing learning rate. So theincrease of weights of connection in B2 is large in comparison with B(templateletter).

Mathematical analysis

 If a output unit has gone through sufficient learning process, itsweights of connection is robust and tolerant in conditions. In this case, weassume 5 fonts could be recognized successfully. As the drawback of the usedalgorithm stated in the section 3.2.4.1, the low number is caused by theserious restraint found in the used algorithm. In other word, even thesophisticated weights of connections could only store the information that letit capable to recognize a limited number of letters.Since this drawback iseliminated by the method of creating more output units in their belongscharacter, the information could be stored in weights of connections isunlimited. And the number of letters which could be recognized ismathematically infinite. Hence the much more larger percentage of success ratein recognition could be given by use of new algorithm.

4.Realization

 4.1 Description of the implementation of thedesign

 In the sequence of the design, the things that are introduced in thebeginning of the design are use-case diagram , the interaction chart andattributes, methods of objects UML. Those information are just auxiliary toolsthat assist project maker to implement the design in the right direction. Thatmeans apart from things mentioned above, the rest of section in the designcould be turned into practice namely implementation.

Pseudo code

 And first one goes to the pseudo code for the key method. In myrealization of the pseudo code, it has been both used in the establishment ofKohonen SOM neuron network and the build of Java code. Logically the former issupposed to be structured first.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The practical Kohonen SOM neuronnetwork is built based on the process declared in the pseudo code step by step.And the data of the pseudo code are changed by values of real numbers such aslearning rate in 0.1 and the value of convergence in 0.01. In order to reallystart the practice of neuron network, the initial weights of connections arepicked randomly between 0 to 1. Since the purpose of making the neuron networkis for the proof that clustering letters in different fonts is practical wherea Kohonen SOM neuron network is in charge to achieve this function. Thereforeall this neuron network has to show is that it is possible to cluster lettersin different font based on the current using pseudo code. And to shorten thetime for make that proof, only a simple neuron network model would be produced.In the model of neuron network, only 2 characters and a 3 x 3 input grid areinvolved. More specifically the two tables showed below illustrate how the gridlooks like and the two characters are A and C. The values of first inputpattern from a1to a9 is 0 1 0 1 1 1 1 0 1. And second input pattern is 0 1 1 10 0 0 1 1.

Once the input pattern and weightof connection are ready. The Kohonen SOM neuron network could work exactly sameas the way defined in pseudo code. The full hand writing logs for the practiceof Kohonen SOM neuron network simple model are available in Appendix [8]. Fromthe bottom of the third picture in Appendix 8, the model realize that pattern 1is the second set of weights of connections and pattern2 is the first set ofweights of connection. Since the weights of connections are randomly selectedfrom initial state. The order for sets of weights of connection will not besame as the order that input pattern are put in for most cases. So the resultshown in fourth picture is correct. Since p3 is recognized by s2(output unit 2)which could recognize pattern1.

 Once finished the establishment of the simple model of neuron network,its simulation system on java is ready to start. As the way used in the modelof neuron network, each step from pseudo code could be achieved by using javamethods. Hence methods Normalization(),UpdateWeightsOfConnections(),Initialization(),stopCondition(),DecideWinner()and saveWeights() are created.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix

Terminologyin used

[1]SOM : self-organizing maps

 

[3] :Theaim of this project is to write a program to implement a Kohonenself-organizing map neural network, and intend to achieve the functionclustering letters from different fonts. To get good understanding ofobjectives of project , the aim and even the answer of difference betweencurrent existing letters recognition software, background information aboutunsupervised learning(Hebb’s rule,Habbian’s rule in Normalization rule,Kohonen SOM) and supervisedlearning(Perceptron) are highly needed.

Hebb’s rule(1949)could be introduced by a neroscience postulate:”Whenan axon of cell A is near enough toexcite a cell B and repeatedly andpersistently takes part in firing it, some growth process or metabolic changetakes place in one or both cells,such that A’sefficiency as one of the cells firing B,is increased.”.[1]And formalization of Hebb’s rule could be the increaseof weight of connection in next time stamp: w^t+1= w^t +∆w^t ;∆w^t= C*a* X^t  where w is weight of connection ,w^t+1  is the weight of next instant,w^tis the instant weight,C is positive coefficient that  represent learning rate,a is the value ofinstant input, X^t is instant output . From the formula weight couldbe changed only if both input value and output value are not equal to 0 at sametime. This shows the correlation nature of Habbian’srule.

Hebb’s rule inNormalization rule is based on Hebb’s rule apparently.But the sum of weight is kept as a constant value. This could cause thestrength of synapse tends to sharpen a neuron’spredisposition and make its firing get better correlated with a cluster ofstimulus patter, stated by Biktasheva.[2]

Kohonen rule is also based on Hebb’s rule. It would apply competitive learning to make sure only oneoutput is fired at every instant when there are several output units fired atsame time stamp.

Rosenblatt(1958) claimed asupervised pattern recognition that neural networks change with experiencecontaining a whole process of setting input,getting output and calculation oferror from output and target output.[3] It is also called error-correctionrule. From the rule , weights of connection in neural network change only iferroneous feedback from that neuron is found.

[4] :

 

 

[5]: :

[6] :

0. Setinput pattern

1. Setthe neurons threshold value θ_{j,j=1......26.}

2. Setrandom initial values for the weights of connections

3. Definethe normalization of the vector  :  ||||=

4. Normalizethe weights of connections : ()_new =()_old

5. Checkif max| - | <= stop constant=0.01 then stop else continue

6. Assumeinstant input values by the input patterns.

7. Computethe instant state of the neuron =

8. Findthe =max(.....)

9. Computethe instant corrections to the weights of connections ∆=C( -),i=1....n;

10. Update the weights ofconnections =+ ∆

11.if the this is the last pair ofinput patterns go to step 3 else go to step 6

 

[7]:

Interface for the web page

Website design would be verysimple and straightforward as showed below. Header is the project name. Andonly 3 functions are accessible.To know the project option will show a briefintroduction of the project on TEXT FIELD which will keep clean until buttonpressed. Same thing would happen on User manual but content in TEXT FIELD wouldbe detailed instructions for use of software. And press Download the softwarebutton will directly go to downloading process.

 

[8]: Hand writing Kohonen SOMtemplate logs