DATA MINING(1) Data mining introduction

Note: this post concludes the content of lecture for COMP 9318 - week 01, School of Computer Science and Engineering, UNSW (s1 - 2017)

Data mining

1. Why we need data warehouse and Data mining?

problems:

1. Data explosion 

2. Drowning in data, but starving for knowledge.

So:

we need data warehouse (to do on-line analytical processing) and data mining (to mining interesting knowledge).

Generally speaking, we need data warehouse and data mining to do data analysis and give a decision support.

2. What is data mining?

Knowledge discovery from data.

Extract of interesting(non-trivial, implicit, previously, unknown,potentially useful) patterns of knowledge from huge amount of data.

3. Where can we use it?

Market analysis andmanagement

Risk analysis andmanagement

DNA and bio-dataanalysis

Text mining and webmining

Stream data mining.

DM vs. KDD

Data mining is one among the steps of KDD (knowledge discovery in databases). it is also the core of KDD process.

Step of KDD process:

1. Learn basic knoledge and select data from dàtabase

2. Data cleaning and preprocessing -> to get a data warehouse, this part may take 60% effort of the whole work.

3. Data reduction andtransformation (find features, dimensionality..)

      -> get relevant data from data warehouse

4. Choose functions and mining algorithm (regression, classification, clustering, association)

      -> this is data mining process, the goal is to get patterns

5. Pattern evaluation and knowledge presentation 

   -> visualization, transformation, removing redundant patterns, (to use)

Data mining functionalities:

discrimination:  generalize, summarize and contrast data characteristics.(总结特征)

Association: e.g., Diaper-> Beer  (寻找关联性)

classification and prediction:

construct models, presentation, clustering, outlier and trend analysis.