Data-mining

Data mining applies statistics and pattern recognizion to large amounts of data from the past to discover (potentially) interesting information and knowledge for future actions.

Problem: separate signal from noise.

The data mining process is also known as knowledge discovery. It is not enough, just to find interesting, previously unknown, patterns in the data, the process must also show what a company has to do.

Data mining bridges the gap between artificial intelligence and databases.

Techniques and tasks

• Anomaly (or outlier) detection (outliers, change and deviation detection)
• Associate rule learning: try to find relationships between variables
• Classification and prediction
• Clustering (Discover groups and structers in the data)
• Affinity grouping/association rules
• Sequence analysis
• Regression: try to find a function that models the data with the least error.
• Time series analysis
• Text mining
• Summarization

New techniques:

• Social network analysis
• Sentiment analysis (aka opinion mining or emotion AI)

A datamining process can be broken into six major phases (cf CRISP-DM)