Chapter 1: History

Unsupervised machine learning, a subfield of artificial intelligence, aims to discover patterns and structure in data without using labeled responses Labeled responses: Data points that have known outcomes or categories. For example, in a dataset of animal images, each image might be labeled as "cat" or "dog." Unsupervised learning does not use these labels. . Its origins trace back to the early work in statistics and pattern recognition Pattern recognition: The process of classifying data based on statistical information extracted from patterns in the data. It involves identifying regularities, trends, or structures within datasets. in the mid-20th century.

• 1950s–1960s: Early clustering Clustering: The task of grouping a set of data points into clusters so that points in the same cluster are more similar to each other than to those in other clusters. methods (e.g., k-means developed by Stuart Lloyd in 1957) and principal component analysis Principal Component Analysis (PCA): A technique used to reduce the number of variables in a dataset by transforming the original variables into a new set of uncorrelated variables (principal components) that retain most of the variance in the data. (PCA) by Karl Pearson (1901) and Harold Hotelling (1933) laid the groundwork.
• 1970s–1980s: Emergence of algorithms like hierarchical clustering Hierarchical clustering: A method of cluster analysis which seeks to build a hierarchy of clusters, either by merging smaller clusters into larger ones (agglomerative) or splitting larger clusters into smaller ones (divisive). and self-organizing maps Self-organizing maps (SOMs): A type of artificial neural network used for clustering and visualizing high-dimensional data by mapping it onto a 2D grid in a way that preserves topological relationships. (Teuvo Kohonen, 1982).
• 1990s–2000s: Advances in density-based clustering Density-based clustering: A clustering approach that groups together data points that are closely packed (dense regions), marking points in sparse regions as outliers. DBSCAN is a popular density-based clustering algorithm. (DBSCAN, 1996), Gaussian mixture models Gaussian mixture models (GMM): Probabilistic models that assume data points are generated from a mixture of several Gaussian distributions with unknown parameters, often used for clustering and density estimation.
  Example: Imagine recording the heights of all people at a theme park, which includes both children and adults. The overall height distribution forms two overlapping "bell curves" (Gaussians), one for children and one for adults. A Gaussian Mixture Model can automatically discover these two groups by modeling the data as a combination of two Gaussian distributions.

  Illustration: The sum of two Gaussian curves (for children and adults) models the overall distribution.
  , and manifold learning Manifold learning: A set of techniques for reducing the dimensionality of data based on the idea that high-dimensional data often lies on a lower-dimensional manifold embedded within the higher-dimensional space. .
  Example: Imagine you have a long, curly ribbon on a table. If you look down from above, it looks like a twisty, two-dimensional shape. Manifold learning helps "unroll" this ribbon into a straight line so you can study its structure more simply.
  

  Illustration: The curly "S" shape (left, high-dim) is mapped to a straight line (right, low-dim).
• 2010s–Present: Unsupervised deep learning Deep learning: A subset of machine learning using neural networks with many layers (deep neural networks) to learn hierarchical representations of data. In unsupervised settings, it includes models like autoencoders and GANs. (autoencoders, GANs) and applications in big data Big data: Extremely large and complex datasets that require advanced methods and technologies to store, process, and analyze. Unsupervised learning helps to find patterns in big data. , image, and text analysis.
  Example: Deep learning models can learn to represent handwritten digits in layers. Early layers might detect simple features like lines or curves; deeper layers combine these to form digit shapes. This allows the model to learn complex structures by stacking many simple ones.

  Illustration: Input features are transformed through multiple hidden layers, creating increasingly complex representations.

Chapter 2: Major Contributors

• Karl Pearson — Introduced Principal Component Analysis (PCA) in 1901.
• Stuart Lloyd — Developed the Lloyd's algorithm (k-means) in 1957.
  Illustration: k-means clustering finds group centers (stars) and assigns points (dots) to the nearest group.
• Teuvo Kohonen — Creator of Self-Organizing Maps (SOMs) in 1982.
• Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu — Developed DBSCAN in 1996.
  Illustration: DBSCAN groups dense regions and marks outliers (crosses) as noise.
• Geoffrey Hinton — Advanced unsupervised deep learning via autoencoders and deep belief nets.
  Illustration: Autoencoder compresses (encodes) data to a bottleneck and reconstructs (decodes) it back.
  Illustration: Deep Belief Net stacks multiple restricted Boltzmann machines for hierarchical feature learning.

  Step 1: Input Data

  The DBN receives raw input data (e.g., pixel values of an image) at the input layer.

  Prev Next

Chapter 3: Major Algorithms

1. k-means Clustering
   Partitions data into k clusters by minimizing within-cluster variance.
2. Hierarchical Clustering
   Builds a hierarchy of clusters using agglomerative or divisive approaches.
3. Principal Component Analysis (PCA)
   Reduces data dimensionality by finding new axes (principal components) that retain most of the data variance.
4. DBSCAN (Density-Based Spatial Clustering of Applications with Noise)
   Groups data points that are closely packed together, marking outliers as noise.
5. Gaussian Mixture Models (GMM)
   Models data as a mixture of multiple Gaussian distributions.
6. Autoencoders
   Neural networks that learn to encode data efficiently, mainly used for dimensionality reduction or feature learning.
7. Visualizing Colors with Self-Organizing Maps (SOM)

   How Self-Organizing Maps (SOM) Work

   1. Initialize a 2D grid of "neurons" (each with a weight vector, e.g. RGB color).
   2. For each data point (e.g., a color), find the "Best Matching Unit" (BMU) — the neuron whose weight is closest to the data point.
   3. Update the BMU and its neighbors to move closer to the data point (using a learning rate and neighborhood function).
   4. Repeat for many iterations, gradually reducing the learning rate and neighborhood size.
   5. Result: Similar colors are mapped to nearby neurons on the grid, forming smooth color regions.

   Let's see a simple example:

   Step 1: Fake Color Dataset
   We'll use 25 random colors (as RGB triples) to mimic a palette:

   [ [237, 50, 55], [ 40, 230, 90], ... , [ 55, 65, 210] ]

   Step 2: Organize on a 2D Grid
   We'll lay out a 5×5 grid (25 neurons), and use a simple SOM algorithm so that each neuron learns to represent one of the input colors. Neighboring neurons on the grid will end up with similar colors.

   Step 3: Result
   The grid below shows how the SOM has organized the original colors into a smooth map—colors that are similar are close together on the grid!

   Each cell shows a learned color.
   Neighboring cells have similar colors, revealing natural groupings.

Chapter 4: Easy-to-Understand Examples

Machine Learning Concepts Quiz