K-Means Clustering

We begin by first creating a simulated dataset with two distinct seperate groups.

set.seed(2)
# Create a 50 x 2 matrix using a random normal distribution
x=matrix(rnorm(50*2), ncol=2) 
# take the first 25 vlaues of the first column and add three 
x[1:25,1]=x[1:25,1]+3
# take the first 25 vlaues of the first column and subtract four
x[1:25,2]=x[1:25,2]-4

Run k-means where k = 2.

km.out=kmeans(x,2,nstart=20)
km.out$cluster
##  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2
## [36] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
plot(x, col=(km.out$cluster+1), main="K-Means Clustering Results with K=2", xlab="", ylab="", pch=20, cex=2)

We know that there are two distinct groups because we created the data, however in the real world we will almost never know the real number of clusters. Below is the same K-Means model with three clusters instead of two.

set.seed(4)
km.out=kmeans(x,3,nstart=20)
km.out
## K-means clustering with 3 clusters of sizes 17, 23, 10
## 
## Cluster means:
##         [,1]        [,2]
## 1  3.7789567 -4.56200798
## 2 -0.3820397 -0.08740753
## 3  2.3001545 -2.69622023
## 
## Clustering vector:
##  [1] 1 3 1 3 1 1 1 3 1 3 1 3 1 3 1 3 1 1 1 1 1 3 1 1 1 2 2 2 2 2 2 2 2 2 2
## [36] 2 2 2 2 2 2 2 2 3 2 3 2 2 2 2
## 
## Within cluster sum of squares by cluster:
## [1] 25.74089 52.67700 19.56137
##  (between_SS / total_SS =  79.3 %)
## 
## Available components:
## 
## [1] "cluster"      "centers"      "totss"        "withinss"    
## [5] "tot.withinss" "betweenss"    "size"         "iter"        
## [9] "ifault"
plot(x, col=(km.out$cluster+1), main="K-Means Clustering Results with K=3", xlab="", ylab="", pch=20, cex=2)

Nstart controls the number of random cluster assignments. When we assign a value greater than 1 to nstart R will utilize multiple random assignments and will return the version with the best performance. Below you can see that the kmeans with an nstart of 20 has a slighlty better within-clustter sum of squares.

set.seed(21)
km.out=kmeans(x,3,nstart=1)
km.out$tot.withinss
## [1] 98.16736
km.out=kmeans(x,3,nstart=20)
km.out$tot.withinss
## [1] 97.97927

Hierarchical Clustering

hc.complete=hclust(dist(x), method="complete") # computes all pairwise dissimilarities and records the largest 
hc.average=hclust(dist(x), method="average") # computes all pairwise dissimilarities and records the average 
hc.single=hclust(dist(x), method="single")# computes all pairwise dissimilarities and records the smallest
par(mfrow=c(1,3))
plot(hc.complete,main="Complete Linkage", xlab="", sub="", cex=.9)
plot(hc.average, main="Average Linkage", xlab="", sub="", cex=.9)
plot(hc.single, main="Single Linkage", xlab="", sub="", cex=.9)

The code below cuts a dendrogram tree into its groups based on the desired number of clusters.

cutree(hc.complete, 2)
##  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2
## [36] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
cutree(hc.average, 2)
##  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 1 2 2
## [36] 2 2 2 2 2 2 2 2 1 2 1 2 2 2 2
cutree(hc.single, 2)
##  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
## [36] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
cutree(hc.single, 4)
##  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3
## [36] 3 3 3 3 3 3 4 3 3 3 3 3 3 3 3
xsc=scale(x)
plot(hclust(dist(xsc), method="complete"), main="Hierarchical Clustering with Scaled Features")

x=matrix(rnorm(30*3), ncol=3)
dd=as.dist(1-cor(t(x)))
plot(hclust(dd, method="complete"), main="Complete Linkage with Correlation-Based Distance", xlab="", sub="")