Support vector machines¶

This example covers Support Vector Machines (SVMs) and two ensemble methods: boosting (via gradient boosting in the gbm package) and random forests in the randomForest package.

Support Vector Machines¶

The default kernel for SVM is radial. In this example, we will use a linear kernel, following up later with a radial kernel. Read the help for svm to find out what kinds of kernels one can use, as well as the parameters of the kernels.

    library(e1071)
    iris = read.table("http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data",
        sep = ",", header = FALSE)
    names(iris) = c("sepal.length", "sepal.width", "petal.length", "petal.width",
        "iris.type")
    model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel = "linear")
    model

    ##
    ## Call:
    ## svm(formula = iris.type ~ sepal.length + sepal.width, data = iris,
    ##     kernel = "linear")
    ##
    ##
    ## Parameters:
    ##    SVM-Type:  C-classification
    ##  SVM-Kernel:  linear
    ##        cost:  1
    ##       gamma:  0.5
    ##
    ## Number of Support Vectors:  77

In[55]:

def svm_iris_linear():
    grid = np.mgrid[3.5:8:500j,2:5:400j]
    gridT = grid.reshape((2,-1)).T
    %R -i gridT colnames(gridT) = c('sepal.length', 'sepal.width')
    %R model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel='linear')
    %R gridT = data.frame(gridT); labels = predict(model, gridT)
    %R -o labels -d iris

    plt.imshow(labels.reshape((500,400)).T, interpolation='nearest', origin='lower', alpha=0.4, extent=[3.5,8,2,5], cmap=pylab.cm.RdYlGn)
    plt.scatter(iris['sepal.length'], iris['sepal.width'], c=[col[t] for t in iris['iris.type']])
    a = plt.gca()
    a.set_xlim((3.5,8))
    a.set_ylim((2,5))

svm_iris_linear()

<matplotlib.figure.Figure at 0x86cc290>

We can see which vectors are support vectors using a plot: the + points are support vectors.

    model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel = "linear")
    plot(iris$sepal.length, iris$sepal.width, col = as.integer(iris[, 5]), pch = c("o",
        "+")[1:150 %in% model$index + 1], cex = 2, xlab = "Sepal length", ylab = "Sepal width")

There is a plot method for SVMs that makes it possible to see the decision boundaries.

    plot(model, iris, sepal.width ~ sepal.length, slice = list(sepal.width = 1,
        sepal.length = 2))

Nonlinear kernel

    model = svm(iris.type ~ sepal.length + sepal.width, data = iris)
    model

    ##
    ## Call:
    ## svm(formula = iris.type ~ sepal.length + sepal.width, data = iris)
    ##
    ##
    ## Parameters:
    ##    SVM-Type:  C-classification
    ##  SVM-Kernel:  radial
    ##        cost:  1
    ##       gamma:  0.5
    ##
    ## Number of Support Vectors:  86

    plot(model, iris, sepal.width ~ sepal.length, slice = list(sepal.width = 1,
        sepal.length = 2))

In[59]:

def svm_iris_kernel(gamma=0.5):
    grid = np.mgrid[3.5:8:500j,2:5:400j]
    gridT = grid.reshape((2,-1)).T
    %R -i gridT,gamma colnames(gridT) = c('sepal.length', 'sepal.width')
    %R model = svm(iris.type ~ sepal.length + sepal.width, data = iris, gamma=gamma)
    %R gridT = data.frame(gridT); labels = predict(model, gridT)
    %R -o labels -d iris

    plt.imshow(labels.reshape((500,400)).T, interpolation='nearest', origin='lower', alpha=0.4, extent=[3.5,8,2,5], cmap=pylab.cm.RdYlGn)
    plt.scatter(iris['sepal.length'], iris['sepal.width'], c=[col[t] for t in iris['iris.type']])
    a = plt.gca()
    a.set_xlim((3.5,8))
    a.set_ylim((2,5))

svm_iris_kernel()

<matplotlib.figure.Figure at 0x86cfbb0>

Nonlinear kernels also have support vectors.

    model = svm(iris.type ~ sepal.length + sepal.width, data = iris)
    plot(iris$sepal.length, iris$sepal.width, col = as.integer(iris[, 5]), pch = c("o",
        "+")[1:150 %in% model$index + 1], cex = 2, xlab = "Sepal length", ylab = "Sepal width")

The parameter $\gamma$ controls the bandwidth of the kernel.

In[61]:

svm_iris_kernel(gamma=3)

<matplotlib.figure.Figure at 0xace76d0>

In[62]:

def svm_iris_polynomial(gamma=0.5, degree=4):
    grid = np.mgrid[3.5:8:500j,2:5:400j]
    gridT = grid.reshape((2,-1)).T
    %R -i gridT,gamma,degree colnames(gridT) = c('sepal.length', 'sepal.width')
    %R model = svm(iris.type ~ sepal.length + sepal.width, data = iris, gamma=gamma, degree=degree, kernel='polynomial')
    %R gridT = data.frame(gridT); labels = predict(model, gridT)
    %R -o labels -d iris

    plt.imshow(labels.reshape((500,400)).T, interpolation='nearest', origin='lower', alpha=0.4, extent=[3.5,8,2,5], cmap=pylab.cm.RdYlGn)
    plt.scatter(iris['sepal.length'], iris['sepal.width'], c=[col[t] for t in iris['iris.type']])
    a = plt.gca()
    a.set_xlim((3.5,8))
    a.set_ylim((2,5))

svm_iris_polynomial(degree=2)

<matplotlib.figure.Figure at 0xace4e90>

    model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel = "polynomial",
        degree = 2, gamma = 0.5)
    plot(model, iris, sepal.width ~ sepal.length, slice = list(sepal.width = 1,
        sepal.length = 2))

In[64]:

svm_iris_polynomial(degree=3)

<matplotlib.figure.Figure at 0xad0db50>

In[65]:

svm_iris_polynomial(degree=4)

<matplotlib.figure.Figure at 0xad10af0>

Table Of Contents

Previous topic

Next topic

This Page

Support vector machines¶