Multi-class classification is not supported by predictive classification models. Perceptron, logistic regression, and support vector machine algorithms were intended for binary classification and do not natively support classification problems with more than two classes.
Splitting the multi-class classification dataset into numerous binary classification datasets and fitting a binary classification model to each is one way of employing binary classification algorithms for multi-classification problems. There are two instances of this strategy:
One-vs-Rest (OvR)
One-vs-One (OvO)
Both of these strategies for multi-class classification are elaborated below.
One-vs-rest is a heuristic method for applying binary classification algorithms for multi-class classification. It entails dividing the multi-class dataset into numerous binary classification problems. After training a binary classifier on each binary classification task, predictions are generated using the most confident model.
This method mandates that each model predicts a class membership probability or a probability-like score. The argmax of these scores is then used to forecast a class.
Given a multi-class classification problem with examples for each class "Iris Versicolor," "Iris Virginica," and "Iris Setosa." The following three binary classification datasets are generated from this:
Binary classification problem 1: Iris Versicolor vs. [Iris Virginica, Iris Setosa]
Binary classification problem 2: Iris Setosa vs. [Iris Versicolor, Iris Virginica]
Binary classification problem 3: Iris Virginica vs. [Iris Versicolor, Iris Setosa]
The following illustration helps to dive into the details of OvR:
One drawback of this approach is that it necessitates the creation of one model for each class. For instance, three classes necessitate three models. This could be a problem with massive datasets (millions of rows), slow models (such as neural networks), or a considerable number of classes (such as hundreds of classes).
This method is frequently used for algorithms that predict numerical class membership probability or score naturally, such as the following:
Logistic Regression
Perceptron
As a result, when employing these algorithms for multi-class classification, the Scikit-learn library's implementation of these algorithms uses the OvR method by default.
One-vs-one is another heuristic method for employing binary classification algorithms for multi-class classification. One-vs-one divides a multi-class classification dataset into binary classification problems.
Unlike one-vs-rest, which divides the dataset into one binary dataset for each class, one-vs-one divides the dataset into one dataset for each class vs. every other class. Likewise, if the binary classification models predict a numerical class membership, such as probability, the argmax of the sum of the scores is predicted as the class label.
Consider the multi-class classification problem with three classes "Iris Versicolor," "Iris Virginica," and "Iris Setosa." Three binary classification datasets are generated from this, as follows:
Binary classification problem 1: Iris Versicolor vs. Iris Virginica
Binary classification problem 2: Iris Versicolor vs. Iris Setosa
Binary classification problem 3: Iris Virginica vs. Iris Setosa
Compared to the one-vs-rest approach mentioned in the preceding section, this has many datasets. The following is the formula for computing the number of binary datasets:
We can see that this gives us the expected value of three binary classification problems for three classes.
The following illustration helps to comprehend the concept of OvO:
This method is traditionally recommended for support vector machines and other kernel-based algorithms. This is because the performance of kernel approaches does not scale in proportion to the size of the training dataset, and employing subsets of the training data may mitigate this effect.
The SVC class in Scikit-learn provides the support vector machine implementation, which enables the one-vs-one approach for multi-class classification problems.
Free Resources