Parametric vs non-parametric models

Parametric vs. non-parametric models

Among the vast possibilities of classifying machine learning models, there exists the divide between parametric and non-parametric models.

Before we can delve into this divide, we must understand what unites all of these models. At their core, every machine learning model attempts to compute a function, f, that is going to take the inputs $x$ and map them to the outputs $y$. This relationship can be summarized as follows:

Contained within the function $f$ are what we call the model's parametersParameters are "configuration variables" that are intrinsic to a model, and are learned entirely from the training data., and figuring them out is what helps us accurately map the inputs $x$ to the outputs $y$. As their names suggest, the divide between the parametric and non-parametric models, therefore, is attributed to the nature of their parameters.

Parametric models

Parametric models are those models that are able to figure out the aforementioned function $f$, which contains a finite and fixed set of parameters, irrespective of the number of training instances. This means that changing the size of the input $x$ is going to have no impact on the number of parameters needed.

In order to make this happen, parametric models attempt to find a mathematical model that can be used to approximate the general mapping from the inputs $x$ to the outputs $y$. Once a mathematical model is found, the training data is then used to find the coefficients (parameters) that will result in the most valid and accurate mapping.

Note: Unfortunately, this methodology works best when the input data is well-defined and predictable.

Here are a few parametric models:

• Linear Regression

• Logistic Regression

• Artificial Neural Networks

Linear Regression

Linear Regression models map the inputs to the outputs using the following generalized linear equation:

Here, $\beta_0$ and $\beta_1$(note how they're finite and fixed in number) act as the model's parameters, which are then used to make the model's predictions.

Note: Click here to learn more about linear regression.

Non-parametric models

Now that we're acquainted with parametric models and the finite number of parameters they have, it would follow that non-parametric models have no parameters, right? Wrong. The term non-parametric actually implies that the number of parameters is flexible and entirely dependent on the training data.

Having a flexible number of parameters that vary with the training data results in a model that functions without having to approximate a mathematical model (unlike linear regression); therefore, these models are immensely flexible as they minimize any assumptions that are made about the data.

It's due to this that non-parametric models are ideal when the data at hand is either too complex or isn't well-defined. However, not approximating a mathematical model does come with its drawbacks. Non-parametric models are more computationally expensive than parametric models, and also get increasingly complex as the amount of data increases (since tracing the effects of parameters becomes more difficult).

Here are a few non-parametric models:

• k-Nearest-Neighbors (kNN)

• Decision Trees

k-Nearest-Neighbors (kNN)

To make a prediction in kNN models, we use the classes of the test instance's k nearest neighbors. As aforementioned, it is the parameters that enable us to make predictions for all models, so we can see that increasing the training instances will thereby lead to an increase in the number of parameters.

Note: Click here to learn more about kNNs.

Comparison

The table below summarizes the comparison between parametric and non-parametric models: