What is forward-mode differentiation?

Forward-mode differentiation is a method used in automatic differentiation. It is a technique that computes numerical derivatives by simultaneously performing elementary derivative operations while evaluating the function. The chain rule is used to update the derivative values at each step.

Let's now understand this concept with an example.

Example

In order to break down functions into elementary steps, evaluation traces are constructed. These traces can be thought of as a record of the individual steps taken to obtain the final results. Let's take the following function as an example:

$f(x, y) = cos(x) + x(e^y)$

To construct the evaluation trace, we will substitute some variables inside the function at each step. To start with, let's substitute $x$with $w_1$ and $y$ with $w_2$ in the above equation.

The equation becomes:

$f(x, y) = cos(w_1) + w_1(e^{w_2})$

Let's now substitute $cos(w_1)$ with $w_3$ and $e^{w_2}$ with $w_4$ in the above equation.

The equation now becomes:

$f(x, y) = w_3 + w_1.w_4$

Finally, we substitute $w_1.w_4$with $w_5$in the above equation and the equation then becomes:

$f(x, y) = w_3 + w_5$

Let’s now evaluate the function when $x = \frac{\pi}{2}$ and $y =1$, and record all the intermediate values in the table below.

Setting the initial conditions

Let's set the initial conditions for the derivatives:

• $w_1'$: $\frac{\partial{w_1}}{\partial{x}} =1$

• $w_2'$: $\frac{\partial{w_2}}{\partial{x}} =0$

By setting the seed values for the derivatives of the variables (in this case, $w_1'$ and $w_2'$), we establish the starting point for the differentiation process. These initial conditions act as the base values from which the derivatives will be computed and propagated forward through the computational graph.

Computing the partial derivative

Let's suppose we want to compute the partial derivative of $y$ with respect to $x$, with $x = \frac{\pi}{2}$ and $y = 1$. We can approach this task by considering one intermediate variable at a time. It's important to note that we are focusing solely on the numerical value of the derivative. For each $w_i$, we calculate $\frac{\partial{w_i}}{\partial{x}}$.

Let’s try to calculate the partial derivative of $w_3$.

Note: We will be using the following two expressions to represent the partial derivative of $w_3$:$w_3'$ or $\frac{\partial{w_3}}{\partial{x}}$.

$\frac{\partial{w_3}}{\partial{x}} = \frac{\partial{cos(w_1)}}{\partial{x}}$

$\frac{\partial{w_3}}{\partial{x}} = -w_1'.sin(w_1)$

substituting $w_1$ with $x$

$\frac{\partial{w_3}}{\partial{x}} = -\frac{\partial{x}}{\partial{x}}.sin(x)$

$\frac{\partial{w_3}}{\partial{x}} = -1sin(x)$

substituting $x$ with $\frac{\pi}{2}$

$\frac{\partial{w_3}}{\partial{x}} = -1sin(\frac{\pi}{2})$

$\frac{\partial{w_3}}{\partial{x}} = -1$

The results of the partial derivatives of $w_4$, $w_5$, and $w_6$ are provided in the table below:

For each intermediate variable, we calculate its derivative by applying derivative rules. Remember that the value of each intermediate variable depends only on the derivatives and values of previous variables.