Linear Regression

Welcome To The World Of Machine Learning

Lets Explore The Linear Regression Model with an example

What is Linear Regression

Linear regression is a supervised machine learning technique where we train a ML algorithm on labeled datasets. Basically, it is a statistical model that identifies the relationship between independent and dependent variables. Here, we predict the target value based on input variables. The linear regression model finds the best-fit line for our model.

Hypothesis For Linear Regression:
\( y = m \times x + c \)

\(y\) : output variable
\(x\) : input variable
\(m\): gradient
\(c\) : bias

Maths Behind Regression

• Forward Propagation

When we train the model, it fits the best-fit line to predict the value of y for a given value of x, by determining the values of m and c. Once we find the optimal values of m and c, we obtain the best-fit line, enabling us to use our model for prediction.

assuming the hypothesis \( y = m \times x + c \)

Here Initially m and c values are randomly assigned

• Optimizing Model Parameters: Deriving the Gradient Descent Updates through Cost Function Differentiation

Cost Function:
The cost function quantifies the error between predicted and true values, aiming to minimize this difference. Mathematically, it is represented as the root mean square error (RMSE) between predicted and true values:

\( \text{cost function} = \frac{{\sum(\text{pred}(i) - y(i))^2}}{2n} \)

cost function differentiation:

\( \frac{\partial(\text{Cost Function})}{\partial(m)} = \frac{1}{n} \sum((\text{pred}(i) - y(i)) \times x(i)) \)
\( \frac{\partial(\text{Cost Function})}{\partial(c)} = \frac{1}{n} \sum(\text{pred}(i) - y(i)) \)

• Gradient descent And Parameter Updation

In gradient descent, the model strives to minimize the cost function by iteratively updating the parameters m and c. Initially, these parameters are randomly initialized and updated iteratively until optimal values are found. Through this process, the model achieves the minimum cost function with optimal values of m and c.

\( m' = m - \alpha \frac{\partial(\text{Cost Function})}{\partial(m)} \)
\( c' = c - \alpha \frac{\partial(\text{Cost Function})}{\partial(c)} \)

Here, \( \alpha \) represents the learning rate, determining the step size in the parameter space during optimization.

Code: Algorithm Implementation in Python


import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Load the dataset
data = pd.read_csv("data_for_lr.csv")

# Split data into training and testing sets
train_input = np.array(data.x[:500]).reshape(500,1)
train_output = np.array(data.y[:500]).reshape(500,1)
test_input = np.array(data.x[500:700]).reshape(199,1)
test_output = np.array(data.y[500:699]).reshape(199,1)

def forwardPropagation(train_input, parameters):
    m = parameters['m']
    c = parameters['c']
    prediction = np.multiply(m, train_input) + c
    return prediction

# Cost function to evaluate the error between predictions and actual values
def costFunction(prediction, train_output):
    cost = np.mean((train_output - prediction) ** 2) * 0.5
    return cost

# Backward propagation function to compute derivatives
def backwardPropagation(train_input, train_output, prediction):
    derivatives = dict()
    df = prediction - train_output
    dm = np.mean(np.multiply(df, train_input))
    dc = np.mean(df)
    derivatives['dm'] = dm
    derivatives['dc'] = dc
    return derivatives

# Function to update parameters using gradient descent
def updateParameters(parameters, derivatives, learning_rate):
    parameters['m'] = parameters['m'] - learning_rate * derivatives['dm']
    parameters['c'] = parameters['c'] - learning_rate * derivatives['dc']
    return parameters

# Train the linear regression model
def train(train_input, train_output, learning_rate, iters):
    # Initialize parameters with random values
    parameters = dict()
    parameters['m'] = np.random.uniform(0, 1)
    parameters['c'] = np.random.uniform(0, 1)

    # List to store loss values for each iteration
    loss = list()

    # Iterate over specified number of iterations
    for i in range(iters):
        # Perform forward propagation
        prediction = forwardPropagation(train_input, parameters)
        # Compute cost
        cost = costFunction(prediction, train_output)
        loss.append(cost)
        print(f'Iterations : {i+1}, loss : {cost}')

        # Plot training data and predictions
        plt.figure()
        plt.plot(train_input, train_output, '+', label='Original')
        plt.plot(train_input, prediction, '-', label='Training')
        plt.legend()
        plt.show()

        # Perform backward propagation
        derivatives = backwardPropagation(train_input, train_output, prediction)
        # Update parameters
        parameters = updateParameters(parameters, derivatives, learning_rate)

    return parameters, loss

# Perform training
parameters, loss = train(train_input, train_output, 0.0001, 10)

# Make predictions on test data
y_predict = test_input * parameters['m'] + parameters['c']

# Plot predicted values against test data
plt.plot(test_input, y_predict, '-')
plt.plot(test_input, test_output, '.')
plt.show()

During training for the best-fit model...

Evaluating Linear Regression Model with MSE

To evaluate a linear regression model using MSE, follow these steps:

Obtain predictions on the test data.
Calculate residuals by finding the difference between actual and predicted values.
Square the residuals.
Compute the mean of the squared residuals to get the MSE.

The mathematical equation for MSE is:

\[ \text{MSE} = \frac{1}{2n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \]

Assumptions of Linear Regression

No Perfect Multicollinearity:
For \( i \neq j \), \( \rho(X_i, X_j) = 0 \)
Homoscedasticity of Residuals:
\( Var(\epsilon_i) = \sigma^2 \)
Independence of Residuals:
\( Cov(\epsilon_i, \hat{\epsilon}_i) = 0 \)
Linearity:
\( Y = m_0 + m_1 X_1 + m_2 X_2 + \ldots + m_p X_p + \epsilon \)
Residuals are Normally Distributed:
\( \epsilon_i \sim N(0, \sigma^2) \)

Limits

Linear regression is commonly used for training a model which can perform prediction on continuous data, and its subsequent prediction is facilitated by determining the best-fit line. This line indicates that we have found values close enough to the input data. However, linear regression has limitations; it struggles to handle outliers effectively, and categorical data encoding is required each time, among other challenges.

LinearRegression 📈🤖