ML Methods (III)

---

1 – Logistic Regression (Cont.)

Logistic Regression is a statistical model used for predicting the probability of a binary outcome (like “Yes” or “No,” “Default” or “No Default”). It belongs to the class of generalized linear models (GLMs).

Why Not Linear Regression?

Standard linear regression is not suitable for predicting probabilities because it can produce output values that are less than 0 or greater than 1, which isn’t possible for a probability. Logistic regression fixes this by using a transformation.

The Logistic Regression Model

The core idea is to model the conditional probability of the response variable. It uses a logit link function, \[logit p=log[1−pp]\], to transform the probability into a value that a linear model can predict.

The linear model for these transformed probabilities is set up as:

\[logit p=β0 +β1 x1 +β2 x2 +…βk xk\]

The glm (generalized linear model) function in R is typically used for fitting this model, specifying family = binomial("logit").

Assessing Model Accuracy

To see how well the model predicts, several metrics are used, often assessed through cross-validation (like 10-fold CV).

Variable Importance

When a model has multiple predictor variables, it’s important to know which ones are most influential in predicting the outcome. The varImp function (from the caret package) can be used to gauge this.
In an example using the Default dataset (predicting debt default using student status, balance, and income), the importance was ranked:

Variable	Overall Importance (from R output)	Interpretation
balance	20.8174258		Most important variable in predicting default.
studentYes	2.2253964	Second most important.
income	0.1776301	Least important.

Confusion Matrix

The confusion matrix compares the actual outcomes to the predicted outcomes to evaluate performance.

Term	Description
True Positive (TP)	Prediction of the right level/event.
False Positive (FP)	Prediction of an event that did not happen (e.g., predicting a customer defaults, but they don’t).
False Negative (FN)	Failure to predict an event that did happen (e.g., failing to predict a customer defaults, but they do).
Accuracy	Overall correctness: \[Total(TN+TP)\]
No Information Rate (NIR)	The accuracy you’d get by simply guessing the most frequent class. A model must have an accuracy significantly better than the NIR to be useful.

In the provided example (where ‘No’ default is the positive class):

Reference	Prediction: No	Prediction: Yes
Actual: No	2885 (True Negative/Positive)	72 (False Positive/Negative)
Actual: Yes	16 (False Negative/Positive)	27 (True Positive/Negative)

• Overall Accuracy: 0.9707.
• No Information Rate: 0.967.

ROC Curves and AUC

Since maximizing accuracy alone can be misleading when classes are imbalanced (like the default data where “No” default is 96.7% of the data) , we look to balance Sensitivity and Specificity.

• ROC (Receiver Operating Characteristics) Curve: Plots the True Positive Rate (Sensitivity) on the y-axis against the False Positive Rate on the x-axis.
  
• AUC (Area Under the Curve): Measures the overall performance. A line representing a random guess is diagonal (lower-left to upper-right). The closer the curve is to the upper-left corner, the better the model is. AUC computes the area under this curve, and the objective is to maximize it.

---

2 : K-Nearest Neighbour (kNN) & Classification

K-Nearest Neighbour (kNN) is a simple, yet powerful, non-parametric classification method.

The Core Idea

The fundamental principle is that things that are alike are likely to have similar properties. The kNN algorithm classifies a new data point based on the class of its closest (“nearest”) neighbors in the training data.

• k: The variable number of nearest neighbors to consider.
  
• Memory-Based: kNN is a memory-based algorithm, meaning it doesn’t create a simplified, closed-form model but relies on the entire training dataset for prediction.

Measuring Similarity: Euclidean Distance

To find the “nearest” neighbors, kNN uses a distance function, typically Euclidean distance, which measures the shortest direct route between two points (observations).

For two instances, p and q, with n features, the distance is:

\[dist(p,q)=(p1​−q1​)2+(p2​−q2​)2+…+(pn​−qn​)2\]​

Choosing k and Data Scaling

The choice of k is crucial as it balances the bias-variance tradeoff.

• Large k: Reduces the impact of noisy data (lower variance) but risks ignoring small, important patterns (higher bias).
  
• Small k: May work best for high-signal data with few noisy features.
  
• Grid Search: An automated approach to test a range of k values (e.g., k=1,3,5,…,99) using a resampling method like k-fold cross-validation to find the best-performing value.

Data Scaling is Essential: Because kNN relies on distance calculations, all predictor variables should be rescaled to a similar range (e.g., centering and scaling) to prevent features with larger numeric ranges (like income) from dominating the distance calculation.

kNN Strengths and Weaknesses

Strengths	Weaknesses
Simple and effective.	Does not produce a descriptive model, limiting feature understanding.
A non-parametric method (makes no assumption about the underlying distribution).	Requires selecting an appropriate $k$ value.
Fast training phase.	Slow classification phase (prediction).
Well-suited for classification where concepts are “difficult to define”.	Problems with missing data (requires additional processing).

---

3 : Forecasting Stock Price Movement using ML (Part-I)

This section introduces using Machine Learning for forecasting stock prices by generating technical indicators from price data.

Introduction to Technical Analysis

Technical Analysis (TA) is the “science of recording… the actual history of trading (price changes, volume of transactions, etc.)… and then deducing from that pictured history the probable future trend”.

Charts and OHLC Data

Forecasting often begins with charting price data, which typically uses OHLC (Open, High, Low, and Close) data along with volume. R packages like quantmod are used to download and plot this data.

Data Field	Description
Open	The price at which a stock first traded when the market opened.
High	The highest price traded during the period.
Low	The lowest price traded during the period.
Close	The last trading price for the period.
Volume	The total number of shares traded during the period.

The main types of charts used are:

• Line Chart
• Candlestick Chart
• OHLC (Bar) Chart
  

Charts with Indicators

Technical Indicators are mathematical transformations of price and volume data used to predict future price movements. The TTR package in R provides many of these.

Simple Moving Average (SMA) and Exponential Moving Average (EMA)

Moving averages show the average price level on a rolling basis.

• SMA (Simple Moving Average): Weighs all candles/days in the period equally (e.g., a 5-day SMA is the average price of the 5 preceding days).
  
• EMA (Exponential Moving Average): Gives exponential weights, meaning it overweights recent days compared to older ones, making it more responsive to new information.
  

Bollinger Bands

Bollinger Bands are composed of three lines:

• A Simple Moving Average (the middle band).
• An Upper Band and a Lower Band.

The upper and lower bands are typically set at two standard deviations (±) from the middle moving average, though this can be modified.

Momentum

Momentum is a measure of the speed or velocity of price changes in a security. It’s used to identify the strength of a price movement.