ADVERTISEMENT
ADVERTISEMENT

Feature Scaling and Normalization in Machine Learning

What is Feature Scaling?

Feature Scaling is a data preprocessing technique that adjusts numerical values to a common scale, ensuring no single feature dominates the model due to different units or magnitudes. It improves the performance and accuracy of machine learning algorithms, especially those based on distance calculations. Feature scaling includes methods like Normalization (Min-Max Scaling) and Standardization (Z-Score Scaling).

What is Normalization?

Normalization is a feature scaling technique that transforms numerical values into a fixed range, usually between 0 and 1 (or -1 to 1). It is useful for datasets with varying scales and is commonly used in Neural Networks and KNN, where feature magnitudes impact model predictions. Normalization ensures that all features contribute equally, preventing bias toward larger values.

Benefits of Feature Scaling

Feature scaling helps machine learning models work better by making all numbers similar in size.

  1. Makes Features Comparable → Prevents big numbers from dominating small ones.
  2. Helps Models Learn Faster → Scaling speeds up training and improves accuracy.
  3. Improves Distance-Based Algorithms → Models like KNN, K-Means, and SVM give better results with scaled data.
  4. Prevents Errors in Calculation → Some models work poorly if numbers have very different sizes.
  5. Ensures Fair Contribution of Features → No single feature becomes too important just because it has larger values.

Benefits of Normalization (Min-Max Scaling)

  1. Brings All Features to the Same Scale → Ensures that large values don’t dominate smaller ones.
  2. Improves Model Performance → Helps distance-based models like KNN and Neural Networks work efficiently.
  3. Speeds Up Training → Reduces computation time, especially for gradient-based optimization algorithms.
  4. Prevents Bias Toward Larger Features → Avoids issues where models give more importance to features with higher magnitude.
  5. Best for Features with Different Units → Works well when dealing with height (cm), weight (kg), or income ($).

Difference Between Feature Scaling and Normalization

  • Feature Scaling is a broader concept, including Normalization and Standardization.
  • Normalization is a specific type of Feature Scaling that rescales data between a fixed range, typically [0,1].
Aspect Feature Scaling Normalization
Definition A general term for transforming data to a standard scale. A specific technique that rescales data to a fixed range (0 to 1 or -1 to 1).
Purpose Ensures all features contribute equally to a model. Prevents larger values from dominating smaller ones.
Techniques Included Normalization, Standardization, Log Scaling, Robust Scaling, etc. Only Min-Max Scaling (rescaling values between 0 and 1).
Formula Varies depending on the technique used (e.g., Z-score, Min-Max, etc.). X′=X-Xmin/Xmax-Xmin
Effect on Data Adjusts values to a uniform scale but does not necessarily fix the range. Compresses values into a strict range (0 to 1).
Best For Any dataset needing rescaled values, depending on model requirements. Algorithms sensitive to magnitude differences (e.g., Neural Networks, KNN).

 

Types of Feature Scaling

1. Normalization (Min-Max Scaling)

Rescales values to a range of [0,1] using the formula:

where:

  • X= Original value
  • Xmin = Minimum value in the dataset
  • Xmax = Maximum value in the dataset
  • Definition: Rescales data values between 0 and 1 (or sometimes between -1 and 1).
  • When to Use: When you want to keep all values within a specific range, especially for algorithms like Neural Networks and KNN.

Example Dataset (Car Prices in $1000s)

Car Price ($1000s)
A 20
B 50
C 35
D 80
E 60

 

Step-by-Step Min-Max Scaling Calculation

  • Find Minimum (Xmin) = 20
  • Find Maximum (Xmax) = 80
  • Apply the Min-Max formula:

2. Standardization (Z-Score Scaling)

  • Definition: Adjusts data so that it has a mean of 0 and a standard deviation of 1.
  • When to Use: When your data follows a normal distribution or for models like Linear Regression and SVM.
  • Example:
    If the average height is 165 cm, and standard deviation is 10 cm, a height of 180 cm becomes +1.5 (meaning 1.5 standard deviations above the mean).

Example of Feature Scaling vs Normalization

Let's take a dataset with "Height (cm)" and "Weight (kg)", which have different scales.

Original Dataset:

Person Height (cm) Weight (kg)
A 180 75
B 160 55
C 170 65
D 150 50
E 175 72 
import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler

# Creating the dataset
data = {
    'Person': ['A', 'B', 'C', 'D', 'E'],
    'Height (cm)': [180, 160, 170, 150, 175],
    'Weight (kg)': [75, 55, 65, 50, 72]
}

df = pd.DataFrame(data)

# Applying Min-Max Normalization
minmax_scaler = MinMaxScaler()
df[['Height (Normalized)', 'Weight (Normalized)']] = minmax_scaler.fit_transform(df[['Height (cm)', 'Weight (kg)']])

# Applying Standardization (Z-score scaling)
standard_scaler = StandardScaler()
df[['Height (Standardized)', 'Weight (Standardized)']] = standard_scaler.fit_transform(df[['Height (cm)', 'Weight (kg)']])

# Displaying the final DataFrame
print(df)

Output:

Person Height (cm) Weight (kg) Height (Normalized) Weight (Normalized) Height (Standardized) Weight (Standardized)
A 180 75 1.000 1.000 1.14 1.14
B 160 55 0.500 0.25 -0.57 -0.92
C 170 65 0.750 0.50 0.29 -0.20
D 150 50 0.000 0.00 -1.43 -1.41
E 175 72 0.875 0.88 0.57 1.39

ADVERTISEMENT
ADVERTISEMENT