What is Correlation?
Understand what correlation measures, Pearson vs Spearman, why correlation is not causation, and how to compute and interpret it in Python for interviews.
Expected Interview Answer
Correlation is a statistical measure, typically Pearson's r, that quantifies the strength and direction of the linear relationship between two variables on a scale from -1 (perfect negative) to +1 (perfect positive), with 0 meaning no linear relationship. Correlation does not imply causation.
Pearson's r captures linear relationships and is sensitive to outliers, while Spearman's rank correlation captures monotonic relationships and is more robust to outliers and non-normal data. In exploratory analysis, a correlation matrix or heatmap is used to quickly scan for strong relationships across many variable pairs before modeling. A high correlation between two variables can arise because one causes the other, because both are driven by a third confounding variable, or simply by coincidence in a small sample, so a controlled experiment is usually needed to establish causation. A correlation near zero doesn't necessarily mean no relationship at all; it can indicate a strong non-linear relationship that Pearson's r fails to capture.
- Quickly quantifies how strongly two variables move together
- Guides feature selection and multicollinearity checks
- Standardized scale (-1 to 1) makes comparisons easy
- Useful first pass before building a formal model
- Highlights candidate relationships worth investigating further
AI Mentor Explanation
Correlation is like plotting a batter's practice-net hours against their match runs and finding they move together, giving a coefficient near +0.8. That strong positive number shows the two rise and fall together, but it does not prove practice alone causes runs — natural talent or fitness could be driving both.
Step-by-Step Explanation
Step 1
Choose the right measure
Use Pearson's r for linear relationships between continuous variables, Spearman's rho for monotonic or ranked relationships.
Step 2
Compute the coefficient
Calculate the correlation value, which ranges from -1 to +1.
Step 3
Interpret the sign
Positive means the variables rise together, negative means one rises as the other falls.
Step 4
Interpret the magnitude
Values near plus or minus 1 indicate a strong relationship, values near 0 indicate a weak or no linear relationship.
Step 5
Check for outliers
A few extreme points can distort Pearson's r, so visualize with a scatterplot first.
Step 6
Never assume causation
Look for confounding variables and consider a controlled experiment before claiming one variable causes the other.
What Interviewer Expects
- Gives the correct range and interpretation of Pearson's r
- Distinguishes correlation from causation with a concrete example
- Mentions Spearman for non-linear or monotonic cases
- Notes sensitivity to outliers
- Mentions visualizing with a scatterplot before trusting the number
Common Mistakes
- Claiming correlation proves causation
- Using Pearson's r on a clearly non-linear relationship
- Ignoring outliers that distort the coefficient
- Confusing a near-zero correlation with 'no relationship at all'
Best Answer (HR Friendly)
“Correlation measures how strongly two things move together, on a scale from -1 to +1. A high correlation just means two variables tend to rise or fall together, not that one is actually causing the other, so you have to be careful about drawing conclusions.”
Code Example
import pandas as pd
df = pd.read_csv("housing.csv")
corr_matrix = df[["square_feet", "price", "age_years"]].corr(method="pearson")
print(corr_matrix)
# Spearman for monotonic, non-linear relationships
spearman_corr = df[["square_feet", "price"]].corr(method="spearman")
print(spearman_corr)Follow-up Questions
- What is the difference between Pearson and Spearman correlation?
- Why doesn't correlation imply causation? Give an example.
- How do outliers affect a correlation coefficient?
- What is a confounding variable?
- How would you test whether a relationship is actually causal?
MCQ Practice
1. What range does Pearson's correlation coefficient fall in?
Pearson's r ranges from -1 (perfect negative linear relationship) to +1 (perfect positive linear relationship).
2. A correlation of -0.9 between two variables means:
A coefficient near -1 indicates a strong negative linear relationship, meaning the variables move in opposite directions, not causation.
3. Which correlation measure is better suited to a monotonic but non-linear relationship?
Spearman's rank correlation captures monotonic relationships even when they aren't linear, unlike Pearson's r.
Flash Cards
What does a correlation of +1 mean? — A perfect positive linear relationship — both variables increase together exactly.
Does correlation imply causation? — No — a strong correlation can arise from confounders or coincidence, not just direct cause and effect.
When should you use Spearman instead of Pearson? — When the relationship is monotonic but not linear, or the data is ordinal or has outliers.
What visualization should precede trusting a correlation number? — A scatterplot, to check for outliers and non-linear patterns.