Datasets, Features, and Labels

Analogy categoryAnalogies

Every machine learning model is trained on a dataset — a structured collection of examples, or samples (also called instances or rows), each describing one observation of the phenomenon being modeled. Each sample is described by a set of features (also called attributes or independent variables), which are the measurable properties used as input to the model. For supervised learning, each sample also has a label (also called the target or dependent variable) — the correct answer the model is trying to predict. Understanding this vocabulary precisely is essential because nearly every scikit-learn function, error message, and tutorial assumes it.

🏏

Cricket analogy: A scorecard is like a dataset: each ball bowled is a sample described by features like line, length, and pace, and in a supervised setup the outcome (runs scored or wicket) is the label the model learns to predict, the way a coach studies deliveries to a batter.

Structuring Data as a Feature Matrix

In practice, tabular datasets are organized as a two-dimensional feature matrix, conventionally called X, where each row is a sample and each column is a feature. For supervised tasks, a separate vector y holds the corresponding label for each row. If you're predicting house prices, X might contain columns for square footage, number of bedrooms, and zip code for each house (each row a house), while y holds the sale price for each of those houses. This X/y separation is the near-universal convention across scikit-learn, and most other ML libraries follow the same layout.

🏏

Cricket analogy: Picture a spreadsheet where each row is a bowler's spell against a top batter, columns X hold pitch type, overs bowled, and field placement, while a separate column y holds the runs conceded that spell — the same X/y split scikit-learn expects.

Feature Types

Features come in different types that affect how a model can use them. Numerical features are continuous or discrete numbers (age, price, temperature) that most algorithms can use directly, sometimes after scaling. Categorical features represent a fixed set of discrete categories (color, country, product type) and typically need encoding into a numeric form before most algorithms can use them. Ordinal features are categorical but have a meaningful order (e.g. 'low', 'medium', 'high'), which affects the best encoding strategy. Distinguishing these types correctly early in a project prevents subtle bugs, like treating a zip code as an ordered number when it is really an unordered category.

🏏

Cricket analogy: A player's strike rate is a numerical feature you can scale directly, the team they play for is a categorical feature needing encoding, and their batting order slot is ordinal because opener ranks meaningfully before number eight.

python

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.DataFrame({
    'sqft': [1200, 1850, 900, 2200],
    'bedrooms': [2, 3, 1, 4],
    'neighborhood': ['A', 'B', 'A', 'C'],   # categorical feature
    'sale_price': [250000, 410000, 190000, 525000],  # label
})

# Conventional split: X = features, y = label
X = df.drop(columns=['sale_price'])
y = df['sale_price']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0
)

print('Feature matrix shape:', X_train.shape)  # (n_samples, n_features)
print('Label vector shape:', y_train.shape)     # (n_samples,)

The terms 'feature' and 'label' come from different traditions — statisticians often say 'independent/dependent variable' or 'predictor/response', while ML practitioners say 'feature/label' or 'input/target'. They refer to the same underlying concepts, so recognizing the synonyms helps when reading papers versus tutorials.

A common pitfall is including the label, or a feature that leaks information about the label (like a proxy variable only known after the outcome occurs), inside the feature matrix X. This causes 'target leakage,' where the model appears to perform extremely well during training and evaluation but fails in production because that leaked information isn't available at prediction time.

A dataset is a collection of samples, each described by features (inputs) and, for supervised tasks, a label (the target output).
Tabular data is organized as a feature matrix X (rows = samples, columns = features) alongside a label vector y.
Numerical, categorical, and ordinal features require different preprocessing before most algorithms can use them.
Target leakage — accidentally including label-related information in the features — produces misleadingly high performance.
The X/y convention is nearly universal across scikit-learn and most other ML tooling.
Precise data vocabulary (sample, feature, label) is a prerequisite for reading ML documentation and code correctly.

Practice what you learned

1. In the conventional X/y notation used by scikit-learn, what does X represent?

2. What is a 'label' in supervised learning?

3. Which feature type requires encoding into numeric form before most algorithms can use it directly?

4. What is 'target leakage'?

5. Why is it a mistake to treat zip codes as an ordinal numeric feature?

Was this page helpful?

Data Cleaning Basics

Topics covered

#Python#MachineLearningBasicsStudyNotes#MachineLearning#DatasetsFeaturesAndLabels#Datasets#Features#Labels#Structuring#StudyNotes#SkillVeris

Datasets, Features, and Labels