Spark MLlib provides a distributed machine learning library built on DataFrames, with a Pipeline API that chains preprocessing transformers and estimators into reproducible, serialisable workflows. For data engineers, MLlib's value is not primarily in its model algorithms but in its preprocessing transformers: scalers, encoders, imputers, and feature assemblers that operate on distributed DataFrames at scale. Running these preprocessing steps in Spark before sampling data down to a single-machine ML framework avoids the bottleneck of loading terabytes of raw features into a single pandas DataFrame.
The MLlib Pipeline API distinguishes between Transformers and Estimators. A Transformer takes a DataFrame and produces a new DataFrame with additional columns — it is stateless and requires no fitting. An Estimator fits to data to learn parameters — the scaler learns mean and standard deviation, the indexer learns category vocabulary — and returns a fitted Transformer. A Pipeline is an ordered sequence of Estimators and Transformers that can be fit once to training data and applied to any compatible DataFrame, guaranteeing consistent preprocessing.