HumanEval
By OpenAI
HumanEval is a benchmark dataset released by OpenAI for measuring the functional correctness of code generated by language models. It consists of hand-written Python programming problems, each given as a function signature plus a docstring…
Definition
HumanEval is a benchmark dataset released by OpenAI for measuring the functional correctness of code generated by language models. It consists of hand-written Python programming problems, each given as a function signature plus a docstring describing the required behavior, paired with unit tests that check whether a generated implementation actually works. Models are scored with the pass@k metric, which estimates the probability that at least one of k sampled completions passes all associated tests.
Overview
HumanEval addresses a gap that older text-similarity metrics such as BLEU left open: a piece of generated code can look nothing like a reference solution and still be entirely correct, or look similar and still fail. Instead of comparing surface text, HumanEval executes the candidate function against a held-out set of unit tests and records a pass or fail outcome, which makes the benchmark a direct proxy for whether the code does what the docstring asked for. Mechanically, each of the problems in the set supplies a function stub, a natural-language docstring, and a small number of assertions that are hidden from the model at generation time. A model produces one or more completions per problem, each completion is executed in a sandboxed interpreter, and pass@k is computed from how many of the k samples pass every test. Because execution rather than text matching is the judge, HumanEval rewards genuinely working logic over superficially plausible code. Among code-generation evaluations, HumanEval sits alongside benchmarks like MBPP and later, harder suites such as HumanEval+, which adds more edge-case tests to catch models that overfit to the original, sparser test suite. Compared to broad reasoning benchmarks like GSM8K or MMLU, HumanEval is narrowly scoped to self-contained Python functions, so it says little about a model's ability to work across multi-file projects or maintain existing code. In practice, HumanEval is used as a standard reporting metric in model release papers and leaderboards, letting teams compare code models on a common, reproducible task. Researchers also use it during training to check whether fine-tuning or reinforcement learning on code actually improves functional correctness rather than just fluency. Its main limitation is saturation: strong models now pass a large majority of the original problems, which compresses the useful signal and has pushed the field toward harder successors and multi-file, real-repository benchmarks. HumanEval also only covers Python and short, self-contained functions, so it does not measure debugging, refactoring, or working within a large existing codebase, and a high score on it does not guarantee reliable behavior on production engineering tasks. Because the problem statements are public, there is also a standing risk that later training corpora absorb the problems or close paraphrases of them, which would inflate scores without reflecting genuine improvement, a concern that has motivated periodic calls for fresh, held-out variants of the benchmark. Teams that rely on HumanEval as their sole signal of coding ability should treat it as one data point among several rather than a complete picture of real-world coding competence.
Key Features
- 164 hand-written Python problems with docstrings and hidden unit tests
- Scores functional correctness through execution rather than text similarity
- Uses the pass@k metric to estimate success across sampled completions
- Runs generated code in an isolated sandbox for safety during evaluation
- Widely reported in model release papers as a standard comparison point
- Has inspired stricter successors like HumanEval+ with expanded test coverage
- Limited to short, self-contained functions rather than full codebases