lm-evaluation-harness
EleutherAIgithub.com/EleutherAI/lm-evaluation-harness
A framework for few-shot evaluation of language models.
Updated 1 h ago · first seen 11 Sept 2026
framework_01M294NC1A8C5XABTXDFE0X1BJ
- Version
- 0.4.13
- T2 · 1 h ago
- Released
- 31 Aug 2026
- T2 · 1 h ago
- License
- MIT
- T2 · 1 h ago
- Stars
- 13,954
- T2 · 1 h ago
Specification
- License
- MIT
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Repository
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Created at
- 28 Aug 2020
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Default branch
- main
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Homepage
- https://www.eleuther.ai
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Kind
- tool
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Latest release
- 31 Aug 2026
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Latest release tag
- v0.4.13
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Latest version
- 0.4.13
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- License name
- MIT License
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Forks
- 3,559
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Releases
- 19
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Stars
- 13,954
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Tags
- 20
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Watchers
- 55
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- PyPI
- lm-eval
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Readme url
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/HEAD/README.md
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Releases url
- https://github.com/EleutherAI/lm-evaluation-harness/releases
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
- Topics
- evaluation-framework, language-model, transformer
Source:GitHub public repositories (HTML, releases.atom, raw files)T2observed 1 h agomedium
Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →
Provenance
Attributed facts
20
Source tiers
T220
Freshest observation
1 h ago
Conflicts
None
- Developed by
- EleutherAI
As of
Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.
Claim history
Licenselicense1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| MIT | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Repositoryrepository_url1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://github.com/EleutherAI/lm-evaluation-harness | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Created atcreated_at1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 28 Aug 2020 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Default branchdefault_branch1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| main | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Descriptiondescription1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| A framework for few-shot evaluation of language models. | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Homepagehomepage1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://www.eleuther.ai | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Kindkind1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| tool | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Latest releaselatest_release_at1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 31 Aug 2026 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Latest release taglatest_release_tag1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| v0.4.13 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Latest versionlatest_version1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 0.4.13 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
License namelicense_name1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| MIT License | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Forksmetric.forks1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 3,559 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Releasesmetric.releases1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 19 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Starsmetric.stars1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 13,954 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Watchersmetric.watchers1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 55 | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
PyPIpypi1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| lm-eval | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Readme urlreadme_url1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/HEAD/README.md | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Releases urlreleases_url1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://github.com/EleutherAI/lm-evaluation-harness/releases | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Topicstopics1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| evaluation-framework, language-model, transformer | → current | current | GitHub public repositories (HTML, releases.atom, raw files)T2 | medium | deterministic |
Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →
EleutherAI/lm-evaluation-harness released v0.4.7
githubEleutherAI/lm-evaluation-harness released v0.4.8
githubEleutherAI/lm-evaluation-harness released v0.4.9
githubEleutherAI/lm-evaluation-harness released v0.4.9.1
githubEleutherAI/lm-evaluation-harness released lm-eval v0.4.9.2 Release Notes
githubEleutherAI/lm-evaluation-harness released v0.4.10
githubEleutherAI/lm-evaluation-harness released v0.4.11
githubEleutherAI/lm-evaluation-harness released v0.4.12
githubEleutherAI/lm-evaluation-harness released v0.5.0.dev1
githubEleutherAI/lm-evaluation-harness released v0.4.13
githubNew framework: lm-evaluation-harness (EleutherAI)
github
| Source | Document | Type | Tier | Last observed | Snapshots |
|---|---|---|---|---|---|
| GitHub public repositories (HTML, releases.atom, raw files) | github.com/EleutherAI/lm-evaluation-harness/releases.atom | feed | T2· Quality secondary | 1 h ago | 1 |
| GitHub public repositories (HTML, releases.atom, raw files) | raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/HEAD/README.md | readme | T2· Quality secondary | 1 h ago | 1 |
| GitHub public repositories (HTML, releases.atom, raw files) | github.com/EleutherAI/lm-evaluation-harness | repository | T2· Quality secondary | 1 h ago | 2 |
Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.