2025-02-14
2023-09-25
2023-09-07
2021-06-28
Updated: ParaCrawl and MultiParaCrawl
2021-06-11
New: MT560 dataset
2021-04-02
2021-02-10
2020-06-30
2020-05-22
2019-10-16
2019-10-14
New corpus: memat (Xhosa/English)
2018-10-06
New version: OpenSubtitles2018
2017-11-06
| Corpus | Sentences | % of OPUS |
|---|---|---|
| OpenSubtitles | 20B | 34.34 |
| NLLB | 13B | 22.10 |
| CCMatrix | 11B | 18.46 |
| MultiCCAligned | 2.2B | 3.80811 |
| ParaCrawl | 1.5B | 2.54657 |
| DGT | 1.1B | 1.85664 |
| MultiHPLT | 897M | 1.52481 |
| XLEnt | 883M | 1.49992 |
| MultiParaCrawl | 789M | 1.33993 |
| LinguaTools-WikiTitles | 487M | 0.82808 |
| CCAligned | 439M | 0.74502 |
| HPLT | 381M | 0.64682 |
| UNPC | 323M | 0.54940 |
| EUbookshop | 279M | 0.47403 |
| EMEA | 243M | 0.41277 |
| GNOME | 225M | 0.38180 |
| KDE4 | 201M | 0.34128 |
| Europarl | 186M | 0.31622 |
| MultiUN | 159M | 0.27097 |
| JRC-Acquis | 147M | 0.25021 |
| TED2020 | 143M | 0.24304 |
| TildeMODEL | 128M | 0.21744 |
| WikiMatrix | 127M | 0.21608 |
| QED | 122M | 0.20761 |
| ParaCrawl-Bonus | 102M | 0.17341 |
| EuroPat | 89M | 0.15138 |
| bible-uedin | 85M | 0.14499 |
| NeuLab-TedTalks | 74M | 0.12590 |
| Samanantar | 50M | 0.08456 |
| Tanzil | 42M | 0.07178 |
| MultiMaCoCu | 28M | 0.04675 |
| JParaCrawl | 26M | 0.04388 |
| MaCoCu | 26M | 0.04338 |
| wikimedia | 23M | 0.03894 |
| giga-fren | 23M | 0.03826 |
| ELITR-ECA | 20M | 0.03427 |
| Anuvaad | 18M | 0.03079 |
| StanfordNLP-NMT | 16M | 0.02706 |
| ECB | 15M | 0.02607 |
| Wikipedia | 13M | 0.02200 |
| SETIMES | 8.8M | 0.01495 |
| Tatoeba | 8.7M | 0.01479 |
| DOGC | 8.5M | 0.01439 |
| WikiTitles | 8M | 0.01364 |
| GlobalVoices | 7.3M | 0.01233 |
| News-Commentary | 6.4M | 0.01092 |
| PHP | 6.1M | 0.01042 |
| MBS | 5M | 0.008543660 |
| SciELO | 3.8M | 0.006412649 |
| Finlex | 3.1M | 0.005290822 |
| infopankki | 2.9M | 0.004985448 |
| JESC | 2.8M | 0.004752673 |
| fiskmo | 2.1M | 0.003567832 |
| ParIce | 2.1M | 0.003562772 |
| GoURMET | 2.1M | 0.003560686 |
| EUconst | 2.1M | 0.003508522 |
| OpenOffice | 2M | 0.003477790 |
| EOPC | 2M | 0.003421877 |
| TED2013 | 1.9M | 0.003235965 |
| EhuHac | 1.8M | 0.003053863 |
| SUMMA | 1.6M | 0.002675213 |
| IITB | 1.6M | 0.002643141 |
| ALT | 1.4M | 0.002396788 |
| Books | 1.3M | 0.002124783 |
| ChuBiCo | 1.2M | 0.002042184 |
| CAPES | 1.2M | 0.001966742 |
| Joshua-IPC | 1.1M | 0.001876319 |
| MIZAN | 1M | 0.001735660 |
| SCB_MT_EN_TH | 988K | 0.001679022 |
| MDN_Web_Docs | 874K | 0.001485737 |
| ECDC | 683K | 0.001161227 |
| Elhuyar | 642K | 0.001091327 |
| EiTB-ParCC | 637K | 0.001082551 |
| TEP | 612K | 0.001039914 |
| KDEdoc | 610K | 0.001037090 |
| WMT-News | 447K | 0.000760098 |
| KFTT | 440K | 0.000748032 |
| pmindia | 422K | 0.000717256 |
| tldr-pages | 370K | 0.000627821 |
| tico-19 | 319K | 0.000542623 |
| memat | 155K | 0.000262937 |
| hrenWaC | 99K | 0.000168199 |
| TedTalks | 86K | 0.000146702 |
| FFR | 82K | 0.000139545 |
| SPC | 68K | 0.000115122 |
| MontenegrinSubs | 65K | 0.000110504 |
| OfisPublik | 63K | 0.000107752 |
| XhosaNavy | 50K | 0.000084916 |
| Bianet | 48K | 0.000082395 |
| WikiSource | 33K | 0.000056547 |
| Salome | 9.4K | 0.000016014 |
| sardware | 6.2K | 0.000010449 |
| ada83 | 4.1K | 0.000007003 |
| InterdialectCorpus | 2.4K | 0.000004157 |
| RF | 1.2K | 0.000001964 |
| komi | Not specified | Not specified |
| liv4ever | Not specified | Not specified |
| Mozilla-I10n | Not specified | Not specified |
| Nunavut_Hansard | Not specified | Not specified |
| translatewiki | Not specified | Not specified |
| Ubuntu | Not specified | Not specified |
