2025-02-14
2023-09-25
2023-09-07
2021-06-28
Updated: ParaCrawl and MultiParaCrawl
2021-06-11
New: MT560 dataset
2021-04-02
2021-02-10
2020-06-30
2020-05-22
2019-10-16
2019-10-14
New corpus: memat (Xhosa/English)
2018-10-06
New version: OpenSubtitles2018
2017-11-06
| Corpus | Sentences | % of OPUS |
|---|---|---|
| OpenSubtitles | 20B | 34.34 |
| NLLB | 13B | 22.10 |
| CCMatrix | 11B | 18.46 |
| MultiCCAligned | 2.2B | 3.80872 |
| ParaCrawl | 1.5B | 2.54697 |
| DGT | 1.1B | 1.85693 |
| MultiHPLT | 897M | 1.52505 |
| XLEnt | 883M | 1.50016 |
| MultiParaCrawl | 789M | 1.34015 |
| LinguaTools-WikiTitles | 487M | 0.82822 |
| CCAligned | 439M | 0.74514 |
| HPLT | 381M | 0.64692 |
| UNPC | 323M | 0.54948 |
| EUbookshop | 279M | 0.47411 |
| EMEA | 243M | 0.41284 |
| GNOME | 225M | 0.38186 |
| KDE4 | 201M | 0.34134 |
| Europarl | 186M | 0.31627 |
| MultiUN | 159M | 0.27101 |
| JRC-Acquis | 147M | 0.25025 |
| TED2020 | 143M | 0.24308 |
| TildeMODEL | 128M | 0.21747 |
| WikiMatrix | 127M | 0.21611 |
| QED | 122M | 0.20765 |
| ParaCrawl-Bonus | 102M | 0.17344 |
| EuroPat | 89M | 0.15140 |
| bible-uedin | 85M | 0.14501 |
| NeuLab-TedTalks | 74M | 0.12592 |
| Samanantar | 50M | 0.08458 |
| Tanzil | 42M | 0.07179 |
| MultiMaCoCu | 28M | 0.04675 |
| JParaCrawl | 26M | 0.04388 |
| MaCoCu | 26M | 0.04339 |
| wikimedia | 23M | 0.03894 |
| giga-fren | 23M | 0.03827 |
| ELITR-ECA | 20M | 0.03427 |
| Anuvaad | 18M | 0.03080 |
| StanfordNLP-NMT | 16M | 0.02706 |
| ECB | 15M | 0.02607 |
| Wikipedia | 13M | 0.02201 |
| SETIMES | 8.8M | 0.01496 |
| Tatoeba | 8.7M | 0.01479 |
| DOGC | 8.5M | 0.01440 |
| GlobalVoices | 7.3M | 0.01233 |
| News-Commentary | 6.4M | 0.01092 |
| PHP | 6.1M | 0.01043 |
| MBS | 5M | 0.008545028 |
| SciELO | 3.8M | 0.006413675 |
| Finlex | 3.1M | 0.005291669 |
| infopankki | 2.9M | 0.004986246 |
| JESC | 2.8M | 0.004753434 |
| fiskmo | 2.1M | 0.003568403 |
| ParIce | 2.1M | 0.003563342 |
| GoURMET | 2.1M | 0.003561256 |
| EUconst | 2.1M | 0.003509084 |
| OpenOffice | 2M | 0.003478346 |
| EOPC | 2M | 0.003422424 |
| TED2013 | 1.9M | 0.003236484 |
| EhuHac | 1.8M | 0.003054352 |
| SUMMA | 1.6M | 0.002675641 |
| IITB | 1.6M | 0.002643565 |
| Books | 1.3M | 0.002125123 |
| ChuBiCo | 1.2M | 0.002042511 |
| CAPES | 1.2M | 0.001967057 |
| Joshua-IPC | 1.1M | 0.001876620 |
| MIZAN | 1M | 0.001735938 |
| SCB_MT_EN_TH | 988K | 0.001679290 |
| MDN_Web_Docs | 874K | 0.001485975 |
| ECDC | 683K | 0.001161413 |
| Elhuyar | 642K | 0.001091501 |
| EiTB-ParCC | 637K | 0.001082725 |
| TEP | 612K | 0.001040081 |
| KDEdoc | 610K | 0.001037257 |
| WMT-News | 447K | 0.000760219 |
| KFTT | 440K | 0.000748151 |
| pmindia | 422K | 0.000717371 |
| tldr-pages | 370K | 0.000627922 |
| tico-19 | 319K | 0.000542710 |
| memat | 155K | 0.000262979 |
| hrenWaC | 99K | 0.000168226 |
| TedTalks | 86K | 0.000146726 |
| FFR | 82K | 0.000139567 |
| SPC | 68K | 0.000115140 |
| MontenegrinSubs | 65K | 0.000110522 |
| OfisPublik | 63K | 0.000107769 |
| XhosaNavy | 50K | 0.000084930 |
| Bianet | 48K | 0.000082408 |
| WikiSource | 33K | 0.000056556 |
| ALT | 18K | 0.000030738 |
| Salome | 9.4K | 0.000016017 |
| sardware | 6.2K | 0.000010450 |
| ada83 | 4.1K | 0.000007004 |
| RF | 1.2K | 0.000001964 |
| komi | Not specified | Not specified |
| liv4ever | Not specified | Not specified |
| Mozilla-I10n | Not specified | Not specified |
| Nunavut_Hansard | Not specified | Not specified |
| translatewiki | Not specified | Not specified |
| Ubuntu | Not specified | Not specified |
| WikiTitles | Not specified | Not specified |
