2023-09-25
2023-09-07
2021-06-28
Updated: ParaCrawl and MultiParaCrawl
2021-06-11
New: MT560 dataset
2021-04-02
2021-02-10
2020-06-30
2020-05-22
2019-10-16
2019-10-14
New corpus: memat (Xhosa/English)
2018-10-06
New version: OpenSubtitles2018
2017-11-06
| Corpus | Sentences | % of OPUS |
|---|---|---|
| OpenSubtitles | 20B | 34.34 |
| NLLB | 13B | 22.10 |
| CCMatrix | 11B | 18.46 |
| MultiCCAligned | 2.2B | 3.80864 |
| ParaCrawl | 1.5B | 2.54692 |
| DGT | 1.1B | 1.85690 |
| MultiHPLT | 897M | 1.52502 |
| XLEnt | 883M | 1.50013 |
| MultiParaCrawl | 789M | 1.34012 |
| LinguaTools-WikiTitles | 487M | 0.82820 |
| CCAligned | 439M | 0.74513 |
| HPLT | 381M | 0.64691 |
| UNPC | 323M | 0.54947 |
| EUbookshop | 279M | 0.47410 |
| EMEA | 243M | 0.41283 |
| GNOME | 225M | 0.38186 |
| KDE4 | 201M | 0.34133 |
| Europarl | 186M | 0.31626 |
| MultiUN | 159M | 0.27100 |
| JRC-Acquis | 147M | 0.25024 |
| TED2020 | 143M | 0.24308 |
| TildeMODEL | 128M | 0.21747 |
| WikiMatrix | 127M | 0.21611 |
| QED | 122M | 0.20764 |
| ParaCrawl-Bonus | 102M | 0.17344 |
| EuroPat | 89M | 0.15140 |
| bible-uedin | 85M | 0.14501 |
| NeuLab-TedTalks | 74M | 0.12592 |
| Samanantar | 50M | 0.08458 |
| Tanzil | 42M | 0.07179 |
| MultiMaCoCu | 28M | 0.04675 |
| JParaCrawl | 26M | 0.04388 |
| MaCoCu | 26M | 0.04339 |
| wikimedia | 23M | 0.03894 |
| giga-fren | 23M | 0.03827 |
| ELITR-ECA | 20M | 0.03427 |
| Anuvaad | 18M | 0.03080 |
| StanfordNLP-NMT | 16M | 0.02706 |
| ECB | 15M | 0.02607 |
| Wikipedia | 13M | 0.02201 |
| SETIMES | 8.8M | 0.01496 |
| Tatoeba | 8.7M | 0.01479 |
| DOGC | 8.5M | 0.01440 |
| GlobalVoices | 7.3M | 0.01233 |
| News-Commentary | 6.4M | 0.01092 |
| PHP | 6.1M | 0.01043 |
| MBS | 5M | 0.008544859 |
| SciELO | 3.8M | 0.006413549 |
| Finlex | 3.1M | 0.005291565 |
| infopankki | 2.9M | 0.004986148 |
| JESC | 2.8M | 0.004753340 |
| fiskmo | 2.1M | 0.003568332 |
| ParIce | 2.1M | 0.003563272 |
| GoURMET | 2.1M | 0.003561185 |
| EUconst | 2.1M | 0.003509015 |
| OpenOffice | 2M | 0.003478278 |
| EOPC | 2M | 0.003422357 |
| TED2013 | 1.9M | 0.003236420 |
| EhuHac | 1.8M | 0.003054292 |
| pmindia | 1.7M | 0.002881996 |
| SUMMA | 1.6M | 0.002675588 |
| IITB | 1.6M | 0.002643512 |
| Books | 1.3M | 0.002125081 |
| ChuBiCo | 1.2M | 0.002042471 |
| CAPES | 1.2M | 0.001967018 |
| Joshua-IPC | 1.1M | 0.001876583 |
| MIZAN | 1M | 0.001735904 |
| SCB_MT_EN_TH | 988K | 0.001679257 |
| MDN_Web_Docs | 874K | 0.001485945 |
| ECDC | 683K | 0.001161390 |
| Elhuyar | 642K | 0.001091480 |
| EiTB-ParCC | 637K | 0.001082703 |
| TEP | 612K | 0.001040060 |
| KDEdoc | 610K | 0.001037236 |
| WMT-News | 447K | 0.000760204 |
| KFTT | 440K | 0.000748137 |
| tico-19 | 319K | 0.000542699 |
| tldr-pages | 258K | 0.000438030 |
| memat | 155K | 0.000262974 |
| hrenWaC | 99K | 0.000168223 |
| TedTalks | 86K | 0.000146723 |
| FFR | 82K | 0.000139564 |
| SPC | 68K | 0.000115138 |
| MontenegrinSubs | 65K | 0.000110520 |
| OfisPublik | 63K | 0.000107767 |
| XhosaNavy | 50K | 0.000084928 |
| Bianet | 48K | 0.000082406 |
| WikiSource | 33K | 0.000056555 |
| ALT | 18K | 0.000030737 |
| Salome | 9.4K | 0.000016017 |
| sardware | 6.2K | 0.000010450 |
| ada83 | 4.1K | 0.000007004 |
| RF | 1.2K | 0.000001964 |
| komi | Not specified | Not specified |
| liv4ever | Not specified | Not specified |
| Mozilla-I10n | Not specified | Not specified |
| MPC1 | Not specified | Not specified |
| Nunavut_Hansard | Not specified | Not specified |
| Ubuntu | Not specified | Not specified |
| WikiTitles | Not specified | Not specified |
