2025-02-14
2023-09-25
2023-09-07
2021-06-28
Updated: ParaCrawl and MultiParaCrawl
2021-06-11
New: MT560 dataset
2021-04-02
2021-02-10
2020-06-30
2020-05-22
2019-10-16
2019-10-14
New corpus: memat (Xhosa/English)
2018-10-06
New version: OpenSubtitles2018
2017-11-06
| Corpus | Sentences | % of OPUS |
|---|---|---|
| OpenSubtitles | 20B | 33.71 |
| NLLB | 13B | 21.69 |
| CCMatrix | 11B | 18.12 |
| MultiCCAligned | 2.2B | 3.73859 |
| MultiHPLT | 1.7B | 2.82215 |
| ParaCrawl | 1.5B | 2.50008 |
| DGT | 1.1B | 1.82274 |
| XLEnt | 883M | 1.47254 |
| MultiParaCrawl | 789M | 1.31547 |
| HPLT | 681M | 1.13537 |
| LinguaTools-WikiTitles | 487M | 0.81297 |
| CCAligned | 439M | 0.73142 |
| UNPC | 323M | 0.53937 |
| EUbookshop | 279M | 0.46538 |
| EMEA | 243M | 0.40524 |
| GNOME | 225M | 0.37483 |
| KDE4 | 201M | 0.33505 |
| Europarl | 186M | 0.31044 |
| MultiUN | 159M | 0.26602 |
| JRC-Acquis | 147M | 0.24564 |
| TED2020 | 143M | 0.23860 |
| TildeMODEL | 128M | 0.21347 |
| WikiMatrix | 127M | 0.21213 |
| QED | 122M | 0.20382 |
| ParaCrawl-Bonus | 102M | 0.17025 |
| EuroPat | 89M | 0.14861 |
| bible-uedin | 85M | 0.14234 |
| NeuLab-TedTalks | 74M | 0.12360 |
| Samanantar | 50M | 0.08302 |
| Tanzil | 42M | 0.07047 |
| MultiMaCoCu | 28M | 0.04589 |
| JParaCrawl | 26M | 0.04307 |
| MaCoCu | 26M | 0.04259 |
| wikimedia | 23M | 0.03822 |
| giga-fren | 23M | 0.03756 |
| ELITR-ECA | 20M | 0.03364 |
| Anuvaad | 18M | 0.03023 |
| StanfordNLP-NMT | 16M | 0.02656 |
| ECB | 15M | 0.02559 |
| Wikipedia | 13M | 0.02160 |
| SETIMES | 8.8M | 0.01468 |
| Tatoeba | 8.7M | 0.01452 |
| DOGC | 8.5M | 0.01413 |
| WikiTitles | 8M | 0.01339 |
| GlobalVoices | 7.3M | 0.01210 |
| News-Commentary | 6.4M | 0.01072 |
| PHP | 6.1M | 0.01023 |
| MBS | 5M | 0.008387693 |
| SciELO | 3.8M | 0.006295584 |
| Finlex | 3.1M | 0.005194237 |
| infopankki | 2.9M | 0.004894437 |
| JESC | 2.8M | 0.004665911 |
| fiskmo | 2.1M | 0.003502700 |
| ParIce | 2.1M | 0.003497732 |
| GoURMET | 2.1M | 0.003495684 |
| EUconst | 2.1M | 0.003444473 |
| OpenOffice | 2M | 0.003414301 |
| EOPC | 2M | 0.003359409 |
| TED2013 | 1.9M | 0.003176892 |
| EhuHac | 1.8M | 0.002998114 |
| SUMMA | 1.6M | 0.002626376 |
| IITB | 1.6M | 0.002594890 |
| ALT | 1.4M | 0.002353034 |
| Books | 1.3M | 0.002085994 |
| ChuBiCo | 1.2M | 0.002004904 |
| CAPES | 1.2M | 0.001930838 |
| Joshua-IPC | 1.1M | 0.001842066 |
| MIZAN | 1M | 0.001703975 |
| SCB_MT_EN_TH | 988K | 0.001648370 |
| MDN_Web_Docs | 874K | 0.001458614 |
| ECDC | 683K | 0.001140029 |
| Elhuyar | 642K | 0.001071404 |
| EiTB-ParCC | 637K | 0.001062789 |
| TEP | 612K | 0.001020930 |
| KDEdoc | 610K | 0.001018158 |
| WMT-News | 447K | 0.000746222 |
| KFTT | 440K | 0.000734376 |
| pmindia | 422K | 0.000704163 |
| tldr-pages | 370K | 0.000616360 |
| tico-19 | 319K | 0.000532717 |
| memat | 155K | 0.000258137 |
| hrenWaC | 99K | 0.000165129 |
| TedTalks | 86K | 0.000144024 |
| FFR | 82K | 0.000136997 |
| SPC | 68K | 0.000113020 |
| MontenegrinSubs | 65K | 0.000108487 |
| OfisPublik | 63K | 0.000105785 |
| XhosaNavy | 50K | 0.000083366 |
| Bianet | 48K | 0.000080891 |
| WikiSource | 33K | 0.000055514 |
| Salome | 9.4K | 0.000015722 |
| sardware | 6.2K | 0.000010258 |
| ada83 | 4.1K | 0.000006875 |
| InterdialectCorpus | 2.4K | 0.000004081 |
| RF | 1.2K | 0.000001928 |
| komi | Not specified | Not specified |
| liv4ever | Not specified | Not specified |
| Mozilla-I10n | Not specified | Not specified |
| Nunavut_Hansard | Not specified | Not specified |
| translatewiki | Not specified | Not specified |
| Ubuntu | Not specified | Not specified |
