Home / Wiki    [Europarl] [Wiki] [Wikibooks] [Wikinews] [Wikipedia] [Wikiquote] [Wikisource]

Wikipedia v1syn

Automatically translated data sets that can be used for data augmentation Translations have been done with models trained on the Tatoeba MT challenge data. We include translations of Wikipedia, WikiSource, WikiBooks, WikiNews and WikiQuote (if available for the source language we translate from). Translations are done on shuffled, de-duplicated data sets and they come in blocks of at most one million sentences per file. The original datasets are taken from cirrussearch wiki dumps. The original back-translations have been checked with the heliport language identification tool and mismatched sentence pairs are excluded from this distribution.

62 languages, 80 bitexts
total number of files: 448
total number of tokens: 7.5G
total number of sentences: 447.7M

Please, acknowledge the Wikimedia Foundation for the data and cite the following paper if you use data from this distribution:
 @inproceedings{tiedemann-2020-ttc,
title = "The {T}atoeba {T}ranslation {C}hallenge -- {R}ealistic Data Sets for Low Resource and Multilingual {MT}",
author = {Tiedemann, J{"o}rg},
booktitle = "Proceedings of the Fifth Conference on Machine Translation (Volume 1: Research Papers)", year = "2020",
publisher = "Association for Computational Linguistics",
url = {https://arxiv.org/abs/2010.06354}
}

Release history:

Download

Below you can download data files for all language pairs in different formats and with different kind of annotation (if available). You can click on the various links as explained below.

License: CC-BY-SA 4.0

You need to download the monolingual corpus files and the standoff alignment files between them:

af ar az bg bn br bs ca ceb cs cy da de el en eo et eu fi fr fy ga gl he hr hu hy ia id ilo it lb lt lv mk ml ms mt nb nds nn pl pt ro ru sk sq sr_Cyrl sr_Latn sv sw ta th tl tr uk ur uz war zh zh_Hans zh_Hant
bs 1.1M bs bs bs
ceb 15.3M ceb ceb ceb
cs cs cs 7.4M cs
en 3.0M 3.5M 19.4k 3.5M 3.6M 3.1M 3.0M 3.1M 3.0M 3.0M 3.1M 3.0M 2.8M 3.1M 3.0M 2.9M 3.0M 3.0M 3.2M 3.1M 3.4M 2.5M 3.5M 2.9M en 2.8M 3.3M 36.3k 2.7M 3.2M 3.0M 2.9M 3.1M 2.2M 3.2M 3.8M 2.8M 3.2M 2.8M 0.2M 2.7M 3.2M 3.1M 2.2M 2.3M 3.0M 2.7M 3.2M 3.3M en 3.5M 3.3M 3.1M 2.3M 2.2M 3.1M 2.8M 3.1M 3.2M 3.3M 3.2M en
hr 2.7M hr hr hr
hu hu hu 0.9M hu
id 4.9M id id id
nb 5.7M nb nb nb
pl pl pl 8.4M pl
ro ro ro 3.1M ro
sk sk sk 2.0M sk
sr_Cyrl 5.2M sr_Cyrl sr_Cyrl sr_Cyrl
uk 13.2M 12.8M 12.8M uk 13.3M 13.9M 13.6M 13.3M 13.7M uk uk
zh_Hans 6.5M zh_Hans zh_Hans zh_Hans
zh_Hant 58.3k zh_Hant zh_Hant zh_Hant
af ar az bg bn br bs ca ceb cs cy da de el en eo et eu fi fr fy ga gl he hr hu hy ia id ilo it lb lt lv mk ml ms mt nb nds nn pl pt ro ru sk sq sr_Cyrl sr_Latn sv sw ta th tl tr uk ur uz war zh zh_Hans zh_Hant


Disclaimer

Notice and take down policy

Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: Take down: We will comply to legitimate requests by removing the affected sources from the next release of the corpus.