Wiki v1syn
Automatically translated data sets that can be used for data augmentation Translations have been done with models trained on the Tatoeba MT challenge data. We include translations of Wikipedia, WikiSource, WikiBooks, WikiNews and WikiQuote (if available for the source language we translate from). Translations are done on shuffled, de-duplicated data sets and they come in blocks of at most one million sentences per file. The original datasets are taken from cirrussearch wiki dumps. The original back-translations have been checked with the heliport language identification tool and mismatched sentence pairs are excluded from this distribution.165 languages, 172 bitexts
total number of files: 601
total number of tokens: 13.0G
total number of sentences: 706.4M
Please, acknowledge the Wikimedia Foundation for the data and cite the following paper if you use data from this distribution: @inproceedings{tiedemann-2020-ttc,
title = "The {T}atoeba {T}ranslation {C}hallenge -- {R}ealistic Data Sets for Low Resource and Multilingual {MT}",
author = {Tiedemann, J{"o}rg},
booktitle = "Proceedings of the Fifth Conference on Machine Translation (Volume 1: Research Papers)", year = "2020",
publisher = "Association for Computational Linguistics",
url = {https://arxiv.org/abs/2010.06354}
}
Download
Below you can download data files for all language pairs in different formats and with different kind of annotation (if available). You can click on the various links as explained below.
License: CC-BY-SA 4.0
Links to sample files from monolingual corpus files (language-ID links) and bitexts (click on 'view')
You need to download the monolingual corpus files and the standoff alignment files between them:
- Links on the language IDs of the top row and first column: zip-files of untokenized monolingual XML files
- Links on the language IDs of the bottom row and last column: zip-files of tokenized monolingual XML files (if they exist)
- Links in the table: Sentence alignment files in XCES Align format (standoff annotation)
Links to zip-files with aligned plain text files, one per language (Moses format).
Links to compressed TMX files, one per language pair.
|
de |
en |
et |
fi |
ru |
sv |
|
ab |
|
0 |
|
|
|
|
ab |
ady |
|
0 |
|
|
|
|
ady |
af |
|
0 |
|
|
|
|
af |
am |
|
0 |
|
|
|
|
am |
an |
|
0 |
|
|
|
|
an |
ang |
|
0 |
|
|
|
|
ang |
ar |
|
0 |
|
|
|
|
ar |
as |
|
0 |
|
|
|
|
as |
ast |
|
0 |
|
|
|
|
ast |
awa |
|
0 |
|
|
|
|
awa |
az |
|
0 |
|
|
|
|
az |
ba |
|
0 |
|
|
|
|
ba |
bar |
|
0 |
|
|
|
|
bar |
be |
|
0 |
|
|
|
|
be |
bg |
|
0 |
|
|
|
|
bg |
bm |
|
0 |
|
|
|
|
bm |
bn |
|
0 |
|
|
|
|
bn |
bo |
|
0 |
|
|
|
|
bo |
br |
|
0 |
|
|
|
|
br |
ca |
|
0 |
|
|
|
|
ca |
ce |
|
0 |
|
|
|
|
ce |
chr |
|
0 |
|
|
|
|
chr |
co |
|
0 |
|
|
|
|
co |
crh |
|
0 |
|
|
|
|
crh |
cs |
|
0 |
|
|
|
|
cs |
|
de |
en |
et |
fi |
ru |
sv |
|
csb |
|
0 |
|
|
|
|
csb |
cy |
|
0 |
|
|
|
|
cy |
da |
|
0 |
|
|
|
|
da |
de |
|
0 |
|
0 |
|
|
de |
dsb |
|
0 |
|
|
|
|
dsb |
el |
|
0 |
|
|
|
|
el |
eo |
|
0 |
|
|
|
|
eo |
es |
|
0 |
|
|
|
|
es |
et |
|
|
|
0 |
|
|
et |
eu |
|
0 |
|
|
|
|
eu |
ext |
|
0 |
|
|
|
|
ext |
fa |
|
0 |
|
|
|
|
fa |
ff |
|
0 |
|
|
|
|
ff |
fi |
0 |
0 |
0 |
|
0 |
0 |
fi |
fj |
|
0 |
|
|
|
|
fj |
fo |
|
0 |
|
|
|
|
fo |
fr |
|
0 |
|
|
|
|
fr |
frr |
|
0 |
|
|
|
|
frr |
fy |
|
0 |
|
|
|
|
fy |
ga |
|
0 |
|
|
|
|
ga |
gl |
|
0 |
|
|
|
|
gl |
gn |
|
0 |
|
|
|
|
gn |
got |
|
0 |
|
|
|
|
got |
gu |
|
0 |
|
|
|
|
gu |
gv |
|
0 |
|
|
|
|
gv |
haw |
|
0 |
|
|
|
|
haw |
|
de |
en |
et |
fi |
ru |
sv |
|
hif |
|
0 |
|
|
|
|
hif |
hsb |
|
0 |
|
|
|
|
hsb |
ht |
|
0 |
|
|
|
|
ht |
hu |
|
0 |
|
|
|
|
hu |
hy |
|
0 |
|
|
|
|
hy |
ia |
|
0 |
|
|
|
|
ia |
ie |
|
0 |
|
|
|
|
ie |
ig |
|
0 |
|
|
|
|
ig |
ilo |
|
0 |
|
|
|
|
ilo |
io |
|
0 |
|
|
|
|
io |
it |
|
0 |
|
|
|
|
it |
iu |
|
0 |
|
|
|
|
iu |
jam |
|
0 |
|
|
|
|
jam |
jbo |
|
0 |
|
|
|
|
jbo |
jv |
|
0 |
|
|
|
|
jv |
ka |
|
0 |
|
|
|
|
ka |
kab |
|
0 |
|
|
|
|
kab |
kk |
|
0 |
|
|
|
|
kk |
kl |
|
0 |
|
|
|
|
kl |
km |
|
0 |
|
|
|
|
km |
kn |
|
0 |
|
|
|
|
kn |
ko |
|
0 |
|
|
|
|
ko |
ku |
|
0 |
|
|
|
|
ku |
kv |
|
0 |
|
|
|
|
kv |
kw |
|
0 |
|
|
|
|
kw |
lad |
|
0 |
|
|
|
|
lad |
|
de |
en |
et |
fi |
ru |
sv |
|
lb |
|
0 |
|
|
|
|
lb |
lfn |
|
0 |
|
|
|
|
lfn |
lg |
|
0 |
|
|
|
|
lg |
lij |
|
0 |
|
|
|
|
lij |
lmo |
|
0 |
|
|
|
|
lmo |
ln |
|
0 |
|
|
|
|
ln |
lo |
|
0 |
|
|
|
|
lo |
lt |
|
0 |
|
0 |
|
|
lt |
lv |
|
0 |
|
|
|
|
lv |
mai |
|
0 |
|
|
|
|
mai |
mdf |
|
0 |
|
|
|
|
mdf |
mg |
|
0 |
|
|
|
|
mg |
mh |
|
0 |
|
|
|
|
mh |
mk |
|
0 |
|
|
|
|
mk |
ml |
|
0 |
|
|
|
|
ml |
mn |
|
0 |
|
|
|
|
mn |
mnw |
|
0 |
|
|
|
|
mnw |
mr |
|
0 |
|
|
|
|
mr |
ms |
|
0 |
|
|
|
|
ms |
mt |
|
0 |
|
|
|
|
mt |
mwl |
|
0 |
|
|
|
|
mwl |
my |
|
0 |
|
|
|
|
my |
myv |
|
0 |
|
|
|
|
myv |
na |
|
0 |
|
|
|
|
na |
nds |
|
0 |
|
|
|
|
nds |
nl |
|
0 |
|
|
|
|
nl |
|
de |
en |
et |
fi |
ru |
sv |
|
no |
|
0 |
|
|
|
|
no |
nov |
|
0 |
|
|
|
|
nov |
ny |
|
0 |
|
|
|
|
ny |
oc |
|
0 |
|
|
|
|
oc |
or |
|
0 |
|
|
|
|
or |
pa |
|
0 |
|
|
|
|
pa |
pag |
|
0 |
|
|
|
|
pag |
pap |
|
0 |
|
|
|
|
pap |
pdc |
|
0 |
|
|
|
|
pdc |
pl |
|
0 |
|
|
|
|
pl |
pms |
|
0 |
|
|
|
|
pms |
ps |
|
0 |
|
|
|
|
ps |
pt |
|
0 |
|
|
|
|
pt |
rm |
|
0 |
|
|
|
|
rm |
rn |
|
0 |
|
|
|
|
rn |
ro |
|
0 |
|
|
|
|
ro |
ru |
|
0 |
|
0 |
|
|
ru |
rw |
|
0 |
|
|
|
|
rw |
sa |
|
0 |
|
|
|
|
sa |
sah |
|
0 |
|
|
|
|
sah |
scn |
|
0 |
|
|
|
|
scn |
sco |
|
0 |
|
|
|
|
sco |
sd |
|
0 |
|
|
|
|
sd |
se |
|
0 |
|
|
|
|
se |
sg |
|
0 |
|
|
|
|
sg |
si |
|
0 |
|
|
|
|
si |
|
de |
en |
et |
fi |
ru |
sv |
|
sl |
|
0 |
|
|
|
|
sl |
sq |
|
0 |
|
|
|
|
sq |
stq |
|
0 |
|
|
|
|
stq |
sv |
|
0 |
|
0 |
|
|
sv |
sw |
|
0 |
|
|
|
|
sw |
ta |
|
0 |
|
|
|
|
ta |
te |
|
0 |
|
|
|
|
te |
tet |
|
0 |
|
|
|
|
tet |
tg |
|
0 |
|
|
|
|
tg |
th |
|
0 |
|
|
|
|
th |
ti |
|
0 |
|
|
|
|
ti |
tk |
|
0 |
|
|
|
|
tk |
tl |
|
0 |
|
|
|
|
tl |
to |
|
0 |
|
|
|
|
to |
tpi |
|
0 |
|
|
|
|
tpi |
tr |
|
0 |
|
|
|
|
tr |
ts |
|
0 |
|
|
|
|
ts |
tt |
|
0 |
|
|
|
|
tt |
ty |
|
0 |
|
|
|
|
ty |
tyv |
|
0 |
|
|
|
|
tyv |
udm |
|
0 |
|
|
|
|
udm |
uk |
|
0 |
|
|
|
|
uk |
ur |
|
0 |
|
|
|
|
ur |
uz |
|
0 |
|
|
|
|
uz |
vec |
|
0 |
|
|
|
|
vec |
vi |
|
0 |
|
|
|
|
vi |
|
de |
en |
et |
fi |
ru |
sv |
|
vo |
|
0 |
|
|
|
|
vo |
wa |
|
0 |
|
|
|
|
wa |
wo |
|
0 |
|
|
|
|
wo |
xal |
|
0 |
|
|
|
|
xal |
xh |
|
0 |
|
|
|
|
xh |
yi |
|
0 |
|
|
|
|
yi |
yo |
|
0 |
|
|
|
|
yo |
zh |
|
0 |
|
|
|
|
zh |
zu |
|
0 |
|
|
|
|
zu |
|
de |
en |
et |
fi |
ru |
sv |
|
Disclaimer
- We do not own any of the text from which the data has been extracted.
- We only offer files that we believe we are free to redistribute. If any doubt occurs about the legality of any of our file downloads we will take them off right away after contacting us.
Notice and take down policy
Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: - Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
- Clearly identify the copyrighted work claimed to be infringed.
- Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.
- And contact the OPUS project at the following email address: opus-project at helsinki.fi.
Take down: We will comply to legitimate requests by removing the affected sources from the next release of the corpus.