Home / Wiki    [Europarl] [Wiki] [Wikibooks] [Wikinews] [Wikipedia] [Wikiquote] [Wikisource]

Wiki v1syn

Automatically translated data sets that can be used for data augmentation Translations have been done with models trained on the Tatoeba MT challenge data. We include translations of Wikipedia, WikiSource, WikiBooks, WikiNews and WikiQuote (if available for the source language we translate from). Translations are done on shuffled, de-duplicated data sets and they come in blocks of at most one million sentences per file. The original datasets are taken from cirrussearch wiki dumps. The original back-translations have been checked with the heliport language identification tool and mismatched sentence pairs are excluded from this distribution.

165 languages, 172 bitexts
total number of files: 601
total number of tokens: 13.0G
total number of sentences: 706.4M

Please, acknowledge the Wikimedia Foundation for the data and cite the following paper if you use data from this distribution:
 @inproceedings{tiedemann-2020-ttc,
title = "The {T}atoeba {T}ranslation {C}hallenge -- {R}ealistic Data Sets for Low Resource and Multilingual {MT}",
author = {Tiedemann, J{"o}rg},
booktitle = "Proceedings of the Fifth Conference on Machine Translation (Volume 1: Research Papers)", year = "2020",
publisher = "Association for Computational Linguistics",
url = {https://arxiv.org/abs/2010.06354}
}

Download

Below you can download data files for all language pairs in different formats and with different kind of annotation (if available). You can click on the various links as explained below.

License: CC-BY-SA 4.0

You need to download the monolingual corpus files and the standoff alignment files between them:

de en et fi ru sv
ab 11.3k ab
ady 2.6k ady
af 1.0M af
am 0.3k am
an 0.1M an
ang 2.3k ang
ar 6.7M ar
as 0.7k as
ast 1.7M ast
awa 2 awa
az 1.7M az
ba 0.5M ba
bar 0.1M bar
be 1.7M be
bg 3.3M bg
bm 1.8k bm
bn 5.3k bn
bo 27 bo
br 0.4M br
ca 7.0M ca
ce 0.4M ce
chr 2.2k chr
co 22.3k co
crh 14.8k crh
cs 7.0M cs
de en et fi ru sv
csb 18.7k csb
cy 0.5M cy
da 2.9M da
de 44.4M 36.0M de
dsb 13.5k dsb
el 3.0M el
eo 2.0M eo
es 20.7M es
et 2.3M et
eu 2.3M eu
ext 18.3k ext
fa 3.9M fa
ff 1.9k ff
fi 5.8M 7.1M 6.7M 6.8M 7.2M fi
fj 2.3k fj
fo 95.5k fo
fr 28.7M fr
frr 21.7k frr
fy 0.5M fy
ga 0.2M ga
gl 1.9M gl
gn 29.3k gn
got 0.1k got
gu 0.2M gu
gv 26.7k gv
haw 7.7k haw
de en et fi ru sv
hif 7.5k hif
hsb 58.0k hsb
ht 78.5k ht
hu 7.1M hu
hy 74.0k hy
ia 28.9k ia
ie 11.4k ie
ig 6.3k ig
ilo 67.3k ilo
io 0.2M io
it 16.0M it
iu 14 iu
jam 1.5k jam
jbo 15.3k jbo
jv 0.3M jv
ka 0.2M ka
kab 10.3k kab
kk 1.5M kk
kl 2.1k kl
km 0.7k km
kn 13.5k kn
ko 0.6M ko
ku 15 ku
kv 29.4k kv
kw 9.4k kw
lad 20.1k lad
de en et fi ru sv
lb 0.4M lb
lfn 33.3k lfn
lg 15.6k lg
lij 11.0k lij
lmo 0.1M lmo
ln 7.0k ln
lo 0.9k lo
lt 1.9M 1.9M lt
lv 1.2M lv
mai 0.3k mai
mdf 5.4k mdf
mg 0.2M mg
mh 24 mh
mk 1.2M mk
ml 91.7k ml
mn 0.2M mn
mnw 0.4k mnw
mr 0.6M mr
ms 1.5M ms
mt 71.0k mt
mwl 63.8k mwl
my 0.9k my
myv 34.4k myv
na 1.2k na
nds 0.3M nds
nl 13.3M nl
de en et fi ru sv
no 4.6M no
nov 2.1k nov
ny 4.6k ny
oc 0.4M oc
or 3.1k or
pa 2.7k pa
pag 2.5k pag
pap 8.1k pap
pdc 4.4k pdc
pl 15.1M pl
pms 0.3k pms
ps 2.1k ps
pt 11.5M pt
rm 76.8k rm
rn 2.5k rn
ro 3.1M ro
ru 20.6M 19.9M ru
rw 3.4k rw
sa 0.3k sa
sah 0.1M sah
scn 56.4k scn
sco 0.1M sco
sd 4.1k sd
se 17.0k se
sg 0.5k sg
si 16.3k si
de en et fi ru sv
sl 1.7M sl
sq 0.7M sq
stq 13.7k stq
sv 14.1M 13.1M sv
sw 0.2M sw
ta 0.1M ta
te 1.2M te
tet 4.6k tet
tg 0.2M tg
th 0.3M th
ti 26 ti
tk 76.3k tk
tl 0.3M tl
to 2.3k to
tpi 1.0k tpi
tr 3.8M tr
ts 4.1k ts
tt 0.4M tt
ty 0.8k ty
tyv 40.4k tyv
udm 28.8k udm
uk 13.5M uk
ur 0.7M ur
uz 0.6M uz
vec 51.0k vec
vi 5.9M vi
de en et fi ru sv
vo 0.3M vo
wa 48.1k wa
wo 10.9k wo
xal 4.6k xal
xh 8.7k xh
yi 3.0k yi
yo 40.7k yo
zh 6.2M zh
zu 6.7k zu
de en et fi ru sv


Disclaimer

Notice and take down policy

Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please: Take down: We will comply to legitimate requests by removing the affected sources from the next release of the corpus.