Home / Query / WordAlign / Wiki    [ada83] [bible] [bianet] [books] [CAPES] [DGT] [DOGC] [ECB] [EhuHac] [Elhuyar] [EMEA] [EUbooks] [EU] [Europarl] [finlex] [fiskmö] [giga] [GNOME] [GlobalVoices] [hren] [JRC] [JW300] [KDE4/doc] [MBS] [memat] [MontenegrinSubs] [MultiUN] [NCv9/v11/v14] [Ofis] [OO/OO3] [subs/16/18] [ParaCrawl] [ParCor] [PHP] [QED] [sardware] [SciELO] [SETIMES] [SPC] [Tatoeba] [Tanzil] [TEP] [TedTalks] [TED] [Tilde] [Ubuntu] [UN] [UNPC] [Wikimedia] [Wikipedia] [WikiSource] [WMT] [XhosaNavy]

TildeMODEL

This is the Tilde MODEL Corpus – Multilingual Open Data for European Languages.

The data has been collected from sites allowing free use and reuse of its content, as well as from Public Sector web sites. The activities have been undertaken as part of the ODINE Open Data Incubator for Europe, which aims to support the next generation of digital businesses and fast-track the development of new products and services. The corpus includes the following parts:

30 languages, 274 bitexts
total number of files: 125
total number of tokens: 1.43G
total number of sentence fragments: 62.44M

Please cite the following article if you use any part of the corpus in your own work: Roberts Rozis, Raivis Skadins, 2017, Tilde MODEL - Multilingual Open Data for EU Languages. Proceedings of the 21th Nordic Conference of Computational Linguistics NODALIDA 2017.

Download

Below you can download data files for all language pairs in different formats and with different kind of annotation (if available). You can click on the various links as explained below. In addition to the files shown on this webpage, OPUS also provides pre-compiled word alignments and phrase tables, bilingual dictionaries, frequency counts, and these files can be found through the resources search form on the top-level website of OPUS.

License: CC-BY - Creative Commons with Attribution

Bottom-left triangle: download files
  • ces = sentence alignments in XCES format
  • leftmost column language IDs = tokenized corpus files in XML
  • TMX and plain text files (Moses): see "Statistics" below
  • lower row language IDs = parsed corpus files (if they exist)
Upper-right triangle: sample files
  • view = bilingual XML file samples
  • upper row language IDs = monolingual XML file samples
  • rightmost column language IDs = untokenized corpus files

bg cs da de el en es et fi fr hr hu is it lt lv mt nl no pl pt ro ru sk sl sq sr sv tr uk
bg viewviewviewviewviewviewviewviewviewviewviewviewviewbg viewviewviewviewviewviewviewviewviewbg
cs ces viewviewviewviewviewviewviewviewviewviewviewviewcs viewviewviewviewviewviewviewviewviewcs
da ces ces viewviewviewviewviewviewviewviewviewviewviewda viewviewviewviewviewviewviewviewviewda
de ces ces ces viewviewviewviewviewviewviewviewviewviewviewde viewviewviewviewviewviewviewviewviewviewviewde
el ces ces ces ces viewviewviewviewviewviewviewviewviewel viewviewviewviewviewviewviewviewviewel
en ces ces ces ces ces viewviewviewviewviewviewviewviewviewen viewviewviewviewviewviewviewviewviewviewviewviewviewviewviewen
es ces ces ces ces ces ces viewviewviewviewviewviewviewes viewviewviewviewviewviewviewviewviewviewes
et ces ces ces ces ces ces ces viewviewviewviewviewviewet viewviewviewviewviewviewviewviewviewviewet
fi ces ces ces ces ces ces ces ces viewviewviewviewviewfi viewviewviewviewviewviewviewviewviewviewfi
fr ces ces ces ces ces ces ces ces ces viewviewviewviewviewfr viewviewviewviewviewviewviewviewviewviewviewfr
hr ces ces ces ces ces ces ces ces ces ces viewviewviewhr viewviewviewviewviewviewviewviewviewhr
hu ces ces ces ces ces ces ces ces ces ces ces viewviewhu viewviewviewviewviewviewviewviewviewhu
is ces ces ces is is
it ces ces ces ces ces ces ces ces ces ces ces ces viewit viewviewviewviewviewviewviewviewviewviewit
lt ces ces ces ces ces ces ces ces ces ces ces ces ces lt viewviewviewviewviewviewviewviewviewviewlt
bg cs da de el en es et fi fr hr hu is it lt lv mt nl no pl pt ro ru sk sl sq sr sv tr uk
lv ces ces ces ces ces ces ces ces ces ces ces ces ces ces lv viewviewviewviewviewviewviewviewviewlv
mt ces ces ces ces ces ces ces ces ces ces ces ces ces ces mt ces viewviewviewviewviewviewviewmt
nl ces ces ces ces ces ces ces ces ces ces ces ces ces ces nl ces ces viewviewviewviewviewviewnl
no ces ces ces no no
pl ces ces ces ces ces ces ces ces ces ces ces ces ces ces pl ces ces ces viewviewviewviewviewviewpl
pt ces ces ces ces ces ces ces ces ces ces ces ces ces ces pt ces ces ces ces viewviewviewviewviewpt
ro ces ces ces ces ces ces ces ces ces ces ces ces ces ces ro ces ces ces ces ces viewviewviewro
ru ces ces ces ces ces ces ces ces ru ces ces ces ru
sk ces ces ces ces ces ces ces ces ces ces ces ces ces ces sk ces ces ces ces ces ces viewviewsk
sl ces ces ces ces ces ces ces ces ces ces ces ces ces ces sl ces ces ces ces ces ces ces viewsl
sq ces sq sq
sr ces sr sr
sv ces ces ces ces ces ces ces ces ces ces ces ces ces ces sv ces ces ces ces ces ces ces ces sv
tr ces tr tr
uk ces uk uk
bg cs da de el en es et fi fr hr hu is it lt lv mt nl no pl pt ro ru sk sl sq sr sv tr uk

Statistics and TMX/Moses Downloads

Number of files, tokens, and sentences per language (including non-parallel ones if they exist)
Number of sentence alignment units per language pair

Upper-right triangle: download translation memory files (TMX)
Bottom-left triangle: download plain text files (MOSES/GIZA++)
Language ID's, first row: monolingual plain text files (tokenized)
Language ID's, first column: monolingual plain text files (untokenized)

language files tokens sentencesbg cs da de el en es et fi fr hr hu is it lt lv mt nl no pl pt ro ru sk sl sq sr sv tr uk
bg 1 85.7k 4.3k3.1k 3.4k 2.7k 0.5k 2.9k 3.4k 3.3k 2.5k 3.5k 1.8k 3.4k 3.3k 3.4k 3.4k 3.5k 3.6k 3.3k 3.6k 3.6k 3.4k 3.5k 3.5k
cs 1 88.9k 5.2k3.1k 3.7k 2.6k 3.7k 3.1k 3.5k 3.5k 3.2k 3.7k 1.8k 3.6k 3.5k 3.6k 3.7k 3.4k 3.8k 3.5k 3.7k 3.1k 4.5k 3.8k 3.8k
da 1 0.1M 6.5k3.4k 3.7k 3.9k 5.0k 4.4k 4.8k 3.5k 4.4k 5.0k 1.6k 3.9k 4.6k 3.9k 4.1k 3.8k 5.2k 3.8k 5.0k 3.4k 3.8k 3.5k 5.2k
de 7 108.8M 5.1M2.7k 2.6k 3.9k 3.2M 4.2M 3.4M 2.0M 3.0M 4.3M 0.7M 1.9M 0.4M 3.4M 2.0M 2.0M 1.9M 3.4M 0.3M 2.0M 3.4M 1.8M 9.5k 2.1M 2.0M 3.1M
el 4 73.5M 3.0M0.5k 3.7k 5.0k 3.3M 5.2k 6.0k 4.2k 4.3k 6.2k 1.8k 4.1k 5.8k 4.0k 4.2k 4.6k 6.3k 4.0k 6.2k 3.6k 4.5k 4.9k 6.0k
en 11 134.9M 5.7M2.9k 3.1k 4.4k 4.3M 5.2k 3.7M 2.0M 3.0M 5.1M 0.7M 2.0M 0.4M 3.7M 2.1M 2.1M 2.0M 3.5M 0.3M 2.1M 1.1M 1.9M 11.0k 2.1M 2.0M 3.1k 2.0k 3.2M 1.6k 1.6k
es 6 102.3M 3.9M3.4k 3.5k 4.8k 3.4M 6.0k 3.8M 4.2k 5.0k 14.1k 1.8k 4.2k 13.6k 4.1k 4.4k 4.6k 6.9k 4.9k 13.5k 3.7k 6.8k 4.6k 5.1k 6.2k
et 5 36.1M 2.1M3.3k 3.5k 3.5k 2.0M 4.2k 2.1M 4.2k 4.7k 4.3k 1.5k 4.1k 4.0k 5.0k 5.4k 4.0k 4.4k 4.0k 4.3k 3.4k 1.0k 4.1k 4.2k 4.4k
fi 5 54.0M 3.2M2.5k 3.2k 4.4k 3.0M 4.3k 3.1M 5.0k 4.7k 4.4k 1.2k 2.9k 4.1k 3.7k 3.8k 2.8k 4.6k 2.8k 4.2k 2.3k 0.8k 3.1k 3.3k 4.7k
fr 6 156.7M 5.6M3.5k 3.7k 5.0k 4.3M 6.2k 5.1M 14.1k 4.3k 4.4k 0.8M 2.0M 0.4M 3.8M 2.1M 2.1M 2.0M 3.5M 0.4M 2.2M 3.6M 1.9M 6.9k 2.2M 2.0M 3.3M
hr 5 15.2M 0.8M1.8k 1.8k 1.6k 0.7M 1.8k 0.7M 1.8k 1.5k 1.2k 0.8M 1.7k 1.7k 1.6k 1.7k 1.7k 1.8k 1.6k 1.8k 1.8k 1.9k 1.8k 1.7k
hu 5 43.0M 2.1M3.4k 3.6k 3.9k 1.9M 4.1k 2.0M 4.2k 4.1k 2.9k 2.0M 1.7k 4.1k 4.1k 4.4k 4.0k 4.4k 4.0k 4.3k 3.6k 4.0k 4.0k 4.2k
is 3 7.4M 0.4M0.4M 0.4M 0.4M
it 5 102.6M 4.0M3.3k 3.5k 4.6k 3.5M 5.8k 3.8M 13.6k 4.0k 4.1k 3.8M 1.7k 4.1k 4.0k 4.2k 4.5k 6.5k 4.8k 13.2k 3.6k 6.8k 4.4k 4.9k 6.0k
lt 6 43.5M 2.2M3.4k 3.6k 3.9k 2.0M 4.0k 2.1M 4.1k 5.0k 3.7k 2.1M 1.6k 4.1k 4.0k 5.4k 3.9k 4.2k 4.1k 4.2k 3.4k 0.9k 4.1k 3.8k 4.1k
lv 7 44.0M 2.2M3.4k 3.7k 4.1k 2.0M 4.2k 2.1M 4.4k 5.4k 3.8k 2.1M 1.7k 4.4k 4.2k 5.4k 4.2k 4.5k 4.1k 4.5k 3.6k 3.8k 4.3k 4.3k 4.4k
mt 4 46.4M 2.1M3.5k 3.4k 3.8k 1.9M 4.6k 2.0M 4.6k 4.0k 2.8k 2.0M 1.7k 4.0k 4.5k 3.9k 4.2k 4.9k 3.9k 4.8k 3.6k 4.2k 4.5k 4.6k
nl 4 87.0M 3.8M3.6k 3.8k 5.2k 3.4M 6.3k 3.5M 6.9k 4.4k 4.6k 3.6M 1.8k 4.4k 6.5k 4.2k 4.5k 4.9k 4.2k 6.5k 3.8k 4.6k 5.2k 6.1k
no 2 5.9M 0.4M0.3M 0.3M 0.4M
pl 6 50.6M 2.3M3.3k 3.5k 3.8k 2.1M 4.0k 2.2M 4.9k 4.0k 2.8k 2.2M 1.6k 4.0k 4.8k 4.1k 4.1k 3.9k 4.2k 5.0k 3.5k 1.1k 4.0k 4.0k 4.1k
pt 6 100.4M 3.8M3.6k 3.7k 5.0k 3.4M 6.2k 1.1M 13.5k 4.3k 4.2k 3.7M 1.8k 4.3k 13.2k 4.2k 4.5k 4.8k 6.5k 5.0k 3.8k 6.7k 4.7k 5.2k 6.3k
ro 5 50.8M 2.0M3.6k 3.2k 3.4k 1.8M 3.6k 1.9M 3.8k 3.4k 2.4k 2.0M 1.9k 3.7k 3.7k 3.5k 3.6k 3.6k 3.9k 3.5k 3.9k 3.5k 3.7k 3.7k
ru 4 0.9M 40.3k9.5k 11.0k 6.8k 1.0k 0.8k 6.9k 6.8k 0.9k 3.8k 1.1k 6.7k
sk 4 49.0M 2.3M3.4k 4.5k 3.8k 2.1M 4.5k 2.2M 4.6k 4.1k 3.1k 2.2M 1.9k 4.0k 4.4k 4.1k 4.3k 4.2k 4.6k 4.0k 4.7k 3.6k 4.5k 4.5k
sl 4 46.9M 2.1M3.5k 3.8k 3.5k 2.0M 4.9k 2.0M 5.1k 4.2k 3.3k 2.1M 1.8k 4.0k 4.9k 3.8k 4.3k 4.5k 5.2k 4.0k 5.2k 3.8k 4.5k 5.0k
sq 1 98.4k 3.0k3.1k
sr 1 51.1k 2.0k2.0k
sv 4 72.0M 3.4M3.5k 3.8k 5.2k 3.1M 6.0k 3.2M 6.2k 4.4k 4.7k 3.3M 1.7k 4.2k 6.0k 4.1k 4.4k 4.6k 6.1k 4.1k 6.3k 3.7k 4.5k 5.0k
tr 1 40.2k 1.6k1.6k
uk 1 39.6k 1.6k1.6k

Note that TMX files only contain unique translation units and, therefore, the number of aligned units is smaller than for the distributions in Moses and XML format. Moses downloads include all non-empty alignment units including duplicates. Token counts for each language also include duplicate sentences and documents.