Wikiv1synSynthetic
Automatically translated data sets that can be used for data augmentation Translations have been done with models trained on the Tatoeba MT challenge data. We include translations of Wikipedia, WikiSource, WikiBooks, WikiNews and WikiQuote (if available for the source language we translate from). Translations are done on shuffled, de-duplicated data sets and they come in blocks of at most one million sentences per file. The original datasets are taken from cirrussearch wiki dumps. The original back-translations have been checked with the heliport language identification tool and mismatched sentence pairs are excluded from this distribution.
Please, acknowledge the Wikimedia Foundation for the data and cite the following paper if you use data from this distribution:
@inproceedings{tiedemann-2020-ttc,
title = "The {T}atoeba {T}ranslation {C}hallenge -- {R}ealistic Data Sets for Low Resource and Multilingual {MT}",
author = {Tiedemann, J{\"o}rg},
booktitle = "Proceedings of the Fifth Conference on Machine Translation (Volume 1: Research Papers)", year = "2020",
publisher = "Association for Computational Linguistics",
url = {https://arxiv.org/abs/2010.06354}
}
Wiki's Numbers
| Languages | Bitexts | Files | Tokens | Sentence fragments |
|---|---|---|---|---|
| 165 | 172 | 601 | 13G | 674M |
Language pairs in Wiki – v1syn
Click a bar to select a pair below.
Downloads
Pick a language pair to see available downloads.
Disclaimer
- We do not own any of the text from which the data has been extracted.
- We only offer files that we believe we are free to redistribute. If any doubt occurs about the legality of any of our file downloads we will take them off right away after contacting us.
Notice and take down policy
Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
- Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
- Clearly identify the copyrighted work claimed to be infringed.
- Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.
- And contact the OPUS project at: opus-project at helsinki.fi.
Take down: We will comply to legitimate requests by removing the affected sources from the next release of the corpus.