MT560 - A Many-to-English Machine Translation Dataset

• Github: https://github.com/thammegowda/006-many-to-eng
• Paper: https://arxiv.org/abs/2104.00290
• Website: http://rtg.isi.edu/many-eng/

Files

• train.raw.tsv.gz : Training data in raw form, before cleaning, deduping and tokenization

• train.v1.eng.tok.gz: good training data, tokenized; english side
• train.v1.src.tok.gz: good training data, tokenized; source size
• train.v1.lang.gz : lang ID of source side sentences
• train.v1.prov.gz : provinance of training record (if you want to see where we got this record)

• train.v1.tok.stats.tsv: stats such as sentence and token count per language

• devs-combo-shuf10k-raw+tok.tgz: dev data that has random 10K sentences for validation
• devtests-raw+tok.tgz : all the dev and test data; both raw and tokenized
• citations.bib: BibTeX of articles which published the datastes collected in this work

Data Size

(head -1 && tail -1) <  train.v1.tok.stats.tsv
Lang    Sents   Source  English
Total   473791770       9001780032      9072887211

$ cut -f1  train.v1.tok.stats.tsv  | grep  '^[a-z]' | wc -l
560

cat  train.v1.tok.stats.tsv  | grep  '^[a-z]' | less  | awk -F '\t' 'int($2) >= 10000' |wc -l
334

cat  train.v1.tok.stats.tsv  | grep  '^[a-z]' | less  | awk -F '\t' 'int($3) >= 1000000' |wc -l
214

Acknowledgements

• OPUS -- Thanks ^ billion
• JW300
• OPUS100
• StatMT.org -- Thanks ^ Billion
• Lots and lots of thanks for the test sets
• NeuLab for TEDTalks
• Japanese-English dataset
• Wikimatrix
• Joshua Indian Parallel Corpus
• PMIndia Corpus
• Paracrawl
• IITB Hindi-English
• TILDE MODEL