MultiCCAligned

This corpus was created from 68 Commoncrawl Snapshots (up until March 2020). The documents are split into sentences based on punctuations and deduplication is performed. No claims of intellectual property are made on the work of preparation of the corpus. The original distribution is available from http://www.statmt.org/cc-aligned/
CCAligned consists of parallel or comparable web-document pairs in 137 languages aligned with English. These web-document pairs were constructed by performing language identification on raw web-documents, and ensuring corresponding language codes were corresponding in the URLs of web documents. This pattern matching approach yielded more than 100 million aligned documents paired with English. Recognizing that each English document was often aligned to mulitple documents in different target language, we can join on English documents to obtain aligned documents that directly pair two non-English documents (e.g., Arabic-French).
Sentence pairs were extracted using similarity scores of LASER embeddings from the document pairs (minimum similarity 1.04, sorted based on decreasing similarity score). It misses some languages not covered by LASER.
This collection has been further processed for making it a multi-parallel corpus by pivoting via English. The original bitexts for English-centric data are available from the CCAligned release. The difference to version 1 is that pivoting now only uses the link with best score in case of alternative alignments for a pivot sentence.

If you use the dataset or code, please cite (pdf), and, please, acknowledge OPUS (bib, pdf) as well for this service. For more information on the sentence pair mining method, see Chaudhary et al., WMT 2019 (bib, pdf). Pivoting is done using OpusTools, see Aulamo et al., LREC 2020 (bib, pdf)