OPUS - an open source parallel corpus

Tools for processing OPUS corpora

Using OPUS corpora with Uplug is very straightforward. Here is a small selection of some simple tools to process parallel corpora from OPUS:

• uplug/tools/readalign

  A simple Perl script to read sentence aligned OPUS corpora and printing them in plain text format to STDOUT. It can also add some very simple HTML tags to show sentence alignments on websites.
  Example usage:
  

  /path/to/uplug/tools/readalign xml/de-sv.ces.gz | less

  (Note that you have to run readalign in the home directory of the corpus in order to match the relative file paths specified in the sentence alignment file xml/de-sv.ces.gz)

• uplug/tools/opus2moses.pl

  A simple Perl script (requires XML::Parser) to convert sentence aligned OPUS corpora to Moses / GIZA++ input format (two separate files for source and target language, one sentence per line, aligned sentences on corresponding lines).
  Example usage:
  

  zcat OPUS/corpus/Europarl3/xml/de-sv.ces.gz | uplug/tools/opus2moses.pl -d OPUS/corpus/Europarl3 -s word:lem:pos -t pos:word -e de-sv.src -f de-sv.trg

  (This will read aligned sentences from the German-Swedish Europarl corpus and writes factors word+lemma+POS-tag for German and factors POS-tag+word for Swedish to the files de-sv.src (German) and de-sv.trg (Swedish))

• uplug/tools/opus-indexer.pl

  Yet another Perl script, this one for indexing OPUS corpora with the Corpus Work Bench (with support for sentence alignment)

Tools used for building OPUS

The following tools have been used for pre-processing, annotation & alignment (not including standard GNU-tools):

• Uplug - tokenizer, sentence-splitter, XML-tools, sentence aligner, word aligner, corpus indexer (using CWB), web search interfaces
  https://bitbucket.org/tiedemann/uplug


• align - sentence aligner (based on Gale&Church, 1993) integrated in Uplug


• hunalign - another sentence aligner also integrated in Uplug


• srt2xml.pl & srtalign.pl - special scripts to convert and align movie subtitles (also included in the latest versions of Uplug)


• OpenNLP & Grok, Jason Baldridge and Gann Bierner
  http://grok.sourceforge.net/
  
  

  tool	language	trained on	trained by
  tagger	English	WSJ+Brown	Gann Bierner
  chunker	English	Penn Tree Bank	Jörg Tiedemann

  
  
• TnT - Statistical Part-of-Speech Tagging, Thorsten Brants
  http://www.coli.uni-sb.de/~thorsten/tnt/
  
  

  tool	language	trained on	trained by
  tagger	German	NEGRA	Thorsten Brants
  tagger	English	WSJ	Thorsten Brants
  tagger	Swedish	SUC	Beáta Megyesi

  
  
• TreeTagger - Decision Tree Tagger, Helmut Schmid
  http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/
  (tokenizer, tagger, lemmatizer)
  
  

  language	trained on	trained by
  German	NEGRA	Helmut Schmid
  English	WSJ	Helmut Schmid
  French		Achim Stein
  Italian		Achim Stein
  Spanish
  Dutch

  
  
• Alpino and Alpino tools for tokenizing (Europarl and OpenSubtitles only) and parsing Dutch (Europarl only)


• TextCat - for language identification (filtering) of movie subtitles


• ChaSen - japanese tokenizer + tagger
  http://chasen.aist-nara.ac.jp/
  (tokenizer POS-tagger lemmatizer sentence splitter)


• QTag - a probabilistic parts-of-speech tagger, Oliver Mason
  http://web.bham.ac.uk/o.mason/software/tagger/
  
  

  language	trained on	trained by
  English	1 Mio words	Oliver Mason
  Portuguese	500 Mio words	Tony Berber Sardinha and Rod de Lima-Lopes

  
  
• recode - convert between various character encodings
  http://www.iro.umontreal.ca/contrib/recode/HTML/
• tidy - validate, correct, and pretty-print XML-files
  http://www.w3.org/People/Raggett/tidy/

The following tools are used for data management:

• IMS Corpus Workbench (CWB)
  http://www.ims.uni-stuttgart.de/projekte/CorpusWorkbench/index.html