Resources

• Corpora
• TEP: Tehran English-Persian Parallel Corpus
• First free Eng-Per corpus
• 4-million tokens on each side
• Sentence Aligned


• Extracted from movie subtitles
• Text domain: informal/conversational
• Total alinged movie subtitles: 1600
• Total number of bilingual sentences: 612086
• Average sentence length: 7.8 words
• Corpus size (ignoring punctuations): About four million words
• Unique words on Persian side: 114275 (17605 with freq. > 10)
• Unique words on English side: 73002 (12716 with freq. > 10)



Note: Usage of this package for any research or non-commercial purposes requires the precondition that you cite the following paper:



• M. T. Pilevar, H. Faili, and A. H. Pilevar, “TEP: Tehran English-Persian Parallel Corpus”, in proceedings of 12th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2011).


• Download TEP corpus, for password please contact us at: nlp {at} ece.ut.ac.ir




• TMC: Tehran Monolingual Corpus
• Largest freely available monolingual corpus for Persian language
• Tokenized
• Suitable for Language Modeling
• More than 250M words in total, ~300K unique words of freq. > 1


• Download TMC corpus, for password please contact us at: nlp {at} ece.ut.ac.ir
  



• Mutual Information

We calculated pair-wise mutual information between english words using 2 gigabytes of Wikipedia documents. This pakage contains two files ‘WordID.txt’ and ‘MI.txt’. In ‘MI.txt’ for each English word a list of english words that have most mutual information with the first English word have been considered. Format of each line of this file is (English word Id:English word Id, mi;English word Id, mi; …), where English word Id come from ‘WordID.txt’.

Pair-wise mutual information between Persian words also have been calculated using Hamshahri and Irna text corpus and have same format as English.



• Download pair-wise mutual information between English words


• Download pair-wise mutual information between Persian words



• Grammar and context sensitive spell checker

Here is a real-world test set for grammatical errors and context sensitive spelling errors for Persian language






• Download Test set for grammatical errors and context sensitive spelling errors of Persian language

[Back to Top]