The Classical Language Toolkit

For @Akhilesh28. (Please assign this to yourself.)

In Sanskrit, a stopword list would include, at least: pronouns and determiners (source), and upasarga (verbal prefix / "preverb" / "preposition") and nipāta (particle) (which I read about here). Also add anything like conjunctions, particles, and interjections.

Just putting together this list shouldn't take more than one week. Let us know if you're having problems. You can post your stopwords here, first, as a "gist": https://gist.github.com/

This ticket is for Jack Duff, with @jtauber generously assisting.

The basic idea is to make a map of Greek letters and their IPA equivalents, something like:

{'α': 'a',
'αι', 'ai',
'ζ': 'zd',
'θ': 'tʰ'}

Obviously, it won't all be so easy, due to proximal characters changing pronunciation (for example, "γ" being IPA "ɡ" but before ["κ", "χ", "γ", "μ"] becoming "ŋ").

If you can get this down for Attic, then consider moving on to other dialects, like Ionic or Koine.

Within the CLTK's architecture, the transliteration maps and logic should go into something like cltk/phonetics/greek/transcription.py. Or consider making a general transcription entry point at cltk/phonetics/transcription.py and then declaring a which language and dialect. I'll leave the implementation details to you two, though.

• ~सः (He)~
• ~स्वयम्(himself)~
• तदीय(theres) -आसम्(be)
• ज्ञा (have) -परि (with) -शक्नोति(can(verb)) -यद्(if) -कतम(which)

add all the words in all their different cases, gender and and all 3 numbers(sin, dual, plural) . If you are doing it right, there must be 72 words exactly for each entity(including a few repetitions). Needs to be careful when it come to verb's word form, they are entirely different structures.

File at: https://github.com/cltk/cltk/blob/master/cltk/stop/sanskrit/stops.py

• I am Scapping Sanskrit - English data from
  • Srimad-bhagavadgita : http://www.gitasupersite.iitk.ac.in/srimad
  • Valmiki Ramayana : http://www.valmiki.iitk.ac.in/

Ping @kylepjohnson

https://te.wikisource.org/wiki contains the classical telugu ithihasas, puranas, vedas, stothras, etc; So I would like to scrape them and add as a new corpus.

Thank you.

To generalize, I observe that there are different approaches to making stopword lists, based either on statistics (most common words, variously calculated) or grammar (definite and indefinite articles, pronouns, etc.) (or some combination).

In doing this ticket, I would like you to do a little research on whether there exist any good lists for OE. If there is one, let's just take it. If not, we can do a little more research about what's right.

I am scraping Raw Classical Hindi Data from http://ltrc.iiit.ac.in/showfile.php?filename=downloads/Classical_Hindi_Literature/SHUSHA/index.html @kylepjohnson

Hi there, It's been months I have been thinking about this and I do not think CLTK contains anything like that. Collatinus and Eulexis are two Lemmatizer and Decliners which are open source (their data is either open or easy to reconstruct. And they are a nice bunch of people).

• Collatinus is in C
  • https://github.com/biblissima/collatinus is the most up to date source code for the flexer / lemmatizer
  • https://github.com/ycollatin/Collatinus-data is the repo for their data (but not up to date I guess ). It seems this is more up to date.
• Eulexis is in php
  • https://github.com/biblissima/eulexis/blob/master/traitement.php For the whole code

I'd be happy to convert the collatinus flexer for CLTK in the long run (give or take few months) but I think Eulexis and the lemmatizer part are out of my scope right now.

What's your opinion on this ? This would help search APIs a lot for text which are not lemmatized.

I've been reading about normalize() and hope it will prevent normalization problems in the future. This builtin method solves the problem of accented characters made with combining diacritics not equaling precomposed characters. Examples of this appear in the testing library, where I have struggled to make two strings of accented Greek equal one another.

Example of normalize() from Fluent Python by Luciano Ramalho (117-118):

>>> from unicodedata import normalize
>>> s1 = 'café' # composed "e" with acute accent
>>> s2 = 'cafe\u0301' # decomposed "e" and acute accent 
>>> len(s1), len(s2)
(4, 5)
>>> len(normalize('NFC', s1)), len(normalize('NFC', s2)) 
(4, 4)
>>> len(normalize('NFD', s1)), len(normalize('NFD', s2)) 
(5, 5)
>>> normalize('NFC', s1) == normalize('NFC', s2)
True
>>> normalize('NFD', s1) == normalize('NFD', s2) 
True

Solutions

1. In core, use normalize with the argument 'NFC', as Fluent Python recommends. Not all Greek combining forms may reduce into precomposed … will need to be tested out.

2. In tests, especially for assertEqual(), check that more complicated strings equal one another. Use normalize('NFC', <text>) on the comparison strings, too, if necessary.

3. Use this to strip out accented characters coming from the PHI, which I don't do very gracefully here: https://github.com/kylepjohnson/cltk/blob/master/cltk/corpus/utils/formatter.py#L94

Docs: https://docs.python.org/3.4/library/unicodedata.html#unicodedata.normalize

The Latin WordNet API mimics the NLTK Princeton WordNet API in all major respects; however because the data is sourced from latinwordnet.exeter.ac.uk (rather than locally) a number of under-the-hood changes were made. Many access methods now return generators rather than lists, and in general the API is now 'lazy' where multiple HTTP requests would cause a bottleneck. The Resnick, Jiang-Conrath, and Lin similarity scoring functions work, but require availability of a corpus-based information content file (forthcoming).

This ticket is for @soumyag213

As discussed by email, you'll port this and related modules, to the CLTK, from the Indic NLP Library.

For a first step, I'd like to see this working in your own repo, which you have started at: https://github.com/soumyag213/cltk-beginning-indo. In the README for this, I would like to see an example of its API. For example, I imagine you showing something like this is the Python shell (BTW I like iPython):

In [1]: from indic_syllabifier import orthographic_syllabify
In [2]: orthographic_syllabify('supercalifragilisticexpialidocious', 'tamil')
Out[2]: 'su-per-cal-i-fra-gil-ist-ic-ex-pi-al-i-doc-ious'

I noticed an anomaly processing Latin text with the default pipeline. The tokenizer fails to separate square brackets from the words they enclose.

text = 'Benedictus XVI [Iosephus Aloisius Ratzinger] fuit papa et episcopus Romanus.'

from cltk import NLP

cltk_nlp = NLP('lat')
cltk_nlp.analyze(text).tokens

Result:

['Benedictus', 'XVI', '[Iosephus', 'Aloisius', 'Ratzinger]', 'fuit', 'papa', 'et', 'episcopus', 'Romanus', '.']

The problem does not occur when the LatinWordTokenizer is used.

from cltk.tokenizers.lat.lat import LatinWordTokenizer

tokenizer = LatinWordTokenizer()
tokenizer.tokenize(text)

Result:

['Benedictus', 'XVI', '[', 'Iosephus', 'Aloisius', 'Ratzinger', ']', 'fuit', 'papa', 'et', 'episcopus', 'Romanus', '.']

Environment: Windows 10 + python 3.9.13 + cltk 1.1.6.

I added the spaCy process with a custom wrapper to translate Token from spacy to Word in cltk. The aim is to be able to use trained models provided by spaCy with CLTK.

In LatinBackOffLemmatizer() and the lemmatizers in its chain I can't seem to find an option to return an empty value (such as in OldEnglishDictionaryLemmatizer()'s best_guess=False option), instead of returning the input value, when the lemmatizer fails to assign a lemma.

Without such an option, it doesn't seem possible to tell successful from unsuccessful lemmatization attempts programmatically, severely limiting the range of the lemmatizer's applications.

Bumps certifi from 2022.5.18.1 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Unicode has two code points for acute accented vowels, one in the Greek and Coptic block and one in the Greek extended block (for omicron they are U+03CC and U+1F79. The list of accented vowels only takes into account the acute accents in the Greek and Coptic block resulting in some vowels not being properly scanned.

>>> from cltk.prosody.grc import Scansion
>>> text_string = "πότνια, θῦμον"
>>> Scansion()._make_syllables(text_string)
[[['πότνι', 'α'], ['θῦ', 'μον']]]

Expected behavior

>>> from cltk.prosody.grc import Scansion
>>> text_string = "πότνια, θῦμον"
>>> Scansion()._make_syllables(text_string)
[[['πο', 'τνι' , 'α'], ['θῦ', 'μον']]]

Desktop

• MacOS 13.0

Latin tokenizer does not separate -que, ne, ve. In line 147 of tokenizers/lat/lat.py I suggest: specific_tokens += [token[: -len(enclitic)]] + ["-"+enclitic] This fixed it for me.

Mac OS 15.7 Python 3.9