Use XMLReader for Open Greek and Latin

I forked the First1KGreek repo and added it to the Greek corpora, which is in TEI.

At the suggestion of @diyclassics I have looked into using the NLTK’s XML corpus reader. Here’s what I see:

In [1]: from nltk.corpus.reader import XMLCorpusReader

In [2]: f = '/Users/kyle.p.johnson/cltk_data/greek/text/greek_text_first1kgreek/
   ...: data/tlg0015/tlg001/tlg0015.tlg001.opp-grc1.xml'

In [3]: import os

In [4]: path, name = os.path.split(f)

In [5]: path
Out[5]: '/Users/kyle.p.johnson/cltk_data/greek/text/greek_text_first1kgreek/data/tlg0015/tlg001'

In [6]: name
Out[6]: 'tlg0015.tlg001.opp-grc1.xml'

In [7]: reader = XMLCorpusReader(path, name)

In [8]: reader.raw()[:100]
Out[8]: '<?xml version="1.0" encoding="UTF-8"?>\n<?xml-model href="http://www.stoa.org/epidoc/schema/latest/te'

In [9]: reader.raw()[:200]
Out[9]: '<?xml version="1.0" encoding="UTF-8"?>\n<?xml-model href="http://www.stoa.org/epidoc/schema/latest/tei-epidoc.rng" schematypens="http://relaxng.org/ns/structure/1.0"?>\n<TEI xmlns="http://www.tei-c.org/'

In [10]: reader.words()[:10]
Out[10]: 
['Ab',
 'excessu',
 'divi',
 'Marci',
 'Herodian',
 'Immanuel',
 'Bekker',
 'European',
 'Social',
 'Fund']

Since .words() returns metadata, we will need something else.

What do others think about parsing TEI XML? Do we need to parse our selves using beautiful soup or the core xml library?

[kylepjohnson]

Another update: I have been forwarded this code, which I’ll use to process the XML: https://github.com/Capitains/HookTest/blob/master/HookTest/build.py#L91

[kylepjohnson]

Update: This repo actually has plain text files in it, already: https://github.com/cltk/First1KGreek/tree/1.1.1603/text

However an XML reader would still be desirable to users, I imagine.