[vie] EnhancementsSee original GitHub issue
I notice a few obvious problems with Vietnamese:
- As far as I can tell, nearly all Vietnamese words have three phonetic pronunciations: “Hà Nội”, "Huế, and “Hồ Chí Minh City”. We should just add dialects to
- As is well-known, the Vietnamese Roman orthography puts a space between every syllable. So our data set is really only monosyllabic Vietnamese words! We could solve this by adding a flag (
--no-skip-space) here, or we could make a language specific-extractor, I suppose.
- Created 3 years ago
Top GitHub Comments
I guess some reasons to skip them include:
- There’s no general guidance for why multiword expressions might be entered in Wiktionary in the first place. Sometimes they’re common compounds or idioms, but occasionally there are even entire phrases.
- There’s no reason to collect multiword expressions for pronunciation purposes, which is our goal here.
You’re right that that function controls it, but I think we probably want it to be scriptable so users can control whether they skip, and we can set it on a per-language basis for the big scrape. So I think the obvious thing to do is to add a flag (in
cli.py) and then make sure it’s passed to
_skip_word; then, disable that for Chinese and Vietnamese.
Potentially we could also make the rules about what gets skipped more expressive—maybe you could pass in a regular expression and if it’s matched, the word is skipped—but that’s probably premature.
Scraping for a Vietnamese dialect from the three that Kyle added in #135 returns results for all three current dialects (when
phonetic=True). Scraping for one of the additional dialects that Yeonju has added returns the same results as scraping for one of the first three dialects added in #135 (when
As far as I can tell this is due to how we define
config.py and the inconsistent HTML used on pages like cầm.
It looks like Vietnamese dialect information (for the three dialects covered in #135 ) is stored within
<i> elments, while the dialect information for the dialects that Yeonju has added is stored as text in
<span class="ib-content qualifier-content"> elements - but not as text in the
<a> elements within those
span elements that we usually target (which is why we get nothing when scraping for one of the dialects Yeonju found while