Grammalecte  Check-in [ec92f6e873]

Overview
Comment:[graphspell][bug] tokenizer: syntax error
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | graphspell | rg
Files: files | file ages | folders
SHA3-256: ec92f6e87328379434d5106478ec89b60c7a81b7c8cc1645ec0f71e5f325b32f
User & Date: olr on 2018-06-30 06:30:01
Other Links: branch diff | manifest | tags
Context
2018-06-30
06:30
[fr] conversion: regex rules -> graph rules check-in: 5cb83eecf5 user: olr tags: fr, rg
06:30
[graphspell][bug] tokenizer: syntax error check-in: ec92f6e873 user: olr tags: graphspell, rg
00:19
[core] ge engine: function for testing token value check-in: 8289f6c423 user: olr tags: core, rg
Changes

Modified graphspell/tokenizer.py from [2adea5dc85] to [b1bcfc3595].

55
56
57
58
59
60
61
62

55
56
57
58
59
60
61

62







-
+
        i = 0
        if bStartEndToken:
            yield { "i": 0, "sType": "INFO", "sValue": "<start>", "nStart": 0, "nEnd": 0, "lMorph": ["<start>"] }
        for i, m in enumerate(self.zToken.finditer(sText), 1):
            yield { "i": i, "sType": m.lastgroup, "sValue": m.group(), "nStart": m.start(), "nEnd": m.end() }
        if bStartEndToken:
            iEnd = len(sText)
            yield { "i": i+1, "sType": "INFO", "sValue": "<end>", "nStart": iEnd, "nEnd": iEnd, , "lMorph": ["<end>"] }
            yield { "i": i+1, "sType": "INFO", "sValue": "<end>", "nStart": iEnd, "nEnd": iEnd, "lMorph": ["<end>"] }