Overview
| Comment: | [graphspell][bug] tokenizer: syntax error |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | descendants | both | graphspell | rg |
| Files: | files | file ages | folders |
| SHA3-256: |
ec92f6e87328379434d5106478ec89b6 |
| User & Date: | olr on 2018-06-30 06:30:01 |
| Other Links: | branch diff | manifest | tags |
Context
|
2018-06-30
| ||
| 06:30 | [fr] conversion: regex rules -> graph rules check-in: 5cb83eecf5 user: olr tags: fr, rg | |
| 06:30 | [graphspell][bug] tokenizer: syntax error check-in: ec92f6e873 user: olr tags: graphspell, rg | |
| 00:19 | [core] ge engine: function for testing token value check-in: 8289f6c423 user: olr tags: core, rg | |
Changes
Modified graphspell/tokenizer.py from [2adea5dc85] to [b1bcfc3595].
| ︙ | ︙ | |||
55 56 57 58 59 60 61 |
i = 0
if bStartEndToken:
yield { "i": 0, "sType": "INFO", "sValue": "<start>", "nStart": 0, "nEnd": 0, "lMorph": ["<start>"] }
for i, m in enumerate(self.zToken.finditer(sText), 1):
yield { "i": i, "sType": m.lastgroup, "sValue": m.group(), "nStart": m.start(), "nEnd": m.end() }
if bStartEndToken:
iEnd = len(sText)
| | | 55 56 57 58 59 60 61 62 |
i = 0
if bStartEndToken:
yield { "i": 0, "sType": "INFO", "sValue": "<start>", "nStart": 0, "nEnd": 0, "lMorph": ["<start>"] }
for i, m in enumerate(self.zToken.finditer(sText), 1):
yield { "i": i, "sType": m.lastgroup, "sValue": m.group(), "nStart": m.start(), "nEnd": m.end() }
if bStartEndToken:
iEnd = len(sText)
yield { "i": i+1, "sType": "INFO", "sValue": "<end>", "nStart": iEnd, "nEnd": iEnd, "lMorph": ["<end>"] }
|