Overview
| Comment: | [graphspell] tokenizer: consider presqu’ and quelqu’ as separate words |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | descendants | both | trunk | graphspell |
| Files: | files | file ages | folders |
| SHA3-256: |
0f0bc77645cf24070746bb77572c8998 |
| User & Date: | olr on 2019-08-30 09:45:02 |
| Other Links: | manifest | tags |
Context
|
2019-08-31
| ||
| 13:50 | [fr] faux positifs et ajustements check-in: 63e9ff5982 user: olr tags: trunk, fr | |
|
2019-08-30
| ||
| 09:45 | [graphspell] tokenizer: consider presqu’ and quelqu’ as separate words check-in: 0f0bc77645 user: olr tags: trunk, graphspell | |
| 09:44 | [fr] ajustement: option chimie check-in: 3a43322a92 user: olr tags: trunk, fr | |
Changes
Modified graphspell-js/tokenizer.js from [2f6a31a87a] to [0f58ee1cf3].
| ︙ | |||
35 36 37 38 39 40 41 | 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | - + |
[/^[,.;:!?…«»“”‘’"(){}\[\]·–—¿¡]/, 'PUNC'],
[/^[A-Z][.][A-Z][.](?:[A-Z][.])*/, 'WORD_ACRONYM'],
[/^(?:https?:\/\/|www[.]|[a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st_-]+[@.][a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st_-]{2,}[@.])[a-zA-Z0-9][a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st_.\/?&!%=+*"'@$#-]+/, 'LINK'],
[/^[#@][a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st_-]+/, 'TAG'],
[/^<[a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st]+.*?>|<\/[a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st]+ *>/, 'HTML'],
[/^\[\/?[a-zA-Zà-öÀ-Ö0-9ø-ÿØ-ßĀ-ʯff-st]+\]/, 'PSEUDOHTML'],
[/^&\w+;(?:\w+;|)/, 'HTMLENTITY'],
|
| ︙ |
Modified graphspell/tokenizer.py from [af7051a739] to [18d4ef9f97].
| ︙ | |||
27 28 29 30 31 32 33 | 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | - + |
r'(?P<FOLDERWIN>[a-zA-Z]:\\(?:Program Files(?: [(]x86[)]|)|[\w.()]+)(?:\\[\w.()-]+)*)',
r'(?P<PUNC>[][,.;:!?…«»“”‘’"(){}·–—¿¡])',
r'(?P<WORD_ACRONYM>[A-Z][.][A-Z][.](?:[A-Z][.])*)',
r'(?P<LINK>(?:https?://|www[.]|\w+[@.]\w\w+[@.])\w[\w./?&!%=+*"\'@$#-]+)',
r'(?P<HASHTAG>[#@][\w-]+)',
r'(?P<HTML><\w+.*?>|</\w+ *>)',
r'(?P<PSEUDOHTML>\[/?\w+\])',
|
| ︙ |