Grammalecte  Check-in [b5ff33ce96]

Overview
Comment:[core] token neutralization
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | core | rg
Files: files | file ages | folders
SHA3-256: b5ff33ce96fa4a652c64d2891a44bcafec39f06f628d9640e21f0d30b47771ef
User & Date: olr on 2018-06-26 09:58:37
Other Links: branch diff | manifest | tags
Context
2018-06-26
12:34
[core] gc engine: new char to merge tokens check-in: 4e6d599062 user: olr tags: core, rg
09:58
[core] token neutralization check-in: b5ff33ce96 user: olr tags: core, rg
09:57
[fr] conversion: regex rules -> graph rules check-in: 1f0c86fd07 user: olr tags: fr, rg
Changes

Modified gc_core/py/lang_core/gc_engine.py from [990cf356cf] to [f034e8dbe9].

899
900
901
902
903
904
905
906

907
908
909
910
911
912
913
914
915







916
917
918
919
920
921
922
899
900
901
902
903
904
905

906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929







-
+









+
+
+
+
+
+
+







        if sWhat == "*":
            # purge text
            if nTokenRewriteEnd - nTokenRewriteStart == 0:
                self.lToken[nTokenRewriteStart]["bToRemove"] = True
            else:
                for i in range(nTokenRewriteStart, nTokenRewriteEnd+1):
                    self.lToken[i]["bToRemove"] = True
        elif sWhat == "_":
        elif sWhat == "=":
            # merge tokens
            self.lToken[nTokenRewriteStart]["nMergeUntil"] = nTokenRewriteEnd
        elif sWhat == "!":
            # immunity
            if nTokenRewriteEnd - nTokenRewriteStart == 0:
                self.lToken[nTokenRewriteStart]["bImmune"] = True
            else:
                for i in range(nTokenRewriteStart, nTokenRewriteEnd+1):
                    self.lToken[i]["bImmune"] = True
        elif sWhat == "_":
            # neutralized token
            if nTokenRewriteEnd - nTokenRewriteStart == 0:
                self.lToken[nTokenRewriteStart]["sNewValue"] = "_"
            else:
                for i in range(nTokenRewriteStart, nTokenRewriteEnd+1):
                    self.lToken[i]["sNewValue"] = "_"
        else:
            if sWhat.startswith("="):
                sWhat = globals()[sWhat[1:]](self.lToken)
            bUppercase = bUppercase and self.lToken[nTokenRewriteStart]["sValue"][0:1].isupper()
            if nTokenRewriteEnd - nTokenRewriteStart == 0:
                sWhat = sWhat + " " * (len(self.lToken[nTokenRewriteStart]["sValue"])-len(sWhat))
                if bUppercase: