Grammalecte  Check-in [8a9ec2fb8e]

Overview
Comment:[build] use ? for token merging, and remove them at build
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk | build
Files: files | file ages | folders
SHA3-256: 8a9ec2fb8eec2fabc29f804b9efff59968778a6e38ae0d81d06812a9443413a1
User & Date: olr on 2018-11-10 12:28:16
Other Links: manifest | tags
Context
2018-11-10
12:53
[fr] màj: emploi du subjonctif check-in: 27de6ef1dc user: olr tags: trunk, fr
12:28
[build] use ? for token merging, and remove them at build check-in: 8a9ec2fb8e user: olr tags: trunk, build
09:56
[fr] nr: auquel, auxquels, etc. + divers check-in: 7d89e90486 user: olr tags: trunk, fr
Changes

Modified compile_rules_graph.py from [eb2b3da492] to [b4ed6d4156].

74
75
76
77
78
79
80



81
82
83
84
85
86
87
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90







+
+
+









def genTokenLines (sTokenLine, dDef):
    "tokenize a string and return a list of lines of tokens"
    lToken = sTokenLine.split()
    lTokenLines = None
    for sToken in lToken:
        # replace merger characters by spaces
        if "␣" in sToken:
            sToken = sToken.replace("␣", " ")
        # optional token?
        bNullPossible = sToken.startswith("?") and sToken.endswith("¿")
        if bNullPossible:
            sToken = sToken[1:-1]
        # token with definition?
        if sToken.startswith("({") and sToken.endswith("})") and sToken[1:-1] in dDef:
            sToken = "(" + dDef[sToken[1:-1]] + ")"