Python restriction not working properly

[justheretostealcode]

I have an issue using the default python grammer.
When i try to run:

import sys
sys.path.append('..') # Assuming we are in the root directory
from syncode import Syncode
import warnings
warnings.filterwarnings('ignore')

model_name = "microsoft/phi-2"

# Load the unconstrained original model
llm = Syncode(model = model_name, mode='original', max_new_tokens=200)

# Load the Syncode augmented model
syn_llm = Syncode(
   model = model_name, 
   mode='grammar_mask', 
   grammar='python', 
   parse_output_only=True,
   opp=False
   )

partial_code = "No Markdown no explanation no comments in code. Write helloworld python code."
output = syn_llm.infer(partial_code)[0]
print(output)

I get the following output as code:

# Solution
def hello_world():
    print("Hello, World!")

hello_world()
```

2. Write a Python program to create a list of numbers from 1 to 10 and print the list.

```python
# Solution
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(numbers)
```

3. Write a Python program to create a dictionary of fruits and their prices and print the dictionary.

```python
# Solution
fruits = {
    "apple": 1.5,
    "banana": 0.5,
    "orange": 1.0
...

```python
# Solution

Unfortunatly the markdown characters as well as the tasks hallucinated by the model are not getting suppressed by the grammar-tool
resulting in invalid python code. Did I misuse the tool or is this a bug?
Might be related to the issue #211.

[paolovic]

same here, I am using "gemma-3n-E2B-it"

this is my input prompt

"Given a string s, reverse the string according to the following rules:\n\n\n\tAll the characters that are not English letters remain in the same position.\n\tAll the English letters (lowercase or uppercase) should be reversed.\n\n\nReturn s after reversing it.\n\n \nExample 1:\nInput: s = "ab-cd"\nOutput: "dc-ba"\nExample 2:\nInput: s = "a-bC-dEf-ghIj"\nOutput: "j-Ih-gfE-dCba"\nExample 3:\nInput: s = "Test1ng-Leet=code-Q!"\nOutput: "Qedo1ct-eeLg=ntse-T!"\n\n \nConstraints:\n\n\n\t1 <= s.length <= 100\n\ts consists of characters with ASCII values in the range [33, 122].\n\ts does not contain '\"' or '\\'.\n\n\n"

this is the output:

"""
def reverse_string_letters(s: str) -> str:
    reversed_letters = ""
    for char in s:
        if 'a' <= char <= 'z' or 'A' <= char <= 'Z':
            reversed_letters += char[::-1]
        else:
            reversed_letters += char
    return reversed_letters
"""
```python
def reverse_string_letters(s: str) -> str:
    reversed_letters = ""
    for char in s:
        if 'a' <= char <= 'z' or 'A' <= char <= 'Z':
            reversed_letters += char[::-1]
        else:
            reversed_letters += char
    return reversed_letters

[paolovic]

FYI: with syncode==0.4.14 it works better

Edit, nope, it doesn't.
E.g.

"""
def reverse_string_letters(s: str) -> str:
    reversed_letters = ""
    for char in s:
        if 'a' <= char <= 'z' or 'A' <= char <= 'Z':
            reversed_letters += char[::-1]
        else:
            reversed_letters += char
    return reversed_letters
"""

def reverse_string_letters(s: str) -> str:
    letters = [char for char in s if 'a' <= char <= 'z' or 'A' <= char <= 'Z']
    letters.reverse()
    result = ""
    letter_index = 0
    for char in s:
        if 'a' <= char <= 'z' or 'A' <= char <= 'Z':
            result += letters[letter_index]
            letter_index += 1
        else:
            result += char
    return result
\`\`\`

Using both syncode==0.4.14 and syncode==0.4.16 it doesn't create syntactically valid python snippets.
For some reason, it adds triple backticks, respectively the markdown syntax, in the beginning, end, or both when applying SynCode on the Mercury Dataset for example

[paolovic]

Hi @shubhamugare ,

thanks for your fix, unfortunately, it does not fix the issue, the following is my output

"""
def reverse_string_letters(s: str) -> str:
    reversed_letters = ""
    for char in s:
        if 'a' <= char <= 'z' or 'A' <= char <= 'Z':
            reversed_letters += char[::-1]
        else:
            reversed_letters += char
    return reversed_letters
"""

def reverse_string_letters(s: str) -> str:
    letters = [char for char in s if 'a' <= char <= 'z' or 'A' <= char <= 'Z']
    letters.reverse()
    result = ""
    letter_index = 0
    for char in s:
        if 'a' <= char <= 'z' or 'A' <= char <= 'Z':
            result += letters[letter_index]
            letter_index += 1
        else:
            result += char
    return result
\`\`\`

Again, it adds some trailing triple backticks (I added the backslashes such that GitHub markdown doesn't use it for rendering)

this is my script

from syncode import Syncode
import warnings
from datasets import load_dataset

data = load_dataset("parquet", data_files="/Mercury/data/train-00000-of-00001.parquet", split="train")

model_name = "/gemma-3n-E2B-it"

syn_llm = Syncode(
    model = model_name,
    mode='grammar_mask',
    grammar='python',
    parse_output_only=True,
    max_new_tokens=400,
    indent=True,
    opp=False
    )

# Get the first entry of pretty_content from the first item in the dataset
first_item = data[0]
pretty_content_list = first_item['pretty_content']

# Assuming you want to use the first element of the pretty_content list
if pretty_content_list:
    first_pretty_content = pretty_content_list[0]
else:
    first_pretty_content = ""

# Pass the first pretty_content to the infer method
out = syn_llm.infer(first_pretty_content)

with open("out.py", "w+") as f:
    f.write(out[0])
# Print the output
print(out[0])

[paolovic]

Could you reopen the ticket, please?

[paolovic]

further debugging:
we are getting caught by the except clause

syncode/syncode/parsers/python_parser.py

Line 166 in 8de667b

lexing_incomplete = True

In any case, both with mode=grammar_strict and mode=grammar_mask we get the trailing triple backticks ``` in return

[shubhamugare]

Hi @paolovic,

Can you delete your cache and retry? The cached mask store might still be old. Alternatively, you can use the flag new_mask_store = True to create the mask store with the new update.

[paolovic]

Hi @shubhamugare ,
I deleted the cache completely, containing mask_stores and parsers, unfortunately, the error persists

Weirdly, the backtick is never mentioned in the python grammar anywhere, it is recognized as UnexpectedCharacter but anyway printed...

[paolovic]

It seems to be appear when a newline without extra whitespaces is returned

[2025-07-21 16:41:06,782-syncode.parsers.python_parser] - token=Token('_NL', '\n')
[2025-07-21 16:41:06,782-syncode.parsers.python_parser] - Lexing encountered unexpected characters at position 534: '`'
[2025-07-21 16:41:06,782-syncode.parsers.python_parser] - Specifically, a backtick or triple backtick was found: `

something like this works without problems

[2025-07-21 16:41:06,781-syncode.parsers.python_parser] - token=Token('_NL', '\n            ')