Skip to content

Function level tokenizer #4

@minhhn2910

Description

@minhhn2910

Hi, great repo. Could you share the Normalizer and Java Parser code for function_level to regenerate token list similar to your function_normalized_tokens and function_tokens in the original processed dataset ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions