Custom in Parse-Then-Place

[Flulike]

Hi @ruick,

We are pleased to share with you that it's possible to use custom text for Parse-Then-Place. Specifically, there are totally 4 steps that you can follow to accomplish this.

1. Adapt the eval_split name in the script, maybe something like mydata.
2. Create a new json file in the folder Parse-Then-Place/data/$dataset/stage2/finetune/mydata.json according to the split name you specified in step 1.
3. Construct the sample based on your custom text in mydata.json. Below is an example:

[
    {
      "text": "a page for end user license agreement. on the top, there is a toolbar with a text button for the back on the left and a text in the middle. below is a text button for the end user license agreement. in the middle, there is a text occupying most of the page.",
      "region_id": "",
      "region_type": "",
      "ir": "[region:SingleInfo]",
      "execution": "",
      "layout_seq_with_completion": "",
      "layout_seq_without_completion": "",
      "plain_layout_seq": "",
      "canvas_width": 1440,
      "canvas_height": 2560
    }
]

4. Test on your custom text using the same command in README! And please check your result in the folder tmp/output/$dataset/stage1/mydata_predictions.json.

Originally posted by @kaoshizhou in #46

I followed author's guidance, and made some changes.
In adapter_eval.sh

if [ "${eval_split}" != "val" ] && [ "${eval_split}" != "train" ]
then
    eval_split="mydata"

In run_parser.py

    eval_split: Optional[str] = field(
        default='mydata',

In dataset.py

        base_name = 'all.jsonl'
        split_file_names = ['train.jsonl', 'val.jsonl', 'test.jsonl', 'mydata.jsonl']

Somehow I still have some issues.

I successfully generated the mydata_predictions.json in tmp/output/rico.

[
  {
    "pred_lf": "[region:SingleInfo[gp:toolbar[gattr:position 'top'][item[el:text button][el:text]]][el:text button[attr:size 'small']][el:image[attr:size 'large']]]",
    "gold_lf": "[region:SingleInfo]",
    "is_set_correct": false,
    "text": "a page for advertisement. on the top, there is a toolbar with a text button for the menu on the left and a text in the middle. below is a text button for the limited time offer. in the middle, there is a promotional image occupying most of the page.",
    "ex_id": "",
    "type": ""
  }
]

And how can I continue stage2 and visualization. I dont have ground truth file of mydata. I run stage2 as shown in readme. Gotten prediction.json like this

[
  {
    "line_id": 0,
    "region_id": "11323",
    "text": "a page for advertisement. on the top, there is a toolbar with a text button for the menu on the left and a text in the middle. below is a text button for the limited time offer. in the middle, there is a promotional image occupying most of the page.",
    "constraint": "singleinfo : text undefined large | text button undefined undefined | [ toolbar top undefined | text undefined undefined | text button left undefined ]",
    "gold_layout_seq": "text 0 12 144 9 | text 8 54 127 149 | text button 7 31 130 10 | text button 3 10 28 13",
    "prediction": [
      "image 7 35 130 47 | text button 41 183 61 7 | [ toolbar 0 8 144 20 | text 5 13 88 9 | text button 127 9 17 17 ]",
      "image 28 60 87 87 | text button 0 184 26 6 | [ toolbar 0 8 144 20 | text 5 13 77 9 | text button 119 9 25 17 ]",
      "image 0 28 144 105 | text button 5 126 30 9 | [ toolbar 0 8 144 20 | text 5 13 56 9 | text button 127 9 17 17 ]",
      "image 5 30 133 183 | text button 41 216 61 8 | [ toolbar 0 8 144 20 | text 5 13 58 9 | text button 113 9 31 17 ]",
      "image 0 28 144 96 | text button 5 127 30 13 | [ toolbar 0 8 144 20 | text 5 13 56 9 | text button 127 9 17 17 ]"
    ]
  }
]

And, also if there is way to get generated images information, such as boxes information.