Performance issues

[nmeylan]

On big json following features are slow:

• Insert new row: due to data structure with json pointer, inserting a new row means we need to iterate over each entry after the new added row to update their json-pointer as the position has changed.

{"skills": [{"copyFlags": {"reproduce": true, "plagiarism": false}}]}

/skills/0 -> {"copyFlags": {"reproduce": true, "plagiarism": false}} 
/skills/0/copyFlags -> {"reproduce": true, "plagiarism": false}
/skills/0/copyFlags/reproduce -> true
/skills/0/copyFlags/plagiarism -> false

Then inserting a new row above row 0, any row after row 0 should be updated like this

/skills/1 -> {"copyFlags": {"reproduce": true, "plagiarism": false}} 
/skills/1/copyFlags -> {"reproduce": true, "plagiarism": false}
/skills/1/copyFlags/reproduce -> true
/skills/1/copyFlags/plagiarism -> false

• Replace in column when criteria match too many row: due to the capability to open multiple view on the same row at different depth of the json object, we have to serialize and parse again the updated row:
  • we need to serialize it so root object of the row can be viewed in "Object table"
  • we then parse it again to update nested objects serialized entry

{"skills": [{"copyFlags": {"reproduce": true, "plagiarism": false}}]}

/skills/0 -> {"copyFlags": {"reproduce": true, "plagiarism": false}}  <- we keep serialized object for depth 3 view 
/skills/0/copyFlags -> {"reproduce": true, "plagiarism": false}  <- we keep serialized object for depth 4 view
/skills/0/copyFlags/reproduce -> true
/skills/0/copyFlags/plagiarism -> false

If we update plagiarism to true /skills/0/copyFlags/plagiarism -> true

we also need to update 
/skills/0 <- require serialization of root object
/skills/0/copyFlags

[nmeylan]

mitigate slow replace by, on json of 350mb, with 310k rows, replacing all 310k rows values:

• Improving serialization: from 6.5sec to 5s
• Enable parallel replacement (using 8 core): from 5s to 1.2s
  It is still slow, but start to be usable