JSSCP bundle upload takes more time

[1t5j0y]

Describe the bug
Bundle uploads, specifically concept saves are taking much longer after 16.0.

To Reproduce
JSSCP bundle upload on Nov 24 took 20 mins
Similar sized bundle for same org on Nov 26 caused db locks a couple of times and the 3rd attempt took 131 mins to complete.

Expected behavior
Bundle upload performance should be same as before.

Developer Checklist
Developer fixing the bug should fill this checklist.

• Does the fix require extensive regression testing?
• Are you mentioning the required scenarios that could be affected?

Analysis:

• JSSCP UAT concepts file(bulkuploads/input/jsscp/db9ad8d9-1c75-4eb0-ac6c-5e91a8bd8b0d-JSSCP_UAT_Org_Updated_1st_Dec_2025.zip) has lesser number of concepts and answers than ashwini(bulkuploads/input/ashwini/804b0698-afb7-40bc-b1fe-f86f7ac641e4-Ashwini-dev-25-11-2025.zip). Ashwini takes only 20-30 mins.
• Same 56 media on 24 and 26th, on 24th took less time around 20 to 30 minutes
• But on 26th we did a release - So something in 16.0 has caused the issue
• Also when importing concepts we are trying to set media by doing updateMediaInfo though it is separately handled when importing conceptmedia folder - though this might not be the cause just sharing that this can be removed.

[himeshr]

Just wanted to check, if the issue is easily reproducible on prerelease or staging envs as well.
Also, did we check if there were any blocking pids at that time.?

[mahalakshme]

• I also think there were additional implementation changes
• We are logging about upload for every concept unlike others where we do on a file basis.
• Independent of above, I agree we need to see if something else can be improved.

[1t5j0y]

@himeshr first 2 bundle attempts yesterday had locking pids from the bundle upload process itself (select on concept_answer was blocking update concept join concept_answer). Killing the blocking process ended the bundle upload.
pg_stat_activity while locked.

No locking pids during the 3rd attempt.. it was just a lot slower. Concepts step took 99% of the time for the job.

Will confirm issue is easily reproducible in prerelease.

[1t5j0y]

Observations so far:

• Performance when workload is inserts of concepts is fine (took 20 mins on prerelease similar to performance on prod for previous runs).
• Performance suffers when workload is updates (i.e. apply same bundle again). Got into a db lock situation on prerelease where one update concept query was blocked by another update concept query.
• No cpu or ebs pressure observed on prod rds db during prod runs that could have caused this issue in prod.
• There is a major cache eviction that happens when all concepts in a bundle are processed.
• Created a prerelease-15-4 db to check if issue is 16.0 specific but have not tested against it yet.

Idea to optimise concepts upload:

• We could leverage metadata diff checker for scenarios like this (update to already existing large org) to only upload concepts that have changes in them.

[1t5j0y]

Didn't find any specific issue that would cause this in 16.0. 15.4 comparison also seemed to be similar.

Parking this for now to see if more such instances are reported.