[Feature Request] Ability to re-scrap all assets at once

[Hybrid512]

Fixing bad scraping issues is a long and unpleasing process for a ROM.
If you need to re-scrap assets, you'll have to re-scrap sometimes 5~6 different assets (boxcover, backcover, fanart, ...) for the same ROM with the same source but you can't do that at once.
Would be great to have an option "Edit all assets" when going to the "Edit rom assets/artwork" menu where, instead of scraping only one element, it would rescrap every element from the same source scraper with the same item ID.

[Wintermute0110]

I've done an implementation of this in the ROM context menu. Can you please test? If you find any problem please report it here.

[Hybrid512]

Hi,

Many thanks for that new feature, it really helps a lot !

It's working great but it could benefit from a few refinements :

• it is acting like chained individual asset scraping method (which it is) but then, it has a wrong behavior when there is no result for an asset (no box back for example) and in this case, there is a blocking popup saying there is no picture for this asset and it is stuck until you validate. When there is no result, asset should just silently be left as is and let the process continue.
• when scraper returns only 1 result for an already existing asset (for example, you already have a fanart set, might it be the good one or not), you get prompted to select the new asset even though there is only 1 result and even if this result is identical to the one you already have set. Would be better if this was ignore in case you get only 1 result with an identical asset and be prompted only if there are 2+ results or if there is only 1 result but with a file that is different from the one already configured.
• when scrapping all assets when there is no assets configured at all for a rom (like after unsetting all assets for example), scraping don't act the same way as when you use the "Scan roms" menu where the assets selection is automatic (or at least, it follows the policy you configured in the options ... I set it to "automatic" in my case). Instead, I get prompted each time there is more than 1 result for an asset. It should follow the same policy as the "Scan roms" option.
• there is a "null" scraper in the scraper list, don't know what it is/does.
• regarding UX, I would prefer to have an "Edit All assets" option in the "Edit ROM -> Edit Assets" menu, at the top of the list than having a dedicated menu, it would make things cleaner to me. That way, you would do "Edit ROM -> Edit Assets -> Edit All assets -> (unset assets / Scrape with ... / Import local image, ..). Would make more sens to me.

Best regards !

[Wintermute0110]

Thanks a lot for your feedback, I really appreciate such a detailed feedback because it helps me with the software design process a lot.

it is acting like chained individual asset scraping method (which it is) but then, it has a wrong behavior when there is no result for an asset (no box back for example) and in this case, there is a blocking popup saying there is no picture for this asset and it is stuck until you validate. When there is no result, asset should just silently be left as is and let the process continue.

I have improved the verbosity when scraping all assets, it should be much smoother now. When scraping a single asset the verbosity is the same.

when scraper returns only 1 result for an already existing asset (for example, you already have a fanart set, might it be the good one or not), you get prompted to select the new asset even though there is only 1 result and even if this result is identical to the one you already have set. Would be better if this was ignore in case you get only 1 result with an identical asset and be prompted only if there are 2+ results or if there is only 1 result but with a file that is different from the one already configured.

The problem is that there is no human way for AEL to know that both images are the same, so if there is a current image and the scraper founds 1 image then user must choose.

when scrapping all assets when there is no assets configured at all for a rom (like after unsetting all assets for example), scraping don't act the same way as when you use the "Scan roms" menu where the assets selection is automatic (or at least, it follows the policy you configured in the options ... I set it to "automatic" in my case). Instead, I get prompted each time there is more than 1 result for an asset. It should follow the same policy as the "Scan roms" option.

Yeah, the scanner always checks if the local assets exists or not. The motto behind the current behavior is that in the scanner should be as automatic as possible, however, when using the context menu your are supposed to know what you are doing. When you use the context menu to unset and asset the asset is not deleted from disk but AEL understands that you want to get rid of it.

there is a "null" scraper in the scraper list, don't know what it is/does.

The Null scraper is a development tool. It is a scraper that never finds a candidate game, never finds any metadata and images are never available. It's simple and it's useful for testing and developing. I have fixed that, now only scrapers that support assets are shown.

regarding UX, I would prefer to have an "Edit All assets" option in the "Edit ROM -> Edit Assets" menu, at the top of the list than having a dedicated menu, it would make things cleaner to me. That way, you would do "Edit ROM -> Edit Assets -> Edit All assets -> (unset assets / Scrape with ... / Import local image, ..). Would make more sens to me.

The way it's implemented now code-wise it's much easier this way. What you propose requires some heavy modifications of current Python code. I will keep the suggestion for a future version.

---

Please try the new version and report back. If you don't find any further issues I will release the new version soon.

[Hybrid512]

Thanks a lot for your feedback, I really appreciate such a detailed feedback because it helps me with the software design process a lot.

Well, thank you above all for doing this great addon and this is the whole purpose of Open Source : contributions and feedback in order to enhance the product.

I have improved the verbosity when scraping all assets, it should be much smoother now. When scraping a single asset the verbosity is the same.

Way better now !

when scraper returns only 1 result for an already existing asset (for example, you already have a fanart set, might it be the good one or not), you get prompted to select the new asset even though there is only 1 result and even if this result is identical to the one you already have set. Would be better if this was ignore in case you get only 1 result with an identical asset and be prompted only if there are 2+ results or if there is only 1 result but with a file that is different from the one already configured.

The problem is that there is no human way for AEL to know that both images are the same, so if there is a current image and the scraper founds 1 image then user must choose.

Well, there could be an "easy" way to solve that point but that would need to modify the code a bit deeper I guess.
Problem is that when you store the picture in the assets, you rename them with the ROM name which is completely understandable as it ease the link between the ROM and the assets files.
BTW, there are 2 issues I see with that method :

• first : if you rename a ROM file, this link is broken
• second : that way, you have no way to know what was the original file downloaded from the scraper.
  Fixing the first problem is not trivial, you could use some kind of fingerprinting based on md5 sum to link the assets to your ROM file (md5sum on the file will give the same hash whatever your file is named) but it would lead to more CPU usage and probably slower application with huge romsets
  The second part could be fixed by storing the original URL of the asset file somewhere in the database, that way, when rescraping the same asset, you would know if it had already been downloaded or if the assets that is stored is the same as the one gathered by the scraper and thus, you could avoid bringing up the popup menu when there is only 1 result from the scraper that is equal to what is already stored in the assets.

regarding UX, I would prefer to have an "Edit All assets" option in the "Edit ROM -> Edit Assets" menu, at the top of the list than having a dedicated menu, it would make things cleaner to me. That way, you would do "Edit ROM -> Edit Assets -> Edit All assets -> (unset assets / Scrape with ... / Import local image, ..). Would make more sens to me.

The way it's implemented now code-wise it's much easier this way. What you propose requires some heavy modifications of current Python code. I will keep the suggestion for a future version.

I can understand that, however, I hope you keep that in mind for a future version because I still think this would be better for the end user.

[Wintermute0110]

second : that way, you have no way to know what was the original file downloaded from the scraper.
Fixing the first problem is not trivial, you could use some kind of fingerprinting based on md5 sum to link the assets to your ROM file (md5sum on the file will give the same hash whatever your file is named) but it would lead to more CPU usage and probably slower application with huge romsets.

The problem is that the image you see in the select dialog for the scrapers is a remote image and the only way to compute a hash of a remote image is to download them. Also, note that in the select dialog AEL always uses thumbnails (image previews), not the original images downloaded. This is true for TGDB and MobyGames, ScreenScraper does not support this so there is no preview in the dialog. The hash for the preview image is different than the hash from the downloaded image.

I think I will keep things as they are now but keep this issue open as a reference for future version.

[Hybrid512]

second : that way, you have no way to know what was the original file downloaded from the scraper.
Fixing the first problem is not trivial, you could use some kind of fingerprinting based on md5 sum to link the assets to your ROM file (md5sum on the file will give the same hash whatever your file is named) but it would lead to more CPU usage and probably slower application with huge romsets.

The problem is that the image you see in the select dialog for the scrapers is a remote image and the only way to compute a hash of a remote image is to download them. Also, note that in the select dialog AEL always uses thumbnails (image previews), not the original images downloaded. This is true for TGDB and MobyGames, ScreenScraper does not support this so there is no preview in the dialog. The hash for the preview image is different than the hash from the downloaded image.

I think I will keep things as they are now but keep this issue open as a reference for future version.

I think you misunderstood what I tried to say.
The hashing mechanism could be a way to "link" downloaded assets to the rom file instead of relying on the rom file name because if you change the rom name, AEL won't be able to find the existing local assets unless you rename them accordingly.
If you base this link on a hash of the rom file, then you won't have this issue but, as I said, it is more CPU greedy ... this was just an idea.

Regarding comparisons between local and remote assets, I was just saying, do this :

• when scraping an asset for the first time, download the file as it is done today (by renaming the file to the rom file name ... and if you could take a look at some of my proposals in [Feature request] Customisable assets dimensions #144 , would be even greater !) and store the file source URL somewhere in a DB (or nfo, xml, ... whatever you use)
• when re-scraping the same asset, compare the scraping URL to the URL you have in your DB, if they are identical AND you have only 1 result from the scraper, then, just silently ignore this asset because you know you already have this file.

[Wintermute0110]

OK, now I completely understand what you mean. I think the solution you propose is too complex for the problem at hand. Keep in mind that Kodi has a 10 feet GUI and that limits what AEL can do a lot. For the scraping of the assets you can use other scraper or external tools in a PC where you have much more advanced features. I think that dealing with ROM renaming it's too much complexity.

The idea behind AEL/AML is the ability to do audits and basic scraping. However, I do not want to convert AEL/AML into a ROM Manager or an advanced scraper. Think on the Kodi music database. You can do artwork scraping and some basic organization of your collection. However, you will need to rely on external tools (tag editor, music organizer, etc.) to organize your music collection.