PascalScript (serialize duplicates) VERY slow Preview.

brian.r.hamilton · 2013-02-15 03:47

Is there a way to Rename without having to do the Preview stage? It takes many hours to generate the New Names via Preview for over 10,000 files.

brian.r.hamilton · 2013-02-15 12:22

I think I found a useful solution in using Shift-F instead of the Pascal serialize duplicates. My project consists of nearly 100,000 files (image files) so speed is important. I am using EXIF meta tags too, so the processing is painfully slow. Much faster without the Pascal script though. May have to try the beta version as I've read this adds a serialize option after Preview. Should I be concerned that it is still beta?

Also, I noticed that ReNamer never uses more than 12.5% CPU when generating the Preview (or maybe when doing anything). I also noticed Windows Explorer seems to have the exact same upper bound when renaming. Is this due to some operating system limitation? I am using Windows 7 SP1. There is no disc thrashing going on, just limited use of CPU for some reason.

den4b · 2013-02-16 13:44

Hi Brian,

The Preview process is the process that generates the new names of the files, while Rename process simply applies them to the actual files. So the answer is NO - you cannot rename without the preview.

There is a number of configuration options which can be adjusted to improve the speed of processing large amount of files, by disabling certain functionality. Open Settings, Preview tab, disabling any of these options will speed up the preview process.

In addition to that, I would suggest staying away from Pascal Script rule and Meta Tags where possible because they are the most time consuming operations. The "serialize duplicates" script was developed simply for educational purposes, while native "fix conflicting new names" option (also accessible via SHIFT+F shortcut) is a much more efficient alternative.

The Beta label normally isn't of a concern. I, for one, always use the latest development version for all filename operations. The Beta however should warn users that some of the new or significantly changed functionality is either not yet well tested or still under development.

Regarding the CPU usage. ReNamer is a single threaded process, which means that it could not use more than a single CPU core in the multi-core architecture. Also, many of ReNamer's operations are slowed down by disk read/write operations, so it's not possible to occupy 100% even of a single core most of the time.

I hope this helps.

brian.r.hamilton · 2013-02-16 18:16

Thank you. Yes, that helps my understanding of a lot of things. I pretty much figured out that Preview must have been doing the actual work & it only makes sense by design. I have already disabled those items. I do need to use the EXIF meta tags though.

Oh yeah, multithreading & hyperthreading. Sorry, I forgot about all this. I didn't fully consider what the information Windows was displaying truly meant & haven't been able to really stay focused on this project like normal. It does seem like ReNamer is an ideal application for multithreading, & would actually be fairly simple to accomplish, but I have never programmed for Windows (just embedded DSPs, with lots of parallelism too). I suspect many more people will be looking for software just like ReNamer (& hope for fast results) in the near future as their digital collections explode in size. Sounds like a perfect "educational" project! :-)

Thanks for a very helpful program!

SafetyCar · 2013-02-17 10:22

Well, the use of EXIF meta tags are directly read from the image file so accessing 100,000 files should take a while. (I don't know how the disk would feel doing that with multithreading...)

Maybe if you could tell the rules or the kind of rules you are using we could see something, for example if you are accessing other meta-tags or the same meta-tag in different rules... Some renaming tasks are very light, but some others especially with so much files and accessing lots of meta-tags can be a pain, but not necessarily because of ReNamer, but for all the work there is to do.

By the way, did it improve disabling the settings in the preview menu?

Last edited by SafetyCar (2013-02-17 10:24)

brian.r.hamilton · 2013-02-17 18:53

I had previously disabled the settings in the Preview menu, so I don't know how much change that made since I didn't try it with them enabled.

I only had 3 Rules initially - Strip the filename, Insert the EXIF date (as the filename), then serialize using the Pascal script. That took nearly 3 hours for a subset of the files; maybe 30,000 or so. Then I removed the Pascal script which cut the time by half at least. I forget the details already.

I needed the EXIF data for the filename to help with sorting & eliminating duplicate photos from a large data loss recovery project. Typical file duplicate finding/eliminating tools helped a lot initially, but are not much help at this point because so many of the recovered files have some degree of corruption. The image data is still valid but the files have fragments attached from the recovery process. Renaming the files helped me see the scope of the problem better so now I can search for a tool to automate weeding out the corrupt files.

Anyway, I don't think multithreading would expose any bottleneck on the disk/file interface but I would be happy if it did!

Last edited by brian.r.hamilton (2013-02-17 18:56)

den4b Forum

#1 2013-02-15 03:47

PascalScript (serialize duplicates) VERY slow Preview.

#2 2013-02-15 12:22

Re: PascalScript (serialize duplicates) VERY slow Preview.

#3 2013-02-16 13:44

Re: PascalScript (serialize duplicates) VERY slow Preview.

#4 2013-02-16 18:16

Re: PascalScript (serialize duplicates) VERY slow Preview.

#5 2013-02-17 10:22

Re: PascalScript (serialize duplicates) VERY slow Preview.

#6 2013-02-17 18:53

Re: PascalScript (serialize duplicates) VERY slow Preview.

Board footer