File Distribution
I have a folder that has several thousand photo files in it. I need to distribute the files into enough folders so that each folder doesn't have more than 5000 files. Is it possible to use Advanced Renamer to do this? I was told it was (by what may have been an unreliable source), so I purchased a Personal Use license. I'm tearing my hair out trying to figure out how to do it, though.
Reply to #1:
Welcome Carolyn!
Yes it is possible in Aren, using a script method (and maybe other ways).
If you are adept at Bash, or Python, or PowerShell -- or if you foolishly trust AI generated code -- those would be faster for just straight moving.
Otherwise, I will post an Aren solution here tomorrow if nobody beats me to it.
Important: Based on bitter experience, might I suggest that you make the limit 2000 files per folder?
Lots of things start to bog, at least on PCs, when you get higher than that.
Also, *** How do you want the new folders organized? ***
EG from: src_folder to...
mypics/batch_01
mypics/batch_02
mypics/batch_03
etc.?
Regards,
Randy
Welcome Carolyn!
Yes it is possible in Aren, using a script method (and maybe other ways).
If you are adept at Bash, or Python, or PowerShell -- or if you foolishly trust AI generated code -- those would be faster for just straight moving.
Otherwise, I will post an Aren solution here tomorrow if nobody beats me to it.
Important: Based on bitter experience, might I suggest that you make the limit 2000 files per folder?
Lots of things start to bog, at least on PCs, when you get higher than that.
Also, *** How do you want the new folders organized? ***
EG from: src_folder to...
mypics/batch_01
mypics/batch_02
mypics/batch_03
etc.?
Regards,
Randy
Reply to #2:
Oh, thank you! I am the farthest thing from adept. :'D Your suggested organization would be great. And 2000 would work just fine!
Oh, thank you! I am the farthest thing from adept. :'D Your suggested organization would be great. And 2000 would work just fine!
Reply to #3:
Hi Carolyn,
It is possible to do, but not exactly "easy" for someone with no coding experience. The MOVE batch mode built into ARen is designed to bulk-move all the files in the "files list" into a new folder, so we have to resort to a javascript snippet to get the job done. I'll walk you through it.
First, read the first two paragraphs on the SCRIPT method in the user manual, just to get some familiarity with the mechanics of the process.
It's here: https://www.advancedrenamer.com/user_guide/method_script
(Don't expect the picture to necessarily line up exactly with what you'll see when you open a new script method)
OK, having read that, and assuming you know enough about ARen to start a new method list and load files (if not let us know, but for brevity I'll skip all that here), open ARen and add a "Script" method (the green "plus-sign" button at the top of the "Renaming methods" section on the left). Once you have a script method, press the "Pre batch script..." button and put this one line in the pre-batch section:
let dirCount = 0 ;
[dirCount is the variable we'll be using to increment for each of your folders; it's in pre-batch so that it will be consistent throughout the process of applying a script to each filename.]
Now press the "Close and apply script" button at the bottom of the pre batch window and you'll return to the main script method.
OK, now we insert our main script-- the one that actually does the moving (it executes once for each filename). Copy and paste the following lines between the dashed lines (*overwrite anything already there!):
// ----------------------------------------------------------
// PREBATCH: let dirCount = 0 ;
let dirBase = item.path + "batch_" ;
let limit = 2000 ; // change if needed/wanted...
if ( index % limit == 0 ) {
++dirCount ;
newDir = dirBase + dirCount.toString() ;
}
item.newPath = newDir ;
return item.newBasename ;
// ----------------------------------------------------------
Some notes: 1) The new directories will be in the directory where the files now reside;
2) you can change the number of files to put in each directory, just change the "2000" in the third line, "let limit = 2000 ;" I tried it with 1500, 2000, and 2500, but it should work to infinity and beyond :) just kidding, of course, 10,000 files is ARen's default limit, though if you are using an SSD drive you can go higher without too much delay (my limit is set at 25,000 on SSD drives).
3) I wouldn't mess with anything else in the script. If you want to change something else, just ask.
4) VERY IMPORTANT! If you decide to do this in more than one big run you will end up putting more files in the folders you made on a preceding run (because the dirCount will always go back to 0 to start in the pre-batch). Maybe Randy's script will have a way around this, and I'll give it some thought too, but in the meantime if you need to do that you'd need to change the Pre-batch script line from "let dirCount = 0 ;" to let dirCount = " (and whatever the last folder number was, i.e., if it was "batch_4" on the last run you would use the line "let dirCount = 4 ;" (without the quote marks, of course).
Once that's done, press the "Apply script" button (top right of script window).
Next, load your files (simplest way is just to drag and drop the files directly; or you can use the built-in loader. It's the folder icon at the far left of the AREn window).
That's all you have to do, other than to press the "Start batch" button, but if I were you I'd click the "Columns" button on the menu and add the "New Path" and "Index" columns if not already visible. You may have to move columns around to get a clear picture of what will happen when you press "Start batch", but since this won't be changing filenames you really should see what effect this script will have before executing it. And remember, you can always press the "undo batch" button to return to the previous state if something goes wrong. Just do it as soon as possible if you need it! After all, I tried it and it worked on 5739 files, but that doesn't mean there couldn't be a slip from cup to lip here. :)
Randy will probably have a better script for you (don't tell him I said so but he's a better javascript coder than I am :) so I wouldn't be mad if you waited for his script; however, within the parameters I've outlined this should work fine for you.
Best wishes and good luck! And if you have more questions don't hesitate... it's been slow around here lately and a good question always perks us up! :) And PS: don't tear your hair out, take it from this bald guy, a few files are not worth it! 8-P
DF
Hi Carolyn,
It is possible to do, but not exactly "easy" for someone with no coding experience. The MOVE batch mode built into ARen is designed to bulk-move all the files in the "files list" into a new folder, so we have to resort to a javascript snippet to get the job done. I'll walk you through it.
First, read the first two paragraphs on the SCRIPT method in the user manual, just to get some familiarity with the mechanics of the process.
It's here: https://www.advancedrenamer.com/user_guide/method_script
(Don't expect the picture to necessarily line up exactly with what you'll see when you open a new script method)
OK, having read that, and assuming you know enough about ARen to start a new method list and load files (if not let us know, but for brevity I'll skip all that here), open ARen and add a "Script" method (the green "plus-sign" button at the top of the "Renaming methods" section on the left). Once you have a script method, press the "Pre batch script..." button and put this one line in the pre-batch section:
let dirCount = 0 ;
[dirCount is the variable we'll be using to increment for each of your folders; it's in pre-batch so that it will be consistent throughout the process of applying a script to each filename.]
Now press the "Close and apply script" button at the bottom of the pre batch window and you'll return to the main script method.
OK, now we insert our main script-- the one that actually does the moving (it executes once for each filename). Copy and paste the following lines between the dashed lines (*overwrite anything already there!):
// ----------------------------------------------------------
// PREBATCH: let dirCount = 0 ;
let dirBase = item.path + "batch_" ;
let limit = 2000 ; // change if needed/wanted...
if ( index % limit == 0 ) {
++dirCount ;
newDir = dirBase + dirCount.toString() ;
}
item.newPath = newDir ;
return item.newBasename ;
// ----------------------------------------------------------
Some notes: 1) The new directories will be in the directory where the files now reside;
2) you can change the number of files to put in each directory, just change the "2000" in the third line, "let limit = 2000 ;" I tried it with 1500, 2000, and 2500, but it should work to infinity and beyond :) just kidding, of course, 10,000 files is ARen's default limit, though if you are using an SSD drive you can go higher without too much delay (my limit is set at 25,000 on SSD drives).
3) I wouldn't mess with anything else in the script. If you want to change something else, just ask.
4) VERY IMPORTANT! If you decide to do this in more than one big run you will end up putting more files in the folders you made on a preceding run (because the dirCount will always go back to 0 to start in the pre-batch). Maybe Randy's script will have a way around this, and I'll give it some thought too, but in the meantime if you need to do that you'd need to change the Pre-batch script line from "let dirCount = 0 ;" to let dirCount = " (and whatever the last folder number was, i.e., if it was "batch_4" on the last run you would use the line "let dirCount = 4 ;" (without the quote marks, of course).
Once that's done, press the "Apply script" button (top right of script window).
Next, load your files (simplest way is just to drag and drop the files directly; or you can use the built-in loader. It's the folder icon at the far left of the AREn window).
That's all you have to do, other than to press the "Start batch" button, but if I were you I'd click the "Columns" button on the menu and add the "New Path" and "Index" columns if not already visible. You may have to move columns around to get a clear picture of what will happen when you press "Start batch", but since this won't be changing filenames you really should see what effect this script will have before executing it. And remember, you can always press the "undo batch" button to return to the previous state if something goes wrong. Just do it as soon as possible if you need it! After all, I tried it and it worked on 5739 files, but that doesn't mean there couldn't be a slip from cup to lip here. :)
Randy will probably have a better script for you (don't tell him I said so but he's a better javascript coder than I am :) so I wouldn't be mad if you waited for his script; however, within the parameters I've outlined this should work fine for you.
Best wishes and good luck! And if you have more questions don't hesitate... it's been slow around here lately and a good question always perks us up! :) And PS: don't tear your hair out, take it from this bald guy, a few files are not worth it! 8-P
DF
DF's method is basically what I had envisioned. My version is below with maybe more spit or polish (at least one of those).
1) First, Aren only does 10K files at a time by default. If you want to try and do more than that at one time, go to:
Settings -> Renaming -> Item limit
And set a higher value.
Just beware that Aren can slow dramatically with large numbers of files at a time,
2) Second, so that Aren doesn't chew up even more time analyzing the files, I recommend checking all four of these while you are doing the this file moving:
Settings -> Media files -> Don't analyze image metadata
Settings -> Media files -> Don't analyze audio metadata
Settings -> Media files -> Don't analyze video metadata
Settings -> Media files -> Don't analyze document metadata
3) Neither DF or I can do this task as well as we would like, because the script currently can't see basic file information on the disk like the number of files and target folders.
Most of the difference between DF's script and mine is trying to kludge around that limitation.
Please go and up-vote DF's Feature request pertaining to this: https://advancedrenamer.fider.io/posts/4/js-love
4) Lastly, for a large quantity of files, you probably have lots of duplicates. You might want to purge them.
There are hundreds of duplicate remover utilities, if you don't already have one. Make sure it uses the recycle bin and shows you the log of each batch.
I use "CloneSpy". Its old but free, fast, powerful, and flexible.
===================================================
== My Script Solution
===================================================
Features:
1) Can remember the last folder used. But only if you always include an "index" file. More below.
2) Zero-pads the folder numbers for better legibility and proper sorting.
EG: batch_001, batch_002, etc.
3) Can move the files to any folder tree you like. More below.
4) We can show you how to run this from a batch file, which means you would just double-click a file/link to run this script repeatedly as needed.
-----------------
Instructions
-----------------
1) (Optional but recommend) Go to the folder that you will be moving files from and create a new text file named
"_Do Not Delete, last folder used 0.txt". It can remain an empty file.
You only need to do this once. The file will be renumbered each time you run the script.
2) Create a new script method ( https://www.advancedrenamer.com/user_guide/v4/method_script ).
3) Paste this next code in the "Pre batch script" and apply it:
//-------------------------------------------------------
// Pre batch script
//-------------------------------------------------------
//--- Most of this is needed because we cannot read folder information...
const maxNumFilesPerDir = 2000; // Hard coded for now. This could also be tracked by the index file.
const indexFilePrefix = "_Do Not Delete, last folder used ";
const indexFileRegex = new RegExp (`^${indexFilePrefix}(\\d+)$`, "i");
let destDirIndex = 0;
let numberOfFiles = app.itemCount;
let numDirsCreated = 0; //-- Total, including preexisting.
//-- Get the last folder-number used, if any.
for (let J = 0; J < numberOfFiles; J++) {
let item = app.getItem (J);
let zFilename = item.name;
if (indexFileRegex.test(zFilename) ) {
let lastDirNumStr = zFilename.replace (indexFileRegex, "$1") ;
destDirIndex = parseInt (lastDirNumStr) || 0;
break;
}
}
//--- We don't know when the index file will appear. So we must precalculate the number of folders that will be used.
numDirsCreated = Math.floor (numberOfFiles / maxNumFilesPerDir) + 1 + destDirIndex;
//-------------------------------------------------------
4) Then paste the following code in the "function(index, item)" block and apply it.
*** Be sure to overwrite/replace anything that was in that input before. ***
//-------------------------------------------------------
// "function( index, item)" script
//-------------------------------------------------------
let subDirPrefix = "batch_"; // Set to taste.
if (indexFileRegex.test(item.name) ) {
//--- This is the special index file.
item.newPath = item.path; //-- Don't move this file.
let newName = indexFilePrefix + String (numDirsCreated).padStart (4, "0");
return newName; //-- But do rename it.
}
//--- For normal files...
if (index % maxNumFilesPerDir === 0) {
destDirIndex++;
}
let destDir = item.newPath + "\\" + subDirPrefix + String (destDirIndex).padStart (3, "0");
item.newPath = destDir; //-- Set the split destination folder for this file.
//-------------------------------------------------------
5) Save the method list as "Move Files in Batches to Size Controlled Folders.aren" to a folder that makes sense to you.
But not the same folder as the pictures! ;)
-----------------
Using it.
-----------------
A) Make sure the "Batch mode:" remains set to "Move."
B) Set the desired base folder in the "Output folder" input near the top center of Aren's window.
For example, if you set "C:\mypics" then the files will be moved like so:
C:\mypics\batch_001
C:\mypics\batch_002
etc.
If you leave this blank, then new folders will be in the same folder that your pictures started in. That's probably not what you want.
C) When you run a batch, add the "Do Not Delete, last folder used...txt" file every time, along with the photos.
This file will be added automatically if you just add the same source folder every time.
D) The order that the files are shown in the window is the oder in which they will be moved/grouped. You can click on the column headers to sort by name, date, etc.
E) As DF said, Aren has an excellent "Undo" feature. (^_^)
F) It's easy to set this up as a shortcut that you can just click on every time to sort a new batch of files. (But please ask a new question, if you want that.)
Regards,
Randy
1) First, Aren only does 10K files at a time by default. If you want to try and do more than that at one time, go to:
Settings -> Renaming -> Item limit
And set a higher value.
Just beware that Aren can slow dramatically with large numbers of files at a time,
2) Second, so that Aren doesn't chew up even more time analyzing the files, I recommend checking all four of these while you are doing the this file moving:
Settings -> Media files -> Don't analyze image metadata
Settings -> Media files -> Don't analyze audio metadata
Settings -> Media files -> Don't analyze video metadata
Settings -> Media files -> Don't analyze document metadata
3) Neither DF or I can do this task as well as we would like, because the script currently can't see basic file information on the disk like the number of files and target folders.
Most of the difference between DF's script and mine is trying to kludge around that limitation.
Please go and up-vote DF's Feature request pertaining to this: https://advancedrenamer.fider.io/posts/4/js-love
4) Lastly, for a large quantity of files, you probably have lots of duplicates. You might want to purge them.
There are hundreds of duplicate remover utilities, if you don't already have one. Make sure it uses the recycle bin and shows you the log of each batch.
I use "CloneSpy". Its old but free, fast, powerful, and flexible.
===================================================
== My Script Solution
===================================================
Features:
1) Can remember the last folder used. But only if you always include an "index" file. More below.
2) Zero-pads the folder numbers for better legibility and proper sorting.
EG: batch_001, batch_002, etc.
3) Can move the files to any folder tree you like. More below.
4) We can show you how to run this from a batch file, which means you would just double-click a file/link to run this script repeatedly as needed.
-----------------
Instructions
-----------------
1) (Optional but recommend) Go to the folder that you will be moving files from and create a new text file named
"_Do Not Delete, last folder used 0.txt". It can remain an empty file.
You only need to do this once. The file will be renumbered each time you run the script.
2) Create a new script method ( https://www.advancedrenamer.com/user_guide/v4/method_script ).
3) Paste this next code in the "Pre batch script" and apply it:
//-------------------------------------------------------
// Pre batch script
//-------------------------------------------------------
//--- Most of this is needed because we cannot read folder information...
const maxNumFilesPerDir = 2000; // Hard coded for now. This could also be tracked by the index file.
const indexFilePrefix = "_Do Not Delete, last folder used ";
const indexFileRegex = new RegExp (`^${indexFilePrefix}(\\d+)$`, "i");
let destDirIndex = 0;
let numberOfFiles = app.itemCount;
let numDirsCreated = 0; //-- Total, including preexisting.
//-- Get the last folder-number used, if any.
for (let J = 0; J < numberOfFiles; J++) {
let item = app.getItem (J);
let zFilename = item.name;
if (indexFileRegex.test(zFilename) ) {
let lastDirNumStr = zFilename.replace (indexFileRegex, "$1") ;
destDirIndex = parseInt (lastDirNumStr) || 0;
break;
}
}
//--- We don't know when the index file will appear. So we must precalculate the number of folders that will be used.
numDirsCreated = Math.floor (numberOfFiles / maxNumFilesPerDir) + 1 + destDirIndex;
//-------------------------------------------------------
4) Then paste the following code in the "function(index, item)" block and apply it.
*** Be sure to overwrite/replace anything that was in that input before. ***
//-------------------------------------------------------
// "function( index, item)" script
//-------------------------------------------------------
let subDirPrefix = "batch_"; // Set to taste.
if (indexFileRegex.test(item.name) ) {
//--- This is the special index file.
item.newPath = item.path; //-- Don't move this file.
let newName = indexFilePrefix + String (numDirsCreated).padStart (4, "0");
return newName; //-- But do rename it.
}
//--- For normal files...
if (index % maxNumFilesPerDir === 0) {
destDirIndex++;
}
let destDir = item.newPath + "\\" + subDirPrefix + String (destDirIndex).padStart (3, "0");
item.newPath = destDir; //-- Set the split destination folder for this file.
//-------------------------------------------------------
5) Save the method list as "Move Files in Batches to Size Controlled Folders.aren" to a folder that makes sense to you.
But not the same folder as the pictures! ;)
-----------------
Using it.
-----------------
A) Make sure the "Batch mode:" remains set to "Move."
B) Set the desired base folder in the "Output folder" input near the top center of Aren's window.
For example, if you set "C:\mypics" then the files will be moved like so:
C:\mypics\batch_001
C:\mypics\batch_002
etc.
If you leave this blank, then new folders will be in the same folder that your pictures started in. That's probably not what you want.
C) When you run a batch, add the "Do Not Delete, last folder used...txt" file every time, along with the photos.
This file will be added automatically if you just add the same source folder every time.
D) The order that the files are shown in the window is the oder in which they will be moved/grouped. You can click on the column headers to sort by name, date, etc.
E) As DF said, Aren has an excellent "Undo" feature. (^_^)
F) It's easy to set this up as a shortcut that you can just click on every time to sort a new batch of files. (But please ask a new question, if you want that.)
Regards,
Randy
Reply to #5:
Thank you both so, so much! In life before children (33 years ago!), I was a programmer, but mostly in a proprietary Pascal-like language. Ancient. I think I'll recognize what's happening in your scripts once I read the primer you recommend this weekend. Again, I thank you!
Thank you both so, so much! In life before children (33 years ago!), I was a programmer, but mostly in a proprietary Pascal-like language. Ancient. I think I'll recognize what's happening in your scripts once I read the primer you recommend this weekend. Again, I thank you!
Reply to #6:
You're welcome! Only two pieces of code I wrote 33 tears ago are still in use (by someone other than myself). Children was the better choice! (^_^)
You're welcome! Only two pieces of code I wrote 33 tears ago are still in use (by someone other than myself). Children was the better choice! (^_^)