Approve the Cookies
This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies. Read More.
OK
Index  •   • New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Register to forums    Log in

 
FORUMS General Gear Talk Computers
Thread started 04 Jun 2015 (Thursday) 15:31
Prev/next
sponsored links
(this ad will go away when you log in as registered member)

Duplicate photo file software

 
alexvtr
Member
Joined Nov 2012
Montreal, QC
Jun 04, 2015 15:31 |  #1

Hi all!

I did some searching (here and on the web) but can't find anything conclusive...

I need some advice on a good duplicate file finder (for Windows).
Like (maybe) some of you, by excess of precaution, I now have multiple duplicates of my photos across different locations (Laptop, desktop, removable storage, cloud, etc.).

MOST of the duplicates are simply redundant but in some cases, files were edited and such, resulting in different file names, dates and size for the same image.

My plan is to dump all my files from different locations on the same drive and run a good program through it.

I tried "Duplicate File Finder Pro" but it doesn't appear to handle multiple duplicates (like more that 2) all that well...

Any other suggestions?
Thanks in advance!


Canon EOS 60D | Canon EF 35mm f/2, EF-S 55-250mm f/4-5.6 IS STM | Canon Speedlite 430 EX

LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as registered member)
cacawcacaw
Goldmember
cacawcacaw's Avatar
2,862 posts
Joined May 2010
Ventura, California
Post has been edited over 2 years ago by cacawcacaw.
Jun 04, 2015 16:38 |  #2

Alex,

When I asked this same question a while back, most people just responded with their technique for post-processing of new photos. Nobody seemed to have a handle on how to organize a big mess of photos that includes some copies, some backups, and groups of photos with some culled and edited and some not.

It's not just a matter of having good duplicate-finding software, you need a procedure that lets you take a storage device, compare it to what you already have in your photo collection and then decide if you need to add those photos to the collection.

If you have a very small and well-organized group of photos, you might be able to do the whole process in Lightroom.

When you consider that some collections of photos have been culled (manually deleting bad photos, which is a job that you don't want to repeat) and that some have been edited in Lightroom (or put into Lightroom collections), the process can get tricky. You certainly don't want to take a culled and edited group of photos and then add un-culled or un-edited files into that group.

I think I figured this out some time ago, but I didn't take good notes. I'll try to describe what I did last time, as I need to do it again with my parents' photo collection.

Koert

p.s. As of a year ago, here are some of the programs that I found useful. There might be some newer programs worth looking at too.

Photo2folders
BreezeBrowser
Downloader Pro
Visipics
Visual Similarity Duplicate Finder Pro (VSDFP)


Replacing my Canon 7D, Tokina 12-24mm, Canon 17-55mm, Sigma 30mm f/1.4, 85mm f/1.4, and 150-500mm with a Panasonic Lumix FZ1000. I still have the 17-55 and the 30 available for sale.

LOG IN TO REPLY
CyberDyneSystems
Admin (type T-2000)
CyberDyneSystems's Avatar
47,623 posts
Gallery: 78 photos
Joined Apr 2003
Rhode Island USA
Jun 04, 2015 17:28 |  #3

Awesome duplicate Photo Finder (really, that's the name)

http://www.duplicate-finder.com/photo.htmlexternal link


GEAR LIST
CDS' HOT LINKS
Jake Hegnauer Photography (external link)

LOG IN TO REPLY
Mark0159
I say stupid things all the time
Mark0159's Avatar
12,710 posts
Gallery: 25 photos
Joined Mar 2003
Hamilton, New Zealand
Jun 04, 2015 17:33 |  #4

CyberDyneSystems wrote in post #17584393external link
Awesome duplicate Photo Finder (really, that's the name)

http://www.duplicate-finder.com/photo.htmlexternal link

I have used this app and it's quite good


Mark
https://www.flickr.com​/photos/52782633@N04 (external link)
Canon EOS 6D | Canon EF 24-105mm f/4L IS USM, EF 17-40mm f/4L USM, EF 50mm f/1.4 USM, EF 100mm f/2.8 Macro USM | Tamron SP 35mm F1.8 Di VC USD | Canon Speedlite 550EX -|- Film | Canon EOS 3 | Olympus OM2 | Zuiko 35mm f2

LOG IN TO REPLY
John ­ from ­ PA
Cream of the Crop
7,381 posts
Joined May 2003
Southeast Pennsylvania
Jun 04, 2015 20:40 |  #5

This article is from 2012 and is primarily for photography. http://www.pcworld.com ...ind-duplicate-photos.htmlexternal link




LOG IN TO REPLY
cacawcacaw
Goldmember
cacawcacaw's Avatar
2,862 posts
Joined May 2010
Ventura, California
Jun 04, 2015 22:28 |  #6

alexvtr wrote in post #17584272external link
... I now have multiple duplicates of my photos across different locations ...

It looks like the Awesome program requires that you look at the duplicates and make a decision for each pair of duplicates. Are you really going to do that 10,000 times?

Visipics automates the process. You set the parameters to determine what is considered a duplicate, and then set priorities to determine which copy should be discarded.


Replacing my Canon 7D, Tokina 12-24mm, Canon 17-55mm, Sigma 30mm f/1.4, 85mm f/1.4, and 150-500mm with a Panasonic Lumix FZ1000. I still have the 17-55 and the 30 available for sale.

LOG IN TO REPLY
nqjudo
Goldmember
nqjudo's Avatar
2,680 posts
Joined Apr 2007
Canada
Post has been edited over 2 years ago by nqjudo.
Jun 04, 2015 22:54 |  #7

cacawcacaw wrote in post #17584716external link
It looks like the Awesome program requires that you look at the duplicates and make a decision for each pair of duplicates. Are you really going to do that 10,000 times?

Visipics automates the process. You set the parameters to determine what is considered a duplicate, and then set priorities to determine which copy should be discarded.

The Awesome program is quite configurable so you can compare duplicates, similar, omit subdirectories, etc. If you did have 10,000 duplicate images I imagine that you definitely would want to take a look and make a decision about each. I certainly wouldn't leave discarding files to any kind of automation.


No photographer is as good as the simplest camera. - Edward Steichen.

LOG IN TO REPLY
cacawcacaw
Goldmember
cacawcacaw's Avatar
2,862 posts
Joined May 2010
Ventura, California
Jun 05, 2015 01:43 |  #8

nqjudo wrote in post #17584737external link
... I certainly wouldn't leave discarding files to any kind of automation.

Just as an example, if you made two extra copies (as backups) every time you did a photo shoot, you don't really want to go through each photo and verify that it's already in your main photo folder. It's way easier to run an automated program.

The problem becomes even more complicated when you have thousands of random images, edited copies, and backup copies spread across a collection of hard disks.

Obviously, the best solution is to never get into this type of situation. But I know there are a lot of photographers who have huge messes on their hands and are ashamed to admit it.


Replacing my Canon 7D, Tokina 12-24mm, Canon 17-55mm, Sigma 30mm f/1.4, 85mm f/1.4, and 150-500mm with a Panasonic Lumix FZ1000. I still have the 17-55 and the 30 available for sale.

LOG IN TO REPLY
alexvtr
THREAD ­ STARTER
Member
Joined Nov 2012
Montreal, QC
Jun 05, 2015 09:52 |  #9

cacawcacaw wrote in post #17584325external link
p.s. As of a year ago, here are some of the programs that I found useful. There might be some newer programs worth looking at too.

Photo2folders
BreezeBrowser
Downloader Pro
Visipics
Visual Similarity Duplicate Finder Pro (VSDFP)

Thanks! I'll look into this and update if I find anything else worthwhile.


Canon EOS 60D | Canon EF 35mm f/2, EF-S 55-250mm f/4-5.6 IS STM | Canon Speedlite 430 EX

LOG IN TO REPLY
cacawcacaw
Goldmember
cacawcacaw's Avatar
2,862 posts
Joined May 2010
Ventura, California
Jun 05, 2015 14:44 |  #10

Found some notes from last year. I'll make edits to this post as I come across problems or better techniques. Feel free to comment, criticize or recommend. These are just rough notes.

Goal:
To develop an efficient procedure for compiling a photo collection.

Problem:
Numerous storage devices with images, many copies and backups mixed in with originals, edited copies, and Lightroom catalogs.

Criteria:
Move all images into a single, organized, folder
Use a single Lightroom catalog for editing, sharing
Maintain backups

Steps:

1) Locate all Lightroom catalogs and consolidate (copy to a new location) all photos edited in Lightroom into a single new catalog. (With a folder structure of Photos/Year/Month/Occa​sion.)

2) Use a program like Photo2folders or BreezeBrowser and/or Downloader Pro to copy every available image file to a second location, without copying exact duplicates. (With a folder structure of AllPhotos/Year/Month/D​ay)

3) Use a program like Visipics or Visual Similarity Duplicate Finder Pro (VSDFP) to remove duplicates from the AllPhotos folder.

4) Set Photos and AllPhotos for periodic backups locally and with Carbonite or Mozy.

5) In Lightroom, Import (copy) small batches from the AllPhotos folder to the Photos folder. (Maybe rename AllPhotos folders that have been imported?)

6) For each batch, use Lightroom to cull, edit, and publish the photos.

7) Newly found photos will be checked (using VSDFP) against the AllPhotos folder and, if they aren't duplicates, moved into the folder.

8) Before erasing any photos from other locations, they'll be checked to make sure they are in the AllPhotos folder.

9) In the future, try to do more culling (deleting obviously unwanted photos) as photos are moved from the camera into the AllPhotos folder.


Replacing my Canon 7D, Tokina 12-24mm, Canon 17-55mm, Sigma 30mm f/1.4, 85mm f/1.4, and 150-500mm with a Panasonic Lumix FZ1000. I still have the 17-55 and the 30 available for sale.

LOG IN TO REPLY
CyberDyneSystems
Admin (type T-2000)
CyberDyneSystems's Avatar
47,623 posts
Gallery: 78 photos
Joined Apr 2003
Rhode Island USA
Jun 05, 2015 14:51 |  #11

I did something similar under your "Criteria" listing back in 2003 and it's worked wonderfully since. No lightroom involved, for me the trick is working with an understanding of the OS file structure.

All photos are in one directory, in dozens (hundreds?) of sub directories, on one drive.
Of course I have multiple copies on multiple hard drives, but at any time I can completely wipe any of those drives knowing that nothing unique is on any of them.


GEAR LIST
CDS' HOT LINKS
Jake Hegnauer Photography (external link)

LOG IN TO REPLY
Bleufire
Goldmember
Bleufire's Avatar
1,182 posts
Joined Mar 2008
California
Aug 23, 2017 00:36 as a reply to post 18433857 |  #12

Sweet thread resurrection Batman! This question is just over two years old!

:-D


5D*Sigma 50/1.4*EF 17-40/4
New to Photography? ----> ENJOY! Canon DSLR! (external link)

LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as registered member)

1,748 views & 0 likes for this thread
Duplicate photo file software
FORUMS General Gear Talk Computers


Not a member yet? Click here to register to the forums.
Registered members get all the features: search, following threads, own gear list and ratings, likes, more forums, private messaging, thread follow, notifications, own gallery, settings, view hosted photos, own reviews and more...


AAA

Send feedback to staff    •   Jump to forum...    •   Rules    •   Index    •   New posts    •   RTAT    •   'Best of'    •   Gallery    •   Gear    •   Reviews    •   Polls

COOKIES DISCLAIMER: This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies. Privacy policy and cookie usage info.

POWERED BY AMASS 1.4version 1.4
made in Finland
by Pekka Saarinen
for photography-on-the.net
Spent 0.00101 for 4 database queries.
PAGE COMPLETED IN 0.02s
Latest registered member is MVPhoto67
838 guests, 325 members online
Simultaneous users record so far is 6106, that happened on Jun 09, 2016