Approve the Cookies
This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and our Privacy Policy.
OK
Index  •   • New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Guest
New posts  •   • RTAT  •   • 'Best of'  •   • Gallery  •   • Gear  •   • Reviews
Register to forums    Log in

 
FORUMS Post Processing, Marketing & Presenting Photos RAW, Post Processing & Printing 
Thread started 17 Apr 2014 (Thursday) 17:28
Search threadPrev/next
sponsored links
(this ad will go away when you log in as a registered member)

Software that can recognize duplicate(ed) RAW files?

 
crammond
Member
59 posts
Joined Mar 2011
     
Apr 17, 2014 17:28 |  #1

First the problem;

A My Book external HDD died. It was full of RAW files. We had it serviced and it was fully restored but it came back with all the files duplicated but with different names.

Is there any kind of a software that could detect the duplicates and allow you to somehow easily delete them?

Thanks.


5D2 | Canon 16-35L 2.8 | Sigma 35 1.4 | Sigma 70-200 2.8 OS | Canon 100L 2.8 | 580EX II

  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)
tonylong
...winded
Avatar
54,657 posts
Gallery: 60 photos
Likes: 542
Joined Sep 2007
Location: Vancouver, WA USA
     
Apr 17, 2014 17:32 |  #2

Hmm, I don't know, what software have you been using with your Raw files?


Tony
Two Canon cameras (5DC, 30D), three Canon lenses (24-105, 100-400, 100mm macro)
Tony Long Photos on PBase (external link)
Wildlife project pics here (external link), Biking Photog shoots here (external link), "Suburbia" project here (external link)! Mount St. Helens, Mount Hood pics here (external link)

  
  LOG IN TO REPLY
ro1313
Member
33 posts
Likes: 3
Joined Mar 2012
     
Apr 17, 2014 18:58 |  #3

Here is something you could try
Set view to detail and sort by date.
If you are lucky....all the dupes will show the same creation date and you can select them and delete
You would have to do this for each folder...but its worth a look




  
  LOG IN TO REPLY
BigAl007
Cream of the Crop
Avatar
7,996 posts
Gallery: 542 photos
Best ofs: 1
Likes: 1613
Joined Dec 2010
Location: Repps cum Bastwick, Gt Yarmouth, Norfolk, UK.
     
Apr 17, 2014 19:01 |  #4

Most file browsers even, let alone image management software can sort the images by the time of exposure. This will display all of the images in the order they were taken, so the duplicates will be next to each other. Then just go through and delete them one at a time. I also have a version control program called winmerge that can check drives for files and compare them, there are others too if you check Google for version control software.

Alan


My Flickr (external link)
My new Aviation images blog site (external link)

  
  LOG IN TO REPLY
ro1313
Member
33 posts
Likes: 3
Joined Mar 2012
     
Apr 17, 2014 19:11 |  #5

They will be listed by the time of exposure but if you right click on a column header you can select creation date and that is the date and time they were put on the Hard Drive




  
  LOG IN TO REPLY
tonylong
...winded
Avatar
54,657 posts
Gallery: 60 photos
Likes: 542
Joined Sep 2007
Location: Vancouver, WA USA
     
Apr 17, 2014 19:50 |  #6

If the copy files have similar "base" names then you can also set your file browser to Sort by Name...


Tony
Two Canon cameras (5DC, 30D), three Canon lenses (24-105, 100-400, 100mm macro)
Tony Long Photos on PBase (external link)
Wildlife project pics here (external link), Biking Photog shoots here (external link), "Suburbia" project here (external link)! Mount St. Helens, Mount Hood pics here (external link)

  
  LOG IN TO REPLY
crammond
THREAD ­ STARTER
Member
59 posts
Joined Mar 2011
     
Apr 18, 2014 06:17 |  #7

I get that you could sort them in the file browser but we are talking about 180.000 photos here..impossible to delete one bye one :)


5D2 | Canon 16-35L 2.8 | Sigma 35 1.4 | Sigma 70-200 2.8 OS | Canon 100L 2.8 | 580EX II

  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
Apr 18, 2014 06:27 |  #8

I would probably write a tiny program that computed MD5 for every file, and then sorted this list. That should group together all identical copies.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
ericcrazyman
Member
190 posts
Gallery: 1 photo
Likes: 12
Joined Apr 2013
Location: New Brunswick, Canada
     
Apr 18, 2014 07:24 |  #9

ACDSEE has a program that looks for same photo files and does not look at file names. It is an older program so it may not be had anymore. Memory fails as to the name of it. It is not shown in any of their new software on their website. It is in their acdsee 5 software bundle, they are at 17 now.


A photograph is worth a thousand words, but Photoshop can produce a thousand lies. Photo editing allowed. I need all the help I can get.

  
  LOG IN TO REPLY
iwannabe
Senior Member
Avatar
250 posts
Likes: 8
Joined Aug 2005
Location: Fort Wayne, Indiana
     
Apr 18, 2014 07:32 |  #10

I use DoubleKiller. Works for me.
http://download.cnet.c​om …29093&dlc=n&par​t=fivemill (external link)


Cameras: Canon 5D Mark III, 7D Mark II, Flashes: Canon 420EX, 580EX1
Lens: Canon 50 1.8II, Canon 85 1.8, Canon 17-55 2.8, Canon 24-70 F4.0,Sigma 30 1.4, Canon 70-200 F4/L IS, Tokina 11-16 2.8II, Tamron 70-200 2.8

  
  LOG IN TO REPLY
DGStinner
Goldmember
Avatar
1,040 posts
Gallery: 24 photos
Likes: 194
Joined Jan 2014
Location: Middlesex, New Jersey
     
Apr 18, 2014 07:35 |  #11

pwm2 wrote in post #16843121 (external link)
I would probably write a tiny program that computed MD5 for every file, and then sorted this list. That should group together all identical copies.

Or download one of the free programs already created which can hash the files using MD5 or SHA1 and compare them with another folder. One example is Karen's Hasher.


Dave Stinner
Gripped 7DII | 6DII
EF 16-35mm f/4L IS USM | EF 24-105mm f/4L | EF 70-200mm f/4L IS USM | EF 50mm f/1.8 STM | EF 100mm f/2.8L Macro IS USM | Rokinon 8mm | Sigma 150-600
Instagram (external link)

  
  LOG IN TO REPLY
ericcrazyman
Member
190 posts
Gallery: 1 photo
Likes: 12
Joined Apr 2013
Location: New Brunswick, Canada
     
Apr 18, 2014 07:53 |  #12

http://www.ashisoft.co​m/duplicate-images-finder.htm (external link)


A photograph is worth a thousand words, but Photoshop can produce a thousand lies. Photo editing allowed. I need all the help I can get.

  
  LOG IN TO REPLY
PixelMagic
Cream of the Crop
5,546 posts
Likes: 4
Joined Nov 2007
Location: Racine, WI
     
Apr 18, 2014 11:45 |  #13

You can use a dedicated duplicate file finder; there are several available but I prefer NoClone (external link).

Another approach would be to move all your images to a clean drive if one is available, then use a downloading utility like Downloader Pro (external link) or ImageIngester (external link) to move them back to their permanent location. Either utility can be configured to read the EXIF metadata in each image and filter out duplicates (e.g. images with the same Created Date and Time).


Gear List

  
  LOG IN TO REPLY
pwm2
"Sorry for being a noob"
Avatar
8,626 posts
Likes: 3
Joined May 2007
Location: Sweden
     
Apr 18, 2014 14:53 |  #14

I would be a bit scared just matching on the EXIF. What if not all files are perfectly restored, so once copy is good and one isn't. The EXIF would say they are the same, but the delete program wouldn't know that one of the files might be broken since the EXIF doesn't tell if the full files are identical.


5DMk2 + BG-E6 | 40D + BG-E2N | 350D + BG-E3 + RC-1 | Elan 7E | Minolta Dimage 7U | (Gear thread)
10-22 | 16-35/2.8 L II | 20-35 | 24-105 L IS | 28-135 IS | 40/2.8 | 50/1.8 II | 70-200/2.8 L IS | 100/2.8 L IS | 100-400 L IS | Sigma 18-200DC
Speedlite 420EZ | Speedlite 580EX | EF 1.4x II | EF 2x II

  
  LOG IN TO REPLY
sponsored links
(this ad will go away when you log in as a registered member)

2,169 views & 0 likes for this thread
Software that can recognize duplicate(ed) RAW files?
FORUMS Post Processing, Marketing & Presenting Photos RAW, Post Processing & Printing 
AAA
x 1600
y 1600

Jump to forum...   •  Rules   •  Index   •  New posts   •  RTAT   •  'Best of'   •  Gallery   •  Gear   •  Reviews   •  Member list   •  Polls   •  Image rules   •  Search   •  Password reset

Not a member yet?
Register to forums
Registered members may log in to forums and access all the features: full search, image upload, follow forums, own gear list and ratings, likes, more forums, private messaging, thread follow, notifications, own gallery, all settings, view hosted photos, own reviews, see more and do more... and all is free. Don't be a stranger - register now and start posting!


COOKIES DISCLAIMER: This website uses cookies to improve your user experience. By using this site, you agree to our use of cookies and to our privacy policy.
Privacy policy and cookie usage info.


POWERED BY AMASS forum software 2.1forum software
version 2.1 /
code and design
by Pekka Saarinen ©
for photography-on-the.net

Latest registered member is SaraiukiSama
1767 guests, 204 members online
Simultaneous users record so far is 15144, that happened on Nov 22, 2018

Photography-on-the.net Digital Photography Forums is the website for photographers and all who love great photos, camera and post processing techniques, gear talk, discussion and sharing. Professionals, hobbyists, newbies and those who don't even own a camera -- all are welcome regardless of skill, favourite brand, gear, gender or age. Registering and usage is free.