₦airaland Forum

1. Do you hash all the files
2. Do you delete by name
3 what would be an efficient way to do this
4 what programming language do you think would be best to do this in
5. How would you determine which of the duplicate to keep?

I would love to see you guys solution to the problem any programming language is allowed. If you using Linux and do it using bash more points to you lol

No googling am only interested in thought process so exact correctness does not necessarily matter. Good luck.

Hello ,

when you mean duplicated files , do you mean the OS allows filename to be same in same directory or different directories , cos I know Windows will not allow same filename in same directory.

If in different directory then , you delete either the old or new one, so date is very critical.

if in different directory then, you have to do a recursive search for each file across all directory,

Any programming language with access to files and directory can do it , C++,C,C#, php, python, java etc

though the OS will also decide choice of programming language

dplus:
Hello ,

when you mean duplicated files , do you mean the OS allows filename to be same in same directory or different directories , cos I know Windows will not allow same filename in same directory.

I different directory but identical files.

I onced wrote one in PHP
when I wanted to delete over 1k music file that contain around 3-5 duplicates each
it's was fun doing
Check it here https://github.com/iambrojohnson/DuplicateFileRemover

Deicide:
1. Do you hash all the files
2. Do you delete by name
3 what would be an efficient way to do this
4 what programming language do you think would be best to do this in
5. How would you determine which of the duplicate to keep?

I would love to see you guys solution to the problem any programming language is allowed. If you using Linux and do it using bash more points to you lol

No googling am only interested in thought process so exact correctness does not necessarily matter. Good luck.

Easy, I could do it with node js file system module plus other modules
You can loop over the directory and find a file "index.js" you then look for an "index-2.js" file or you store the contents in a variable and compare it, delete the second file if they have the same content

Fourpockets:
Easy, I could do it with node js file system module plus other modules
You can loop over the directory and find a file "index.js" you then look for an "index-2.js" file or you store the contents in a variable and compare it, delete the second file if they have the same content

How would you check for same content?

Iambro:
I onced wrote one in PHP
when I wanted to delete over 1k music file that contain around 3-5 duplicates each
it's was fun doing
Check it here https://github.com/iambrojohnson/DuplicateFileRemover

How fast was it?

Deicide:
How would you check for same content?

Using the FS module

Deicide:
How fast was it?

depends on the total files.
The program even won phpclass.org 2020 award.

Iambro:
depends on the total files.
The program even won phpclass.org 2020 award.

Wow that's impressive.

I think you have answered your own question. The best way is to hash all the files, this is done recursively from the root directory using a sha1-sum (you could use other hasing algorithms if you like), Linux provides a sha1-sum hash by default, then you loop through all your hashes, if you encounter a duplicate hash, you delete either of them, the date of file creation and filename doesn't matter since the content are the same. All this can be done with a bash script but I'd have to google to come up with a working solution. Even at that, my script might not be efficient because I'm not so versed at bash scripting.

Luckydonalds:
I think you have answered your own question. The best way is to hash all the files, this is done recursively from the root directory using a sha1-sum (you could use other hasing algorithms if you like), Linux provides a sha1-sum hash by default, then you loop through all your hashes, if you encounter a duplicate hash, you delete either of them, the date of file creation and filename doesn't matter since the content are the same. All this can be done with a bash script but I'd have to google to come up with a working solution. Even at that, my script might not be efficient because I'm not so versed at bash scripting.

You could use python though.

Deicide:
1. Do you hash all the files
2. Do you delete by name
3 what would be an efficient way to do this
4 what programming language do you think would be best to do this in
5. How would you determine which of the duplicate to keep?

I would love to see you guys solution to the problem any programming language is allowed. If you using Linux and do it using bash more points to you lol

No googling am only interested in thought process so exact correctness does not necessarily matter. Good luck.

i no sabi o

dodgelord:
i no sabi o

There is a question on leetcode "deleted duplicate" but instead of files you would do it with an array. You could use that to practice.