To follow up on what >>6 said, a long time ago there used to be a website called ia.desync.com (later 0chan.org) which used some sort of crawler script to download all the images off of 4chan and a few other boards that don't come to mind now. When dealing with duplicate images he implemented what >>5 said regarding MD5's. Unfortunatley the site doesn't exist anymore because he ran into problems.

As far as I know problem he encountered was not bandwidth, but the fact that he had so many illegal (child porn probably being at the top) and copyrighted images that he was at huge legal risk. I'm 99% sure that is what caused him to take the site down, since several times he was complaining about legal threats from HentaiKey.

At one point he announced that he had over a quarter million images on his server, which is an insane amount. That was two or three years ago, and since anon-imageboards have grown so much since then, I doubt there will be any more image archives in the future, especially since CP has become a worsening problem. I hope that helps anyone who was wondering.