That's incredibly inefficient because the cost of storage is much lower than the cost of CPU and memory. There's really no reason to give them the benefit of the doubt. The odds of them fucking up are much greater than the odds of them being intelligent cryptologists.
Indeed. When you're doing deduplication, you typically want to have large runs of data, so small blocks just hurts things way too much as the overhead of tracking those small blocks quickly adds up to more than the savings you get by deduplicating those small blocks. Typically, if you do a block-level deduplication, you use whatever the native file system or hard drive block size is, which is usually somewhere between 512 and 4096 bytes. You also would want to be able to consolidate runs of identical blocks together as bigger blocks are more efficient than smaller blocks with respect to their tracking overhead.
Here's the thing -- deduplication isn't all that different from lossless compression like you'd find in zip files and such. Many of the same issues appear in both as they all need to be able to find like data spread out across one or multiple files so that you're not storing redundant data. Deduplication's magic mostly comes from being able to do the compression and decompression, if you will, on the fly and across multiple files on the entire storage system. Not just the file system, as you may have some sort of clustered storage tech where you're storing the data across multiple individual file systems but want the deduplication applied across the entire cluster.
Indeed. You can almost think of deduplication as putting every single file into one giant zip file while keeping enough metadata around so that you can extract individual files from the one gigantic zip.
In the old MegaUpload if 100 people uploaded the same exact file, this method could save a ton of storage space with only a small cost. If the new MEGA truly encrypts every upload with different keys, then 100 people uploading the same exactly file will actually produce 100 completely different files. That will make deduplication almost worthless.
Comments
Here's the thing -- deduplication isn't all that different from lossless compression like you'd find in zip files and such. Many of the same issues appear in both as they all need to be able to find like data spread out across one or multiple files so that you're not storing redundant data. Deduplication's magic mostly comes from being able to do the compression and decompression, if you will, on the fly and across multiple files on the entire storage system. Not just the file system, as you may have some sort of clustered storage tech where you're storing the data across multiple individual file systems but want the deduplication applied across the entire cluster.
In the old MegaUpload if 100 people uploaded the same exact file, this method could save a ton of storage space with only a small cost. If the new MEGA truly encrypts every upload with different keys, then 100 people uploading the same exactly file will actually produce 100 completely different files. That will make deduplication almost worthless.
Well, so much for that.