Data deduplication saves storage by keeping one copy of identical data and replacing repeats with references. It works on files or smaller blocks. It clashes with ordinary encryption, because encrypting the same file twice with different keys gives different output. Schemes that allow both give up some secrecy, so decide what you can accept.
How does deduplication work?
The system splits data into chunks, computes a fingerprint (a cryptographic hash) of each, and checks whether that fingerprint is already stored. If so, it stores a pointer instead of the data. Backup systems benefit most, since nightly backups contain mostly unchanged data. A constructed example: 50 virtual machines built from the same image share most of their blocks, so a backup repository might store that common part once.
Deduplication can happen at the client before upload, which saves bandwidth, or at the storage target after arrival.
Why does encryption get in the way?
With standard encryption, every user uses a different key and often a random value for each encryption. Two copies of the same document become two unrelated blobs. The storage system sees no matches, so it saves nothing.
What are the options?
Encrypt after deduplicating. The storage system sees plaintext, compares and then encrypts what it keeps. Savings are kept, but the provider sees your data unless you control the system.
Message locked encryption. Bellare, Keelveedhi and Ristenpart formalized this primitive: the encryption key is derived from the message itself, which provides a way to achieve secure deduplication, or space efficient secure outsourced storage. Identical files give identical ciphertext, so the server can deduplicate without reading them.
The cost is inherent. If equal files produce equal ciphertext, someone who can guess a file can encrypt the guess and check whether it exists in the store. That gives an attacker a confirmation tool for predictable content, such as a standard form with only a few unknown values. It does not suit low entropy data.
No deduplication for sensitive data. Sometimes the right answer is to pay for the storage.
What about side channels?
When deduplication happens on the client before upload, the speed or the response can reveal whether a file was already stored. A user could learn that someone else has uploaded the same file. This is a privacy leak from the saving itself, so shared multi tenant services need controls on it.
How does it relate to ransomware and backups?
Deduplicated backup stores concentrate data. A bad actor with write access can damage a lot at once, because many backups reference the same blocks. Protect the repository with immutability, separate credentials and retention locks, and test restores.
When is deduplication worth it?
It pays off most where data repeats heavily: backups, virtual machine images and shared document stores. It pays little for already compressed or encrypted media, and it adds processing and metadata overhead. Measure the ratio on a sample before committing, since claimed savings depend entirely on your data.
How to check or apply this
- Decide what you are trying to protect: stored data from the provider, from other tenants or from outsiders.
- Check whether deduplication runs on the client or the server, and who holds keys.
- Avoid message locked schemes for data an attacker could guess.
- Keep deduplication within one tenant or trust domain.
- Put backup repositories behind immutability and separate administrative credentials.
- Run restore tests regularly and check integrity of the shared blocks.
- Document the trade-off for auditors.
Quick check
Can deduplicated data be encrypted?
Yes, but you need to choose how: the server sees plaintext, or a message locked scheme where the same input gives the same output, with the guessing risk that brings.
Is deduplication the same as compression?
No. Compression shrinks data inside a file. Deduplication removes repeated data across files or blocks.
Sources
- Bellare, Keelveedhi and Ristenpart, Message-Locked Encryption and Secure Deduplication (IACR ePrint 2012/631)
- CISA, #StopRansomware Guide
Last reviewed: 18 October 2026.
