Whoever said storage is cheap, never had to buy and manage that storage.

I’ve been meaning to write in some depth about the challenges of storing digital photos for quite some time, unfortunately, this article isn’t it. However, with rising camera resolutions, and the respective increase in image file sizes, finding room to store, backup and properly manage a large volume of image files is becoming increasingly expensive and difficult.

Beyond practicing shutter discipline and being aggressive with our editing, there is one more aspect that is worth considering, and that’s compression.

Lossy vs. Lossless Compression vs. Photographers

Compression comes in two basic flavors, lossy and lossless. The difference between the two is that lossless compression produces a file that can exactly reproduce the original source material, while a lossy file throws “data” away to produce a more compressible file.

Lossy compression has always had something of a bad wrap in the photographic world, even when it didn’t actually throw away anything worth keeping. Strictly speaking, most photographers, and most of the people who will be most vocal about not using lossy formats don’t really understand what’s going on behind the scenes. That’s not to say we should all start shooting in JPEG now, because we shouldn’t. But we shouldn’t be so quick to write off “lossy” compression when it’s properly modeled, and fits the needs.

I don’t purprot to be an expert in compression, but without going into excruciating detail, it suffices to say, if the “lossy” algorithm is properly modeled to exploit the behavior of light, the distribution of data in an image, or the psychology and physiology of the human viewing the image it’s certainly possible to produce a “lossy” compression algorithm that is visually indistinguishable from a lossless one.

Lossy DNG

So what does lossy DNG actually do. Well I can tell you it’s not a DNG version of Nikon’s Lossy NEF. That is it doesn’t simply use a non-linear mapping function to convert 12 or 14 bits of data into 8 or 9. In fact, it’s considerably more complicated than that, on the flip side, it does produce some incredibly tiny files with no perceptible loss of image quality and still some headroom for post processing.

So how does it work?

According to an Adobe employee’s comments in this thread, it works something like this. The RAW file is demosaiced, to produce a RGB file similar to a TIFF or BMP. The color channels are then remapped using a 1D perceptual mapping algorithm, so that their values are spread across the full range of possible values. Some noise is added—dithering—in specific places as necessary to insure that the now remapped data doesn’t become blocky. The image is then compressed using the same compression algorithm that JPEG files normally uses.

The result is a file that’s often 1/3 or 1/4 the size of the source file, that retains more fidelity than a JPEG would and has sufficient data to allow the image to be processed to a limited degree but more extensively than a JPEG.

In addition to the processing and file size advantages, being stored in a DNG container also means that the file retains the DNG’s checksums to preserve image integrity.

Finally, Adobe did handle the two edge cases, converting a JPEG to a lossy DNG and re-compressing a lossy DNG as a lossy DNG. In both cases, the image data is copied bit-for-bit instead of being decompressed and recompressed.

Size v. IQ

Given that it’s lossy the real question is how much do I really lose versus how much do I really gain. Well lets start with a simple case, in the past when I wanted to save space with a snapshot kind of image, I would export the image as a high quality JPEG, to the same folder and replace the raw with the JPEG.

The RAW from my 10.1MP 1D mark 3, and this was a best case ISO 100 file, was 11.31MB. The JPEG produced by Lightroom that I would have used in the past, is 5.4MB. The lossy-DNG is only 3.83MB. But the real question is how do the images compare?

• RAW
• Lossy DNG
• aRGB JPEG 100
• aRGB JPEG 83

Judging the comparative quality of images is a tad more difficult. To my eye, I can’t detect a difference in detail between a 5MB JPEG 100, a 2MB JPEG 83, a 3.3MB lossy DNG, and the origioal 14-bit RAW. There is a color shift in the two JPEGs due to the color space they’re rendered in (the lossy DNG is still in the same capture color space as the RAW).

Of course these images are unprocessed, and certainly not heavily processed, which isn’t something I intend to be doing with any regularity with how I plan on using lossy DNGs. However, how the lossy-DNGs handle processing is something that I want to look at eventually. More specifically, I’m interested to see how the noise added during the quantization process ultimately effects image noise—though I’m not really expecting it to.

For me at least, lossy DNGs are definitely a useful tool that wasn’t there in previous Lightroom Versions. So far they seem to be smaller and offer more processing headroom than a quality 100 JPEG, and preserve the checksum feature of regular DNGs. Of course the real utility is based in a large part on how much storage you have and how much of your library is less than “serious” business. Granted, it’s not like any appreciable amount of my library is going to end up being converted from lossless RAWs to lossy DNGs, but as it stands on average every 3 files that I compress, I get space for another uncompressed RAW. Which in my opinion is a better storage solution than just throwing the images away.