Some Raw Truth about RAW Converters

Some Raw Truth about RAW Converters

Your RAW conversion program may be damaging your image
by
Rob Nelson

Introduction

Until recently, I held RAW conversion programs as basically sacrosanct.  I realized that I was confusing that feeling with the purity of the RAW image itself.  The idea is that the RAW image is the pure CCD image, so when we ask for it from our camera, and through whichever RAW converter we’re using, we expect a faithful version of the RAW image.

As it is, all RAW conversion programs must do some work to get the image to you.  Demosaicing, some limited auto-levels, and typically a gamma curve.  Added onto that is some color balancing, and perhaps some other functions.

But, after what should be innocuous operations, we expect a RAW file that is not just fairly representative, but very faithfully representative of the RAW image. That’s why we use RAW.

Many RAW editors — free and expensive ones both — will actually damage your image before you even see it, in many ways.

In Part I, I discuss Black and White points in the histogram, as well as range compression.  These are issues a RAW conversion program should handle carefully, and it turns out that many don’t.

This article is for Sagelight users and non-Sagelight users alike.  For Sagelight users, this article answers a question that comes up every once-in-a-while: “How come my (automatically processed) RAW image looks plain compared to other RAW conversion programs?“.  For non-Sagelight users, this can tell you what to look for in your preferred RAW conversion program, whatever it is.

How I Ended Up Getting on my Soap Box about RAW conversion

The other day, a user sent me a couple images converted from their original RAW format.  One was a Sagelight image, and the other was from a fairly expensive package.   When I looked at the two pictures, I didn’t need to look at the labeling to see which one was from Sagelight and which one was from the other package.

With the Sagelight image, I saw exactly what I expected: A well-formed histogram without range compression on either side.  By now, I think it’s probably obvious that the user liked the other image better.   After making a convincing argument that Sagelight was doing it right, and this other RAW conversion program had, in fact, irreparably damaged their image, I decided to write an article on it.

This isn’t the first time I went through this time-consuming process of convincing someone that their RAW converter damaged their image, but it was the first time I decided to take a look at a number of RAW conversion programs to see what is going on under the hood with these things.  It’s been a known factor that Sagelight does things differently than a lot of RAW editors, and this is for a reason.  Not so much to be different, but because most RAW editors take liberties with your picture that they shouldn’t.

Before I researched it, I didn’t realize the extent to which many RAW conversion programs will damage your image before you even see it.

I’m from the old school of things, I guess.  Sagelight, like many editors, makes many assumptions about pictures to help you along with your image and to make things easier.  Sagelight takes the viewpoint of not doing anything for you automatically that will harm your image, such as cutting black-and-white points too much. But every editor is different, and they should be. There’s enough room for many editors in this field.  But, RAW images… Well, that’s a different story.

All Raw Converters Are Not Created Equal

With RAW, however, I thought all editors were the same.  With Sagelight, I make the assumption that if you’re using a RAW image then you’re serious.  Otherwise, you could just accept the JPEG from the camera after it has done a host of image processing functions on it — after all, even with a 24-bit JPEG there is still plenty of room to edit your picture and make it look more to your own liking.

By definition, RAW is much more about subtlety.  It’s the little things that give you the room for expression. RAW has a higher dynamic range as well as a higher bit-depth.  With RAW, you also get more control.  The normal in-camera processing takes away this control from you.  With RAW, I want to control the sharpening, the noise reduction, the auto-levels, and anything else that can make a different in my picture.

After all, you wouldn’t take an expensive sports car and put 87 octane gas in, would you?  RAW is the same way — if you’ve bothered to go through the steps to get the sports car, then you also assume the mechanic you take it to won’t just treat it like a regular car.

What I found with RAW Converters

After going through this with a number of users privately, I decided to look into a number of RAW converters — some free and some very expensive.  In many cases, I found some significant issues that contradict the entire reason we use RAW in the first place.

I will be discussing 5 different programs in this article, Program A, Program B, Program C, Program D, and Sagelight.  All of these are retail programs (some expensive and well-known), except for one which is a free but, surprisingly, scored better than the other A-D retail programs in the list.   For the record, I did not test Adobe products, mostly because many people who use other editors, including Sagelight, do so to find an alternative to Adobe or because Adobe Photoshop just costs too darn much.

The RAW portions of Sagelight were written with the old-school mentality of not doing anything more than is necessary to the image, and then giving you the control to back out of it or change whatever Sagelight is doing.

RAW images start in life as very dark pictures, and it’s necessary to do some operations — like Gamma curves and some levels adjustment — before you see the picture.  After that, with a RAW image, an editor should be doing nothing.  It’s your picture, and your options should not be limited by destructive behavior.

Cutoff Shadows

First, let’s look at the two pictures I was sent plus another editor that was referred to in the letter. The first from Program A:

Sample1-A

A little dark, but realistically, this is more typical of what a RAW image really looks like before it a gamma curve is applied — most RAW images look very dark natively.

Program B:
sample1-B

Now, that’s a pretty good looking picture. It has fairly nice contrast.  Maybe a little dark, but definitely better — from a visual perspective than Program A

Now Sagelight:

sample1-sl

Well, this doesn’t look nearly as nice as the picture from Program B, does it?  Nor should it!

Let’s see why.  Let’s take a look at the histograms:

sample2-histograms

Programs A& B have damaged this image in a significant manner.  On both the histograms for programs A & B, the black points have been cut off completely.  This can cause problems.  The low point data, especially when at 16-bits per-channel, is very valuable, and this data is now gone in the RAW conversions from Programs A & B.

Let’s look at what can be recovered here:

sample1-closeup-sl

In this image, since Sagelight kept this data, we can recover a lot of the shadow information to make a nice even picture.  This data is either lost or so highly compressed as to be essentially unrecoverable with programs A & B.

Highlights

I will get to this in the next example, but as you can see Sagelight has preserved the upper highlights.  It’s just a little red line, but sometimes this can make all of the difference in a picture.  Much of the time, since the histogram represents the entire picture, this data is fairly meaningless.

However, in other cases, this also represents a focal point — a specular reflection that will become completely flat.

Overall Picture

In this case, visually, the sample from Program B looked the best.  It may have damaged the picture for some editing, but since it looks better, how much will it take to get Sagelight to look as good or better?  Simple:  One slider movement in the RAW editing mode (the contrast slider):

sample1-sl2

After one slider movement, we have the same picture as Program A — but better.  Notice that the background, though dark, is still visible.  It was cutoff in both program A and B.  You can’t see it here, but the textured board of the background is completely visible in this image.  Also, we can still recover the dark areas near the scarf to create an image with a much broader dynamic range.

Blown Highlights

RAW pictures have much more dynamic range than you would think, especially after the treatment with some RAW conversion programs.  Here are two samples, one from Sagelight, and one from Program A, which happens to be a very well-known, expensive package:

The image to the right represents the blown highlights in this image based on the each color channel.  As you can see, its quite extensive.  In the orange banner to the left-center of the picture, for example – it looks fine, but is actually blown out in the reds.  if you wanted to change the texture of this area you would run into problems,

This was the impetus for writing this article.  This RAW conversion would normally make you think these areas were blown out in the RAW image when they are perfectly intact:

sample2-sl

Here, the Sagelight version — again looking less visually interesting for the moment — has absolutely no blown highlights, as shown in the image to the right.

Program B also had similar blown highlight problems, but not as many as Program A.

Let’s look at a side-by-side comparison:

sample2-sbs

As you can see, in the picture to the left, the brownish tiles above the doors and windows are completely blown out and unrecoverable — they are simply gone and blown out from the image data.  You can also see that the upper sky area, nice and blue in the Sagelight image, is also blown out and white.  Even areas that are still blue suffer from a range-compression issue making them basically unrecoverable.  The highlights on the girl’s shirt are also becoming blown out.

I leave it as an exercise that the Sagelight image can be made into the more visually impressive image — sans blown highlights on the wall and in the sky — with a simple Contrast Slider movement in Sagelight.

Highlights and Range Compression

Let’s look at the histograms for Programs A, B, and Sagelight for this image:

sample2a-histograms

With Program A, it’s clear that the highlights are blown in this histogram and that this data is unrecoverable.  With Program B, it’s also clear that some highlights are blown (but not as many as Program B).

With the Sagelight image, you can see two things:  The right edge of the histogram has a nice, natural slope downward showing that this image has a full range of highlights that don’t need to be cut off.

Also, you can see that the histogram is moved inward a little on both sides.  This is because there are some low-level shadows and highlights that are too minimal to be displayed in the histogram.

Sagelight is preventing two things here:

1. Cutting off Specular Reflections

By not cutting off the very top of the unseen — but, nevertheless, existing — highlights, Sagelight is preventing what could be a specular highlight from becoming flat.  This can be a reflection off of a window, sunglass, or somewhere else — one of the major elements that gives a picture depth, or, conversely, makes a picture seem flat.

2. Range Compression

Range Compression is not a subject that is generally discussed with image editing, but this is a problem that makes it difficult to deal with highlights that are still present in the image.  First, non-16-bit-per-channel editors have little chance here.  Second, this is where terms like “Highlight Recovery” come from — the idea of getting this compressed data uncompressed in the histogram.

It’s better to just avoid it in the first place, at least where possible.

Even though most of the highlights are present in the histogram of Picture B, it’s still very difficult to deal with the compressed highlights.  In many cases, it’s just not practically possible and where it is possible, it’s a lot of work — and it’s still hard (if not impossible) to get the same results you’d get if they weren’t compressed in the first place.

Coming in Part II

In part II, I show more destructive tendencies of some RAW conversion editors, which were even more surprising to me than what I’ve showed here in part I. I will show where some conversion programs smooth, sharpen, and otherwise modify your image in ways that are not only destructive, but once again contradict the reason you’re using RAW in the first place.

Conclusion

RAW editors have become somewhat sacrosanct in our minds, because RAW images are properly sacrosanct.  Until recently, I thought that was true (with the RAW editor part).  But, I found that many RAW converters — some very expensive and popular — aren’t treating your images they way you might think and, in many cases, are actually damaging your image and limiting what you can achieve.

Advertisements

13 thoughts on “Some Raw Truth about RAW Converters

  1. Is there a good reason for not mentioning the names of the other RAW converters in this comparison? You’re doing a good job of highlighting your product’s strengths, but without knowing what you’re comparing to, I won’t know how to reproduce these results, nor what programs remain to be compared with. I see no good reason to hide the names, these are factual comparisons, and only show your product favorably.

    Thanks,

    Oshyan

  2. Thanks! A great discussion.

    However, I believe many users actually still might want a very contrasty, vibrant, and sharpened default output (with some more headroom to adjust for overexposure) instead of an as-intact-as-possible but somewhat flat rendition. Good intentions aside this might require some guidance to new users.

  3. Hi, Luc.

    The reason I didn’t name the programs is because I am the author of an editor, and I really wanted to be as objective as possible. If Istarted naming names, it would look more like I am more writing articles like this because I have something to sell, and that I am just trying to denigrate the competition.

    Plus, this is just one issue. Many of these packages are fine packages and have lots of things to offer. I may not like what is going on with the RAW images, but I think it’s just professional courtesy there. There’s a lot of things you could pick on Sagelight not doing as well as it could, and it would just be awful if we all did that to each other.

    So, I guess to answer your question is that I’m trying to write things like that that stay as objective as possible, considering my position as the author of Sagelight, as well as showing some professional courtesy by not ‘flaming’ specific editors.

    Rob

  4. Thanks for a very well writen article which I enjoyed very much. If you are saying that all is needed in Sagelght is a little contrast to bring the RAW image to life, then I will stick with Sagelight. I have certainly learnt quite a lot from reading this and other articles in the blog.

    • Hi, Chris.

      Thanks. Yes, that’s basically the idea — just use the Contrast Control. The one in the RAW (i.e. Pro Quick Edit Mode) will give basically the same results, but without killing the highlights or the black points. I’ve also found that decreasing the gamma and then using the fill light (which is a different formula than the fill light in the Quick Edit Mode) is also a good combination.

      I’ll be revamping the RAW tools in just the next couple weeks, and I will be putting in more controls to control the black/white points and to see what’s going on (i.e. by adding the histogram in the RAW mode).

      In part II of the document, I’ll be discussing another difference. A lot of RAW converters also smooth and then sharpen the image. I really don’t like the idea of sharpening an image until later in the process, because this can add problems when you want to add color or do more than just basic or minimal curves.

      But, that’s another difference you might see — some images from other RAW converters may look sharper than the Sagelight version, because Sagelight doesn’t pre-sharpen the image for you. I leave that for the Unsharpen Mask, where you have more control.

      Rob

  5. I did my own comparisons of the highlight recovery tools in several RAW converters a while ago with a challenging test picture from a German blog: “Ansichten eines Sturkopfs” (http://www.die-ritters.de/blog/bibble-5-0-angetestet-und-verglichen.sturkopf). I have added Sagelight to this comparison and the highlight recovery performs indeed as good as indicated in the blog post discussion above (one has to use the “>> File >> Open RAW file with Highlight Recovery Options” menu). In my eyes Sagelight is one of two converters with superior highlight recovery (together with Scarab Darkroom). Please note that for this test I did focus my efforts only on recovering the best highlights. The images are rendered quite differently in all other aspects by most converters.
    The test images can be downloaded in full size from here.
    http://picasaweb.google.com/luc0815/RAWConverterTestHighlightRecovery#
    The Sagelight example is the last one of eight conversions.

  6. Fantastic article Rob.

    I’m curious as to why the other converters show a high dark peak in each channel at the dark end, in the first pic ??? One reaches 100%, while the equivalent peak in Sagelight is quite low.

    I agree with Oshyan. I would be useful to name the other editors, as does every other comparison on the net. Saying “product A” makes it sound like a fake soap commercial.

    I wonder if the poor conversion is why a lot of the multi-image HDR has grown ? One would expect a need for multiple EVs with camera jpgs, but if the dynamic range of RAW’s is limited by the converter, it generates an HDR requirement in pics with a high dynamic range.

    • Hi, adb.

      Thanks. The left peaks are two things happening. I am not sure which peak you’re referring to — the left peak or the v-shape just to the right of the edge that also turns into a peak.

      In the first case, this is the black points getting cut off and information being lost. In the other case (the v-shaped or right-peak) is the dynamic range being compressed, making it difficult to deal with the shadows.

      You can see the effect by looking at the large concentrated area — it stays the same in basically all pictures (though it moves around), but the information to the left of it gets very compressed (and cut off) in the non-Sagelight pictures. This is why the typical Sagelight RAW file can look a little washed out at first, because it’s saving this information. An example of that area is in the closeup where I brought out the tie area on the bear.

      Let me know if that’s what you meant, or if I got your meaning all wrong. 🙂

      Soap Commercial. With the Product A, B, etc… I know it does sound like a soap commercial, but as someone who sells an editor, I don’t want to name names for many reasons. One, it’s just professional courtesy. Two, I have nothing against these editors, and since there are so many editors out there, it would be unfair to just single out two or three specific editors because these happen to be the ones I tested on. I’m glad to see people have verified the issue, as well. Also, I really didn’t want to make it about Sagelight = better specifically, as much as I wanted to make it a comentary on RAW processing that surprised the heck out of me. But, since I am the author of Sagelight, I had to use it as the example, as this is an area where I concentrated specific RAW functionality that I wanted to point out (since I was there).

      With HDR: You really hit an area where I have been thinking the same thing. I didn’t want to be controversial with the article, but when I realized that RAW is more capable than some processors would allow (at least easily), I started thinking the same thing with regard to HDR. I am a believer in using multiple exposures, and will have that available in Sagelight soon (at least in some minimal form), but, from what I’m seeing with the range of RAW images, especially with newer cameras, as well as more advanced techniques (such as anisotropic or NL means noise reduction), I think you can do a lot more with RAW in the HDR area where nornmally multiple exposures might come up in the thought process.

      Rob

  7. Pingback: Why I didn’t name Editors in the RAW Editor Article « Sagelight Image Editor Blog & Newsletter

  8. Funny, that particular image was provided to me by the purveyors of my favorite processor … or maybe it’s merely a standard licensed test image.

    Nice rendering of it by Sagelight. The reference about native “raw” image-files (excepting the file-format conversion case of the notorious Adobe DNG Converter) being “irreparably damaged” by being presented as non-modified input image-files to “raw” processors seems only appropriate in reference to what a given “raw” processor can be adjusted to do with the tone-levels of the image-information as they relate to the black-level of minimum luminance.

    It certainly is true that it is helpful when the camera exposure levels have been “exposed to the right” to the extent where the “raw” image information itself emerges from the noise into the lower tone-levels.

    It is possible (but not optimum in terms of the net image signal/noise ration) to “slide” that “shadow” information back up into the (linear) histogram view (using a processor’s “Exposure Comp/Bias”).

    However, a 16-bit logarithmic histogram display (such as displayed with using Histogrammar 1.2) typically reveal a great deal of detail that the (nearly universally 8-bit only) linear histogram displays of “raw” processors fail to reveal to the user.

    Adjusting the white-point (in my processor) merely expands the RGB color channels in a geometric manner from zero value (without shifting or raising the black-point).

    Additionally, there are many instances in my own processing of wide dynamic range shots (such as landscapes) where somewhat decreasing magnitudes of tone-levels near zero are not inappropriate. (Granted), one wants the ability to control thee things!

    With a higher dynamic-range image sensor, it may not overly degrade the composite image dynamic-range to adjust decreasing magnitudes of tone-levels towards zero. A tad of this approach can supplement the sparing use of Noise Reduction algorithms, and, as well, add a nice “downward-expansion” tail at tone-levels just below where Gamma correction curves will begin to increase the noise (as well as the signal) gain of the tone-curve transfer-function.

    By inspection of the 16-bit TIF output of my favorite “raw” processor (using the logarithmic histogram display of Histogrammar 1.2), I can see that a great measure of the detail-information in the processed images occurs as contrasts between color-channel elements at mid and upper tone-levels.

    Don’t get me wrong – I am a pixel-peeping image-quality maniac! Due to all factors involved, the ability of the optical lens system, the image-sensor, and the configuration of the “raw” processor all matter in the ability of a image-processing chain to resolve what the human eye is better at – perceiving small contrasts at low tone-levels with acuity.

    Indeed, little comes easy between photo-sensor wells filling with electrons and the composite noise-floor raging below … and Nature exacts an invariable price for the non-linear machinations with which we massage the intact “raw” data with the “smoke and mirrors” of finding beauty within the bytes … 🙂

  9. Rather than effects that can (depending on the individual color-channels signal levels existing in the camera’s source “raw”) be mitigated or eliminated entirely by user-adjustments of the processing parameters of (at least some, including my favorite) “raw” processor(s) …

    … this issue (linked below) *does* in fact constitute “irreparable damage” – not to the unmodified “raw” source image-file itself, but to the (in plenty of cases) important EXIF data retained in (at least, TIF) image files emerging out of processing by Sagelight itself:

    http://www.sagelighteditor.com/discussion/viewtopic.php?f=9&t=21199

    It would be great to see this bit of local housekeeping addressed!

  10. Pingback: RAW Converters as of July, 2010

  11. Pingback: New to GF1 -workflow question - Page 2 - Micro Four Thirds User Forum

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s