Should we let our DSLRs do dark-frame subtraction in the camera?

I threw away the boring entry that was here and wrote a much more important one — at least important for my fellow astrophotography enthusiasts. This is probably the last I'll write until around Independence Day because I'm getting busy with students finishing theses and other urgent pursuits. Happy end-of-month!

The following is about a subtle mathematical point that Sander Pool raised on the DeepSkyStacker mailing list. At first I didn't understand him; as I said, it's subtle. I thank him for taking the time to clarify it. Read on...

Whenever a DSLR takes an exposure longer than about 1 second, it's desirable to subtract a matching dark frame from it. That is a matching exposure taken with the lens cap on and/or the shutter closed. The reason is that all of the sensor pixels have a small amount of electrical leakage that makes them give nonzero readings even when there is no light. Some of them have quite a bit of leakage, and that's why a long-exposure picture often seems to be peppered with brightly-colored specks.

Most DSLRs will take and subtract a dark frame for you automatically. They call it long-exposure noise reduction and it's a menu setting. Take your 5-minute exposure, and then the shutter closes and the camera takes another 5-minute exposure, performs the subtraction, and stores the result in the raw and JPEG files.

Astrophotographers often do it a different way. We do batch dark-frame subtraction, where we take a series of exposures (usually many of the same celestial object), then a matching set of dark frames, and we do the subtraction with software. The software first combines all the dark frames into one averaged one, then applies it to all the images.

This has two obvious practical advantages. You can save time by taking the dark frames in a batch, at the end of the session, rather than in between exposures. My camera often finishes up the dark frames, under automatic control, with the lens cap on, while I'm taking down the other equipment or even during the ride home. And the other advantage is that if the software is smart enough, and one of the dark frames happens to have been hit by a cosmic ray (causing a non-reproducible bright spot), it can recognize the bright spot as deviant and throw it away automatically.

But the question was: Apart from those two points, is there any advantage to doing things one way or the other? Which is best?

The obvious appeal of in-camera dark-frame subtraction is that it's automatic, and you can be sure the dark frame is a good match to the image. It is taken immediately afterward, with the sensor at the same temperature and all the settings the same.

With that in mind, I asserted that it makes no difference which method you use, as long as the numbers of frames are the same and (in the batch method) the dark frames are combined with simple averaging.

Not exactly. Here's the subtle part.

Usually, we taken multiple exposures of the same celestial object and then align and stack the images. That is, we allow for variations in telescope tracking by having the software align the images before combining them.

Consider now a situation where there are 3 image frames and 3 dark frames, and consider the pixel at location (x,y) in the finished picture.

If alignment has been done, pixel (x,y) in the finished picture corresponds to three different pixel positions in the original images. Call them (x₁,y₁), (x₂,y₂), and (x₃,y₃).

Now then. If the dark-frame subtraction was done in the camera, then each of these image pixels, (x₁,y₁), (x₂,y₂), and (x₃,y₃), was corrected against the corresponding position in one dark frame.

But if the dark-frame subtraction was done as a batch, then each of them was corrected against the average of all the dark frames in that particular pixel position. And that should be a much more accurate estimate of the actual leakage value at that point in the image, because it smooths out random fluctuation.

So the batch method wins — but only if you are aligning images before stacking them. If the images are already perfectly aligned, so that each pixel in the finished picture corresponds to the same pixel position in all of the input frames, then the techniques are equivalent, because averages are averages.

I think the practical difference will be small and will be most evident when the number of dark frames is moderately small (like 3 to 6). Maybe someone will do an experiment and give us some actual results.

Permanent link to this entry

Poetic meter

I read "Locksley Hall" last evening.
Now its rhythm haunts my thought.
All my words come out trochaic,
All this pesky poem's fault!

Tennyson's "Locksley Hall" is a long poem about several subjects, and I think parts of it are better than others. It is well known to amateur astronomers and to Star Trek fans because of small parts they have seen quoted.

Permanent link to this entry

Miscellany

There seems to be another Turing legend going around, which is that he created computer programming. Not exactly. We owe a great deal to Charles Babbage and Konrad Zuse. But Turing's mathematical model of computability (the Turing machine) and his paper on whether machines can have intelligence are still vital parts of the curriculum.

(I remember coming across some students once who thought a "touring machine" was a robot that explores its environment. Not at all the same thing...)

Meanwhile, my disdain for "hipster photography" has made it into the campus newspaper. (The interviewer came and sought me out.) I think something was lost from the article during the editing process, because I'm not talking about sharing photos, I'm talking about filtering them to introduce artificial defects.

My technological discovery today: All latex interior paint today, including "flat," is glossier than the paint of 10 or 20 years ago. It also requires less of a primer or undercoat, and it's more durable. But if you touch up an old paint job with new paint that matches the color perfectly, there will still be a difference in sheen. (Reportedly, this is not only due to changes in paint technology, but also the fact that paint loses glossiness as it ages.)

We are in full-on summer frenzy mode, with everybody rushing around wanting to do things, and I may not be writing every day for a while.

Permanent link to this entry

Alan Turing — legend or reality?

Today is the 100th birthday of Alan Turing, the man who laid the mathematical foundations for computer science as we know it, as well as inventing cryptographic techniques that helped our side win World War II.

Turing died young, in 1954. Most people believe he killed himself by eating an apple poisoned with cyanide, despondent because of the way he had been persecuted and mistreated because of his homosexuality.

That is widely said to be the reason why the trademark of Apple Computer is a partly-eaten apple.

Now a new investigation is calling the suicide verdict seriously into question. The death looks more like an accident, though even murder cannot be ruled out. (Think about it — the Cold War was raging and someone on the Russian side may well have wanted to deprive NATO of a cryptographic genius.)

Fact: The apple was never actually tested for cyanide.

Fact: There was cyanide nearby, from an electroplating experiment that Turing was doing in his home lab, and there was a strong smell of cyanide in the air. A chemical-handling accident seems likely.

Fact: Nobody knows whether the cyanide entered Turing's body by ingestion, inhalation, or other means.

Fact: There was no evidence of a suicidal state of mind. Yes, Turing was treated wretchedly — but he seems to have held up well, perhaps expecting to be vindicated in future years.

So the theory that anti-gay psychologists hounded Turing to death is actually hard to sustain.

As often happens, the version of the event that makes the best story is not the version that best accords with the facts.

Permanent link to this entry

There's a hole in the library...

The lighting was very harsh, but here you see workmen cutting one of two large holes in the western wall of the ground (2nd) floor of the UGA Science Library. There are going to be more windows.

Permanent link to this entry

Microsoft Installer fails with error 2203
Internal error 2203 when installing Java
Error 2203 on C:\Windows\Install

This morning (June 20) I couldn't install Cisco AnyConnect on my 64-bit Windows machine. The installer kept crashing with an error code of 2203 and a reference to a nonexistent file in C:\Windows\Install.

Here's how I fixed it.

Summary: Username SYSTEM (the installer) needs Full Control of C:\Windows\Install and of the Temp folder in your local application data folder.

Details:

(1) Click on the start button and, in the search space at the bottom of the menu, type the mysterious incantation:

%temp%\..

This will open up your local application data folder, which will look something like this:

(2) In that folder, find a folder called temp. Right-click on it and choose Security.

If SYSTEM is already in the list of names, and, when you choose SYSTEM, Full Control is checked, you don't have the problem that I had. In my case, SYSTEM was not on the list of names. I had to click "Edit" to add SYSTEM and give it Full Control, and that fixed the problem.

It's also a good idea to delete the files and folders inside the temp folder, since they are no longer needed and may have the wrong permissions.

You should also check the folder C:\Windows\Install. SYSTEM should have Full Control of that one also.

Permanent link to this entry

Out with the old, in with the new!

The red shag carpet of 1974 is hardly appropriate for the new century, so we replaced it, finally! Both of our dogs are puzzled.

I should add that in the second picture, the carpet isn't completely installed yet! Of course it came out as flat and neat as any wall-to-wall carpet. Melody is thrilled.

Permanent link to this entry

How to get good service from a bank

The news media are full of stories about how terrible banks are. Reports of bad customer service are everywhere. Yet I continue to get consistently good service from my bank (SunTrust, in Athens, Georgia). What's my secret?

I have the crazy idea that if I want the bank to act human, I have to act human.

That is, I'm not just a number, an online connection, or an invisible face in the drive-through. Every couple of weeks, I go inside the bank branch to make a deposit, and I say hello to a few of the staff. Because it's a big bank that switches people around, I never know quite all of them, but I figure it's in my best interest for several of them to know me by sight.

Maybe this is one of my mid-20th-century quirks — something out of the 1950s — but it works. Over the years, not only have they helped me out of awkward situations (such as a mistyped online transaction — our fault — that was way too big), they've spontaneously come up with a lot of good things, including a mortgage refinancing deal that I didn't ask for and didn't even know was available.

Am I wasting their time? I don't think so. I suppose they might save a tiny bit of time if I only turned up online and didn't walk in and say hello. But the point is, if I don't want them to be impersonal to me, I shouldn't be impersonal to them.

Permanent link to this entry

An academic mistake

Many of the people running American universities seem to assume that if more scientists apply for federal research grants, there will be more grants.

And that is not true. There is no reason for the supply of grants to rise with the demand. It's not like saying that if more people buy cars, more cars will be manufactured.

Even if the federal budget were in good shape, research grants wouldn't expand that way. They might expand modestly, but the government is under no obligation at all to give more grants just because more people are applying for them! As it is, the supply is going to shrink because of necessary budget cuts.

Another reason I'm glad not to be dependent on federal grants as so many scientists are.

Permanent link to this entry

Thought for the day

Which do you think will impress me more, your knowledge of detailed and obscure facts, or your ability to make worthwhile things happen?

The astute reader will note that I've been busy doing a lot of things other than blogging. I hope to get back to the Daily Notebook soon.

Permanent link to this entry

Rutherford Hall demolition

One of the more disconcerting sights at the University today was this. Rutherford Hall, a women's dormitory built in the 1930s, is being torn down and replaced by a larger building built in the same style, reportedly to be given the same name.

I have no particular personal ties to Rutherford; I suppose some of my friends in my student days must have lived there, but I don't know which ones; in fact I've never set foot in the building. And now it's clear that I'm never going to.

[Addendum:] Even when pulverized and perceived just as dust in the air, Rutherford Hall still smells like an old dorm!

Permanent link to this entry

Odd thing of the day

Take a good look at this can of spray paint and guess what color of paint is in it.

Gold, maybe? That's what I thought, until I sprayed some of it. Then I took a closer look...

The work of a prankster? No; a whole shipment of silver spray paint reached the local Home Depot with gold-colored caps. They offered to let me return it, but since I'll eventually use the silvery paint too, I decided to keep it.

Permanent link to this entry

Sun, Venus, and clouds

In Athens, clouds almost blocked our view of the transit of Venus. Almost. The cloud cover had frequent narrow breaks — the cracks in a mackerel sky — and I was able to take a number of pictures like this one, showing the sun partly obscured by clouds, with Venus plainly visible.

We also looked at the sun through a #12 welder's glass (originally purchased for the 1970 eclipse) and clearly saw Venus as a dot, not a speck.

There you have it — the last transit of Venus in our lifetimes.

This was taken with a Celestron 5 and full-aperture solar filter, set up near Hardman Hall at the University of Georgia. We had planned on Myers Quad, but there was too much dust in the air from the demolition of Rutherford Hall taking place at the time.

Permanent link to this entry

The transit of Venus

In case you haven't heard it anywhere else, tomorrow evening (Tuesday, June 5), Venus will pass between us and the sun. As seen from Georgia, the transit will start just after 6 p.m. and will end after sunset (which means we won't see all of it).

For full coverage see astronomy sites such as this one as well as Wikipedia. In brief:

• It's rare because the orbits of the earth and of Venus are not in the same plane; thus Venus usually passes north or south of the sun rather than right in front of it. The earth and Venus both have to be in the right parts of their orbits, and as it works out, we get a couple of Venus transits and then a century-long hiatus. This transit is the second one of a pair. The next one is in 2117.
• To see it, you need the same precautions as for viewing a solar eclipse, except that the transit of Venus is much harder to see. You can probably see it without magnification (through a safe solar viewer), but it won't be prominent — just a dot on the sun. You probably can't see it by pinhole projection (not sharp enough), but you might want to try.
• Transits of Mercury and Venus are important because, by timing them, astronomers can measure positions precisely. This is how we check our knowledge of Venus's orbit and of our own.

Melody and I plan to photograph it from Myers Quadrangle on the University of Georgia campus. But we can't let you look through our telescope — it will be fully occupied with photography.

Permanent link to this entry

Will your American electrical appliances work in other countries?

See this. It's something I had been meaning to write for 20 years.

Permanent link to this entry