MK IV Dark images, hot pixels and cosmic rays
Andrew Bennett:
andrew.bennett@ns.sympatico.ca
Revision 4: 2000 Jan 4: Dark subtraction
Revision 3: 1999 Dec 23/25: Hot pixels
Revision 2: 1999 Nov 21:Cosmic Rays, Hot Pixels
Revision 1: 1999 Jan 21 follows below: I have
not changed it. The statistics of hot pixels found more
rrecently contradict the conclusion in
Revision 1 that the noise distribution is gaussian. I've left the
old stuff in so you can have a good laugh ...
Analysis of CD5 data.
1999 Dec 23 analysis of hot pixels
Elsewhere I show my tentative identification of
edge pixels for CD5. I have
selected column 2040 (counting from zero) as a "Dark"
estimate and column 2 as an "Unused" reference level. Below,
1999 Nov 21, I describe how
I selected "Hot" pixels on the
dark images. I have used this method, slightly cleaned up,
to make ASCII files of the locations of all the detected
"Hot" pixels.
Added Dec 25:
That's one file for V,
Vindex.txt, and one for I,
Iindex.txt. In these files, the coordinates
X1 and X2 are for the image area and run [0..2031,0..2031].
The amplitudes for the three dark images are in units of the
estimated standard deviation for the particular image. This
is estimated from the probable error. The lower limit
is a little different from that described below.
End of addition.
I have
written code to extract the pixel values at these locations
in all the images and get the result into a spreadsheet. The
pixel value logged is relative to the mean of 4 surrounding
pixels.
Values found for these "Hot" pixels are remarkably
consistent and not visibly affected by noise sources such
as stars. The correlation coefficient between values from
two images ranges from 0.85 for two dark images up to
0.998 for two adjacent I images having rather high values
for their "Hot" pixels.
The mean value of the "Hot" pixels varies considerably
from image to image. I have plotted it against the measure
of dark current obtained from the edge pixels, median of
column 2040 - median of column 2. First for V:
I think the correlation is remarkably good! Now for the
I images:
Ha! One low point! That's H4R1442.889; the sky background is
so high that the "Hot" pixels are saturating.
I conclude:
1) The edge pixels provide a good estimate of the "Hot" pixel
values.
2) "Hot" pixel activity varies by a large factor from image to
image. This is presumably a result of temperature variation. One
would expect a large improvement if lower temperatures could be
achieved without returning to the ice crystal problem. As it is,
correction should be possible using a scaled Dark image, as I
hinted in an earlier post to the mailgroup.
3) The "Hot" pixels are consistent. I hypothesize that the
intermittent ones seen on the dark images when creating the
index (reported below) are mostly the result of an odd
phenomenon I read about in a book on CCD's. Yes - I do
sometimes read books. This is the phenomenon of hot pixels
turning on rather abruptly as the temperature is raised. And
conversely, turning off rather abruptly as the temperature is
lowered. If one assumes, as the data suggests, that the dark
images were taken at a lower temperature than most of the star
images, some "Hot" pixels would be inactive. This nicely explains
the lower correlation between dark images compared to that between
star images. This also suggests that the improvement on cooling
could be rather great.
Dark Subtraction
I have shown above that the edge pixels
imply that temperature varied quite a lot during the CD5
measurements and that improved dark correction ought to be
possible using a scaled dark image. I have now got round to
trying this out on file H4R1446.853. This was selected as being
representative of those images where the hot pixel activity was
greater than it was for the dark images.I subtracted a scaled
version of my median dark image and ran a strip from the resulting
image (flattened and stripped of "Bright" sources as usual) through
the MK III version of "Star" with the following results.
Factor Noise, adu No. of stars > 20 s.d.
1.0 116 328
1.25 107 355
1.5 102 362
1.75 105 348
2.0 116 309
4.0 260 85
So there is some improvement to be had. Now: could
sombody remind me how to arrive at the optimum factor (1.5 in
this case) without having to run the entire analysis six
times? My programs won't run with Windows and "Star" won't
run without so I'm doing a lot of booting.
1.5 is pretty close to square root(average "hot" pixel in image /
average "hot" pixel in all three dark images) = 1.55. Um! Needs
a bit of hand-waving to justify using average instead of median
for the dark images, since the one I use is the median of the three,
not the average. I think I can hand-wave my way into the
square root ...
1999 Nov 21 analysis of cosmic rays and hot pixels
Image Median iqr s.d. CR Total Hot Intermittent Hot
VDARK15 9531 33 24 )
VDARK16 9538 36 27 ) 735 943 11
VDARK17 9584 75 56 )
IDARK15 9790 51 38 )
IDARK16 9814 66 49 ) 347 922 20
IDARK17 9990 197 146 )
Median is calculated from a grid of about 1600 points.
iqr is the interquartile range = 2 x probable error (p.e.).
s.d. is standard deviation calculated from p.e.
CR (Cosmic Ray) is > 13.5 s.d (20 p.e. or 10 iqr) on one image.
Hot Pixel is > 6.7 s.d. (10 p.e) on all images and > 13.5 s.d
on at least one.
Intermittent Hot Pixel is > 13.5 s.d. (20 p.e.) on only two images
or a Hot Pixel with Max/(Min+10p.e.) > 2.
Distribution of Hot Pixels.
The X's mark the intermittent hot pixels. Some are rather
bright!
Distribution of Cosmic Rays: total for 3 images.
Old Stuff
Revision 1: 1999 Jan 21
The Good News: Noise
After eliminating obvious cosmic rays (I am using the
term 'Cosmic Rays' as an all-embracing term of abuse) the
dark images dark, 11ba, 11bb, 1b-26a, d142355, d142359,
d150004 etc, appear more or less indistinguishable from gaussian
noise. There is, as Tom pointed out, a bit of a ramp
at the start of each image but this is small and appears
consistent from image to image and should be completely
eliminated by any sort of dark subtraction so there is no
need to fix it.
After much manipulation, I have come up with various estimates
of the electronic noise ... the best ones agree pretty well with
Tom's. Damn! All that effort and not even anything to argue about.
Bennett: 12.2 electrons.
Tom: 12.3 electrons.
See Chapter and Verse below
With the eye of faith, optimum contrast and strategic use
of peripheral vision, I have almost convinced myself there is
something consistent with a period of 4 pixels. I propose to
spend the next month getting my FFT routines to run on image
data so I can really get nit-picky. Whatever it is, it is at
such a low level that it will have no effect in the real
world. Congratulations, Tom! I think you got things grounded!
The Mixed News: Gain
The other 2 images (11fa, 11fb) from which Tom calculated
the gain are a different story. The following clip is from 11fa
and has been put through a logarithmic compression to bring up
the contrast in the noise. OK - I could have done a better job.
Next time.
Yes - those things are the dreaded frost crystals. I felt sure
when I calculated the rms difference between the full images and got an
answer that agreed with Tom's that we must both be wrong! But
painstaking avoidance of all the icefloes gave exactly the same
answer and the final removal of one last cosmic ray reduced the
rms by half a percent giving a final reciprocal gain estimate of:
Bennett: 3.0 electrons/e- (actually 2.96 but I rounded it up)
Tom: 2.9
There! Hard work pays off.
The statistics are not as straightforward for these bright images;
histograms show strong odd/even bias or in another case a pattern
with a period of 4 adu. Not to be confused with the spatial period
of 4 pixel mentioned above. There is also more than a hint of a
spatial pattern at 16 pixels at right angles to the 4 pixel one.
Lots of scope for future nit-picking. On the other hand, the pixels
appear pretty uniform. One interpretation of the statistics is that
pixel sensitivity is random with an rms variation of around 1% which
ain't too bad. We are going to need good flats with less ice.
The Bad News: Cosmic Rays
The bad new is that the cosmic rays are mostly not single
pixels. In the main, they look just like well focussed stars
or like galaxies or close pairs. This means that it is going
to be very difficult to get rid of them without losing several
classes of real objects that we are supposed to be looking for.
Ouch!
Part of 11ba. Ok - this doesn't come out too well. I got some
improvement by jiggling the brightness and contrast on my monitor.
Take the image away and look at it with some real software.
The images started out as TIF and had to be JPEG'd to get them
on the Net. Minimum compression has left them bearing a slight
resemblance to the original. If you want the real thing, I will
e-mail it.
The same part of 11bb.
The difference between the two previous images. Mid grey means
no difference. The white dots are cosmic rays from image 11ba and
the black ones are from image 11bb. Even on this scale, I can
tell on my display that at least the black ones are blobs
rather than dots. I used bidirectional logarithmic compression
to bring up the noise.
Electronic Noise Estimation.
Files d142359 and d150004
Selected regions, no large cosmic rays
Image Mean var sd
d142359 62.0 7.9
d150004 42.0 6.5
Difference 84.6 9.2
If we now scale the variance of the difference in the
ratio of the separate variances (a hairy process!):
Scaled var sd
d142359 50.4 7.1
d150004 34.1 5.8
Now extrapolate to zero dark current assuming d142359
got two units and d150004 got 1 unit (OK - it should be
100:51 with a readout time of 50 seconds but I'm too lazy to
go back to the spreadsheet and fix the factor)
(Method 1 below)
var sd
Extrapolated 17.8 4.2 = 12.2 electrons taking gain as 2.9 adu/e-
Compare Tom's estimate of 12.3 using files 11ba and 11bb
Wow! The same answer from a different data set. Too good to
be true.
Alternatively, (Method 2 below)
using gain = 2.9
Mean variance from Difference 42.3
Estimated dark current noise 17.4
Electronic var 24.9 sd 5.0 = 14.4 electrons
Which is not very different.
Dark current noise variance is estimated as 1.5 ((1+2)/2) times
change in mean (adu) times gain (adu/electron - the reciprocal
of the "gain" Tom used).
Files 11ba, 11bb, 11fa and 11ba, as used by Tom
Using the whole images, cosmic rays on both 11ba and 11bb
plus icicles all over 11fa and 11fb
Average var sd Average sd
11ba -24650.7 39.4 6.3 11fa -3066.8 478
11bb -24637.1 27.5 5.2 11fb -3006.7 482
Diff. -13.6 65.7 8.1 Diff. -60.1 121
R 0.0175 R 0.9682
The low value of the correlation, R, for 11ba and11bb is
excellent news - there isn't much there except uncorrelated
noise. The high value for 11fa and 11fb says that in this
case most of the junk is fixed: the icicles didn't move much
between the exposures.
Estimating the dark current noise as above - assuming the
exposures are the same:
Mean variance from Diff. 32.8
Estimated dark current noise 7.1
Electronic var 25.8 sd 5.1 = 14.7 electrons
Using selected bits avoiding most cosmic rays
Average var sd Average sd
11ba -24650.8 23.1 4.8 11fa -3014.3 238
11bb -24637.1 29.7 5.5 11fb -2952.3 239
Diff. 51.7 7.2 121
R 0.0197 R 0.8707
The correlation between 11fa and 11fb is still high. Most of
the signal is variation in pixel sensitivity.
For dark files, 11ba and 11bb:
Mean variance from Diff. 25.9 adu^2
Estimated dark current noise 7.1
Electronic var 18.8 sd 4.3 = 12.6 electrons
So getting clear of (most of) the cosmic rays brings the
answer back pretty close to Tom's.
The major difference in going to the selected, cleaner, area
is a big reduction in the variance of the individual flat files.
The variance of the difference is essentially unchanged - a
coincidence. Modelling the total noise as the sum of random
variation in pixel sensitivity plus the root n electron noise
plus the electronic noise we can estimate:
For flat files, 11fa and 11fb:
Total variation 238 digits
Pixel variation 222 digits = 1.03% rms
Noise part 86 digits
Estimating (reciprocal) gain as change in average value between
the dark files and the illuminated ones divided by change in
variance gives 2.96 e-/adu which rounds up to 3.0. Hooray!
A different answer from Tom's 2.9.
Theory
The noise is modelled as a random superposition of electronic
noise plus the random fluctuations in the dark and photoelectric
charge. If the total charge is on average n electrons, the variance
is taken as n electrons and the standard deviation as root(n).
Define
d dark electrons in unit time: 2d in twice the time
f photoelectrons in unit time for the flat image
e electronic noise
A Gain, digits per electron
V0 Digits out for zero electrons
Then
Case Mean digits Variance digits^2
Dark d V0+Ad d+e A^2(d+e)
Dark x2 2d V0+2Ad 2d+e A^2(2d+e)
Flat d+f V0+A(d+f) d+f+e A^2(d+f+e)
So from the digitized output one can estimate
A = (var(Flat) - var(Dark))/((mean(Flat) - mean(Dark))
e = (2var(Dark) - var(Dark x2))/A^2
which is the first method used above. Since the variances
are not that well determined, especially with the odd cosmic
ray, this is not the best method.
Alternatively one can proceed as follows giving the second
method used above.
Ad = mean(Dark x2) - mean(Dark)
A^2(1.5d+e) = (var(Dark) + var(Dark x2))/2
e = {(var(Dark) + var(Dark x2))/2
- 1.5A(mean(Dark x2) - mean(Dark))}/A^2
Bennett TASS home page
The Bennetts' Home Page