TN 0049: Mark IV Dark images and cosmic rays
Andrew Bennett:
andrew.bennett@ns.sympatico.ca
Jan 21, 1999
Revision: #1 990121
Key Words: CCD electronics
The Good News: Noise
After eliminating obvious cosmic rays (I am using the
term 'Cosmic Rays' as an all-embracing term of abuse) the
dark images dark, 11ba, 11bb, 1b-26a, d142355, d142359,
d150004 etc, appear more or less indistinguishable from gaussian
noise. There is, as Tom pointed out, a bit of a ramp
at the start of each image but this is small and appears
consistent from image to image and should be completely
eliminated by any sort of dark subtraction so there is no
need to fix it.
After much manipulation, I have come up with various estimates
of the electronic noise ... the best ones agree pretty well with
Tom's. Damn! All that effort and not even anything to argue about.
Bennett: 12.2 electrons.
Tom: 12.3 electrons.
See Chapter and Verse below
With the eye of faith, optimum contrast and strategic use
of peripheral vision, I have almost convinced myself there is
something consistent with a period of 4 pixels. I propose to
spend the next month getting my FFT routines to run on image
data so I can really get nit-picky. Whatever it is, it is at
such a low level that it will have no effect in the real
world. Congratulations, Tom! I think you got things grounded!
The Mixed News: Gain
The other 2 images (11fa, 11fb) from which Tom calculated
the gain are a different story. The following clip is from 11fa
and has been put through a logarithmic compression to bring up
the contrast in the noise. OK - I could have done a better job.
Next time.
Yes - those things are the dreaded frost crystals. I felt sure
when I calculated the rms difference between the full images and got an
answer that agreed with Tom's that we must both be wrong! But
painstaking avoidance of all the icefloes gave exactly the same
answer and the final removal of one last cosmic ray reduced the
rms by half a percent giving a final reciprocal gain estimate of:
Bennett: 3.0 electrons/e- (actually 2.96 but I rounded it up)
Tom: 2.9
There! Hard work pays off.
The statistics are not as straightforward for these bright images;
histograms show strong odd/even bias or in another case a pattern
with a period of 4 adu. Not to be confused with the spatial period
of 4 pixel mentioned above. There is also more than a hint of a
spatial pattern at 16 pixels at right angles to the 4 pixel one.
Lots of scope for future nit-picking. On the other hand, the pixels
appear pretty uniform. One interpretation of the statistics is that
pixel sensitivity is random with an rms variation of around 1% which
ain't too bad. We are going to need good flats with less ice.
The Bad News: Cosmic Rays
The bad new is that the cosmic rays are mostly not single
pixels. In the main, they look just like well focussed stars
or like galaxies or close pairs. This means that it is going
to be very difficult to get rid of them without losing several
classes of real objects that we are supposed to be looking for.
Ouch!
Part of 11ba. Ok - this doesn't come out too well. I got some
improvement by jiggling the brightness and contrast on my monitor.
Take the image away and look at it with some real software.
The images started out as TIF and had to be JPEG'd to get them
on the Net. Minimum compression has left them bearing a slight
resemblance to the original. If you want the real thing, I will
e-mail it.
The same part of 11bb.
The difference between the two previous images. Mid grey means
no difference. The white dots are cosmic rays from image 11ba and
the black ones are from image 11bb. Even on this scale, I can
tell on my display that at least the black ones are blobs
rather than dots. I used bidirectional logarithmic compression
to bring up the noise.
Electronic Noise Estimation.
Files d142359 and d150004
Selected regions, no large cosmic rays
Image Mean var sd
d142359 62.0 7.9
d150004 42.0 6.5
Difference 84.6 9.2
If we now scale the variance of the difference in the
ratio of the separate variances (a hairy process!):
Scaled var sd
d142359 50.4 7.1
d150004 34.1 5.8
Now extrapolate to zero dark current assuming d142359
got two units and d150004 got 1 unit (OK - it should be
100:51 with a readout time of 50 seconds but I'm too lazy to
go back to the spreadsheet and fix the factor)
(Method 1 below)
var sd
Extrapolated 17.8 4.2 = 12.2 electrons taking gain as 2.9 adu/e-
Compare Tom's estimate of 12.3 using files 11ba and 11bb
Wow! The same answer from a different data set. Too good to
be true.
Alternatively, (Method 2 below)
using gain = 2.9
Mean variance from Difference 42.3
Estimated dark current noise 17.4
Electronic var 24.9 sd 5.0 = 14.4 electrons
Which is not very different.
Dark current noise variance is estimated as 1.5 ((1+2)/2) times
change in mean (adu) times gain (adu/electron - the reciprocal
of the "gain" Tom used).
Files 11ba, 11bb, 11fa and 11ba, as used by Tom
Using the whole images, cosmic rays on both 11ba and 11bb
plus icicles all over 11fa and 11fb
Average var sd Average sd
11ba -24650.7 39.4 6.3 11fa -3066.8 478
11bb -24637.1 27.5 5.2 11fb -3006.7 482
Diff. -13.6 65.7 8.1 Diff. -60.1 121
R 0.0175 R 0.9682
The low value of the correlation, R, for 11ba and11bb is
excellent news - there isn't much there except uncorrelated
noise. The high value for 11fa and 11fb says that in this
case most of the junk is fixed: the icicles didn't move much
between the exposures.
Estimating the dark current noise as above - assuming the
exposures are the same:
Mean variance from Diff. 32.8
Estimated dark current noise 7.1
Electronic var 25.8 sd 5.1 = 14.7 electrons
Using selected bits avoiding most cosmic rays
Average var sd Average sd
11ba -24650.8 23.1 4.8 11fa -3014.3 238
11bb -24637.1 29.7 5.5 11fb -2952.3 239
Diff. 51.7 7.2 121
R 0.0197 R 0.8707
The correlation between 11fa and 11fb is still high. Most of
the signal is variation in pixel sensitivity.
For dark files, 11ba and 11bb:
Mean variance from Diff. 25.9 adu^2
Estimated dark current noise 7.1
Electronic var 18.8 sd 4.3 = 12.6 electrons
So getting clear of (most of) the cosmic rays brings the
answer back pretty close to Tom's.
The major difference in going to the selected, cleaner, area
is a big reduction in the variance of the individual flat files.
The variance of the difference is essentially unchanged - a
coincidence. Modelling the total noise as the sum of random
variation in pixel sensitivity plus the root n electron noise
plus the electronic noise we can estimate:
For flat files, 11fa and 11fb:
Total variation 238 digits
Pixel variation 222 digits = 1.03% rms
Noise part 86 digits
Estimating (reciprocal) gain as change in average value between
the dark files and the illuminated ones divided by change in
variance gives 2.96 e-/adu which rounds up to 3.0. Hooray!
A different answer from Tom's 2.9.
Theory
The noise is modelled as a random superposition of electronic
noise plus the random fluctuations in the dark and photoelectric
charge. If the total charge is on average n electrons, the variance
is taken as n electrons and the standard deviation as root(n).
Define
d dark electrons in unit time: 2d in twice the time
f photoelectrons in unit time for the flat image
e electronic noise
A Gain, digits per electron
V0 Digits out for zero electrons
Then
Case Mean digits Variance digits^2
Dark d V0+Ad d+e A^2(d+e)
Dark x2 2d V0+2Ad 2d+e A^2(2d+e)
Flat d+f V0+A(d+f) d+f+e A^2(d+f+e)
So from the digitized output one can estimate
A = (var(Flat) - var(Dark))/((mean(Flat) - mean(Dark))
e = (2var(Dark) - var(Dark x2))/A^2
which is the first method used above. Since the variances
are not that well determined, especially with the odd cosmic
ray, this is not the best method.
Alternatively one can proceed as follows giving the second
method used above.
Ad = mean(Dark x2) - mean(Dark)
A^2(1.5d+e) = (var(Dark) + var(Dark x2))/2
e = {(var(Dark) + var(Dark x2))/2
- 1.5A(mean(Dark x2) - mean(Dark))}/A^2