In the last section, I quote some of
Tom Droege findings for dark images, and background information on dark current,
hot pixels, and image column formats. I also quote some of Andrew Bennett's findings
on dark images. This Tech Note therefore also serves to document their work and
to highlight open issues.
Documentation from the file accompanying these images says the
exposures were "about 200 seconds and vary". Tom has said (Jan 2000) it
takes about 45 seconds to read out the array, and that the minimum effective
dark current time is 46 seconds across the array. Tom has suggested (Dec 1999
maillist correspondence) that there may have been
some additional 10 second delays during the exposure of some of the images
in this CD-ROM data set. He has also said that he thought the temperature
of the CCD was fairly stable, but as it was not regulated at the time he
can't confirm the temperature. (Subsequent work as reported by Tom in TN 65
suggests a temperature roughly around -10C to 0C.)
Tom has also said in previous correspondence (28 Sept 1999):
"Disk 5 contains 7 successive frames in V and I on 5 nights spread out over
10 days. The first three nights are sequential, and pretty good. The last
2 are spaced 2 and 4 days apart and have moon and clouds. Should be good
data for developing software."
To reference the values reported in this Tech Note, Tom Droege's TN 65
specifies that the ADC is scaled and offset such that it ranges from -32768 to
+32767. The scale factor is roughly 2.4 e- counts per ADU, and the CCD442A has
a reported full-well saturation value of 80,000 e-. Therefore, Tom says an ADU
value of -25500 is roughly the dark current level for a short (15 second) exposure,
and with the scaling noted that saturation would be around +8000 ADU's.
The values as I have reported here do not
include the BZERO value in the image files of 32768: this is an offset
that must be subtracted to restore the original value of the pixels as read.
The FITS dark image files did not include this BZERO value: I edited the files
to include it but BZERO (32768) IS NOT SUBTRACTED FROM MY DATA IN THIS NOTE.
I will resolve this as I report some of Tom's data in
the latter part of my note.
Inspection of the early columns of all the images suggests that columns
1, 3, and 5 (from 0) are likely to be "dark pixels", that is pixels
that are covered in the CCD chip and which provide a dark reference
for each row. Also, solumns 0, 2, and 4 have average values and standard
deviations that are
also different from columns 1, 3, and 5, and from the imaging columns in
the dark images: their use and origin is not clear, and this report
will not examine them until their origin is made clear. As of Jan 2000,
Tom says these issues are not a priority to resolve. I report Tom's representation
of them as of September 1999.
Tom asked in 1999 for any indications of CCD pixel clocking transfer
inefficiencies. However, a set of dark exposures of roughly IDENTICAL exposure
time will NOT be informative of any losses due to clocking transfer
inefficiency. If there are any losses, they will "cancel out" if the
CCD is cleared at the same rate it is read out. Also, as dark current
depends on temperature as well as time, values of dark current cannot
be used to determine both time AND temperature. However, one could
reasonably assume that images with similar ADU values for covered pixels
have similar exposure times at similar temperatures. The covered pixels
for H3 images were very consistent, as were the covered pixels for H4 but
with a different value.
Table 1 below is a summary of column features for the dark images in CD #5.
Details of these measurements will be discussed.
All values above are in ADU's, and BZERO (32768) has not been subtracted. Column 5 is
of particular signifigance as a reference column. This will be discussed in a later section.
A typical set of values by columns for one image is in Table 2 below for VDARK15.FTS.
For dark image VDARK15.FTS, columns of pixels were read and averaged by column for each dark. A standard deviation value
was also determined, and a count of pixels per standard deviation. As the number beyond three
standard deviations were small, 2 to 6, and presumably due to "hot pixels",
I computed another distribution without those pixels and provide another
average and standard deviation. The consistent result of this recalculation was a change for a few ADUs
in the average, but a drop in the standard deviation to 1/4th or 1/5th its prior
value. IDARK17 had many more of these outliers. Again, all values are in ADUs WITHOUT BZERO
(32768) SUBTRACTED.
As for the other dark images (Table 1), the non-covered column values
for VDARK range from about 42300 ADUs to 42500 ADUs; for IDARK from about 42550 to
42900 ADU's. The range is much less within any single dark image, as shown in Table 2.
The VDARK darks have a consistent trend in column dark current by column, with
an additional step increase of about 50 ADU's for VDARK17. Likewise, IDARK17
shows a 50 ADU increase over IDARK15 and IDARK 16 by column. Typical standard
deviations for VDARK15 and 16 columns is under 200 ADUs generally, for VDARK17
it is under 300 ADUs. For IDARK15 and 16 it is also under 200 ADUs, for IDARK17
it is under 500 ADUs. (Removal of outliers reduces this considerably as noted.)
VDARK darks showed a dropoff
from columns 10 and 20 to around columns 400 to 600 of about 75 ADUs, followed
by a slightly decreasing slope to around column 1800, then a 15 to 20 ADU rise up
to column 2000. Typical values in the "flat" area (column 1200) are 42299 for VDARK15,
42310 for VDARK16, 42360 for VDARK17.
IDARK darks showed a brief drop to column 200, a faster rise to column 1800, then
a sharp rise at columns 2000, 2010 and 2020. The drop is about 20 to 30 ADUs; the
slow rise is about 70 ADU's for IDARK15 and 16, 160 ADUs for IDARK17. The sharp
rise is 110 ADUs, 160 ADUs, and 390 ADUs for IDARK15, 16, and 17 respectively.
The values in column 1200 are 42564, 42581, and 42767 for IDARK15, 16 and 17.
The * for VDARK16 in H3 indicates that the standard deviation for those dark columns
was 377 to 657 ADU's; after removing ONE outlier, the st. dev. dropped to 9 to
15 ADU's, and so I report the revised average value. It seems clear that,
outliers and exceptions notwithstanding, the dark columns are consistent by camera,
and consistent across the images taken by each camera.
Table 3 shows that for H3 files, the value varied from 40636 to 40656 ADUs. The
standard deviation was
below 3 ADUs. The value varied by TWO ADUs for a given image. There was
one H3 file that was a exception (H3R1446.866). For H4 files, the range was 40804
to 40821 ADUs, with variation in each image by two ADU's: there were two exceptions
(H4R1439.880, H4R1440.979) The
exceptions in both sets had very large standard deviations and higher ADU values.
I also compared column 5 and column 500 in the image files. Of course the variation
was higher as the imaged column includes sky background and star images. The standard
deviation was typically 600 to 1000 ADUs. But over the 2000-pixel run per column,
the background should probably dominate. Indeed, for H3 files
the average for column 500 was around 45000 to 46000 ADUs for most images, 48000
to 51000 ADU's for the rest. For H4 files the numbers were 47000 to 51000 ADUs
for most images, and up to 61000 ADUs for the rest. The shift would appear by
inspection to be consistent with greater backround levels (clouds?) as reported
in these images. Removal of several to a dozen outliers consistently RAISED the
average by tens of ADU's but dropped the new standard deviation to 100 to 300 ADUs
in general.
I again note that, as BZERO VALUES WERE NOT SUBTRACTED from
the values in the image files, as whereas there was NO ATTEMPT TO TREAT VALUES
BELOW BZERO (32768) AS "WRAP AROUND" VALUES, there may be some skew in the image
values read from column 500. However, given a standard deviation of under 1000,
and given maybe a dozen outliers beyond three standard deviations (under 3000 from 45000
to 51000), I would not expect these values to change much if these conditions were
added. Another way to look at this: in a single column of an image, to stumble over
a dozen stars or so makes some sense.
The trend by rows for VDARK darks is a drop of about 100ADUs from the first rows to
row 400; a flat region to end rows past 2000. Standard deviations for these rows are
200 to 500 ADUs, with 5 or 6 outliers at three sigma. Row 2034 is about 500 ADUs
higher than preceeding rows.
1)Row 0 is substantially higher in ADUs and in noise(3 times higher standard
deviation) than row 1. Row 1 is also higher than subsequent rows in ADU and
standard deviation. Some TASS members have chosen to
reject row 0 data from consideration, and my findings would support that and also to
consider rejecting row 1.
2) For the VDARK darks, EARLY rows and early columns (up to number 400) show a drop-off in ADU
values with increasing column or row number. In the IDARK dark images, those same rows and
columns are relatively flat. Instead, the LAST rows and columns
rise in ADU value: by 100 ADUs for row 1800 to 2000, by 200 ADUs or more from
column 1800 to 2000.
One explanation for a bright corner in VDARK images
would be electroluminscence from the CCD amplifying electronics. This occurs
when transistor junctions act as LED's and produce
photons. But it is not clear why IDARK would have a "bright corner" in the OPPOSITE
CORNER from VDARK, one would expect the phenomena to be either consistent or absent.
Another explanation came from Tom Droege in TASS discussion in 2000: some glue or material on
the edges of the CCD cover plate may have scattered additional light in those areas. This
would account for the differences between CCD devices.
The effect amounts to a few hundred ADUs out of a range roughly around 42700 ADUs.
Less the BZERO value of 32767, that leaves an actual ADU count of about 10000:
the noise would be about the square root of that or 100 ADUs. So in any case the
effect can be reduced by raising the dark level "floor" for the entire image,
with a slight reduction in available range.
Ted Woodhouse reported a similar bright corner when he constructed a flat from the
CD #5 images, which was subsequently used by Michael Richmond et al. Ted reported this
on Dec 4 1999 in the TASS maillist, his work is presumably referenced on the Mark IV
section of the TASS Web site.
3) The columns 1, 3, and 5 appear to be VERY consistent in value
across not only the dark images but the exposed star images: their standard deviation
is typically 2 or 3 ADUs and the ADU values themselves around 40637 for VDARK and
40804 for IDARK. As the images and darks are of identical exposure time and supposedly
identical temperatures, the consistency is expected to a point. Tom Droege
has also suggested there is a dark covered column at column 2034, but in CD #5 they
were apparently not available in H4 or IDARK image so I have ignored them here.
Tom has said that resolving the format of the leading and trailing
columns is "not a priority". I can't do any more work on them until they are
identified. Meanwhile it is reasonable to assume that the first several columns
and the last few columns are not image columns; and that columns 1, 3, and 5
are usable as covered dark columns.
It would be helpful if dark images of other exposure times and temperatures were available.
Without such data, a conversion factor cannot be determined for dark covered pixel values
to typical dark current values across the image. But the three darks per camera are
very consistent other than an offset value within each set, presumably due to variations
in time or temperature. (However, see Tom Droege's subsequent comments on this later in
this report.)
4) Averaging across rows or columns and eliminating pixels above 3 sigma from that average,
serves to elimate hot pixels in the reecomputed average.
For the most part, it seems there are 1 to 6 pixels
per row or column that are more than 3 sigma from the average. Inspection of the data
shows almost ALL of these are include very bright pixels above 10 sigma, but that
would include row 0 in the by-column data (see point 3) above). Incidently, I'd
like to thank Andrew Bennett for his use of three sigma and other statistical arguements,
which suggested the methods for my work here.
Point 2) above suggests a dark current noise of a few hundred
ADUs but VDARK17 and especially IDARK17 seem to have variation well above that (see
Table 4 vs. Table 5). Other TASS members have also suggested
that IDARK17 was noisy. One goal of this work was to establish methods to define a
background level and to check for consistency of background in a quantative way.
1) Tom Droege reported some work on dark current of recent images
on Jan 2000: my comments are in italics.
2) Tom Droege's Tech Note 65 reports at length on dark current, hot pixels
temperature control, and ice in the Mark IV camera as of May 2000. But prior to that TN he
offered another report which is also very informative, so I quote it here
at length:
3) Here is Tom's report on the early and late columns in the Mark IV.
This is all the info available as of Jan 2000 (and I know of nothing new
up to June 2000).
Andrew Bennett has worked extensively on Mark IV images from disk #5.
He produced a composite dark image with some of the intermittant hot spots
removed that TASS members found useful, and has discussed his "hot pixel"
findings at length. Below is a sample of his overall dark current findings.
His Web site has a number of his reports, and he has a Tech Note as well on
the TASS site.
Background
Many other people have also worked on
these images to review the column information, dark current, pixel noise,
and of course other issues. The TASS Web page has a section on "Mark IV"
that references some of that work. It can also be found in the Tech Notes,
and in the archives of TASS email correspondence. I'd be glad to directly
reference here any relevant information upon request.
Analysis
Inspection of dark images
Darks from one camera should not be used for dark subtraction or comparison
on another camera, so it is important to identify the images accordingly.
The FITS parameter "filter" for the H4 image files is "I", and for the H3 files
it is "V". Therefore the "VDARK" darks are from the H3 camera, and the "IDARK"
darks are from the H4 camera. Also, the "VDARK" dark frames are apparently
from the same camera as the "H3" images as both have a dark column 2041.
Likewise, the "IDARK" dark frames and the "H4" images both have a
column 2041 which is IDENTICAL to column 2042.
Comparisons by column
Table1: Summary of column averages and features
VDARK15 16 17 IDARK15 16 17
column averages 42300 to 42500 42550 to 42900
typical st. dev. <200 <200 <300 <200 <200 <500
col 5 for darks 40637, 40639*, 40643 40803, 40806, 40809
col 5 for images 40636 to 40656 40804 to 40821
col 5 st. dev 1.7 to 1.9 1.7 to 1.9
col 5 vs. col 10 +2000 +2000
trends from col. 10 drop, slow fall, slight rise drop, slow rise, sharp rise
flat area (col 1200) 42299 42310 42360 42564 42581 42767
Table 2: vdark15.fts by columns
col mean st dev >3 st.dev. new mean new st dev
1 40636.5 1.85 4 40636.4 1.79
3 40636.9 1.9 4 40636.9 1.86
5 40637.7 1.85 6 40637.7 1.81
10 42383.9 174 2 42379.7 48.6
20 42379.7 193 2 42375.2 47.7
100 42346.4 162.8 2 42342.6 36.7
200 42324.5 137.8 3 42320.9 32
400 42311 127.8 3 42307.4 25.7
600 42306.3 112.7 2 42303.7 24.1
800 42305.4 122.4 4 42303.1 24.3
1000 42301.2 103.7 2 42298.7 22.9
1200 42299.6 111.8 2 42297 22.2
1400 42298.3 112.3 2 42295.7 22.5
1600 42299 99.7 3 42296.4 21.6
1800 42298.7 115.2 3 42295.6 19.7
2000 42316.2 120.4 2 42313.3 25.9
2010 42319 122.9 3 42315.5 26.4
2020 42319.3 107.9 2 42316.7 25.8
VDARK vs. IDARK overall
Dark image column 5 vs. star image columns 5, 500
For all darks, Table 2 shows there is an increase of about 2000 ADU's from covered
column 5 to the first uncovered columns (column 10). The covered columns 1, 3, 5 have a
values from 40636 to 40821, with a standard deviation
of 1.7 to 1.9 ADUs. I compared the image's columns 1, 3, & 5 to column 10
in the dark images AND the exposed images, with results as shown in
Table 3:
Table 3: Column 5 values versus column 10
column 1,3,5 column 10 for V/IDARK15, 16, 17
H3: 40636 to 40656 VDARK: 40637, 40639*, 40643
H4: 40804 to 40821 IDARK: 40803, 40806, 40809
Comparisons by Rows
For all six dark images, row 0 is strikingly higher than the other rows by 500 to
1000 ADUs. Its standard deviation is also higher. A typical set of values by rows
for IDARK15.FTS is in Table 4 below:
Table 4: idark15.fts by rows
row avr st. dev >3 st. dev. new avr. new st. dev.
0 47275.69 1077.03 5 47306.77 575.25
1 42546.61 340.71 5 42540.36 60.66
2 42570.74 136.65 4 42570.93 48.1
3 42574.33 144.66 4 42574.3 47.62
5 42574.83 143.49 5 42574.46 49.52
10 42572.25 142.64 4 42572.26 46.28
20 42571.49 142.09 4 42571.51 45.87
100 42565.61 143.55 5 42565.25 43.76
200 42561.47 143.23 5 42561.02 41.95
400 42558.28 141.71 4 42558.25 41.5
600 42561.29 144.25 5 42560.97 42.7
800 42565.05 145.96 6 42564.21 42.74
1000 42566.58 140.85 5 42566.38 45.15
1200 42570.86 146.57 5 42570.48 48.33
1400 42572.66 150.29 5 42572.17 49.11
1600 42577.06 149.67 5 42576.7 52.41
1800 42587.67 156.78 5 42587.22 59.77
2000 42633.5 168.84 6 42632.85 90.74
2010 42637.99 173.48 5 42637.46 95.93
2020 42641.49 170.2 7 42640.62 93.51
2030 42648.56 179.23 7 42647.38 98.08
2031 42646.81 171.72 8 42645.61 97.11
2032 42646.98 176.8 5 42646.4 97.41
2033 42646.62 182.42 5 42645.89 98.46
2034 44025.78 858.11 46 43961.74 691.51
2035 42517.72 260.79 5 42512.77 20.63
2036 42517.42 133.2 4 42517.4 17.09
The trend for IDARK rows is a slight drop of 10-20
ADUs to row 400; a slight rise of about 30 to 120 ADUs to row 1800; then a rise
of 50 to 190 ADUs to row 2000. IDARK17 has the highest rise rates, and also an
overall higher ADU count over IDARK15 and 16 by about 200 ADUs. The standard
deviation for IDARK17 is typically 250 ADUs vs. 150 ADUs for IDARK15 and 16.
I would say that IDARK17 is "noisier" than the other IDARK darks. Table 5 is
IDARK17 measurements by columns, compare it to Table 4 for IDARK15.
Table 5: idark17.fts by rows
row avr st. dev >3 st. dev.new avr. new st. dev.
0 53791.13 21628.07 0 53791.13 21628.07
1 42764.95 312.66 7 42769.69 188.36
2 42819.96 217 14 42816.65 163.23
3 42829.61 230.47 19 42823.69 160.26
5 42831.11 241.1 18 42823.85 162.84
10 42823.11 224.76 10 42820.76 162.78
20 42820.83 223.79 13 42817.36 159.11
100 42790.19 218.65 9 42788.06 148.65
200 42771.96 226.47 11 42767.47 140.86
400 42755.61 211.6 17 42750.11 132.59
600 42769.47 226.33 19 42762.1 140.25
800 42783.62 243.3 12 42777.9 148.69
1000 42791.08 220.8 23 42783.52 149.23
1200 42807.56 243.82 21 42799.36 160.27
1400 42815.48 255.42 17 42808.12 163.08
1600 42834.26 257.44 20 42825.88 174
1800 42875.65 291.42 21 42865.49 199.41
2000 43064.48 383.44 30 43046.36 306.01
2010 43086.39 411.84 26 43068.82 329.6
2020 43099.69 400.22 34 43077.94 316.57
2030 43128.56 433.37 28 43108.38 340.74
2031 43123.17 416.54 25 43106.42 343.86
2032 43123.26 421.91 31 43102.24 330.12
2033 43120.56 435.25 21 43105.96 345.65
2034 48243.05 3054.75 43 48038.74 2541
2035 42526.4 653.95 2 42538.32 75.25
2036 42551.38 137.3 4 42551.36 28.52
Findings for CD #5, and comments
Software
The C program for reading Mark IV FITS files and reporting by columns is
mkivcols.c. It compiles under Turbo C++ for MS-DOS
(sorry) but may well compile under Windows and Linux with some mods. It read FITS
header values for rows and columns. It does NOT read the BZERO value but
one could add that in. It produces a range of statistics, including binning results
by 50 rows and showing how they vary from the column average. It produces a file with
various results depending on what the programmer choses to report, which is why I
offer it in source code. The program mkivrows.c performs a
similar service by rows. While I provide these programs only to document my methods and to
suggest implementations, the MS-DOS executable are also provided, namely
mkivcols.exe and
mkivrows.exe.
These programs should be available on the TASS Web site
where this Tech Note was obtained, contact that site if the Web links fail in this document.
Related work by Tom Droege
Date: Mon, 17 Jan 2000 16:53:07 -0600
To: tass@listserv.wwa.com
From: Tom Droege
Date: Wed, 19 Jan 2000 20:20:45 -0600
To: tass@listserv.wwa.com
From: Tom Droege
Date: Mon, 27 Sep 1999 22:56:20 -0500
From: Tom Droege
Related work by Andrew Bennett
Date: Mon, 22 Nov 1999 16:02:10 GMT
From: Andrew Bennett