[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Processing Data
Since I find myself trying to answer this privately and frequently, let me
try to outline how the data should be processed. I write this only because
no one has written a cook book TN that outlines how the data should be
handled. Some of you will notice a different procedure from that which I
last recommended. This is because I have thought more about it.
Do not think that I know what I am doing. I have arrived at the procedure
below by thinking about it. Experts, please comment!!!
The data disks contain darks and sky exposures. Those being worked on by
the Data Reduction Group contain a dark followed by 4 exposures of the same
sky. I will call these nx for the dark, and na, nb, nc, and nd for the
successive exposures of the same star field (to a few pixels). For this
discussion, n=1 is the first exposure set of the night, and for some data
sets n goes as high as 48. All the exposures were of the same length.
In general the process below assumes that all images were taken under the
same conditions. That is, the CCD temperature did not change, the exposure
times did not change, etc.. One has to watch out for other variations. I
am running in the suburbs. Sometimes my neighbors shine bright lights at
my telescope. Sometimes they turn these off at some time, say
midnight. One really needs a way to sort out data taken under similar
conditions. Arne does not have to worry about this sort of thing. But I
do, and the Data Reduction Group will need to do something.
OK, assuming a set of data that has been filtered to be a matched set of
nx, na, nb, nc, and nd exposures, here is what I would propose to do
(obviously the V and I cameras are done separately):
1) Take some nx's. I would first take the mean of each, and throw out any
that deviated from the others. I have some old data on my TOM
machine. Here is what I found:
File ext. Mean Value
.567 -25272.7
.578 -25428.3
.589 -25433.4
.600 -25423.5
.610 -25443.4
.621 -25435.1
I would clearly throw out the .567 point as it was probably taken while
cooling down. A less negative value for .567 pretty much confirms this, it
has more dark current. The dark current variation looks like a sigma of 5
or so by eye after cooldown. Much less than the sigma of the data pixel to
pixel.
Picking a set of ni - the group above from .578 to .621 would probably be
enough, compute the median of the set. For the statistically impaired (me)
this is just done with some computer program that does it. I looks at the
frames pixel by pixel and forms a new frame where each pixel is the mid
value pixel from the 5 frames above. For me, 5 seem quite enough to remove
the cosmic rays and gives a dark frame that looks nice. If you want, use a
larger number of the darks. But there are only 8 on a disk. No reason
not to use all the disks for an evening where the temperature has
stabilized. Note that you should probably throw away the data for the
darks that are well away from the mean. This means the first set of data
frames for the day 1853 above. But note that the sigma for the dark frames
above is 35-40, so if a dark is 1 sigma from the mean of the darks, one can
probably keep it's data. One of the problems in the data reduction is to
figure out how to make this sort of decision about the data on the fly.
Once this median is taken, save it as the "Dark Frame" for the data set.
2) Now we work on the Flat Frame. We generate a flat frame by taking the
median of a bunch of sky exposures. For the disks that have been sent out,
the na, nb, nc, and nd frames are of the same star field. The stars move
very little between frames. This means that one can only use one of each
exposure set for the flat. Now collect as many sky exposures as you can
get taken under the same conditions. You will just have to look at them to
see if they are any good. The way I would do this is to use the nb or nc
exposures. Do not use the na exposures as they are looking at a tree
limb. I have found that I need at least 16 to get a good flat. More is
probably better. I would first take the mean of each frame and throw out
any for which the mean is significantly different from the
others. Significant is relative. Since the sky frames have a sigma of
order 50-100, I would not reject a 25 or so mean difference.
OK, I just tried day and here is what I got from the first two:
Frame Mean Sigma
1853.584 -21868.7 56.2
1853.595 -19455.0 75.0
Looking at the data, the moon is coming and it is just a streaky mess. So
I would not use these frames.
Moving to run 1824, there is less variation:
Frame Mean Sigma
1824.703 -22637.4 42.1
1824.716 -22578.3 42.8
1824.729 -22810.4 41.4
1824.742 -23091.4 39.6
There is a clear trend appearing. It is getting darker. So I would move
to a time later in the evening and try to get 16 or so frame sets in a row
where the variation is as small as possible. Note that Data Disk 16 was
specifically set up with the best data I had at the time for this
purpose. It does, however, not follow the exposure sequence of the Data
Reduction Group disks.
Once one has selected a suitable group of 16 or more sky frames which seem
to be representative, subtract the Dark Frame from 1) from each, and take
the median. Look at it. If you can still see stars in it, use more
frames. In fact it will be instructive for someone who has not done this
to take various numbers of frames for the median and to look at the
results. Display programs are pretty tricky and are intended to accentuate
variations. You can still see stars in one that might be quite acceptable
for use. There is no use going down too deeply into the noise.
Experts might comment on whether it is better to subtract the dark frame
from each frame used before the median is taken, or to subtract it from the
median after it is formed. I figure this order gives less apparent noise
in the median. I am not sure that this results in less real noise. I was
once a graduate student in mathematics and know enough to suspect that:
median[(1-k), (2-k), ... (n-k)] is not guaranteed equal to median
[1, 2, ... n] - k for all data sets.
Once this is median is taken save it as the Flat Frame for the data
set. Note that this frame has had the dark current subtracted from
it. The pixels should all be greater than 1 (by a lot) since each pixel is
seeing the sky. Variations represent gain variations of the system. They
might be dirt on the CCD, variation in sensitivity over the field of the
lens, or anything else that affects the gain. This assumes that the sky is
equally bright in all directions. At least it assumes that the variations
in the brightness of the sky over the 4 x 4 degree field are negligible.
3) To view a frame, take the raw frame and subtract the Dark Frame from
1). This removes the dark current. Now divide it by the Flat Frame from
2). This removes the gain variations. What is left is what is possible to
correct, I think. There are still lots of variation. A large effect is
the neighbors lights. I can do nothing about them. For best work I would
start at dawn and work back. There is also sky light from Chicago. A
cloud between me and Chicago could change the sky a lot, even though it is
clear at the moment looking through the telescope. So there are a lot of
problems to solve.
For the Mark IV system, the dark current should stay pretty constant for
runs taken at the same temperature. I think dark current subtraction will
not be much of a problem. It should vary significantly less than the noise.
Flat fields should also be the same over time. They represent dirt and
optics and such. One should be able to use the same one unless something
changes. One thing to come out of the Data Reduction Group work should be
some measurements of what needs to be done with dark and flat fields.
Tom Droege