[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Cleaning Out Clouds
I've a C program which will generate statistics on a region of a fits file.
It uses CFITSIO. I've not used it for a while, and it needs a little
cleaning... If you want, I can clean it up by this weekend...
Rob
-----Original Message-----
From: Tom Droege [mailto:tdroege2@earthlink.net]
Sent: September 21, 2002 11:17 PM
To: tass@listserv.wwa.com
Subject: Cleaning Out Clouds
I have spent the last several weeks mucking about with the pipeline
attempting to find the cause of the scatter at bright magnitudes. I am
fearless, so I have tried changing everything I could get my hands on. I
have also checked a few things. I have been running the computer full time
trying things.
1) I did two different linearity checks. The chip I checked is linear to
about 43,000 ADU. Somewhat more than the 80,000 e- (32000 ADU) that is
advertised. Indications are that the newer chips are not so good. So I
will set the linearity limit at 32,000 counts up from the present
20,000. I will get out a tech note on this.
2) I tried using different sets of reference stars. Using a smaller more
accurate set fools you at first. It makes the mag vs mag sigma plot look
tighter. But it does this by increasing the error of the fainter
stars. So the largest catalog gives the best distribution.
3) I found a light leak in the flat field box at IR. I thought this would
explain everything. But a new flat field made with a sealed up box in
darkness did not make an improvement. The old one was as good as the
new. (It was made in the tower during daytime with the doors
closed. Pretty dark.) Making flats from a large number of sky images
seemed to be a little better than the flat box. This as viewed by looking
at the cleaned .fits images from ccdproc. But it does not seem to be a
large improvement.
4) I tried different apertures. Again, this gave a misleading
result. Larger apertures tightened up the grouping of the mag vs mag sigma
plot but again did this by increasing the error at the faint end. An
aperture of 4 is near optimum. There is the possibility that 3 will be
slightly better.
5) Yet to be checked, the scatter as a function of location in the
sky. The idea is to make sky flats from a narrow range of telescope
pointing. Another thing yet to be checked is the quality of the darks. I
give little hope that either of these things will do anything.
OK, now the thing that was successful.
Get rid of clouds.
To do this, I looked at the dark subtracted and flatfielded images as
produced by ccdproc. I looked with DS9 in the line analyze horizontal and
vertical cut mode. I just rejected any frame where the base line varied
more than the noise level. This eliminated about half of one data set but
significantly cleaned up the mag vs mag sigma plot. This with images that
really looked pretty good when you looked at them. I had previously tried
a cruder scheme to reject frames with no significant result. I soon
realized that there was nothing easy to do with the raw images. You needed
to dark subtract and flat field them. So why not use the result of the
pipeline?
OK, here is what I propose to do:
1) Run the pipeline through the ccdproc step using a light box flat.
2) Look at the resulting dark subtracted and flat fielded .fits images
with some program. Discard the images where there is any gradient. This
with some parameters to set so that I can tune the process.
3) Restart the pipeline from the make flat step and use the remaining
images to make a sky flat if there are enough left, otherwise use the light
box flat. (Which is not so bad.)
Does anyone want to write a program that does 2? I can do the rest. It
appears to be worth doing. One just looks in a specified directory and
uses your favorite program to somehow look at each image and determine if
they have any significant gradient to some specification, then delete the
.fits files that fail. When you look at cloudy images, there are white
smudges that can be in any location. So you have to cover most of the
images. I would think fitting three or four top to bottom and side to side
lines would do. Just fit second order? lines and reject images where the
deviation from flat was greater than some number, like the noise level. Or
compute the mean image and some random small areas. Delete the image if
the small area means differ from the image mean by more than the noise
level. Or some such. You do it and you get to do something clever.
If no one takes it on I will have a go at it. Michael has shown me how to
use the tools in the xvista package, and I can use these (somehow) to do
2). But you are letting me write code and that is probably a big mistake.
Comments from experts?
Tom Droege