Image Storage

[Tass Mailing List <tass@mail.alembic.net>]

Date: Thu, 04 Jan 2007 13:40:39 -0600
From: Thomas F. Droege <droege@fastmail.fm>
To: tass@tass-survey.org
Subject: Image Storage

I think the last thing we want to do is to store my archive disks on a
big server somewhere.  The attached jan_2_3.png is data from the last
two nights. It is an example of real data.  Some data was taken between
0 and 200 degrees in ra.  You will notice big holes.  This is because
there was a near full moon on these two nights.  I just start taking
images at dusk, and run until dawn taking images.  About 2000 images are
stored on the 6 DVDs that I wrote.  Note that there is no data from tom1
after the moon passed.  This is because tom1 failed to find the home
limit switch in the bright moonlight and just died.  My guess is that
about 70% of these images are junk.  The pipeline just throws out images
where the sky background from the moon is too large, or where the
background is too irregular (clouds), or where it does not come to an
astrometric solution.

Do we really want to store images on a public archive that are 70% junk?
Further, these are not dark subtracted or flat fielded images.  The
data to do this is on the disk, as are the .list files.

Seems to me a better plan would be to call up the .list files and look
at the data they contain in order to decide whether to load an image.
The first thing to look at is the "bad" flag.  This means that it does
not meet very broad criteria for good data.  Next one can look at the
sky background level.  This will be high for images taken at dawn and
dusk.  Next we might look at the star count for this image.  One might
require that an image contain a high percentage of the possible star
count to be loaded.  Many other things come to mind. Note also that the
.list file contains the astrometric solution for the center of the
image.  The fits header just contains my guess to get the solution
started. The loading process might want to update the .fits header with
the correct values.

This holds true for the current data, 2005, 2006, 2007.  Data from 2003
and 2004 has other problems.  First it was written as dark subtracted
and flat fielded images.  This solves one problem.  But the images for a
night are spread over a number of CDs.  I will have to look at some of
the disks, but I think they just contained images at the start.  Later I
saved other things if I had room.

I think it requires some discussion before we jump ahead and load disks
into a file.  What do you all think?  Just because we only load
relatively good images to the file does not mean that the archive disks
are not available to a determined researcher.  The process of loading
the disks should produce a comprehensive catalog of all the images.
Someone may want to look at "moon" data which is full of streaks and
other funny artifacts when a star was doing something really interesting
on that night.

Tom Droege
-- 
  Thomas F. Droege
  droege@fastmail.fm

jan_2_3.png