[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Image Storage
- To: Undisclosed recipients: ;
- Subject: Re: Image Storage
- From: Tass Mailing List <firstname.lastname@example.org>
- Date: Thu, 04 Jan 2007 17:20:52 -0800 (PST)
Date: Thu, 04 Jan 2007 20:02:59 -0500
From: Doug Welch <email@example.com>
To: Tass Mailing List <firstname.lastname@example.org>
Subject: Re: Image Storage
I meant Cortez, not Cortex! :)
Tass Mailing List wrote:
> Date: Thu, 04 Jan 2007 17:38:55 -0500
> From: Doug Welch <email@example.com>
> To: Tass Mailing List <firstname.lastname@example.org>
> Subject: Re: Image Storage
> Hi Tom,
> Well - I have to disagree. To paraphrase Cortex "Load them all and let
> the computer sort it out." For instance, one could ingest the pipelined
> data and then update the image record in the db (not the header) -
> identified by JD+filter or some other ruse - with the number of stars
> measured. Finally, do a pass through to disable all images records in
> the db which do not pass some minimum production standard.
> My basic point is that any workaround which requires someone to be on
> call to load CD/DVDs will prevent extensive use of the images and
> photometry from TASS. The dark-subtraction and flat-fielding are
> single-time operations. Once done, they stay done and the images can be
> used from then on.
> Tass Mailing List wrote:
>> Date: Thu, 04 Jan 2007 13:40:39 -0600
>> From: Thomas F. Droege <email@example.com>
>> To: firstname.lastname@example.org
>> Subject: Image Storage
>> I think the last thing we want to do is to store my archive disks on a
>> big server somewhere. The attached jan_2_3.png is data from the last
>> two nights. It is an example of real data. Some data was taken between
>> 0 and 200 degrees in ra. You will notice big holes. This is because
>> there was a near full moon on these two nights. I just start taking
>> images at dusk, and run until dawn taking images. About 2000 images are
>> stored on the 6 DVDs that I wrote. Note that there is no data from tom1
>> after the moon passed. This is because tom1 failed to find the home
>> limit switch in the bright moonlight and just died. My guess is that
>> about 70% of these images are junk. The pipeline just throws out images
>> where the sky background from the moon is too large, or where the
>> background is too irregular (clouds), or where it does not come to an
>> astrometric solution.
>> Do we really want to store images on a public archive that are 70% junk?
>> Further, these are not dark subtracted or flat fielded images. The
>> data to do this is on the disk, as are the .list files.
>> Seems to me a better plan would be to call up the .list files and look
>> at the data they contain in order to decide whether to load an image.
>> The first thing to look at is the "bad" flag. This means that it does
>> not meet very broad criteria for good data. Next one can look at the
>> sky background level. This will be high for images taken at dawn and
>> dusk. Next we might look at the star count for this image. One might
>> require that an image contain a high percentage of the possible star
>> count to be loaded. Many other things come to mind. Note also that the
>> .list file contains the astrometric solution for the center of the
>> image. The fits header just contains my guess to get the solution
>> started. The loading process might want to update the .fits header with
>> the correct values.
>> This holds true for the current data, 2005, 2006, 2007. Data from 2003
>> and 2004 has other problems. First it was written as dark subtracted
>> and flat fielded images. This solves one problem. But the images for a
>> night are spread over a number of CDs. I will have to look at some of
>> the disks, but I think they just contained images at the start. Later I
>> saved other things if I had room.
>> I think it requires some discussion before we jump ahead and load disks
>> into a file. What do you all think? Just because we only load
>> relatively good images to the file does not mean that the archive disks
>> are not available to a determined researcher. The process of loading
>> the disks should produce a comprehensive catalog of all the images.
>> Someone may want to look at "moon" data which is full of streaks and
>> other funny artifacts when a star was doing something really interesting
>> on that night.
>> Tom Droege