Re: Image Storage

[Tass Mailing List <tass@mail.alembic.net>]

Date: Thu, 04 Jan 2007 20:02:59 -0500
From: Doug Welch <welch@physics.mcmaster.ca>
To: Tass Mailing List <tass@mail.alembic.net>
Subject: Re: Image Storage


I meant Cortez, not Cortex! :)

Tass Mailing List wrote:
>
> Date: Thu, 04 Jan 2007 17:38:55 -0500
> From: Doug Welch <welch@physics.mcmaster.ca>
> To: Tass Mailing List <tass@mail.alembic.net>
> Subject: Re: Image Storage
>
>
> Hi Tom,
>
> Well - I have to disagree. To paraphrase Cortex "Load them all and let
> the computer sort it out." For instance, one could ingest the pipelined
> data and then update the image record in the db (not the header) -
> identified by JD+filter or some other ruse - with the number of stars
> measured. Finally, do a pass through to disable all images records in
> the db which do not pass some minimum production standard.
>
> My basic point is that any workaround which requires someone to be on
> call to load CD/DVDs will prevent extensive use of the images and
> photometry from TASS. The dark-subtraction and flat-fielding are
> single-time operations. Once done, they stay done and the images can be
> used from then on.
>
> Cheers,
> Doug
>
> Tass Mailing List wrote:
>> Date: Thu, 04 Jan 2007 13:40:39 -0600
>> From: Thomas F. Droege <droege@fastmail.fm>
>> To: tass@tass-survey.org
>> Subject: Image Storage
>>
>> I think the last thing we want to do is to store my archive disks on a
>> big server somewhere.  The attached jan_2_3.png is data from the last
>> two nights. It is an example of real data.  Some data was taken between
>> 0 and 200 degrees in ra.  You will notice big holes.  This is because
>> there was a near full moon on these two nights.  I just start taking
>> images at dusk, and run until dawn taking images.  About 2000 images are
>> stored on the 6 DVDs that I wrote.  Note that there is no data from tom1
>> after the moon passed.  This is because tom1 failed to find the home
>> limit switch in the bright moonlight and just died.  My guess is that
>> about 70% of these images are junk.  The pipeline just throws out images
>> where the sky background from the moon is too large, or where the
>> background is too irregular (clouds), or where it does not come to an
>> astrometric solution.
>>
>> Do we really want to store images on a public archive that are 70% junk?
>> Further, these are not dark subtracted or flat fielded images.  The
>> data to do this is on the disk, as are the .list files.
>>
>> Seems to me a better plan would be to call up the .list files and look
>> at the data they contain in order to decide whether to load an image.
>> The first thing to look at is the "bad" flag.  This means that it does
>> not meet very broad criteria for good data.  Next one can look at the
>> sky background level.  This will be high for images taken at dawn and
>> dusk.  Next we might look at the star count for this image.  One might
>> require that an image contain a high percentage of the possible star
>> count to be loaded.  Many other things come to mind. Note also that the
>> .list file contains the astrometric solution for the center of the
>> image.  The fits header just contains my guess to get the solution
>> started. The loading process might want to update the .fits header with
>> the correct values.
>>
>> This holds true for the current data, 2005, 2006, 2007.  Data from 2003
>> and 2004 has other problems.  First it was written as dark subtracted
>> and flat fielded images.  This solves one problem.  But the images for a
>> night are spread over a number of CDs.  I will have to look at some of
>> the disks, but I think they just contained images at the start.  Later I
>> saved other things if I had room.
>>
>> I think it requires some discussion before we jump ahead and load disks
>> into a file.  What do you all think?  Just because we only load
>> relatively good images to the file does not mean that the archive disks
>> are not available to a determined researcher.  The process of loading
>> the disks should produce a comprehensive catalog of all the images.
>> Someone may want to look at "moon" data which is full of streaks and
>> other funny artifacts when a star was doing something really interesting
>> on that night.
>>
>> Tom Droege
>>
>>
>> ------------------------------------------------------------------------
>>
>