Re: Database

[Chris Albertson <chrisalbertson90278@yahoo.com>]

aah@nofs.navy.mil wrote:
> 
> Paul,
>   Thanks for your clarification.  While it is true that
> you don't need to update for every star, and can use
> a batch transaction, I see two problems:  the ~18 daily
> updates will be posted at different times depending on
> the site; and the database has to be available to the
> outside sometime during the day!  So you cannot take it
> off-line for extensive periods while updating.  So in
> this sense, the database is 'real-time'.  You might
> be able to ping-pong and have two versions running, one
> that accepts outside queries and is updated once/day from
> the other, which is doing the data entry.

You don't have to "ping pong" that is one reason we want to
use a true DBMS system like Postgres.  It allows concurent
access for as many users as you like and keeps stuff straight
so that a reader for example nevers sees a "half record" in
the process of being input.  We can teach it that frames are
"atomic" and are added "all or nothing at all" so if a reader sees
a frame he will always know ALL of it's data is there.
Yes the complexity needed to make simultainious reading and
writting by possably hundreds of users is great but it done
for us by Postgres' quarter million lines of code.
I expect that we will run one input process for each camera
site and have as many as six going at once.  Postgres can
take advantage of multiple CPUs and scales well.  Actually
Oracle and DB2 are better but unless we qualify as "Personal use"
as they define it the fee is in five figures.


>   Archiving needs to be done periodically.  We had a few
> cases where transactions needed to be backed out; this
> is sometimes easier if you just restore the previous day's
> master files.
>   I haven't discussed the quality control issues:  removing
> bad data, determining extinction/transformation coefficients
> for each site or night, etc.
>   What I am getting at is that, as with all large survey
> efforts like MACHO or SDSS, data flow, archiving, quality
> control and access are all difficult without personnel.
> We don't have postdocs or FTEs, so all effort is voluntary.
> The easier you make that effort, the more likely the
> process is to succeed.  Otherwise, the more likely it is
> that systems will not be run or else the sites will process
> and keep their own data.  These are all obvious points,
> but I just want people to keep them in mind.
> Arne

-- 

--
   Chris Albertson             
   chrisalbertson90278@yahoo.com
   Redondo Beach, California
   home: 310-376-1029
   cell: 310-990-7550