[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Time for Software/databases

"Creager, Robert S" wrote:
> I'm going to toss an idea out and see if anyone cringes.  Does anyone have a
> DSL/Cable computer connection which run Linux/PostGreSQL and is on all the
> time?  If so, I believe I can write some software which would allow for a
> secure central database server.  Or are we talking too much data from each
> TASS site that those of us stuck in the modem age wouldn't be able to update
> the central TASS database?  Pardon if this is an uninformed question, but
> I'm uninformed :-).

Maybe you are new to the list but there IS a TASS database
on line now.  It is running PostgreSQL too.  And yes you
can do limited searches via a web interface or unlimited
queries from any Postgres client or ODBC.  Michael Richmond
runs the server on his office Linux PC which is also the
home of www.tass-survey.com.  I wrote some of the software he 
uses.  He fixed it.  I run a copy of the database here on
my PC but with a small subset (about 1/4) of the data.

I'd like to change a few things for the Mk IV database.
Mainly off load some CPU intensive processing to other
computers away from Michael's and make what I'll call
"non-central" databases.  Don't know the details yet but
All the database would be "peers" in that they'd run the same
software but one (Michael's) would be the biggest peer
in that it holds all of the data while the others hold less
than all.

My idea is that camera operators (or someone working with
a camera operator) processes his local data into his local
peer database.  At intervals he transfer data that is in
the local database to Michael and pulls an updated master
TASS star catalog back to his local site.  The transfer
media could be either the Internet of CD-ROMS in the mail.

If you'd like to write software go ahead there is lots of
work to do.  I am going to work in real-time control software
before I re-visit data base software.  I would however be
happy to share what I've learned writing Mk III DBMS software.

As for the amount and rate of data just do the math.
N systems * 2 cameras * M (exposures per night) * 
(fraction of clear nights) * K (stars per image)
Maybe I missed something but you get the Idea.  I think the
answer is in the "few hundred millions of rows" by a year
or so.  Assume a decade lifetime for the Mk IV system
and we will be into a few billion row tables. 

One more thing.  I like Postgres but I want to write
the new software so it is independent of the database
used.  That way someone could us MySQL or even IBM's DB2
or Oracle.  If we use Perl DBI, ODBC or JDBC we don't
tie ourselves to one DBMS.  Problem is with this is you
loose some of Postgres' nice features.  So maybe we just
make some of the code DBMS independent.
   --Chris Albertson             home: chrisja@jps.net        
     Redondo Beach, California   work: calbertson@logicon.com