Re: We need to be 1,000 times faster.

[Tom Droege <droege@wwa.com>]

I was just talking to someone about the Sloan.  The Fermilab computing 
group designed the data acquisition system.  They nearly met the 
schedule.  As a result, they were 3-5 years too early and the system is 
designed around computers so obsolete that they have to be special ordered 
for spares.  They find that they can spend x dollars and get 20% spares for 
the old design.  The same x dollars would buy 1/5 the number of computers 
that would run 5x as fast and do the whole job.  So they are faced with 
buying something slightly different and upgrading software, etc., or buying 
old stuff for spares and replacements.

It seems like it will always be thus in the computer biz.

I say don't worry about it.  We can now just barely do the Mark III 
job.  By the time that the Mark IVs are spitting out all that data, we will 
just barely be able to handle it.  Note that I thought about all this when 
I started this project.  If anything the computer cost for the project had 
decreased relatively from what I expected.  When I started this, I was 
running with 20Mbyte disk drives.

What we don't want to do is to be tied to any particular bit of 
hardware.  Sloan has a lot of special hardware.  We only have the need for 
a single ISA slot.  You can still buy computers with ISA slots (probably 
not for long).  We only need this on the very front end so we should be 
able to make do with old PCs as the telescope controller.

So I say don't be afraid of the size of the problem.  It will get smaller 
with time.  Let's take good data and do what we can to reduce it - at least 
to calibrated star lists.  I think each of us can do that at our sites with 
present technology.  We can then always process some of it to a data 
base.  We just do what we can.  If this data base is useful, pressure will 
grow to process all the data.  A way will be found.  Data in astronomy has 
time value.  You can never again take data for the year 2000.  So if we 
have it in an archive, some of it will be wanted.

The big problem is to figure out how to store the data so that it can be 
processed at a later date.

Tom Droege

At 11:52 AM 8/4/00 -0700, you wrote:
>Some problems with Chris' scheme, though in general
>I agree that distributed database processing is what
>will be needed.  First, a site such as NOFS or Tom's
>6X will produce more data per year than is currently in Michael's
>entire database.  If Michael has problems updating it, then
>at the same level each site will have problems in maintaining
>their own individual database.  Second, while
>FITS tables format is certainly a convenient,
>standard way to keep data available that might be
>needed for queries, such tables are not SQL-accessible
>and I wonder how a centralized server could handle them
>in an efficient manner.  The total database is not that
>much different in size from SDSS.  Let's see what ARNE
>brings to the table.  Perhaps the time really has come
>to get Microsoft, etc. involved.
>Arne