[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Validation


Tom,

My "law" of computing is that the difficulty of a project is
proportional to C to the Nth power (C^N) where N is the number
of computers you use to solve the problem.  OK if you have a
big problem you need N computers but you can try to make C
small, "C" is the size/complexity (number of interfaces
that need to be configured) of one computer.

If you haven't messed around with setting up NFS mounts it may be
worth your effort.  You put all the software and catalog data on
one computer.  Then using NFS make the directory containing this
to appear on all other PCs.  Advantages are:

1) You only set up the software, catalog and scripts once

2) Because of #1 you be be pretty much certain that you are
   using the same software, scripts amd catalog data on all the
   computers.

3) You have reduced "C" by a large factor.

You could mount your data (raw images and processed data) directories
across al the PCs too but don't try it unless Michael tells you he
has thought about "concurency problems"  So, keep the data
seporate but I think you'd be better off with a single installation
of the software.


At some later date it might be fun to figure out how best to put
a true "cluster" of computers to work on theis probllem.  It is
clear that this is exactly the type of problem that clusters were
ment to solve.  With this methom "C" in above forula approaches
one for most of the PCs.

The way these work is simple.  There are N "worker PCs" and one
"taskmaster PC".  workers after finishing one job ask the taskmaster
for another.  It gives the worker the name of a raw image file and
a location to put the reduced data.  Worker PC typically don't even
have operating systems and may not even have hard drives.  They boot
off the network.  Typically the "taskmaster" supplies the boot
image.  So building a hugh array is easy as the worker PCs are what
they call "zero configuration"  Just connect the Ethernet and
turn on the power.  The computer animation industry uses this
method.  They'll build arrays with a couple hundred CPUs
They'd go nuts if they had to install and maintain a couple hundred
copies of a complex software system.  One of the first places to
setup something like this was at Fermi Lab. They are getting to
be common now.



--- Tom Droege <tdroege2@earthlink.net> wrote:
> I am trying to get several networked computers all set up to process
> image 
> files to star lists.  It is a struggle.  So far I am yet to get
> consistent 
> answers.  Sigh!   Mostly it is my fault in that I don't have all the 
> constants set to the same value.
> 
> I have at least one computer which has some individualistic
> tendencies.  It 
> will load 27 8MB images and then quit like it has loaded everything
> on the 
> disk.  This without complaint and even though there are 80 images on
> the 
> disk.  It has exactly the same set up as another computer that works 
> perfectly.  Possibly this is some well known problem.  Mandrake 9.0,
> 256 MB 
> of main memory, plenty of disk menory.  Note this computer was set up
> with 
> exactly the same sequence as another that works without problem.  The
> only 
> difference is the size and shape of the disk partitions and the funny
> 
> computer has "only" 256MB of memory.  Possibly the Mandrake set up
> program 
> in it's wisdom set some buffer different on this computer?  I was
> running a 
> large (15,000 square foot computer room) computer installation when
> there 
> was probably not 256 MB of memory in the world.
> 
> Today I will set up with a master pipeline on one computer and just
> move it 
> to the other computers.
> 
> I am working on a tech note that will describe everything in the set
> up so 
> that later we can go back and see how this run was processed.
> 
> I know that this is a well known problem for some of you, but I have
> to 
> work out how to do it from scratch.
> 
> There is lots of good data to process once I get everything working 
> smoothly.  I have close to 360 degrees at -6 to + 18. There is
> roughly 
> 10-20 days of observations for each star.  There should be about 1
> million 
> stars in the list.  This should be enough data to show lots of medium
> and 
> long period variables.
> 
> Each day I get closer to being able to run 3 systems.  Sigh!  It is
> always 
> something as those of you with systems know.  The current problem is
> that 
> (even though Dan asked me if he was installing the right ones) I
> somehow 
> had the wrong pulleys put on TOM2 and TOM3.
> 
> However, once I get one running, it just keeps going.  Taking data
> with 
> TOM1 is routine.  I just open it up and run.  Then I transcribe the
> data to 
> CD.  Every so often I look at the desiccant and change it when pink. 
> Even 
> the focus now holds constant.
> 
> Tom Droege
> 
> 
> 


=====
Chris Albertson
  Home:   310-376-1029  chrisalbertson90278@yahoo.com
  Cell:   310-990-7550
  Office: 310-336-5189  Christopher.J.Albertson@aero.org
  KG6OMK

__________________________________________________
Do you Yahoo!?
Yahoo! Platinum - Watch CBS' NCAA March Madness, live on your desktop!
http://platinum.yahoo.com