Michael Richmond
richmond@astro.princeton.edu
Dec 1, 1996
I went up to Vermont for Thanksgiving, and discovered that the liquid which cools the TE units had frozen in its plastic tubing! I guess this shouldn't have been a surprise, since it gets pretty cold up there, and the tubing runs for 40 feet from a basement, under the lawn, to the cameras (and back).
Fortunately, there was a solution. I found two hair dryers, placed one at each end of the PVC pipe running from camera to basement, and let them run for about an hour. It was quick work to thaw out the exposed tubing near the camera itself, but it took quite a while to get rid of the ice in the middle of the tubing. At last, however, after turning on the pump, I saw the bubbles move. Phew.
So, I picked up 7 gallons of antifreeze (the non-toxic kind) and added them to the 25 gallons of water (which already had a little antifreeze in it). In addition, I've left the pump running, hoping that if the water flows 24 hours a day, all winter long, it's less likely to freeze.
While I was there, my brother and I drilled a hole in the floor so that we could run a telephone wire down from the first floor to the basement. I'll pick up a modem sometime this week, and install it on my next trip up to Vermont. My hope is that I can keep the PC running Linux continuously, programming it to fire up the (soon to appear) Linux data-aq during the night, reduce the data from images to star lists in the morning, and finally transfer the star lists over the phone to a machine down here in New Jersey.
Jure Skvarc
SKVARC@eros.ijs.si
Dec 1, 1996
This week I downloaded images Chris Albertson put on his FTP server. I have remark or two considering the format of images.
I suspect I am a bit responsible for the first thing. Namely, the BZERO value in the FITS header is set to 32768. This is nice if we want to have pixel values from 0 to 65535, what is the only natural and I also wrote this value in the Technical note 4. However, when I wanted to see these images using Fitsblink program, the CFITSIO library complained because of the overflow. The reason is that it supports only signed 16-bit integers which have range from -32768 to 32767. I made a workaround in the program, but I wonder shouldn't image files be more compatible with CFITSIO library. This would mean that we put BZERO back to 0. Of course, it is always possible to handle this problem in a program which reads images, but it introduces some complications. What do others think about this?
The other question I have is what is on the left and right side of the image. On both sides there is a strip of pixels with values which are very different from values of the sky and stars. If these are values of the covered pixels or anything like that I suggest to remove them or to put them in an extension of the FITS file. I think that now they only confuse programs which read such images.
Also I would like to announce Fitsblink 1.2. There are no functional changes from the previous version but I removed some bugs and made some code-cleaning (and new bugs, I suppose). Now you can read TASS images as they are (with BZERO=32768). It is also easier to download sources because they are separated from test images. Along with binary versions for HPUX and linux they can be found on the locations: http://kronos.ijs.si/~jure/fitsblink/fitsblink.html and http://www2.ijs.si/~skvarc/fitsblink/fitsblink.html
I will appreciate bug reports, suggestions and other remarks to my email. Thank you.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 1, 1996
> This week I downloaded images Chris Albertson put on his FTP server. I have > remark or two considering the format of images. > > I suspect I am a bit responsible for the first thing. Namely, the BZERO > value in the FITS header is set to 32768. This is nice if we want to have > pixel values from 0 to 65535, what is the only natural and I also wrote > this value in the Technical note 4. However, when I wanted to see these > images using Fitsblink program, the CFITSIO library complained because of the > overflow. The reason is that it supports only signed 16-bit integers which > have range from -32768 to 32767. I made a workaround in the program, but I > wonder shouldn't image files be more compatible with CFITSIO library. This > would mean that we put BZERO back to 0. Of course, it is always possible to > handle this problem in a program which reads images, but it introduces some > complications. What do others think about this?Well, the problem here is that the CFITSIO library is bugged: it is not 100% compatible with the FITS standard in that the standard does not disallow the use of data with 16 significand bits (instead of 15): the conversion we do (using a BZERO equal to 32768) is perfectly all right for the FITS standard. So, rather than throw away one bit in every pixel we have, let's correct the CFITSIO library, I say.
> The other question I have is what is on the left and right side of the > image. On both sides there is a strip of pixels with values which are very > different from values of the sky and stars. If these are values of the > covered pixels or anything like that I suggest to remove them or to put them > in an extension of the FITS file. I think that now they only confuse > programs which read such images.Sorry, but I disagree: the files generated by the data acquisition program (tm3get10) are *raw* data files, and as such contain *all* the acquired data. Sure enough, those files were not designed to be seen. These strips of pixels (some of which are not pixels at all, but temperature, voltage or current measurements) are to be used during the image reduction steps, i.e.: estimation of an appropriate dark line (this requires the temperature data and may make use of the blind pixels strips left and right of the image), substraction of said dark line, division by a standard gain line (i.e.: "flat fielding"), removal of damaged lines (this is where voltage measurements are useful: data taken with the nominal +5 volts down to +3 volts are pretty much useless), etc.
A fairly complete description of these strips of pixels is to be found in the data acquisition program's documentation, i.e.: tm3get10.htm, available at Michael's TASS home page on the WEB. If you can't get it from there, please tell me: I'll email a copy of that doc to you.
Naturally enough, these strips of non-imaging pixels should be removed after that stage, and an appropriate temperature entry made in the FITS header instead, giving the average temperature and its standard deviation (both in celsius). That work, however, should not be done by the data acquisition program. It is the job of the image reduction program to transform these raw data files into clean image files. Then the data reduction program should extract light source coordinate files from these clean image files, etc.
> Also I would like to announce Fitsblink 1.2. There are no functional > changes from the previous version but I removed some bugs and made some > code-cleaning (and new bugs, I suppose). Now you can read TASS images > as they are (with BZERO=32768). It is also easier to download sources > because they are separated from test images. Along with binary versions for > HPUX and linux they can be found on the locations: > http://kronos.ijs.si/~jure/fitsblink/fitsblink.html and > http://www2.ijs.si/~skvarc/fitsblink/fitsblink.htmlThanks for a nice program!
Chris Albertson
chrisa@wavenet.com
Dec 1, 1996
> This week I downloaded images Chris Albertson put on his FTP server. I have > remark or two considering the format of images. > > I suspect I am a bit responsible for the first thing. Namely, the BZERO > value in the FITS header is set to 32768. This is nice if we want to have > pixel values from 0 to 65535, what is the only natural and I also wrote > this value in the Technical note 4. However, when I wanted to see these > images using Fitsblink program, the CFITSIO library complained because of the > overflow. The reason is that it supports only signed 16-bit integers which > have range from -32768 to 32767.Read on, I think I agree with you. BUT...
I just pulled out my copy of the FITS standard. I think the FITS files written by Norman's program are technically correct by the standard. Also by the standard I think the CFITSIO library must be in error. Here is my reasoning:
The software (fits reader) I use does the correct thing. If BZERO and BSCALE are present and BSCALE is non-zero it returns type floats. Is there a way to get CFITSIO to give you floats? Using floats is the best approach because (as you know) it is possible for a valid FITS file to overflow a 16-bit signed int.
All that said, I see no reason not to set BZERO = 0.0 or to simply remove the B* header lines completely. From my point of view it make no difference what the B* lines are set to.
I do think the array should continue to contain ADU values exactly as they came off the ADC chip.
> The other question I have is what is on the left and right side of the > image. On both sides there is a strip of pixels with values which are very > different from values of the sky and stars. If these are values of the > covered pixels or anything like that I suggest to remove them or to put them > in an extension of the FITS file. I think that now they only confuse > programs which read such images.
Joe Ronchetti
jronchetti@earthlink.net
Dec 1, 1996
When I convert fits values to true pixel values I get a range from 0 to 65000 or so. The majority of pixels seem to be between 9000 and 14000. What is the lowest value to consider above dark current and what is the highest to consider below hot pixels and such?
Chris Albertson
chrisa@wavenet.com
Dec 1, 1996
Joe Ronchetti brings up an interesting question.
At first I thought that the range 0..65000 came from the fact that I was including the "edges", the peudo-pixels along the edges of the raw data files that contain things like masked cells and instrumentation values.
It does seem very odd that the (12 or 14 bit I forget) wide ADC can generate ADU values with such a wide range (0..65K should require 16bits)
But no, I can find very small subsections of the image that contain the full 0..65K range. For example, in 30t0337_269 there are several bright stars that are saterated. These stars come right up from the background and then contain a "hole" in the middle. The hole is a bunch of zero ADU values in the center of the star. Typicaly the 65K ADU values are directly adjacent to the 0 ADU values. I think it may be caused by the ADC chip wrapping when it is fed a voltage it can not handle. I am not an expert on how this process works.
Here is an example: The index numbers assume the first cell = 1 (not 0) My first step on reading the file is to convert to floating point. It avoids numeric problems. (taken from 30t0337_269)
231 232 233 234 235 236 237 238 239 240 241
584 1.4E4 1.4E4 1.6E4 1.8E4 2.2E4 0. 2.5E4 2.1E4 1.7E4 1.5E4 1.4E4
583 1.4E4 1.5E4 1.7E4 2.2E4 3.1E4 0. 3.7E4 2.7E4 2.0E4 1.6E4 1.4E4
582 1.4E4 1.6E4 1.9E4 2.9E4 4.8E4 0. 5.3E4 3.3E4 2.2E4 1.7E4 1.5E4
581 1.4E4 1.6E4 2.1E4 3.4E4 6.3E4 0. 6.6E4 3.9E4 2.4E4 1.7E4 1.5E4
580 1.4E4 1.6E4 2.2E4 3.7E4 0. 0. 6.6E4 4.0E4 2.4E4 1.7E4 1.5E4
579 1.4E4 1.6E4 2.2E4 3.8E4 0. 0. 6.6E4 4.0E4 2.4E4 1.7E4 1.5E4
578 1.4E4 1.6E4 2.2E4 3.7E4 0. 0. 6.2E4 3.6E4 2.2E4 1.7E4 1.5E4
577 1.4E4 1.6E4 2.0E4 3.3E4 6.0E4 0. 5.1E4 3.1E4 2.1E4 1.6E4 1.5E4
576 1.4E4 1.5E4 1.8E4 2.6E4 4.1E4 5.2E4 3.7E4 2.5E4 1.9E4 1.6E4 1.4E4
575 1.3E4 1.4E4 1.6E4 1.9E4 2.5E4 2.9E4 2.6E4 2.1E4 1.7E4 1.5E4 1.4E4
574 1.3E4 1.4E4 1.4E4 1.6E4 1.8E4 1.9E4 1.9E4 1.7E4 1.5E4 1.4E4 1.4E4
Anyone know how these high valued pixels get into the file?
These are not the normal case. A histogram of the image portion of the file shows what one would expect. A peak right at the sky background value (about 12k) then dropping smoothly to near zero at about 32K but extending flat from there to 65K. The few ADU=0 points are to few to show up on the graph.
This is right in line with Joe Ronchetti's results. The vast majority of the pixels are in the range between 9000..14000. These are easy to understand the "interesting" ones to me are those in the above table.
Joe Ronchetti
jronchetti@earthlink.net
Dec 1, 1996
How do I convert the crval1 and crval2 to dec and ra?
Chris Albertson
chrisa@wavenet.com
Dec 1, 1996
For reference here is part of a header:
CTYPE1 = 'DEC'
CRPIX1 = 1.0
CRVAL1 = 0.0000
CDELT1 = 0.0038197
CROTA1 = 0.0
CTYPE2 = 'RA'
CRPIX2 = 1.0
CRVAL2 = 344.166659
CDELT2 = 0.0038197
CROTA2 = 0.0
It says,
(CTYPE1) "along dimension 1 units are DEC"
(CRPIX1) Reference pixel is pixel #1 (the first)
(CRVAL1) The reference pixel is looking at DEC=0.0000
(CDELT1) Each pixel after the reference is looking
0.0038197 DEC units farther
The header does not say it, but the units in the DEC direction
are decimal degrees.
If you are looking at the files I posted on logicon.com I think the value in CRVAL1 is wrong. I think norman may have fixed this. In real life, Pixel 1 is not looking at 0.0 (the equator) as the header implies, pixel 400 is looking there. Pixel #1 is realy looking at about -1.5 DEC
The header above says the image is above the equator but in reality it is centered on the equator.
Dimension 2 is in the RA direction. Pixel #1 is looking at RA=344.166659 (in decimal degrees) each pixel looks 0.0038197 degrees farther in RA.
The CROTA1/CROTA2 lines say the image is not rotated or skewed.
If you need to convert from RA in degrees to RA in HH:MM:SS it is easy. Just remember that there are 360 degrees or 24 hours in a circle.
Tom Droege
droege@fnal.gov
Dec 1, 1996
>But no, I can find very small subsections of the image that contain the >full 0..65K range. For example, in 30t0337_269 there are several bright >stars that are saterated. These stars come right up from the background >and then contain a "hole" in the middle. The hole is a bunch of zero >ADU >values in the center of the star. Typicaly the 65K ADU values are >directly >adjacent to the 0 ADU values. I think it may be caused by the ADC chip >wrapping when it is fed a voltage it can not handle. I am not an expert >on how this process works.Don't blame the poor ADC, it works very nicely. Blame my electronics. Or better still blame the CCD chip as that is where the problem arises. I will have to think about how to cure this in the next version, but for now you all will have to do it in software. This is of course why we have software instead of doing everything in software. Software is so felxible you know. ;^)
Here is how the problem arises. We clear the charge from the output capicitor in the CCD with the reset pulse. We now sample the voltage on the output capicitor in the CCD and store it in Hold circuit #1. The charge in the next cell is now shifted into the output capicator and we measure its charge and store it in hold circuit #2. An operational amplifier now takes the difference between the voltage on hold circuit #2 and hold circuit #1. This voltage (after amplification) is sent to the ADC. A bias voltage sets the dark difference so that there is about 8000 counts (out of 65000) for dark. This value normally assures that there are no negative differences that the ADC would see as below zero and thus read as zero, losing the value.
When a very bright light shines on the CCD, it does really funny things. The point in time where the reset value is taken becomes full scale. It just does not reset properly. This independant of how hard you drive it or how long, etc.. I think the output circuit is light sensitive. It is worse than that. The reset value can be larger than the light value under these conditions. Too complicated to draw what happens, but I have looked at it many times on a scope. When this happens, the "dark" (sample #1) sample becomes larger than the light (sample #2) and the difference is negative. The ADC properly converts this as zero.
While I can imagine "fixing" this in hardware, it is really a saturated chip problem. I suppose I could sense this and force the ADC to full scale, but probably this is better done in software, what do you all think?? There is always an "edge" problem in hardware which might be hard to cure.
So for now, a zero in the center of full scale readings should be set to full scale. Note that you have already lost calibration when the value goes above about 40,000 counts. 8000 counts is zero. 80,000 electrons divided by 2.5 electrons per ADU gives 32000 counts for the true linear range. Readings above this require bleeding between CCD cells, or charge in a well more that the stated full well capacity. Note that as I understand it, the full well capacity stated on the data sheet is really the "linear full well capacity". Above that value there still may be charge trapped but the voltage generated is no longer a linear function of the light.
I have been less active than usual here because I have been busy with other things. I am now officially "half" retired. I have hired a full time technician to help me at home in this work. The addition to my house is nearly complete, and I have actually swam in the pool. So my new life is starting! I am still trying to get this mail list switched over to Glenn who has agreed to manage it. I just haven't succeeded in attracting the attention of Dave Vrona at wwa.com. No wonder, he is the president of wwa.com and is living the exciting life of being the president of a hot start up. I lived that life once (a company called Applied Logic). We were going to manufacture "computer power" in central stagions much like electric utilitys. You should read the prospectus. By now "time sharing" was gong to be a multi-billion dollar industry. Goes to show you what stock analysts know. Think about Applied Logic when you are tempted to invest in Netscape or ... We had 350 employees at one time.
Jure Skvarc
SKVARC@eros.ijs.si
Dec 2, 1996
Thanks to Norman and Chris I see that I was in error when I assumed that those FITS files were "final version" i. e. with dark frame subtracted etc. However, I still think it may be useful to put somewhere in the FITS header information where is the "sky area" in the image.
As for CFITSIO, I don't think it does anything wrong. When it stores pixel values to memory it just checks if everything is OK and short int format I use to read values has range -32768 to 32767. But it is still possible to read data with BITPIX=16 as 32-bit integers or floating point numbers. Of course, you either spend two times more space to store such images or you have to copy data to 16-bit array later. I guess I will probably go for 32 bit integers at expense of doubling memory consumption.
Joe Ronchetti
jronchetti@earthlink.net
Dec 2, 1996
I need to know how to come up with accurate dark values to subtract from each pixel.
Glenn Gombert
ggombert@mail.erinet.com
Dec 5, 1996
I just downloaded an GREAT Windows95 software package for doing image analysis under Windows95 from the Automated Telescope Facility at the University of IOWA. It has a full featured set of routines for doing such things as SuperNova Searches, looking for new variable stars, and much, much more! Check it out at:
"http://www-astro.physics.uiowa.edu/software.html"
The package takes up 6.5 megabyte file in compressed PKZIP format and takes a LONG time to download so be prepared to wait a while...
The information was supplied courtsey of Dennis diCicco the Senior Editor of CCD Astronomy, they are going to do a review of the package in the next issue of CCDA.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 5, 1996
I have placed a new DOS data acquisition program on the "incoming" ftp directory of "storm.fnal.gov", here are the files to be found there:
- tm3get11.exe the DOS executable as compiled by WATCOM C/C++ 10.6
(and let me tell you that this compiler is no joy to use)
- tm3get11.c the main source file
- tass_dos.1 the first subsidiary source file, #included in tm3get11.c
- tass_dos.2 the second subsidiary source file, #included in tm3get11.c
- tm3get11.htm the documentation - it has a change history near the
beginning, just after the introduction - Please, read it
if you want to use tm3get11 or the FITS files it produces.
- tass_mak.wat a makefile for WATCOM C/C++
- tass_lnk.wat a link file for WATCOM C/C++
- tass_mak.bor a makefile for Borland C/C++
- tass_lnk.bor a link file for Borland C/C++
- tass.rc an *example* configuration file for tm3get11
I'm sorry to say so, but the last version of tm3get10 was bugged - its stack was WATCOM's default size, i.e.: 2 Kb and it overflew whenever closing a set of files after opening the next set, with the usual strange behavior due to a stack overflow... I've since changed the stack allocation to 16 Kb, which is quite enough.
Tm3get11 has a pair of new features: its FITS headers are now clearly commented and there is a way to automatically add your own entries to these headers, for instance to indicate what filter you used on this particular CCD, etc.
Tm3get11 is also more clearly modular, with all its DOS dependencies localized within a pair of files, i.e.: tass_dos.1 and tass_dos.2 (what else?). I've been quite ill, hence my productivity has been low, which may or may not explain both the bug I left in the last tm3get10 and the delay in delivering a Linux version of tm3get11. Rest assured that I'm working on it with all of (what's left of) my energies. And yes, it will be the same tm3get11 that you now see running under DOS, but running (much more smoothly, I hope) under that much better of OS'es: Linux.
I'm too tired to keep working tonight, so bye for now!
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 6, 1996
Nick asked some good questions whose answers might interest some more of us. So, here are my answers: Dear Nick, you asked:
> I am setting linux up on the second drive of the tass data collection > computer. I was planning to use redhat linux. What version kernel are you > running, and are you using the new release of the XFree86 (I think it is > 3.2)?I am not too sure for the Linux kernel: I'm running DOS on the same machine right now and can't check without loosing my telecom access. I think it is kernel 1.3.0 or some such number between 1.3.0 and 1.3.21 (I'm sure it is less than 1.3.21). XFree86: I am using release 3.2, 'coz previous releases didn't work with my ATI Graphics Expression video card.
> Can you let me know when you might be ready for someone to alpha test > your code? I would like to look it over to see how it might integrate > into the remote control approach we are setting up at the lab.I'll tell everybody on the list as soon as it stops freezing & crashing my Linux box, promised! But you might prefer to tell me how you intend to go about providing access to both the FITS data files (not in real-time, but maybe nearly so, as one is produced every few minutes by each camera in a TASS triplet) and the real-time display (further info on that further down in this message).
> (I want to > collect the rough image into a periodically updated graphics file, and > send it to a heavy duty httpd server that will be accessable to the > outside world. (We are setting up the apl observatory home page to have > outside access). The observatory had two relatively low power computers > (486) one collecting data, the other managing a limited use homepage. The > home page server can run a script to convert a tass collected rough image > (4 bit pixel) into a gif or jpeg file, and then send it to the sparc 20 > that the lab uses to manage its outside web pages. I need to see how your > program sets up the image display so I can taylor the web server routines.Well, what tm3get11 does is this: as soon as it receives a completed (800 pixels) line from the TASS device driver, it copies it to the currently open FITS file (or files when in the overlap area between two successive files). Then it computes a one-dimensional histogram of the image pixels in that line, in order to find its median, which it then divides by 16, discarding 4 LS bits. Then it similarly discards the 4 least significand bits from all image pixels to get 12-bits pixel values, substracts the above calculated median (replacing negative results with 0) and replaces all results above 255 with 255, thus getting 8-bits pixel values. It then uses a look-up table (with 256 entries) to find the 4-bits approximate log of these pixel values and uses these logs to update the display. So, in order to satisfy your desire to make a fast display available for downloading, I may copy any of the following data to a Linux named pipe (or some similar device - I think streamio might also be appropriate - I'm open to suggestions here) for use by your programs:
I think the first one is redundant as new FITS files become available every few minutes. The second one is not too useful, I think. The third one is half the size of the raw data and might be nice for gif or jpeg files (I don't recommend the latter: it's a slow compression method and I have seen it generate a lot of artifacts on stellar fields - you might instead consider using the new "free" image file standard that is currently taking the WEB by storm, it compresses well and doesn't produce the obnoxious artifacts that plague jpeg). The last one is already four times smaller than the raw data, with two pixels per byte. Even for this format I would recommend avoiding jpeg, though.
So, what do you want tm3get11 to do for you?
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 6, 1996
Norman,
I agree with your suggestion about JPEG. If you save 4 bit logs (what you see on the screen), that may be sufficient for determining if everything is working OK. (I can live with 8 bit or 4 bit). I like the idea of limiting the file size at the data collection machine. Since I have a second computer that is maintaining the home page, it will retrieve the latest file from the data collection machine. I would like to keep it as low overhead as possible.
One idea (just for discussion). If you are creating the 4 bit per pixel, 2 pixel per byte output from the data collection machine, you can output that to a sockets interface that will send it via the lan to the data collection computer. The matching routine will collect the data stream and output a series of gif files and home page command signals (either by semaphore files or by mail box). The home page will use the latest display file in its update to the user monitoring the system. The output pipe could have as a format:
time stamp, line number, temperature 1, temperature 2, 4-bit encoded pixels (newline)
The receiver program will take the pixel data, add it with previously collected data, and when 10 or 20 seconds goes by, convert it to GIF, output the file, and then take the time stamp, line number, temperature 1 and 2 and output that to a text file. The home page will have a task that is monitoring for the new text file, read it, output the data, and direct the GIF to the user currently logged into the home page (refreshing the screen automatically).
This way, the home page will have access to the temperature values, able to detect problems, issue shut down to 12 volts power supply if necessary, send out emergency e-mail, without interfering with the data collection. If it becomes overloaded, it just will have a data dropout.
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 6, 1996
Chris Albertson wrote:
> Well, I said a conversion program would be easy to write... > > Should anyone want to periodically scan a directory for new *.fts files, > convert those files to *.gif files then move both the gif and fts files > to new directories, the following will do that for you. It should run > on any UNIX or Linux system. Most systems will already have the > required > utilities installed. If not they are easy to get off the 'net. >Most if not all of the utilities are standard with linux (redhad edition and slackware are my two data points). I planned to do something simpler. Since the raw pixels are available (downloaded from the data aquisition system as 4 bit per pixel data, I can use a library to create a GIF file. I think we are talking a few seconds (I will have to rig a set of test data to develop and check the routine. I would like the data aquisition computer to spend all of its time doing the data collection, and leave the re-format to the home page server.
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 6, 1996
Chris Albertson wrote:
> I'd also extend this Idea to the UNIX real-time display. Make > it a seperate non-real-time program. It would accept pixels from > tm3get11 scale and display them on an X11 screen The display program > should be the one to scale the pixels to the properties of the display > device and the user's preference. This removes any X11 display overhead > from your real-time program. > > If you have not yet written the UNIX display code, perhaps you could > skip it and simply blast the pixels out a named pipe or socket. This > would be much easyer to do then any calls to actualy draw thoses pixels. > > Now you need another program that reads from a socket or named pipe and > does whatever it wants with those pixels, converts them to low-res GIF, > or writes them to an X-display, or does some kind of real-time comet or > supernova hunting or whatever... In any case I think the pixel scaling > should be done by the end that reads the pipe. BTW I would have made > the > FITS file writer just another one of those pipe readers too. > > .... > > Then a GIF writer would look a lot like a FITS writer: they both just > read from a pipe and write to disk. An SVGA displayer would look like > and X11 displayer, they both just read from a pipe and write to thier > display. With sockets, any of these "readers" could run on any > computer on the network that understands sockets, even Win95, Sun, > or Linux. > > Another advantage. When the Mk IV comes out you only have to redo the > part that reads from the camera and stuffs pixels into the pipe.
Chris,
This all makes sense. If we are piping the 4 bit pixels to a socket interface, then the slow data aquisition computer can just pipe the data to the faster X windows display system. If you are displaying it via a slow modem link on that fast X-11 host, then the speed differential will not effect (or should not effect) the slower data aquisition computer. It would fit in nicely with the httpd approach of providing netscape access to the latest GIF image from the telescope.
We will see if nfs access to the fits files has any effect on data aquisition.
Since we do not have a high speed computer taking the data, I was not planning on making the FITS files available until after the observing session. That also depends on the APL network management people allowing us storage for off-site ftp access. (We think that will happen, but they will not give their blessing until they see the whole system).
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 6, 1996
Said Chris, replying to Nick about the use of separate programs to transform FITS files (or 4-bits display files) into downloadable GIF files:
> I'd also extend this Idea to the UNIX real-time display. Make > it a seperate non-real-time program. It would accept pixels from > tm3get11 scale and display them on an X11 screen The display program > should be the one to scale the pixels to the properties of the display > device and the user's preference. This removes any X11 display overhead > from your real-time program.Well, yes and no... I'll have to explain the architecture of the Linux implementation of tm3get11 so we can start talking about one and the same thing, I guess. See further down in this message.
> If you have not yet written the UNIX display code, perhaps you could > skip it and simply blast the pixels out a named pipe or socket. This > would be much easyer to do then any calls to actualy draw thoses pixels.Well, possibly, but I'm not too sure. The DOS version of tm3get11 already contains a lot of display code (now neatly separated into a pair of DOS-only included files: tass_dos.1 and tass_dos.2), so writing the corresponding interface for X11 would not be that much trouble - the interface is pretty clean, anyway, and there is a subset of X11 that's fairly easy to handle too (with fixed-size windows without mousy interface and similar restrictions).
> Now you need another program that reads from a socket or named pipe and > does whatever it wants with those pixels, converts them to low-res GIF, > or writes them to an X-display, or does some kind of real-time comet or > supernova hunting or whatever... In any case I think the pixel scaling > should be done by the end that reads the pipe. BTW I would have made > the FITS file writer just another one of those pipe readers too.Well, I see we really need to share an identical view of tm3get11-under-Linux, so here goes its architecture:
> Well, here I am telling you what to do when you are doing all the work. > But if the program where not monolithic some of us could contribute a > "pipe reader" or two and all you'd have to do is read data from the > hardware and stuff the pipe.As you can see, it is not that monolithic... ;^) If there is something to be gained by separating the X11 real-time display code (which I'm not sure there is), I could do that, and if someone out there is interested in doing that part, I would readily accept such help (but I get to specify the interface data format - and I am known to be a buff of packetizing, so be warned: it would look like packetized message passing via a named pipe or whatever similar stuff is available under Linux).
Much more interesting would be to separate a possible remote access facility from the bulk of tm3get11, i.e.: some separate program that would accept remote requests, translate them into pseudo keypresses & insert them within the stream on keypresses coming from the real keyboard to tm3get11 (on one side) and send the 4-bits image lines and textual data for real-time display to both the X11 driver and the appropriate remote display facility...
Such a program would relieve tm3get11 from all its user interaction parts and thus considerably simplify its code. I have written a similar program a few years ago, when XENIX was all the Unix that PC owners could (quite expensively) enjoy... 't was fun writing, but then there was no image display facility, no X11 to interface to, etc... Much easier than this job! Although the remote control facilities were PC's running a special TSR program under DOS that disguised them as VT100 terminals for XENIX purposes while letting them run whatever program their owners wanted (for DOS purposes), and that TSR was indeed quite a challenge to write. ;^)
> Then a GIF writer would look a lot like a FITS writer: they both just > read from a pipe and write to disk. An SVGA displayer would look like > and X11 displayer, they both just read from a pipe and write to thier > display. With sockets, any of these "readers" could run on any > computer on the network that understands sockets, even Win95, Sun, > or Linux.And tm3get11 becomes a TASS server program. As I wrote above, I would prefer to cut neatly all user interactions from such a server and place them in a separate program (that would process both remote operations and X11/SVGAlib displaying and local keyboard input, exchanging only messages and data with the tm3get11 server) - any takers for such a "user interactions" program? And by the way, it would have to be able to run on a networkless Linux station, which means that (maybe regrettably) sockets would not do it (unless you know something I don't - a distinct probability, that!).
> Another advantage. When the Mk IV comes out you only have to redo the > part that reads from the camera and stuffs pixels into the pipe.No advantage on the current architecture, that: I just have to redo the part that poll the Control Card and maybe change 10 to 20 % of the device driver code to become Mark IV ready... ;^)
Chris Albertson
chrisa@wavenet.com
Dec 6, 1996
Thanks a lot for the explaination of Linux tm3get. rather then quote you I'll try to explain your design in my own words. See if I am correct. I'll try to explain it for non-UNIX types as this is going over the TASS mail list too.
Question for the reast of the group: Would the ability to easyly write "clients" as discussed below be a valueable feature?
So here goes a further refinment of my version of a Linux TASS system:
First by some magic of kernel hacking and device driver writing by Norman we will have a "special file" called "/dev/tassmk3" Now by the above magic this file is connected to the TASS camera hardware and a normal user program can open and read from it as if it were a normal file on a disk. (Close enough) Now when you read from the "special file" you get 3x800=2400 raw pixels of data at a time. This special file is esentualy an extension of the Linux OS.
Now here is where my design would differ sightly from Norman's plan. (It may seem like a large departure at first but at the end of this message I argue that it is not.) Instead of having tm3get11 directly read from the special file /dev/tassmk3, I'd write a "pixel server". A pixel server would read from /dev/tassmk3 and then supply those pixels to any number of clients via a socket interface. The pixel server would act as a "Y" or even an octopus supplying pixels to any number of "users" These users or "pixel clients" would read raw pixel data from thier socket and do whatever they whished with the data. I can see that there could be several usfull clients:
From a C programmer's point of view sockets are easy to use. Basicaly, when you create a socket you are given a "file descriptor". You do reads and writes on the file descriptor as if it where a file. So the "pixel clients" look a lot like programs that could otherwise directly read the special file /dev/tassmk3.
The pixel server simply acts as a multiplexor. It does no other processing. (Well maybe buffering a few lines would be a good idea.)
Now one more importent point about sockets. When you (a client) connect to a socket you say something very close to
connect (FD, SOCKNAME)
This is like a file open. FD is a file descriptor and SOCKETNAME is like a file name. But here is the good part: the syntax of SOCKETNAME is like this: "HOSTNAME:SOCKET". HOSTNAME can be any host on the the local Ethernet LAN or any host on the Internet. Of cource HOSTNAME could also be the name of the local machine. In that case the server and client would be co-hosted.
Sockets are not just a UNIX thing. MS Windows supports them and so does VAX/VMS and other OSes too. So those clients (#s 1 ~ 6) could run on anything, even Win95, Win NT, OS2, SunOS, you name it. So if some Microsoft fan realy wanted to see stars scroll by on his Win95 screen he could simply write a Win95 display client using the winsock.dll library. In reality, serveral clients would run on the same box as the server. If one of them became a CPU hog it could be moved ___without even a recompile___ to a second Linux machine on the local net.
Another thing about sockets is that unlike files, sockets are bi-directional. You can both read and write to them after they are connected. So a client, lets call it a "Status and Control Client" could have a simple way to send data (key strokes) back to the pixel server. The pixel server then would do who-knows- what with any incomming data. (Maybe make some iocntl calls on the device?)
Now some details: to establish a socket connection (I'll skip some parameters for now. See /usr/include/sys/socket.h and man pages for details)
There is some room for protocol design. Perhaps each client could be required to send an "ID String" first thing after the connect. Then the server would do a read after the accept. It would then know something about the client connected via that FD.
Now back to Norman's design. Because reading from a socket is very much like reading from a device file, his tm3get11 program could easly become a pixel client. I see only a few lines of code different from a tm3get11 that talks to the device directly vs. one that talks to a socket. Both are done with normal reads. But with the client/server achitechure other clients could be added that run along side tm3get11 and agment or even over time replace it's features.
Michael Richmond
richmond@astro.princeton.edu
Dec 7, 1996
Norman asked "What do you want tm3get11 do to?", so I'll give my response. It would appear that I'll be one of the less demanding users :-)
My desire is to have a single computer do all the following steps:
The idea is to have a remote site with an unattended computer. Therefore, image display isn't necessary, but the ability to start/stop the data acquisition program without pressing any keys _is_ necessary; complicated command-line switches, or interrupts, or signals, or anything would be okay by me for ocntrolling the data acquisition program.
To use Chris Albertson's terminology, I would like to have a "FITS server". Moreover, I'd like to have one that doesn't require a human to press keys (although, of course, it will be handy to be able to do that for commissioning or special purposes).
I wouldn't be surprised if each person with a camera wants to use it in a slightly different way. I hope that we don't put too many demands on our Gaffeur :-) But the impression I get on the tm3get11 design leads me to believe that it won't be hard to make the program do what I want, and I'll be able to modify it if necessary.
Chris Albertson
chrisa@wavenet.com
Dec 7, 1996
Michael,
Have you noticed the UNIX "crontab" feature? It is a feature built into every UNIX system. Basically it is a table containning a schedule and some program names. You fill in the table and then the system will automatically run the programs at the specified time.
The programs can be run one-shot or periodically with a period specified in anything from seconds to months. You should be able to do what you want without any extra work on Norman's part. One of the reasons for porting to Linux is access to system features like this one.
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 8, 1996
Chris Albertson wrote:
> Question for the rest of the group: Would the ability to > easyly write "clients" as discussed below be a valueable > feature? > ... > Now here is where my design would differ sightly from Norman's > plan. (It may seem like a large departure at first but at the > end of this message I argue that it is not.) > .... > > Now back to Norman's design. Because reading from a socket > is very much like reading from a device file, his tm3get11 > program could easly become a pixel client. I see only a > few lines of code different from a tm3get11 that talks to > the device directly vs. one that talks to a socket. Both > are done with normal reads. But with the client/server > achitechure other clients could be added that run along side > tm3get11 and agment or even over time replace it's features.I like your idea, and I think that you and Norman are thinking along the same lines (with your e-mail supplying much of the implementation details). Another advantage of this concept, is that the applications working off the socket interface can be hosted on lower power (cheap or free) computers. From the standpoint of the APL observatory, I can beg and borrow many old 486 computers that are being surplused at the Lab, and distribute the processing among them. There is a traffic overhead issue which may not be a big factor if we are able to do multicast of sockets rather than multiple socket broadcasts of the data.
> Hmmm. I wasn't aware that the dark current changed by so much. > I wonder if there can be differences from camera to camera, orThe data I have is all from one triplet. I see conciderable differances between the CCD chips. On order of 100 ADU. Night-to-night variation of same CCD about the same. The exact relationship between the values of the "edge" or "masked" pixels and the image pixels appears to depend on the chip.
> even depending on the settings. I've certainly noticed that > when I start running, it takes some time for the temperature of the > chip to equilibrate -- and the time it takes can vary quite a bit,What I am calling "dark current" is realy "dark curent plus bias curent" The Mk III adds about 8000 ADUs just to insure we do not get negative numbers. Much of the drift both short term and night to night may be due to drifts in the bias current. Either way this still needs to be subtracted out. As I said in a previous post it looks like this can be done to within 1 ADU at least for the test data I have.
To Michael and anyone else with a working camera,
If you would send some dark frames from your system (put lens cap on) I'll see if the method applies to other triplets. Take some data both early just after the temp stabilizes and late after in has been stable for some time. (Please modify the tass.rc file so your serial number is correct.)
> depending on the temperature setting. I'll have to look over a > set of long dark frames very carefully.What I do is average all image pixels one a scane line and plot the average vs. scan line number. See ftp://ftp.logicon.com/open/chris/dark.gif for an example plot.
Chris Albertson
chrisa@wavenet.com
Dec 8, 1996
> > Question for the reast of the group: Would the ability to > > easyly write "clients" as discussed below be a valueable > > feature? > > Chris, > > I like your idea, and I think that you and Norman are thinking along > the > same lines (with your e-mail supplying much of the implementation > details). Another advantage of this concept, is that the applications > working off the socket interface can be hosted on lower power (cheap > or > free) computers. From the standpoint of the APL observatory, I can beg > and borrow many old 486 computers that are being surplused at the Lab, > and distribute the processing among them. There is a traffic overhead > issue which may not be a big factor if we are able to do multicast of > sockets rather than multiple socket broadcasts of the data.Yes I left out some detail in my socket write-up. The "socket" call, which creates the socket allows you to choose a protocol "famaily" and a protocol. Basically the choises are UDP or TCP. UDP allows a broadcast where you send the packet once and all clients see it. TCP adds error checking and garenteed in order delivery but is point to point so you need to send the packet once per client. This is a design issue. With a bit of work you can have it both ways were the clients pick the protocol.
When you are on an _isolated_ Ethernet network trafic is not that big an issue. I would not dump TASS pixel data on your lab's LAN. Ethernet by nature is non-deterministic and is not so good for even soft-real-time unless it is very lightly loaded. If you are going to connect a bunch of 486s together for TASS. I would set up a seperate Ethernet and then stick two Ethernet cards in one of the 486s and let that machine route traffic between the TASS LAN and the lab's LAN. You gain a lot at the cost of only one extra Ethernet card. (A 486 with a few Ethernet cards and serial ports running Linux is the world's cheapest router)
If you have not guessed, this is what I've been doing at work lately: Sockets. The curent project uses loads of them on a few rooms full of computers. So I've learned how to use a hammer and now everything looks like a nail.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 10, 1996
Tm3get11 for Linux: discussion, architecture, specs.
OK, gang, I'm waiting for comments / offers to write some client program / suggestions for the socket interface specs / etc. There are quite a few question marks here above, let's see some answers!
Chris Albertson
chrisa@wavenet.com
Dec 10, 1996
First I don't mind people disagreeing with me. My thinking on this subject is evolving. In fact I've changed my mind on a few things. I expect I will again . The two main things I am thinking now are
Specific comments follow:
> 1) discussion: > - as Chris has aptly summarized, there are a number of advantages to a > tm3get11 implemented as a server (and accessed via a socket): it becomes > easy to provide newer/fancier client programs > (like GUIs or remote control > programs) and these clients need not be changed when > going from Mark 3 to > Mark 4 etc. Furthermore, while one person is implementing the server, > others can implement various clients inasmuch as the interface has been > clearly defined and fully frozen. > > - there is, however, one point on which I disagree strongly with Chris: to > my eyes, the tm3get11 server should not pass the raw data to clients via > the socket interface - it should create the raw FITS files and only pass > the preprocessed data with 4 bits per pixel > (maybe offer a choice between > 8 and 4 bits per pixel, but I don't know if that is a good idea) via the > socket interface, plus some textual data to be processed or displayed > (i.e.: the contents of the top 3 lines of text and the contents of the > focussing matrix). It should also accept commands from the socket > interface (both commands equivalent to the current keyboard commands and > new commands to control the Auxiliary Data In and Out ports).I think now that the "pixel server" should log the raw data itself also. because.
Now what should pass over the socket. IMO just enough data for the client but certainly not less then enough. Now that we all agree on that statement :-) let's see what it means.
> - why should the tm3get11 server not give direct access to the raw data > lines and why should it store them in the raw FITS files itself?I think you are mixing two subject. Who stores the raw pixels and who as access to them. Access is required for anything like client "c" above. Maybe not truely raw pixels but something like them. Perhaps with a crude dark curent corection applied or some such pre-processing applied.
> Quite simply, to guarantee complete uniformity of format for the raw FITS > files for all TASS installations. If that is not done by the server, > everybody and his sister will want to have its own client program storing > these raw files in his own favourite formatI agree. the server writes the raw FITS file. But it could also send out 16-bit pixels to those who need them for other reasons.
> (say, stripping the non-pixel > data off the file, for instance, or using a fancier header format with > angles specified - say - in radians or in hours instead of decimal > degrees). To put things bluntly: either I keep the format quite uniform > by providing the software that writes these files, or we will be unable > to exchange raw data files among ourselves in no time flat.I like to see the FITS files be uniform too but I hope we do not learn to depend on that uniformity. This would mean that a change would be impossable because it would break lots of data reduction code. A good FITS reader should be able to handle any odd FITS file thrown at it. For exampe Michael Richmond sent me some TASS files that are not at all like the ones tm3get writes. I was still able to make a average dark current vs. time plot with his files.
> - I'm sorry to have to say things this bluntly, but I have some experience > (not as much as Tom, but still over 22 years of it) working with teams of > both programmers and scientists (beeing one of both gangs) and one thing > I certainly learned: either someone freezes file formats and interfaces > by implementing them himself, or soon teamwork breaks down under the > weight of a multitude of slightly incompatible file formats and > interfaces. Hence, the tm3get11 server will write the raw FITS files. > Any program that must access the raw data will have to read these files.For a project as loosely organized as this one that is how they work: The first person to do something sets the standard. However the standards that last the longest as those that are robust and flexible. I think FITS helps us that way.
> 2) architecture: > > - Linux kernel modification > * I'm using Linux kernel 1.3.20 (is this what you wanted to know, Nick?) > * this part is written and undergoing tests right nowMy only concern here is that when new Linux kernel come out will they break your patch file? I assume the clock tick stuff has been stable for a long time. I'm hopingyou are not making yoursef a life-long job of updating the patch for every new kernel version.
> - /dev/tassmk3 device driver > * it is a character-based device > providing a one-line-at-a-time interface > (when using a single camera) or a > three-lines-at-a-time interface (when > using a full triplet) to the hardware for the server > * this part is also written and undergoing tests now > > - tm3get11 server program > * the server reads and processes the tass.rc configuration file > * it then sets-up the TASS triplet and starts > collecting data and storing it in raw FITS files > * it prepares the 4-bits per pixel display lines, > the 3 top lines of text and the focussing matrix data > * it provides access to these image and textual data via the socket > interface > * it also provides access to the programmable facilities of the device > driver (the Auxiliary Data In and Out ports) via the socket interface > * it is also controllable via the socket interface (update TEC and VCO > ADC's, choose one camera for focussing, start/stop focussing calcs, > clear/ramp-up the CCDs, start/stop storing data file, shut-down, etc.) > * I'm writing this part - luckily enough, I wasn't too advanced here and > do not have to throw away any of my code to make it a server > * THE SOCKET INTERFACE HAS STILL TO BE DEFINED (see further down) > > - X windows client program > * must be able to display images from one camera or from 3 cameras > simultaneously, together with the usual top 3 lines of textThat is one design. It could also be that a display client shows only one camera and you simply run three clients, one for each camera. After all some lucky person may happen to have three spare computers and want to use three CRTs one for each CCD. If I were to run a TASS system at my work we would have endless UNIX mmachines on which to run clients.
I am not arguing for any client design now. But I do think the server should not constrain the design of any clients. With X11 the display could be done in overlapping windows.
> * must be able to display the focussing matrix together with the image > from a single camera > * must provide the user with a full set of commands (to update the TEC > and VCO ADC's, to choose one camera for the focussing calculations, > to start/stop the focussing calculations, to clear/ramp-up the CCDs, > to start/stop storing raw data files, to shut-down the server, etc) > to be transmitted to the tm3get11 server program via the socket > interface > * who wants to write this part?I agree all the above requirements must be met. However I see no reason to require one client to do all of them. For example how about a display client, focus client, and control and status client?
> - telecontrol client program > * must build current/recent GIF or PGS files > (from the 4-bits image lines) > to be accessed by the WEB or FTP > * must build current/recent WEB or TEXT pages showing the top 3 lines of > text with (when appropriate) the contents of the focussing matrix > * must provide the remote user with commands similar to these provided > by the X windows client, plus commands to > download current/recent pages and image files > * who wants to write this part? > > - svgalib client program > * must be equivalent to the X windows client program > * may possibly not be necessary (would be useful on smaller machines - > like 486'es - without X windows server) > * who wants to write this part? > > 3) socket interface specifications: > - should the server provide access to more than one client at a time?This is easy to do. But you can run out of resources on a small mmachine. You could use an environment variable to define the upper limit. Yes I think you need more then one or the only client that could run would be some kind of "do everything" supper client. It would defeat the purpose of using a server.
> - should we use datagrams or streams?My idea is to let the client specify. There could be two sockets, the client connects with the one it wants.
First thing any client must do after connecting is write on the socket a "request for pixel data." In the request would be the number of bits per pixel and wether it wants image, "edge" or both types of pixels. Maybe alllso the type od scaling to be performed.
Version one of the server could limit the options the client may ask for to say only one option. Later versions could offer clients more.
> - what sockaddr (socket address) should we bind to the socket? > - what data format should we exchange across the socket interface, i.e.: > how do we packetize the data we must exchange? > - here is what the server must provide to the clients: > * one image line (768 pixels of 4 bits) with its camera number (0, 1 or 2) > and line number since beginning of night (long int)Or 8-bit or 16-bit data as requested by the client.
> * top 3 lines of text > * focussing data (5 lines of text) > * contents of the Auxiliary Data In port > - here is what the clients must provide to the server: > * data for the Auxiliary Data Out port > * user commands to update the TEC and VCO ADC's, to choose one camera > for the focussing calculations, to start/stop focussing calculations, > to clear/ramp-up the CCDs, to start/stop storing raw data files, to > shut-down the server, etcAlong with the "request for pixel data" would be commands for the above as well.
> OK, gang, I'm waiting for comments / offers to write some client program / > suggestions for the socket interface specs / etc. There are quite a few > question marks here above, let's see some answers!Well I think the first think to write is a "dummy server" one that always serves the same line of pixels. The next program would be a "dummy client" that just reads the pixels and dumps them in the old bit bucket. These two programs would define the interface and the dummy client would serve as a model for other clients. In fact it would mmake writing a new client easy.
Tom Droege
droege@fnal.gov
Dec 11, 1996
> - why should the tm3get11 server not give direct access to the raw data > lines and why should it store them in the raw FITS files itself? > Quite simply, to guarantee complete uniformity of format for the raw FITS > files for all TASS installations. If that is not done by the server, > everybody and his sister will want to have its own client program storing > these raw files in his own favourite format (say, stripping the non-pixel > data off the file, for instance, or using a fancier header format with > angles specified - say - in radians or in hours instead of decimal > degrees). To put things bluntly: either I keep the format quite uniform > by providing the software that writes these files, or we will be unable > to exchange raw data files among ourselves in no time flat.I pretty much agree with Norman. He who writes the software gets to set the standard. I would rather live with a fairly good standard than a lot of very good ones! ;^) On the other hand you all have an opportunity to persuade Norman to your way of thinking. I am sure that if we all get together and decide that we really want thins done in one particular way that Norman will listen.
Norman also listed a number of things ending with:
* who wants to write this part?
I have enough parts here and on order to make a few more single cameras. I will try to get a camera to anyone who needs one to work on software.
Glenn Gombert
ggombert@mail.erinet.com
Dec 12, 1996
I have been doing some more experimenting with the Automated Telescope Tool Kit from the University of IOWA and have tried several of the tools out on several of the TASS images taken here in Dayton several weeks ago. One of the first calibration steps to using the ATF Tools is to calculate the "WCS" (World Coordinate System) values that are inserted into the FITS header of the images that it is desired to use the tool kit with. The WCS coordinates appear to be a very power set of information that allows many sophisticated measurements to be made from sets of images that have the WCS coordinates added to the FITS image headers. The set of programs that I downloaded over the weekend operate under Windows95/NT . I am in the process of installing the same set of tools that operate under Linux.
The procedure that I used to get the RA and DEC of the images that had the WCS information added to the header was from Michael Richmonds automated astrometry server. Once the approximate RA and DEC of the each TASS image field center was entered into the header the "WCS" program in the AFTTool kit which detected (automatically) the stars in the FITS image and then extracted reference stares from the HGSC CDROM selected the 25 brightest stars from each image and performed a pattern match on the selected stars from the reference image and the HGSC. Once this was accomplished the WCS FITS header parameters were calculated and added to the FITS image header.
Adding WCS parameters to FITS images allow some sophisticated image analysis to be performed in an automated fashion, such as specifying certain variable stars which to produce light curves for and checking images taken during different nights to check for new stars that were present in one image and not another (SuperNova Searches, etc.). The reset of the tools in the "tool kit" depend on the WCS coordinates being added to the FITS image headers.
Also over the weekend I downloaded and installed the immatch (image matching package) that is an add-on to IRAF. The immatch package also has a number of functions to use and manipulate FITS format images that have the correct WCS parameters setup in the header. In short the "WCS parameters " appear to be very useful information to have in TASS images in order to automate the data reduction process that confronts the mountain of data that results from just one evenings operation (120 megabytes collected Saturday night / Sunday morning).
Maybe Michael Richmond can expound on the "philosophy" behind the World Coordinate System fields in FITS format images and how they might be fully utilized to enhance the automated data analysis of TASS images that we will want to pursue in the future. It might be a good thing to add to a future version of "tm3get" in the future to calculate and insert these parameters into FITS header images. This could also greatly help to compare images taken at different TASS locations / camera's as more and more data is collected once these parameters are added to all image headers.
Glenn Gombert
ggombert@mail.erinet.com
Dec 12, 1996
While surfing the WEB looking up "info" on the World Coordinate System I can across a very interesting program developed by the European Southern Observatory called "SkyCat" which has some very interesting features AND it has been ported to Linux! Below are some of the features that I copied from the SkyCat Home Page.
I am too tired tonight to download and install it but it looks VERY interesting...
------------------------------------------------
SkyCat is a tool that combines visualization of images and access to catalogs and archive data for astronomy.
See the SkyCat FAQ for more information.
SkyCat features
Open and visualize a variety of FITS images including support for World
Coordinate System (WCS),
interactive measurement of offsets and other standard visualization
functions (SAOimage-like);
Overlay and edit color graphic objects on the image, like `tagging'
sources with text, arrows, circles or other
graphic elements such as masks;
Display a world coordinates based grid over the image with the
resolution given in arcseconds.
Display a compass indicating where north and east are in the image,
based on the world coordinates
information.
PostScript color printing of the display (image + graphics);
Access and load an image from a network server of the Digitized Sky
Survey scans;
Access and load catalog information from a number of popular
astronomical catalogs like the HST Guide
Star Catalog and others;
Access local user catalogs.
Save catalog data locally.
Overlay catalog sources on an image;
Interact with Netscape to display more object information when
available (support for Mosaic may be
addded in a future release);
Access the observations catalog from the NTT, HST and CFHT Science
Archives;
Access to SIMBAD and NED both as name resolvers as well as for
information on known objects;
Retrieve preview and other compressed images and decompress them on the
fly.
Retrieve and plot tabular preview data for a selected object as an X/Y
graph.
Calculate, display and plot the center position, FWHM, angle and other
information for a selected
star/object.
Load compressed images in hcompress, gzip or unix compress format.
Access SkyCat features from a remote process via socket interface or Tk
"send".
Access image header and data (FITS format) via SysV shared memory or mmap.
Interact with WWW browser to access catalog documentation and other
documents.
Things we are working on
Save canvas graphics possibly as a fits extension on the image file.
User customizable symbols to "label" objects on the canvas.
Load lists of catalogs from sites (ESO, CADC, CDS, local) and and allow
user to select his/her preferred
default catalog list.
Save user preferences for look-up tables and other settings.
Add or edit WCS parameters for a given image.
Caveats
Astrometry: SkyCat handles the astrometric positions by translating pixel positions into equatorial coordinates (RA, DEC). This translation is based on WCS (World Coordinate System) FITS keywords that are included in the image header and that give the astrometric solution for the image. The accuracy of the solution varies because, in many cases, an approximation is used. Colors: If you are running netscape or some other color intensive applications before you start SkyCat, you may get some color flashing when you move the mouse in and out of the SkyCat image window. This is because SkyCat is using a private colormap to get enough colors to display the image. You can get around this problem by starting netscape with the "-ncols" option: for example:
netscape -ncols 60 &
You can also control how many colors SkyCat uses via the color dialog in the View menu.
About the software
SkyCat was developed by ESO's Data Management and Very Large Telescope (VLT) Project divisions with contributions from the Canadian Astronomical Data Center (CADC).
The tool was originally conceived as a demo of the capabilities of the class library that we are developing for the VLT.
This library consists so far of two major subsystems:
The Real-time Display (RTD) image classes with the RTD widget
(see The Messenger, 81, 1995), and
the Catalog Interface classes with the CatSelect widget.
These constitute the basic building blocks of SkyCat. Early demonstrations of the tool to users were received so enthusiastically that we decided to polish things up and produce the current version.
Java vs Tcl/Tk
Some users might wonder why we did this development in the Tcl/Tk environment rather than in Java. The main reason is the pragmatic need to get the functionality implemented in time for when the VLT comes on-line (mid 1997). However, great care has been taken to develop as much as possible in the short time with object oriented languages (C++, [incr Tcl]), having in mind that the future lies in tele-scripting rather in the distribution of binary code.
SkyCat mailing list
A mailing list has been setup to support a wide collaboration on the SkyCat/RTD/CatLib project. Many people have shown insterest in participating in such a venture. You can sign up with that list here. A hypermail archive is also available.
Distribution and support
The SkyCat binaries are freely available to any users who want to download and use the software at their own risk. Users to wish to modify the source code are should contact malbrech@eso.org.
SkyCat is available as an executable for these platforms:
Solaris-2.5 (SunOS-5.5) Solaris-2.4 (SunOS-5.4) SunOS-4.1.3 HP-UX-B.10.10 HP-UX-A.09.05 Linux (courtesy of S. Sani)
ESO will maintain the Sun Solaris and HP versions in the longer term. They are the platforms on which VLT software will run. ESO does not have the resources to port to and maintain SkyCat on any other platforms. We will be glad to redistribute any port that other people or groups may support but decline any responsibility for them.
Chris Albertson
chrisa@wavenet.com
Dec 13, 1996
As I said, I'd offer to write a client server socket interface for the Linux version of tm3get. I have started work on a prototype. The prototype just shows that all the required system calls work under Linux and do what is expected.
The prototype consists of two demo programs. Nothing TASS specific yet. My point was just to see how much effort it would take. The basic setup took only a few hours.
I have a stand alone server and a matching client. All they do is connect then send each other messages then quit.
The client looks for an environment variable called TM3GET_HOST. If present it will read the value associated with TM3GET_HOST and connect with a server running on that machine. If the environment variable is not present, the client will connect with a server running on the local machine.
Both server and client look (by use of getservbyname) at either the local /etc/services file to get the proper port number for the "tm3get" service or if NIS is set up and has a services map getservbyname() will look there.
The system uses Internet Domain sockets bound to a port defined in /etc/services. After doing some research I think we realy should use the TCP protocol. It would not be a lot of work to add UDP capabilty later. The problem with UDP that error checking, re-tries, and timeouts must all be handled be the client and servers. TCP makes this the responceability of the system. UDP is something we can add later (with no need to remove TCP) if required and the change will be transparent.
The above all seems to work. I need to add better error checking. Currently the software checks for an error after every system call but just prints the error and quits. Not what I call "production quality" I still need to test odd things like a connection between a Sun Workstation client and a Linux server accross the Internet but this "should" work. Yes it should be possible to control a TASS camera located anywere in the world from your home computer given enought bandwidth.
Still to be determined is the format of the data passed between client and server. The prototype simply sends a big binary buffer with an ASCII test message like "Hello server" or "Hello client" stuffed into it.
My proposel is that:
Code zero is reserved to indicate so-called "out of band" data where the scan line actually contains some kind of message of command from server to client. I am sure we will think of a use for this.
I think this is simple but allows for expansion later. At first the server would offer only one or two differnt choices of scan line type.
Chris Albertson
chrisa@wavenet.com
Dec 13, 1996
As they used to say on Monty Python, "And now for something completely different."
Please don't take this too seriously. This is intended as pure entertainment. After hearing about all the pros and cons of various proposed camera designs I think I have thought of a new type. Just wondering, is this a new idea and would it work?
A while back someone on this list described a drift scan camera he had built that used a mirror. The CCD camera was faceing backward to the mirror but offset from the mirror's optical path so as not to obstruct the incomming light. There was some optical distortion but nothing that software could not iron out later.
My idea is to extend this. Make a _ring_ of CCD chips a little larger then the mirror and one focal lenght away. An 18 or 24 inch mirror is expensive but if it could be shared by 20 of so CCD chips the total cost could be lower. With a mirror that large there is room for two concentric rings.
So here is a way to use 50 or so cheap CCD detectors to make the equivelent of that 2 square inch, multi-megapixel CCD we can not afford.
I think this design could be improved at little cost: Double the focal lenth of the mirror then place a small positive lens just in front of the CCD to bring the _efective_ focal lenght back down to what it was. This reduces the offset angle from the mirror's axis and (we hope) optical distortion. Lens tilt and off-axis shift could re-align the plane of focus back onto the chip's plane. (Those of you who do large format photography should be able to visulize this better.) Now one more improvement, don't double the mirror's focal lenght but keep the lenses. Now you can have a large diameter f/2.0 optical system made with a standard issue dobsonian mirror.
I got this idea while reading about "non imaging optics" It turns out there is a whole class of optics that compute functions other then the one to one mapping that camera and telescope optics perform. Now I was thinking that if no one were ever to look at an image but were to simply dump it into a computer for analysis there would be no need to make such nice distortion free images. The above is a very mild example. I think all you realy need to do is channel the light flux from the sky onto the detector in some understandable way.
John Cooper
JBCooper3@aol.com
Dec 13, 1996
> I think this design could be improved at little cost: Double > the focal lenth of the mirror then place a small positive lens > just in front of the CCD to bring the _efective_ focal lenght > back down to what it was. This reduces the offset angle from > the mirror's axis and (we hope) optical distortion. Lens tilt > and off-axis shift could re-align the plane of focus back onto > the chip's plane. (Those of you who do large format photography > should be able to visulize this better.) Now one more improvement, > don't double the mirror's focal lenght but keep the lenses. Now > you can have a large diameter f/2.0 optical system made with a > standard issue dobsonian mirror.It's always easier to gun down someone than to come up with an original idea. That said, here goes (out comes six-shooter)... Two main concerns (one of which can be eliminated by throwing money at the problem) are that the usable FOV may not be large enough to fit more that a few chips in (inlcuding packaging constraints); and the auxiliary lens proposal doesn't give f/2-type images, since the cone of light that each chip uses is smaller than f/2.
Gunning me down is welcome, I don't mind being proven incorrect at all (especially if it betters The Cause!)
Joe Ronchetti
jronchetti@earthlink.net
Dec 13, 1996
Has anyone named one star on any tass image? If so send me the image and identify the star.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 13, 1996
> Please don't take this too seriously. This is intended as pure > entertainment. After hearing about all the pros and cons of > various proposed camera designs I think I have thought of a > new type. Just wondering, is this a new idea and would it work?Some details should be ironed out of your design, but it is indeed a workable concept, and an interesting one at that - it would be perfect, for instance, for an amateur space telescope. It would also be great for an automated amateur telescope.
> My idea is to extend this. Make a _ring_ of CCD chips a little > larger then the mirror and one focal lenght away. An 18 or 24 > inch mirror is expensive but if it could be shared by 20 of so > CCD chips the total cost could be lower. With a mirror that large > there is room for two concentric rings.Well, it depends. The distortion on the outer ring would be much larger than on the inner ring, unless one used a slightly hyperbolic mirror - and these are more difficult to make (meaning more expensive) than parabolic ones or (cheaper still) spherical ones. Now, using a spherical mirror of f/16 or lesser aperture would still give an excellent image all around the ring, with quite acceptable distortion.
> So here is a way to use 50 or so cheap CCD detectors to make the > equivelent of that 2 square inch, multi-megapixel CCD we can not afford.It is indeed feasible, but there is a drawback: such a system would image a ring in the sky, with a huge blind spot at the center of the ring. When you want to image something, you have to offset the telescope so that what you want to see falls on one of the CCDs.
> I think this design could be improved at little cost: Double > the focal lenth of the mirror then place a small positive lens > just in front of the CCD to bring the _efective_ focal lenght > back down to what it was. This reduces the offset angle from > the mirror's axis and (we hope) optical distortion. Lens tilt > and off-axis shift could re-align the plane of focus back onto > the chip's plane. (Those of you who do large format photography > should be able to visulize this better.)This would also effectively multiply the price of the system by a large number. a small positive lens would magnify the distortion unless it was adequately corrected, i.e.: a small triplet - With a triplet, one could indeed choose the lenses in such a way as to largely compensate the distortion caused by the mirror offset.
> don't double the mirror's focal lenght but keep the lenses. Now > you can have a large diameter f/2.0 optical system made with a > standard issue dobsonian mirror.No, no, definitively no. A standard issue dobs mirror (say an 18 inches f/4.5 first surface parabolic) is already a tad expensive (to put it mildly), but it is also a very highly distorting optical system when used off-axis. Lenses to correct the gross distortion and at the same time reduce the focal length would cost an arm and a leg - each CCD would have to be fitted with such lenses, each set costing at least as much as the standard issue mirror. Your off-center ring telescope cannot be inexpensive unless one uses very small apertures (f/16 or smaller) and no correcting optics.
> I got this idea while reading about "non imaging optics" It turns > out there is a whole class of optics that compute functions other > then the one to one mapping that camera and telescope optics perform.Right, but the one to many mappings provided by imperfect optics such as the one you propose are useless unless one has a lot of computer power to deconvolve the images thus obtained. Think in terms of one Pentium per CCD chip. The simpler the mapping (i.e.: the nearer to "one to one"), the cheaper and faster it gets to extract usable image data. The Hubble mirror saga is a good case in point.
> Now I was thinking that if no one were ever to look at an image but > were to simply dump it into a computer for analysis there would be > no need to make such nice distortion free images. The above is a > very mild example. I think all you realy need to do is channel the > light flux from the sky onto the detector in some understandable way.As I said, the simpler the mapping, the cheaper the calculations required to make the data usable.
Another drawback of your system is that it can only be used in stare mode, or in stare-and-skip mode, unless you have found a way to make sure that all the CCD chips are perfectly parallel to each other.
All in all, I think your idea is certainly worth investigating in more detail, but in order to keep its price down, one would have to restrict oneself to non-corrected single mirrors, preferably of small numeric aperture (an 18" f/16 mirror has a focal length of 288" or 24 feet (about 8 meters), talk of a long tube!
Finally, there is a small remark that ought to be stated: at best, one could build a third of a ring around the mirror, because the remainder of the ring would obstruct the optical path for the CCDs on the first third.
Joe Ronchetti
jronchetti@earthlink.net
Dec 13, 1996
Why place the chips in a ring? Instead a grid of chips collecting light from the center of the mirror would give a larger image of the selected sky. Some data would be lost between the chips but overall I think it would be acceptable.(Correct me if I'm missing something.)
Bob Goff
goffaxe@azstarnet.com
Dec 13, 1996
I'd like to throw my hat into the ring. How about an offset matrix like they have at the ccdti at kitt peak.? On axis, of course ;}} ( note the beard )
Chris Albertson
chrisa@wavenet.com
Dec 13, 1996
> > Please don't take this too seriously. This is intended as pure > > It is indeed feasible, but there is a drawback: such a system would image a > ring in the sky, with a huge blind spot at the center of the ring.I was thinking of a TASS style survey device. The goal was to maximize the square inches of both silicon and glass
> No, no, definitively no. A standard issue dobs mirror (say an 18 inches > f/4.5 first surface parabolic) is already a tad expensive (to put it mildly),Realy, I figured just a few hundred dollars. Anyway I'll make a one CCD version someday out of my CB245 and a maybe 6" mirror. It would only cost me the price of a mirror. With luck we'll have analysis software long before I set up the hardware.
> Right, but the one to many mappings provided by imperfect optics such as > the one you propose are useless unless one has a lot of computer power to > deconvolve the images thus obtained. Think in terms of one Pentium per CCD > chip.We have huge amounts of computting power going to waste at my office. There are multi-CPU UltraSPARC and SGI boxes sitting idle 12 hours a day (or night) Just one of those UltraSPARC CPUs is like 5 or 8 Pentiums. Has 1/2 a gig of RAM too. 6 foot tall DEC Alpha servers upstairs along with VAXes. Well you get the point. Hundreds of mega- FLOPS setting Idle all over the country and mostly at night. Yes I know, way hard to put this to use.
> As I said, the simpler the mapping, the cheaper the calculations required to > make the data usable. >I read about one optical system that projects a Fourier transform. The math was way over my head and I don't remember the details. It was part of a device used by the Air Force to process Synthetic Apiture Radar data. We were building a digital replacement for it.
> Finally, there is a small remark that ought to be stated: at best, one could > build a third of a ring around the mirror, because the remainder of the ring > would obstruct the optical path for the CCDs on the first third.Yah, I see no way out of that.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 14, 1996
I wrote:
>> It is indeed feasible, but there is a drawback: such a system would image a >> ring in the sky, with a huge blind spot at the center of the ring.And you replied:
> I was thinking of a TASS style survey device. The goal was to maximize the > square inches of both silicon and glassRight. Taking into account the unavoidable optical path obstruction caused by an off-axis ring, as explained in my previous message, such a device could image one third of a ring in the sky. If all CCDs are kept parallel to each other, one could indeed use drift scan to gether data along a relatively large swath in the sky. There would be one problem with that, though: your system would require a very small aperture, as explained in my previous message, something like f/16 or fainter, hence it would have a long focal length, hence the sky would drift really fast in its field of view, hence your system would gather only very little light in drift scan mode. I think it would be more useful in stare and skip mode.
>> No, no, definitively no. A standard issue dobs mirror (say an 18 inches >> f/4.5 first surface parabolic) is already a tad expensive (to put it mildly) > Realy, I figured just a few hundred dollars. Anyway I'll make a one CCD > version someday out of my CB245 and a maybe 6" mirror. It would only > cost me the price of a mirror. With luck we'll have analysis software > long before I set up the hardware.Looking at the ads in a 6 months old Sky and Telescope, I see that Nova Optical Systems sells an 18 inches parabolic between f/4 and f/10 for US$ 1800, for instance. I don't call that cheap. Their 24 inches was US$ 3650. Matter of fact, anything above 13 inches is quite expensive in their price list.
>> Right, but the one to many mappings provided by imperfect optics such as >> the one you propose are useless unless one has a lot of computer power to >> deconvolve the images thus obtained. Think in terms of one Pentium per CCD >> chip. > We have huge amounts of computting power going to waste at my office. > There are multi-CPU UltraSPARC and SGI boxes sitting idle 12 hours a day > (or night) Just one of those UltraSPARC CPUs is like 5 or 8 Pentiums. > Has 1/2 a gig of RAM too. 6 foot tall DEC Alpha servers upstairs along > with VAXes. Well you get the point. Hundreds of megaFLOPS setting Idle > all over the country and mostly at night. > Yes I know, way hard to put this to use.The difficulty resides in the fact that solving an inverse optical problem (that's what you are thinking about) is relatively easy when the light sources are point sources (stars, for instance) and the PDF is constant over the field of view, but becomes much more difficult when the light sources are extended (galaxies) or the PDF varies with position in the field of view (which would be the case in your off-axis third-of-a-ring camera). The difficulty reaches its worst when both conditions are met.
>> As I said, the simpler the mapping, the cheaper the calculations required to >> make the data usable. > I read about one optical system that projects a Fourier transform. The > math was way over my head and I don't remember the details. It was part > of a device used by the Air Force to process Synthetic Apiture Radar > data. We were building a digital replacement for it.Right. I have used such optical Fourier transformers for crystallographic work ('t was 23 years ago ;^). I know these kinds of setup quite well. What we used then was this: a point source (a laser is better, due to its better coherency) is set at the front focus of a first lens, thus projecting the image of the point source to the infinite. At the back focus of that lens, we place the image (a transparency) to be Fourier transformed. At twice the same distance, i.e. still further away from the first lens, we place a second lens with the same focal length as the third lens, so that our transparency is at its front focus and its image is projected to the infinite. At the back focus of that lens, we have (superimposed on each other) both an image of the point source (easy to block with a single black dot on a transparent mask) and the Fourier transform of our transparency. Matter of fact, the image of the point source is just the DC (zero frequency) component of that Fourier transform. Naturally enough, the system worked better with monochromatic light and well corrected lenses, still better with a laser. One important point: the more extended the point source, the worse the result, because an extended source causes a spreading of the frequencies in the Fourier transform, hence a blurring of that transform. It is not easy to see how to apply a similar process to starlight, I'm afraid.
Glenn Gombert
ggombert@mail.erinet.com
Dec 15, 1996
Proposal For a TASS Automated Variable Star Monitoring Project
Using several of the programs in the WCS "ToolKit" it should be fairly easy to implement an elementary variable star observing project with some minor modifications of existing stuff that should get us down the road to "automated" data reduction of TASS image data, and also set the ground work for more advance projects in the near future. The procedure that I would like to propose is outlined in two parts below. I invite comments questions/ criticism of the approach and would like to see some more concrete use made of the TASS data that is being collected here in Dayton as well as other places.
The first step would be to calculate a general astrometric solution to each TASS image that could is collected . The program WCS.exe from the ATFTool Kit looks like it just might do the job with some additions to the FIT header file that is written out by "tm3get".
The WCS.exe program basically does three things:
For the above to work properly it needs to know the approximate center of the image and several other parameters such as the pixel scale in arc seconds (for TASS images this would be 13.8 arc seconds/pixel) in both the X&Y directions. Norman's program "tm3get" calculates the value of RA based upon the time and the fact that the middle camera is pointed directly south. The RA and DEC values might have to be "fine tuned", perhaps with and offset factor that is entered in for RA and DEC that would have to be determined by each camera location and used to correct for misalignment etc. The center of the filed does not have to be "exact" for the pattern matching algorithm in WCS.exe to work properly. The fact that TASS images are "wide-field" images as compared to say ST-6 images taken at 72 inch focal length should help a lot. There need to be enough of an overlap region between the TASS image and the field center for the pattern matching algorithm to work properly.
I have tried the "WCS.exe" program out over the last few days and it appears to work properly with several TASS images that were centered using Michael Richmonds astrometry server as well as several ST-6 images taken through a 16 inch F/4.5 Newtonian telescope. The pattern matching algorithm appeared to work properly on such popular objects a NGC891 and NGC4565 when the center of the galaxy was used as the center of the field, often times when the object itself was not really centered in the field. The nice thing about the program is that it can (automatically) detect stars in the image, select a region from the HGSC CD-ROM and then do a pattern matching operation between the two to calculate the necessary rotation parameters for the WCS FITS header file variables.
Once this has been accomplished it would then be possible to use the "VSMON.exe" Variable Star Monitoring Program to monitor specific variable stars along the 0 degree DEC path by specifying which objects that we were interested in the RA & DEC parameters and a "reference" star in the same color band located somewhere in the image that has had the WCS parameters calculated for it. The input reference file that the VSMON.exe program uses as input is just a text file with the "target" and "reference" stars defined for each observation that it is desired to make. The program then calculates the differential magnitude ( in the defined color band) and prints out one line of output for each variable on the input list that is found in the image.
The list could star out small with 15-20 stars that it is desired to make light curves of and then expand the list as the first few measurements that are made automatically are verified. This is not as sophisticated as some of the things that we would like to do with TASS image data but would be good first step to possibly providing a general astrometric solution to each TASS image as well as automating some of the photometric solution to measure some variables that are to be present along the path of zero degree declination.
The above program outlined above should be doable with some modifications to the programs and hardware that we currently have with out having to undertake major new development efforts. This would require some careful calibration of each TASS camera site to ensure that the calculated center of the field of each TASS image is as accurate as possible but this seems that it would be possible with some calibration of each site. Hopefully a project like this (less ambitious than making measurements on every star found in a TASS image) would start the flow of good measurements made from the TASS installations that are collecting data at a good rate.
I will try and put together a TASS Applications Note on how the WCS parameters are actually calculated from TASS images. I invite comments, questions, suggestions, etc on the above..
Tom Droege
droege@wwa.com
Dec 14, 1996
When I first started building cameras, I burned out the CCDs. Now I seem to be able to move a CCD assembly from one camera to another and have not hurt one in some time. This is encouraging. Now there are other things that break. Slowly, I am learning this art.
Nick Barnes
nickb@harlequin.co.uk
Dec 16, 1996
> We have huge amounts of computting power going to waste at my office. > There are multi-CPU UltraSPARC and SGI boxes sitting idle 12 hours a > day (or night) Just one of those UltraSPARC CPUs is like 5 or 8 > Pentiums. Has 1/2 a gig of RAM too. 6 foot tall DEC Alpha servers > upstairs along with VAXes. Well you get the point. Hundreds of mega- > FLOPS setting Idle all over the country and mostly at night. > Yes I know, way hard to put this to use.Actually, not too hard to put at least some of it to use. Look at the way the crypto people get millions of MIPS-years together for crypto challenges. For a start, I suspect that on the TASS list alone there are people with access to MFLOP-years per night. On my own I could probably put together a GFLOP-hour per night, most nights. Give me code, data, and a motivation to use those idle cycles....
Joe Ronchetti
jronchetti@earthlink.net
Dec 16, 1996
> So far we have not worried about declination. Most of us get it to > 1/2 degree or so. Plenty accurate if you are using an atomatic > star finding program. Note that we probably never be accurate > enough to try to locate stars by time and camera position. The local > earth moves around too much.I have written the majority of a program that displays raw tass images, allows you to process the image to expose the faintest stars, shows the right ascention and declination of the visible star and when clicked on will identify the star and pertinant information.
Programatically it works but without accurate header information the co-ordinates will not be correct and the identity of the star will not be correct.
If all agree with Tom, then there is no need to continue with the program. If anyone believes that header info and the alignment of the camera can be improved and you wish to work with me to accomplish the task then let me know. I don't have a camera so I need someone with a camera to be willing to work on these issues and make the images available to me for test purposes.
Nick Beser
beser@aplcomm.jhuapl.edU
Dec 16, 1996
The APL TASS observatory will try to have the headers correctly identified. I am also working up image alignment software to calibrated images, so I hope to have very accurate pointing information in the header. If you have some software that can help with that, release it to the group. It may not be used as is, but could be incorporated with other analysis software.
Chris Albertson
chrisa@wavenet.com
Dec 16, 1996
I think Joe Ronchetti in all his recent postings brings up a VERY important point: We (collectively) have working cameras but when you look at the data it is hard to know what part of the sky the camera was looking at. It will be very hard to do any kind of real science if we are unable to attach names to the stars we see. So far it appears that not one person on this mail list is able to do this for the recent crop of FITS format images.
That sounds pretty bad but I suspect an error in the tm3get software.
About pointing accuracy requirements:
I think plus or minus 1/2 degree in DEC and RA is about as good as we can hope for in the raw FITS files. A camera operator may notice how far off his aim point is and simply correct the tass.rc file rather then move the camera. So I think over time the raw header files will improve in accuracy but never perfect as the true aim point moves around based on whatever is supporting the camera. Still any data processing software should allow for a big error box on the aim point.
It would appear however that some posted FIT files have gross errors the the header data. (The ones I put up on ftp.logicon.com for example) These errors are much greater then a 1/2 degree pointing uncertainty. It would be good to locate the source of these errors. Could it be inaccurate values in the TASS.RC file or an error in the TM3GETxx program? The values in these files are what led me to suspect an error in the program.
So, clicking on a star image in a raw FITS file will never give perfect possition. This should be expected just as you cannot map ADU values diectly to stellar magnitude. This is what the word "raw" in raw data file means. But...
One of the next steps in data processing is to compute an _exact_ transformation between X,Y and RA, DEC. In general with a wide 3 deg field of view this transform may not even be linear especialy if the system is ever moved up from the equator. The File header could be up dated after this exact transform is computed. The FITS standard allows for non-linear (second order) transforms to be specified in the header.
You compute the exact transformation by matching the observed stars to those listed in an astrometric star catalog. Software for doing this uses a "simalar triangels" technique but still depends on a rough transformation for a starting place, orientation and image scale. In order for this to work we realy do need to get #1 correct to about 1/2 degree or so.
Tom Droege
droege@fnal.gov
Dec 17, 1996
I assume these are the files that you got from me. I know that I only put in very rough values for RA and Decl. into Norman's setup file. I assumed I would refine them later. I only set out to get them good enough so that I could find my way on a star chart.
There are calibrations to be done, but they are not difficult or even impossible.
Joe Ronchetti
jronchetti@earthlink.net
Dec 17, 1996
Thanks to the recent input from the group I have been able to identify stars from file(30t0337_287.fits). The bright streak star is not a star but saturn. Other stars in the field seem to match my HSG catalogue.
Please verify.
Chris Albertson
chrisa@wavenet.com
Dec 17, 1996
I'm putting this on the list as thier may be interested observers hopfully willing to contribute ideas or even work. For example, does anyone know the details about SAOimage's interface?
Molhant Norman wrote:
> Dear Chris, > > > When you write RA in decimal degrees is the range 0..360 or > > is it -360..+360 I assumed the former but then it was pointed > > out to me that there are files with negative RA values. > > Tm3getnn does not normalize the result of its RA calculations, so that, yes, > indeed, the result is between -360 and +360 degrees instead of in the range > 0 to 360. > > > I should just be able to divide decimal degrees by 15 to get > > decimal hours. Correct? > > After normalizing, yes.Why not have the program normalize it? It is an easy operation and negative RA values confused the heck out of everyone. But now that I know, I can just keep adding 360 'till I get a positive value them divide by 15. But everyone will have to do this as the first step in processing. Why not do it in tn3getnn?
> > That client/server stuff is comming along nicely. I want the sample > > client to be very easy to read and understand by programmers who > > may not know a lot about either sockets, networking or UNIX. > > Great! For myself, I am one of these programmers. I used UNIX when I wasWell the client is running at about 6 lines of comments to each C line. I try to explain what needs to be done and why before each step. My purpose for the sample client is to make clients easy to write. It is close to done.
> > Still some work to do on the server side. I want it to handle > > multiple clients and simple commands from them. > > I want to read it, 'coz without it my part of the work will keep going slowly > (the bloody thing will be hard to test: I don't have a network, just a single > Pentium without any kind of network card...).From your point of view the server code will likely be two C functions: 1) SetupSocket. 2) SendScanLine. #1 would be called durring initialization the second after each time you write to disk. If you are in a hurry I could send you a dummer server but with the same calling interface. What is your schedule? does the device driver work yet? I was kind of hoping it would take you weeks to debug and I could just kick back :-)
At this point in time the server is operational but to crude for use. I am testing mostly with just one computer. the client/server defaults to both programs on the same box unless you tell it otherwise. Well what realy happens is you run the server where ever you want. You then start up some clients but you need to tell the clients the hostname of the server. (BTW this is just like the way FTP and TELNET work and you can FTP and TELNET to your own system) You should be able to run on one box. That is the normal mode I envision except for people running at a site with tens or hundreds of idle UNIX workstations.
BTW I got a note from Nick Barnes saying he had access to 4 TFLOP per night of processing capability and could move a few tens of megabytes in and out per night too. So, he says if anyone needs images deconvolved let him know.
> > Glenn Gombert has e-mailed me pointers to two programs that look > > very much like what I've been calling a display client. They > > allow data display and control of a CCD camera via a socket > > interface. Neither are directly useable but show that others > > have used to same approach. > > Could you forward these references to me? I'd like to look at the code... > As far as I know, we don't have any volunteer to program the first X-window > client, so I'll probably have to do that too...I was hopping for a volunteer too. It may be best to research tools/libraries for writting X11 programs. They should be simple to use. I think Motif is out. I have heard good things about Xforms. maybe others are good too. I do not want to program at the X toolkit level. It would take hundreds of lines to do anything simple. Need to look around some. I think this is where net surfing will pay off. I have done some X11 programming but have yet to find an easy way.
Another Idea: We may even find we can use SAOimage or something like it directly without even a re-compile. SAOimage already has an interface where by other programs can draw into it's display area. In fact that is it's whole reason for being. I don't think it can handle scrolling so we would have to update the whole display every few minutes or so. Could save a lot of work as our display client would only have to shuffel data between the tm3get server and SAOimage. SAOimage has a nice set of tools built into it.
As for _control_ of the server from a client, we could write a second client to do this. I think version one could just be text based. This would take only a few hours to write. You would have to provide a function call in tm3get for me to call from inside "SendScanLine()" I'd call this for every commad received from a client.
> Hopefully, your prototype client will be easy to expand.The client calls a function "ProcessScanLine(...)" In the sample client ProcessScanLine simply prints a message for each line recieved. I would think a display program would place code to update the sceen inside "ProcessScanLine(...)". This function is in it's own dot.c file. So I'm hopping all you would Need to do is re-write that one function. I could also encapulate the sending of commands back to the server too.
I'll publish as soon as it is ready. The client maybe before the server. Maybe a week give or take?
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 18, 1996
>> Tm3getnn does not normalize the result of its RA calculations, so that, yes, >> indeed, the result is between -360 and +360 degrees instead of in the range >> 0 to 360.And you kindly replied:
> Why not have the program normalize it? It is an easy operation > and negative RA values confused the heck out of everyone. But > now that I know, I can just keep adding 360 'till I get a positive > value them divide by 15. But everyone will have to do this as the > first step in processing. Why not do it in tn3getnn?I did not realize the calculations could give a result outside the 0:360 range until you (and later Joe) pointed this out to me. It never happened here. I will add the appropriate normalizing code to tm3get11 and post an updated version this coming weekend. I'm sorry for the trouble this caused.
> Well the client is running at about 6 lines of comments to each C line. > I try to explain what needs to be done and why before each step. My purpose > for the sample client is to make clients easy to write. It is close to done.Great news!
> From your point of view the server code will likely be two C functions: > 1) SetupSocket. > 2) SendScanLine. > #1 would be called during initialization > the second after each time you write to diskI'm afraid it will have to be a little more complicated than that:
> If you are in a hurry I could send you a dummer server but with the > same calling interface.I'm not in that hurry, but Michael and Nick may well be.
> What is your schedule? does the device driver work yet? I was kind > of hoping it would take you weeks to debug and I could just kick back > :-)Debugging is going along much faster now: the driver does not hang my machine any more. ;^)
> At this point in time the server is operational but to crude for use. > I am testing mostly with just one computer. the client/server defaults > to both programs on the same box unless you tell it otherwise. Well what > really happens is you run the server where ever you want. > You then start up some clients but you need to tell the clients the hostname > of the server. (BTW this is just like the way FTP and TELNET work and you > can FTP and TELNET to your own system) > You should be able to run on one box. That is the normal > mode I envision except for people running at a site with tens or > hundreds of idle UNIX workstations.You still need to configure network support in your kernel in order to use sockets, right? I have configured SLIP and PPP support, though I don't use them, so I think I can use sockets.
> BTW I got a note from Nick Barnes saying he had access to 4 TFLOP > per night of processing capability and could move a few tens of > megabytes in and out per night too. So, he says if anyone needs images > deconvolved let him know.Fine idea!
>> Could you forward these references to me? I'd like to look at the code... >> As far as I know, we don't have any volunteer to program the first X-window >> client, so I'll probably have to do that too... > I was hoping for a volunteer too. It may be best to research tools/libraries > for writing X11 programs. They should be simple to use. I think Motif is > out. I have heard good things about Xforms. maybe others are good too.I still have to explore the tools/libraries to see where I want to go... But yes, Motif beeing a commercial package, I think it is out.
> Another Idea: > We may even find we can use SAOimage or something like it directly > without even a re-compile. > SAOimage already has an interface where by > other programs can draw into it's display area. In fact that is it's > whole reason for being. I don't think it can handle > scrolling so we would have to update the whole display every few minutes > or so. Could save a lot of work as our display client would only have to > shuffel data between the tm3get server and SAOimage. SAOimage has a > nice set of tools built into it.I'm sorry to say that this does not sound too practical: SAOimage is a HUGE package - I'd rather have as small a client as I can get! As for scrolling, who needs it? Scrolling eats a lot of time and the circulating display I use in the DOS version of tm3get11 is clean enough and fast enough for us, isn't it?
> As for _control_ of the server from a client, we could write a second > client to do this. I think version one could just be text based. > This would take only a few hours to write. You would have to provide > a function call in tm3get for me to call from inside "SendScanLine()" > I'd call this for every command received from a client.What a wierd way of doing things! This is quite Macintosh-like, don't you know? As I expect the server to send very many other things besides the scan lines, I also expect to use a somewhat different architecture, where the server waits for commands from the socket with a shortish time-out, then checks whether the device driver has finished accumulating CCD data for a line, if that is not the case it returns to wait for a command from the socket, otherwise it reads the CCD data from the device driver, stores it in the appropriate files, processes these data to build the display lines and update statistics and focussing calculations, then send the appropriate infos (display lines, stats, focussing matrix) to the clients that posted corresponding requests via the socket. At the end of this, the server goes back to waiting for a command from the socket.
> The client calls a function "ProcessScanLine(...)" In the sample client > ProcessScanLine simply prints a message for each line recieved. > I would think a display program would place code to update the sceen > inside "ProcessScanLine(...)". > This function is in it's own dot.c file. So I'm hopping all you would > need to do is re-write that one function. I could also encapulate the > sending of commands back to the server too.Again, I expect to use a slightly different architecture for the client, where it sends commands/requests to the server and reacts appropriately to its replies, but I'll have to wait and see your code before I freeze my client's architecture.
> I'll publish as soon as it is ready. The client maybe before the > server. Maybe a week give or take?Early January would be nice indeed! Thanks for the good job, Chris!
Nick Barnes
nickb@harlequin.co.uk
Dec 18, 1996
> > BTW I got a note from Nick Barnes saying he had access to 4 TFLOP > > per night of processing capability and could move a few tens of > > megabytes in and out per night too. So, he says if anyone needs images > > deconvolved let him know. > > Fine idea!These numbers are ballpark, but I expect I would be able to get 10 hours on 10 machines, each of which will do about 10 MFLOPs/sec. That's 1 GFLOP-hour, or about 4 TFLOPs. I'd be surprised if there weren't other TASS people with similar (or much greater) resources at their disposal; this is just spare Unix workstation cycles floating around one site of a mid-size software house (if I knew and trusted Windows, there'd doubtless be more available). I'd be prepared to spend a couple of hours getting the code to compile the first time around. I'm assuming it would be portable C.
The FTP estimate is ball-park, too. At a quiet time, on a good day, with a tail wind, I can get 10Kbytes+/sec to or from a well-connected site; I can imagine spending up to twenty or thirty minutes of this overnight.
All this is, of course, predicated on my having an informal chat with the relevant people here.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 18, 1996
Here are some details you asked for on the tm3get for Linux as I see it.
Modification to the kernel clock timer interrupt:
instead of a patch file, I'll make both "before" and "after" files available for downloading, so that everyone will be able to adapt this for his own version of Linux. Only 1 file is changed and all changes are clearly marked in the source code.
Device driver and documentation:
it is as clear and to the point as I can make it: I want to be able to read and understand it after leaving it alone for 5 years, so it is documented with a lot of detail (as is the kernel clock timer mod), but not so much that the structure is drowned under a deluge of comments.
Linux 1.3.20 kernel:
this is the most recent version I happen to have sitting on a CDROM here, and also the only one that worked with my hardware (I have two different 1.2.X kernels on another pair of CDROMs, but none of them runs on my machine). I have no idea of the size this Linux takes on my machine, as I don't know how to ask Linux for disk usage statistics. Anyway, I have loaded whatever source (i.e.: not much of it, except for the kernel) I could find on that CDROM, so I suppose I am far from the smallest possible setup.
TASS server:
- read and process the "tass.rc" configuration file
- open the "/dev/tassmk3" device driver and do the necessary setup
- create a socket that will be used to receive commands from clients
- main loop
* wait for a command from the socket, with shortish time-out
= if there is a command, process it as follows
+ new client login
. create a socket for this client
. reply with new socket address
+ old client logout
. reply with acknowledge
. remove the socket for this client
+ write to auxiliary TASS port
. execute the write
. reply with acknowledge
+ read from auxiliary TASS port
. read from that port
. replay with the byte thus read
+ set/update/recall the VCO or the TEC DAC value
. execute the command
. reply with the corresponding new value
+ clear/ramp-up the CCDs
. switch to the appropriate CCD mode
. reply with acknowledge
+ permanent request for data (see further down for an explanation)
. note the requirements for this client
. reply with acknowledge
* check if the device driver has a line (or triple line) ready
= if so
+ get the line (or triple line)
+ save it in the current FITS file
+ reduce it to a displayable line (or triple line)
+ accumulate data for status and focussing
+ for any client requiring display line data, send it
+ for any client requiring status data, send it
+ every 5th line,
for any client requiring focussing data, send it
+ when closing a set of FITS files,
for any client requiring a notification, send it
* check if data collection ended
= if so, exit from main loop
- send end-of-data-collection message to any remaining clients
- remove all sockets
- close the device driver
- end server
Permanent request for data:
- 3 flags (CCD0flag, CCD1flag, CCD2flag) set to 1 to require the
corresponding displayable line data, set to 0 when not requiring them
- 1 flag (STATUSflag) set to 1 to require status data, set to 0 otherwise
(status data = the top 3 lines in the DOS version of tm3get11)
- 1 flag (FOCUSflag) set to 1 to require focus data from camera 0, set to 2
to require focus data from camera 1, set to 3 to require focus data from
camera 2, set to 0 otherwise (this requires 2 bits)
- 1 flag (FITSflag) set to 1 to require notification of closed FITS files,
set to 0 otherwise
- for a total of 7 bits
Packet format used to exchange data via the sockets:
- 2 bytes: length of packet in bytes (itself included in the count),
stored MSB first (network style)
- 1 byte: packet type or command code
- 1 byte: packet subtype or command subcode or parameter
- 2 bytes: packet serial number or data line serial number,
modulo 65536 (unsigned short int),
stored MSB first (network style)
- n bytes: facultative packet data (format depends on the packet type)
Sorry, no CRC: either the socket (network interface) provides CRC data
protection at a lower ISO level in the network (say in the transport layer),
in which case a CRC would be superfluous here, or there is no data protection
on the network, in which case we should change networks. ;^)
Anyway, CRC calculations are at best cumbersome, at worst hard to follow and
in all cases a complete loss of time for us, as we will not program any kind
of error recovery in the clients anyway.
TASS client: (this is an exemple only)
- Send client login request via the socket (reply is another socket address)
then open the socket given in reply
- Send a permanent request for data (say for camera 1, status, focus and
FITS file notifications)
- main loop
* wait for data coming from the socket (with a shortish time-out)
= if there is a data packet, process it as follows:
+ displayable line:
display it
+ status info:
display it
+ focussing data:
display it
+ FITS file notification:
display the file names of the FITS files just closed
+ command acknowledge:
flag the command as done
+ requested data (TEC or VCO DAC value or auxiliary input port data):
do whatever is appropriate (probably just display it) then flag
the corresponding command as done
+ end-of-data-collection message:
notify the user and exit from main loop
* check for a keyboard action or some similar event (i.e.: button click)
(this could be done instead using a call-back mechanism, i.e.:
registering with the GUI which function to call when such or such event
occurs, so we need not write an event dispatcher)
= if there is one, process it and in some cases send the appropriate
command to the server and mark that command as pending
one of these commands might be a client logout request, in which case
wait for acknowledge then exit from main loop
- close the socket connection
- end client
X windows client: I'm wondering whether one could write one in Tcl/Tk, as that seems to be one really easy way to harness X windows...
Chris, you wrote:
>Well if you don't like event driven call-back style programing you are going >to hate X-windows. That is all it is. You pass off pointers to functions >to be called when various events happen. The "system" calls these in >responce to button clicks or other user action.The point is not whether I like event-driven software or not, the point is that I don't want to rewrite the main event dispatcher yet another time. The call-back mechanism you are describing is exactly what I'd like to have. Under the MAC OS (and also under MS-Windows) I have had to write too many event dispatchers, 'coz the fools who designed these systems decided to force all event notifications down the gizzard of the currently active program (even pure system events, at least for the MAC OS). No, call-back is no problem, assuming we can use it all over the board. In fact, I find writing event-driven programs easier (they are just large collections of finite-state automata, sometimes interconnected - very easy to program indeed) than writing the same program in the usual sequential way (i.e.: as a single huge finite-state automaton with an enormous connection graph), inasmuch as I don't have to rewrite yet again the main event dispatcher, for sure!
>If you compile your own Linuux Kernel you need to add support for networking >but almost all distributed kernels have this. You do not need PPP, SLIP or >Ethernet. All you need is the "127.0.0.1 localhost" line in your >/etc/hosts file, the so called loopback device. If you can TELNET or even >PING into your own machine you are set up correctly.Anyway, I must recompile the kernel to use the clock interrupt mod, so I could reconfigure my system any which way needed for this project.
Chris Albertson
chrisa@wavenet.com
Dec 18, 1996
I'm replying to the easy stuff first. I'll look over the rest in more detail.
Molhant Norman wrote:
> this is the most recent version I happen to have sitting on a CDROM here, > and also the only one that worked with my hardware (I have two different > 1.2.X kernels on another pair of CDROMs, but none of them runs on my > machine). I have no idea of the size this Linux takes on my machine,type "df" it will show you all mounted partitions, thier size and how much space is consummed and available
Try "du -s * in any directory containing subdirectories. It will tell you how much space each subdirectory is using.
df and du have many opptions, see man pages.
I can put a very minimal Linux on one floppy disk. A development environment, no graphics, can be smashed into 80MB.
Linux, X11 window system, all Linux utilities I can find, my own web server, word procesors and speadsheets and a couple very large astronomy packages fit in 600MB.
I once set up a fully configued Linux system at our office with only one old 20MB MFM drive on a 20Mhz 386. All the rest of the files not needed to boot were stored on a Sun SPRC file server.
> X windows client: > I'm wondering whether one could write one in Tcl/Tk, as that seems to be > one really easy way to harness X windows...That is a good choise too. Tcl/Tk is available for MS Windows too. so there is the chance that what you do is portable. Tcl/Tk does make ugly stick-like buttons and things and runs slower then Xforms or Motif. Lots of good things have been done with it. The guy in the next cube at work is a real fan.
Tcl is realy a scripping language. I've always though it could make the basis of a good data analysis system. If you wrote a bunch of building block functions the Tcl interpeter could string them together.
> Chris, you wrote: > > >Well if you don't like event driven call-back style programing you are going > >to hate X-windows. That is all it is. You pass off pointers to functions > >to be called when various events happen. The "system" calls these in > >responce to button clicks or other user action. > > The point is not whether I like event-driven software or not, the point > is that I don't want to rewrite the main event dispatcher yet another time. > The call-back mechanism you are describing is exactly what I'd like to have. > Under the MAC OS (and also under MS-Windows)I think it is the same story with Tcl/Tk. You supply functions that it calls. Kind of the way the world is going OOD and all that.
Chris Albertson
chrisa@wavenet.com
Dec 18, 1996
Norman and other software fans:
Basically I think the structure of your server is good. I have suggested some functions that I could supply. There were a couple technical problems, but mostly my suggestions seperate the client/server socket stuff out of the TASS driver stuff. If I am to contribute much there will have to be fewer and cleaner interfaces. I also know what fuctions I have written and have suggested where you could drop them in.
I liked your term "permenent request". I think if we generalize the idea we can eliminate some special purpose code you had.
Do you have any buffers in your Linux device driver? If not and TM3GET is late reading a line it falls on the floor. If you could queue only three scan lines then TM3GET could be seconds late and still catch up by doing a few quick reads.
Molhant Norman wrote (with editing):
> Dear Chris, Nick, Dr. S-100 and all other TASSers, > > Here are some details you asked for on the tm3get for Linux as I see it. > > Modification to the kernel clock timer interrupt: > > Device driver and documentation: > > Linux 1.3.20 kernel:Above ok by me.
> TASS server: > - read and process the "tass.rc" configuration file > - open the "/dev/tassmk3" device driver and do the necessary setup > - create a socket that will be used to receive commands from clientsI've got a function that does this, you can use it. It also does what I talk about just below. (accept)
> - main loopOne step that needs to be added here: "Accept" any new clients see man accept. needs to be called in non-blocking mode.
We have a choise here. We can accept all waiting clients in accept's pending queue or pull off up to one new client each time we get to this spot. I vote for one new client per loop for performence reasons.
We also need to call this "accept" function _above_ the main loop too. This way we connect with the bulk of any waiting clients out side or sort-of-real-time main loop.
Again I can supply the function that does this: creats new sockets and manages a table of open connections.
> * wait for a command from the socket, with shortish time-outYou can check without the wait. This is good as we will need to chech _each_ client socket. Commands will come in via the socket that gets created for each connection by the "accept" system call. So this becomes a loop over all open sockets in a client table.
> = if there is a command, process it as followsI can supply a function that polls all the open sockets (and only open ones) then calls your "Process Command" function. I can also handle socket related stuff (clients logging out etc.)here and give only the driver related commands to your "Process Command".
A way you may like better may be this: I'll supply a function called "Get_Command" You call it at this point in the loop. It will poll all open sockets, process any commands it knows how to do and return those it cannot handle. This is a clearer division of labor. You should never see a "logout" command.
> + new client login > . create a socket for this clientThe create socket needs to be done above but clients should still send a logon as their first command. Don't know what to do with a login message. Maybe just build a status display of logged in clients. This is one command "Get_Command" proceses internally and would not return to you.
> . reply with new socket addressClient does not need this. Does not notice the change on the server side.
> + old client logout > . reply with acknowledge > . remove the socket for this clientSame as logon. Get_Command does this itself.
> + write to auxiliary TASS port > . execute the write > . reply with acknowledge > + read from auxiliary TASS port > . read from that port > . replay with the byte thus read > + set/update/recall the VCO or the TEC DAC value > . execute the command > . reply with the corresponding new value > + clear/ramp-up the CCDs > . switch to the appropriate CCD mode > . reply with acknowledgeAll of the above would be returned by Get_command for you to handle.
> + permanent request for data (see further down for an explanation) > . note the requirements for this client > . reply with acknowledgeGet_Command needs access to the list of open sockets. "permanent requests" need to be stored in this same list too. Get_Command should track these requests and maintain the table of clients/sockets/perm requests. The Table may need to be global. I don't know. Fewer globals the better. So I would not return a Perm Request to you.
> * check if the device driver has a line (or triple line) ready > = if so > + get the line (or triple line) > + save it in the current FITS file > + reduce it to a displayable line (or triple line) > + accumulate data for status and focussing > + for any client requiring display line data, send itI'll supply a SendLine function. It could run through the table of perm. requestes giving each client whatever it asked for. Maybe I should call it "BroadCastScanLine"?
> + for any client requiring status data, send it > + every 5th line,I had not thought about this "every 5th line" idea. I like it. We could generalize the concept. Say, as part of the perm. request a client specifies a "Filter" value. The filter causes the data update rate to be cut by a factor of N.
Now we make "status data" just an other type of perm request and if the client specified filter=5 it works as you say.
> for any client requiring focussing data, send it > + when closing a set of FITS files, > for any client requiring a notification, send itWe could also make focus and notification a type of perm. request too. We may have to limit the client's choise of filter values for some types of perm requests. I am thinking of a data structure that is a list of requests for each socket.
If you place the current scan line(s), last computed focus matrix, status values, and whatever else in a global buffer, then if you simply call "BroadCastScanLine" I'll look up the perm request tables and ship data as required. This should make it easy for you. You just keep a set of buffers filled with current data and make certain to call BroadCastScanLine once per scan cycle. Then call "GetCommand", process any commands then check if the next scan line is ready.
SIDE ISSUE: If you built the FITS files in a tmp directory (/tmp/tass) then mv'd them after they where closed to where ever they realy go. You would never have a half written file after a crash. Also a program (like the script I posted) could watch this directory for new files and not require a client interface at all
> * check if data collection ended > = if so, exit from main loop > - send end-of-data-collection message to any remaining clients > - remove all socketsI think I could supply this to. A close down function that loops over all open clients.
> - close the device driver > - end server > > Permanent request for data: > - 3 flags (CCD0flag, CCD1flag, CCD2flag) set to 1 to require the > corresponding displayable line data, set to 0 when not requiring them > - 1 flag (STATUSflag) set to 1 to require status data, set to 0 otherwise > (status data = the top 3 lines in the DOS version of tm3get11) > - 1 flag (FOCUSflag) set to 1 to require focus data from camera 0, set to 2 > to require focus data from camera 1, set to 3 to require focus data from > camera 2, set to 0 otherwise (this requires 2 bits) > - 1 flag (FITSflag) set to 1 to require notification of closed FITS files, > set to 0 otherwise > - for a total of 7 bitsI think we can generalize this a bit. I want to think ahead and plan for the future Mk IV, V, CB245 and what ever. I like your term "permentent request". I think we should allow for many types of these. maybe we could just define a 16-bit field as a request type so we never run out
> Packet format used to exchange data via the sockets: > - 2 bytes: length of packet in bytes (itself included in the count), > stored MSB first (network style) > - 1 byte: packet type or command code > - 1 byte: packet subtype or command subcode or parameter > - 2 bytes: packet serial number or data line serial number, > modulo 65536 (unsigned short int), > stored MSB first (network style) > - n bytes: facultative packet data (format depends on the packet type) > Sorry, no CRC: either the socket (network interface) provides CRC data > protection at a lower ISO level in the network (say in the transport layer), > in which case a CRC would be superfluous here, > or there is no data protection > on the network, in which case we should change networks. ;^) > Anyway, CRC calculations are at best cumbersome, at worst hard to follow and > in all cases a complete loss of time for us, as we will not program any kind > of error recovery in the clients anyway.OK, I can do without the CRC. I was using the belt and suspenders engineering principle, also thinking (when I wrote it) about use of UDP. Just stick with TCP for the reason you pointed out: we have no time to do retries and no place hold messages waiting for an ACK. Could you imagine the complexity? One queue per open socket of packets sent but not ACK'd another queue of outbound packets ready to go but not sent. We would also have to use a sliding window ACK system as we could not wait for each ACK before we sent the next packet or network latency would kill us. I agre let's not do this.
This will work for a command line client. For a X11 GUI the structure realy has to change. For example, you need to do a redraw when someone moves a window off of ours but only id the X-server has no backing store or has run out.
> TASS client: (this is an exemple only)Figure out where the server is and what port it is listening on.
> - Send client login request via the socket (reply is another socket address) > then open the socket given in replyNo, Client just calls connect and that's the end of it. when connect returns it is connected.
I have a function done that does all of the above including finding the server and parceing the arg list.
> - Send a permanent request for data (say for camera 1, status, focus and > FITS file notifications)A realy nice client may send these based on settings in a dialog box and a button press. Our first client would have the above hard coded.
> - main loop > * wait for data coming from the socket (with a shortish time-out) > = if there is a data packet, process it as follows:Again we can use a zero timeout. Also I was going to do a set of simpler clients. Each would only do a subset of what you have below.
Also some X11 libraries do not allow a "main loop". No problem as you could set up a timmer and poll the socket a few times a second and pull _all_ queued messages out if you find any. return if none.
My "sample client" just prints a line if it gets a message, but so far looks just like this.
> + displayable line: > display it > + status info: > display it > + focussing data: > display it > + FITS file notification: > display the file names of the FITS files just closed > + command acknowledge: > flag the command as done > + requested data (TEC or VCO DAC value or auxiliary input port data): > do whatever is appropriate (probably just display it) then flag > the corresponding command as done > + end-of-data-collection message: > notify the user and exit from main loop > * check for a keyboard action or some similar event (i.e.: button click) > (this could be done instead using a call-back mechanism, i.e.: > registering with the GUI which function to call when such or such event > occurs, so we need not write an event dispatcher)You will likely not be able to check for a key press or click. There are hundreds of things that can happen: user turns you into an Icon, re-sizes window, bring us up to the "top" or forground, mouse pointer crosses pixel boundary, Cut and past from type-in field, ect, ect... Most events are handled by the library (Tk, Xforms, Motif....) but you must supply functions to be called where you desire other then default action.
The c++ libraries make you supply methods for objects that you instansiate from system supplied classes. Even farther removed from the polling loop structure.
> = if there is one, process it and in some cases send the appropriate > command to the server and mark that command as pending > one of these commands might be a client logout request, in which case > wait for acknowledge then exit from main loopYes but you just code the "quit" logic into the callback function associated with the button you labeled "quit" and have faith that by magic the function is called whenever the button is clicked.
Yes you could go down to the event level but all you get is a mouse left button down event followed by a mouse left button up event then you get to figure which part of the screen the pointer was over when the mouse button when down Those widget libraries save a lot of work but you have to stay within the framework of which ever one you pick. That is the reason I'd like to study them in detail before selecting one.
> - close the socket connection > - end client > > X windows client: > I'm wondering whether one could write one in Tcl/Tk, as that seems to be > one really easy way to harness X windows...Another answer i just thought of: See if you can't download some programs that were built with the various X11 libraries, Tk, Motif, xforms and so on. Some of these simply make better quality applications then others. For example, lesstif has been known to crash X-servers.
Tcl is an interpeted langage much like the UNIX shell. In fact you can use it for a shell. Tk is an add-on that has commands like "make dialog box" "make button". When you make a GUI object you also specify a Tcl procedure to be executed when a user operares on one of these GUI objects. If you use the notation (programming language)/(X11 Library) then all you do is choose Tcl/Tk, C/Motif, C/xforms, or even Ada95/xforms, C++/Qt, or C++/(Microsoft Foundation Class) yuk.
(I am _not_ recommending this) Most of my experience is with C/XView. XView is from Sun and is sometimes called OpenLook. I think it is very clean look. Not the current Win95/Motif "in your face" style. If you have Linux you likely have OpenLook too as it is included in most distributions. Problem is the amount of work it takes as it is very low-level. It is in the "land of the 600 line Hello World programs"
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 19, 1996
Dear Joe and fellow TASSers,
> I have tried your patch of adding 360 to the -RA but in the subsequent > pictures, as time passes, instead of reducing the RA as the earth rotates > it is increased because -9 + 360 = 351 , - 12 + 360 = 348 which has us > going the wrong direction. > <-------EAST......WEST-------> > 348, 351.................. -9,-12I assume you mean the opposite: RA should indeed increase as the earth rotates (as any elementary astronomy book will tell you - I just checked in "The Astronomical Companion" by Guy Ottewell, an inexpensive introductory book that I heartily recommend).
Now, if I understand correctly what you are trying to say here, the file with RA = -9 is an earlier image than the file with RA = -12 (i.e.: the RA = -12 image was taken after the RA = -9 image), so that the given RA in the file decreases with time when it should be increasing.
I think I get it.
First, let me reply to your previous message:
You told me that the calculations gave 23h 23m 18s while the image matched 0h 36m 35s, is that it?
Ok, now 23h 23m 18s = 350.825 degrees = -9.175 degrees
and 0h 36m 35s = 9.146 degrees
So, down to the precision of the camera alignment, this looks like a sign
error in the RA calculations in tm3get10 and tm3get11.
Now, if that is so, it would clearly explain why you see RA decreasing as the earth rotates and images follow each other: RA should increase instead, the error beeing due to a wrong sign somewhere in my formula.
That beeing the case, I'll make a correction in the next release of tm3get.
By the way, my thanks and congratulations to you for having found an important error in tm3get - the many astronomers on this list and the few TASS camera operators didn't detect it, neither did I (the author of tm3get) - shame on us!
In the mean time, for all files generated by prior and current versions of tm3get, the RA should first be normalized to the 0:360 range (by adding 360 when negative), and then be substracted from 360 to give the actual RA.
I.e.: if given_RA < 0.0 then add 360.0 to given_RA endif true_RA = 360.0 - given_RA
This should solve that problem. I hope. Can anyone check the resulting true_RA on a few TASS FITS files? Thanks a lot!
Tom Droege
droege@FNAL.GOV
Dec 19, 1996
Merry Christmas and Happy New Year to all tassers.
They are shutting down Fermilab over Christmas, so I will not regularly read mail sent here until the 2nd. I will see tass postings at the wwa.com address from home.
Meanwhile, I am chugging away at the 2k x 2k camera. I predict first light in March.
Glenn Gombert
ggombert@mail.erinet.com
Dec 14, 1996
I think that I am rapidly becoming convinced of the desirablility to have WCS FITS Header file coordinaes in the header files of all future TASS images.
Once they are stored and inserted in FITS Headers this will then allow many of the types of analysis tasks that we wish to do with this sort of images be done in an automted manner.
Below is some information on a library of routines that should be of great interest to TASS software developers, along with a brief description of what kind of functions can be performed with this type of coordinate information. It will make some intersting reading to say the least to see what kind of functions can be (automaticlly) performed with this type of coordinate information.
I will propose an inplemetation plan over the weekend of how we might implement and make use of WCS coordinates in the automated analysis of TASS data.
Click here for information on WCSTools
Michael Gutzwiller
DeepSky@gnn.com
Dec 19, 1996
> Now, if that is so, it would clearly explain why you see RA > decreasing as > the earth rotates and images follow each other: RA should increase > instead, > the error beeing due to a wrong sign somewhere in my formula. > > That beeing the case, I'll make a correction in the next release > of tm3get. > > By the way, my thanks and congratulations to > you for having found an > important > error in tm3get - the many astronomers on this list and the few > TASS camera > operators didn't detect it, neither did I (the author of tm3get) - shame > on us! > > In the mean time, for all files generated by prior and current > versions of > tm3get, the RA should first be normalized to the 0:360 range (by > adding 360 > when negative), and then be substracted from 360 to give the > actual RA. > > I.e.: if given_RA < 0.0 then add 360.0 to given_RA endif > true_RA = 360.0 - given_RA > > This should solve that problem. I hope. > Can anyone check the resulting true_RA on a few TASS FITS files? > Thanks a lot! >I looked at a couple of files I took recently and the RA did increase by the expected amount from image to image. I have not seen negative values in my files so there may be some other problem generating them but for the files I sampled the RA was indeed going in the right direction.
I can't verify whether the computed RA matches the image absolutely. Lately I've been batting .000 using Michael Richmond's astrometry e-mail server. I'll let you know when I do have a good match.
and in a subsequent message ...
I just checked some more files I acquired with tm3get11. By identifying Orion's belt and examining the FITS header in several files I can say that the RA values in all the files I have examined are correct. The RA values I checked are correct in sign, trend, and absolute value and are properly adjusted for each camera.
Whatever may have caused the negative RA values does not seem to affect the RA values I am getting.
Rafael Barbera
rbarb@astro.gea.cesca.es
Dec 20, 1996
Some time ago, i post about a program to analys the variable stars data. At this time, the only version was in Spanish. Now, we have an english version, and an english home page. IF you want to take a look:
URL: http://www.gea.cesca.es/~rbarb/aveint.html
Joe Ronchetti
jronchetti@earthlink.net
Dec 20, 1996
Michael and All,
I would like to process your images through my program. I have been able to identify the starfields on several of Tom's images but because of the irregularities I'm not sure at this point if I am chasing errors in my program or the irregularities.
This goes for everyone taking images. If you can make them available for download or send me sets of three via e-mail. I would like to have a gig of reliable images to process and identify. I am close to starting the database construction of the program and the more images I have, preferably contiguous, to crunch through the better to find bugs.
Joe Ronchetti
jronchetti@earthlink.net
Dec 20, 1996
I have been hearing about gif. Is there consensus on the matter. Has anyone considered fractal compression? Iterated systems claims to get 100:1 reduction and nonpixelizing zoom. I'm considering adding this format to the Tass image processor I am writing.
If one of you have the gif format specs I will add that format also.
Joe Ronchetti
jronchetti@earthlink.net
Dec 20, 1996
I am running Slackware on my linux machine and am thinking about changing to Red Hat. How about a few opinions since I am just learning about linux.
Paul Bartholdi
Paul.Bartholdi@obs.unige.ch
Dec 20, 1996
I think that some of you should come to this meeting next may to present your work and, why not, get in touch with other people working on variable starswith all kind of means.
If you need any further information, do not hesitate to contact me or Dr Grenon from Observatoire de Geneve
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Paul Bartholdi Paul.Bartholdi@obs.unige.ch +
+ Observatoire de Geneve tel +41 22 755 2611 +
+ CH-1290 Sauverny fax +41 22 755 3983 +
+ Switzerland www http://obswww.unige.ch/ +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
============================================================================
Draft 11 Dec 1996
PRELIMINARY MEETING SCHEDULE
-----------------------------
of the
86th AAVSO SPRING MEETING
on
VARIABLE STARS : NEW FRONTIERS
in
SION and ST. LUC, SWITZERLAND
May 26 - 31, 1997
for more information, see
http://www.aavso.org/meetings/coverlet.html
Alain Maury
maury@ocar01.obs-azur.fr
Dec 20, 1996
A message from Emmanuel Bertin:
ftp://ftp.iap.fr/pub/from_users/bertin/sextractor/
The name of the archive is sex_1.2b5a.tar.gz. It contains the source and
makefiles for Linux PCs, SUN/OS, DEC-Alpha, SUN/SOLARIS and HP/UX
compilers. Just go into the ``source'' directory extracted from the
archive and type there ``make -f Makefile.xxx'' where xxx corresponds to
your machine type.
This is still a beta version, but should now be in principle stable
enough for use. Also, I prefered to release it in the present test
before adding new parameters/estimators in order to be sure that they
would lie on a solid ground.
If you choose you use it, please keep me informed about any bug you
might encounter, and any suggestion you might have. I think I will be
able to continue developing SExtractor thoughout the first half of 1997.
The documentation has not been updated yet, but there is a README which
describes most changes.
You mailed me quite recently about a stable SExtractor version for
PC/Linux or having SExtractor running faster or having the possibility
to measure accurate pixel-to-pixel colours with SExtractor. This is just
to tell you that SExtractor 1.2beta5a is now available at
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 20, 1996
> I have been hearing about gif. Is there consensus on the matter. Has anyone > considered fractal compression? Iterated systems claims to get 100:1 > reduction and nonpixelizing zoom. I'm considering adding this format to the > Tass image processor I am writing. > > If one of you have the gif format specs I will add that format also.Fractal Compression from iterated systems is a proprietary algorithm, and the compressor can be purchased for a fee from them. They have decompressors for PC (possibly macs), but I don't think they have unix or linux decompressors. They also play a little loose with the compression ratio's, so you will not see 100:1. I would stay away from it.
GIF is also priorietary, but is widely used and is supported by both netscape and microsoft explorer. It can also be generated rather quickly. The source code is available on the net (look at the pbmplus utilities).
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 20, 1996
> I am running Slackware on my linux machine and am thinking about changing > to Red Hat. How about a few opinions since I am just learning about linux. >I am using Redhat linux on our tass workstation. It came complete with the apache web server, and smb, plus all of the tools. It was not as bad as slackware to install (install from dos or windows).
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 20, 1996
We attempted to do a first light at the APL/TASS Observatory tonight. We were running the tm3get11 software (pulled down tonight from storm.fnal.gov). We modified tass.rc with the correct lat/lon, and all three cameras enabled. The data files with the door closed looked about what we expected (good dark current measurements). However, after the doors were opened, the sensors became saturated. As far as we know, there should have been no light source within the field of view. We have not gone through a detailed focus test as of yet. The lens of each sensor is set to infinity.
Are there any settings in tass.rc that affect the sensitivity of the camera?
We tried the C option to do focusing, but we did not see any change.
We can use some advice to help with the setup.
I can ftp the three files up to storm.fnal.gov.
Herb Johnson
hjohnson@pluto.njcc.com
Dec 20, 1996
*>I am running Slackware on my linux machine and am thinking about changing *>to Red Hat. How about a few opinions since I am just learning about linux.
Linux Journal often praises their Linux packaging and support. Check
http://www.ssc.com/lj
for info on Linux Journal - The current issue has an index of articles too!
Herb Johnson
hjohnson@pluto.njcc.com
Dec 20, 1996
*>I have been hearing about gif. Is there consensus on the matter. Has anyone *>considered fractal compression? Iterated systems claims to get 100:1 *>reduction and nonpixelizing zoom. I'm considering adding this format to the *>Tass image processor I am writing. *> *>If one of you have the gif format specs I will add that format also.I have a little experience with compression. While the TASS data is 12-bit, it likely can stand to be compressed without loss. Any graphic format compression for archival purposes should be lossless: I imagine a 50% compression is reasonable. Any graphic format for non-data bearing images is a different matter. My impression is that mostly lossless methods have been considered. An old standby is Huffman compression, which replaces common values with minimum size codes. But ZIP and gzip seem *awfully* common and available across many systems.
Fractal compression is generally lossy: they may have lossless versions. As fractal compression is highly dependent on image features, I think an image of dots will not gain much by fractal compression.
Norman Molhant
molhant@ERE.UMontreal.CA
Dec 20, 1996
Joe wrote:
> *>I have been hearing about gif. Is there consensus on the matter. Has anyone > *>considered fractal compression? Iterated systems claims to get 100:1 > *>reduction and nonpixelizing zoom. I'm considering adding this format to the > *>Tass image processor I am writing. > *> > *>If one of you have the gif format specs I will add that format also.
And Herbert kindly replied:
> I have a little experience with compression. While the TASS data is 12-bit, > it likely can stand to be compressed without loss. Any graphic format > compression for archival purposes should be lossless: I imagine a 50% > compression is reasonable. Any graphic format for non-data bearing images > is a different matter. My impression is that mostly lossless methods > have been considered. An old standby is Huffman compression, which replaces > common values with minimum size codes. But ZIP and gzip seem *awfully* > common and available across many systems. > > Fractal compression is generally lossy: they may have lossless versions. > As fractal compression is highly dependent on image features, I think > an image of dots will not gain much by fractal compression.Right.
There are ways to compress stellar images without loss (Rice encoding is one, have a look at the compression software that Michael has made available on the TASS home page), and there are standard image file formats with and without compression. For astronomical data, nothing yet beats FITS as a standard (without compression), so that's what we use. ;^) When it comes to displaying pretty images, JPEG is not bad, GIF is well known, and there are other formats, like TIFF etc. One new format is slowly taking the Internet by storm and has proved both powerful and adaptable, that is PNG, whose definition is available on the WEB (use Alta Vista to search for PNG + graphics + format, you'll find it). Source code for the PNG format is also available via the WEB, the ad-hoc sitescan be found starting from the PNG definition pages. PNG beeing a non-proprietary format and beeing also quite good at lossless compression of images (and very good at lossy compression of images too), may I suggest that we use that format for pretty images preferably to GIF, JPEG and others?
Joe Ronchetti
jronchetti@earthlink.neT
Dec 20, 1996
> There are ways to compress stellar > images without loss (Rice encoding is one, > have a look at the compression software that > Michael has made available on > the TASS home page), and there are standard image file formats with and > without compression. For astronomical data, nothing yet beats FITS as a > standard (without compression), so that's what we use. ;^) > When it comes to displaying pretty images, > JPEG is not bad, GIF is well known,For the record, I am interested in pursuing compression and secondary formats which will retain the integrity of the images and also reduce there size. Pretty pictures are not the issue. Download times and hard drive space are the issue.
I agree that perverting the image is not worth it. If hcompress is lossless then why are we not using it?
> and there are other formats, like TIFF etc. > One new format is slowly taking the Internet by storm and has proved both > powerful and adaptable, that is PNG, whose definition is available on the > WEB (use Alta Vista to search for PNG + graphics + format), > you'll find it). > Source code for the PNG format is also available via the WEB, the ad-hoc > sitescan be found starting from the PNG definition pages. > PNG beeing a non-proprietary format and beeing also quite > good at lossless > compression of images (and very good at lossy compression of images too), > may I suggest that we use that format for > pretty images preferably to GIF, > JPEG and others?I am for PNG if it will do the job. Any other opinions?
Chris Albertson
chrisa@wavenet.com
Dec 20, 1996
> I agree that perverting the image is not worth it. If hcompress is lossless > then why are we not using it?I have been pushing for hcompress every since I found out about it. It does _both_ loss-less and lossey compression. So it can serve all purposes.
However you notice that when I posted Tom's data files on the FTP site I did not hcompress them. Why, not everyone has an h-uncompresser. Simple.
That could be fixed by posting the software where ever the data is posted but I don't know if anyone has gotten it to work under DOS.
hcompress is at it's best when allowed to be lossey. Lossless compression of TASS data is limited to about reduction to 55% of the uncompressed file size. So all good lossless compresors do about the same job. The current king of the hill of the lossless compressors "bzip" (no longer gzip) gets just under this size. However, hcompress has the option to go much farther.
I just did an experiment. I compressed 3232t0346_806.fits by a factor of about 50 from 1.4MB to 32K or to 0.337 bits per pixel. I then uncompressed it. To the eye you can still see the same stars and image defects like the now famiar "black holes" in the brighter stars. in fact compressing and decompessing is a quick and dirty way to remove visual noise from the image. The hcompress algorithum saves space be tossing out random background values. The author also claims that compressed images are still photometricaly and astrometricaly "correct". So highly compressed images are also still processable to a degree. It realy does give you the best of both worlds.
So I say yes, use it _but_ then you must also post a decompressor that _everyone_ can use. If someone has a 32-bit DOS C compiler it should be easy. One would have to convert the UNIX style argument processing to DOS style. I doubt it would work with a 16-bit DOS compiler.
The algorithum is also documented both in a paper and in the code itself. One could do a re-write for Win95 but understanding the wavelet transform used would take a bit of study.
Joe Ronchetti
jronchetti@earthlink.net
Dec 21, 1996
This subject may be more important to me that others because I am camera-less and depend on the kindness of others for data. However, I think it is time, with eight camera sites capable of producing 100mb per night, that is almost a gig a night, that serious discussion and a decision be made as to the official type of compression and the official type of archive media be established.
If there are reasons why this can not be addressed at this time, please let me know.
If not, then this would a good time to establish these standards.
Herb Johnson
hjohnson@pluto.njcc.com
Dec 21, 1996
I'm going to "lean into" this discussion of images and file formats. While I obviously have my own prejudices, I certainly respect the work being done and discussed by those actively programming in the TASS project, so no offense or disrespect is meant. Hope my views and comments are helpful.
*> Joe Ronchetti wrote (with editing): *> *>> I agree that perverting the image is not worth it. If hcompress is lossless *>> then why are we not using it? *> *> I have been pushing for hcompress every since I found out about it. *> It does _both_ loss-less and lossey compression. So it can serve *> all purposes. *> *> However you notice that when I posted Tom's data files on the *> FTP site I did not hcompress them. Why, not everyone has an *> h-uncompresser. Simple.There is the argument in a nutshell. Remember that Tom Drouge's computer runs MS-DOS stuff; his programs are written in QBASIC (I don't recall if he uses Windows). As I'll argue below, images are for a different audience and serve a different purpose than *data*.
*>Lossless compression *>of TASS data is limited to about reduction to 55% of the uncompressed *>file size. So all good lossless compresors do about the same job.This is true. But, *if we accept* the premise that "images" are not data, then lossy compression is acceptable. My impression is that we have *data* archives, in FITS format; and then we have some images, which are NOT data. I can appreciate the desire to think of images as valuable in themselves. Excuse me if I express some disappointment in the following story:
I was an early member of another maillist, "CCD", which a few years ago dealt with the technical issues of design and use of CCD's for astronomy. Today it's mostly about acquiring color images and how to build Cookbook cameras. A few years ago, the popular astronomy magazines dealt with CCD and film camera techniques. Now they are filled with overcolorized images of Messier objects and star trails. There is no shortage today of domains which persue ever-prettier pictures.
My point here is to suggest that images are popular, but they should not replace good data; nor should extensive resources be devoted to both producing "images" AND preserving their data integrity. If the *data archives* are available, and programs to convert that data are available, then others are free to "repackage" the data into whatever image formats satisfy their particular priorities. Providing images in popular image formats is a reasonable courtesy, as I see it. Highly compressable images are a reasonable consideration for bulk storage reasons: if *images are not data*, then lossy compression is not a problem.
*>I just did an experiment. I compressed 3232t0346_806.fits *>by a factor of about 50 from 1.4MB to 32K or to 0.337 bits per pixel. *>I then uncompressed it. To the eye you can still see the same stars *>and image defects like the now famiar "black holes" in the brighter *>stars. in fact compressing and decompessing is a quick and dirty *>way to remove visual noise from the image. The hcompress algorithumNot to pick nits, this is a good experiment, but would simply reducing the dynamic range of the image from 12 bits to 8 bits accomplish the same thing?
*>So I say yes, use it _but_ then you must also post a decompressor that *>_everyone_ can use. If someone has a 32-bit DOS C compiler it should *>be easy. One would have to convert the UNIX style argument processing *>to DOS style. I doubt it would work with a 16-bit DOS compiler.I appreciate you accept the concept that image formats must be assessable. But the process you mention is not easy to those who not do this. COnsider the presumption that one is familiar with UNIX, MS-DOS, C, 16-bit vs. 32-bit, etc..... It comes down to archiving a set of decomression programs for Windows, Unix, MS-DOS; and the time it takes to write, debug, and support them. Not to "tell my grandmother how to suck soup", but is this a priority?
Herb Johnson
hjohnson@pluto.njcc.com
Dec 21, 1996
Glad my comments were helpful.
*> When it comes to displaying pretty images, *> JPEG is not bad, GIF is well known, *> and there are other formats, like TIFF etc.TIFF seems to be out of favor, probably because it allows too many varieties or "tags". GIF and JPEG are supported everywhere.
*> One new format is slowly taking the Internet by storm and has proved both *> powerful and adaptable, that is PNG, whose definition is available on the *> WEB .... *> may I suggest that we use that format for pretty images preferably to GIF, *> JPEG and others?I'm reluctant to concurr to use a new graphics format. As the priority of TASS is to do measurements and collect data, such *images* as are created and archived should be in a conservative format. Of course, folks can do what they wish with TASS data (within the agreed sharing data rights) beyond the archives and primary distributions. But one argument that would support the use of additional image formats would be: do they compress much better than GIF or JPEG? Actual tests would be more persuasive than speculations.
Peter McCullough
pmcc@astro.uiuc.edu
Dec 21, 1996
Stardial uses the hcompress algorithm of White et al. A factor of 10 is about as much compression that can be done without starting to lose SNR significantly. A quick warning to novices: the standard hcompress codes expect 16 bit integer data...they will not complain about doing other formats of data, but sometimes those data are not decompressable afterwards! So practice before you delete the original data!
Another warning, relevant to any lossy compression: stacking lossy-compressed images doesn't give the same sort of SNR improvement with N frames as losslessly compressed images. A moments thought makes it clear why this is.
For many applications, images stacks are not of interest, so the above comment is not always relevant.
One last thing. White et al are working on a progressive decompressor based upon hcompress's simple wavelet xform. The progressive quality is important if you want the most significant part of the image and then you are intrigued and are willing to wait for higher and higher fidelity. This algorithm should be done by now, by I haven't heard yet of its public release (but I haven't asked Rick White in a long while).
Tom Droege
droege@wwa.com
Dec 21, 1996
Dear Nick,
Good news! I assume that with the doors closed that you were getting dark values like -20,000 ADU. I assume that you are running cooled. If not, the count will be much higher. As I recall, uncooled the cameras will just about saturate with the normal drift scan exposure.
Note the scale goes from about -20,000 to +32767.
You will find focus far from the infinity mark. I would start out by setting the 2.5 on the orange (meter) scale opposite the fourth mark (16) to the left of the fiducial mark. This is thus at about 2.75 on the meter scale.
If you have not fired up the cooling, be sure to do so. I would start with a setting of 0.5 volts, and work down from there as the telescope cools down. Do not run over 8 amps total on the power supply. If you turn on with the cooling command set at one volt, then this will turn off the second stage cooling completely. Then adjust the power supply for about 6 amps from it. This will probably require about 12 volts from the power supply. Be sure the sooling water is on when you do this, and run the program so that you can monitor the temeratures. With just the first stage running you should get down to 0 C or so. Now you can start lowering the command voltage. At about .4 or so you should se the second stage start to turn on.
Another possibility is that you do not have the clock set at the right speed. A good starting period is 0.916 seconds. You will have to experiment to find where value to set in Normans program. I tried to write the correct value on the control card.
Oh! One more reason that the telescope will saturate is if there are clouds. Cloud cover will surely saturate the camera if there is light around anywhere.
Tell me more and I will think up more things to look at.
Tom Droege
droege@wwa.com
Dec 21, 1996
It seems to me that we are not primarily in the business of making pretty pictures. Of course, all are free to publish neat stuff when you get it. Our main picture publishing will be "finder charts" to locate what we have discovered. These can be done with any standard format as they will be small.
Seems to me that we "compress" by converting our massive raw data files to star lists. This should be of order 1000/1. In this process we loose a lot of information. Seems to me that if someone wants to look at my raw data, then either I have to mail him a tape, or he has to send me a program to process it for what he wants.
It seems to me then that compression algorithms are not very relavent to this work.
Having said that, at this stage in the game where we want to exchange big data file for test, a nice lossless compression standard would be useful. But later I hope we don't have to do such things.
Chris Albertson
chrisa@wavenet.com
Dec 21, 1996
I agree 100%. Image is not data. The subject came up when I wanted to e-mail an image to illustrait a point. I hate emailing a multi megabyte file. Compressed 100:1 a TASS raw frame is only 15K and the guy on the other end can see it with his normal data analysis software.
It would be nice to have the ability to say (in e-mail) "What do you see at pixel [345,631] in attached file?" and then send an attached 100:1 compressed image. This is the only use I see for compressing TASS data files. A GIF works but kills the data as data and only reduces the size to about 50%. So I liked hcompress.
I think you missed just a little math. See below. Well you said you were nit picking so can I :-)
Herbert R Johnson wrote:
> I'm going to "lean into" this discussion of images and file formats. > *>I just did an experiment. I compressed 3232t0346_806.fits > *>by a factor of about 50 from 1.4MB to 32K or to 0.337 bits per pixel. > *>I then uncompressed it. To the eye you can still see the same stars > *>and image defects like the now famiar "black holes" in the brighter > *>stars. in fact compressing and decompessing is a quick and dirty > *>way to remove visual noise from the image. The hcompress algorithum > > Not to pick nits, this is a good experiment, but would simply reducing > the dynamic range of the image from 12 bits to 8 bits accomplish the > same thing?No, I got 640 kilopixels into a 32 Kilobyte file. _MUCH_ better then 8 bits/pixel. As stated, uncompressed file looks reasonable.
The author claims the uncompressed file is still "data" in that the ADU values are the same and in the same locations _and_ the uncompressed file is still FITS.
I did do emperical tests with lossless compression. The winners were within a few percent points but I like hcompress as it has the option to go lossey.
BTW, It appears that there may already be a DOS version. We are testing.
Nick Beser
beser@aplcomm.jhuapl.edu
Dec 22, 1996
Thanks to Marty Pittinger and Norman Molhant, we had a very productive test run last night (Saturday night). Marty will be following up with a detailed e-mail report with his findings, however let me summarize:
We did see stars, We have both a focus and alignment problem (no surprise) We may have found some bugs in tm3get11 (I will let Norman report on that).
We will be taking care of the focus and alignment problems during the week, and the next clear night we will attempt another run (forecast is for cloudy weather).
I wish to express my thanks to Norman for his assistance to Marty. We tied Norman up via phone conference while Marty performed the leg work. This was a several hour exercise, and we appreciate his support.
We have some minor re-engineering work on the housing (to fix the focus, stability, and posible reflection problems. I expect Bernie to work that out during the week.
I asked Marty to document his notes, since it may be useful to others who have to setup the camera to see what we went through. I will forward his e-mail when I get it.
Michael Gutzwiller
MGutzwille@aol.com
Dec 22, 1996
Dear Norman et al.,
It seems that while I get very good values for RA in the FITS headers generated by the tm3get11 program (sometimes within 0.03 degrees) the value for Declination is incorrect in both CRVAL1 and CDELT1.
In particular CDELT1 should be negative since as NAXIS1 (X) increases, declination decreases. Alternatively CROTA1 could be 180 instead of 0 but I would rather change CDELT1.
There are two things wrong with the calculation of CRVAL1. tm3get11 determines the value of CRVAL1 by taking the value in tass.rc and subtracting 1.4668 from it. 1.4668 is simply CDELT1 (0.0038197) times half the width of the image (384). Instead the correct value should be 1.5202 (0.0038197*398) added to the value in tass.rc. It should be added for the same reason that the sign of CDELT1 is wrong. It should be 398 pixels since the image includes the 14 non-imaging pixels at the beginning of each row. Alternatively CRPIX1 could be 15 instead of 1 but I would rather change CRVAL1 instead.
Herb Johnson
hjohnson@pluto.njcc.com
Dec 22, 1996
Thanks for your comments, Chris. Just to clarify one point:
*>> *>stars. in fact compressing and decompessing is a quick and dirty *>> *>way to remove visual noise from the image. The hcompress algorithum *>> *>> Not to pick nits, this is a good experiment, but would simply reducing *>> the dynamic range of the image from 12 bits to 8 bits accomplish the *>> same thing?
...the "thing" being the *removal of noise*, not the compression; sorry I was not clear. Compression has its own merits, but as TASS people are noting, imaging is a seperate issue and not the primary objective of the project. In all this email discussion, priorities can get confusing. A liability of conventional e-mail is that all discussions look the same, all equally important, with length, content, and prior knowledge the only clues!
Chris Albertson
chrisa@wavenet.com
Dec 22, 1996
> Alternatively CRPIX1 > could be 15 instead of 1 but I would rather change CRVAL1 instead.Thinking ahead just a bit, Suppose that one wanted to write out a _processed_ FITS image file. One that is stripped of the non-image edge pixels and dark/flat corrected. We'll need to set CRPIX1 to something in that file too. I'll vote to always make it the first _image_ pixel. So CRVAL1=15 is not that ugly in the raw file as it helps to document the fact that there are non-image pixels.
Chris Albertson
chrisa@wavenet.com
Dec 22, 1996
Herbert writes
"A liability of conventional e-mail is that all discussions
look the same, all equally important..."
See the tag on the subject line. I'll try to flag my posts
so that those of you who are bored to death about technical
software issues can identify and skip this kind of stuff.
Maybe we need to be more carefull with subject lines??
Single subject e-mails would help too.
Tcl/Tk Technical Correction:
I want to retract what I posted some days ago about Tcl/Tk. I had a discussion with my friend the Tcl "expert". I have also done some reading. I an not asking anyone to do anything, just corectioning an earlier post. I now think Tcl does deserve a good look. (summary in last paragraph)
Here is what I found out:
There is an interpeter called "wish". It interpets commands in the Tcl language plus extensions to Tcl (called Tk) for doing graphical user interfaces. There are versions of wish available for most computers including UNIX and MS Windows.
Tcl/Tk (wish) does not draw stick figure objects as I said earlyer. It is in fact almost Motif complient. Motif is a "look and feel" standard. It is also the name of one specific library commonly used to implement Motif complient user interfaces. However an interface may adopt the Motif look without use of the library. Tcl/Tk helps with this.
For those who do not know what Motif looks like, it is close to the Windows 95 look.
To create a Tcl application you write a script in Tcl/Tk. Unlike most other interpreted languages you (the programmer) can add features to "wish", the command interpeter. You can add routines written in C to wish which become new Tcl commands. So it is easy to add functionality to Tcl. In fact that is how Tk was built. If you added a few commands to read/write FITS files for example then writing a FITS viewer in Tcl would be much easyer.
The Tcl/Tk command to create and display a button looks a lot like this:
Button -text "PUSH HERE" -command My_Button_Function
When wish reads this line it will display a button labled "PUSH HERE". When the user clicks on the button the Tcl procedure called "My_button_Function" will be called by the wish command interpeter. (Notice the Tcl syntax is very much like the UNIX shell's syntax and there are more qualifiers that could go on the Button command to specify for example the button's size or location)
Tcl/Tk speed: If you run on a 486 drag and drop and other interactive graphics may not keep up with the mouse. It acts a bit jerky. If you do normal dialog operations it is fast enough. Also if you have a Pentium or Sun SPARC it is fast enough.
Tcl scripts are portable between UNIX X11 and MS Windows to a degree. Some things just do not apply to one system of the other.
Best thing about Tcl/Tk: The "hello world" program is small enough that you can type it in at the command prompt without even an editor and run it without a compile and link step. More like programming in csh or bash then in C.
Tcl/Tk deserves a good look from those writing software for this group. Not only can it be portable but more importently it allows a group of loosely organized programmers to work together. The way one creates an application is to first extend the Tcl interpeter with new functionality then write the application itself as a script. The way a group can work together is to split up the first part. One person writes extensions for image processing, another for FITS I/O, another for Image disply, networking and so on. Each of these extensions are self conntained and there does not need to be much technical interaction between the programmers except when one of them finishes and says "I've added the following new command ...."
Chris Albertson
chrisa@wavenet.com
Dec 22, 1996
Plodding away on the software. One more little part done. I'll post this kind of stuff now and then just in case anyone sees a "show stopper" someplace.
I now have a set of functions for managing a "client table". There are functions for adding/removeing clients and for adding and removeing "Permenent Requests" for each client. An other routine that scans the table and lets whatever sends the data know what to do and in what order.
Assumptions:
Tom Droege
droege@wwa.com
Dec 23, 1996
A good idea. For the moment, I am trying to read everything. But there may come a time when sorting is indicated. I particularly like the idea of "single topic" posts. I know that I tend to ramble once I start typing, this tends to obscure important points and no one notices what I wanted to say.
For starters:
HARDWARE SOFTWARE ON LINE OFF LINE DATA BASE DATA REDUCTION GENERAL NEWS
Would someone like to pick up this task, propose a nice list, and make it a technical note?
I would make it a "courtesy standard" for posts. I have read some lists where any post withoug the proper heading got a nasty response from the moderator. I welcome all to this list whether they follow rules or not.
Herb Johnson
hjohnson@pluto.njcc.com
Dec 24, 1996
Tom concurred on "single-topic" posts and suggested a list of topics for purposes of easy recognition as follows below. I agree this would be for convenience, not for exclusion; and the number of topics should be few. I add to the suggestions. - Herb
*> [Tom's list] *> HARDWARE *> SOFTWARE *> ON LINE *> OFF LINE *> DATA BASE *> DATA REDUCTION *> GENERAL *> NEWSFor "software: on line and off line" how about instead:
"Control Software" (drivers, test programs, etc. that "talk"
to the camera hardware directly)
"System Software" (most of the Linux stuff these days, excluding
analysis, data reduction, etc., but including
remote access stuff, operating system specific
stuff.)
Does the "data base" stuff include discussions of archiving? Seems a good place to put it.
For "hardware" how about instead:
"Camera" (all camera hardware, optics, filters, setup and use,
repair and upgrade)
"Electronics" (other hardware not directly part of the camera, like
computer hardware and site electronics"
Other categories:
"Observations" (what we are looking at or should be looking at, site
or observing conditions, etc. including observation reports
"Astronomy" and astronomy-related posts)
"Mark IV" (stuff related to the upcoming camera design, which everyone
should look at just to give feedback)
"Site Reports" (just as it says, these probably should migrate to the archives
as a series. These could represent our most "public" face to
those initially interested in observations.)
There will be some overlap, but there always is! The rest of the subject title will clarify it.
Glenn Gombert
ggombert@mail.erinet.com
Dec 26, 1996
While serfing the Astronomcial WEB this past several days I came upon a very interesting URL that proposes a "standard" protocol to transfer data, etc from an astronomcial server and its remote clinet.
The URL is:
http://vizier.u-strasbg.fr/doc/asu.htm
It appears to be used as the basis of such things as the ESO "SkyCat" tool among others, it may contain some interesting information & concepts for the TASS remote client interface...
Glenn Gombert
ggombert@mail.erinet.com
Dec 30, 1996
Several weeks ago I modified the cooling system for the triplet that I have been operating here in Dayton and I thought that I would pass along the modification(s) that I made in case some other folks would like to try the same thing.
I modified the cooling system to use a copper coil from the Richard Berry "CookBook" design that was purchased from University Optics. I made the cooling system a "closed cycle" system filling the system with windshield wash fluid and pumping the mixture with a "Little Giant" pump. The copper cooling coil is used in a five gallon bucket of water filled with antifreeze. I used this arrangement to take data the night of 12/14 all night long. The cooling system worked very well with the increase in temperature from waste heat from the TASS triplet almost equally being offset during the night by the drop in outside temperature (as I recall the temperature reached the mid 20's outside). The result was that the current through the TEC cooler stayed relatively constant over the course of the night and thus maintained very good temperature regulation.
I have not tried this combination at lower temperatures but I think that it should work out quite well. I will post the results of a run at lower temperatures as the course of the Winter progresses.