We received our copy of the report of the DEIMOS Software PDR review board in late April 1996. Our initial response to that report was contained in the software section (2.4) of DEIMOS Quarterly Report No. 8, which was issued in mid-July. (That section is reproduced at the end of this Appendix.) That response described what we intended to do regarding the issues raised by the review board.
This appendix represents an updated response that describes what we have done during the 15 months since the PDR to address those same issues. For ease of reference, points will be addressed in the order in which they appear in Steve Kent's email of Apr 23, 1996 (a concatenation of the reviewers' notes).
The review board identified the following 5 items as most important:
We have decided to use the 2nd generation SDSU controllers, as advised. We have received and tested our first set of 2nd generation controller boards, and the results of these tests were reported at the DEIMOS CCD review held May 20, 1997. Copies of the report from that review are available, and selected chapters (3: CCD System Overview, 5: Science CCD Controller, and 6: Data Capture) of that report are available via the WWW at the following URL: http://www.ucolick.org/~kibrick.
The choice of quick-look display has remained a difficult one. While we hope to use the NOAO ``PIE server" distributed real-time image display tool (currently under development) for DEIMOS, it is not yet available and may conceivably not be released in time for commissioning. Our interim plan is to use Keck Figdisp for engineering tests, and as a fall-back display for commissioning. Meanwhile, CARA personnel are investigating the latest release of ESO/RTD (which was used for demo purposes at our PDR) as a candidate for the new Keck Guider display. RTD may emerge as an alternative fall-back QL display if CARA's results are positive. The review board advised us to be prepared to simplify the requirements for this display. We are prepared to do so, and the contingency of commissioning with Figdisp will represent an implicit simplification of those requirements.
The review board asked for a better definition of the requirements and specifications for the database, and suggested a need to better understand the interface with CARA. The CDR documents include a detailed specification of database function and schema design as requested. Details of procedural and technical dependencies on CARA are still under discussion, so we cannot at this time state how much unification of telemetry, environmental and weather logging, etc. will take place at Keck by the time of DEIMOS commissioning.
With respect to management, the board recommended that we be prepared to prioritize requirements in case descoping of effort is required to stay within budget. While we are prepared to do so, a prioritized list of requirements has not yet been generated. As described later, obtaining a sufficient allocation of software staff time is currently a more serious management problem than budgetary constraints.
The board suggested that DEIMOS should take the lead in getting CARA to commit to providing the guider functionality that DEIMOS requires. Lick and CARA have recently begun a joint effort (headed by Jerry Nelson) to develop a new Guider system for Keck. DEIMOS project staff will be minimally impacted by this effort, but will have input into the new guider specification. Another member of the Lick scientific software staff (Deich) who is not currently assigned to the DEIMOS project will be assigned to this effort at approximately 15%-time.
The review board made five suggestions for the CDR:
High-level functional diagrams of the DEIMOS hardware/software system and certain procedural elements (like slitmask fabrication) are provided in the CDR documents, as requested. These diagrams show data flow as well as interdependence. While additional work remains to provide a complete detail-level interconnectivity diagram of all subsystems, that work is in progress.
Decisions about actual brands and models of computing hardware will be made as late in the project as possible, because we have taken considerable effort to ensure that the software we develop is and will continue to be portable to most likely platforms. However, we will provide at the CDR our performance and memory requirements for whichever vendor's product is eventually purchased.
The CCD controller issues have recently been resolved, and a second-generation SDSU system will be used. A detailed hardware design for the CCD controller was presented and accepted at the DEIMOS CCD Review held on May 20. The purchase order for this hardware has been issued, and the first shipment of boards should arrive in July. In the interim, software development will proceed using the initial set of evaluation boards that we received in March. We expect to receive the final shipment of controller boards in September.
We have re-invited all of the reviewers from the PDR panel, as requested.
We have attempted to abstract the main points of each of the individual reviewer's comments. These are shown as italicized text. In cases where the same comment or concern was raised by more than one reviewer, usually only the first occurrence will be listed and answered.
The project has taken effectively both of George's suggestions with regard to image display software. We have identified an existing image display project involving a large subset of the astronomy community (NOAO RTD) and although DEIMOS cannot be said to ``take the lead" in this effort, we are actively participating in and monitoring the progress of the project. Doug Tody came to Lick last December for initial discussions about this collaboration, and Steve Allen paid a return visit to NOAO in March. Results of these discussions are described in section I.4.1.
In the meantime, we are also enhancing an existing product (Keck Figdisp) for immediate engineering use and as our insurance policy should the NOAO software not ship in time for our needs. Keck Figdisp is the image display server currently used by three of the first-generation Keck instruments (HIRES, LRIS, NIRC). It was recently ported from SunOS to Solaris by John Cromer of CIT (at no cost to the DEIMOS project) and has subsequently been ported to Dec Alpha/Digital Unix and Intel-586/Linux by Steve Allen. We do not intend to enhance Keck Figdisp to meet the full range of DEIMOS image display requirements at this time, and will only make the minimal enhancements necessary to allow its use during the engineering and test phase of the project. If, by October, it appears that delivery of the NOAO RTD will be significantly delayed, then further enhancements to Keck Figdisp will be considered at that time.
The scope and utility of the database component of DEIMOS have been more accurately defined. Our work assisting CARA in the automatic ingestion of STB (an image archiving system) headers into a relational database has proved useful in clarifying DEIMOS' needs and our practical capabilities in the realm of image location and retrieval. The database as repository of authoritative keyword information and instrument design information has proven practical (as the many pages of generated documentation in the CDR packet may show). The role of the database in slitmask fabrication and documentation, pipeline data reduction, and online information and help systems has been realistically defined.
There should be a supported Keck ``approved" pathway to deriving slit mask coordinates.
This is a worthwhile goal, but requires coordination with CARA. No action has yet been taken on this suggestion.
The panel suggest a more convenient approach (than HTML) to printable documents be provided...Possible ideas are Frame, LateX, or other word processors with HTML converters.
The DEIMOS software team has adopted LaTeX for all its formal documentation. The keyword documentation generators produce both HTML and LaTeX directly; handwritten documents are written in LaTeX and converted to HTML using publicly-available tools. A standard format is imposed on all project documentation, and all handwritten documents are maintained under CVS for revision control and ease of access.
CCD controller hardware requirements (such as matching amplifier gains to 1%) should be relaxed if equivalent functionality can be provided via hardware.
Software equivalents will be implemented if the corresponding hardware functions prove too challenging. However, preliminary tests with MIT Lincoln Lab CCDs and the second-generation SDSU controllers indicate that amplifier gain matching to the 1% level may well be achievable.
Of all the suggestions made by the PDR review board, this last item is the one that should have been taken more seriously. Kibrick is the one responsible for managing the day to day activities of the Lick scientific programming group overall, and is also responsible for managing the DEIMOS software effort. In addition, Kibrick has been deeply involved in the selection and testing of the second-generation CCD controller boards and in the overall hardware design for the DEIMOS CCD controller. He is also responsible for the management of the software effort for the Keck ESI Spectrograph, as well as several recent upgrades to the Keck HIRES Spectrograph. One of these (the HIRES image rotator upgrade) took considerably more effort than was budgeted, partly because it required additional effort to adapt to the changes in the Keck Telescope control interface that resulted from retrofitting the Keck-2 system onto Keck-1 at the beginning of this year. Further, Kibrick has ongoing responsibilities to support both telescope and instrument control systems for both the Shane 3-m and Lick 1-m Telescopes at Mt. Hamilton, including completion of the commissioning of the MOS and PFCAM instruments for Mt. Hamilton.
While all of the members of the DEIMOS software team have obligations to other projects, Allen and Clarke have been able to devote the majority of their time to the DEIMOS software effort during the past year. Phillips has been able to devote 50% time, and Tucker 40%. (In addition, Tucker has had assistance with the Galil Motor Controller software from Jim Burrous of the Lick Electronics Shop.) For the period October 1996 through May 1997, the percentage of Kibrick's hours devoted to DEIMOS was 21.4%, with the bulk of that time expended in the months of April and May.
While excellent progress has been made by the other members of the DEIMOS software group (as evidenced by their respective chapters of this document), many of the overall software management functions (e.g., documentation of top-level software design, overall software planning, establishment of standards, budgeting, scheduling, and reporting) have not received adequate attention due to Kibrick's obligations to other projects. This lack of attention is readily apparent in those sections of this CDR document for which he is responsible. While UCO Director Miller has recently made efforts to temporarily disengage Kibrick from non-DEIMOS activities, this disengagement did not occur until April 1, and is currently slated to expire following the DEIMOS software CDR. In addition, between April 1 and late May, most of Kibrick's time (while directed towards DEIMOS) was spent on preparations for the DEIMOS CCD Review, at the expense of preparing for this CDR.
In hindsight, it is obvious that our expectations regarding the fraction of Kibrick's time that would be available for DEIMOS software project management were overly optimistic. As Kent so aptly noted, DEIMOS activities (in this case, management activities) did slip because other activities (typically those impacting scheduled telescope time) assumed higher priority. Clearly, the current arrangement is not adequate, and a greater allocation of effort towards this area is urgently needed. This, and the question of overall software staffing levels for the DEIMOS project, is one of the more urgent items that needs to be addressed by this CDR, and for which suggestions from the review board are particularly desired.
The availability of office space is a related concern which impacts our decisions regarding staffing. Both the overall DEIMOS project manager (Cowley) and Kibrick have felt for some time that the combined software demands of DEIMOS and ESI indicated the need for additional software staff. While that need has been partially offset by the transfer of Jim Burrous from Mt. Hamilton to Santa Cruz, Jim is attached to the Lick Electronics shop and his primary responsibilities involve electro-mechanical testing and integration. However, Jim has made very significant contributions to the Galil motor controller firmware used by DEIMOS, ESI, and PFCAM, and we should plan to make as much use of his software talents as possible.
Nonetheless, both DEIMOS and ESI could benefit from the addition of at least 1 more programmer, or even a summertime student worker. However, the scientific programming group (currently housed in temporary offices in Thimann Labs) has no additional office space in which to locate a new hire. In fact, our repeated requests for additional office space have been consistently denied due to a global lack of office space within the Observatory and the campus in general. Our group will be moving back into the Natural Sciences 2 building (the location of our offices prior to November 1995) now that the seismic repairs on that building are complete, and this might provide a window of opportunity for acquiring one small office.
However, even this is doubtful, since current plans call for cutting the square footage assigned to the Observatory by 1900 square feet. This severe shortage of office space needs to be considered when proposing any plans to augment the software staff available to work on the DEIMOS and/or ESI projects. It should also be noted that negotiations involving the allocation of office space have consumed an inordinate amount of staff time throughout the Observatory. While these negotiations have had negligible impact on most of the Lick Technical Facilities (since they are housed in buildings in which space is not being reallocated to other campus units), they have represented a significant drain of time on the scientific programming group (especially Kibrick) and on the staff of the CCD Lab (especially Stover).
An area that did not appear explicitly on any budget but which might absorb a lot of time is generic support for infrastructure
Kent was also correct on this point. The DEIMOS software group has made a significant investment in software infrastructure during this past year, as should be apparent from the relevant sections of this document. While this was an extremely worthwhile investment, it certainly involved more hours than we would have predicted at the PDR. While the most significant portions of this investment have been completed, we will be careful to budget for an on-going commitment to infrastructure-related activities.
The biggest single item that I found missing in the PDR documentation was a diagram showing all the subsystems with their interconnectivity... Such a diagram is a must for the CDR.
Considerable effort has been applied towards this task, especially towards development of effective tools which utilize information from the DEIMOS design database to automatically generate such diagrams. Major portions of such an overall diagram have been completed using these tools, and are included in this document. Additional diagrams will be presented at the CDR. However, it is clear that further effort is still required to provide a truly complete overall diagram, and the absence of such a diagram from the initial release of this document is entirely due to insufficient availability of Kibrick's time.
It was stated during the review that files would be used to exchange information if the database were not available...The use of two databases, one for ``operational" needs and one for offline scientist queries is also the approach chosen by Sloan.
In most cases where access to database-resident information is required for processes such as slitmask fabrication or design, pipeline data reduction, etc., a level of isolation has been introduced; authoritative data are maintained and resident in an online database, but the applications run against human-readable extract files (some in standard formats such as FITS table extensions). In this way the applications are decoupled from dependence on particular database characteristics; only the relatively simple extract generators need be changed if the database engine or requirements change. These extract files can, at need, be generated by other means, or manually customized by users. We are considering this policy of isolation not as an interim measure, but as a permanent design feature. Remote access to database information can be arranged by access to extract generators via Web pages, with the generated extract downloadable to the remote user's site.
A very useful activity is to construct a data flow diagram for the whole software system.
We have tools in hand which create, maintain, and generate paper copy of information flow diagrams. (See diagrams included in CDR documents). We concur that all major subsystems should be sketched, examined, and refined before code is written. While considerable effort has been applied to this task, additional work remains.
It is not clear if the display software is intended only for use at the mountain or if it should also run on user's home workstations.
Our current plan is that it should be usable at both. Assuming the NOAO RTD effort is successful, that display server should ultimately become available as a supported part of IRAF, and thus should be available on user's home workstations.
Keck Figdisp should also run on user's home workstations, and it has been ported to several platforms including SPARC/SunOS, SPARC/Solaris, DEC Alpha/Digital Unix, and Intel-586/Linux. However, convenient GUI access to Keck Figdisp requires purchase of a DataViews runtime license which cost approximately $100 per machine, and which is available for all of the above platforms except for Linux. In the absence of DataViews, all Figdisp functions are still accessible via mappable function keys.
How much input will the user community have into the specification and requirements for this (Quick Look display) and other functions such as spectrograph control? (Is the opportunity passed?)
While the opportunity is not yet passed with respect to the Quick Look display, it is not clear how to obtain greater input from the user community. Many have been invited to the software CDR, but few have chosen to attend. The same was true of the PDR. All of our PDR documents have been and will continue to be accessible via the WWW, as will the documents from the CDR. Kibrick has sent surveys to all Keck users at UC requesting suggestions for improvements/additions to Figdisp, but has received only a handful of responses. We consider such input extremely valuable, and would like to find a way to encourage further participation by the DEIMOS user community. Suggestions are solicited from the review board regarding how best to obtain additional input and review from the DEIMOS user community.
The control room layout plans assume that the observer is at the telescope, yet a desire has been expressed that remote observing be supported.
Once the instrument is fully commissioned, our long-term plan is that DEIMOS will be operated remotely from Keck HQ in Waimea. Upgrades to the network bandwidth between the Keck summit and Waimea are now in progress that should make this feasible.
Sloan has used CVS for code management for several years...
The DEIMOS software group thanks the review board for recommending CVS. We have converted our pre-existing SCCS-based source repositories to CVS, and are using CVS to maintain all DEIMOS software and documentation, including the chapters of this document.
Configuration management is a thornier problem...and means tracking not only versions of a particular software package, but also tracking the dependencies of one package on another, possibly for multiple platforms simultaneously.
Configuration management is indeed a thorny issue for instrument control systems. Lick and CARA personnel are collaborating on a revised build and release structure for the CARA KTL library code. We hope that this project will ensure easier porting, co-development, and configuration management of KTL-based instrument control systems. If there is sufficient interest, the current state of this collaboration can be discussed at the CDR.
The biggest potential problem in supporting multiple platforms may be the interface between the VME crate and the instrument computer; although this interface uses ``standard" network protocols, problems peculiar to each platform may crop up.
This is one of several reasons why our default plan is to run Solaris on the CPU in the VME crate. In doing so, the VME crate becomes indistinguishable (from the point of the OS) from a standard Sparcstation 5. Only if serious problems develop with this plan will we switch to running VxWorks in the VME crate, in which case this issue of ``peculiarity" may become relevant.
The hardware requirements were not discussed. They should be a significant item for the CDR...One should buy a system with plenty of CPU, memory, and disk. One should avoid saving on hardware at the expense of making the software more complicated.
Hardware requirements will be presented at the CDR. Certainly we would like to have plenty of compute resources, but we have a fixed budget for the instrument computer hardware. By deferring this purchase as long as possible, we hope to acquire the most powerful machine we can afford.
If funding is found for the second beam, will that funding include upgrades to computing to accomodate the doubling of the data volume, or is the current hardware supposed to support two beam?
Depending on the CCD readout rate (which depends on the characteristics of the CCDs ultimately selected for the DEIMOS mosaic), the current system may be barely adequate to support both beams. In seeking funding for the second beam, we will attempt to acquire funds for additional computing hardware to more adequately support the second beam. However, since the level of funding that might ultimately be obtained for a second beam is unknown, we can't provide a firm answer to this question.
Of all the various processes that make up the HIRES and LRIS software suite, the MUSIC ``traffic" process has been a rock. It is easily the most robust and stable piece of control software we have running now.
The DEIMOS software team has elected to stay with the music/traffic instrument control software rather than migrate to the newer EPICS system currently in use by CARA for telescope control. We concur with John's assessment of music and traffic.
WCS Issues and FITS keywords: Seems to imply an enormous number of keywords for the various file headers...
Steve Allen's latest revisions of FITS WCS representation use FITS table extensions rather than enormous numbers of indexed keywords. This reduces the WCS problem to the case for existing single-CCD images.
Spectrograph Control:
Multiple levels of transaction logging are being provided for the commands between the control computer and motor controller. Keywords have been implemented to track the revision level of the software that runs in the computer. The DEIMOS motor control software will be maintained under CVS. Automated procedures have been implemented for downloading the firmware from the control computer to the motor control.
I do not believe the project should be delayed in order to meet the 1% gain-matching requirement or dual-amp readout.
It won't. The CCD controller design presented at the DEIMOS CCD review on May 20 provides for a phased-implementation approach by which we can initially support single-amplifier readout and then upgrade to dual-amplifier readout at a later date without the need for any rewiring of the CCD dewar or the CCD controller. Our initial purchase order to SDSU for CCD controller boards includes only enough video processing boards to support single-amplifier readout. A more detailed discussion of these issues can be found in Chapter 5 of the document from the DEIMOS CCD review, which can be obtained at http://www.ucolick.org/~kibrick/deimos-ccd/chapter5.html.
Data Storage Formats: Observerrs are going to spend more time trying to figure out the data storage formats than they did collecting the data. Adequate software tools should be supplied to address this complexity... The team should be prepared to back off some of the requirements to meet the delivery schedule.
During the ensuing year we have conferred with other teams constructing mosaic detectors (notably NOAO). It is now clear that the image section from each CCD amplifier should be stored in its own IMAGE extension. I.e., there should be no archival image format where the separate image sections are combined into a single FITS array. This removes the need for the extremely complex set of indexed keywords which describe the geometries of image sections within a combined array.
The display server should deal sensibly with conflicting client programs and be able to distinguish between a real-time client and a non real-time one...Does a new image begin to overwrite the screen while a quick-look procedure is still processing the previously-displayed image?
We have not yet explicitly addressed this issue in our design, and this is a question on which additional input from the DEIMOS user community would be highly desirable. Our instinct is to abort any such in-progress procedure and return an error condition from that procedure so as to not delay display of the newly-available image. However, this may not be what some observers desire. Further suggestions from the review board regarding alternative solutions to this problem would be appreciated. In particular, it would be useful to know the current plans for how the NOAO RTD server will address this problem.
The degree of isolation between database and applications described above should address John's concern about impacts on observing-time performance. Software has been implemented and tested to perform sanity/consistency checks across those portions of the database which maintain the system design description and the keyword database. Where feasible, we will attempt to implement similar consistency for the other portions of the database, and to provide proper documentation to insure the CARA staff are able to run these as necessary. Some of these consistency checks may be run automatically via cron jobs. We have tried in the CDR document to make more accurate estimates of the ongoing commitment required of both Lick and CARA staff to keep the information management system functional.
Our upfront, highlighted, can't-emphasize-it-enough recommendation is to manage the code development via CVS.
As mentioned above, the DEIMOS team has settled on CVS as our source code control standard and we are well pleased with it.
The presentation should have contained a data flow diagram...
As mentioned above, data flow diagrams are included in the CDR, although this effort is not yet complete. It should be noted that these diagrams are well integrated with database so that they can be properly maintained as the software evolves. The software to generate these diagrams from the design model in the database will be a part of the delivered system, so CARA should be able to regenerate these diagrams on demand.
Possible compute platforms should be specified and a proposed set of benchmarks presented at the CDR.
Benchmarks of typical image reduction operations are included in the CDR.
Quick Look: The proposed combination of a very heavily binned image with a panned/zoomed mouse-driven display of small regions at full pixel resolution might not be adequate for either engineering tests or commissioning. It might be useful to also produce instrument statistics...
We do not see an obvious alternative to this approach, and this combination remains our default plan.
Keck Figdisp, which we plan to use for initial engineering tests, already provides many of the statistics capabilities requested. We plan to implement a more generalized statistics process as part of the data capture pipeline.
Simulated images might be very useful for testing the software.
Allen has generated simulated images using the IRAF artdata package, and these are being used to the performance of Keck Figdisp on DEIMOS-sized images.
CCD Readout Software: The biggest software risk area seems to be the DSP and the low level C programming required to readout the mosaics...The DEIMOS team should focus on developing the DSP and low level C code needed to operate the second generation SDSU controller.
For this reason, a significant fraction of Kibrick's time was devoted to tests of the second-generation SDSU controller boards following their delivery in late March, in order to have results in time for the May 20 DEIMOS CCD Review, and to ensure that the major purchase order for this equipment could be issued prior to a May 31 fiscal closing deadline for orders of this size.
Our existing engineering software tools for operating the SDSU controllers have been updated and tested with the second-generation controllers boards. We can now successfully program and generate valid CCD clocking waveforms and bias voltages using the second-generation boards, and have used a high-speed oscilloscope to verify the proper operation of the video processing signal chain. Our VME crate (currently SunOS-based) is reliably communicating with the second-generation SDSU timing board and utility board. We are sufficiently confident with this new hardware and with our ability to generate DSP software for it that we have sought (and obtained) from the DEIMOS CCD Review Board permission to place a large order for second-generation SDSU controller boards.
More Prototyping Encouraged
The DEIMOS team has used the working HIRES instrument as a prototyping test platform for the GUI design tools. The DEIMOS GUI software has been adopted as the standard for ESI and PFCAM as well, so that there will be deployed interfaces for these projects prior to DEIMOS commissioning, and the DEIMOS UI will benefit from our experiences with the other two instruments. The interface software, now in its second major release, has been upgraded to run in ``no-KTL-service" mode so that prototyping for DEIMOS can precede access to the working instrument. Colour pictures of some early DEIMOS interface designs were distributed at the May 12th Keck Software Coordinating Committee (SCC) meeting, and a more complete set are in the CDR. We concur that user interface prototyping and early user feedback are very important.
Initial tests of Keck Figdisp using simulated DEIMOS images are in progress by the DEIMOS software staff. However, we have not yet had potential DEIMOS users take this system on a test drive. This needs to be done soon.
Mask Generation: CARA and DEIMOS programmers should work together to achieve as much commonality as practical between CARA's ``sky" planning tool and DEIMOS' mask planning tool, which have much in common. Both should use common formats for star lists.
This is a desirable goal, but will require some commitment of resources from CARA if it is to be achieved. In particular, it would be useful if Al Conrad (the author of CARA's ``sky" program), could spend some time in Santa Cruz to work with Drew Phillips (the author of the DEIMOS mask planning tool) to help achieve such commonality. Methods for funding the costs of such collaboration need to be explored.
The DEIMOS team should review their commitment to IRAF (for the mask planning software) and consider a switch to IDL.
The DEIMOS team conducted a survey of the potential DEIMOS user community vis-a-vis their preferences between IRAF and IDL. The results of the survey were completely inconclusive; as a result, the team is maintaining its current preference for IRAF as the development environment for certain end-user tools such as pipeline data reduction and slitmask design.
Spectrograph Control:
Although a recent telecon with CARA regarding cabling may have briefly touched on this issue, this should be flagged as an action item to confirm that the necessary cabling will be in place.
Extensive testing with the Galil DMC-1500 series 8-axis controllers was conducted earlier this year in conjunction with tests of the Lantronix ETS8/P terminal servers. These tests were conducted both to establish the reliability of RTS/CTS hardware handshaking under conditions of maximum traffic and to measure overall throughput of the complete system when operating multiple serial channels simultaneously at the maximum controller baud rate of 19.2 Kbaud. Based on the results of these throughput tests under worst-case loading, we believe the serial connection to the Galil controllers will provide the necessary bandwidth for our application. Operational tests using this system during PFCAM commissioning in March confirm this at least for the case of operating 3 stages under actual observing conditions.
The biggest issue with remote observing is the need to duplicate the shared guider console. This is not possible with the existing xguide/cam system.
This problem should be flagged as an action item to be addressed by the Lick/CARA collaboration that is working on the new guider system.
DEIMOS slit mask alignment requires 0.1" offsetting accuracy, which is not available on Keck-1. For Keck-2, an engineering test to demonstrate 0.1" offsetting accuracy should be written up and run during Keck-2 engineering time.
We do not know if CARA conducted this engineering test, and if so, the result. However, subsequent analysis of the offsetting accuracy required for DEIMOS slitmask alignment indicates that 0.1" is overly ambitious.
The planned multiple subraster feature will aid slit mask alignment. The GUI to control it will be a challenge and prototyping should begin ASAP...In addition, perhaps CARA should extend the field overlay popup in ``sky".
No prototyping of this GUI has yet occurred, nor has this activity been explicitly scheduled. This is a concern. However, the specification of the position of the alignment boxes should come directly from the slit mask design, and thus the observer should not need to interactively specify the location of the alignment boxes. This simplifies the GUI requirements considerably, since the main function needed is simply a toggle to switch between the normal readout mode and the alignment box readout mode. When a given mask is loaded into the spectrograph, its bar code will be scanned, which identifies the relevant slit mask design from which the alignment box positions can then be automatically obtained.
With regard to the extension of the field overlay popup in ``sky", this deserves further exploration with CARA, since the display of the alignment box positions with respect to the guider field could be quite useful. For this to occur, CARA will need to make some of Conrad's time available (preferably in Santa Cruz), but his time is typically in very short supply.
The delta PA during slit mask alignment should be negligible, although for unknown reasons this was not the case for LRIS on Keck 1.
It would be useful to hear from the CARA representatives or from Cromer whether this mystery has been resolved with the move of LRIS to Keck 2.
CCD Control:
As noted earlier, we have focussed on the DSP and low-level C code and are now successfully communicating with and operating the second-generation SDSU controller boards. This effort has proceeded better than expected, and reflects a significant improvement in robustness between the first and second-generation SDSU controller hardware.
Our current CCD controller design provides for a phased implementation. The first-phase yields the suggested configuration (single-amplifier readout from eight amplifiers via a single timing board) while allowing a future upgrade to dual-amplifier readout simply by adding addition controller boards and changing jumpers and moving interconnect cables between different connectors.
Based on latest recommendations we have received from Mark Sirota at CARA, it appears that the fiber-through-the-wrap noise problems that LRIS had experienced on Keck-1 have been resolved by the replacement of the 50-micron fiber in the cable wrap by 62.5-micron fiber. Both the first and second generation SDSU timing boards use 62.5-micron fiber, and the fiber in the Keck-2 cable wrap is 62.5-micron. Accordingly, we are still planning to locate the CCD VME crates in the control room, and to connect these to their corresponding SDSU timing boards within the DEIMOS instrument via the fiber through the Keck 2 cable wrap. In general, Sirota is recommending that we use fiber links through the cable wrap in place of other types of cabling wherever possible.
In addition, since we are currently planning on using 100-Mbit/sec twisted-pair fast Ethernet to connect the CCD VME crate to the instrument computer, and since the telescope cable wrap probably exceeds the practical length limit over which this type of fast ethernet can operate, this also argues in favor of locating the CCD VME crate in the control room. (However, if we were forced to move the CCD VME crate back to Nasmyth, we could switch from 100-Mbit/sec fast ethernet to a fiber-optic-based ethernet, although we would need to confirm that it would operate correctly over the 62.5 micron fiber in the telescope cable wrap.) Finally, since Sun has announced that they are withdrawing support for diskless clients after Solaris version 2.5, we are now planning to install a local SCSI disk drive within the CCD VME crate so that it can boot standalone rather than from the instrument computer. This provides yet one more reason for locating the CCD VME Crate in the control room rather than at Nasmyth focus.
Data Storage Formats:
DEIMOS does not plan to use datacube (i.e., NAXIS greater than 2) FITS representations for its mosaic images. The DEIMOS team has been participating in FITS standards discussions, mosaic image discussions, and the NOAO RTD effort in an attempt to ensure compatibility with other large detector array projects. We will attempt to extend the existing HIRES/LRIS mechanism for dealing with the overscan regions from each chip, but the detailed design for this is not yet complete.
Image Display and Quick Look Reduction:
SAOtng is no longer a primary contender for the DEIMOS image display. Keck Figdisp is display server we will be using for engineering tests, and our fallback for commissioning. While IDL-based display tools appear to be the choice of newer Keck IR instruments, there are currently no Keck optical instruments now in operation or planned for which IDL-based display tools are embedded within the data acquisition system. It is also unclear whether the Keck IR instruments will be using a common IDL-based tool, or several different tools which simply happen to have been built using IDL. As noted earlier, our survey of the DEIMOS user community did not elicit any obvious consensus that we should switch to IDL-based software either for slit mask design software or for Quick Look data analysis.
Information Management and Data Archiving:
Some discussion has taken place between CARA and Lick about the feasibility of a joint telemetry and archiving venture in support of not only DEIMOS but other Keck instruments. This discussion is not complete at this time.
A pilot effort succeeded in importing HIRES log data into the Sybase database at Keck. However, more polished end-user tools (for analysis, query, visualization) are needed to ensure acceptance and common use of such engineering databases. Should a joint venture as described above actually take place, development or acquisition of such tools would be a high priority.
George's comments on IRAF's present and future are noted, and were taken into account as the DEIMOS team pondered the inconclusive survey results mentioned above.
This initial response to the DEIMOS Software PDR Review Board Report was published as section 2.4 of DEIMOS Quarterly Report Number 8, which was issued in mid-July 1996. Although a more formal, detailed report (for which Kibrick was responsible) was planned at that time, it was never prepared. The first detailed response to the PDR Review Board Report is section 2 of this Appendix.
I have annotated this initial response to show how our statements, estimates, and projections (made in July 1996) compare with today's reality. (My annotations are italicized.) I believe this comparison provides a useful reality-check which should be used to calibrate the reliability of similar statements, estimates, and projections made in this CDR document.
In general, these comparisons show that the software tasks planned for Allen, Clarke, Phillips, and Tucker were completed as planned, but those planned for Kibrick slipped significantly due to competing demands from other projects as well as delays in the scheduled delivery of CCD controller hardware.
While a similar comparison of software projections versus reality could be made for the software sections of subsequent DEIMOS quarterly reports (numbers 9-11), I think the overall conclusions would be the same.
At the start of this quarter we received the review board's report for the DEIMOS software PDR that was held on March 22. The review board was generally satisfied with the preliminary design and the completeness of most of the material presented. The review board had specific recommendations addressed to a number of areas, and our efforts during the last quarter involved several of these:
Other review board recommendations will be addressed in our formal response to their report.
The scheduling of the DEIMOS software effort since the PDR has been primarily short-term, based on relatively close day-to-day coordination among the DEIMOS software staff. While we are very pleased with the software progress since the PDR, it is clear that longer-range planning of the DEIMOS schedule is still needed, and that close coordination with the ESI software scheduling is also required. Toward this end, Bigelow and Kibrick have met twice to iterate on the ESI schedule, and Kibrick and Faber have several meetings scheduled for late July to refine the DEIMOS software schedule. An updated long-term software schedule for DEIMOS will be released following those meetings. Note: Such an updated long-term software schedule was never prepared.
We are still planning on holding the software CDR near the end of 1996, and will try to have the actual date established by the end of the next quarter. Note: Due to late delivery of the second-generation SDSU CCD controller hardware, the software CDR was first delayed to March 1997, and then delayed again until June 1997 to insure that it occurred following the DEIMOS CCD Review.
This was identified as the highest risk area for the DEIMOS software effort, and the board recommended focusing on development of DSP and low-level code for the second generation SDSU CCD controller as soon as possible. We agree with this recommendation, and will commence work on the prototype DSP software as soon as receive the beta-test version of the second-generation CCD controller boards from SDSU. This hardware was successfully tested at SDSU during a Gemini-related review held on May 23, and it achieved a per-pixel readout time of 2.4 microseconds/pixel reading out a 1K x 1K CCD. Subsequently, some manufacturing problems were discovered involving the reliability of chip sockets and the soldering of surface mount components. SDSU is working with the board manufacturers and assemblers to resolve these problems. Expected delivery of this hardware has now slipped approximately one month, from mid-August to mid-September. Should this delivery slip further, we will likely begin preliminary tests of our fallback plan of using the first-generation SDSU CCD controller hardware. The test CCD dewar should be completed well before software testing begins, and several engineering grade, thick, 2K x 4K CCD Orbit CCD chips are available for testing in that dewar. Significant effort on design and prototyping of the DEIMOS CCD controller software will thus commence near the end of the next quarter.
Notes:
The board recommended putting more effort into early prototyping of DEIMOS user interfaces. We agree, and have begun assembling software toolkits to be used for this effort. We plan to use Tcl/Tk-based tools for UI prototyping, and numerous Tcl/Tk-related tools have been acquired, (ported where necessary), installed, and tested under Solaris 2.4.5. To support tests of these UI prototypes using existing Keck-style hardware and to also support the eventual development of the actual DEIMOS control and UI tasks, Clarke succeeded in porting the Keck Task Libraries (KTL), the complete HIRES keyword libraries, and Lupton's KTL-Tcl to Solaris 2.4.5. (The latter was also ported to Tcl version 7.5). This porting effort proved to be much harder than anticipated, due to a number of serious difficulties with the existing Keck-1-style make/build procedures provided by CARA as part of the KTL distribution, as well as the use of non-portable programming constructs within the KTL sources. As a result, we intend to develop alternative make/build procedures that will be easier to maintain and to use, and which will be portable between different operating systems, compilers, and hardware architectures. We also plan to follow the recommendation of the review board and convert to using CVS rather than SCCS (as was used for KTL and other Keck-1 software) for source code management. Note: In collaboration with CARA, we have developed these alternative make/build procedures.{
To confirm the integrity of this porting effort and to provide a proof-of- concept of using these Tcl/Tk-based tools to build GUIs for instruments, Clarke successfully used this ported software to develop a working, albeit rudimentary, GUI for the HIRES image rotator. This GUI was used to operate portions of the actual rotator hardware currently in Santa Cruz (awaiting shipment to Keck). The DEIMOS software team has also reviewed the existing HIRES rotator GUI (specified by Tytler and implemented at CARA using the commercial GUI-builder software ``Dataviews") and has identified a number of aspects of the design that can be improved for the DEIMOS GUI that will be used to control instrument rotation.
The review board expressed some concerns regarding our plan to use IRAF-based packages as our primary software for DEIMOS quick-look reduction and display, and suggested that we at least investigate other alternatives such as IDL. We prepared a questionnaire to better gauge the needs and preferences of prospective DEIMOS users, as well as the anticipated availability and support for IRAF and IDL as those users' sites. This questionnaire was distributed via e-mail at the end of the quarter to approximately 150 DEIMOS users at UC, CIT, UH, and NASA. The final date for the return of these questionnaires is August 1. (As of July 14, we had received 35 responses, but these have not yet been tabulated.)
Note: The results from the IRAF/IDL questionnaire were briefly described in DEIMOS Quarterly Report Number 9.
We have also completed a series of image display and quick look reduction tests on large images using IRAF, and have nearly completed corresponding benchmarks with IDL. Preliminary results do not indicate significant performance differences that would commend one package over the other.
The review board identified the need for close coordination between the DEIMOS team and CARA software developers with respect to several key areas:
During this past quarter, we put considerable effort into item C. Clarke and Kibrick visited CARA for several days in late April to work with CARA software staff on various DEIMOS-related issues and to develop a proof-of-concept of the utility of using a database package (i.e., Sybase) for managing and archiving both science images as well as engineering/ environmental data from instruments. (Clarke also watched Faber's LRIS observing run, and submitted a report to Faber in May describing various problems with the existing GUIs and observing procedures.) Clarke developed and installed a working prototype for ingesting the FITS headers from existing Keck science (and guider) images into a operational database. This tool proved sufficiently successful that it has been adopted by CARA as operational software, and the FITS headers of all of the HIRES, LRIS, and NIRC images that have been saved to Exabyte tape over the last several years via the STB backup software have now been stored into this database; headers for newly acquired images are being added to the database as part of the STB backup procedure. Using this database, relatively complex queries regarding the FITS headers of Keck images taken to date can now be rapidly answered.
Note: Clarke made a follow-up visit to CARA in January 1997 to make refinements to the STB header ingestion software and to provide training to CARA staff in the operation and maintenance of this suite of software.
Clarke also implemented and installed a prototype database for managing the HIRES engineering/environmental data, and all of this data for the last few years has been ingested. In addition, Clarke installed several Tcl/Tk-based GUIs to allow non-expert users to construct database queries to the FITS header database, and a prototype web-based UI for querying the database of HIRES engineering/environmental data. These tools have been well-received at CARA, although the web-based UI and other querry tools need further refinement.
These database prototypes represent a subset of the database implementation that we plan for DEIMOS. Their successful installation, operation, and reception at CARA serve to validate both the feasibility and utility of using database packages (such as Sybase) for managing these kinds of data from astronomical instruments, and represent the first step in developing at CARA the necessary infrastructure for supporting such database applications as we intend to deliver as part of the DEIMOS software.
The review board recommended that we allocate sufficient software effort early on towards defining and constructing an adequate software infrastructure to support DEIMOS software development. We agree, and during the past quarter considerable effort was devoted to acquiring, building, installing, and testing not only the GUI development tools noted above, but general-purpose software development tools including compilers (e.g., gcc), debuggers (e.g., gdb, ddd), source code utilities (e.g., tkdiff), code maintenance software (e.g., RCS and CVS), and Sybase-related tools. In addition, in response to the review board's recommendation that we not develop our documentation directly in HTML (as was done for our PDR documents), several documentation tools (tex, latex, latex2html) were installed which will allow us to develop documents using LaTeX which can then be translated automatically to HTML for posting to the web. CARA will also be providing to DEIMOS a version of the Frame package, including an option for generating HTML from Frame documents.
Note: DEIMOS stayed with the stated plan to use LateX for document generation, and have not actively used the Frame package provided by CARA for generation of software documentation.
Many of these general-purpose infrastructure tools were subsequently used for the porting and building efforts described above in the section on ``Early Prototyping". All of these tools are now operational under Solaris 2.4.5, and many are also operating under Digital Unix (formerly OSF) on the DEIMOS Dec Alpha test-bed machine, radec. Porting/installing the remainder of these tools to Digital Unix is planned for completion in the next quarter. Note: the porting of these tools across all of our supported platforms was successfully completed.
The review board recommended that we generate and experiment with simulated DEIMOS data early on. We agree, and plan to start generating such data towards the end of the next quarter. In preparation for this task, as well as to support the overall design and documentation effort, Clarke has designed and implemented a database for managing the large number of Keck keywords that will be used to control and document the operation of DEIMOS, and has developed an exhaustive specification of legitimate FITS keyword syntax. As a proof-of-concept, this database was successfully used to model all of the existing HIRES keywords. This database will be used to automatically generate HTML documents of keywords, as well as code to generate sample DEIMOS FITS headers.
In parallel, Allen has designed and documented all of the coordinate transforms needed to map between sky coordinates, focal plane coordinates, and both spectral and imaging pixels of the detectors of the mosaic. The FITS keywords and associated FITS extension tables for representing these transforms have been defined, and the means for representing these within the keyword database have been designed. Allen has also worked to finalize the coordinate system labeling and detector/amplifier numbering conventions, and was worked with Osborne to insure that the consistent use of these conventions on all DEIMOS engineering and mechanical design drawings.
Abstracts for two papers, which describe the keyword database and the coordinate transform designs, are being submitted for the Astronomical Data Analysis and Software Systems (ADASS) conference scheduled for late September. Note: These two papers were presented at the ADASS conference. On-line copies can be obtained from
http://www.ucolick.org/~sla/deimos/index.html
We anticipate that Clarke, Allen, and Phillips will continue working on DEIMOS at roughly the same level of effort during the next quarter. Clarke will be concentrating on overall data-flows and global design issues, definition of database schema, and GUI prototyping. An internal software review of the data-flows and database schema is now scheduled for July 30.
Allen will begin software prototyping of the coordinate transformations and their representation in FITS image headers, with the goal of generating simulated DEIMOS FITS images that include all of the relevant DEIMOS keywords and header extension tables. This simulated data will be used to evaluate various quick image display and data reduction alternatives. Allen and Clarke each plan to present a paper at ADASS.
Phillips will continue his efforts on slit mask design software, and will begin work on integrating into that software the coordinate transform and FITS header mechanisms that Allen has defined. Phillips and Allen will also spend some time evaluating the capabilities of IDL (and the user-contributed IDL astronomy libraries). We will try to arrange for Phillips to visit with at least one astronomy research group which conducts optical observations at Keck and which uses IDL as its primary quick look reduction tool. With assistance from the group, Phillips will test the ability of IDL and existing IDL astronomy packages to perform typical quick look image display and data reduction functions on existing LRIS multi-slit data.
Note: Allen, Phillips, and Clarke did work on these activities as planned, and the internal DEIMOS review was held on July 30.
The availability of Kibrick and Tucker to the DEIMOS software effort will be significantly constrained during the next quarter, due to vacations, HIRES image rotator installation and commissioning, several Keck AO CDRs (for which Kibrick is serving on the review board), and commissioning of the Lick Prime Focus Camera (PFCAM) on the Shane 3-m in late September. DEIMOS CCD software design and prototyping is now anticipated to commence near the end of the the next quarter (9), coincident with the expected delivery of the beta-test second-generation SDSU controller hardware. DEIMOS-specific motor control software effort will also commence at that time, building on common motor controller software developed for PFCAM.
Note: The availability-constraints imposed by the HIRES image rotator upgrade and the subsequent retrofit of the Keck-2 Telescope Drive and Control System (DCS) onto Keck-1 in reality stretched out through early March 1997, i.e., well in quarter 11. CCD software prototyping did not commence until late March 1997.
During the next quarter, we anticipate purchasing (with DEIMOS funds) an additional 4 GB of disk to support Solaris-based software development for DEIMOS, as well as a laptop computer (running Linux) that will be split-funded with DEIMOS, ESI, and UCO/Lick funds. We are currently obtaining pricing for Dec Alpha memory, with the goal of adding an additional 256 MB to the DEIMOS test-bed machine, radec, this fall to support tests involving simulated DEIMOS images. We plan to complete our formal response to the PDR review board report during the next quarter, following the tabulation of results from our IRAF/IDL questionnaire, and Phillips' evaluation of IDL capabilities.
Note: the proposed hardware was acquired, except the RAM upgrade for radec was increased to bring the total memory to 1GB. The laptop has proved to be extremely valuable to our porting efforts, as well as to the productivity of our visits to Keck for collaborations with CARA staff. The formal response to the PDR review board report was not completed as planned, although the results from the IRAF/IDL questionnaire were briefly described in DEIMOS Quarterly Report Number 9.