WECdata7

WECdata meeting Ångström Laboratory, Uppsala, 2001 November 14 Minutes compiled by CCH
Present :		Tobia Carozzi, Toby Champion, Tomas Chust, Yvonne de Conchy, Michel Dedieu, Anders Eriksson, Tobias Eriksson, Emanuel Le Guirrec, Christopher Harvey, Per-Arne Lindqvist, Bjorn Lybekk, Milan Maksimovic, Jean-Gabriel Trotignon, Simon Walker
This meeting took place in the middle of the week of WEC ISDAT demonstrations and hands-on problem solving organised in Uppsala. The prime purpose of the meeting was to evaluate ISDAT V2.7.2 and decide: what corrections and improvements can be incorporated in release V2.8 what improvements are needed but cannot be incorporated before V2.9 During the course of the meeting it was decided that V2.8 should be released on November 23 2001, and V2.9 towards the end of 2001. The following points were covered: 1 - Current Status 1.1 - ISDAT 1.2 - WEC server 1.3 - EFW 1.4 - STAFF-SA 1.5 - STAFF-SC 1.6 - WHISPER 1.7 - DWP 1.8 - WBD 1.9 - EPHEMERIS 1.10 - IGR 1.11 - CSDS data 1.12 - IDFS converter 1.13 - Flatfile interface to ISDAT 1.14 - CIS 2 - Future Developments 2.1 - Documentation 2.2 - The TWiki tool 2.3 - Other Developments 2.4 - Distribution Policy 2.5 - Miscellaneous 3 - New Releases 3.1 - ISDAT 2.8 3.2 - ISDAT 2.9 4 - Actions 5 - Splinter Meetings 1 - Current Status 1.1 - ISDAT Have integrated the latest TED version 2.4.3. This includes fixes for timing issues "as an extra functionality", and has been compiled on several systems. The next step is to see how this new functionality can be used. In the absence of a TED representative, there was some uncertainty as to what the new features actually do, and why they are optional. It was recalled that it has always been foreseen that it will be possible to enhance the relative timing precision for inter-spacecraft correlations, but that the increase in precision requires : additional data from DWP which requires careful preparation and will not be provided on a regular basis, and an additional dbh computation effort which is not justified except for high time resolution inter-spacecraft correlations. A splinter group was set up to address timing issues: discover what the new TED does reiterate the timing issues, perhaps by updating Simon Walker's document "WEC Timing", CL-DWP-TN-0002, Issue 0, Rev. 2 dated 31st March 1998. examine how optional timing enhancements should be implemented, for example, by having a flag to indicate that higher timing resolution exists, and a button to select its use. This group should meet on Friday, November 16. To facilitate configuration control, Simon Davis should be made familiar with ISDAT. Toby indicated that he may be able to give Simon a hand. PAL has noted a timing problem in the ESOC data. Successive packets should be separated by 5.15 s, but there are times when the inter-packet separation is significantly different. Sometimes extra packets are inserted between packets uniformly separated by 5.15 s. This problem was first communicated to ESA during commissioning, but it has been noticed recently that it still exists, with disastrous consequences for the science, especially with respect to inter-spacecraft correlation. PAL will contact ESA about this. 1.2 - WEC server The limitation of not being able to plot over NM/BM (normal mode / Burst Mode) boundaries still exists. ISDAT returns only data from the mode in which the experiment was at the start of the selected period. Burst memory dumps occur at least once per day. Anders Lundgren has agreed to fix this, but it cannot be done in time for V2.8 ; but every effort will be made to fix it for V2.9, this will discussed with Anders Lundgren. Error messages should be made more specific and made available for propagation to higher levels. The client software should be modified so that at least the user's current configuration can be saved when the server goes down. 1.3 - EFW The EFW dbh is relatively stable in the sense that it does not crash. The handling of continuous time series data is complete, but burst and sweep mode data tools need further development. Calibration in the frequency domain has not been implemented. The transfer function is relatively flat and Fourier transformation to the frequency plane as done for STAFF-SC is overkill. A simple solution is to offset the data sample time-tags by the group delay time, which is approximately 2 1/2 data samples (in both NM and BM). This is equivalent to transforming to the frequency domain, correcting for the phase but not the amplitude of the transfer function (which is nearly flat), and transforming back to the time domain. This modification will be incorporated into ISDAT V2.8. 1.4 - STAFF-SA A logical instrument for the complex cross-spectral matrix has been created, as has a new instrument which includes all 27 frequencies of the cross-spectral matrix (this facilitates the plotting of full 27-frequency spectrograms). The real and the imaginary parts of this matrix can be recovered by using the line command "iscmd". The generalisation of the operators REAL and IMAGINARY to the STAFF-SA data, although difficult, was eventually incorporated. It is not possible to produce spectrograms for one component of the cross-spectral matrix of level 1 data, nor is it possible to obtain the spectrograms for the real and the imaginary parts of elements of the cross-spectral matrix of level 1 data. What is really needed is a polarisation operator for STAFF-SA, which produces physically meaningful parameters which can be plotted as spectrograms. Milan Maksimovic noted, however, that the necessary development effort is difficult to justify now because such a tool, PRASSADCO, has already been produced, independently of ISDAT, by Andrei Santolec when at LPCE. YdC has noted a problem when restoring a saved configuration. The time selection is not saved; worse, after restoring the configuration, it is necessary to enter the time manager to select the desired period, then enter IGR and restore the configuration a second time. It works, but involves lots of unnecessary clicking. There is a need to implement dynamic use of STAFF calibration server. The STAFF calibration software presently looks at a calibration file whose version number is "hard-coded" into the STAFF software. This is unsatisfactory: STAFF software should use the most recent calibration file. 1.5 - STAFF-SC The dbh is now working as intended, the error of sign error has been corrected and the results are now correct. The server presently requires the presence of a file called "infile_dbh" situated in the users home directory. This file should be moved elsewhere, preferably by V2.8. 1.6 - WHISPER No problem other than the "internal error" message which occurs from time to time. And the plotting problem is an IGR problem, should check whether or not the data is segmented, then use the option 600 columns is it is not segmented. Whisper wants to plot active and passive spectra in the same panel. Presently IGR can plot up to four time series in the same panel, but can plot only one spectrum (when it's variation with frequency, not time). Also, WHISPER would like to zoom the spectrum in the frequency direction. MM would like to plot a WHISPER spectral density at a single frequency as a function of time. 1.7 - DWP People are now using the dbh, most of which is working well. 1.8 - WBD There is no data yet. There have been a lot of problems with DSN data acquisition, and the ISDAT format has not yet been established. But it is anticipated that data distribution will start very soon. 1.9 - EPHEMERIS Five new logical instruments have been added : LT, MLT, L-shell, inv-lat and cov-mtx. The ephemeris module requires a FORTRAN compiler, and the site.def file must be changed to indicate that this compiler exists. SW is starting to insert the covariance matrix calculations which appear on the CD-ROM. The L-shell depends upon the magnetospheric conditions, and the module assumes that the magnetopause boundary is at 11 R_E, to be compatible with the JSOC value. PAL says that he found that this module was not easy to install; he had a problem with the spin phase. SW replied that an old bug seemed to have been re-introduced due to confusion in the use of CVS system, but he will sort this out rapidly. It was emphasised that CVS is a development tool: no development is done on the trunk, which is characterised by periodic releases of tar files. Users who are not active software developers should install each release (c.f., very few tar files were shipped for V2.7.2) if they want to do science with the best available version of ISDAT. 1.10 - IGR Most of IGR is working satisfactorily in its current version, 3.2. New features include : labels below the axes flatfile format when there are several plots. Still not so easy to get the data into Matlab, may be fixed later can replace top text by your own text. can reduce size of PostScript files by reducing resolution new operators, for example averaging and inverse cosine calculator updated to perform addition, subtraction and multiplication of spectra yesterday's bug concerned motif windows, a small bug next release should include new requirement from WHISPER next release will include "png" output in order to be able to include figures in PowerPoint presentations. In Sussex IGR is used to browse the data, and Matlab to do the analysis. The next development is to make the operators accessible to the APIs. In other words, it should be possible to obtain data through an operator, the present getdata( ) obtains data only directly from the corresponding dbh. Tobia will look into this. 1.11 - CSDS data There is presently a problem when creating the index files for CSDS data files, the procedure crashes on Solaris systems. Also, it will not work with CDF V2.5 (the version used by ISTP), but it does work with V2.6 which is the official version of CDF for Cluster. 1.12 - IDFS converter Pavel has the IDFS source code, but he is not allowed to make it available; he can, however, make binaries available to WEC members. SwRI does not maintain rigorous version control, but Pavel does. PEACE still has problems with calibration, but some data products are available. There are very few IDFS value added-files available in Sussex. These files can be directly used by Pavel's IDFS format converter. The Sussex group can take responsibility for the ISDAT interface to PEACE, possibly even providing a PEACE ISDAT server. CCH mentioned that David Winningham has mentioned the desire "to have a WEC science interface". This does not exist, it is up to interested non-WEC scientists to contact the WEC PI concerned ; PAL will contact David Winningham. 1.13 - Flatfile interface to ISDAT This uses the same ASCII format as "IGR" and "ISCMD", but it is not yet functional. It is planned that metadata will be handled via an XML-type markup. 1.14 - CIS CCH mentioned that Pavel Travnicek developed a CIS interface when he was in ESTEC. CCH will investigate the current status of this software. 2 - Future Developments 2.1 - Documentation It was noted that WECdata is the application of ISDAT to WEC. There is an ISDAT documentation tree, but there is now an urgent need for WECdata documentation, especially a WECdata "how-to" document designed to help scientists who are not members of the WEC data working group. The documentation tree should include : Instrument parameter (i.e., logical instrument) description Experience (good and bad) gained during earlier installations Tutorials List of known bugs Validated IGR configuration files Description of the batch operating mode It may also be useful to update documents on the ISDAT server, such as "The Structure of the WEC/ISDAT Data", CWD-OBSPM-DD-001, Issue 1, Rev. 6, dated 1994 March 24 "WEC Timing", CL-DWP-TN-0002, Issue 0, Rev. 2 dated 31st March 1998 The different sections will be shifted to the ISDAT distribution, from where the installer should make them available to his users and provide them with the high level entry URL 2.2 - The TWiki tool Toby Champion suggested the use of TWiki, a Web-based groupware application to help prepare the documentation with a minimum of effort. The TWiki home page is http://www.twiki.org/ Toby and Tobias will work together to install this system on the IRF-U Web server at http://space.irfu.se/isdat/twiki. Meanwhile, that address contains a link to Toby Champion's TWiki installation, which it was agreed to use until the IRF-U installation is completed. WEC members can access this page with User : guest Password : isdat Here you can find an explanation of how to use the system. To find WECdata documents, follow the link from the first page of this site, then the link to the topic WecDataMeeting. 2.3 - Other Developments ISDAT will be used in Uppsala for Rosetta and Mars Express. These missions will use PDS headers. ISDAT may be used to access all data from other PDS missions. There is also activity with the frame of EuroGRID. 2.4 - Distribution Policy The tar files are the primary method of distribution. CVS is for the developers. When a new distribution comes out, the tar file should be installed. The distribution procedure should be secure, to make sure that nobody gets a dbh server by default. 2.5 - Miscellaneous If you have overlapping DDS and CD-ROM data, you should destroy the DDS data. There should be a WEC data "hands-on" session in ESTEC at the time of the workshop in March 2002. It may also be useful to have a WEC ISDAT demonstration ofr PIs and other scientists not directly involved in the WEC data working group. Perhaps the simple use of ISDAT during the workshop would be sufficient. For points 2 and 3 an adequate external Internet connexion is required in ESTEC. 3 - New Releases 3.1 - ISDAT 2.8 ISDAT V2.8 should be released at the end of next week (November 21). It should be possible to eliminate most known bugs by then, except the NM/BM problem. This needs the collaboration of Anders Lundgren, and will be postponed until V2.9. Anders Eriksson has provided Anders Lundgren with example test data. The following fixes should be included in V2.8. This list was noted during the meeting, but a more detailed list was prepared using the TWiki tool later during the week. The latter list can be consulted at the URL http://space.irfu.se/isdat/twiki/. On this TWiki server each bug has been assigned an ID in the form of a character string which is both descriptive and uniquely identifies it. These IDs do not appear on the following list because, at the time of the meeting, they did not exist. EFW time fixes STAFF spectral matrix elements IGR WHISPER segmentation problem IGR several spectrum plots zoom into spectrograms coordinate transform if it is fixed despin in with caveat ephemeris bug removed resample operator if bug can be fixed new TED get rid of the strange STAFF-SC file change default access control try to eliminate CSDS index generation crashes include data dependent coordinate system transform, with a caveat saying it is not tested. CT for tensors (later) when despinning several satellites gets mixed up CT messes up STAFF-SC data default time format in IGR list of available operators in IGR help help now works for labels possibility to include user-defined colour scales label error PS output of labelling has wrong scaling Try to recover as much info as possible when dbh crashes resample operator Real and Imaginary operators to work on tensors arg(z) operator exists (atan2) frequency scale will become available on first IGR update WECHK server index generation problem corrected 3.2 - ISDAT 2.9 This version should be released before Christmas, and include : Seamless crossing of NM/BM file boundaries averaging operator for spectra EFW Langmuir probe sweep timing EFW current sweep V is factor 8 too low All playbacks are not blanked burst timing wrong Offsets not handled in optimum way (2.9 or later) V_sc and spin fits should be available in IGR Many useful parameters should be made available in IGR 4 - Actions
No.	For		Description
1	TCh, TC		Familiarise Simon Davis with ISDAT
2	PAL		Inform ESA/ESOC of the timing problem in the ESOC data
3	AE		Solve the NM/BM boundary problem with the help of Anders Lundgren
4	STAFF		Implement dynamic use of calibration files
5	STAFF-SC		Eliminate need for the "infile_dbh" file in the users home directory
6	TC		Explore possibility to obtain API data via an operator instead of directly from the dbh
7	Sussex		Follow the interface between IDFS and ISDAT
8	CCH		Discover status of CIS dbh
9	All		Contribute text and validated IGR configuration files to the WEC ISDAT "user-friendly" documentation
10	CCH, SW		Update documents CWD-OBSPM-DD-001 and CL-DWP-TN-0002
11	All		Install new versions (starting with V2.8) from the tar file, to ensure ISDAT-wide coherence (CVS is only for active development work)
12	All		As part of each ISDAT installation, make the associated user-friendly documentation easily available to the end user via a high-level URL.
13	TC		Verify ISDAT software distribution procedure
14	AE		Organise a "hands-on" ISDAT session in ESTEC before or after the workshop planned during the week starting March 4, 2002
15	AE		Request adequate external Internet connectivity to support this activity
16	All		Implement the bug fixes for V2.8, and developments for V2.9
5 - Splinter Meetings WEC data group members should try to discuss the following points during the meeting. N.B., this list is NOT exhaustive ! Timing: what does the latest TED release actually include how does WBD enhance the relative timing precision it improves the known published ESOC experimental errors can it correct the unexplained ESOC timing blunders Error tracing Operator to interpret STAFF-SA data STAFF-SA restore-configuration problem Remove requirement for anomalous STAFF-SC file in the user's home directory Demo of DWP wave/particle correlator dbh Documentation and user-friendliness WG CVS management Coordinate transforms Resampling operator : segmented data, Bartlett window, the IGR joining operator, Definition of data-related coordinate systems Problem of crashing when indexing CSDS PP files

1 - Current Status

1.1 - ISDAT

1.2 - WEC server

1.3 - EFW

1.4 - STAFF-SA

1.5 - STAFF-SC

1.6 - WHISPER

1.7 - DWP

1.8 - WBD

1.9 - EPHEMERIS

1.10 - IGR

1.11 - CSDS data

1.12 - IDFS converter

1.13 - Flatfile interface to ISDAT

1.14 - CIS

2 - Future Developments

2.1 - Documentation

2.2 - The TWiki tool

2.3 - Other Developments

2.4 - Distribution Policy

2.5 - Miscellaneous

3 - New Releases

3.1 - ISDAT 2.8

3.2 - ISDAT 2.9

4 - Actions

5 - Splinter Meetings