Rocket I
S
D
A
T

Page maintained by
Reine Gill

The Isdat tutorial.

Author: Tobias Eriksson
Official url of this page: http://www.space.irfu.se/isdat/using/Getting_started.html


Prerequisites

This paper assumes that you already have built the isdat clients to use. They are built the same way as the server but with only clients and isdatbase checked out from the CVS. You also need a working Isdat server Server to contact, either locally on your machine or remote on any machine on your network. You don't need access to any data since we use the test database that is available in all Isdat servers.

To set these things up please refer to the documentation in the source code release. Specially the INSTALL file in the isdat root directory and various papers in the doc directory.


What is Isdat?

Isdat (current version 2.7.1) is a client-server application to access high resolution time series data from different space missions. The server part (data base handler) keeps track of all data and unpacks new data files as they are added into the database. The client part fetches data from the server and presents them to the user. The main advantage of this design is that all data is only kept in one place in the institution (or on the Internet) reducing maintenance, improving data integrity and allowing fast data distribution to all users.


The Clients

At the moment there are 16 clients in the Wec Isdat distribution. All of different status and functionality, some (like igr) are graphical applications intended to display and perform various calculations on the data. This type of clients are started from ctm (Client and Time Manager) who's main function is to manage the clients and to help you select a start time and a stop time and provide this to the clients so they know which data to fetch.

Another class of clients is the so called API:s. These have only one sole use, to provide a bridge between the Isdat server and third party software such as Matlab and IDL. With the help of these handy clients you are free to do almost any calculations and visualizations on the data in the Isdat database.


Getting started using iscmd.

iscmd is a simple and flexible way of getting data out of the Isdat Database. It is a simple shell program that takes a number of command line arguments and displays tabulated data on standard output so it can piped to any other command line tool or simply being displayed on the screen. iscmds status is someware between an stand alone client and a API for the shell programing language. You can use it both ways and thous it makes a good introduction to the Isdat system.

In order for iscmd (and any other Isdat client) to work there are some things to set up. First your shell must find the iscmd program. When Isdat compiles it does not copy or link its executables to any of the folders were the shell normally looks for programs. This has to be set up manually by setting a number of shell environment variables. Our first variable, ISDAT_HOME, makes sure that the Isdat clients can find themselves. It must point to the base of the Isdat installation. If you downloaded Isdat by CVS and installed it under /usr/local then typing:

ISDAT_HOME=/usr/local/isdat_root; export ISDAT_HOME;

would do the job. Now we can use the ISDAT_HOME variable to extend the PATH variable in a correct way.

PATH=$PATH:$ISDAT_HOME/bin; export PATH;

It's important that the $PATH: string is included on the right side of the equality sign. Otherwise the shell won't be able to find any of the normal UNIX commands any longer. The final step is to tell iscmd were to look for its missing half, it must find the Isdat server. This is done by setting ISDAT_DATABASE. In order to do this you must know on witch machine the server is running and witch Isdat database number it has assigned to it. When you have this information proceed with:

ISDAT_DATABASE='computername':'DBnumber'; export ISDAT_DATABASE;

Now you should be ready to run Isdat clients. Naturally all this is not what you would like to type into your computer every morning you come to work. And of course there is a simple way to get around it. In the Isdat distribution there is a skeleton script for setting all environment variables needed. Copy it to your home directory:

cp $ISDAT_HOME/scripts/my_isdat_profile ~/.my_isdat_profile

or if you are running c-shell:

cp $ISDAT_HOME/scripts/my_isdat_profile.csh ~/.my_isdat_profile

Now edit this file according to the above recipe and include the following row in one of your startup files.

. ~/.my_isdat_profile (bash, The first dot is important.)
source ~/.my_isdat_profile (c-shell)

For Bash users a typical startup file is ~/.profile and for c-shell users ~/.cshrc. But beware there are numerous exceptions to this.

ls -A --ignore=*

Will give you a listing of all your "dot-files" in Linux, on other os ls -a will have to do. If you don't have any of the above files then check your local documentation for your correct startup file.

With all this done you are finally ready to start looking at data. iscmd has four modes of operation. Simply typing iscmd at the prompt will give you an short help message together with an listing of the command line options for the different modes. The first option to use is q, it stands for query. It's purpose is to let you explore the structure of the isdat data tree. In many ways this tree reminds of the file system of your local hard drive, it has a root from were everything originates as the directory's in a file system. Let us call Isdats "directory's" for logical instruments. The big difference in this comparison is that when you can put datafiles in any directory. Isdat data only shows up in the leaf of the data tree. A Isdat logical instrument can only contain data or other logical instruments not both.
Thats the theory, let us now explore.

iscmd q /

Will give you an listing of the databases on your Isdat server. There will be one entry called Test and if you are working in Wec most probably another one called Cluster. To explore further you can use the slash '/' separator between entry's. Thous

iscmd q /Test

will give you the next lever in the Test logical instrument tree. Give yourself some time to familiarize yourself with your local Isdat data tree. When you encounter an logical instrument that is empty it probably isn't. Try

iscmd c 'your search path'

to view the contents of the data set in this logical instrument. Although, there is a hatch here. Even though iscmd c reports that there should be data in a specific time interval it's not guaranteed to be there. This is be course the Isdat server only looks for the existence of datafiles and does not check their contents when reporting data available. This is an issue for future Isdat versions.

Now let us look at a specific event. In the Test database there is 100 sec of an built in square wave signal. It is accessed by:

iscmd d /Test/*/wave/square/wf 1993-08-01 15:10:00.0 100

After a small delay you will see a lot of data pass through the screen. If you want to view it, pipe it to less.

iscmd d /Test/*/wave/square/wf 1993-08-01 15:10:00.0 100 | less

Now you will be able to see the top of the dataset. It tells you a great deal about the data, its place in the Isdat database, its type, Isdat server version and so on. Just before the data there is an row describing the type of data. In this case Time and Volt. The time is relative from the requested start time. Time is the main key that all data in Isdat is ordered after so this row will always be there. The unit of next column will of course vary depending on the requested data, for some measurements there will be several columns since these are multi dimensional.

Let's examine the above command row more carefully. The d option tells iscmd to ask the Isdat server for data and if there is any iscmd delivers it to standard out. Second we have the full qualified logical instrument path to the data. Third the start time for the wanted data set and last the duration of the data set in seconds. iscmd is a bit strict about the format of the start time it should be:

YYYY-MM-DD hh:mm:ss.nsec

Where MM is the two digit number for the month and mm two digit number for the minute. The only optional part of the above time is the nanoseconds.

An alternative way of calling iscmd is with the isdat url syntax. In that case the database is added before the isdat logical instrument path, thous eliminating the need of the ISDAT_DATABASE shell variable. In that case the above command would look like:

iscmd d isdat://localhost:10/Test/*/wave/square/wf 1993-08-01 15:10:00.0 100

if the Isdat server is running on a machine called localhost on DB port 10. For now iscmd is the only isdat client that supports this syntax.

Well asci data might not be very informative, piping it to xgraph might give us a better show.

iscmd d /Test/*/wave/square/wf 1993-08-01 15:10:00.0 100 | xgraph

xgraph is avaliable for download from http://www.cs.toronto.edu/~radford/xgraph.html

Installation is straight forward, at least on Suse Linux, un tar source package, go there, type:

xmkmf; make; make install; make install.man;

I really recommend this small handy program.


Lesson two, use Igr.

One of the most common ways of accessing the isdat database is through the Igr client. To start Igr type ctm &. This will start the client and time manager. Your first quest will be to select the same data set as used with iscmd. On the left side on the ctm panel there is three drop down boxes. Project, Member and Instrument. These corresponds to the first three levels of the Isdat logical instrument tree. The default settings might be Cluster, 1, efw. If so, change to Test, * and wave. Next to select a time interval you use the start, stop and interval boxes on the right. We will continue to work with the above data (1993-08-01 15:10:00.0 interval 100s). You will now discover that to type text into isdat text field you must always have the cursor positioned over the text field, for people used to the Windows environment this can be a bit confusing in the beginning. Next start igr from the drop down menu, Clients -> General -> Igr.

Igr is a client program for plotting and looking at Isdat data, once you got a grip of the main idea the program it gets quite easy to use. The two abstract main components to work with is quantities and plots. A quantity is a storage box (named q0 - q132) into whom you can put a data set from any of the logical instruments. You can then make a number of operations and calculations on these quantities. Finally you bind the result to a named plot for displaying your result. There are two control panels for doing this tasks. First the quantity dialog (Select -> quantity, or Ctrl-Q), second the Calculator. At any moment just one quantity and one plot can be active, activating an quantity means that it's possible to fil it with data. The active quantity and panel are displayed right under the drop down menus.
Use the quantity dialog to select a quantity that data will be put into, q0 will do fine. The param menu contains the various logical instruments in Test, click param -> * -> wave -> square -> wf to fill q0 with square data for the period you just specified in ctm. Now to display the data we use the calculator to bind q0 to a plot panel. Type p0y = q0 'return' in the text field directly under the drop down menu in the calculator. You will see p0y = q0 being added to the big box below. This a simple formula telling igr that the y-axel of plot p0 should be given the values of q0 (p*x also exist, it defaults to the time of the data set). It is possible to write more complex formulas using several quantities, variables and the built in formulas like despin and fft.
After defining a formula for a plot the way of displaying it is to click with the right mouse button in the corresponding plot, this has to be done every time a formula or quantity has been changed. If you have room on you desktop keep the calculator open, this will help you keep track of what your plots actually contains.

It is possible to add more plot panels to igr. Graphics -> panel -> create will show up a dialog box with buttons for each one of 30 possible plot panels. The first panel corresponds to the top one on the left side. Pressing one of the highlight buttons will add one panel in that place. Click so you have four panels avaliable for plotting. In the right upper corner of each plot there is four buttons, think of these as shortcuts to a plot and quantity. When you click one of these you will change both the current plot and quantity. For example, if you press the last button in the left upper plot you will activate plot 3 and quantity 3. Let's read in some data in q3. Now you will see a new row being added to the calculator. Right click in the panel to see the plot. This is the fast way if you just want to look at raw data.


More complicated plots!

Let us now try to make a fft spectrogram on our event. This is a more complicated operation and will require several steps. First exit igr and goto Options -> clean client start in ctm, relaunch igr. This will assure that all quantities and plots are cleared (the default behavior of igr is to start in the same state as it was stopped in). Now select quantity 0 (Select -> quantity) and fill it with square test data (Test -> * -> wave -> square -> wf). In the drop down menu Operator you will see igrs built in operators. To make a fft spectrogram plot we will use the fft operator. The fft operator takes some optional additional arguments that customize its behavior. This is done by marking the used operator with the mouse and then Clicking Edit -> Operator. In this case we want to tell the fft operator to sweep over the signal and transform it in many small overlapping pieces. In the options dialog box change mode to Manual, fft size to 64, real samples to 64 and shift to 32 also change spectra to All. Don't press the apply button in the calculator at this stage this would erase those fft options just chosen and restore the default ones. Instead press left mouse button in panel p0 to view the results. As you see there are now many transformed signals in the same plot. To view all these as a spectrogram instead just Click Graphics -> Plot -> Spectra. If you feel happy with your plot and would like to save it to a file or print it. Choose File -> print, here you have the option to save the panel to a postscript file or a gif file or simply to print it.

For comments, suggestions etc.

reine.gill@irfu.se