This document describes the procedure used to reduce the XTE clock data, and produce a set of clock coefficients that can be used for analysis of high precision timing data. The procedure uses IDL in several stages, however no programming knowledge of IDL is needed to go through the pipeline.
The overall procedure is:
You should be able to match the following procedures to this flow diagram:
There are several requirements that must be met before this procedure can be started.
Here is a more detailed description of these files.
IDL access - you must have access to IDL 5.1 or higher in the lab. This is usually accomplished by having code like this in your .cshrc file which sets up default paths and environment variables for IDL:
if (-e /software/IDL/idl.5.1/idl_setup) \ source /software/IDL/idl.5.1/idl_setup limit datasize unlimited
This second command is needed because of the large dataset that needs to be worked with.
Markwardt programs in IDL path - you must have access to the Markwardt IDL programs, and they must be at the front of your IDL path. The easiest way to do this is to place the following command after the above IDL setup command:
setenv IDL_PATH +/home/lhea/craigm/lib/idl:${IDL_PATH}
Markwardt apstract.pl - this Perl script is optional. It allows one to quickly extract the relevant apid files from the XTE archive and copy them into a comprehensive directory structure.
Raw clock offset data - the raw clock offset data has filenames of the form "FPclock...", and is available from the SOF. The original USCCS files supplied by the MOC must be processed by XFF to make FITS files. You must have this data in a directory accessible to you. Each of the clock files is ingested into one single clock "summary" file.
Clock frequency telemetry data - the clock frequency data is stored in XTE apid 0x7b, and has files of the form FH7b_???????-???????. They must be extracted from the XTE archive, or from a daily SOF cron job. These files are ingested into one single clock frequency "summary" file.
Clock offset "summary" file - the "summary" file is a history of the clock offset data stored in one FITS file. This is an intermediate product generate by HKSUMM and used by the clock fitting application. The procedure to generate and update this file are described below. The typical name for this file is clockdt0.fits.
Clock frequency "summary" file - this is the equivalent summary file for the clock frequency data. The typical name for this file is clockfs0.fits. This file is used to create or update the clock settings file.
Clock settings file - this is an intermediate file derived from the clock frequency summary file. This ASCII file contains a list of mission times, and frequency settings. The typical name for this file is clockset.dat.
Previously fitted coefficients - you need at least one set of already fitted coefficients to begin. When you fit additional data the new coefficients will be appended to the old ones, and a new file will be created. The typical name for this file is clockfit-YYYY-MM-DD.fits, where YYYY/MM/DD is the date of generation.
Any "fudged" clock data - this file, fudgeclock.dat, is optional. If there is any data which does not appear in the MOC-generated files, then it must be entered by hand in this file.
You must first get the raw data from the SOF or the XTE archive. The raw data includes the clock offset data (FPclock), and the clock frequency settings (FH7b). Craig has automatic scripts that run in the SOF which extract this data, but they appear in different places.
The clock offset data is computed from the MOC-supplied files. A daily cron job converts the clock data to FITS format. A date check is applied to be sure the file name is correct (XFF does not do this). The files are deposited in /socops/FITS/craigm/clock/FITS.
The clock frequency data is generated from the XTE science data and placed in daily real-time summary directories with many other apids. The files can be found in /socops/FITS/craigm/hk/D????-??-??-??R/D????-??-??-??R/spsdu/FH7b*.
These files should be copied over to a work area accessible to you on your own computer. Alternatively, if you have access to PCALPH1 in the lab, you can go to /xtepca/p07/hk and /xtepca/p12/craigm to find them, however these paths will change soon.
Creation of the summary files is a one-time only procedure (or, at worst, once per year). It involves starting IDL and running the following commands.
tlimits = [63071940D, (2002 - 1994)*365.25d*86400d] hksumm, 'clockfs0.fits', /create, hktype='FREQSTD', tlimits=tlimits, timedel=600D hksumm, 'clockdt0.fits', /create, hktype='CLOCK', tlimits=tlimits
This makes two summary files for the clock frequency and offset data. The time range is from the beginning of 1996 to 01 Jan 2002, but it should be easier to see how to make a larger file by replacing 2002 with a later year.
If at a later date you wish to enlarge an existing summary file, then use the same commands, but compute a larger value of TLIMITS, and replace the /CREATE keyword by the /GROW keyword:
year = 2003 ;; 01 Jan 2003 tlimits = [63071940D, (year - 1994)*365.25d*86400d] hksumm, 'clockfs0.fits', /grow, hktype='FREQSTD', tlimits=tlimits, timedel=600D hksumm, 'clockdt0.fits', /grow, hktype='CLOCK', tlimits=tlimits
The precise time limits do not matter, but making a ten year file can use a lot of disk space.
The summary files in principle need only to be created once. After creating the files, you must populate them with data. The commands are very similar, but you must supply a directory where the data can be found. You need to update this data every time you wish to analyze clock data.
The clock offset data is updated in the following way.
datadir = '/xtepca/p07/hk' ;; Reset to appropriate directory hksumm, 'clockdt0.fits', /merge, hkdir=datadir+'/clock/FITS'
The clock frequency data is updated with these commands. The commands are slightly more complicated because the Unix "find" command was used to first locate only the apid FH7b files.
datadir = '/xtepca/p07/hk' ;; Reset to appropriate directory spawn, 'find clockfs -follow -name "FH7b*" -print', infiles hksumm, 'clockfs0.fits', /merge, hkfiles=datadir+'/'+infiles
Be aware that the HKSUMM program will only ingest new files. It keeps track of all of the files that ingests, and does not duplicate any work. Therefore, it is safe to give the same directories every time.
The next step is to take the clock frequency summary file and combine it with the existing clockset.dat file. The purpose of the clockset.dat file is to list every clock frequency transition during the mission. This data is used to match discontinuities in the clock offset data. If you receive notification from the XTE MOC, you can enter the settings by hand into this file. The format of the file is one line per clock frequency change, and the columns are: TIME FREQSTD FREQOFF, where TIME is the time of measurement, either in mission days, in MET, or expressed in the "SOF" format (any format will work); FREQSTD is the frequency standard setting (an integer from 2300 to 3500); and FREQOFF is the TIMEZERO offset.
The clock frequency summary file contains a list of every recorded clock frequency setting, not just the transitions. The VERICLOCK IDL program is used to cross-check the summary file and clockset.dat, to be sure that they are both consistent.
When you run the program, it will probably flag the fact that new frequency settings have been entered since the last run. It will suggest any new entries that must be made (or corrections). Here is how to run the program:
vericlock, setfile='clockset.dat', fsfile='clockfs0.fits'
And the typical output will look like this:
Reading clock frequency settings...done
Reading frequency standard telemetry...done
WARNING: the following times were not found in clockset.dat
2844.73264 3011 3.37843132
2852.60069 3010 3.37843132
2853.74653 3009 3.37843132
2875.69097 3012 3.37843132
Unless there is something wrong, you should be able to cut and paste these lines into clockset.dat directly.
You may manually add clock data if it doesn't appear in the telemetry. An example of this is the year 2000 rollover where a bug in the XTE MOC prevented the clock data from being provided. The MOC operators computed a few sample points manually, and those points are entered into fudgeclock.dat. Normally you will not have to adjust any of the values in fudgeclock.dat.
The format of the file is one line per value: TIME, OFFSET (usec), ERROR (usec). Here is the current contents of fudgeclock.dat, for the year 2000:
2194.812 +40.0 0.01 2196.694 +49.6 0.01 2201.874 +65.0 0.01 2203.926 +73.0 0.02
The first column is mission day, the second column is the time offset in microseconds, and the third column is the uncertainty value. I have set the errors smaller than their formal values, because otherwise the other data in the same interval dominates the fit.
The more or less final stage is to fit the clock data using all of the files that you have created so far. The command sequence should look something like this:
clockfit, setfile='clockset.dat', dtfile='clockdt0.fits', fitfile='clockfit-2001-12-16.fits'
The SETFILE keyword indicates the file containing the clock frequency settings, and should normally be 'clockset.dat'. The DTFILE keyword indicates the summary file containing clock offset data, and should be what you named it above in Creating the Summary Files. The FITFILE keyword indicates the name of the file containing the polynomial coefficients from the previous fitting session. Usually the FITFILE keyword is the only keyword which changes from one session to the next.
This procedure will read the clock summary files (both the offset and frequency standard files) and display the results graphically. The clock offset describes the difference between the spacecraft timestamped Mission Elapsed Time (MET) and a ground-based ephemeris time.
The application should look something like this.
Most of your attention will be devoted to the upper graphics panels and the "Fit" button. The upper panel is shows the clock offset data, and the current model. The middle graphics panel shows the residuals and a +/- 2 usec band. The bottom widgets allow you to navigate and control the interface.
The ideal situation is where the clock offset is zero. However, the XTE clock is known to have a non-linear frequency drift with time. Given a long enough period of time, the clock offset would drift far from the true value.
The user has the option of fitting the resulting data to a parameterized curve. The curve is assumed to be a piecewise continuous quadratic polynomial. By default, continuity of the function value itself is guaranteed. The FS settings also determine the amount of any discontinuities in the function derivative. Thus, the only free parameters are the clock second derivative values.
The user should first perform a crude fit of the most recent data
by pressing the ![[FIT]](gif/fit1but.gif)
button.
In the next step the user should perform a more refined fit. This
is accomplished by partitioning the data more finely. First, select
the ![[Partition]](gif/partbut.gif)
radio button, and then
partition the data using the mouse in the data or residuals
panels. The behaviors of the mouse buttons are:
Generally speaking, new partitions should be added at points of inflection in the residuals. When the new partitions have been added, the user should perform one last "FIT" before finishing up.
There are two important things to remember when fitting. First, the user should include a few data intervals preceding the new data. Even though a coefficient is local to a particular time segment, because of the continuity relationship, a coefficient will end up affecting its neighboring segments as well.
Second, the user *MUST* remember to fit the data all the way to the end of the most recent data. That means that if the user changes the fitted coefficients from the earliest part of the mission, they must go through the entire data set and refit everything. Before quitting the user should be warned if this hasn't been done.
It should be possible to achieve a precision of +/-2 microseconds.
The bottom bar below the plot windows are used to navigate through
the mission timeline. It looks something like this:
 |
 |
|
| Adjust Left | Shift Left / Right | Adjust Right |
The "<" and ">" buttons nudge the boundaries of the plot left or right respectively. The amount of "nudging" is to the next frequency setting interval.
The "<--" and "-->" buttons shift the entire display one screen to the left or right, including some data for continuity.
The user can also fill in the desired left or right boundaries by hand using the text boxes.
Two "save" options exist, as shown by the above image. The first button produces a FITS file with all of the fitted polynomial coefficients. This file is needed the next time CLOCKFIT is run, so it is vitally important to save your work here before quitting.
It is also to generate a "tdc.dat" type file, which can be used as input to the other high precision timing XTE tools.
Here are some things to keep in mind before and after a clock fitting session.
clockfit-2001-12-16.fits - (FITS) fitted clock coefficients - valid up to Mission Day 2800.
clockset.dat - (ASCII) clock frequency settings - valid up to Mission Day 2875.
fudgeclock.dat - (ASCII) manually entered data - valid for Y2K problem.
tdc-2001-12-16.dat - (ASCII) automatically generated tdc.dat file (based on clockfit file above).
In order to get started I have put all of the files and scripts needed into one directory. Here is how it works:
The canned directory I have provided supplies a directory called "clockfs" which contains the mission history of FH7b files up to mission day 2865. These are fmerge'd to save space. The clock offset data is kept in the "clockdt" subdirectory. All of your work will be in the clockproc/ directory itself. It is okay to merge the data from this directory with the SOF data retrieved at a later time.
Here are abbreviated procedures that summarize the detailed steps above.
clockfit, setfile='clockset.dat', dtfile='clockdt0.fits', fitfile='clockfit-YYYY-MM-DD.fits'
This whole procedure should take about 5-10 minutes.