Easy INTEGRAL Data Analysis

Last updated: 2005-03-10

INTEGRAL data anlsyis can be performed with ISDC's Off-line Science Analysis (OSA) package, and the data analysis manual is provided by ISDC.

In this page, we explain an alternative and hopefully easier way to analyze INTEGRAL ISGRI and JEMX data. Still we are dependent on OSA, but we run OSA components differently from standard OSA methods (i.e., running og_create, ibis_science_analysis and jemx_science_analysis).

We provide simple c-shell scripts isgr_scw_analysis.csh and jemx_scw_analysis.csh, in which we call ISDC and ftools directly to perform necessary tasks, rather than through "ISDCroot scripts". People who prefer shell scripting to GUI interface might find our scripts useful.

OSA installation

Obtain OSA package from http://isdc.unige.ch/index.cgi?Soft+download. Binary installation is easiest. If you choose source installation, I suggest installation without CERN ROOT. From time to time, ROOT compilation can be tricky, and ROOT is used in OSA only for GUI and ISDCroot. You can analyze INTEGRAL data without ROOT.

To configure without ROOT, "ac_stuff/configure --without-cern-root". Also, do not forget "unsetenv ROOTSYS", if you had accidentally set up ROOTSYS environmental variable. However, when doing "source $ISDC_ENV/bin/isdc_init_env.csh", ROOTSYS environmental parameter may be required (this shall be fixed soon). Untill this problem is fixed, you can set just a dummy value, such as "setenv ROOTSYS abcdefg" or whatever.

Install the IC (Instrument Characteristic) package and the catalog following the standard procedure. Currently, you need to install *all* the IC and catalogs, even if you would analyze only one or two instruments. SPI and OMC occupy most volumes of the IC and catalog data, while many people will analyze only ISGR and JEMX. Soon, this requirement of the entire IC will be mended, and you will be required to install only ISGRI and JEMX related callibration files to analyze ISGRI and JEMX data.

LHEASOFT/FTOOLS

I assume you have the latest LHEASOFT/FTOOLS package installed.

Event Data Download

You may obtain the INTEGRAL data as a PI, but let's consider the case that you are going to use archival data. You may search for and download archival data through ISDC browse. Or, if you already know the data you want, you may directly access the anonymous ftp directory. I will explain the latter method; this method has an advantage that you can download only files you really need for you analysis.

For example, I would like to download only "scw.fits" (Science Window index file) and jemx and isgri related files for all the Science Windows (ScWs) in Revolution 55.

  cd /local/data/subaru4/ebisawa/INTEGRAL/scw/0055
  lftp -c "open ftp://isdcarc.unige.ch/arc/rev_2/scw/0055 && mirror -I *0.001/j*fits* -I *0.001/i*fits* -I *0.001/swg.fits* -X *[12].001/"
lftp is a very useful tool for this kind of purpose. In this example, I downloaded only files which are necessary to analyze ISGRI and JEMX from pointing Science Windows (i.e., excluding slews). Note, pointing Science Window names always end with *0.001 (001 is the processing version number, which should be 000 for the first processing), so I excluded directories ending with 1.001 or 2.001.

Auxiliary Data Download

Some auxiliary data are needed and should be put in the proper directory.
  mkdir -p  $REP_BASE_PROD/aux/adp/0055.001
  cd $REP_BASE_PROD/aux/adp/0055.001
  lftp -c "open ftp://isdcarc.unige.ch/arc/rev_2/aux/adp/0055.001 && mirror -I attitude* -I orbit* -I time* -X logs/"
  mkdir -p  $REP_BASE_PROD/aux/adp/ref
  cd $REP_BASE_PROD/aux/adp/ref
  lftp -c "open ftp://isdcarc.unige.ch/arc/rev_2/aux/adp/ref  && mirror -I *"

Analysis of a single Science Window data

After properly downloading ISGRI and JEMX data, You need to set up several environmental variables. Here is my example. Of course, actual values will be dependent on your environment.
  setenv REP_BASE_PROD /local/data/subaru4/ebisawa/INTEGRAL
  setenv ISDC_REF_CAT "/local/data/subaru4/ebisawa/INTEGRAL/cat//hec/gnrl_refr_cat_0017.fits[1]"
  setenv ISDC_ENV /local/data/subaru4/ebisawa/INTEGRAL/osa_sw-4.2
  source $ISDC_ENV/bin/isdc_init_env.csh
  setenv COMMONLOGFILE +common_log.txt
The last COMMONLOGFILE setting is not required but useful to keep a log with the name "common_log.txt".

Then, modify the initial part of isgr_scw_analysis.csh and jemx_scw_analysis.csh to specify the Science Window data location. After that, in any directory, run these scripts. Several OSA executables are executed, and you will see the following files in the current directory.

  GNRL-OBSG-GRP.fits              INDEX_j_ima_cor_intensity.fits    isgri_mosa_res.fits              jmx2_rsti_ima.fits
  GNRL-SCWG-GRP.fits              INDEX_j_ima_shadowgram.fits       isgri_pif.fits                   jmx2_sky_ima.fits
  INDEX_ii_cat_extract.fits       INDEX_j_ima_src_find.fits         isgri_sky_ima.fits               jmx2_srcl_arf.fits
  INDEX_ii_shadow_build.fits      INDEX_j_src_spectra.fits          isgri_sky_res.fits               jmx2_srcl_bsp.fits
  INDEX_ii_shadow_ubc-BIN_I.fits  isgri_catalog.fits                isgri_spectrum.fits              jmx2_srcl_cat.fits
  INDEX_ii_shadow_ubc-BIN_S.fits  isgri_catalog.reg                 isgri_srcl_res.fits              jmx2_srcl_cat.reg
  INDEX_ii_skyimage.fits          isgri_cor_shad.fits               isgri_srcl_res.reg               jmx2_srcl_res.fits
  INDEX_ii_spectra_extract.fits   isgri_detector_shadowgram.fits    isgri_srcl_res_significant.fits  jmx2_srcl_res.reg
  INDEX_j_bin_evts_spectra.fits   isgri_efficiency_shadowgram.fits  jmx2_evts_shd.fits               jmx2_srcl_spe.fits
  INDEX_j_cat_extract.fits        isgri_grou_shd.fits               jmx2_find_res.fits
  INDEX_j_ima_basic_recon.fits    isgri_mosa_ima.fits               jmx2_full_dsp.fits
Those INDEX*fits files are temporary index files which are needed to run ISDC tasks. Files which have names like i*fits and j*fits are the output files. They are identical to what you get by running the standard ISDC tools ibis_science_analysis and jemx_science_analsys. Note, user GTI selection capability has not implemented in these c-shell scripts. You will have to implemnet "gti_merge" task for this purpose (or, just use standard ISDC programs ibis_science_analysis or jemx_science_analysis).

Combine refults from many Science Windoes

isgr_scw_analysis.csh and jemx_scw_analysis.csh provide a simple way to analys a single ScW data. You may run these scripts on many Science Windows (you can use "foreach" loop), after that you can combine Science Window results. You can use ISDC tools spe_pick, lc_pick, image_mosaic or ii_skyimage for this purpose. We will also provide simple scripts to combine many Science Window results by calling these ISDC tools.

Simple Mosaic with "varmosaic" ftool

To create precise mosaic images from many Science Windows, you will have to take into account of several instrumental specific things, such as background correction and vignetting correction. Tools to take care of these effects shall be provided by ISDC. Meanwhile, the "varmosaic" generic ftool provides a simple way to make a mosaic image from many INTEGRAL Scinece Windows.

Standard Scinece Window images are already created by ISDC pipe-line, and found in the ISDC archives. You may download these images, and run varmosaic. We will explain this method below.

To obtain all the JEMX2 Science Window images of Revolution 55:

  lftp -c "open ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001 && mirror -I ssj2_00550???0010/scw/005500???010.001/jmx2_sky_ima.fits.gz"
Now you have the following files under the current direcotry:
   ssj2_005500020010/scw/005500020010.001//jmx2_sky_ima.fits.gz
   ssj2_005500030010/scw/005500030010.001//jmx2_sky_ima.fits.gz
   ssj2_005500040010/scw/005500040010.001//jmx2_sky_ima.fits.gz
   ...
   ssj2_005500900010/scw/005500900010.001//jmx2_sky_ima.fits.gz
   ssj2_005500910010/scw/005500910010.001//jmx2_sky_ima.fits.gz
   ssj2_005500930010/scw/005500930010.001//jmx2_sky_ima.fits.gz
You can check the pointing directions of these images:
  foreach file (*/scw/*/*fits.gz)
  set RA  = `fkeyprint ${file}+2 CRVAL1 | grep Sky | awk '{printf("%10.3f\n",$3)}'`
  set DEC = `fkeyprint ${file}+2 CRVAL2 | grep Sky | awk '{printf("%10.3f\n",$3)}'`
  set TSTART = `fkeyprint ${file}+2 TSTART | grep Start | awk '{printf("%15.3f\n",$3)}'`
  echo $file $TSTART $RA $DEC
  end

  ssj2_005500020010/scw/005500020010.001/jmx2_sky_ima.fits.gz 245.613 -60.150
  ssj2_005500030010/scw/005500030010.001/jmx2_sky_ima.fits.gz 241.603 -60.315
  ...
  ssj2_005500630010/scw/005500630010.001/jmx2_sky_ima.fits.gz 253.820 -31.727
  ssj2_005500640010/scw/005500640010.001/jmx2_sky_ima.fits.gz 256.097 -33.165
  ssj2_005500650010/scw/005500650010.001/jmx2_sky_ima.fits.gz 147.064 -48.030
  ssj2_005500660010/scw/005500660010.001/jmx2_sky_ima.fits.gz 146.925 -47.900
  ..
  ssj2_005500910010/scw/005500910010.001/jmx2_sky_ima.fits.gz 39.422 67.267
  ssj2_005500930010/scw/005500930010.001/jmx2_sky_ima.fits.gz 39.422 67.267
You might also want to check the scheduling information at http://www.rssd.esa.int/integral_webapps/index.jsp?past?0?2253

OK, we decide to make a mosaic image from the GCDE pointings. We choose the following Scinece Window image files, and put the list in the "listfile"

  ssj2_005500480010/scw/005500480010.001/jmx2_sky_ima.fits.gz
  ssj2_005500490010/scw/005500490010.001/jmx2_sky_ima.fits.gz
  ssj2_005500500010/scw/005500500010.001/jmx2_sky_ima.fits.gz
  ssj2_005500510010/scw/005500510010.001/jmx2_sky_ima.fits.gz
  ssj2_005500520010/scw/005500520010.001/jmx2_sky_ima.fits.gz
  ssj2_005500530010/scw/005500530010.001/jmx2_sky_ima.fits.gz
  ssj2_005500540010/scw/005500540010.001/jmx2_sky_ima.fits.gz
  ssj2_005500550010/scw/005500550010.001/jmx2_sky_ima.fits.gz
  ssj2_005500560010/scw/005500560010.001/jmx2_sky_ima.fits.gz
  ssj2_005500570010/scw/005500570010.001/jmx2_sky_ima.fits.gz
  ssj2_005500580010/scw/005500580010.001/jmx2_sky_ima.fits.gz
  ssj2_005500590010/scw/005500590010.001/jmx2_sky_ima.fits.gz
  ssj2_005500600010/scw/005500600010.001/jmx2_sky_ima.fits.gz
  ssj2_005500610010/scw/005500610010.001/jmx2_sky_ima.fits.gz
  ssj2_005500620010/scw/005500620010.001/jmx2_sky_ima.fits.gz
  ssj2_005500630010/scw/005500630010.001/jmx2_sky_ima.fits.gz
  ssj2_005500640010/scw/005500640010.001/jmx2_sky_ima.fits.gz
We run varmosaic:
  
  varmosaic listfile jemx2_rev55_gc.fits
Now you see a rather smooth jemx mosaic image as shown below. GX349+2 is clearly seen. This is the Primary of the file, which is the Significance map of the first energy band (varmosaic creates mosaics of significance map, intensity, variance, and exposure map).

Note, ftools/cfitsio has a nice capability that they can access remote files directly through Intenet. Consequently, you do not have to donwload invididual Scinece Window images, but specify the remote files at ISDC archives directly in the "listfile".

For example, you may specify the following in the "listfile",

  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500480010/scw//005500480010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500490010/scw//005500490010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500500010/scw//005500500010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500510010/scw//005500510010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500520010/scw//005500520010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500530010/scw//005500530010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500540010/scw//005500540010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500550010/scw//005500550010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500560010/scw//005500560010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500570010/scw//005500570010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500580010/scw//005500580010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500590010/scw//005500590010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500600010/scw//005500600010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500610010/scw//005500610010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500620010/scw//005500620010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500630010/scw//005500630010.001/jmx2_sky_ima.fits.gz
  ftp://isdcarc.unige.ch/arc/rev_2/obs_jmx/0055.001/ssj2_005500640010/scw//005500640010.001/jmx2_sky_ima.fits.gz

then varmosaic opens and reads these remote files on the fly. You will get exactly the same image above as when local files are input.