CDS Data

This is a massive undertaking!

The general procedure is to download a cdrom from the sohoftp website. This can take 20 - 30 minutes (depending on data volume). The telemetry (tm) files are generated from this using the xcdrom gui. The following step involves creating fits files with fitsgen.


Getting xcdrom data from SOHO's ftp site

Use the routine /home/master/cds_ftp/cds_get_data.pro to download cdroms. This involves a 2-pass wget procedure (because the default maximun depth for wget is limited to 5 - which isn't enough).

CIDL> add_path,"$HOME/cds_ftp",/expand
CIDL> .compile cds_get_data (optional)
CIDL> cds_get_data , cd_num = '5774'

Downloads the xcdrom data from; ftp://sohoftp.nascom.nasa.gov/pub/data/telemetry/LZ/SO_ALL_LZ_5774/. It is saved in the directory; /data4/level0/LZ/SO_ALL_LZ_5774/.


xcdrom GUI Settings

Note: There is a bug in the routine "load_cdrom.pro" that validated a cdrom record had a valid date - this was set to the end of 2011. Hence cdroms after this date would cause the xcdrom tool to hang. However, Bill Thompson sent an updated version which removed this problem. Since it is not yet on the ssw distribution on the solar machine, it has to be compiled manually to override the existing version (the version of the routine was incremented from v13 to v14). Until this makes its way in the ssw distro, use this IDL command before running xcdrom;
CIDL> .compile load_cdrom
This assumes you're running CIDL from the home directory - which is where the updated version resides.

The following sets up the required environment variables and invokes the gui;

CIDL> setenv , 'CDS_CDROM_DATA_W=/disks/newsolar3/newcdsfits/tm'
CIDL> setenv , 'CDS_CDROM_DATA=/disks/newsolar3/newcdsfits/cdrom/SO_ALL_LZ_5774'
CIDL> xcdrom , drive = '/disks/newsolar3/newcdsfits/cdrom/SO_ALL_LZ_5774/'

Press the following buttons in sequence;

  1. Register
  2. Find gaps
  3. Load


Inventory of RAL /cdsfits/arch1 to arch32 directories

qlds = readcdsfits('s12345r00')

Acquired using the standard command; $ls arch34 | wc -l

Directory Sym Link "" "*.Z" "*.fits"
arch1 - 5000 5000 0
arch2 - 3239 3239 0
arch3 - 2963 2963 0
arch4 - 2291 2291 0
arch5 - 1546 1546 0
arch6 - 2595 2595 0
arch7 - 1425 1425 0
arch8 - 3022 3022 0
arch9 - 7684 7684 0
arch10 - 2074 2074 0
arch11 - 2795 2795 0
arch12 - 3290 3289 0
arch13 - 1762 1762 0
arch14 - 1976 1976 0
arch15 - 3926 3926 0
arch16 - 2595 2595 0
arch17 - 2152 2152 0
arch18 - 1400 1400 0
arch19 - 1837 1836 0
arch20 - 2730 2730 0
arch21 - 2144 2144 0
arch22 - 2250 2250 0
arch23 - 2736 2736 0
arch24 - 1787 1787 0
arch25 - 9549 9549 0
arch26 - 9355 9355 0
arch27 - 29730 29730 0
arch28 - 21655 21655 0
arch29 - 94391 94390 0
arch30 - 33430 24339 9089
arch31 /cdsfits2/arch31 35766 0 35766
arch32 /data3/arch32/ 25030 0 25029
Total 324125 254235 69884

The arch30 problem!

Using IDL routine "/home/master/cds_ftp/cds_num_files_in_arch_folders.pro". However, there are duplicates. For example; "s35243r42.fits" and "s35243r42.fits.Z" in "/cdsfits/arch30". Up until "arch30" (it looks like) all files are zipped (*.Z) - and then after that (arch31 and arch32) they are not zipped.