Running the pipeline#
After installation of both pycrires and EsoRex (see Installation section), we are now ready to start processing CRIRES+ data with the pipeline!
Downloading data#
First, we need to download a dataset by using the CRIRES Raw Data Query Form that accesses the ESO Science Archive Facility. It is recommended to process data that has been obtained with a single observing block (OB), which comprises typically one hour of telescope time.
There are various ways to search for the data with the query form. For example, we can search for the science data (i.e. setting DPR CATG to SCIENCE) of a specific target and observing night. After searching, the relevant data files can be selected and requested from the archive. Then, on the next screen, the associated raw calibration files can be selected. Finally, the data can be either directly downloaded as ZIP archive or through a shell script.
Important
It is important to download the Associated raw calibrations (i.e. not the processed calibrations) from the ESO archive. Processing of the calibration data occurs by the pipeline while the use of processed calibrations is currently not supported.
After downloading the raw data, the compressed FITS files (i.e. ending with .fits.Z) should be uncompressed (e.g. with uncompress in a terminal on macOS). The folder that contains the raw data should be renamed to raw, which is a requirement for initializing the working environment in the next section.
Processing data#
The pycrires package provides a Python wrapper for the EsoRex recipes of CRIRES+ and MolecFit. The pipeline functionalities are all embedded in the Pipeline class.
The instance of Pipeline requires a path in which the raw folder is located. This path will be used as working environment in which all the pipeline data will be stored, such as input and configuration files for the recipes and the calibrated data products.
After creating a Pipeline object, we can run all the methods for processing both the calibration and science data. Each method calls a single recipe of the EsoRex pipeline and produces output which is then typically used by the next method.
A configuration file is generated when calling a Pipeline method for the first time. If needed, the configuration file can be adjusted, which will then be used when rerunning the same method.
Below, there is an full example for reducing and calibrating the CRIRES+ data.
Tip
The pipeline will automatically download any missing calibration files (e.g. dark frames with specific DIT). After downloading and uncompressing, it is important to rerun the pipeline such that the calibration files are included with the data reduction.
Tip
Set verbose=True to print the output from EsoRex to check for potential issues with the reduction.
import pycrires
pipeline = pycrires.Pipeline('./')
pipeline.rename_files()
pipeline.extract_header()
pipeline.cal_dark(verbose=False, create_sof=True)
pipeline.util_calib(calib_type='flat', verbose=False, create_sof=True)
pipeline.util_trace(plot_trace=True, verbose=False, create_sof=True)
pipeline.util_slit_curv(plot_trace=False, verbose=False, create_sof=True)
pipeline.util_extract(calib_type='flat', verbose=False, create_sof=True)
pipeline.util_normflat(verbose=False, create_sof=True)
pipeline.util_bpm_merge(verbose=False, create_sof=True)
pipeline.util_calib(calib_type='une', verbose=False, create_sof=True)
pipeline.util_extract(calib_type='une', verbose=False, create_sof=True)
pipeline.util_genlines(verbose=False, create_sof=True)
pipeline.util_wave(calib_type='une', poly_deg=0, wl_err=0.1, verbose=False, create_sof=True)
pipeline.util_wave(calib_type='une', poly_deg=2, wl_err=0.03, verbose=False, create_sof=True)
pipeline.util_calib(calib_type='fpet', verbose=False, create_sof=True)
pipeline.util_extract(calib_type='fpet', verbose=False, create_sof=True)
pipeline.util_wave(calib_type='fpet', poly_deg=4, wl_err=0.01, verbose=False, create_sof=True)
pipeline.obs_nodding(correct_bad_pixels=True, extraction_required=True, check_existing=False, verbose=False, create_sof=True)
pipeline.run_skycalc(pwv=1.)
pipeline.plot_spectra(nod_ab='A', telluric=True, corrected=False, file_id=0)
Tip
The obs_nodding method has a long runtime with extraction_required=True. In case the 1D spectra from its output are not required, it is recommended to set extraction_required=False, for example when a 2D extraction will be done with custom_extract_2d (see below). The extracted spectra by obs_nodding should be ignored in that case since they will have a low accuracy. That is fine, because the extraction will be done separately with the dedicated method that maintains the spatial dimension.
Next, for spatially-resolved targets (e.g. directly imaged exoplanets), there are dedicated methods for extracting 2D spectra (so maintaining the spatial dimension):
pipeline.custom_extract_2d(nod_ab='A', max_separation=2.0)
pipeline.fit_gaussian(nod_ab='A', extraction_input='custom_extract_2d')
pipeline.correct_wavelengths_2d(nod_ab=nod_ab, input_folder='fit_gaussian',
collapse_spatially=True, collapse_exposures=True, accuracy=0.02)
pipeline.correct_wavelengths_2d(nod_ab=nod_ab, input_folder='correct_wavelengths_2d',
collapse_spatially=True, collapse_exposures=True, accuracy=0.005)
pipeline.correct_wavelengths_2d(nod_ab=nod_ab, input_folder='correct_wavelengths_2d',
collapse_spatially=True, collapse_exposures=True, accuracy=0.001)
pipeline.remove_starlight(nod_ab, input_folder='correct_wavelengths_2d',
telluric_mask=(0.6, 1.6), svd_broadening_kernel=True)
pipeline.remove_systematics(nod_ab, n_modes=5, input_folder='remove_starlight')
model_flux, model_wavel = pycrires.load_bt_settl_template(t_eff=1500.0, log_g=4.0, vsini=10.0)
pipeline.detection_map(model_flux, model_wavel, rv_grid=np.linspace(-100, 100, 201), vsini_grid=None,
nod_ab=nod_ab, input_folder='remove_systematics', error_weighted=False)
Or, for unresolved targets (e.g. transiting exoplanets), the 1D spectra are already extracted by the obs_nodding method so we only need to apply the additional wavelength correction:
pipeline.correct_wavelengths(nod_ab='A', create_plots=True)
pipeline.plot_spectra(nod_ab='A', telluric=True, corrected=True, file_id=0)
pipeline.export_spectra(nod_ab='A', corrected=True)
Finally, for removing any intermediate data products and freeing up some disk space:
pipeline.clean_folder(keep_product=True)