Current file format (2017+)

Image data saved into a CXI format compliant file (*.cxi)

Processing results are saved into a separate HDF5 file (*.h5)

Events are synchronised between both files (same stack position in both files corresponds to the same event)



Results files are much smaller than CXI files because they do not contain the entire image for each event:

-rw-rw----+ 1 barty xu       13G May 15 02:39 mfxn1116-r0040-c00.cxi

-rw-rw----+ 1 barty xu      269M May 15 02:39 mfxn1116-r0040-c00.h5



This has several benefits:

  1. Benefits of a small number of large files are retained

  2. Results file can be more flexibly defined to suit FEL data without having to force data into the Nexus standard

  3. Results and metadata can be worked on without needing to copy the (large) .cxi file

  4. Image file complies more closely to standard CXI and Nexus files



HDF5 is a well supported industry standard container data format that can be directly read by any decent scientific data analysis software. 


Useful:

  1. > h5ls -h

  2. usage: h5ls [OPTIONS] file[/OBJECT] [file[/[OBJECT]...]

  3. ....

 

REsults file (.h5)

Broadly speaking:

    /instrument/...    = data read directly from the instrument (encoder values, gas detectors, photon energy, etc)

    /event_data        = results which appear for each event (peak lists, radial average stack, etc)  

    /run_data            = results accumulated across the entire run (detector sums, virtual powder, pixel histograms)  

    /cheetah/...         = things pertaining to cheetah internals (configuration, threadID, event number, etc)



Data locations are designed to be discoverable using the h5ls command, for example:

    > h5ls -r cxij4716-r0059-c00.h5/instrument

    /detector0               Group

    /detector0/EncoderValue  Dataset {524/Inf}

    /detector0/position      Dataset {524/Inf}

    /ebeamCharge             Dataset {524/Inf}

    /ebeamL3Energy           Dataset {524/Inf}

    /ebeamLTUAngX            Dataset {524/Inf}

    /ebeamLTUAngY            Dataset {524/Inf}

    /ebeamLTUPosX            Dataset {524/Inf}

    /ebeamLTUPosY            Dataset {524/Inf}

    /ebeamPkCurrBC2          Dataset {524/Inf}

    /eventTimeString         Dataset {524/Inf}


    > h5ls -r cxij4716-r0059-c00.h5/run_data

    /detector0               Group

    /detector0/class1        Group

    /detector0/class1/mean_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}

    /detector0/class1/mean_non_assembled_detector_corrected Dataset {1480, 1552}

    /detector0/class1/mean_radial_average_detector_and_photon_corrected Dataset {1187}

    /detector0/class1/mean_radial_average_detector_corrected Dataset {1187}

    /detector0/class1/sigma_non_assembled_detector_and_photon_corrected Dataset {1480, 1552}

    /detector0/class1/sigma_non_assembled_detector_corrected Dataset {1480, 1552}

    /detector0/class1/sigma_radial_average_detector_and_photon_corrected Dataset {1187}

    /detector0/class1/sigma_radial_average_detector_corrected Dataset {1187}


Or using an interactive HDF5 file browser as shown below:

 

Image data is saved in the format of the Coherent X-ray Imaging Data Bank (CXIDB) which is loosely based on NeXus.

A small amount of results are saved so as to not immediately break compatibility with CrystFEL




 

> h5ls -r mfxn1116-r0040-c00.cxi

/                        Group

/LCLS                    Soft Link {instrument}

/entry_1                 Group

/entry_1/data0           Soft Link {/entry_1/instrument_1/detector_1}

/entry_1/experiment_identifier Dataset {1959/Inf}

/entry_1/instrument_1    Group

/entry_1/instrument_1/detector_1 Group

/entry_1/instrument_1/detector_1/data Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/data}

/entry_1/instrument_1/detector_1/description Dataset {SCALAR}

/entry_1/instrument_1/detector_1/detector_corrected Group

/entry_1/instrument_1/detector_1/detector_corrected/data Dataset {1959/Inf, 1480, 1552}

/entry_1/instrument_1/detector_1/detector_corrected/experiment_identifier Soft Link {/entry_1/experiment_identifier}

/entry_1/instrument_1/detector_1/detector_corrected/mask Dataset {1959/Inf, 1480, 1552}

/entry_1/instrument_1/detector_1/detector_corrected/mask_shared Dataset {1480, 1552}

/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_max Dataset {1480, 1552}

/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_min Dataset {1480, 1552}

/entry_1/instrument_1/detector_1/distance Dataset {1959/Inf}

/entry_1/instrument_1/detector_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}

/entry_1/instrument_1/detector_1/mask Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask}

/entry_1/instrument_1/detector_1/mask_shared Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared}

/entry_1/instrument_1/detector_1/mask_shared_max Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_max}

/entry_1/instrument_1/detector_1/mask_shared_min Soft Link {/entry_1/instrument_1/detector_1/detector_corrected/mask_shared_min}

/entry_1/instrument_1/detector_1/x_pixel_size Dataset {1959/Inf}

/entry_1/instrument_1/detector_1/y_pixel_size Dataset {1959/Inf}

/entry_1/instrument_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}

/entry_1/instrument_1/source_1 Group

/entry_1/instrument_1/source_1/energy Dataset {1959/Inf}

/entry_1/instrument_1/source_1/experiment_identifier Soft Link {/entry_1/experiment_identifier}

/entry_1/result_1        Group

/entry_1/result_1/data   Soft Link {peakTotalIntensity}

/entry_1/result_1/nPeaks Dataset {1959/Inf}

/entry_1/result_1/peakMaximumValue Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakNPixels Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakSNR Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakTotalIntensity Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakXPosAssembled Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakXPosRaw Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakYPosAssembled Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/peakYPosRaw Dataset {1959/Inf, 1024/Inf}

/entry_1/result_1/powderClass Dataset {1959/Inf}

/instrument              Group

/instrument/detector_1   Group

/instrument/detector_1/EncoderValue Dataset {1959/Inf}

/instrument/detector_1/SolidAngleConst Dataset {1959/Inf}

/instrument/detector_1/position Dataset {1959/Inf}

/instrument/eventTimeString Dataset {1959/Inf}

/instrument/fiducial     Dataset {1959/Inf}

/instrument/machineTime  Dataset {1959/Inf}

/instrument/photon_energy_eV Dataset {1959/Inf}

/instrument/photon_wavelength_A Dataset {1959/Inf}

 

IMage data file (.CXI)