CrystFEL

Information

Documentation

partialator: scaling and post-refinement of partial reflections

Synopsis

partialator -i input.stream -o output.hkl -y pointgroup [options] ...
partialator --help

Description

partialator merges reflections by scaling and post refinement, accounting for the partialities of the reflections. That means that it models the geometry of diffraction for each pattern (crystal orientation, unit cell parameters, X-ray bandwidth and so on) and attempts to optimise the geometrical parameters to make the fully integrated intensities calculated using the model agree as closely as possible between the many patterns.

See the usage cases below for examples of commands to merge reflections in different ways, for example with and without partiality or scaling.

In addition to the output reflection list, partialator will write a file called partialator.params which contains the scaling factors determined for each crystal. After each iteration, a file will be written called pgraph-itern.dat which contains the observed and calculated partialities for a randomly chosen set of "free" reflections which were not included in the refinement.

partialator will also write datasets merged using two halves of the total data using the same filename as you give for -o or --output, with 1 and 2 appended. For example, partialator.hkl1 and partialator.hkl2. Compare these two files with each other to calculate figures of merit such as Rsplit and CC1/2. See the section on Custom Dataset Splitting for more discussion on this topic.

Options

-i filename --input=filename

Give the name of the input stream.

-o filename --output=filename

Give the name of the output file. The default is --output=partialator.hkl.

-y pointgroup --symmetry=pointgroup

Merge according to symmetry pointgroup.

-n n --iterations=n

Run n cycles of scaling and post refinement.

--no-scale

Disable the scaling part of the refinement calculation.

--no-Bscale

Disable the Debye-Waller part of the scaling calculation.

--no-pr

Disable the orientation/physics model part of the refinement calculation.

--no-deltacchalf

Disable rejection based on deltaCChalf.

-m model --model=model

Specify the partiality model. See the list below for possible choices.

-j n

Run n analyses in parallel.

--polarisation=type

Specify the polarisation of the incident radiation. type can be horiz or vert to indicate 100% polarisation of the electric field in the horizontal plane or vertical plane respectively. Setting type to none completely disables the polarisation correction (see the note below). Alternatively, type can be a direction followed by a percentage polarisation fraction. For example, 45deg90 means that 90% of the radiation is polarised with its electric field in a direction 45 degrees from horizontal, and 10deg100 means that all the radiation is polarised at 10 degrees from horizontal. The angle is specified clockwise from horizontal as viewed along the beam direction, i.e. as shown by the CrystFEL GUI. The beam is unpolarised when the fraction is 50% (equal parts of the radiation have their electric field in the specified plane). If the polarisation fraction is 100%, it can be omitted. For example 10deg or horiz.

Note that --polarisation=none is not the same as, for example, --polarisation=vert50. In the first case, the polarisation correction will be completely disabled. In the other case, the incident beam will be unpolarised, but the polarisation of the diffracted radiation will still be corrected for (the factor of (1+cos²(2theta))/2 or, equivalently, (2-sin²(2theta))/2).

The default is --polarisation=horiz.

--no-polarisation

Synonym for --polarisation=none.

--max-adu=n

Include reflections only if their peak values were less than n. That means, n is the saturation value of the detector. The default is infinity, i.e. no cutoff.

--min-res=n

Merge crystals only if they diffract to beyond n Angstroms resolution. The default is infinity, i.e. all crystals are included. The resolution is taken from the diffraction_resolution_limit line in the stream.

--min-measurements=n

Include a reflection in the output only if it appears at least least n times. The default is --min-measurements=2.

--push=res=n

merge reflections which are up to n nm^-1 higher than the apparent resolution limit of each individual crystal. n can be negative to merge lower than the apparent resolution limit. The default is --push-res=inf, which means no resolution cutoff at all.

--start-after=n

Ignore the first n crystals in the input. The default is --start-after=0, i.e. start at the beginning.

--stop-after=n

Stop processing after n crystals have been successfully read. The default is --stop-after=0, which means to process all the patterns from the start point to the end of the input (see --start-after).

--no-free

Disable cross-validation (for testing only).

--custom-split=filename

Read a list of filenames, event IDs and dataset IDs from filename. See the section on Custom Dataset Splitting below.

--max-rel-B=n

Reject crystals if the absolute values of their relative Debye-Waller ("B") factors are more than n square Angstroms. The default is --max-rel-B=100.

--output-every-cycle

Write out the per-crystal parameters and reflection lists after every cycle of refinement, instead of only at the end. The intermediate reflection lists and parameter filenames will be prefixed with iterN_ (note the trailing underscore), where N is the iteration number. If you use --custom-split, intermediate results will also be output for each custom datset.

--no-logs

Do not write the extensive log files needed for plotting contour maps and spectrum graphs. This makes the process a lot faster, but you probably do want these logs to check that post-refinement is working reasonably.

--log-folder=folder

Specify the location of the log folder (see --no-logs). The default is --log-folder=pr-logs.

-w pg

Set the apparent symmetry of the crystals. The ambiguity operator will be determined by comparing this to the actual symmetry.

If you prefer (or the scenario demands it), you can specify the ambiguity operator directly using --operator.

--operator=op

Specify the indexing ambiguity operator. Example: --operator=k,h,-l.

If you prefer, you can specify the ambiguity operator by specifying the apparent symmetry using -w.

--force-bandwidth=bw --force-radius=R --force-lambda=l

Set the X-ray bandwidth, initial profile radius or wavelength for all crystals before proceeding, overriding the values from the stream. Bandwidth is given as a fraction, i.e. --force-bandwidth=0.0013 means 0.13 percent (approximate FWHM). Radius is given in nm^-1. Wavelength is given in Angstroms.

--harvest-file=fn

Write a list of parameters to fn, in JSON format. This is intended to be used for harvesting data into a database system.

--unmerged-output=fn

Write unmerged, but otherwise fully correct, intensities to fn.

Partiality models

The available partiality models are:

xsphere

The volume of intersection between a sphere centered on each reciprocal lattice point and the part of reciprocal space excited by the Ewald sphere taking into account the finite bandwidth and convergence angle. A "source coverage factor" is included to take into account the spectral brightness of the effective source for the reflection.

This model is the same as that described in Acta Cryst. D71 (2015) p1400.

unity

Fix all partialities at 1.

offset

Treat the radiation as perfectly monochromatic, and calculate the partiality as exp(-t^2/R^2), where t is the excitation error and R is the reflection radius (taking mosaicity into account). This model is similar to that used by XDS.

ggpm

An analytical overlap integral modelling the reflection profiles as Gaussian functions, and the radiation spectrum as a sum of Gaussians.

Usage cases

Merging without scaling, partialities or post-refinement:: partialator -i my.stream -o my.hkl -y mypointgroup --model=unity --iterations=0
Merging without partialities or post-refinement, but with scaling: partialator -i my.stream -o my.hkl -y mypointgroup --model=unity --iterations=1
(Use a higher number of iterations to increase the accuracy of scaling, but at a cost of more CPU time and possibly more rejected crystals)
Merging with partialities, but without post-refinement and without scaling:: partialator -i my.stream -o my.hkl -y mypointgroup --model=xsphere --iterations=0
Mering with partialities, with scaling but without post-refinement:: partialator -i my.stream -o my.hkl -y mypointgroup --model=xsphere --iterations=1 --no-pr
Merging with partialities, post-refinement and scaling:: partialator -i my.stream -o my.hkl -y mypointgroup --model=xsphere --iterations=1
(Use a higher number of iterations to increase the accuracy of scaling and post-refinement, but at a cost of more CPU time and possibly more rejected crystals)
Merging with partialities and post-refinement, but without scaling:: This would be a strange thing to want to do, however:
partialator -i my.stream -o my.hkl -y mypointgroup --model=xsphere --iterations=1 --no-scale
(Use a higher number of iterations to increase the accuracy of post-refinement, but at a cost of more CPU time and possibly more rejected crystals)

Custom dataset splitting

When performing a time-resolved experiment (for example), it is preferable to ensure that the data for all time points has been processed identically. Rather than processing each time point independently with separate runs of partialator, it is better to process them all together and do the splitting into time points just before the final output. Consider, for example, the case of simple scaling (without a B factor): when merging independently, the resulting datasets would probably end up with different overall scaling factors. When comparing the results, you would need to take this difference into account. In practice, most programs can do that job easily, but what about if a B factor is included? And what if partialities are included - how unique is the solution?

With partialator --custom-split, you can provide a separate text file containing a list of filenames, event numbers and dataset names, one event (detector frame) per line, with the fields separated by exactly one space. For each unique dataset name, a separate reflection list will be output. All crystals will be refined together, but they will be merged according to the dataset names you give. The parameters (scaling factors, partialities etc) determined during the joint refinement will be applied. For each dataset, a separate pair of split half-datasets will also be written, allowing you to calculate figures of merit such as Rsplit and CC1/2 for each one.

If the overall output filename (given with -o or --output) were merged.hkl, then a dataset named dataset would be written to merged-dataset.hkl. The corresponding half-datasets would be written to merged-dataset.hkl1 and merged-dataset.hkl2.

Note that the filenames and event names must match exactly what is written into the stream as the Image filename and Event, taking into account options such as indexamajig --prefix and --basename. You should therefore check that the numbers of crystals in each dataset, which will be written on the terminal by partialator, match your expectations and that no patterns have been "lost". There is no requirement for every event in the list to appear in the stream, nor for every event in the stream to belong to one of the datasets. If an event is listed for more than one dataset, the results are "undefined".

If you do not have event IDs for your data, i.e. if you have one detector frame per input file, simply leave out the event IDs from the custom split file.

Finally, note that the main and all custom split datasets, and also all the half-datasets, are subject to --min-measurements.

Author

This page was written by Thomas White.

Reporting bugs

Report bugs to taw@physics.org

Copyright and Disclaimer

Please read the AUTHORS file in the CrystFEL source code distribution for a full list of contributions and contributors.

CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CrystFEL. If not, see http://www.gnu.org/licenses/.