Data Access, Processing and Transfer

Automatic Data Processing

If the "start XDS" box is ticked in the Data Collection Tab of the CrystalControl GUI, then the automatic data processing is enabled and XDS runs in the background.

The results of the data processing is twofold:
- pre: as for a preprocessing, limited to all the frames included in the first angles: 5, 15, 45°...etc.
- full: as for a full data processing, i.e. including all the frames

Results of automatic data processing are available within the internet browser at the beamline as well as the one in the remote session.

Manual Data Processing

Prerequisites

Local users can perform manual data processing as described at the beamline. Users having remote access can also process their data only if they are through their DESY Science Accounts. Users may apply for science accounts to analyze data measured at Petra III or Flash on resources located at the DESY computer center. Science accounts have a standard lifetime of 3 years (renewable).

Applying for a Science Account:
1) the user submits a request to the beamline scientist/manager
2) the user receives a pre-filled form by Email within a few business days from the DESY-FS Administrator
3) the form has to be signed and sent back to the Administrator (hard-copy as scan/photo by email/fax in pdf format)
4) the user receives the initial credentials from the beamline scientist/manager who forwarded the account request. If on site at DESY, the user can authenticate him-/herself by valid ID card or passport at FS-EC's Administrators to retrieve the initial password for the account.

Accessing Data

There are two methods to access DESY computers from outside using a scientific account:

       Command line: in a Linux terminal or Cygwin (windows) type:
       - ssh username@desy-ps-ext.desy.de
        enter your Science Account password
       - ssh desy-ps-cpu or desy-ps-gpu
       - cd /asap3/petra3/gpfs/p11/YEAR/data/YourBeamTimeID

       Graphical User Interface: in a web browser type:
       - https://max-display.desy.de:3443/
       - enter your Science Account credentials
       - launch e.g. an XFCE session
       - ssh to desy-ps-cpu or desy-ps-gpu
       - cd /asap3/petra3/gpfs/p11/YEAR/data/YourBeamTimeID

Windows users: the advantage of using max-display comes to the fact of opening the GUIs of the crystallographic programs directly in the session, max-display gives a better performance compared to other X servers such as MobaXterm, X-win32 or Xming.

Resources Usage and Constraints

Please read the following instructions carefully:


       - desy-ps-cpu or desy-ps-gpu are interactive and shared nodes, which means users must not run intense jobs on those machines. i.e. NO USE of more than 4 threads / 40 GB RAM for processing
       - these nodes are for small jobs, code development and job submission
       - the nodes are shared between all external users i.e. whatever a user does will affect other users working on these machines as well
       - demanding compute jobs have to be executed in the compute cluster based on SLURM typically as batch jobs (i.e. without GUI)
       - the resources are for work on data taken at DESY Photon Science (i.e. PETRA III or FLASH) or for work for DESY
       - user follow the compute rules as agreed by signing the form to get an account, i.e. in view of local software deposit etc..
       - these resources cannot be used for any kind of commercial purposes
       - misuse of the resources can/will lead to an account revocation

Crystallographic and Scientific Software Packages

Software packages might be available out-of-the box as system installs or as part from the OS default repository, others are available via module functionality. A description on how to enable/use a particular software package is given on confluence. Crystallographic software packages are in the Photon Science section.

In a terminal type (as examples):
       module avail #to check what programs and versions are available
       module load xray # sets the environment for xds and helper GUIs such as XDSAPP
       module load ccp4/7 # sets the environment for ccp4 version 7
       module load phenix # sets the environment for phenix

Once the correct module is loaded, the program can be called via the command line by typing its shortcut or alias (xdsapp, phenix, albula....etc)

Data Transfer

1. Data are transferred locally at P11 via Rsync. The Local Contact sets up the backup on the users' external Hard Drive.
       - For Remote Session Users, an external HDD can be sent with the Dewar.

2. Data are also available for download via the Gamma portal as it is described in here.

3. A third option is to use Globus Online: this is available exclusively for Users with Science Accounts and an Institute Subscription to the Globus Group:
       - check with your local administrator if your institute is a subscriber of the Globus Group. If so, log in into Globus Online by choosing your Organization and Identity Provider
       - create an Endpoint on your machine by downloading the suitable "Globus Connect" for you operating system
       - choose DESY in the "Collection Search" and browse to your Beamtime folder
       - start transferring data in the file manager section
       - further instructions can be found on FS-EC's confluence page