Software

Members of FS-SC develop, maintain, or contribute to many software projects across a range of disciplines. The following are some of our more major and visible efforts.

Many projects are hosted under the DESY FS-SC GitLab project  while some have their own identity elsewhere.

  • ASAP::O. We develop and maintain the ASAP::O data system in collaboration with central IT. The broad purpose is to process data ‘in flight’ rather than ‘in files’. Contact: Tim Schoof and Marc-Olivier Andrez

  • ASAP::O workers are programs which process data streamed by ASAP::O, including: NeXuS writer, Eiger connector, seedee library for data serialisation. Contact: Diana Rueda

  • AMARCORD is database application designed to gather metadata about chemicals, runs and analysis results of serial crystallography (SX) experiments. Contact: Philipp Middendorf

  • CrystFEL CrystFEL is a suite of programs for processing data from serial crystallography experiments, performed at synchrotron and X-ray free-electron laser facilities, as well as using an electron microscope. Contact: Thomas White

  • HiDRA is a high performance data multiplexing tool for transferring files/data from beamline PCs to the data centre. Contact: Tim Schoof.

  • PLUG2 is a program for data analysis and reduction for XPCS experiments, primarily at P10. Contact: Vijay Kartik

  • OM, developed in collaboration with SLAC/LCLS, is a framework for the development of programs that can monitor x-ray imaging experiments in real-time. It is also a set of ready-to-use monitoring programs. It is in use at P11, P09 and REGGAE at DESY, and elsewhere. Contact: Alexandra Tolstikova

  • ROCK-IT: is a Helmholtz-funded project that aims to develop all necessary tools for the automation and remote access of in-situ and operando experiments. ROCK-IT stands for “Remote, Operando Controlled, Knowledge-driven, and IT-based”. Some of these tools are not yet public (ongoing licensing discussion).

    • Asapo Bluesky Connector is a connector that reads documents published by Bluesky to a ZMQ stream and publishes the corresponding data to Asapo. Contact: Marc-Olivier Andrez
    • XAS Workers is a collection of Python-based workers designed to process in Near real-time X-ray Absorption Spectroscopy (XAS) experimental data. Contact: Diana Rueda
    • XAS Sample Alignment is a Python toolkit for automated sample positioning in X-ray Absorption Spectroscopy (XAS) experiments. Contact: Diana Rueda (internal link: sdlkfl)
    • Data Visualizer is a library for visualizing and analysing data acquired during experiments in near real-time. Contact: Marc-Olivier Andrez
    • Pipeline Manager

We’re also developing several other projects that currently don’t have an easily identifiable public identity or dedicated repository. For example:

  • Full in-memory processing pipeline for serial crystallography experiments at P11 / DESY.
  • Geometry refinement of diffraction data at P02.2 (Thomas White, Alexandra Tolstikova)
  • Accelerating XRF data processing using the methods contained in GeoPIXE (Christoph Rosemann with P25)
  • ISPyB installation at P11 (Olga Merkulova)
  • Automated analysis and calibration of XAS-type data (Diana Rueda)
  • Automatic submission of jobs to the Maxwell cluster (Mento by Vijay Kartik)

Quantitative impact:

  • For in-memory SFX processing at P11, we reduced the computation from 1/3 of Maxwell to two nodes required to handle the Eiger 16M at 133Hz frame rate through code profiling and optimisation.
  • Add: Tensor tomography, ask Mikhail.
  • For XPCS with PLUG2 at P10, one core of one lab PC is used to process data approximately as fast as the multithreaded optimized Matlab program used for offline processing.