A scalable transient detection pipeline for the Australian SKA\n Pathfinder VAST survey

The Australian Square Kilometre Array Pathfinder (ASKAP) collects images of\nthe sky at radio wavelengths with an unprecedented field of view, combined with\na high angular resolution and sub-millijansky sensitivities. The large quantity\nof data produced is used by the ASKAP Variables and Slow Transients (VAST)\nsurvey science project to study the dynamic radio sky. Efficient pipelines are\nvital in such research, where searches often form a `needle in a haystack' type\nof problem to solve. However, the existing pipelines developed among the\nradio-transient community are not suitable for the scale of ASKAP datasets.\n In this paper we provide a technical overview of the new "VAST Pipeline": a\nmodern and scalable Python-based data pipeline for transient searches, using\nup-to-date dependencies and methods. The pipeline allows source association to\nbe performed at scale using the Pandas DataFrame interface and the well-known\nAstropy crossmatch functions. The Dask Python framework is used to parallelise\noperations as well as scale them both vertically and horizontally, by means of\na cluster of workers. A modern web interface for data exploration and querying\nhas also been developed using the latest Django web framework combined with\nBootstrap.\n