Intelligent

Transformation

Delivery

Orchestration

## iDDS

Intelligent optimize the workflow

## Intelligent

Fine grained data transformation

Remote data transformation/reduction

On-demand production of analysis format data transformation

## Transformation

Deliver only when needed

Fine grained tape carousel

## Delivery

Orchestration of WFMS and DDMS with generalized workflows

## Orchestration

A general service to extract and abstract common functions of WFMS and DDMS

Flexibility to experiment agnostic

Flexibility to support different use-cases and backends with plugin structure

## Intelligent to transform and deliver needed data to consumers

Components structure and Layers Architecture

## Architecture

Main Components

## Components

Transformer To transform input data from one collection to another collection. With plugin structure to support different transform type and different backends.
Transporter To manage collections with DDM backend.
Conductor To notify/schedule consumers to consume new transformed data in a fine-grained granularity.

Layer and Plugin Architecture

## Layers and Plugins

Abstract Layers Hide the complexity of different logics to implement general API
Plugins To support different transforms and backends
Restful interface Interface to communicate with clients

Experiment agnostic based on generation

Abstract for ochestration

Maintainability and extensibility with plugins

## Features

Applications that iDDS would like to serve

## Potential Applications

- ### Event Streaming Service

  Split files into fine grained events and stream the events as inputs for EventService.

- ### Fine grained tape carousel

  Aggregate tape stage-in requests and immediate processing of partially staged data.

- ### On-demand DAOD production

  Passive creation of analysis format data rather than proactively keeing them on DISK, reduction of expensive disk usage while increaing of cheap CPU usage.

- ### Fine grained remoted data transformation/reduction

  Remotely transform/reduce data and then deliver only needed data to consumers.