Daily ETL Pipeline

System ↔ System

Extract, transform, validate, and load with retry handling.

4 nodes · 4 edgesdata

apiclidb

Visual

Extract from APIapi

↓sequential→ Transform & Clean

↓fallback→ Transform & Clean

Transform & Cleancli

↓sequential→ Schema Validation

Schema Validationsystem

↓conditional→ Load to Database

Load to Databasedb

Open in Editor

uc-data-pipeline.osop.yaml

osop_version: "1.0"
id: "etl-pipeline"
name: "Daily ETL Pipeline"
description: "Extract, transform, validate, and load with retry handling."

nodes:
  - id: "extract"
    type: "api"
    subtype: "rest"
    name: "Extract from API"
    retry_policy:
      max_attempts: 3
      backoff: "exponential"
    timeout_sec: 60

  - id: "transform"
    type: "cli"
    subtype: "script"
    name: "Transform & Clean"

  - id: "validate"
    type: "system"
    name: "Schema Validation"

  - id: "load"
    type: "db"
    name: "Load to Database"
    security:
      risk_level: "medium"

edges:
  - from: "extract"
    to: "transform"
    mode: "sequential"
  - from: "transform"
    to: "validate"
    mode: "sequential"
  - from: "validate"
    to: "load"
    mode: "conditional"
    when: "validation.passed == true"
  - from: "extract"
    to: "transform"
    mode: "fallback"
    label: "Use cached data on API failure"

←

Wellness Partner Integration

Production Deployment

→