每日 ETL 管線

System ↔ System

擷取、轉換、驗證並載入資料，含重試處理機制。

4 個節點 · 4 條連接data

apiclidb

視覺化

從 API 擷取資料api

↓sequential→ 轉換與清理

↓fallback→ 轉換與清理

轉換與清理cli

↓sequential→ 結構描述驗證

結構描述驗證system

↓conditional→ 載入至資料庫

載入至資料庫db

在編輯器開啟

uc-data-pipeline.osop.yaml

osop_version: "1.0"
id: "etl-pipeline"
name:"每日 ETL 管線"
description:"擷取、轉換、驗證並載入資料，含重試處理機制。"

nodes:
  - id: "extract"
    type: "api"
    subtype: "rest"
    name: "從 API 擷取資料"
    retry_policy:
      max_attempts: 3
      backoff: "exponential"
    timeout_sec: 60

  - id: "transform"
    type: "cli"
    subtype: "script"
    name: "轉換與清理"

  - id: "validate"
    type: "system"
    name: "結構描述驗證"

  - id: "load"
    type: "db"
    name: "載入至資料庫"
    security:
      risk_level: "medium"

edges:
  - from: "extract"
    to: "transform"
    mode: "sequential"
  - from: "transform"
    to: "validate"
    mode: "sequential"
  - from: "validate"
    to: "load"
    mode: "conditional"
    when: "validation.passed == true"
  - from: "extract"
    to: "transform"
    mode: "fallback"
    label: "Use cached data on API failure"