--- name: goldsky-io/turbo-architecture source: https://app.decimal.ai/s/goldsky-io-turbo-architecture@1/SKILL.md source_sha256: 89d9cafc339c --- # Turbo Pipeline Architecture Help users make architecture decisions for Turbo pipelines — source types, data flow patterns, resource sizing, sink strategies, and multi-chain deployment. ## Triggers Invoke this skill when the user: - Asks "how should I structure my pipeline" or "what's the best approach for..." - Needs to choose between dataset and kafka source types - Wants to send data to multiple destinations - Is planning a multi-chain deployment - Asks about resource sizing (s, m, l) - Wants to split or combine pipelines - Asks about `parallelism`, sink performance, or throughput - Needs to decide between one big pipeline vs several smaller ones - Mentions `/turbo-architecture` ## Agent Instructions ### Step 1: Understand the Requirements Ask the user about: 1. **What data?** — Which chain(s), which events/datasets, historical or real-time only? 2. **Where does it go?** — Database, webhook, streaming topic, multiple destinations? 3. **How much volume?** — Single contract or all chain activity? How many events/sec? 4. **Latency needs?** — Real-time dashboards (sub-second) or analytics (minutes OK)? ### Step 2: Recommend an Architecture Use the patterns and decision guides below to recommend a pipeline architecture. Reference the templates in `templates/` as starting points: - `templates/linear-pipeline.yaml` — Simple decode → filter → sink - `templates/fan-out-pipeline.yaml` — One source → multiple sinks - `templates/fan-in-pipeline.yaml` — Multiple events → UNION ALL → sink - `templates/multi-chain-templated.yaml` — Per-chain pipeline pattern ### Step 3: Hand Off to Implementation Skills After the architecture is decided, direct the user to: - **`/turbo-pipelines`** — To build and deploy the YAML - **`/turbo-transforms`** — To write the SQL transforms - **`/goldsky-secrets`** — To set up sink credentials --- ## Source Types ### Dataset Sources — Historical + Real-Time Use `type: dataset` when you need to process historical blockchain data and/or continue streaming new data. ```yaml sources: my_source: type: dataset dataset_name: base.erc20_transfers version: 1.2.0 start_at: earliest # or: latest ``` **Best for:** Raw logs, transactions, transfers, blocks — anything where you need historical backfill or decoded event processing. **Available EVM datasets:** `raw_logs`, `blocks`, `raw_transactions`, `raw_traces`, `erc20_transfers`, `erc721_transfers`, `erc1155_transfers`, `decoded_logs` **Non-EVM chains also supported:** Solana (`solana.*`), Bitcoin (`bitcoin.raw.*`), Stellar (`stellar_mainnet.*`) ### Kafka Sources — Real-Time Streaming > **Note:** Kafka sources are used in production pipelines but are **not documented** in the official Goldsky docs. Use with caution and contact Goldsky support for topic names. Use `type: kafka` when consuming from a Goldsky-managed Kafka topic, typically for continuously-updated state like balances. ```yaml sources: my_source: type: kafka topic: base.raw.latest_balances_v2 ``` **Best for:** Balance snapshots, latest state data, high-volume continuous streams. **Key differences from dataset sources:** - No `start_at` or `version` fields - Optional fields: `filter`, `include_metadata`, `starting_offsets` - Delivers the latest state rather than historical event logs ### When to Use Which | Scenario | Source Type | Why | | --------------------------------- | ----------- | -------------------------------------------- | | Decode contract events from logs | `dataset` | Need `raw_logs` + `_gs_log_decode()` | | Track token transfers | `dataset` | `erc20_transfers` has structured data | | Historical backfill + live | `dataset` | `start_at: earliest` processes history | | Live token balances | `kafka` | `latest_balances_v2` is a streaming topic | | Real-time state snapshots | `kafka` | Kafka delivers latest state continuously | | Only need new data going forward | Either | Dataset with `start_at: latest` or Kafka | --- ## Data Flow Patterns ### Linear Pipeline The simplest pattern. One source → one or more chained transforms → one sink. ``` source → transform_a → transform_b → sink ``` **Use when:** You have a single data source, single destination, and straightforward processing (decode, filter, reshape). **Example:** `templates/linear-pipeline.yaml` — raw logs → decode → extract trade events → postgres **Resource size:** `s` or `m` ### Fan-Out (One Source → Multiple Sinks) One source feeds multiple transforms, each writing to a different sink. ``` ┌→ transform_a → sink_1 (clickhouse) source ──────┤ └→ transform_b → sink_2 (webhook) ``` **Use when:** You need different views or subsets of the same data going to different destinations — e.g., balances to a warehouse AND token metadata to a webhook. **Example:** `templates/fan-out-pipeline.yaml` — one Kafka source → fungible balances to ClickHouse + all tokens to a webhook ```yaml transforms: fungible_balances: type: sql primary_key: id sql: | SELECT ... FROM latest_balances balance WHERE balance.token_type = 'ERC_20' OR balance.token_type IS NULL all_tokens: type: sql primary_key: id sql: | SELECT ... FROM latest_balances balance WHERE balance.token_type IN ('ERC_20', 'ERC_721', 'ERC_1155') sinks: warehouse: type: clickhouse from: fungible_balances # ... webhook: type: webhook from: all_tokens # ... ``` **Resource size:** `m` (multiple output paths) ### Fan-In (Multiple Events → One Output) Multiple event types decoded from the same source, normalized to a common schema, then combined with UNION ALL into a single sink. ``` ┌→ event_type_a ──┐ source → decode ┤ ├→ UNION ALL → sink └→ event_type_b ──┘ ``` **Use when:** You want a unified activity feed, combining trades, deposits, withdrawals, transfers, etc. into one table. **Example:** `templates/fan-in-pipeline.yaml` — one raw_logs source → decode → multiple event-type transforms → UNION ALL → ClickHouse **Resource size:** `l` (complex processing with many transforms) ### Multi-Chain Fan-In Multiple sources from different chains combined into a single output. ``` source_chain_a ──┐ source_chain_b ──┼→ UNION ALL → sink source_chain_c ──┘ ``` **Use when:** You want cross-chain analytics or a unified view across chains. ```yaml sources: eth_transfers: type: dataset dataset_name: ethereum.erc20_transfers version: 1.0.0 start_at: latest base_transfers: type: dataset dataset_name: base.erc20_transfers version: 1.2.0 start_at: latest transforms: combined: type: sql primary_key: id sql: | SELECT *, 'ethereum' AS chain FROM eth_transfers UNION ALL SELECT *, 'base' AS chain FROM base_transfers ``` **Resource size:** `m` or `l` depending on chain count --- ## Templated Multi-Chain Deployment When you need the **same pipeline logic** across multiple chains, create separate pipeline files per chain rather than one multi-source pipeline. This gives you: - Independent lifecycle (deploy/delete per chain) - Independent checkpointing (one chain failing doesn't block others) - Clearer monitoring per chain **Pattern:** Copy the pipeline YAML and swap the chain-specific values: | Field | Chain A (base) | Chain B (arbitrum) | | ------------------ | --------------------------------- | ------------------------------------- | | `name` | `base-balance-streaming` | `arbitrum-balance-streaming` | | `topic` | `base.raw.latest_balances_v2` | `arbitrum.raw.latest_balances_v2` | | Source key | `base_latest_balances_v2` | `arbitrum_latest_balances_v2` | | Transform SQL | `'base' AS chain` | `'arbitrum' AS chain` | | Sink table | `base_token_balances` | `arbitrum_token_balances` | **Example:** `templates/multi-chain-templated.yaml` — shows the base chain version; duplicate for each chain. **When to use templated vs multi-source:** | Approach | Pros | Cons | | ----------------------- | --------------------------------------------- | ----------------------------------- | | Templated (per-chain) | Independent lifecycle, clear monitoring | More files to manage | | Multi-source (one file) | Single deployment, cross-chain UNION possible | Coupled lifecycle, harder to debug | --- ## Resource Sizing | Size | Workers | When to Use | | ---- | ------- | -------------------------------------------------------------- | | `s` | 1 | Testing, simple filters, single source/sink, low volume | | `m` | 2 | Multiple sinks, Kafka streaming, moderate transform complexity | | `l` | 4 | Multi-event decoding with UNION ALL, high-volume historical backfill, complex processing | **Rules of thumb from production pipelines:** - Simple filter + single sink → `s` - Kafka source + multiple sinks OR multiple transforms → `m` - Raw log decoding + 5+ event types + UNION ALL → `l` - Historical backfill of high-volume data → `l` (can downsize after catch-up) --- ## Sink Strategy ### Choosing a Sink Type | Destination | Sink Type | Best For | | -------------------- | -------------------- | --------------------------------------------- | | Analytics queries | `clickhouse` | Large-scale aggregations, time-series data | | Application DB | `postgres` | Row-level lookups, joins, application serving | | Real-time aggregates | `postgres_aggregate` | Balances, counters, running totals via triggers| | Event processing | `kafka` | Downstream consumers, event-driven systems | | Notifications | `webhook` | Lambda functions, API callbacks, alerts | | Data lake | `s3_sink` | Long-term archival, batch processing | | Serverless streaming | `s2_sink` | S2.dev streams, alternative to Kafka | | Testing | `blackhole` | Validate pipeline without writing data | ### Multi-Sink Considerations - Each sink reads from a `from:` field — different sinks can read from different transforms - Sinks are independent — one failing doesn't block others - Use different `batch_size` / `batch_flush_interval` per sink based on latency needs ### Sink Parallelism ClickHouse and other sinks support a `parallelism` setting: ```yaml sinks: my_sink: type: clickhouse from: my_transform parallelism: 1 # number of concurrent writers # ... ``` Use `parallelism: 1` for most cases. Increase only if the sink can handle concurrent writes and you need higher throughput. ### Webhook Sinks Without Secrets Webhooks can use a direct URL instead of a secret when no auth headers are needed: ```yaml sinks: my_webhook: type: webhook from: my_transform url: https://my-lambda.us-west-2.on.aws/ ``` --- ## Pipeline Splitting Decisions ### One Pipeline vs. Multiple **Use one pipeline when:** - All data comes from the same source - Transforms share intermediate results (e.g., a shared decode step) - You want atomic deployment of the whole flow **Split into multiple pipelines when:** - Different data sources with no shared transforms - Different lifecycle needs (one is stable, another changes frequently) - Different resource requirements (one needs `l`, another needs `s`) - Different chains with independent processing (templated pattern) ### Keeping Pipelines Focused A pipeline should ideally do **one logical thing**: | Pipeline | Focus | | ------------------------------- | ----------------------------------- | | `dex-trades` | Trade events → Postgres | | `dex-activities` | All activity types → ClickHouse DWH | | `token-balances` | Token balances → Postgres | | `base-balance-streaming` | Base balances → ClickHouse + webhook | Even though trades are a subset of activities, they're separate pipelines because they serve different consumers (application DB vs data warehouse). --- ## Related Skills - **`/turbo-pipelines`** — Build and deploy pipeline YAML configurations - **`/turbo-transforms`** — Write SQL transforms (filtering, decoding, UNION ALL, etc.) - **`/goldsky-datasets`** — Discover available blockchain datasets and chain prefixes - **`/goldsky-secrets`** — Set up credentials for sinks - **`/turbo-monitor-debug`** — Monitor pipeline health and debug issues - **`/turbo-lifecycle`** — List, delete, and manage pipeline state