/build-data-pipeline

name: build-data-pipeline
description: Build a Sui data indexer or analytics pipeline. Use when the user wants to index Sui events, build a pipeline, or query Sui RPC data.

Preamble (run first)

# Suiperpower telemetry. Routes through the CLI so projects.json + Convex stay in sync.
# Silent on failure so the skill never blocks. Opt out: SUIPERPOWER_TELEMETRY=off.
#
# AGENT NOTE: when this skill finishes, run the matching completion command:
#   suiperpower track build-data-pipeline build completed
# Or use "failed" / "aborted" if it ended that way.
command -v suiperpower >/dev/null 2>&1 && suiperpower track build-data-pipeline build started >/dev/null 2>&1 &
true

If `TEL_PROMPTED` is `no`, before doing real work, ask the user:

> Help suiperpower get better. We track which skills get used and how long they take. No code, no file paths, no PII. Change anytime in `~/.suiperpower/config.json`. > > A) Sure, anonymous > B) No thanks

Write the answer to `~/.suiperpower/config.json` `telemetryTier` field and create `~/.suiperpower/.telemetry-prompted`. Then continue.

What this skill does

Helps the user read, index, and monitor on-chain Sui data. Picks the right data access method (GraphQL RPC, event polling, custom indexer, or gRPC), implements the queries or indexer, and verifies that real data comes back before declaring done.

When to use it

• The user needs to read on-chain state (balances, objects, transaction history).
• The user wants to listen for events emitted by a Move module.
• The project needs a backend indexer for analytics, leaderboards, or historical data.
• The user asks about GraphQL queries on Sui.
• The user wants to monitor specific transaction types or object changes.

When NOT to use it

• If the user has not picked a project yet, use `find-next-sui-idea` first.
• If the user has not scaffolded a project, use `scaffold-project` first.
• If the user is writing Move code (not reading data), use `build-with-move`.
• If the user is composing transactions (writing, not reading), use `ptb-composer`.
• If the user wants DeepBook market data specifically, use `deepbook-orderbook` (it has DeepBook-specific queries).

If you activated this and the user actually wants something else, consult `skills/SKILL_ROUTER.md` and hand off.

Inputs

• A Sui project (TS app, backend service, or standalone script).
• Optional: `.suiperpower/build-context.md` from `scaffold-project`. Read it if present.
• The data requirement: what on-chain data the user needs and why.

If unclear, interview the user for:

• What data do you need? (events, object state, transaction history, balances)
• How fresh must the data be? (real-time, near-real-time, periodic batch)
• What volume do you expect? (a few queries vs. indexing all events for a module)
• Is this for a frontend, a backend analytics service, or a standalone script?

Outputs

• Query code or indexer implementation matching the chosen method.
• For GraphQL: working `.graphql` queries or inline TS query strings.
• For event polling: a polling loop with cursor tracking and persistence.
• For custom indexer: Rust project scaffolded with `sui-indexer-alt-framework`.
• Append to `.suiperpower/build-context.md`:

  ## build-data-pipeline session, <timestamp>
  - method: <graphql | event-polling | custom-indexer | grpc>
  - data targets: <what is being indexed/queried>
  - network: <mainnet | testnet | devnet>
  - polling interval: <if applicable>
  - third-party services: <if applicable>
  - open issues: <list>

Workflow

1. **Context gathering**

• Read `.suiperpower/build-context.md` if it exists.
• Identify which Move modules or event types the user cares about.
• Confirm the data freshness and volume requirements.

2. **Pick the data access method**

• Use the decision table below. Confirm the choice with the user before writing code.
• For hackathon projects, default to GraphQL RPC + event polling. Custom indexers are overkill for a hackathon.

3. **Implement queries**

• For GraphQL: write queries against `https://graphql.testnet.sui.io/graphql` (or mainnet). Respect the limits: 5KB query size, 40s timeout, 300 nodes max, 50 items per page, cursor-based pagination.
• For event polling: implement `queryEvents` with cursor tracking, descending order, and a persistence mechanism for the last-seen cursor.
• For custom indexer: scaffold the Rust project with `sui-indexer-alt-framework`, configure the PostgreSQL schema, and write the pipeline handler.
• For gRPC: configure the gRPC client against the Sui gRPC endpoint.
• See `references/data-access-methods.md` for code patterns.

4. **Add event polling if monitoring is needed**

• If the project needs to react to events in near-real-time, layer event polling on top of any read method.
• Implement a poll loop with configurable interval (default: 5s for testnet, 1s for mainnet).
• Track the cursor between polls to avoid reprocessing.
• Handle the case where events arrive faster than the poll interval (pagination within a poll).

5. **Test with real data**

• Run every query against the target network and verify non-empty results.
• For event polling, trigger a real event (or use a known recent event) and confirm it appears.
• For custom indexers, run against a small checkpoint range and verify rows land in PostgreSQL.

6. **Optimize and harden**

• Add error handling for network failures (retry with backoff).
• Add pagination handling for queries that may exceed 50 results.
• Set appropriate timeouts on all network calls.
• Review `references/data-pipeline-pitfalls.md` and fix any matching issues.

7. **Writeback**

• Append session details to `.suiperpower/build-context.md`.

8. **Closing handoff**

• If `.suiperpower/intent.md` exists and the session was non-trivial (new module, new sponsor integration, or material changes to public functions), recommend `verify-against-intent`