/new-project

name: new-project
description: "Build a new AI agent with Olakai monitoring from scratch — project setup, SDK integration, KPI configuration, and end-to-end validation"
metadata:
  revision: 1
  updated-on: "2026-03-10"
  source: maintainer
  tags: "olakai,new-project,agent,monitoring,kpi,governance"

Build a New AI Agent Project with Olakai

This skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.

Prerequisites

Before starting, ensure: 1. Olakai CLI installed: `npm install -g olakai-cli` 2. CLI authenticated: `olakai login` 3. API key for SDK (generated per-agent via CLI — see Step 2.2)

Why Custom KPIs Are Essential

Olakai's core value is **tracking business-specific KPIs for your AI agents**. Without KPIs, you're tracking events without gaining actionable insights.

**What you can measure with KPIs:**

• Business outcomes (items processed, success rates, revenue impact)
• Operational data (step counts, retry rates, execution time)
• Quality indicators (error rates, user satisfaction signals)

**Without KPIs configured:**

• No dashboard KPIs beyond basic token counts
• No aggregated performance views
• No alerting thresholds
• No ROI calculations

> **Every agent should have 2-4 KPIs that answer: "How do I know this agent is performing well?"**

> **KPIs created here belong to this specific agent only.** If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.

Understanding the customData to KPI Pipeline

Before diving into implementation, understand how data flows through Olakai:

SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData

How It Works

1. **customData** (SDK): Raw JSON you send with each event 2. **CustomDataConfig** (Platform): Schema defining which fields are processed 3. **Context Variables**: CustomDataConfig fields become available for formulas 4. **KPI Formula**: Expression that computes a value (e.g., `SuccessRate * 100`) 5. **kpiData** (Response): Computed KPI values returned with each event

Critical Rules

| Rule | Consequence | |------|-------------| | Only CustomDataConfig fields become variables | Unregistered customData fields are NOT usable in KPIs | | Formula evaluation is case-insensitive | `stepCount`, `STEPCOUNT`, `StepCount` all work in formulas | | NUMBER configs need numeric values | Don't send `"5"` (string), send `5` (number) | | KPIs are unique per agent | Each KPI belongs to exactly one agent — create separately for each |

Built-in Context Variables (Always Available)

| Variable | Type | Description | |----------|------|-------------| | `Prompt` | string | The prompt text sent to the LLM | | `Response` | string | The LLM response text | | `Documents count` | number | Number of attached documents | | `PII detected` | boolean | Whether PII was detected | | `PHI detected` | boolean | Whether PHI was detected | | `CODE detected` | boolean | Whether code was detected | | `SECRET detected` | boolean | Whether secrets were detected |

Step 1: Design the Agent Architecture

1.1 Determine Agent Type

**Agentic AI** (Multi-step autonomous workflows):

• Research agents, document processors, data pipelines
• Track as SINGLE events aggregating all internal LLM calls
• Focus on workflow-level KPIs (total tokens, total time, success/failure)

**Assistive AI** (Interactive chatbots/copilots):

• Customer support agents, coding assistants, Q&A systems
• Track EACH interaction as separate events
• Focus on conversation-level KPIs (per-message tokens, response quality)

1.2 Design Your KPI Schema (CRITICAL)

**Design your KPIs BEFORE writing any SDK code.** This ensures only meaningful data is sent and tracked.

Step A: Identify Business Questions

What do stakeholders need to know about this agent?

• "How many items does it process per run?"
• "What's the success/failure rate?"
• "How efficient is each execution?"

Step B: Map Questions to Data Fields

| Business Question | Field Name | Type | KPI Formula | Aggregation | |-------------------|------------|------|-------------|-------------| | Throughput | ItemsProcessed | NUMBER | `ItemsProcessed` | SUM | | Reliability | SuccessRate | NUMBER | `SuccessRate * 100` | AVERAGE | | Error count | SuccessRate | NUMBER | `IF(SuccessRate < 1, 1, 0)` | SUM | | Correlation | ExecutionId | STRING | (for filtering only) | - |

Step C: Plan Your customData Structure

// ONLY include fields you'll register as CustomDataConfigs
customData: {
  // Business KPIs
  ItemsProcessed: number,  // Count of items handled
  SuccessRate: number,     // 0-1 success ratio

  // Performance KPIs
  StepCount: number,       // Number of workflow steps

  // Identification (for filtering, not KPIs)
  ExecutionId: string,     // Correlation ID
}

> **IMPORTANT**: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but **cannot be used in KPIs**.

What NOT to Include in customData

The Olakai platform automatically tracks these — do NOT duplicate them:

| Already Tracked | Where | Don't Send As customData | |-----------------|-------|--------------------------| | Session ID | Main payload | `sessionId` | | Agent ID | API key association | `agentId` | | User email | `userEmail` parameter | `email`, `userEmail` | | Timestamp | Event metadata | `timestamp`, `createdAt` | | Request time | `requestTime` parameter | `duration`, `latency` | | Token count | `tokens` parameter | `tokenCount` | | Model | Auto-detected | `model`, `modelName` | | Provider | Client config | `provider` |

**customData is ONLY for:** 1. **KPI variables** — Fields you'll use in formula calculations 2. **Tagging/filtering** — Fields you'll filter by in queries

Step 2: Configure Olakai Platform

2.1 Create a Workflow (Required)

> **Every agent MUST belong to a workflow*