/more_training_data

name: Generate More Training Data
allowed-tools: Bash, Read, Write
description: Analyze data patterns and generate additional synthetic training data

Generate More Training Data

This command analyzes patterns in existing data files (CSV, JSONL) and generates additional synthetic training data based on those patterns. Uses bash commands or inline uv python to append data efficiently without loading large files into memory.

Instructions

• IMPORTANT: You can use inline astral uv python code (with any libraries you need) for data processing. Use `uv run python --with pandas --with <whatever library you need> -c "import pandas as pd; print(\"whatever you want here\")"`
• Example: `uv run --with pandas --with faker python -c "import pandas as pd; from faker import Faker; fake = Faker(); print(fake.name())"`
• IMPORTANT: Both bash commands and uv python are acceptable - choose the most efficient approach for each task
• IMPORTANT: When generating synthetic data, ensure variety and realistic patterns

Variables

DROPPED_FILE_PATH: [[FILE_PATH]] DROPPED_FILE_PATH_ARCHIVE: agentic_drop_zone/training_data_zone/drop_zone_file_archive/ DATA_OUTPUT_DIR: agentic_drop_zone/training_data_zone/data_output/<date_time>/

• This is the directory where all generated data will be saved
• The date_time is the current date and time in the format YYYY-MM-DD_HH-MM-SS

NUM_NEW_ROWS: 25

• Default number of new data rows to generate
• Can be overridden if specified in the dropped file

SAMPLE_SIZE: 50

• Number of rows to sample for pattern analysis (keeps context window small)
• Use Read with only a specific number of rows to keep the context window small

Workflow

• Create output directory: `DATA_OUTPUT_DIR/<date_time>/`
• Determine file format by extension (.csv or .jsonl)
• Copy the original file to output directory: `cp DROPPED_FILE_PATH DATA_OUTPUT_DIR/<date_time>/original_<filename>`

Pattern Analysis Phase

• Extract a sample for analysis (to keep context window small):

**Option 1: Using bash commands**

• For CSV: `head -n SAMPLE_SIZE DROPPED_FILE_PATH > DATA_OUTPUT_DIR/<date_time>/sample.csv`
• For JSONL: `head -n SAMPLE_SIZE DROPPED_FILE_PATH > DATA_OUTPUT_DIR/<date_time>/sample.jsonl`

**Option 2: Using uv python**

  # For CSV
  uv run --with pandas python -c "
  import pandas as pd
  df = pd.read_csv('DROPPED_FILE_PATH', nrows=100)
  df.to_csv('DATA_OUTPUT_DIR/<date_time>/sample.csv', index=False)
  print(f'Sampled {len(df)} rows for analysis')
  "
  
  # For JSONL
  uv run --with pandas python -c "
  import pandas as pd
  df = pd.read_json('DROPPED_FILE_PATH', lines=True, nrows=100)
  df.to_json('DATA_OUTPUT_DIR/<date_time>/sample.jsonl', orient='records', lines=True)
  print(f'Sampled {len(df)} rows for analysis')
  "

• Read and analyze ONLY the sample file to determine:
• Data schema/structure
• Field types and patterns
• Value distributions and constraints
• Any relationships between fields

**For CSV files:**

• Identify column headers from first line
• Detect data types for each column (numeric, text, date, boolean, etc.)
• Analyze value ranges for numeric columns
• Identify patterns in text fields (emails, phone numbers, IDs, etc.)
• Check for categorical values and their distributions

**For JSONL files:**

• Parse each line as a separate JSON object
• Identify all keys and their data types
• Detect enumerated values and their frequencies
• Identify any ID patterns or sequences
• Note: Each line must be a complete, valid JSON object

Data Generation Phase

• Based on the pattern analysis, generate `NUM_NEW_ROWS` new data entries
• Write the new data to a separate file:
• For CSV: `DATA_OUTPUT_DIR/<date_time>/new_rows.csv` (without headers)
• For JSONL: `DATA_OUTPUT_DIR/<date_time>/new_rows.jsonl`

**Example using uv python with faker:**

  # Generate synthetic CSV data
  uv run --with pandas --with faker python -c "
  import pandas as pd
  from faker import Faker
  import random
  fake = Faker()
  
  # Generate 25 rows of synthetic data based on analyzed patterns
  data = []
  for i in range(25):
      row = {
          'id': i + 1000,  # Continue from existing IDs
          'name': fake.name(),
          'email': fake.email(),
          'age': random.randint(22, 65),
          'department': random.choice(['Engineering', 'Marketing', 'Sales', 'Support'])
      }
      data.append(row)
  
  df = pd.DataFrame(data)
  df.to_csv('DATA_OUTPUT_DIR/<date_time>/new_rows.csv', index=False, header=False)
  print(f'Generated {len(df)} new rows')
  "

• Generated data should:
• Follow the same structure as the original
• Maintain realistic value distributions
• Preserve relationships between fields
• Use similar patterns for IDs, dates, etc.
• Include variation to avoid exact duplicates
• Maintain data integrity constraints observed in original

Append Phase

• Create the extended dataset by appending new rows to a copy of the original:

**Option 1: Using bash commands**

For CSV:

  # Copy original to extended file
  cp DATA_OUTPUT_DIR/<date_time>/original_<filename> DATA_OUTPUT_DIR/<date_time>/extended_data.csv
  
  # Append new rows (skip header if present in new_rows.csv)
  tail -n +2 DATA_OUTPUT_DIR/<date_time>/new_rows.csv >> DATA_OUTPUT_DIR/<date_time>/extended_data.csv

For JSONL:

  # Copy original to extended file
  cp DATA_OUTPUT_DIR/<date_time>/original_<filename> DATA_OUTPUT_DIR/<date_time>/extended_data.jsonl
  
  # Append new rows
  cat DATA_OUTPUT_DIR/<date_time>/new_rows.jsonl >> DATA_OUTPUT_DIR/<date_time>/extended_data.jsonl

**Option 2: Using uv python**

For CSV:

  uv run --with pandas python -c "
  import pandas as pd
  
  # Read original and new data
  original = pd.read_csv('DATA_OUTPUT_DIR/<date_time>/original_<filename>')
  new_rows = pd.read_csv('DATA_OUTPUT_DIR/<date_time>/new_rows.csv', header=None,