/building-ioc-defanging-and-sharing-pipeline

name: building-ioc-defanging-and-sharing-pipeline
description: Build an automated pipeline that ingests raw IOCs (URLs, IPs, domains,
  emails), normalizes and deduplicates them, then produces defanged renderings for
  safe human reading alongside canonical STIX 2.1 bundles distributed via TAXII servers,
  MISP, or email reports. Use when preparing indicators of compromise for safe analyst
  sharing or automating threat intel distribution to TAXII/MISP feeds.
domain: cybersecurity
subdomain: threat-intelligence
tags:
- ioc
- defanging
- threat-sharing
- stix
- pipeline
- indicator
- automation
- threat-intelligence
version: '1.0'
author: mahipal
license: Apache-2.0
nist_csf:
- ID.RA-01
- ID.RA-05
- DE.CM-01
- DE.AE-02
mitre_attack:
- T1071.001
- T1583.001
- T1105
- T1566.002

Building IOC Defanging and Sharing Pipeline

Overview

IOC defanging modifies potentially malicious indicators (URLs, IP addresses, domains, email addresses) to prevent accidental clicks or execution while preserving readability for analysis and sharing. This skill covers building an automated pipeline that ingests raw IOCs from multiple sources, normalizes and deduplicates them, applies defanging for safe human consumption, converts them to STIX 2.1 format for machine consumption, and distributes through TAXII servers, MISP instances, and email reports.

When to Use

• When deploying or configuring building ioc defanging and sharing pipeline capabilities in your environment
• When establishing security controls aligned to compliance requirements
• When building or improving security architecture for this domain
• When conducting security assessments that require this implementation

Prerequisites

• Python 3.9+ with `defang`, `ioc-fanger`, `stix2`, `requests`, `validators` libraries
• MISP instance or TAXII server for automated sharing
• Understanding of IOC types: IPv4/IPv6, domains, URLs, email addresses, file hashes
• Familiarity with STIX 2.1 Indicator patterns and TLP marking definitions
• Access to threat intelligence feeds for IOC ingestion

Key Concepts

IOC Defanging Standards

Defanging replaces active protocol and domain components to prevent execution: `http://` becomes `hxxp://`, `https://` becomes `hxxps://`, dots in domains/IPs become `[.]`, `@` in emails becomes `[@]`. This is critical for sharing IOCs in reports, emails, Slack channels, and paste sites where auto-linking could trigger network connections to malicious infrastructure.

IOC Normalization

Raw IOCs from different sources come in inconsistent formats. Normalization involves converting to lowercase, removing trailing slashes and whitespace, extracting domains from URLs, resolving URL encoding, validating format correctness, and deduplicating across sources.

STIX 2.1 Indicator Patterns

STIX patterns express IOCs in a standardized format: `[ipv4-addr:value = '203.0.113.1']`, `[domain-name:value = 'malicious.example.com']`, `[url:value = 'http://evil.com/payload']`, `[file:hashes.'SHA-256' = 'abc123...']`. Each indicator includes valid_from, indicator_types, confidence, and optional TLP markings.

Workflow

Step 1: Build IOC Extraction and Normalization

import re
import hashlib
from urllib.parse import urlparse, unquote
from datetime import datetime

class IOCExtractor:
    """Extract and normalize IOCs from text."""

    PATTERNS = {
        "ipv4": r'\b(?:(?:25[0-5]|2[0-4]\d|1\d{2}|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d{2}|[1-9]?\d)\b',
        "domain": r'\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}\b',
        "url": r'https?://[^\s<>"{}|\\^`\[\]]+',
        "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
        "md5": r'\b[a-fA-F0-9]{32}\b',
        "sha1": r'\b[a-fA-F0-9]{40}\b',
        "sha256": r'\b[a-fA-F0-9]{64}\b',
    }

    WHITELIST_DOMAINS = {
        "google.com", "microsoft.com", "amazon.com", "github.com",
        "cloudflare.com", "akamai.com", "example.com",
    }

    def extract_from_text(self, text):
        """Extract all IOC types from free text."""
        # Refang any already-defanged indicators first
        text = self._refang(text)
        iocs = {"ipv4": set(), "domain": set(), "url": set(),
                "email": set(), "md5": set(), "sha1": set(), "sha256": set()}

        for ioc_type, pattern in self.PATTERNS.items():
            matches = re.findall(pattern, text)
            for match in matches:
                normalized = self._normalize(match, ioc_type)
                if normalized and not self._is_whitelisted(normalized, ioc_type):
                    iocs[ioc_type].add(normalized)

        # Remove domains that are part of URLs
        url_domains = set()
        for url in iocs["url"]:
            parsed = urlparse(url)
            url_domains.add(parsed.netloc)
        iocs["domain"] -= url_domains

        total = sum(len(v) for v in iocs.values())
        print(f"[+] Extracted {total} unique IOCs from text")
        return {k: sorted(v) for k, v in iocs.items()}

    def _refang(self, text):
        """Convert defanged indicators back to active form."""
        text = text.replace("hxxp://", "http://").replace("hxxps://", "https://")
        text = text.replace("[.]", ".").replace("[@]", "@")
        text = text.replace("[://]", "://").replace("(.)", ".")
        return text

    def _normalize(self, value, ioc_type):
        """Normalize an IOC value."""
        value = value.strip().lower()
        if ioc_type == "url":
            value = unquote(value).rstrip("/")
        elif ioc_type == "domain":
            value = value.rstrip(".")
        return value

    def _is_whitelisted(self, value, ioc_type):
        """Check if IOC is in whitelist."""
        if ioc_type == "domain":
            return value in self.WHITELIST_DOMAINS
        if ioc_type == "url":
            parsed = urlparse(value)
            return parsed.netloc in self.WHITELIST_DOMAINS
        return False

extractor = IOCExtr