Intelligence

View Markdown

Intelligence

Introduction

Ittybit natively supports Intelligence for common AI use cases.

Intelligence files are File Objects with a kind of speech, nsfw, description, tags, or prompt.

{
  "id": "file_qrstuvwx1234",
  "object": "intelligence",
  "kind": "speech",
  "type": "application/json",
  // ... other props
  "url": "https://you.ittybit.net/example/speech.json",
  "ref": "speech",
}

Media objects can contain one or more Intelligence files.

{
  "id": "med_abcdefgh1234",
  "object": "media",
  "kind": "video",
  "files": [
    // ... other files
    {
      "id": "file_qrstuvwx1234",
      "object": "intelligence",
      "kind": "speech",
      "type": "application/json",
      "url": "https://you.ittybit.net/example/speech.json",
      "ref": "speech",
    },
    {
      "id": "file_qrstuvwx5678",
      "object": "intelligence",
      "kind": "description",
      "type": "application/json",
      "url": "https://you.ittybit.net/example/description.json",
      "ref": "description",
    }
  ],
  // ... other props
}

Intelligence files are formatted as JSON with a consistent schema containing rich data about the associated media.

Customers use intelligence files to power features like:

  • Transcripts
  • NSFW Filtering
  • Tags and Descriptions
  • Semantic Search
  • Recommendations
  • Open Graph Previews

Usage

Some customers use ittybit's stored JSON files directly. For example, fetching a speech file and using the text property to display a transcript.

const response = await fetch('https://you.ittybit.net/example/speech.json')
const data = await response.json()
console.log(data.text)

Other customers generate intelligence files and then store the values in their database for later use.

const item = database.get('articles', { id: 'abcd-1234-efgh-5678' })

return (
  <OpenGraph
    title={item.title}
    description={item.description}
    image={item.urls.thumbnail}
  />
)

Kinds

Intelligence files can be one of the following kinds:

  • speech
  • description
  • nsfw
  • prompt

Speech

Speech files contain a transcript of the spoken content in a video or audio file.

Speech

{
  "id": "file_qrstuvwx1234",
  "object": "intelligence",
  "kind": "speech",
  "detected": true,
  "speakers": 2,
  "language": "en",
  "text": [
    "Hello, and welcome to UkeTube. I'm Jesse Doe.",
    "And I'm John Doe. Today we're going to be learning Sandstorm by Darude.",
    // ... continues
  ],
  "confidence": 0.72,
  "timeline": [
    {
      "index": 0,
      "start": 12.00,
      "end": 14.50,
      "detected": true,
      "speaker": 0,
      "text": "Hello, and welcome to UkeTube. I'm Jesse Doe.",
      "confidence": 0.89
    },
    {
      "index": 1,
      "start": 14.80,
      "end": 18.28,
      "detected": true,
      "speaker": 1,
      "text": "And I'm John Doe. Today we're going to be learning Sandstorm by Darude.",
      "confidence": 0.91
    },
    // ... continues
  ],
  "created": "2025-01-01T01:23:45Z",
  "updated": "2025-01-01T01:23:45Z",
}

See the Speech Tasks section for more information about creating new speech files.

Speech Tasks

Description

Get a text description of the content in an image, video, or audio file.

{
  "id": "file_qrstuvwx5678",
  "object": "intelligence",
  "kind": "description",
  "detected": true,
  "language": "en",
  "title": "Designing at Scale: Airbnb's Journey with Design Systems",
  "description": "Karri Saarinen, formerly of Airbnb, presents a compelling case for design systems as a solution to the increasing complexity of product development in the digital age. He argues that design systems are not just about aesthetics, but are crucial tools for organizational efficiency, product consistency, and scalability.",
  "tags": ["Design systems", "Airbnb", "Product development", "Scalability", "UI/UX", "Accessibility", "React Sketchapp", "Organizational change"],
  "confidence": 0.96,
  "created": "2025-01-01T01:23:45Z",
  "updated": "2025-01-01T01:23:45Z",
}

See the Description Tasks section for more information about creating new description files.

Description Tasks

NSFW

Detect if a video or image contains explicit content.

NSFW

{
  "id": "file_qrstuvwx9012",
  "object": "intelligence",
  "kind": "nsfw",
  "detected": true,
  "nudity": 0.82,
  "sexual": 0.24,
  "violence": 0.64,
  "gore": 0.04,
  "confidence": 0.82,
  "created": "2025-01-01T01:23:45Z",
  "updated": "2025-01-01T01:23:45Z",
}

See the NSFW Tasks section for more information about creating new nsfw files.

NSFW Tasks

Prompt

Prompt

{
  "id": "file_qrstuvwx3456",
  "object": "intelligence",
  "kind": "prompt",
  "detected": true,
  "language": "en",
  "prompt": "Identify the objects in this image. Please respond with JSON and include a bounding box and a confidence score for each item.",
  "response": [
    {
      "label": "White soccer socks",
      "boundingBox": [80, 160, 240, 320],
      "confidence": 0.98
    },
    {
      "label": "Uhlsport Football",
      "boundingBox": [240, 120, 400, 320],
      "confidence": 0.95
    },
    // ... continues
  ],
  "confidence": 0.96,
  "created": "2025-01-01T01:23:45Z",
  "updated": "2025-01-01T01:23:45Z",
}

See the Prompt Tasks section for more information about creating new prompt files.

Prompt Tasks

Tracks

Previous Page

Metadata

Next Page

On this page