Orchestrator Agent using Nevermined's Payments API (TypeScript)

Related Projects

This project is part of a larger workflow that explores the interconnection between agents and how can they communicate and work together. Please, refer to these projects in order to have a full view of the whole process

1. Movie Orchestrator Agent⁠:

   • Coordinates the entire workflow, ensuring smooth task execution across agents.

2. Movie Script Generator Agent⁠:

   • Generates movie scripts and characters descriptions based on input ideas.

3. Video Generator Agent⁠:

   • Generates realistic character videos based on their descriptions.

Workflow Diagram:

Table of Contents

• Orchestrator Agent using Nevermined's Payments API (TypeScript)
  • Related Projects
    • Workflow Diagram:
  • Table of Contents
  • Introduction
  • Getting Started
    • Prerequisites
    • Installation
  • Environment Variables
  • Architecture Overview
    • Plans and Agents
      • Relationship Diagram:
    • Workflow and Steps
    • In this example:
  • How to create your own complex agent
    • 1. Generating Steps
    • 2. Sub-Agent Task Management
    • 3. Validating Tasks and Completing Steps
    • 4. Managing Balance and Payments
    • 5. Handling Input and Output Data
  • License

Introduction

The Orchestrator Agent leverages Nevermined's Payments API to enable structured workflows with clear payment and task execution flows. The Payments API allows agents to:

• Manage payment plans: Create tasks for agents under the same or different payment plans.
• Handle balance checks: Ensure sufficient balance in a plan and purchase additional credits when necessary.
• Orchestrate workflows: Assign tasks to agents, monitor their progress, and validate results through callbacks.

This project demonstrates a real-world implementation of these concepts, integrating multiple sub-agents:

• Script Generator Agent (same plan)
• Character Extractor Agent (same plan)
• Image Generator Agent (different plan)

Getting Started

Prerequisites

• Node.js (v14 or higher)
• TypeScript globally installed
• Nevermined API Key
• Sub-agent DIDs for specific tasks

Installation

1. Clone the repository:

   git clone https://github.com/nevermined-io/movie-orchestrator-agent.git
   cd orchestrator-agent
   

   Copy

2. Install dependencies:

   npm install
   

   Copy

3. Configure environment variables: Copy .env.example to .env and populate it:

   NVM_API_KEY=your_nevermined_api_key
   NVM_ENVIRONMENT=testing # or staging or production
   PLAN_DID=your_plan_did
   IMAGE_GENERATOR_PLAN_DID=your_image_plan_did
   AGENT_DID=your_agent_did
   SCRIPT_GENERATOR_DID=your_script_did
   IMAGE_GENERATOR_DID=your_image_generator_did
   

   Copy

4. Build and run the project:

   npm run build
   npm start
   

   Copy

Environment Variables

Architecture Overview

Plans and Agents

In Nevermined, a Plan (PLAN_DID) represents a subscription that allows agents to execute tasks. Plans have credits that are consumed as tasks are executed.

• Agents (AGENT_DID) are entities that execute specific tasks. These agents may:
  • Be under the same plan, sharing the plan's balance (e.g., Script Generator and Character Extractor).
  • Be under a different plan, or even belong to different builders, requiring separate permissions and credits (e.g., Image Generator).

Example:

• Plan A:
  • Includes the Orchestrator Agent and its sub-agents:
    • Script Generator Agent
    • Character Extractor Agent
• Plan B:
  • Includes an external Image Generator Agent.

Relationship Diagram:

[Plan A: PLAN_DID] ----------- [Orchestrator Agent]
                                    |-- [Script Generator Agent]
                                    |-- [Character Extractor Agent]

[Plan B: IMAGE_GENERATOR_PLAN_DID] -- [Image Generator Agent]

Workflow and Steps

1. Init Step:

   • Always the first step in a workflow.
   • Creates subsequent steps (e.g., generateScript, generateImagesForCharacters).

2. Step Lifecycle:

   • Balance Check: Ensure the plan has sufficient credits.
   • Sub-Task Creation:
     • Retrieve access permissions for the sub-agent.
     • Create a task for the sub-agent with specific input data.
   • Task Validation:
     • Validate the task status via logs.
     • Mark the task as completed only when all associated sub-tasks are finished.

In this example:

1. Init Step:
   • Defines and schedules subsequent steps.
2. Generate Script:
   • Uses Script Generator to create a story and a list of character descriptions.
3. Generate Videos for Characters:
   • Assigns tasks to Video Generator for creating character cinematic videos.

How to create your own complex agent

1. Generating Steps

The handleInitStep function initializes the workflow by defining the subsequent steps:

export async function handleInitStep(step: any, payments: any) {
  const scriptStepId = generateStepId();
  const videoStepId = generateStepId();

  const steps = [
    { step_id: scriptStepId, task_id: step.task_id, name: "generateScript", predecessor: step.step_id },
    { step_id: videoStepId, task_id: step.task_id, name: "generateVideoForCharacters", predecessor: scriptStepId },
  ];

  await payments.query.createSteps(step.did, step.task_id, { steps });
}

All steps are interlinked using predecessor, maintaining the execution order.

2. Sub-Agent Task Management

Within each step, sub-tasks are created for specific agents:

export async function handleStepWithAgent(
  step: any,
  agentDid: string,
  agentName: string,
  planDid: string,
  payments: any
) {
  const hasBalance = await ensureSufficientBalance(planDid, step, payments);
  if (!hasBalance) return;

  const accessConfig = await payments.query.getServiceAccessConfig(agentDid);
  const taskData = { query: step.input_query, name: step.name };

  await payments.query.createTask(
    agentDid,
    taskData,
    accessConfig,
    async (data) => {
      const taskLog = JSON.parse(data);

      if (taskLog.task_status === "Completed") {
        await validateScriptGenerationTask(
          taskLog.task_id,
          agentDid,
          accessConfig,
          step,
          payments
        );
      }
    }
  );
}

3. Validating Tasks and Completing Steps

When a step involves multiple tasks (e.g., generating images), all tasks must complete successfully before marking the step as completed:

export async function handleImageGenerationForCharacters(step: any, payments: any) {
  const characters = JSON.parse(step.input_artifacts);
  const tasks = characters.map((character) =>
    queryAgentWithPrompt(step, generateTextToImagePrompt(character), "Image Generator", payments)
  );

  await Promise.all(tasks);

  await payments.query.updateStep(step.did, {
    ...step,
    step_status: "Completed",
    output: "All image generation tasks completed successfully.",
  });
}

4. Managing Balance and Payments

The orchestrator ensures sufficient balance before executing tasks:

export async function ensureSufficientBalance(planDid: string, step: any, payments: any): Promise<boolean> {
  const balance = await payments.getPlanBalance(planDid);
  if (balance < 1) {
    const orderResult = await payments.orderPlan(planDid);
    if (!orderResult.success) {
      await payments.query.updateStep(step.did, {
        ...step,
        step_status: "Failed",
        output: "Insufficient balance and failed to order credits.",
      });
      return false;
    }
  }
  return true;
}

5. Handling Input and Output Data

Input and Output Parameters:

• step.input_query: Primary input for a step.
• step.output: Standard output of the step.
• step.input_artifacts: Artifacts passed from the previous step.
• step.output_artifacts: Artifacts generated by the current step.

Flow:

• Outputs (output/output_artifacts) from one step are automatically used as inputs (input_query/input_artifacts) for the subsequent step.

Example:

• Script Generator Step outputs a script (output).
• Character Extractor Step uses this script as its input_query.

License

Copyright 2024 Nevermined AG

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.