Docent

Docent is an optional local trajectory annotation service plus a Control Tower CLI for managing collections, uploading evaluation data, and sharing collections with collaborators.

Local Service

Docent is not started by default. To include it:

docker compose --profile docent up -d

Docent is a third-party tool by Transluce for summarizing, clustering, and searching over agent transcripts. We've found it useful for analyzing trajectories, but it is not part of Control Tower. See the Docent repo and Docent docs for support and documentation.

Docent runs entirely locally with its own PostgreSQL database. Accounts are created on first sign-up and stored there.

Getting started: Go to http://localhost:3001 and create an account.

CLI alternative:

1. Sign up: curl -X POST http://localhost:8888/rest/signup -H 'Content-Type: application/json' -d '{"email":"you@example.com","password":"pass"}'
2. Log in: curl -X POST http://localhost:8888/rest/login -H 'Content-Type: application/json' -d '{"email":"you@example.com","password":"pass"}'
3. Create an API key and manage collections at http://localhost:3001

Local Environment Variables

Add these to your main .env:

DOCENT_SERVER_URL=http://localhost:8888
DOCENT_WEB_URL=http://localhost:3001
DOCENT_API_KEY=<your-api-key-after-signup>

For Docent's LLM features, copy .env.docent.example to .env.docent and add your API keys. Only that file is passed to the Docent containers:

cp .env.docent.example .env.docent
# Add your ANTHROPIC_API_KEY and/or OPENAI_API_KEY

Installation

The CLI is part of the Control Tower project. Make sure you have the required dependencies installed:

uv sync --extra docent

Quick Start

1. Connect to Docent: set DOCENT_API_KEY, DOCENT_SERVER_URL, and DOCENT_WEB_URL in your .env (see Local Environment Variables above)

2. Create a collection:

   ct docent create --name "My Evals"

3. Upload evaluation logs:

   ct docent upload logs/my_eval.eval

4. Add a collaborator:

   ct docent collaborator-add colleague@example.com

Commands

All commands follow the pattern: ct docent <command> [options]

Commands fail with a clear error if DOCENT_API_KEY, DOCENT_SERVER_URL, or DOCENT_WEB_URL is missing from your environment.

Collection Management

list

List all your collections.

ct docent list

Options:

• --search TEXT - Filter collections by name (partial match)

Example:

ct docent list --search "linux"

create

Create a new collection.

ct docent create --name "Collection Name"

Options:

• --name TEXT - Collection name (required, will prompt if not provided)
• --description TEXT - Optional description

Example:

ct docent create --name "Linux Evals" --description "Evaluation runs for Linux tasks"

delete

Delete a collection.

ct docent delete [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Options:

• --force - Skip confirmation prompt

Examples:

# Interactive selection
ct docent delete

# Direct deletion
ct docent delete abc123 --force

Upload Evaluation Data

upload

Upload evaluation logs to a collection.

ct docent upload PATH

Arguments:

• PATH - Path to a .eval file or directory containing .eval files

Options:

• --collection-name TEXT - Use existing collection by name (errors if no collection matches)
• --collection-id TEXT - Use existing collection by ID

Examples:

Upload a single file (will prompt for collection):

ct docent upload logs/my_eval.eval

Upload all files in a directory:

ct docent upload logs/

Upload to a specific collection by name:

ct docent upload logs/my_eval.eval --collection-name "Linux Evals"

Upload to a specific collection by ID:

ct docent upload logs/my_eval.eval --collection-id abc123

Sharing & Permissions

All sharing commands support both interactive and direct modes. If you don't provide a collection ID, you'll be prompted to select from your available collections.

collaborator-add

Add a collaborator to a collection by email.

ct docent collaborator-add EMAIL [--collection-id ID]

Arguments:

• EMAIL - Email address of the user to add (must have a Docent account)

Options:

• --collection-id TEXT - Collection ID (will prompt if not provided)

Permission granted: read access

Examples:

# Interactive collection selection
ct docent collaborator-add colleague@example.com

# Direct
ct docent collaborator-add colleague@example.com --collection-id abc123

make-public

Make a collection publicly accessible (read-only).

ct docent make-public [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Examples:

# Interactive selection
ct docent make-public

# Direct
ct docent make-public abc123

After making public, anyone with the link can view the collection at: <DOCENT_WEB_URL>/dashboard/<collection_id> (the command prints the URL)

make-private

Remove public access from a collection.

ct docent make-private [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Examples:

# Interactive selection
ct docent make-private

# Direct
ct docent make-private abc123

collaborators

List all collaborators for a collection.

ct docent collaborators [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Examples:

# Interactive selection
ct docent collaborators

# Direct
ct docent collaborators abc123

Output shows:

• Public access status
• User collaborators with their email and permission level
• Organization collaborators (if any)

collaborator-remove

Remove a user's access to a collection.

ct docent collaborator-remove EMAIL [--collection-id ID]

Arguments:

• EMAIL - Email address of the collaborator to remove

Options:

• --collection-id TEXT - Collection ID (will prompt if not provided)

Examples:

# Interactive collection selection
ct docent collaborator-remove colleague@example.com

# Direct
ct docent collaborator-remove colleague@example.com --collection-id abc123

permissions

Check your permission level on a collection.

ct docent permissions [COLLECTION_ID]

Arguments:

• COLLECTION_ID - Optional. Will prompt to select if not provided.

Permission levels:

• admin - Full control (can manage sharing, delete collection)
• write - Can modify collection contents
• read - Can view collection

Examples:

# Interactive selection
ct docent permissions

# Direct
ct docent permissions abc123

Interactive Collection Selection

For all commands that operate on a collection, if you don't provide a collection ID, you'll see an interactive prompt:

Select a collection:
  1. Linux Evals (ID: abc123) - Evaluation runs for Linux tasks
  2. Agent Tests (ID: def456)
  3. Production Runs (ID: ghi789) - Weekly production evaluations
Select option (1-3): 

This makes it easy to work with collections without memorizing IDs.

File Structure

src/control_tower/trajectories/docent/
├── collections.py       # Collection management (list, create, delete)
├── docent_connection.py # Connection management and configuration
├── parsers.py           # Input parsing and .eval file processing
├── prompts.py           # Interactive prompts for CLI
├── sharing.py           # Collection sharing and permissions
├── upload.py            # Eval log upload helpers
└── README.md            # Symlink to docs/docent.md (this file)

src/control_tower/cli/
└── docent.py            # CLI entry point and command definitions

Environment Variables

The CLI uses the following environment variables (set them in .env; there are no defaults):

• DOCENT_API_KEY - Your Docent API key
• DOCENT_SERVER_URL - Docent API URL (e.g., http://localhost:8888)
• DOCENT_WEB_URL - Docent Web UI URL (e.g., http://localhost:3001)

Error Handling

The CLI provides helpful error messages for common issues:

Not configured:

Missing required environment variables: ... Set DOCENT_API_KEY, DOCENT_SERVER_URL, and DOCENT_WEB_URL in your .env

User not found:

Error: User with email 'user@example.com' not found

Insufficient permissions:

Error: You don't have admin permission on this collection

Tips

1. Use collection names for upload: If you have a collection named "Linux Evals", you can use --collection-name "Linux Evals" instead of looking up the ID.

2. Scripting: For automation, always provide --collection-id to skip interactive prompts.

3. Batch uploads: Point upload at a directory to upload all .eval files at once.

4. Check before sharing: Use permissions to verify you have admin access before trying to share.

5. List collaborators: Use collaborators to see who already has access before sharing.