Files Management

Overview

LlmTornado provides comprehensive file management capabilities across multiple AI providers including OpenAI, Google, and Anthropic. You can upload files for use with assistants, vision models, and other AI features, as well as retrieve, list, and delete files. File management is essential for building applications that work with documents, images, and other media.

Quick Start

Here's a simple file upload example:

using LlmTornado;
using LlmTornado.Common;
using LlmTornado.Files;

TornadoApi api = new TornadoApi("your-api-key");

// Upload a file
HttpCallResult<TornadoFile> uploadedFile = await api.Files.Upload(
    "path/to/document.pdf", 
    FilePurpose.Assistants);

Console.WriteLine($"File uploaded with ID: {uploadedFile.Data.Id}");

// Retrieve file information
TornadoFile? retrievedFile = await api.Files.Get(uploadedFile.Data.Id);
Console.WriteLine($"File name: {retrievedFile?.Filename}");

Prerequisites

Before using file management features, ensure you have:

1. The LlmTornado package installed
2. A valid API key with file access permissions
3. Files in supported formats (PDF, images, text, etc.)
4. Understanding of file purposes and provider limitations

Detailed Explanation

File Purposes

Files can be uploaded for different purposes:

OpenAI Purposes

• Assistants - Files for assistant knowledge base
• Vision - Images for vision analysis
• FineTuning - Training data for model fine-tuning
• Batch - Batch processing input files

File Operations

The library supports complete CRUD operations:

• Upload - Add files to provider storage
• Get - Retrieve file metadata
• List - Get all files for your account
• Delete - Remove files from storage
• Download - Retrieve file content (provider-dependent)

Provider Support

Different providers have different file capabilities:

OpenAI

• Full CRUD support
• File purposes: assistants, vision, fine-tuning, batch
• File size limits and format restrictions

Google (Gemini)

• Upload, retrieve, list, delete
• Used with multimodal models
• Automatic file expiration

Anthropic (Claude)

• Upload, retrieve, list, delete
• Document analysis capabilities
• Downloadable file support

Basic Usage

Upload a File

// Upload to OpenAI
HttpCallResult<TornadoFile> result = await api.Files.Upload(
    "documents/sample.pdf", 
    FilePurpose.Assistants);

if (result.Data != null)
{
    Console.WriteLine($"Uploaded: {result.Data.Id}");
    Console.WriteLine($"Filename: {result.Data.Filename}");
    Console.WriteLine($"Size: {result.Data.Bytes} bytes");
}

Upload to Specific Provider

// Upload to Google
HttpCallResult<TornadoFile> googleFile = await api.Files.Upload(
    "documents/sample.pdf", 
    provider: LLmProviders.Google);

// Upload to Anthropic
HttpCallResult<TornadoFile> anthropicFile = await api.Files.Upload(
    "documents/sample.pdf", 
    provider: LLmProviders.Anthropic);

Console.WriteLine($"Google file ID: {googleFile.Data.Id}");
Console.WriteLine($"Anthropic file ID: {anthropicFile.Data.Id}");

Retrieve File Information

// Get file metadata by ID
TornadoFile? file = await api.Files.Get("file_id_here");

if (file != null)
{
    Console.WriteLine($"ID: {file.Id}");
    Console.WriteLine($"Filename: {file.Filename}");
    Console.WriteLine($"Purpose: {file.Purpose}");
    Console.WriteLine($"Size: {file.Bytes} bytes");
    Console.WriteLine($"Created: {file.CreatedUnixTime}");
}

List All Files

// List OpenAI files
TornadoPagingList<TornadoFile>? files = await api.Files.Get(
    provider: LLmProviders.OpenAi);

if (files is not null)
{
    Console.WriteLine($"Found {files.Items.Count} files.");
    
    foreach (TornadoFile file in files.Items)
    {
        Console.WriteLine($"{file.Id} - {file.Filename}");
    }
}

Delete a File

// Delete from OpenAI
HttpCallResult<DeletedTornadoFile> deleteResult = await api.Files.Delete("file_id");
Console.WriteLine($"Deleted: {deleteResult.Data?.Deleted}");

Advanced Usage

Upload with Purpose

// Upload for assistants
HttpCallResult<TornadoFile> assistantFile = await api.Files.Upload(
    "knowledge_base.pdf", 
    FilePurpose.Assistants, 
    provider: LLmProviders.OpenAi);

// Upload for fine-tuning
HttpCallResult<TornadoFile> trainingFile = await api.Files.Upload(
    "training_data.jsonl", 
    FilePurpose.FineTuning, 
    provider: LLmProviders.OpenAi);

Console.WriteLine($"Assistant file: {assistantFile.Data.Id}");
Console.WriteLine($"Training file: {trainingFile.Data.Id}");

Paginated File Listing

// List files with pagination
TornadoPagingList<TornadoFile>? files = await api.Files.Get(
    new ListQuery(100), 
    provider: LLmProviders.Google);

if (files is not null)
{
    Console.WriteLine($"Total files: {files.Items.Count}");
    Console.WriteLine($"Has more: {files.HasMore}");
    
    foreach (TornadoFile file in files.Items)
    {
        Console.WriteLine($"{file.Id}: {file.Filename} ({file.Bytes} bytes)");
    }
}

Download File Content

Some providers support file content download:

// List Anthropic files
TornadoPagingList<TornadoFile>? files = await api.Files.Get(
    new ListQuery(100), 
    provider: LLmProviders.Anthropic);

if (files?.Items.Count > 0)
{
    // Find downloadable file
    TornadoFile? downloadable = files.Items.FirstOrDefault(x => x.Downloadable);

    if (downloadable is not null)
    {
        // Download content
        string content = await api.Files.GetContent(
            downloadable.Id, 
            provider: LLmProviders.Anthropic);
        
        Console.WriteLine($"Downloaded {content.Length} characters");
        await File.WriteAllTextAsync("downloaded_file.txt", content);
    }
}

Multi-Provider File Management

Manage files across different providers:

// Upload same file to multiple providers
string filePath = "documents/important.pdf";

HttpCallResult<TornadoFile> openAiFile = await api.Files.Upload(
    filePath, 
    FilePurpose.Assistants, 
    provider: LLmProviders.OpenAi);

HttpCallResult<TornadoFile> googleFile = await api.Files.Upload(
    filePath, 
    provider: LLmProviders.Google);

HttpCallResult<TornadoFile> anthropicFile = await api.Files.Upload(
    filePath, 
    provider: LLmProviders.Anthropic);

Console.WriteLine("File uploaded to all providers:");
Console.WriteLine($"OpenAI: {openAiFile.Data.Id}");
Console.WriteLine($"Google: {googleFile.Data.Id}");
Console.WriteLine($"Anthropic: {anthropicFile.Data.Id}");

File Upload with Error Handling

async Task<TornadoFile?> SafeUploadFile(string path, FilePurpose purpose)
{
    try
    {
        // Check if file exists
        if (!File.Exists(path))
        {
            Console.WriteLine($"File not found: {path}");
            return null;
        }

        // Check file size
        FileInfo fileInfo = new FileInfo(path);
        if (fileInfo.Length > 50 * 1024 * 1024) // 50 MB limit
        {
            Console.WriteLine("File too large (max 50MB)");
            return null;
        }

        // Upload file
        HttpCallResult<TornadoFile> result = await api.Files.Upload(path, purpose);
        
        if (result.Data != null)
        {
            Console.WriteLine($"Successfully uploaded: {result.Data.Id}");
            return result.Data;
        }
        else
        {
            Console.WriteLine("Upload failed: No data returned");
            return null;
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Upload error: {ex.Message}");
        return null;
    }
}

// Usage
TornadoFile? uploaded = await SafeUploadFile("document.pdf", FilePurpose.Assistants);

Cleanup Old Files

Automatically clean up old files:

async Task CleanupOldFiles(int daysOld = 30)
{
    TornadoPagingList<TornadoFile>? files = await api.Files.Get();
    
    if (files is null) return;

    long cutoffTime = DateTimeOffset.UtcNow.AddDays(-daysOld).ToUnixTimeSeconds();
    
    foreach (TornadoFile file in files.Items)
    {
        if (file.CreatedUnixTime < cutoffTime)
        {
            HttpCallResult<DeletedTornadoFile> result = await api.Files.Delete(file.Id);
            
            if (result.Data?.Deleted == true)
            {
                Console.WriteLine($"Deleted old file: {file.Filename}");
            }
        }
    }
}

// Clean up files older than 30 days
await CleanupOldFiles(30);

File Validation

Validate files before upload:

bool ValidateFile(string path, out string error)
{
    error = string.Empty;
    
    if (!File.Exists(path))
    {
        error = "File does not exist";
        return false;
    }
    
    FileInfo info = new FileInfo(path);
    
    // Check size (50 MB max)
    if (info.Length > 50 * 1024 * 1024)
    {
        error = "File too large (max 50MB)";
        return false;
    }
    
    // Check extension
    string[] allowedExtensions = [".pdf", ".txt", ".docx", ".jpg", ".png"];
    if (!allowedExtensions.Contains(info.Extension.ToLower()))
    {
        error = $"Unsupported file type: {info.Extension}";
        return false;
    }
    
    return true;
}

// Usage
if (ValidateFile("document.pdf", out string error))
{
    HttpCallResult<TornadoFile> result = await api.Files.Upload(
        "document.pdf", 
        FilePurpose.Assistants);
}
else
{
    Console.WriteLine($"Validation failed: {error}");
}

Best Practices

1. File Organization

• Use descriptive filenames
• Tag files with metadata when possible
• Implement a file tracking system
• Clean up unused files regularly

2. Size Management

• Compress files before uploading when possible
• Split large files into smaller chunks
• Monitor storage usage
• Delete files after use

3. Error Handling

• Validate files before upload
• Handle network errors gracefully
• Implement retry logic for transient failures
• Log upload/delete operations

4. Security

• Don't upload sensitive data without encryption
• Use secure file paths
• Validate file contents
• Implement access controls

5. Provider-Specific Considerations

• Understand provider file limits
• Use appropriate file purposes
• Handle provider-specific errors
• Monitor provider quotas

Common Issues

Upload Failed

• Issue: File upload returns error
• Solution: Check file size, format, and API permissions
• Prevention: Validate files before upload

File Not Found

• Issue: Cannot retrieve uploaded file
• Solution: Verify file ID and provider
• Prevention: Store file IDs after upload

Storage Quota Exceeded

• Issue: Cannot upload more files
• Solution: Delete old or unused files
• Prevention: Implement automatic cleanup

Unsupported File Type

• Issue: Provider rejects file format
• Solution: Convert to supported format
• Prevention: Validate file types before upload

API Reference

Files Endpoint

• Upload(string path, FilePurpose? purpose, LLmProviders? provider) - Upload file
• Get(string fileId, LLmProviders? provider) - Get file by ID
• Get(ListQuery? query, LLmProviders? provider) - List files
• Delete(string fileId, LLmProviders? provider) - Delete file
• GetContent(string fileId, LLmProviders? provider) - Download content

TornadoFile

• string Id - Unique file identifier
• string Filename - Original filename
• FilePurpose? Purpose - File purpose
• long Bytes - File size in bytes
• long CreatedUnixTime - Upload timestamp
• bool Downloadable - Whether file can be downloaded

FilePurpose

• Assistants - For use with assistants
• Vision - For vision models
• FineTuning - For model fine-tuning
• Batch - For batch processing

HttpCallResult<T>

• T Data - Result data
• string Response - Raw response
• bool Success - Whether call succeeded

Related Topics

• Assistants - Using files with assistants
• Vision - Image analysis
• Vector Stores - File-based vector stores
• Chat Basics - Core functionality