WitEngine

The Challenge of Distributed Computing

Imagine you need to render an animated film, process millions of documents, or train a machine learning model. Your single computer would take weeks. The obvious solution — use multiple machines working together — introduces extraordinary complexity:

• Network communication — How do machines talk to each other?
• Task partitioning — How do you divide the work fairly?
• Load balancing — How do you prevent fast machines from sitting idle?
• Failure handling — What happens when a machine crashes mid-task?
• Hardware diversity — How do you handle different CPUs, GPUs, and operating systems?

Traditional approaches require months of infrastructure work before any actual computation begins.

WitEngine eliminates this complexity.

The Intelligent Orchestrator

WitEngine acts as an intelligent orchestrator — you describe what you want to compute, and it figures out how to distribute it efficiently. Just as a taxi dispatcher assigns cars based on location and traffic, WitEngine assigns tasks based on machine capabilities, current load, and performance characteristics.

From Complex Infrastructure to Simple Scripts

Without WitEngine, distributing a rendering task requires:

❌ Setting up message queues (RabbitMQ, Kafka)
❌ Implementing worker processes with health monitoring
❌ Building a custom task scheduler
❌ Handling serialization for complex data
❌ Managing partial failures and retries
❌ Months of development and testing

With WitEngine:

Job:RenderAnimation(FrameCollection:frames)
{
    ~ Distribute rendering across all available machines ~
    ImageCollection:images = Grid.ForEach(frame in frames) 
        => Renderer.RenderFrame(frame);
    
    Video:result = Video.Compose(images);
}

The Grid.ForEach call handles everything: finding available machines, dividing frames based on performance, sending data, collecting results, and handling failures.

Smart Distribution with Benchmarks

One of WitEngine's core innovations is benchmark-based load distribution. When a machine joins the cluster, it runs a standardized benchmark that measures its computational capabilities. This score becomes the basis for intelligent task allocation.

How It Works

Consider an office with diverse hardware:

When you submit a job with 1000 frames to render, WitEngine doesn't split them equally (250 per machine). Instead, it distributes proportionally:

• Developer Workstation: 420 frames
• Designer Mac: 310 frames
• Office Laptop: 170 frames
• Old Desktop: 100 frames

The result? All machines finish at approximately the same time. No powerful machine sits idle waiting for slower ones to complete.

Dynamic Rebalancing

Benchmarks aren't static. WitEngine continuously monitors actual performance:

• If a machine performs better than expected, it receives more work
• If a machine slows down (thermal throttling, competing processes), work is redistributed
• If a machine goes offline, its pending tasks are automatically reassigned

This ensures optimal utilization even as conditions change throughout a long-running job.

Two Distribution Strategies

WitEngine offers two strategies optimized for different workload patterns:

Balanced Strategy (Push Model) — Work is divided upfront based on benchmark scores. Each node receives its full batch immediately. Best for predictable tasks with consistent execution times.

Queued Strategy (Pull Model) — Nodes pull tasks from a shared queue as they complete previous work. Faster nodes automatically do more. Best for variable execution times or unpredictable workloads.

ProcessingOptions:opts = ProcessingOptions.Create("Queued");
Results:r = Grid.ForEach(item in items, opts) => Process(item);

Core Design Principles

Pattern-Based Distribution

WitEngine is built on a fundamental insight: most distributed workloads follow predictable patterns. Whether you're rendering frames, processing files, or running simulations:

1. Take a collection of inputs
2. Apply the same operation to each input
3. Collect the outputs

WitEngine provides primitives that capture these patterns, handling all complexity behind the scenes.

Fault Tolerance

When a machine fails during processing:

• Its incomplete tasks are automatically reassigned
• No manual intervention required
• Results are never lost
• The job continues without interruption

Heterogeneous Support

WitEngine works across:

• Different operating systems (Windows, Linux, macOS)
• Different hardware (x64, ARM, with or without GPU)
• Different network configurations (LAN, WAN, cloud)

Each machine contributes what it can, and WitEngine orchestrates them into a unified computational resource.

Supported Platforms

A single job can utilize servers, workstations, laptops, and mobile devices — each receiving work proportional to its capabilities.

Key Features

Plugin Architecture

WitEngine is built on a modular plugin system where everything is a plugin — even built-in functionality comes from Controllers. This means custom types and operations are first-class citizens.

Each Controller provides:

• Variables — Data types usable in scripts (e.g., Int, Matrix, Image)
• Activities — Operations on data (e.g., Int.Add, Matrix.Multiply, Image.Resize)

Built-in Controllers include primitives (Int, String, Bool), control flow (If, Loop, ForEach), distributed computing (Grid.ForEach), and linear algebra (Matrix, Vector).

Capability-Based Node Selection

Not every node can run every task. Activities can declare hardware requirements:

[RequiresCpu(MinLogicalCores = 8)]
[RequiresMemory(MinRamMb = 16384)]
[RequiresGpu(MinVRamMb = 8192, Features = GpuFeatures.Cuda)]
public class WitActivityTrainModel : WitActivityTransform { }

The orchestrator automatically filters nodes based on:

• OS — Platform, version
• CPU — Architecture, core count
• Memory — Available RAM
• GPU — Type, VRAM, features (CUDA, OpenCL, Metal)
• Custom properties — User-defined capabilities

Only compatible nodes receive the work.

The Foundation of a Larger Ecosystem

WitEngine is the core engine of the WitCloud ecosystem — a platform that turns any collection of devices into a unified computational resource. While WitEngine provides the distributed computing primitives, WitCloud adds device management, job orchestration, and enterprise features.

WitEngine provides the foundation: the scripting language, distributed execution primitives, benchmark system, and fault tolerance. WitCloud builds on top with device enrollment, job scheduling, user management, and monitoring dashboards. OmnibusCloud is the global public instance of WitCloud — a marketplace where anyone can contribute idle computing resources or consume computational power.

Use Cases

• Rendering & Animation — Distribute frame rendering across hundreds of machines
• Scientific Computing — Run simulations at scale without supercomputer access
• Data Processing — Process massive datasets in parallel
• Machine Learning — Distribute training across available resources
• Build Systems — Parallelize compilation across developer machines
• Batch Processing — Process thousands of files simultaneously

Getting Started

WitEngine is a core component of the WitCloud/OmnibusCloud platform. While the framework itself is proprietary, you can extend it with custom controllers and activities.

For developers:

• Visit witengine.io for documentation and tutorials
• Explore WitEngine.Public on GitHub — sample controllers and a limited SDK for developing and debugging your own extensions

The public SDK allows you to build custom controllers that integrate with WitEngine, test them locally, and deploy them to WitCloud or OmnibusCloud environments.