Parallel Computing in Julia

Julia offers three distinct, composable layers of parallelism: asynchronous tasks (@async, @sync, and coroutines via Channels) for overlapping I/O-bound work like network requests within a single thread; multithreading (Threads.@threads, Threads.@spawn) for CPU-bound work split across cores that share the same memory space within one process; and multiprocessing via the Distributed standard library (addprocs, @distributed, pmap) for spreading work across entirely separate Julia processes, which may run on different machines and communicate by explicit message-passing rather than shared memory. Choosing the right layer matters because they have very different overhead and failure characteristics — threads are cheap to spawn but share mutable state (and thus risk data races), while processes are heavier to start but fully isolated, making them safer for large, independent chunks of work or for scaling beyond a single machine.

Threads.@threads splits the iterations of a for loop across however many threads Julia was started with (set via the -t flag, e.g. julia -t 4, or the JULIA_NUM_THREADS environment variable), giving each thread a contiguous chunk of the iteration range to execute in shared memory with no communication overhead between threads for independent work. Because all threads share the same heap, writing to a shared mutable variable (like accumulating into a single sum without synchronization) from multiple threads simultaneously is a data race that produces silently wrong, non-deterministic results rather than an error — the loop body must either write to disjoint memory locations per iteration, or use synchronization primitives like Threads.Atomic or a ReentrantLock when a shared value truly must be updated.

The Distributed standard library adds worker processes with addprocs(n) (or connects to remote machines via SSH with addprocs([("host1", n), ("host2", n)])), each an entirely separate Julia process with its own memory space and no shared state by default, so data must be explicitly sent across using @spawnat, remotecall, or the higher-level @distributed macro and pmap function. Because workers don't automatically have access to code or packages loaded only in the main process, @everywhere using PackageName and @everywhere function myfunc(...) ... end are required to make definitions available on every worker before using them in a distributed computation, and pmap in particular is well suited to embarrassingly parallel workloads where each unit of work is independent and roughly equal in cost.

Threads are the right choice when work is CPU-bound, fine-grained, and needs to share large in-memory data structures without the cost of copying or serializing them — spawning a thread is cheap (microseconds) and there's no message-passing overhead for read-only shared data. Processes are the right choice when you need true fault isolation (a crash in one worker process doesn't take down the whole computation), when scaling beyond a single machine's core count across a cluster, or when working with libraries that aren't thread-safe, since giving each unit of work its own process sidesteps shared-mutable-state hazards entirely at the cost of higher startup overhead and the need to explicitly serialize data across process boundaries.