When One Computer Is Not Enough

When a problem needs more memory, storage, or processing power than a single machine can provide, or when one machine would simply take too long, we split the work across many computers connected by a network. This is called a distributed system: a system whose parts sit on different networked computers that coordinate by sending messages to each other. The idea is to divide a big problem into smaller tasks, give each task to a different machine, and then combine the results.

Historically, this let researchers tackle very hard problems without buying an expensive supercomputer — for example SETI@home, one of the earliest and best-known projects, which used volunteers' home computers to analyze radio signals. However, splitting the work creates new problems. The machines have to communicate, stay in sync, and keep working even when some of them break. The main challenges are handling components that run concurrently, the absence of a single shared clock, and parts that can fail independently of each other.

No. In practice the speedup is always less than 1,000x, for a few reasons. First, not every part of a problem can be split up — some steps must happen in order. Amdahl's Law describes this: if even ten percent of a program has to run sequentially, the best possible speedup is ten times, no matter how many processors you add.

Second, the computers have to talk to each other over a network, and sending messages is far slower than a single computer accessing its own memory. Third, coordination itself costs time: the more processors you use, the more overhead goes into managing them. Finally, with 1,000 machines, some are almost always slow or broken, so the system has to spend effort detecting and working around failures.

So 1,000 computers are much more powerful than one, but they behave like a team that has to coordinate, not like a single giant brain.

Yes, but only under certain conditions. This is known as the consensus problem. The version with lying computers is called the Byzantine Generals Problem, described in 1982 by Leslie Lamport, Robert Shostak, and Marshall Pease using the story of army generals who must agree to attack or retreat while some of them may be traitors.

If computers only crash (stop responding) but never lie, algorithms like Paxos and Raft can still reach agreement as long as a majority of machines are working. If some computers can lie or send conflicting information, the problem is much harder. The classic result is that agreement is possible only if fewer than one third of the machines are dishonest. That means at least 3f + 1 computers are needed to tolerate f liars.

Real systems use these ideas today. For example, Bitcoin deals with this problem through Proof of Work, which makes it too expensive for dishonest participants to rewrite the shared record.

Mostly not on your own device. Your phone or laptop mainly acts as a “client”: it sends requests and displays results. The heavy computation happens on servers in large data centers owned by companies like Google, Meta, Microsoft, or Amazon.

When you ask ChatGPT a question, your message travels to a data center where large AI models run on clusters of powerful GPUs, and the answer is sent back. A Google search is split across thousands of machines that each hold part of the search index. Instagram stores photos on many servers and uses content delivery networks (servers placed around the world) so images load from a location near you. Online games are usually a mix: your device draws the graphics and handles your controls, while a game server keeps track of the shared world and makes sure all players see the same game state.

So “the cloud” is really a huge distributed system of physical computers in buildings around the world.

Some possibilities I think are exciting: much more detailed climate and weather simulations that could predict extreme events earlier and more precisely; simulations of the human body at the level of cells or even molecules, which could speed up drug discovery (projects like Folding@home already do a small version of this); and real-time global systems for things like disaster response, traffic, or energy grids that coordinate millions of devices at once without a single point of failure.

It could also make scientific research more open. Anyone with an idea could use enormous computing power as if it were one reliable computer, without needing to be an expert in distributed systems. Finally, it could lead to much larger and more capable AI systems, along with shared records (for health, finance, or voting) that stay correct even when some machines fail or are attacked.