When One Computer Is Not Enough: The Fascinating World of Distributed Systems
What happens when the problem you want to solve becomes too big for any one computer?
Parallel systems are usually used. This is where the task is split up into smaller problems, and then each part of the task is shared to a computer, and every computer is connected together via a network.
Suppose 1,000 computers work together. Do you now have one computer that is 1,000 times more powerful? Why or why not?
No, you do not. This is because it takes time to transfer data between any two computers; in essence, the data transfer rate limits how strong the 'distributed' computer is. The main idea of Amdahl's law is "that the speedup of a system is limited by the portion of the program that cannot be parallelized (the sequential part)."
Can 1,000 computers agree on something if some of them fail or even lie?
Yes, they can. This is known as the Byzantine Generals Problem. If the number of 'traitors' is more than 1/3rd of the total system, then the problem fails.
When you use ChatGPT, Google, Instagram, or an online game, where is the computation actually happening?
The computation is stored and distributed across the whole world. The places that store this data are known as data centers, and large cloud providers handle the transfer of data.
If you could make millions of computers behave like one dependable machine, what could humanity build that we cannot build today?
Basically everything. We could solve any physics equation, find solutions to the millennial questions, and more. We could achieve atomic-level simulations of particle accelerators and weather, and have precise forecasting.