distributed systems questions

1. What happens when the problem you want to solve becomes too big for any one computer?

Answer: What happens is we distribute the problem across multiple systems and see if the shared resources of these systems can solve this problem. That is the key principle of distributed systems.

2. Suppose 1,000 computers work together. Do you now have one computer that is 1,000 times more powerful? Why or why not?

Answer: No. 1,000 computers working together is still 1,000 computers working, not 1 computer. This is because of mainly structural and hardware limitations: each computer still governs its own resources, e.g. one computer cannot directly access the memory of another computer, and there is also a network latency between the computers, which can be crucial when speed is important. There is also something called the Amdahl's Law, which basically means some tasks cannot be distributed to be completed in parallel and have to be computed in a specific sequence that takes a fixed amount of time.

3. Can 1,000 computers agree on something if some of them fail or even lie?

Answer: They can, and that's called Byzantine fault tolerance, named after the Byzantine generals problem. The general rule is that less than one third of computers may fail or lie for the system to reach consensus.

4. When you use ChatGPT, Google, Instagram, or an online game, where is the computation actually happening?

Answer: In the cloud, i.e. data centers with hundreds of thousands of computers. These data centers use the principle of distributed systems for different purposes. For ChatGPT, data centers provide GPU resources that it needs to generate billions of responses to billions of prompts every day. For Google searches, these data centers are mainly used for CPU and storage for indexing. For Instagram, they use so-called Content Delivery Networks scattered across the globe, which cache content in different places so that loading it is fast enough wherever the user is located. For online games, some of the computing (e.g. graphics and physics) happens on your device, but other computing (e.g. match data and player positions) happens in those data centers. Due to the rise in AI demand, the number of hypermassive data centers is now rapidly increasing, which means increasingly powerful distributed systems will be able to solve a few more problems before they kill our planet.

5. If you could make millions of computers behave like one dependable machine, what could humanity build that we cannot build today?

Answer: We could cure cancer, solve climate change, predict disasters with much more accuracy, find more laws of physics, locate aliens, possibly create one decent mobile banking app.

references: