The Fascinating World of Distributed Systems | Preparation for Speaker Prof. Hammoud
When a problem becomes too big for a single computer, you can break down the task into smaller pieces and have multiple computers do the tasks and work together, essentially distributing the tasks to multiple machines over a network. This is what is called a distributed system.
No, 1,000 computers working together does not mean that the system is 1000x more powerful. This is because the computers must work together as they are all working on the same task, just different parts of it. This means that they need to communicate across cables or networks, which takes time and causes delays.
The more computers you add to a system, the greater the latency increase per computer. Eventually, the amount of processing time decreased will be less than the increase in latency by adding another computer, meaning there is a maximum practical limit.
Yes, 1,000 computers can agree to something even if some fail or lie, a concept known as Byzantine Fault Tolerance. In order to achieve this consensus, they must use specialized mathematical protocols.
However, there is still a limit as to how many failures or liars there can be within a system before the entire thing fails. This limit is 1/3 of the total number of computers. For example, if 333 computers in a 1,000-computer system are faulty, the system is able to safely handle them. Problems arise, however, when this becomes 334 as it exceeds the 1/3 limit.
The computation in these big services or apps is split into two main places: remote data centers (the cloud) and your local device (your phone, computer, or console). However, the vast majority of the heavy lifting happens in remote data centers.
Applications like ChatGPT, Google, or Instagram have almost all of their processing done on their own data centers.
Games divide the work more evenly. Things like graphics, animation, and physics are done on the client side (your device), while multiplayer synchronization is done on the server side.
If this were possible, we would unlock an insane amount of things. Simulations scaling beyond the planet could be done hyper-accurately and in real-time, predicting things like weather and natural disasters almost perfectly before they even happen.
Also, not just scaling up, but also down into the molecular level. Drugs could be discovered instantly by testing billions of chemical compounds to cure the rarest diseases in human history. Stuff like global infrastructure—coordinating every supply chain, vehicle, and energy grid on the Earth to eliminate traffic jams and food/energy waste. We would even be able to compute plasma containment data to stabilize nuclear fusion as a power source worldwide.