Research Update — Week 5
When One Computer Is Not Enough: The Fascinating World of Distributed Systems
The computer is not able to compute the problem by itself so the combined power of multiple computers must be used. The problem can be divided into smaller pieces and distributed among many computers, and this approach is called distributed computing. Each computer or node, works on part of the task and communicates its results to the others through a network. The problems for which this method may be used include: scientific simulations, training AI, analysing large data sets in medicine or astrology, and searching huge collections of files.
Not necessarily because there are many limitations when solving a problem by splitting it up into parts. One limitation is that the problem sometimes can't be divided evenly. Some parts of the problem may depend on others therefore those must be solved sequentially by one computer rather than in parallel by multiple. Also, communication time between computers limits the efficiency. It may take computers more time to communicate their results to each other than to individually compute their tasks.
An example of these limitations in action is if a task consists of 900 independent calculations plus 100 steps that must happen sequentially, 1,000 computers cannot make the entire task 1,000 times faster. This is related to Amdahl's law: the maximum speedup of a parallel program is limited by the fraction of the task that cannot be parallelized.
There are two main scenarios:
The other computers cannot always tell whether a computer has crashed, is very slow, or is deliberately lying. For example, if a message arrives late, the others do not immediately know whether the problem is with the sender or the network is simply slow. To combat this, computers use special rules called consensus protocols to reach agreement. These protocols may use voting, repeated messages, identity checks, and backup computers. However, there are limits. The FLP impossibility theorem shows that if messages can be delayed indefinitely, no system can guarantee that it will always reach a decision when even one computer might crash.
Usually, the computation happens in data centers, not primarily on your phone or computer. Your device still performs some work (for example, displaying the interface, capturing a photo, decoding video, or handling local input) but the large-scale search, storage, recommendation, and AI computation generally happens remotely.
For the above examples:
When using ChatGPT, the model's response is generated by servers containing specialized processors. When using Google, a search request is processed by Google's distributed infrastructure and data centers. When using Instagram, collections of servers in data centers store and deliver content, process requests, and manage databases.
We can build whatever was previously too time consuming or too large to compute.
Possible achievements include: