What happens when the problem you want to solve becomes too big for any one computer?

When a problem we have becomes too large for a single computer to solve, the problem is broken down into smaller sections and distributed amongst multiple computers on the same network. These computers then simultaneously work on their parts of the problem to eventually solve it.

Suppose 1,000 computers work together. Do you now have one computer that is 1,000 times more powerful? Why or why not?

No, 1,000 computers working together is not 1000x more powerful than a single computer. This is due to several reasons: firstly, as the number of computers solving a single problem increases, they spend more time communicating and less time actually working on the problem. Secondly, in accordance with Amdahl’s law, there exist parts of a problem which simply cannot be broken down into smaller segments and exist as independent functions which take a fixed amount of time and power. Finally, realistically speaking, the failure rate is bound to increase with such a large number of computers.

Can 1,000 computers agree on something if some of them fail or even lie?

Yes, this is possible; however, it can only be done under certain limits of computing consensus. First, regarding failing: as long as more than 50% of computers continue to function properly, agreement can be achieved. Second, regarding lying: this is a case of Byzantine Generals Problem – here, as long as the number of honest computers is strictly above 2/3rds of the total count, the function can survive the lying models.

When you use ChatGPT, Google, Instagram, or an online game, where is the computation actually happening?

When one uses these various online services, the computation tasks are divided between the device locally and a remote server. Most of the heavy tasks are completely done at the servers, which are massive data centers located all over the world. This is particularly prevalent in artificial intelligence model computation, where text prediction, image generation, and the actual work all happens at the servers, and the local device merely acts as a receiver for this data.

If you could make millions of computers behave like one dependable machine, what could humanity build that we cannot build today?

Having such a reliable, immensely powerful machine would eradicate the single most time-consuming task developers face today: error management and data syncing. With this dependable machine, the focus of developers would shift from technical fixes to actually innovating and creating applications. Some examples of what humanity could build with such a device include an extremely precise cellular model of the human body, an incredibly accurate weather tracking system, and an omnipresent, continuously learning AI system. These breakthroughs could fix major problems faced in our daily lives – health management, disaster prediction, and data synchronization respectively.

Find out more about Distributed Systems and Computation with these links!

An intricate introduction to Distributed Systems
How cloud computing on servers really works
A futuristic outlook on how much smarter our computers might get