Load Balancing
Load Balancing
What is load balancing?
- Spread requests evenly across multiple servers
- Prevents overload on any one server
- The foundation of scalability and reliability
The moment you add load balancing, a server changes from “a one-of-a-kind asset” into a part you can swap out at any time. When you deploy, you pull servers out one at a time, push the new version, and put them back, and users notice nothing. The old habit of posting a maintenance notice and shutting the service down at dawn disappears, and you become a team that deploys on a weekday afternoon. It is the point where the way people work changes before the infrastructure does.
There are two main layers at which traffic is split. L4 forwards based on IP and port only, which is fast but blind to what a request contains. L7 reads URLs, headers, and even cookies, so it can route /api to API servers and /img to image servers. Routing rules vary too: round robin cycles through servers in order, least connections sends traffic to the server with the fewest active connections, and IP hash pins the same user to the same server. For services with uneven processing times, least connections works best. On top of that, health check interval and failure threshold, for example removing a server after 3 consecutive failures at 10-second intervals, determine how fast you respond to outages.
That said, a load balancer does not add servers by itself. Autoscaling adjusts the number of servers automatically, a CDN serves images and video from locations close to users, and failover hands over to a standby system when the primary system goes down entirely. Only when all three work together can you actually withstand real traffic. If you put a load balancer in front of just two servers, both will go down together under heavy load.
Sessions are where things break most often in practice. If login state lives in a server’s memory, users get logged out every time a request lands on a different server, and the support tickets about vanished shopping carts start pouring in. Under pressure, teams often patch it with sticky sessions that pin each user to one server, but when that server dies, every user on it gets kicked out at once. The standard fix is to move sessions into a shared store such as Redis. Health checks are another trap: if they only confirm that a port responds, an application that has lost its database connection still reports as “healthy,” and requests keep flowing to a dead server.
Seungtae ParkA hands-on AX instructor who teaches AI business development and marketing automation by building them in practice
Seolyeong ParkGenerative AI and AX training expert with 17 years of hands-on experience and 200+ lectures, helping non-specialists put AI to work right awayA networking technology that evolved from the need to handle traffic for large-scale web services and has become a basic building block of cloud infrastructure.
Consider a hypothetical case. Company A, a household-goods e-commerce business with about KRW 40 billion (≈ US$29,926,700) in annual revenue and 180 employees, saw orders surge during its sale event in the first week of every month. At the March event, a single web server hit 98% CPU just 10 minutes after launch, and the checkout page stayed down for 12 minutes. That day alone, over 300 cancellations and inquiries piled up.
- When Park, the infrastructure team lead, dug through the logs, there were already 2 servers, but because it was DNS round robin, requests kept going to the one that had stopped. “There was no mechanism at all to check whether a server was alive.”
- They put a cloud L7 load balancer in front, scaled the web servers to 4, and changed the health-check path to verify the DB connection as well, so a server is automatically removed after 3 failed checks at 10-second intervals.
- When logins kept dropping in the first test, Dev Lead Kim pointed out that “sessions live in server memory,” and over two days the team moved session storage to Redis.
- In a load test with 8,000 concurrent users, round robin pushed CPU on certain servers up to 90%, so the team switched the distribution method to least connections and moved image requests off to a CDN.
- During the June sale, the site stayed up with 12,000 concurrent users, average response time fell from 1.8 seconds to 0.6 seconds, and checkout-failure inquiries dropped from about 300 to 4.