₦airaland Forum

Welcome, Guest: RegisterLoginWith GoogleTrendingRecentNew

Stats: 3,330,557 members, 8,446,010 topics. Date: Wednesday, 15 July 2026 at 10:28 PM

Toggle theme

The Night The Server Refused To Sleep - Programming - Nairaland

Nairaland ForumScience/TechnologyProgrammingThe Night The Server Refused To Sleep (155 Views)

1 Reply

The Night The Server Refused To Sleep by toamutiltech(op): 8:36pm On Apr 04
At 11:47 PM, the office lights of a small fintech startup in Abuja were still on.
Most of the team had gone home hours earlier, but Daniel remained in front of his laptop, staring at a dashboard full of red alerts.
Daniel was the backend developer responsible for the company’s payment system.
And something was very wrong.
The transaction queue was growing rapidly.
Normally, the system processed payments in seconds, but tonight the delay had climbed to almost two minutes per transaction.
In fintech, two minutes might as well be two hours.
Customers were already complaining.
Daniel opened the system logs.
Thousands of payment requests were coming in through the API, and the server CPU usage had jumped to 95%.
At first, he suspected a DDoS attack, where attackers flood a server with fake requests.
But the requests looked legitimate.
They were all coming from real users.
Daniel leaned back and thought.
Earlier that week, the team had released a new feature: automatic wallet transfers between users.
The idea was simple.
A user could schedule recurring transfers, and the system would execute them automatically.
The feature worked perfectly during testing.
But production traffic was a different beast.
Daniel opened the code responsible for processing transfers.
The logic worked like this:
Receive transfer request
Check sender balance
Deduct amount
Add amount to receiver wallet
Record the transaction
Everything looked correct.
But something bothered him.
He noticed the system was performing multiple database queries for each transfer.
One query to check the balance.
Another to deduct funds.
Another to update the receiver’s wallet.
Another to log the transaction.
Under heavy load, this meant the database was handling thousands of separate operations per second.
The database wasn’t under attack.
It was simply overwhelmed.
Daniel quickly wrote a script to simulate high traffic locally.
Within minutes, the same slowdown appeared.
He had found the problem.
The transfer process needed to run inside a single database transaction.
A database transaction ensures that multiple operations happen as one unit. If one step fails, everything is rolled back, keeping the data consistent.
More importantly, it reduces the number of database calls.
Daniel refactored the code.
Now the process looked like this:
Start transaction
Lock sender wallet row
Verify balance
Deduct funds
Credit receiver
Record transaction
Commit transaction
This approach ensured that the system handled transfers atomically and efficiently.
He deployed the fix.
For a moment, nothing changed.
The queue was still long.
Then slowly, the numbers began to drop.
1200 pending transactions.



Finally, the dashboard turned green.
The system was processing payments in under 300 milliseconds again.
Daniel smiled and stretched.
Just as he was about to close his laptop, a message popped up from the CTO.
“Great work tonight. Monitoring shows the system stabilized. What happened?”
Daniel replied:
“Database bottleneck. Too many queries per transaction. Fixed using transactional processing and row locking.”
A minute later she responded.
“This is why good backend engineers are hard to find.”
Daniel shut down his laptop and stepped outside.
The city was quiet.
Most people would never know how many things had to work perfectly for a simple payment to succeed.
But Daniel knew.
Sometimes the difference between chaos and stability was just a few lines of better code.
And tonight, the server could finally sleep.

Re: The Night The Server Refused To Sleep by Alphabyte3: 10:16pm On Apr 04
Maybe a multiple cdn server might fix it if it due to DDoS attack. Most big site use it with hybrid cloud and geolocation to avoid all parts of the site or app downtime
Re: The Night The Server Refused To Sleep by Shomek(m): 11:23pm On Apr 04
toamutiltech:
At 11:47 PM, the office lights of a small fintech startup in Abuja were still on.
Most of the team had gone home hours earlier, but Daniel remained in front of his laptop, staring at a dashboard full of red alerts.
Daniel was the backend developer responsible for the company’s payment system.
And something was very wrong.
The transaction queue was growing rapidly.
Normally, the system processed payments in seconds, but tonight the delay had climbed to almost two minutes per transaction.
In fintech, two minutes might as well be two hours.
Customers were already complaining.
Daniel opened the system logs.
Thousands of payment requests were coming in through the API, and the server CPU usage had jumped to 95%.
At first, he suspected a DDoS attack, where attackers flood a server with fake requests.
But the requests looked legitimate.
They were all coming from real users.
Daniel leaned back and thought.
Earlier that week, the team had released a new feature: automatic wallet transfers between users.
The idea was simple.
A user could schedule recurring transfers, and the system would execute them automatically.
The feature worked perfectly during testing.
But production traffic was a different beast.
Daniel opened the code responsible for processing transfers.
The logic worked like this:
Receive transfer request
Check sender balance
Deduct amount
Add amount to receiver wallet
Record the transaction
Everything looked correct.
But something bothered him.
He noticed the system was performing multiple database queries for each transfer.
One query to check the balance.
Another to deduct funds.
Another to update the receiver’s wallet.
Another to log the transaction.
Under heavy load, this meant the database was handling thousands of separate operations per second.
The database wasn’t under attack.
It was simply overwhelmed.
Daniel quickly wrote a script to simulate high traffic locally.
Within minutes, the same slowdown appeared.
He had found the problem.
The transfer process needed to run inside a single database transaction.
A database transaction ensures that multiple operations happen as one unit. If one step fails, everything is rolled back, keeping the data consistent.
More importantly, it reduces the number of database calls.
Daniel refactored the code.
Now the process looked like this:
Start transaction
Lock sender wallet row
Verify balance
Deduct funds
Credit receiver
Record transaction
Commit transaction
This approach ensured that the system handled transfers atomically and efficiently.
He deployed the fix.
For a moment, nothing changed.
The queue was still long.
Then slowly, the numbers began to drop.
1200 pending transactions.



Finally, the dashboard turned green.
The system was processing payments in under 300 milliseconds again.
Daniel smiled and stretched.
Just as he was about to close his laptop, a message popped up from the CTO.
“Great work tonight. Monitoring shows the system stabilized. What happened?”
Daniel replied:
“Database bottleneck. Too many queries per transaction. Fixed using transactional processing and row locking.”
A minute later she responded.
“This is why good backend engineers are hard to find.”
Daniel shut down his laptop and stepped outside.
The city was quiet.
Most people would never know how many things had to work perfectly for a simple payment to succeed.
But Daniel knew.
Sometimes the difference between chaos and stability was just a few lines of better code.
And tonight, the server could finally sleep.
Nice write up 👍
Keep it up t
1 Reply

The Night The Server CrashedThe Night The Server CrashedHow Do You Deploy Laravel Lumen 5.7 On The Server234

Unlock Superior Results With The Right Pressure Washing EquipmentCrypto Profit KitHow To Master Debugging Like A Pro