Building Fast Is Easy. Building Predictably Fast Is Hard.

Photo by Luke Chesser on Unsplash

What Building Go Microservices in Production Taught Me About Performance

Over the years, I’ve spent a lot of time building backend systems, working with Go, microservices, PostgreSQL, Redis, message brokers, and Kubernetes.

One thing I’ve learned repeatedly is that performance problems are rarely where you initially think they are.

When an API becomes slow, my first instinct is no longer to look for a "slow piece of code."

I look at the entire request.

Where did the time go?

Was it waiting for PostgreSQL?

Was a connection unavailable?

Did we make three synchronous service calls?

Was Redis missing?

Was a downstream service slow?

Did a retry make things worse?

Was the pod CPU-throttled?

Or were we simply doing more work than the request actually needed?

This shift in perspective has probably been one of the most valuable things I've learned while working with backend systems.

Here are some of the lessons that have stayed with me.

---

1. Most Performance Problems Aren't Go Problems

I really like Go for backend development.

The language is relatively simple, concurrency is built into the ecosystem, the standard library is strong, and it is very practical for building network-heavy services.

But I've learned not to blame the application language when a service is slow.

I've seen situations where engineers immediately start optimizing Go code while the actual request is spending most of its time waiting for a database or another service.

For example, imagine an API that looks like this:

Request
   ↓
Go Service
   ↓
PostgreSQL
   ↓
Another Service
   ↓
External API

The Go code might take 5–10 ms.

But the external API might take 300 ms.

Optimizing those 10 ms doesn't meaningfully improve the user experience.

This sounds obvious, but under pressure, it's surprisingly easy to forget.

My approach now is simple:

Measure the request before optimizing the implementation.

---

2. I Learned to Look at the Entire Request Path

In a microservice architecture, a request rarely belongs to one service.

A typical request might look something like:

Client
  ↓
API Gateway
  ↓
Authentication
  ↓
Payment Service
  ↓
Merchant Service
  ↓
PostgreSQL
  ↓
Message Broker

When something becomes slow, every step becomes a suspect.

One of the most useful habits I've developed is asking:

"Where exactly is the time being spent?"

That question is much more useful than:

"Why is this API slow?"

Once you break the request into pieces, the problem usually becomes much easier to reason about.

For example:

Authentication       4 ms
Merchant lookup      8 ms
Payment processing  22 ms
PostgreSQL           35 ms
External service    180 ms
Response              3 ms
---------------------------
Total               252 ms

Now we have a direction.

The 22 ms payment-processing code probably isn't the problem.

The 180 ms dependency is.

This is where metrics and distributed tracing have become extremely valuable in my day-to-day work.

---

3. Microservices Made Me More Careful About Network Calls

One of the biggest differences between building a monolith and building microservices is that function calls become network calls.

Inside a process:

user := getUser(id)

is cheap.

Across services:

Service A
   ↓ HTTP/gRPC
Service B

there is serialization, networking, connection handling, processing, and deserialization.

One service call isn't necessarily a problem.

The problem starts when they accumulate.

I've seen request flows that effectively become:

A → B → C → D → E

Every service adds latency and another failure point.

This changed the way I think about service boundaries.

I don't believe a system becomes better simply because it has more microservices.

Sometimes fewer, better-defined services are a much better architecture.

---

4. Database Performance Has Surprised Me More Than Once

When an API is slow, application code gets blamed very quickly.

But PostgreSQL is often where the real work is happening.

One query can dominate the latency of an otherwise efficient service.

For example:

SELECT *
FROM transactions
WHERE merchant_id = $1
ORDER BY created_at DESC
LIMIT 50;

If this query isn't supported by an appropriate index, the application can be perfectly optimized and still be slow.

I've learned to become very comfortable with:

EXPLAIN ANALYZE

Instead of guessing, I want to know what PostgreSQL is actually doing.

Is it performing a sequential scan?

Is it using the expected index?

Are the estimated rows wildly different from the actual rows?

Is a sort operation expensive?

Are we returning far more data than necessary?

These questions usually lead to better solutions than simply increasing server resources.

---

5. Connection Pools Can Quietly Become Bottlenecks

This is another issue that is easy to miss.

Imagine a service with plenty of CPU and memory.

Everything looks healthy.

But requests are still slow.

Then you look at the database connection pool.

Maybe the application has:

Max connections: 20

while hundreds of requests are trying to perform database operations simultaneously.

Now requests are simply waiting.

The database itself might not even be overloaded.

The application is just waiting for a connection.

This taught me an important lesson:

Resource utilization isn't enough. You also need to understand resource contention.

CPU, memory, database connections, goroutines, network connections, and queue capacity can all become bottlenecks.

---

6. More Goroutines Don't Always Mean More Performance

When I started working more deeply with Go, one of the most attractive things was how easy it was to introduce concurrency.

You can turn this:

process(a)
process(b)
process(c)

into something concurrent very easily.

But eventually I learned that concurrency has a cost.

Suppose we create hundreds of goroutines and all of them need PostgreSQL.

We haven't increased the database's capacity.

We've simply increased the number of clients competing for it.

The same applies to downstream APIs.

If a service can handle 1,000 requests per second, sending 5,000 concurrent requests to it doesn't make the dependency faster.

It makes the dependency unhappy.

So these days, I think about concurrency together with capacity.

The question isn't:

"How many goroutines can Go create?"

The better question is:

"How much parallelism can the entire system safely handle?"

---

7. Retries Are One of Those Features That Can Make Things Worse

Retries are useful.

A temporary network failure happens, so we try again.

Sounds reasonable.

But production taught me to be much more careful with them.

Imagine a downstream service normally receives:

1,000 requests/sec

Now it starts experiencing problems.

If every request retries three times, the dependency may suddenly see several times the original traffic.

The service becomes more overloaded.

More requests fail.

More retries happen.

And the problem gets worse.

I've learned to treat retries as part of the system's load model.

When using retries, I think about:

• Maximum retry count
• Exponential backoff
• Jitter
• Timeouts
• Which errors are actually retryable
• Whether the operation is idempotent

Sometimes retrying is exactly what saves the request.

Sometimes retrying is what causes the incident.

---

8. Idempotency Is Not Just a Payment Concept

Working with financial transactions made idempotency especially important for me.

Consider this:

Client
   │
   │ Create Payment
   ▼
Server
   │
   │ Payment succeeds
   ▼
Network failure

The client never receives the response.

What does it do?

It retries.

Now the server receives the same request again.

Without idempotency, we have a potentially dangerous situation.

This is why I like using an idempotency key for operations where duplicate execution matters:

Idempotency-Key: 9d8f...

The system can associate the key with the operation and safely handle retries.

The broader lesson is important:

In distributed systems, "the client didn't receive the response" does not mean "the operation didn't happen."

That distinction affects both reliability and system design.

---

9. I've Become More Comfortable Moving Work Asynchronously

One of the easiest ways to reduce API latency is to stop doing unnecessary work during the request.

Imagine a payment request that performs:

Validate payment
Update balance
Write transaction
Send SMS
Send email
Update analytics
Generate report

Does the customer really need to wait for all of these?

Usually not.

A better approach might be:

Payment API
    │
    ├── Critical transaction
    │
    └── Message Broker
           ├── Notification
           ├── Analytics
           └── Reporting

This can significantly improve response time.

But asynchronous processing doesn't eliminate complexity.

It introduces different problems:

• Duplicate messages
• Message ordering
• Consumer failures
• Retries
• Dead-letter queues
• Eventual consistency

I've learned that asynchronous processing is powerful, but it should be used deliberately.

---

10. Redis Taught Me That Fast Doesn't Always Mean Simple

Redis can make an enormous difference to application performance.

A database query that takes milliseconds can potentially become a much faster cache lookup.

But introducing Redis also means introducing another consistency problem.

Now you have:

PostgreSQL
     +
Redis

The question becomes:

"Which one is the source of truth?"

And then:

"When does Redis get updated?"

And:

"What happens if Redis contains stale data?"

Caching taught me that performance improvements often introduce new correctness and operational concerns.

I now think about caching in terms of:

latency + consistency + invalidation + failure behavior

rather than simply:

"Redis is fast."

---

11. Kubernetes Doesn't Fix Architecture

Kubernetes is excellent at running distributed applications.

But putting an application inside Kubernetes doesn't automatically make it scalable.

Suppose we have:

3 Pods
   ↓
PostgreSQL

and increase it to:

30 Pods
   ↓
PostgreSQL

The application now has ten times more instances.

But if PostgreSQL is the bottleneck, we've potentially made the problem worse.

The same applies to:

• Redis
• External APIs
• Message brokers
• Connection pools
• CPU
• Network bandwidth

I've learned to think about scaling as:

Where is the bottleneck moving when I scale this component?

That's a much more useful question than simply asking how many pods we should run.

---

12. Performance Usually Means Finding the Bottleneck

Over time, I've become less interested in theoretical optimization and more interested in finding the actual bottleneck.

When something is slow, my investigation usually starts with:

What changed?
      ↓
What does the metric say?
      ↓
Where is the latency?
      ↓
What resource is saturated?
      ↓
Can I reproduce it?
      ↓
Can I profile it?
      ↓
What is the smallest change that can fix it?

For Go, that might mean using:

go test -bench=.

or profiling with:

go tool pprof

For PostgreSQL:

EXPLAIN ANALYZE

For distributed systems:

• Metrics
• Logs
• Traces
• Queue depth
• Connection-pool statistics

The tools are different, but the mindset is the same:

Don't guess. Measure.

---

13. The Most Important Performance Lesson

If I had to summarize what I've learned about backend performance in one sentence, it would be:

Don't optimize the code until you understand the system.

A 10 ms optimization doesn't matter if the request waits 500 ms for another service.

Adding more goroutines doesn't help if the database connection pool is exhausted.

Adding more pods doesn't help if PostgreSQL is already saturated.

Adding retries doesn't help if the dependency is already overloaded.

Adding Redis doesn't help if the application has a poor cache invalidation strategy.

And rewriting perfectly good Go code doesn't help if the real problem is an inefficient SQL query.

The best performance improvements I've seen usually came from identifying where the system was actually spending its time.

---

14. What I'm Still Learning

One thing I like about backend engineering is that there is always another layer.

You solve an application bottleneck and discover a database bottleneck.

You solve the database bottleneck and discover a connection-pool problem.

You increase capacity and discover a consistency problem.

You introduce asynchronous processing and discover an ordering problem.

You fix that and discover an observability gap.

There is rarely a final state where everything is "solved."

That's probably what keeps the work interesting.

The technologies will continue to change—Go versions will evolve, infrastructure will change, new databases and messaging systems will appear—but the fundamental questions remain:

Where is the bottleneck?

What happens when something fails?

How does the system behave under load?

What trade-off are we making?

How do we know the change actually improved things?

Those are the questions I find myself asking more and more as I work on backend systems.

And, for me, that's what good backend engineering is really about.

Not writing the most clever code.

Not using the most fashionable technology.

But building systems that behave predictably, efficiently, and reliably when the real world starts putting pressure on them.