Key Achievements at a Glance​

5000 CCU Capacity Test​

• 🚀 5,000 CCU Peak Capacity: Perfectly maintained 5,000 concurrent active WebSocket connections.
• ⚡ 14,000 ops/sec Broadcast: Handled massive throughput at peak capacity, pushing 2.6+ million updates in 10 minutes.
• 📉 30% Latency Improvement: Significantly improved real-time responsiveness, reducing P50 streaming latency from >1s down to 500ms.
• 🧠 Optimized Hardware Footprint (~2 CPU Cores): Reduced CPU usage by up to 10% (stabilizing at 2 cores), sustaining the absolute maximum load with high efficiency.
• 🛡️ Zero Resource Leaks & 33% Less RAM: Slashed memory consumption from 300MB to 200MB (a 33% reduction on BC11) while ensuring perfect Goroutine cleanup when 5,000 users abruptly disconnect.

Metric	Before	After
CPU BC11(G): 2.2 -> 2 (10% reduction) BC15(Y): 2.1-> 2 (5% reduction)
Memory BC11(G): 300MB -> 200MB (33% reduction) BC15(Y): 512 MB -> 512 MB (0% reduction)
P50 Streaming latency (Y) >1s -> 500ms (50% reduction)

1500 CCU Sudden Spike Test​

• 🚀 1,500 CCU Peak Capacity: Perfectly maintained 1,500 concurrent active WebSocket connections under sudden spike test conditions.
• 📉 Up to 40% Latency Reduction: Significantly improved real-time responsiveness under heavy load, slashing P99 streaming latency from 1s to 600ms (40% faster) and P99 subscription latency to 320ms (36% faster).
• 🧠 Optimized Hardware Footprint: Drastically improved compute efficiency, reducing CPU usage by up to 50% (stabilizing at just 0.6 to 0.8 cores) to sustain maximum load.
• 🛡️ Zero Leaks & Robust Resilience: Slashed peak Goroutine count by 76% (from 37k down to 9k) and memory consumption by 37% (270MB down to 170MB), ensuring perfect resource cleanup even when 1,500 users abruptly disconnect.

Metric	Before	After
Memory BC11(G): 90MB -> 90MB (0% reduction) BC15(Y): 270MB -> 170MB (37% reduction)
CPU BC11(G): 1.2 -> 0.8 (33% reduction) BC15(Y): 1.2 -> 0.6 (50% reduction)
Subscription P99 Latency 500ms -> 320ms (36% reduction)
Streaming P99 Latency 1s -> 600ms (40% reduction)
Goroutine Count at peak 37000 -> 9000 (76% reduction)

Load Test Results​

• Capacity Test (5,000 VUs): Perfectly maintained 5,000 concurrent active WebSocket connections. The system successfully distributed over 260+ million real-time price updates within 10 minutes, with a peak broadcast rate of 14,000 ops/sec. It achieved this using an optimized hardware footprint of only ~2 CPU cores (a 10% reduction) and 200MB RAM (a 33% reduction). Furthermore, P50 streaming latency was reduced by 50% down to 500ms, with zero memory or goroutine leaks after the test ended.
• Spike Test (1,500 VUs): Successfully handled massive connection churns, proving the system's robust resilience under sudden traffic surges and drops. Compute efficiency was drastically improved, requiring only 0.6 to 0.8 CPU cores (up to 50% reduction) and slashing peak Goroutine count by 76% (from 37,000 down to 9,000). Real-time responsiveness was significantly enhanced, achieving a 320ms P99 subscription latency (36% faster) and 600ms P99 streaming latency (40% faster).

Note: I intentionally set up 200ms frequency to batch the price updates to simulate the real-world market volatility.

Challenges & Solutions​

Throughout the testing phases, several critical concurrency and resource management issues were exposed and resolved:

1. Race Conditions & Panics​

• Issue: panic: close of closed channel occurred when multiple goroutines (API, background cleanup, WebSocket close) attempted to deactivate the same session concurrently.
• Solution: Made the deactivateSession function idempotent using a session-level lock (if !session.IsActive { return }) before cleaning up resources.

2. Resource & Goroutine Leaks​

• Redis Pub/Sub Leak: Connections leaked when stream.Send() errors bypassed the session deactivation. Fixed by ensuring deactivateSession is called within a defer statement.
• WebSocket Goroutine Leak: The API gateway accumulated up to 40k leaked goroutines for only 1.3k connections due to improper context cancellation and unhandled TCP drops. Fixed by:
  • Returning and properly deferring the cancel function from custom WithTimeout contexts.
  • Implementing explicit conn.Close() on context completion instead of relying on blocked actions in select default cases.
  • Adding Ping/Pong handlers with read deadlines (conn.SetReadDeadline) to gracefully detect and close dead connections that didn't send a TCP FIN packet.
• Long-tail Resource Leak: Calling close(session.PriceStream) caused panics if background goroutines were still interacting with it. Fixed by using context.CancelFunc to unblock receivers and letting the Garbage Collector (GC) safely reclaim the unclosed channels.

3. Performance Tuning & Micro-optimizations​

• O(1) Map Lookup: Replaced the array-based priceChannels management with a nested map (map[string]map[string]chan) keyed by sessionID. This O(1) deletion significantly reduced overhead and lock contention during high-churn disconnection events.
• Timestamp Type Conversion: Avoided expensive string-to-int parsing (strconv.ParseInt) by changing the Protobuf timestamp type to int64. This micro-optimization, combined with the structural changes, was the key driver in slashing the P99 latency down to 600ms and lowering overall CPU usage by up to 50%.

Key Takeaways​

1. Let GC Handle Data Channels: "Never close a channel from the receiver (Session) side. Only close it from the sender, or let GC handle it." Use Context to signal cancellations instead of explicitly closing data channels that might still have concurrent senders, avoiding close of closed channel panics.
2. Avoid Blocking in select default: "In a loop, never put blocking functions (like network reads, channel receives) in the default case of a select!" It defeats the purpose of the select statement and blocks the goroutine from receiving the context cancellation signal.
3. Context Cancellation Responsibility: "Who calls, who is responsible for cancellation." Always defer the cancel function in the exact same scope where the context was created to guarantee execution.
4. WebSocket Lifecycle Management: HTTP connections upgraded to WebSockets require explicit, manual closure. They do not automatically tear down when the HTTP context is done, nor do they reliably emit TCP FIN packets if the client abruptly disconnects or loses network. Always implement application-level Ping/Pong heartbeat and zombine cleanup mechanisms.

Check the series of posts:​

• Streaming Engine to 5k+ Users (0) - Set up
• Streaming Engine to 5k+ Users (1) - Debugging a Race Condition in SessionManager.
• Streaming Engine to 5k+ Users (2) - Debugging Leaked Redis Pub/Sub Connections
• Streaming Engine to 5k+ Users (3) - Debugging Leaked Goroutine with Websocket
• Streaming Engine to 5k+ Users (4) - Shifting to Capacity Test
• Streaming Engine to 5k+ Users (5) - Performance Tuning
• Streaming Engine to 5k+ Users (6) - Micro-optimization
• Streaming Engine to 5k+ Users (7) - Debugging Long-tail Resource Leaks

There is a long-tail resource leak issue that may affect the performance. It happened when the server was idle for a while after a load test without restarting the server.

Conclusion​

I fixed the long-tail resource leak issue. No more error logs are shown after the load test.

If you are interested in the testing process, please continue reading.

Observations​

Long-tail Resource Leaks​

goroutine 23228 [chan receive]:
lzstock/mods/bc11-market-monitor/useCases.(*RedisSubscriptionManager).listenToRedis(0xc000463e90, {0xc00030c350, 0x3}, 0xc00098f7a0?)
  /build/mods/bc11-market-monitor/useCases/RedisSubscriptionManager.go:88 +0x65
created by lzstock/mods/bc11-market-monitor/useCases.(*RedisSubscriptionManager).AddSubscription in goroutine 23223
  /build/mods/bc11-market-monitor/useCases/RedisSubscriptionManager.go:54 +0x345

goroutine 23206 [select]:
lzstock/mods/bc11-market-monitor/useCases.(*StreamStockPriceUseCase).StreamPriceUpdates(0xc00013eee0, 0xc0004c19e0, {0x114bc30, 0xc0028285f0})
  /build/mods/bc11-market-monitor/useCases/StreamStockPrice.go:49 +0x4d0
lzstock/mods/bc11-market-monitor/controllers.(*Controller).StreamPriceUpdates(0xc00304c780?, 0xf4cb40?, {0x114bc30?, 0xc0028285f0?})
  /build/mods/bc11-market-monitor/controllers/StockPrice.go:52 +0xd7
lzstock/shared/go/protos/bc11_market_monitor._RealTimePriceMonitoringService_StreamPriceUpdates_Handler({0xea5ce0, 0xc00011a2d8}, {0x114a8b0, 0xc00304c780})
  /build/shared/go/protos/bc11_market_monitor/bc11_market_monitor.service.pb.go:474 +0x110
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000478000, {0x1147190, 0xc00295d0b0}, 0xc00125c600, 0xc00047a000, 0x1837b00, 0x0)
  /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1722 +0x12e8
google.golang.org/grpc.(*Server).handleStream(0xc000478000, {0x1147830, 0xc000326000}, 0xc00125c600)
  /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1846 +0xb47
google.golang.org/grpc.(*Server).serveStreams.func2.1()
  /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1061 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 1582
  /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1072 +0x11d

Root Cause Analysis: Long-tail Resource Leaks​

Sender: RedisSubscriptionManager's broadcast loop.

Receiver: gRPC's StreamPriceUpdates.

Owner: sessionManager (because it creates the channel using make(chan)).

I got a mixed suggestions that I should close the channel actively or rely on GC to close it. I decided to leave it out there. Unfortunately, it is not a good idea. That was the reason for this issue.

// Session Manager
// 1. Detach this session's channel from the global Redis listener
for _, ticker := range session.Tickers {
    sm.redisSubManager.RemoveSubscription(ticker, sessionID) 
}

// 2. Close the channel
// close(session.PriceStream)

The easiest way to fix this issue is to uncomment the line that closes the channel. However, it is a bad smell. Relying on execution order to safely close a channel is a very fragile design

Solution​

Never Close Data Channel, use context instead

// SubscribeStockPriceUseCase

func (u *SubscribeStockPriceUseCase) Executes(dto *dtos.StartPriceMonitoring) (err error) {
  defer appErrs.Recover(&err)
  if len(dto.Req.Tickers) == 0 {
    return appErrs.ErrEmptyTickers
  }

  // Create a monitoring session
  session := &session{
    ID:         uuid.New().String(),
    ...
    PriceStream: make(chan *schema.MarketPrice, 1000),
+   ctx    context.Context
+   cancel context.CancelFunc
  }

  return nil
}

func (sm *sessionManager) deactivateSessionLocked(sessionID string) {
  session := sm.sessions[sessionID]
  ...
+ session.cancel()
  ...
}

func (u *StreamStockPriceUseCase) StreamPriceUpdates(dto *dtos.StreamPriceUpdates, stream pb.RealTimePriceMonitoringService_StreamPriceUpdatesServer) (err error) {

  ...
  for {
    select {
    case price := <-session.PriceStream:
      ...
    case <-flushTicker.C:
      ...
+   case <-session.ctx.Done():
+     return nil
    case <-stream.Context().Done():
      return return stream.Context().Err()
  }
}

Result​

Rule: Receiver and Manager should not close the data channel.

Solution: Use RemoveSubscription to block the data source, let PriceStream be collected by GC gracefully. Use context.CancelFunc to replace close, wake up and end the zombie Goroutine stuck in the select.

There is a type conversion issue that may affect the performance. Let's fix it and observe whether the performance is improved or not.

Conclusion​

I fixed the type conversion issue. The P99 latency is reduced from 700ms to 450ms roughly and CPU usage is reduced a bit from from 0.8 to 0.5 roughly.

If you are interested in the testing process, please continue reading.

Optimization​

message MarketPrice {
  string ticker = 1; // "TSLA"
  TradePrice trade = 2;
  PriceChange price_change = 3;
- string timestamp = 4;
+ int64 timestamp = 4;
}

now := time.Now().UnixMilli()
for _, result := range stock.Results {
- if sourceTs, err := strconv.ParseInt(result.Timestamp, 10, 64); err == nil {
-   latency := float64(now - result.Timestamp)
- }
+ latency := float64(now - result.Timestamp)
}

Result​

CPU & Memory

CPU usage is reduced a bit from from 0.8 to 0.5 roughly (Yellow line).

Streaming Latency

P99 latency is reduced from 700ms to 450ms roughly (Green line).

I am alomst done with the load test. There is just one last piece of the puzzle to fix that is so called Long-tail resource leak. I will cover it in the next article.

There is a minor issue in the price channel management that may affect performance when conection rapidly built and destroyed. Let's fix it and observe whether the latency meter could be improved or not.

Conclusion​

I updated the price channel management to use a map with sessionID. It is more efficient than the previous array. The P99 latency is reduced from above 1 second to around 800ms.

If you are interested in the testing process, please continue reading.

Spike Test​

To oberve whether the performance is improved or not, a spike test is useful than capacity test. Because only in the extreme environment, the spike test can test whether the RLock will cause lock contention, and whether the O(1) Map deletion will help CPU save power.

export const options = {
  stages: [
    { duration: '3m', target: 1000 },  // Ramp-up: from 0
    { duration: '1m', target: 1000 },  // Steady state: observe baseline
    { duration: '30s', target: 1500 }, // Spike: Sudden influx of concurrent users
    { duration: '1m', target: 1500 },  // Sustained peak: test Lock Contention
    { duration: '10s', target: 0 },    // Recovery: Rapid scale down
  ],
};

Capacity Test Baseline​

type RedisSubscriptionManager struct {
  ...
  priceChannels map[string][]chan *schema.MarketPrice                       
	priceChannels map[string]map[string]chan *schema.MarketPrice
}

func (rsm *RedisSubscriptionManager) AddSubscription(ticker string, sessionID string, stream chan *schema.MarketPrice) {
- rsm.priceChannels[ticker] = append(rsm.priceChannels[ticker], stream)

+ if rsm.priceChannels[ticker] == nil {
+ 	rsm.priceChannels[ticker] = make(map[string]chan *schema.MarketPrice)
+ }
+
+ rsm.priceChannels[ticker][sessionID] = stream
}

func (rsm *RedisSubscriptionManager) RemoveSubscription(ticker string, sessionID string) {
- channels := rsm.priceChannels[ticker]
- for i, ch := range channels {
-   if ch == streamToRemove {
-     // 從 slice 中移除該 channel
-     rsm.priceChannels[ticker] = append(channels[:i], channels[i+1:]...)
-     break
-		}
-	}
+ subs, exists := rsm.priceChannels[ticker]
+ if !exists {
+   return
+ }
+ // Safely and accurately remove the specific session
+ delete(subs, sessionID)

- if len(rsm.priceChannels[ticker]) == 0 {
+ if len(subs) == 0 {
+   delete(rsm.priceChannels, ticker)
+ }
}

Observations​

1. 99th Percentile Latency (Streaming Flow)​

P99 latency of streaming flow is reduce a bit from above 1 second to around 800ms.

2. CPU​

CPU usage is reduced a bit but not much.

CPU & Memory, Fanout Drop, Latency

CPU & Memory	Baseline
	Upgrade to PriceMap
Fanout Drop	Baseline
	Upgrade to PriceMap
Latency	Baseline
	Upgrade to PriceMap

Next Step​

I am almost done with the load test. In the next article, I am going to fix a type conversion issue that may affect the performance.

Next step I test the last part of the load test. Let's do the capacity test with 5000 users CCU.

Conclusion​

The system can now handle 5000 users CCU within 5 mins and hold for another 5 mins. All of them will continuously receive message through websocket.

If you are interested in the testing process, please continue reading.

Capacity Test Round​

export const options = {
  stages: [
    { duration: '5m', target: 5000 },  // 5 minutes ramp-up to 5000 VU
    { duration: '5m', target: 5000 },  // Maintain 5000 users stable (quiet period)
    { duration: '30s', target: 0 },    // Test end, graceful shutdown within 30 seconds
  ],
};

Observations​

1. K6 Result ✅​

The test was 100% successful. No failed requests.

2. BC11 Active WS Connection ✅​

The green line climbed slowly and precisely stopped at 5000, maintaining stability for 5 minutes, and then dropped vertically to zero instantly. This proves that your k6 script executed exactly as planned, and the server perfectly handled these 5000 long connections.

3. Goroutine Count ✅​

After the test ended, all Goroutines were zero leaked and instantly recycled.

4. Messages / Fanout ✅​

These people received nearly 2.6+ million (ws_msgs_received) real-time stock price updates within 10 minutes! Your server's Fanout peak reached 12K ~ 14K ops/s (processing 14,000 broadcasts per second).

5. CPU & Memory ✅​

Memory: The Go memory pool was like a standard textbook (ramp-up -> enter stable plateau period -> maintain available water level after test end). No Memory Leak.

CPU: The peak was approximately 2.5 cores. Using 2.5 CPUs to handle 5000 users and processing thousands of broadcasts per second is extremely efficient!

Result​

After a series of thorough underlying Goroutine lifecycle and parallel lock reconstruction, our push engine finally faced the ultimate test: 5000 CCU capacity test (Capacity Test).

In this test, the system not only perfectly maintained 5000 active WebSocket connections but also successfully distributed 2.6+ million real-time price update messages within a few minutes, with a peak broadcast rate of 14,000 ops/sec. The most exciting part was that when 5000 users simultaneously disconnected, the system resources were precisely recycled like a textbook, with more than 30,000 Goroutines instantly returning to zero, completely eliminating Memory/Goroutine Leak. This proves that our architecture not only has high concurrent processing capabilities but also has extremely stable and scalable capabilities.

Until now, both spike test and capacity test are successful. Let's shift to the next article to see if there are any other issues or performance tuning opportunities.

The real show begin. Let's do the spike test with 1500 VUs and see what will be happened.

Conclusion​

I spotted a lot of goroutine leaked. I fixed it. The system can now handle 1500 VUs without any issues.

This is the most time consuming part of the load test.

If you are interested in the debugging process, please continue reading.

Spike Test Round​

export const options = {
  stages: [
    { duration: '3m', target: 1000 },  // Ramp-up: from 0
    { duration: '1m', target: 1000 },  // Steady state: observe baseline
    { duration: '30s', target: 1500 }, // Spike: Sudden influx of concurrent users
    { duration: '1m', target: 1500 },  // Sustained peak: test Lock Contention
    { duration: '10s', target: 0 },    // Recovery: Rapid scale down
  ],
};

Observations​

Gateway Memory was leaking ❌​

It was wired having 35000-40000 goroutine with only 1300 WS connections during the peak.

Fan-out arch is successful ✅​

Message successful rate was met around 4.5k ops/s, while drop rate is almost zero, together with message pushed rate. It means inside BC11 no message is blocked during the test.

Hardware limit is far from reached ✅​

Service's CPU didn't meet limit​

According to the graph, CPUs are almost idle (CPU reach 1.3 core * 2 = 2.6 / 8 cores) and it usually indicates that our system is meeting I/O blocking or Deadlock.

Redis didn't have load pressure ✅​

Based on the graph, I can confirm that I successfully shifted pressure to service from redis by fan-out arch through a one-ticker-one-connection architecture.

Redis PubSub Connection was leaking ✅​

The Redis PubSub connection number is back to baseline.

Debugging​

I thought the issues may be related to the low level network or wrong configuration. So I tried to by pass Docker Desktop Proxy and increase all the congfiguration limits that I could find to see if it can resolve the issue.

By pass Docker Desktop Proxy​

In case port proxy is a bottleneck when docker desktop forward inside docker desktop. I created a new temporary K6 container firing test inside of docker desktop to bypass the possible proxy issue.

  k6:
    image: grafana/k6:latest
    labels:
      - "load-test"
    volumes:
      # load local script
      - ./k6/:/scripts   
    networks:
      - lzstock-network
    profiles:
      - tools

docker compose -p load-test run --rm k6 run /scripts/load-test.js

gRPC MaxConcurrentStreams​

I also increased the MaxConcurrentStreams to 65535 to prevent connection drop.

g.ServerInstance = grpc.NewServer(
  ...
  grpc.MaxConcurrentStreams(65535),
)

TCP Backlog​

Linux's TCP connection may be jammed. To prevent connection drop, this value is needed.

bc11:
  sysctls:
    - net.core.somaxconn=65535 

File Descriptor Limit / ulimit​

In Linux, each Websocket Connection is a TCP Socket, and each socket needs a File Descriptor. In case of disruption of connection, I updated the upper limit.

bc11:
  ulimits:
    nproc: 65535
    nofile:
      soft: 65535
      hard: 65535

Root Cause Analysis with pprof​

Everything mentioned above didn't resolve the issue. So I started to analyze the pprof results.

go tool pprof http://localhost:50052/debug/pprof/goroutine
go tool pprof -http=:8082 pprof.app.goroutine.pb.BC15-0.gz

At the top right of the pprof screenshot, it shows unexpected high value of goroutine which was 17505 (100%). The runtime.gopark means the Goroutine is putting to sleep because it is waiting on a select statement. The custom withTimeout function is the reason that cuased the leak. A anaymous background Goroutine (func1) wasn't canceled cleanly. So it got stuck waiting forever.

Solution​

failed to cancel websocket connection​

First attempt

The implicit cancel call is inside WithTimeout(). It is fine if it is for HTTP connection, but not quite good for websocket, which relies on explicit cancel call to prevent leaked goroutine. HTTP connection is upgraded to a long live state after websocket connection. So it is necessary to close connection manually instead of fully relying on Done() method in the framework.

Fasthttp reuse RequestCtx and put it back to sync.Pool. When it called upgrader.Upgrade(), this conneciton is over from Fasthttp's perspective. It will put the RequestCtx back to the pool. So the goroutine that is waiting on the ctx.Done() will be executed wrongly.

Original v.s. Fixed Code is this:

ctx := LZContext.New(routingCtx).WithTimeout(5 * time.Minute)

func (cb *LZStockContext) WithTimeout(timeout time.Duration) (context.Context){
  ...
  ctx, cancel := context.WithTimeout(context.Background(), timeout)
  ...
- go func() {
-   select {
-   case <-cb.routingCtx.RequestCtx.Done():
-     cancel() // cancel when HTTP request end
-   case <-ctx.Done():
-     return
-   }
- }()  
  ...
- return ctx
+ return ctx, cancel
}

In the upper layer or the caller level. A caller should be responsible for canceling the context.

- ctx := LZContext.New(routingCtx).WithTimeout(5 * time.Minute)
+ ctx, cancel := LZContext.New(routingCtx).WithTimeout(5 * time.Minute)
+ defer cancel()

Second attempt

In both handleGRPCStream() and listenWebSocketMessages(), I put select + default and called two blocked action in default case. I set read deadline to 15 minutes in listenWebSocketMessages(), and called stream.Recv() in handleGRPCStream(). Either of them will be blocked if the context Done() was called.

So, I removed select + default and listened to context Done() in the upper layer and call conn.Close() if the context was done. So the connection will be closed explicitly when the context was done.

Original v.s. Fixed Code is this:

func handleGRPCStream(ctx context.Context, conn *websocket.Conn, req *pb.StreamPriceUpdatesReq, sessionID, investorID string) error {
  ...
  for {
-   select {
-   case <-ctx.Done():
-     return ctx.Err()
-   default:
      stock, err := stream.Recv()
      if err != nil {
        if err == io.EOF {
          return nil // 正常結束
        }
        return fmt.Errorf("failed to receive from stream: %w", err)
      }

      if err := sendToWebSocket(conn, stock, sessionID, investorID); err != nil {
        return fmt.Errorf("failed to send to WebSocket: %w", err)
      }
    }
  }
}

func listenWebSocketMessages(ctx context.Context, conn *websocket.Conn, sessionID, investorID string) error {
  for {
-   select {
-   case <-ctx.Done():
-     return ctx.Err()
-   default:
      // Set read timeout
      conn.SetReadDeadline(time.Now().Add(15 * time.Minute))

      messageType, message, err := conn.ReadMessage()
      if err != nil {
        if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway, websocket.CloseAbnormalClosure) {
          return fmt.Errorf("WebSocket read error: %w", err)
        }
        return fmt.Errorf("WebSocket connection closed: %w", err)
      }

      // Handle received messages
      if messageType == websocket.TextMessage {
        if err := handleWebSocketMessage(conn, message, sessionID, investorID); err != nil {
          log.Printf("Error handling WebSocket message: %v", err)
          continue
        }
      }
    }
  }  
}

func PricePusher(conn *websocket.Conn, routingCtx *routing.Context, err *error) {
  g, ctx := errgroup.WithContext(ctx)
+ go func() {
+   <-ctx.Done()
+   if conn != nil {
+     conn.Close()
+   }
+ }()

  // Handle gRPC stream
  g.Go(func() error {
    return handleGRPCStream(ctx, conn, req, sessionID, investorID)
  })

  // Monitor WebSocket connection
  g.Go(func() error {
    return monitorWebSocketConnection(ctx, conn, sessionID, investorID)
  })

  // Listen WebSocket messages
  g.Go(func() error {
    return listenWebSocketMessages(conn, sessionID, investorID)
  })
}

Third attempt

After K6 test finished, it will abruptly stop the test. It may not dispatchs TCP FIN packet to the server. So the server may not be able to close the connection gracefully. To address this issue, the server should actively check the connection status by sending a ping message periodically. If the client side didn't respond within 60 seconds with a pong message, the server will close the connection.

const (
	pingPeriod = 30 * time.Second
	pongWait   = 60 * time.Second
)

func monitorWebSocketConnection(ctx context.Context, conn *websocket.Conn, sessionID, investorID string) error {
  ...
+ ticker := time.NewTicker(pingPeriod)
  ...
}

func listenWebSocketMessages(conn *websocket.Conn, sessionID, investorID string) error {
  // Set up initial read deadline
+ conn.SetReadDeadline(time.Now().Add(pongWait))
+ conn.SetPongHandler(func(string) error {
+  // Extended read deadline to 60 seconds as long as the client side responds with a pong message.
+   conn.SetReadDeadline(time.Now().Add(pongWait))
+   return nil
+ })  

  for {
-   // Don't set read deadline in the loop.
-   conn.SetReadDeadline(time.Now().Add(15 * time.Minute))
  }
}

func handleWebSocketMessage(conn *websocket.Conn, message []byte, sessionID, investorID string) error {
  var wsMsg WSMessage
  switch wsMsg.Type {
    case "ping":
      // Add extended read deadline to 60 seconds as long as the client side sends with a ping message.
+     conn.SetReadDeadline(time.Now().Add(pongWait))
      ...
  }
}

Result​

Gateway Memory was not leaking anymore ✅​

In comparison with the previous goroutine count, it was reduced from 35000-40000 to 1000-2000.

Verified with K6 Result​

There was 46 errors out of 22833 WS connections and 46 errors out of 22835 Websocket connections. So the availability was 99.8%.

# 46 Error / 22833 WS connection
ERRO[0172] [VU 259] WS Error:  websocket: close 1006 (abnormal closure): unexpected EOF  source=console 
...

There were 46 errors out of 22835 HTTP connections. So the availability was 99.8%.

# 46 Error / 22835 HTTP connection
ERRO[0362] [VU 1496] HTTP Failed! Status: 0, Body: null  source=console
...
ERRO[0369] [VU 428] HTTP Failed! Status: 0, Body: null   source=console

What is Status: 0, Body: null?

It happened near the end of loading test. Status: 0, Body: null means no TCP connection, the request didn't even touch the server.

Only one error was caused by 404 error. It can be ignored during these kind of high load test.

# 1 / 22835 HTTP connection
ERRO[0275] [VU 743] HTTP Failed! Status: 404, Body: Route not found  source=console

Now the spike test is successful. Let's shift to capacity test in the next article. I will test 5000 users CCU within 5 mins and hold for another 5 mins. All of them will continuously receive message through websocket, of course.

Following up on my last post, here is how to fix the Redis Pub/Sub connection leak.

Conclusion​

The leaking problem has been fixed. The system is ready for the larger scale test.

If you are interested in the debugging process, please continue reading.

Smoke Test for Spike Test Round​

The parameters are the same as the last one.

Observations​

Redis PubSub Connection was leaking ❌​

Ideally, these two logs should be paired.

2026-04-02 13:49:29.640 | 2026/04/02 17:49:29 Created Redis PubSub connection for BKR
2026-04-02 13:49:29.640 | 2026/04/02 17:49:29 Closed Redis PubSub connection for BKR (no more subscribers)

Based on the count of orphaned logs, I can confirm the issue.

Statistics:
- Total 'Created' events: 1897
- Total 'Closed' events : 1804
- Net active (at log end): 93

Debugging​

Initially, I thought the reason may be multiple goroutine are trying to deactivateSession that cause race condition which block clean up process. So I spent a lots of time modify sessionManager lock scope. However, none of them are related. I skipped these attempts and demonstrated what was exactly happened.

Quick answer, there is actually a missing entry point of deactivateSession() that be triggered by close signal of websocket connection. And I accidiently put cleanup logic only in one of many exist returns and forget that the branch is not the only exist.

func (u *StreamStockPriceUseCase) StreamPriceUpdates(dto *dtos.StreamPriceUpdates, stream pb.RealTimePriceMonitoringService_StreamPriceUpdatesServer) (err error) {

...

for {
    select {
    case price, ok := <-session.PriceStream:
      if !ok { return nil } 
      batchMap[price.Ticker] = // ... logic ...

    case <-flushTicker.C:
      if len(batchMap) == 0 { continue }
      // ... convert to proto ...
      
      if err := stream.Send(update); err != nil {
        return err
      }
      clear(batchMap)

    case <-stream.Context().Done():
      // Should not be called here
->    u.sessionManager.deactivateSession(dto.Req.SessionId)
      return stream.Context().Err() 
    }
  }
}

Root Cause Analysis: The Missing Entry Point​

Session may not be deactivated if stream.Send() returns an error. The function without calling deactivateSesssion() will leak the Redis Pub/Sub connection.

Solution: Put deactivateSession() in defer​

func (u *StreamStockPriceUseCase) StreamPriceUpdates(dto *dtos.StreamPriceUpdates, stream pb.RealTimePriceMonitoringService_StreamPriceUpdatesServer) (err error) {
+  defer func() {
+   u.sessionManager.deactivateSession(dto.Req.SessionId)
+  }()
...

for {
    select {
    case price, ok := <-session.PriceStream:
      if !ok { return nil } 
      batchMap[price.Ticker] = // ... logic ...

    case <-flushTicker.C:
      if len(batchMap) == 0 { continue }
      // ... convert to proto ...
      
      if err := stream.Send(update); err != nil {
        return err
      }
      clear(batchMap)

    case <-stream.Context().Done():
-     u.sessionManager.deactivateSession(dto.Req.SessionId)
      return stream.Context().Err() 
    }
  }
}

Result​

Redis PubSub Connection was not leaking ✅​

Let's re-run the smoke test again.

The Pub/Sub connection number is back to baseline. The issue has been fixed!! Perfect!!

Next step, let's do the spike test with 10 times of traffic and see what will be happened.

To confirm system stability using a smoke test with a small number of VUs.

Conclusion​

Spotting a race condition in SessionManager. Fixed it. The system is ready for the larger scale test.

If you are interested in the debugging process, please continue reading.

Smoke Test for Spike Test Round​

I reduced the VUs to a small number to start

export const options = {
  stages: [
    { duration: '3m', target: 100 },  // Ramp-up: from 0
    { duration: '1m', target: 100 },  // Steady state: observe baseline
    { duration: '30s', target: 150 }, // Spike: Sudden influx of concurrent users
    { duration: '1m', target: 150 },  // Sustained peak: test Lock Contention
    { duration: '10s', target: 0 },   // Recovery: Rapid scale down
  ],
};

Observations​

K6 Test Results ❌​

The success rate of HTTP sessions created (via Simple POST) is only 97%.

Performance Degradation ❌​

The P99 latency for the registerSession endpoint spiked to nearly 400ms, indicating severe lock contention or that the system is struggling under load.

Close of closed channel ❌​

The Go server repeatedly threw a fatal error: panic: close of closed channel.

2026/03/31 19:22:55 close of closed channel
2026/03/31 19:22:55 Stack: File: /build/mods/bc11-market-monitor/useCases/SessionManager.go:70 Func: (*sessionManager).registerSession
File: /build/mods/bc11-market-monitor/useCases/SubscribeStockPrice.go:45 Func: (*SubscribeStockPriceUseCase).Executes
File: /build/mods/bc11-market-monitor/controllers/StockPrice.go:22 Func: (*Controller).StartPriceMonitoring
File: /build/shared/go/protos/bc11_market_monitor/bc11_market_monitor.service.pb.go:464 Func: lzstock/shared/go/protos/bc11_market_monitor._RealTimePriceMonitoringService_StartPriceMonitoring_Handler.func1
File: /build/shared/go/infra/grpcServer/interceptor.go:18 Func: (*Server).UnaryInterceptor
File: /build/shared/go/protos/bc11_market_monitor/bc11_market_monitor.service.pb.go:466 Func: lzstock/shared/go/protos/bc11_market_monitor._RealTimePriceMonitoringService_StartPriceMonitoring_Handler
File: /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1431 Func: (*Server).processUnaryRPC
File: /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1842 Func: (*Server).handleStream
File: /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1061 Func: (*Server).serveStreams.func2.1()
File: /go/pkg/mod/google.golang.org/grpc@v1.75.0/server.go:1072 Func: (*Server).serveStreams.func2 in goroutine 102
2026/03/31 19:22:55 start price monitoring: Error: register session: Error: close of closed channel

Debugging​

Thanks to the clear error logs, I can easily locate the issue. The message indicated the file path, line number, and function name.

...
2026/03/31 19:22:55 Stack: 
File: /build/mods/bc11-market-monitor/useCases/SessionManager.go:70 
Func: (*sessionManager).registerSession
...

One of the entries calling deactivateSession is the registerSession function.

func (sm *sessionManager) registerSession(session *session) (err error) {
  sm.mu.Lock()

  ...

  if len(userSessions) >= sm.maxSessionsPerUser {
    ...
    sm.mu.Unlock()
->  sm.deactivateSession(oldestSessionID)

    sm.mu.Lock()
    ...
  }

  ...
  sm.mu.Unlock()
}

Inside the deactivateSession function, it is calling close() on a channel that has already been closed will instantly cause a panic.

func (sm *sessionManager) deactivateSession(sessionID string) {
  sm.mu.RLock()
  session := sm.sessions[sessionID]
  sm.mu.RUnlock()

  if session == nil {
    return
  }

  // Lock for session modification
  session.mu.Lock()

  session.IsActive = false
  traderID := session.TraderID

  // Remove Session Channel from RedisManager
  for _, ticker := range session.Tickers {
    sm.redisSubManager.RemoveSubscription(ticker, session.PriceStream)
  }
->close(session.PriceStream)

  session.mu.Unlock()
  
  // Lock for sessionsByUser map modification
  sm.mu.Lock()

  delete(sm.sessions, sessionID)

  if userSessions, exists := sm.sessionsByUser[traderID]; exists {
    for i, id := range userSessions {
      if id == sessionID {
        sm.sessionsByUser[traderID] = append(userSessions[:i], userSessions[i+1:]...)
        break
      }
    }
    if len(sm.sessionsByUser[traderID]) == 0 {
      delete(sm.sessionsByUser, traderID)
    }
  }
  sm.mu.Unlock()
}

The root cause is there multiple goroutines as entries calling deactivateSession simultaneously. Let's break down the problem step by step.

Root Cause Analysis: The Race Condition​

In this case, the issue is most likely generated by multiple goroutines calling deactivateSession simultaneously, leading to a classic Race Condition.

The Callers:

• API Goroutine (registerSession): When a user reaches their maximum allowed concurrent sessions, the system actively kicks out their oldest session.
• Background Goroutine (cleanupExpiredSessions): A cron-like job that periodically scans for and removes timed-out sessions.
• Websocket Connection Close: When a user closes the websocket connection, the system will call deactivateSession to clean up the session.

Solution: Idempotent Deactivation​

Instead of trying to prevent these callers from executing simultaneously, I modified deactivateSession to be idempotent—meaning it can be safely called multiple times without causing errors.

func (sm *sessionManager) deactivateSession(sessionID string) {
  ...
  // Acquire Session-level lock
  session.mu.Lock()

+ if !session.IsActive {
+   session.mu.Unlock() 
+   return // Session was already deactivated by another goroutine
+ }
  
  // Mark as deactivated immediately
  session.IsActive = false
  traderID := session.TraderID
  ...
}

Result​

K6 Test Results ✅​

The success rate of HTTP sessions created is now 100%.

Performance Degradation ✅​

The P99 latency for the registerSession endpoint stayed low, indicating that all session creations finished without blocking.

No more close of closed channel ✅​

No more close of closed channel panic. Perfect!!

Next: Leaked Redis Pub/Sub Connections​

Everything looks solid. But new issues are beginning to surface. I found Redis Pub/Sub connections didn't close after all sessions dropped. Let's debug it.

Test Goal​

The primary goal of this load test is to prove that the real-time streaming engine remains stable and responsive under massive traffic. The test specifically evaluates the system's ability to seamlessly handle two core user flows simultaneously:

• Flow 1: Session Creation. Users successfully hitting the REST API to create a monitoring session for their chosen stock tickers.
• Flow 2: Real-Time Streaming. Users establishing a WebSocket connection using that generated session ID to continuously receive live price updates.

Metrics​

To truly prove the architecture's resilience, I looked at specific Grafana charts to track my custom application metrics and OS-level hardware metrics...

1. Active WebSocket Sessions​

bc11_active_ws_sessions

• Definition: A Counter metric tracking the exact number of active monitoring sessions currently registered in the core engine.
• Why do we want this metric: I need to know the exact capacity and load on the server at any given second.
• What this metric can tell us: During the load test, this graph should scale perfectly in sync with the k6 Virtual Users (VUs) connecting. Most importantly, it must drop back to exactly zero when the test finishes. If it stays above zero, it tells me I have "zombie" sessions and a memory leak.

2. Active Redis Pub/Subs​

bc11_active_redis_pubsubs

• Definition: A Counter tracking the number of active channels subscribed to the Redis server.
• Why do we want this metric: To monitor the connection overhead on the database layer. Every connection takes up resources, and Redis has strict connection limits.
• What this metric can tell us: This proves my 1-to-1 Ticker-to-Goroutine architecture is working. Even if 5,000 users subscribe to "TSLA", this gauge should only show 1. It tells me the system is successfully multiplexing and preventing database exhaustion.

3. Subscription Update Duration​

bc11_subscription_update_duration_seconds

• Definition: A Histogram measuring the time (in seconds) it takes to process a user creating a session or changing their stock subscriptions.
• Why do we want this metric: Because updating subscriptions requires a write lock (sync.RWMutex). I need to measure lock contention in the subscription flow.
• What this metric can tell us: If the P99 latency spikes here, it means thousands of goroutines are waiting in line to read or write state. Keeping this under sub-millisecond levels tells me the lock granularity is optimized and the code is truly thread-safe.

4. Internal Fan-out & Backpressure​

bc11_fanout_messages_total

• Definition: A Counter tracking messages broadcasted from Redis into the internal Go channels, labeled by success or dropped.
• Why do we want this metric: To observe the internal streaming engine's health and the effectiveness of the backpressure mechanism.
• What this metric can tell us: A spike in the dropped line during extreme load is a feature, not a bug. It tells me the non-blocking channel design successfully sheds load. The system is intentionally dropping stale ticks for slow clients instead of blocking the main broadcast loop and crashing the server.

"What happens if a client is on a terrible 3G network, their WebSocket buffer fills up, and my Go server cannot push data to them fast enough?"

If I use standard blocking channel operations, a single slow client could block my internal listenToRedis broadcasting goroutine. This would cause a cascading failure, delaying price updates for thousands of users with perfectly good internet connections.

To prevent this, I implemented Backpressure via Non-blocking Channels.

Instead of waiting, I use Go's select statement with a default case. If a user's specific channel buffer is full, I intentionally drop the message:

// Fan-out broadcast to all channels subscribed to this Ticker
for _, sessionChannel := range subscribers {
    select {
    case sessionChannel <- parsedPrice:
        // Success: Buffer has space
        metrics.FanoutMessagesTotal.WithLabelValues("success").Inc()
    default:
        // BACKPRESSURE: Channel is full. 
        // Drop the stale price rather than blocking the main broadcast loop.
        metrics.FanoutMessagesTotal.WithLabelValues("dropped").Inc()
    }
}

5. Messages Pushed to Frontend​

bc11_messages_pushed_total

• Definition: A Counter tracking the total number of stock price updates successfully pushed out to the end users over the WebSocket.
• Why do we want this metric: To measure the actual business throughput and verify data delivery.
• What this metric can tell us: Watching this number climb steadily confirms the streaming engine is actively doing its job. It proves the server isn't just holding idle connections, but is successfully routing massive amounts of data to the clients.

6. End-to-End Push Latency​

market_monitor_e2e_latency_ms

• Definition: A Histogram measuring the total time (in milliseconds) from when a price is generated to when it is successfully pushed to the WebSocket.
• Why do we want this metric: This is the ultimate user experience metric. Having fast internal processing doesn't matter if the final delivery to the trader is slow.
• What this metric can tell us: It proves the entire pipeline—from Redis injection to batching, and finally to the network flush—meets the strict real-time trading requirement. If the P95 stays under 50ms, the architecture is a success.

7. System Errors & Panics​

market_monitor_system_errors_total

• Definition: A Counter catching system-level errors and recovered panics, categorized by the component where the error occurred.
• Why do we want this metric: Because silent failures in background goroutines are a nightmare to debug. I need a tripwire for unexpected crashes.
• What this metric can tell us: If this increments during a load test, it exposes a flaw in the error handling or an unexpected edge case (like writing to a closed channel) that needs an immediate fix.

System Health Metrics (CPU, RAM, and Goroutines)​

• container_cpu_usage_seconds_total (CPU): * Definition: A metric from the Docker Engine showing how many CPU cores the container is actively burning.
  • Why/What it tells us: It verifies that the high concurrency is handled efficiently by Go's scheduler. If latency is high but CPU is low, I know I have a lock contention issue, not a hardware limit.
• go_memstats_heap_inuse_bytes vs go_memstats_sys_bytes (RAM):
  • Definition: Tracks the gap between what the Go runtime is actively using (Heap) versus what the OS has reserved for the process.
  • Why/What it tells us: It allows me to observe the Garbage Collection (GC) sawtooth pattern. It confirms that my GOMEMLIMIT configuration prevents the process from experiencing an Out-Of-Memory (OOM) kill under peak load.
• go_goroutines:
  • Definition: Tracks the total number of goroutines currently existing in the Go runtime.
  • Why/What it tells us: A flat plateau during the test means stable concurrency. The count dropping back to the baseline immediately after the test proves zero goroutine leaks.

Setting Up Load Test Scripts​

To validate my system, I needed to simulate a highly volatile market opening. I used k6 as my load-generation weapon of choice.

My goal was to saturate both the inbound WebSocket connections and the internal Redis Pub/Sub broadcasting channels simultaneously. I prepared two test scripts to test the system under extreme load and capacity. Both scripts are use the same mock price injection and observability although they have slightly different test parameters that will be explained in the following sections.

• Mock Price Injection: My BC10 service injects 859 prices, randomly generates new prices within a 1% variation rate, and then inserts them into Redis.
• Observability: Prometheus scraping Go runtime metrics, visualized via Grafana.

1. Spike Test Script:​

The Test Parameters:

• Target Load: Scaling up to a peak of 1,500 concurrent Virtual Users (VUs) establishing WebSocket connections.
• Traffic Behavior: Each Virtual User (VU) acts like a real, impatient trader. They run in a continuous loop: first, they hit the REST API to create a session for a random batch of 1 to 5 tickers. Next, they open a WebSocket connection, listen to live price updates for a random period (5 to 20 seconds), and then abruptly disconnect. After a quick 1 to 3-second break, they loop back and do it again (capped at 100 iterations per VU). This simulates massive "connection churn" to test if the server cleans up resources properly.
• Traffic Stages:
  • Ramp-up: A gradual 3-minute climb to a baseline of 1,000 VUs.
  • Baseline Plateau: A sustained 1-minute steady state at 1,000 VUs.
  • The Spike: A sudden 30-second surge injecting 500 more users, pushing the total to 1,500 VUs.
  • Peak Plateau: Maintaining the extreme 1,500 VU load for 1 minute to test for lock contention.
  • Recovery: A rapid 10-second scale-down to 0 VUs to observe how quickly the system releases memory.

// Tickers pool (from mockPriceInit.go)
const TICKER_POOL = [
    ...
];

// Helper function: Randomly select Tickers
function getRandomTickers() {
  const count = Math.floor(Math.random() * 5) + 1; // Randomly select 1-5
  const shuffled = [...TICKER_POOL].sort(() => 0.5 - Math.random());
  return shuffled.slice(0, count);
}

// Helper function: Generate random integer
function randomInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1) + min);
}

// Define stress test script: Extreme load test (Ramp-up -> Spike -> Recovery)
export const options = {
  stages: [
    { duration: '3m', target: 1000 },  // Ramp-up: 3 minutes to ramp up to 5000
    { duration: '1m', target: 1000 },  // Stable at 5000 to observe baseline performance
    { duration: '30s', target: 1500 }, // Spike: 30 seconds to spike to 15000 concurrent
    { duration: '1m', target: 1500 },  // Continue for 1 minute extreme load, test Lock Contention
    { duration: '10s', target: 0 },     // Recovery: 10 seconds to recover from load
  ],
};

// =========================================================
// Function 1: Responsible for calling HTTP API to get Session ID
// =========================================================
function createSession(tickers) {
    const apiUrl = 'http://bc15:8080/v1/sub_stock_price';
    const payload = JSON.stringify({ tickers: tickers });
    
    const httpParams = {
        headers: { 
            'Content-Type': 'application/json',
            'Authorization': TOKEN 
        },
    };

    const res = http.post(apiUrl, payload, httpParams);
    check(res, { 'HTTP Session created (200)': (r) => r.status === 200 });

    if (res.status !== 200) {
        console.error(`[VU ${exec.vu.idInTest}] HTTP Failed! Status: ${res.status}, Body: ${res.body}`);
        return null;
    }

    try {
        return res.json('session_id'); 
    } catch (e) {
        console.error(`[VU ${exec.vu.idInTest}] Failed to parse JSON. Error: ${e}`);
        return null;
    }
}

// =========================================================
// Function 2: Responsible for creating WebSocket and closing it at a random time
// =========================================================
function runWebSocketTest(sessionId) {
    const wsUrl = `ws://bc15:8080/ws?session_id=${sessionId}&token=${TOKEN}`;
    const waitTime = randomInt(5, 20); // Randomly stay 5-20 seconds
    
    const wsRes = ws.connect(wsUrl, {}, function (socket) {
        socket.on('open', () => { 
            // After a random number of seconds, actively disconnect
            socket.setTimeout(function () {
                socket.close();
            }, waitTime * 1000);
        });

        socket.on('message', (msg) => {
            // console.log(`[VU ${exec.vu.idInTest}] Received message`);
        });

        socket.on('error', (e) => {
            if (e.error() != 'websocket: close sent') {
                console.error(`[VU ${exec.vu.idInTest}] WS Error: `, e.error());
            }
        });
    });

    check(wsRes, { 'WS status is 101': (r) => r && r.status === 101 });
}

// =========================================================
// Main Entry Point
// =========================================================
export default function () {
    // Each VU will continue to execute until the test ends
    while (true) {
        const tickers = getRandomTickers();
        const sessionId = createSession(tickers);
        
        if (sessionId) {
            runWebSocketTest(sessionId);
        }
        
        // After each reconnection, take a short break
        sleep(randomInt(1, 3));
        
        // Check if the test time has been exceeded
        if (__ITER >= 100) break; // Prevent infinite loop, actually controlled by options.duration
    }
}

(Note: To prevent my authentication microservice from bottlenecking the load test, I implemented a strict BYPASS_AUTH environment variable coupled with a non-production safety lock. This allowed k6 to instantly establish thousands of connections without dealing with heavy JWT handshakes.)

2. Capacity Test Script:​

The Test Parameters:

• Target Load: Scaling up to a massive peak of 5,000 concurrent Virtual Users (VUs) establishing long-lived WebSocket connections.
• Traffic Behavior: Each Virtual User (VU) acts like a dedicated, long-term investor. Instead of aggressive churning, they connect once and stay online. First, they hit the REST API to create a session for a random batch of 1 to 5 tickers. Next, they open a WebSocket connection and continuously listen to live price updates for the entire duration of the test (up to 15 minutes). This simulates a sustained, high-volume data streaming scenario without connection drops, testing the absolute capacity of our broadcasting engine.
• Traffic Stages: * Ramp-up: A gradual 5-minute climb from 0 to 5,000 VUs.
  • Baseline Plateau: A sustained 5-minute steady state maintaining the peak 5,000 VU load. This provides a clear, noise-free window to observe system performance under maximum capacity.
  • Recovery: A rapid 30-second graceful scale-down to 0 VUs to observe how efficiently the system releases memory.

const TICKER_POOL = [
    ...
];

function getRandomTickers() {
    const count = Math.floor(Math.random() * 5) + 1;
    const shuffled = [...TICKER_POOL].sort(() => 0.5 - Math.random());
    return shuffled.slice(0, count);
}

// =========================================================
// 1. Adjust script: Gradually ramp up to 5,000 users and maintain stability
// =========================================================
export const options = {
  stages: [
    { duration: '5m', target: 5000 },  // Gradual ramp-up to 5,000 users
    { duration: '5m', target: 5000 },  // Maintain 5,000 users stable (quiet period)
    { duration: '30s', target: 0 },    // Test end, graceful shutdown within 30 seconds
  ],
};

function createSession(tickers) {
    const apiUrl = 'http://bc15:8080/v1/sub_stock_price';
    const payload = JSON.stringify({ tickers: tickers });
    const httpParams = { headers: { 'Content-Type': 'application/json', 'Authorization': TOKEN } };

    const res = http.post(apiUrl, payload, httpParams);
    check(res, { 'HTTP Session created (200)': (r) => r.status === 200 });
    if (res.status !== 200) {
        console.error(`[VU ${exec.vu.idInTest}] HTTP Failed! Status: ${res.status}`);
        return null;
    }
    try { return res.json('session_id'); } catch (e) { return null; }
}

function runWebSocketTest(sessionId) {
    const wsUrl = `ws://bc15:8080/ws?session_id=${sessionId}&token=${TOKEN}`;
    
    // =========================================================
    // 2. Remove random disconnections, keep connections until test end (e.g. 10 minutes)
    // =========================================================
    const waitTime = 900; // 600 seconds (10 minutes)

    const wsRes = ws.connect(wsUrl, {}, function (socket) {
        socket.on('open', () => { 
            // Here we no longer randomly disconnect, keep it online
            socket.setTimeout(function () {
                socket.close();
            }, waitTime * 1000);
        });

        socket.on('message', (msg) => {
            // Silent receive
        });

        socket.on('error', (e) => {
            if (e.error() != 'websocket: close sent') {
                console.error(`[VU ${exec.vu.idInTest}] WS Error: `, e.error());
            }
        });
    });

    check(wsRes, { 'WS status is 101': (r) => r && r.status === 101 });
}

// =========================================================
// 3. Entry Point：Remove while(true), each VU only executes once connection
// =========================================================
export default function () {
    const tickers = getRandomTickers();
    const sessionId = createSession(tickers);
    
    if (sessionId) {
        runWebSocketTest(sessionId);
    }
    
    // Here we don't need while(true), k6 will automatically manage the lifecycle of VU based on options.duration
}

const TICKER_POOL = [
    "A", "AA", "AAL", "AAP", "AAPL", "ABBV", "ABEV", "ABNB", "ABT", "ACGL", "ACI", "ACM", "ACN", "ADBE", "ADI", "ADM", "ADP", "ADSK", "AEE", "AEG", "AEM", "AEP", "AER", "AES", "AFG", "AFL", "AGCO", "AGL", "AIG", "AIZ", "AJG", "AKAM", "ALB", "ALC", "ALGN", "ALL", "ALLE", "ALLY", "ALNY", "AMAT", "AMC", "AMCR", "AMD", "AME", "AMGN", "AMH", "AMP", "AMT", "AMZN", "AN", "ANET", "ANSS", "AON", "AOS", "APD", "APH", "APO", "APP", "APTV", "AQN", "AR", "ARE", "ARES", "ARGX", "ARMK", "ARW", "ASML", "ASTI", "ASX", "ATEYY", "ATO", "AVB", "AVGO", "AVTR", "AVY", "AWK", "AXP", "AZN", "AZO", "BA", "BABA", "BAC", "BAH", "BAM", "BAP", "BAX", "BBD", "BBVA", "BBWI", "BBY", "BCE", "BCH", "BCS", "BDX", "BEN", "BEP", "BG", "BHP", "BHVN", "BIDU", "BIIB", "BILI", "BILL", "BIO", "BIP", "BJ", "BK", "BKNG", "BKR", "BLDR", "BLK", "BMO", "BMRN", "BMY", "BNS", "BNTX", "BP", "BR", "BRKR", "BRO", "BSAC", "BSBR", "BSX", "BSY", "BTI", "BUD", "BURL", "BWA", "BX", "BXP", "BZ", "C", "CABO", "CACC", "CAE", "CAG", "CAH", "CAR", "CARR", "CASY", "CAT", "CB", "CBOE", "CBRE", "CBSH", "CCEP", "CCI", "CCJ", "CCK", "CCL", "CDNS", "CDW", "CE", "CF", "CFG", "CFR", "CG", "CGNX", "CHD", "CHDN", "CHE", "CHKP", "CHRW", "CHT", "CHTR", "CHWY", "CI", "CIEN", "CINF", "CL", "CLF", "CLVT", "CLX", "CM", "CMA", "CMCSA", "CME", "CMG", "CMI", "CMS", "CNA", "CNC", "CNI", "CNP", "CNQ", "CNXC", "COF", "COIN", "COLD", "CONOCOPHILLIPS", "COO", "COST", "CP", "CPB", "CPNG", "CPRT", "CPT", "CQP", "CRH", "CRL", "CRM", "CRWD", "CSCO", "CSGP", "CSL", "CSX", "CTAS", "CTRA", "CTSH", "CTVA", "CUBE", "CUK", "CVE", "CVS", "CVX", "CZR", "D", "DAL", "DAR", "DASH", "DB", "DBX", "DD", "DDOG", "DE", "DELL", "DEO", "DG", "DGX", "DHI", "DHR", "DIS", "DLB", "DLO", "DLR", "DLTR", "DOCS", "DOCU", "DOV", "DOW", "DOX", "DPZ", "DRI", "DT", "DTE", "DUK", "DVA", "DVN", "DXC", "DXCM", "E", "EA", "EBAY", "EBR", "EC", "ECL", "ED", "EDU", "EFX", "EIX", "EL", "ELAN", "ELP", "ELS", "EMN", "EMR", "ENB", "ENPH", "ENTG", "EOG", "EPAM", "EPD", "EQH", "EQIX", "EQNR", "EQR", "EQT", "ERIC", "ERIE", "ES", "ESLT", "ESS", "ET", "ETN", "ETR", "ETSY", "EVRG", "EW", "EWBC", "EXAS", "EXC", "EXPD", "EXPE", "EXR", "F", "FANG", "FAST", "FCX", "FDS", "FDX", "FE", "FERG", "FFIV", "FHN", "FICO", "FIS", "FITB", "FMC", "FMS", "FMX", "FNF", "FNV", "FOXA", "FRFHF", "FRT", "FSLR", "FTNT", "FTS", "FTV", "FUTU", "G", "GD", "GDDY", "GDS", "GE", "GFI", "GFL", "GFS", "GGB", "GGG", "GIB", "GILD", "GIS", "GL", "GLD", "GLOB", "GLPI", "GLW", "GM", "GMAB", "GME", "GNRC", "GOOGL", "GPC", "GPN", "GRAB", "GRFS", "GRMN", "GS", "GSK", "GTLB", "GWLIF", "GWW", "H", "HAL", "HAS", "HBAN", "HCA", "HD", "HDB", "HEI", "HES", "HIG", "HII", "HLT", "HMC", "HOLX", "HON", "HPE", "HPQ", "HRL", "HSBC", "HSIC", "HST", "HSY", "HTHT", "HTZ", "HUBB", "HUBS", "HUM", "HWM", "IAU", "IBKR", "IBM", "IBN", "ICE", "ICL", "ICLR", "IDXX", "IEP", "IEX", "IFF", "IFNNY", "IHG", "ILMN", "IMO", "INCY", "INFY", "ING", "INTC", "INTU", "INVH", "IP", "IPG", "IQV", "IR", "IRM", "ISRG", "IT", "ITUB", "ITW", "IVZ", "IX", "J", "JAZZ", "JBHT", "JBL", "JCI", "JD", "JHX", "JKHY", "JLL", "JNJ", "JNPR", "JPM", "KB", "KBR", "KDP", "KELLANOVA", "KEP", "KEY", "KEYS", "KHC", "KIM", "KKR", "KLAC", "KMB", "KMI", "KMX", "KNX", "KO", "KOF", "KR", "L", "LAD", "LAMR", "LBRDA", "LBTYA", "LCID", "LDOS", "LEA", "LECO", "LEGN", "LEN", "LH", "LHX", "LI", "LII", "LIN", "LKQ", "LLY", "LMT", "LNC", "LNG", "LNT", "LOGI", "LOW", "LPLA", "LRCX", "LU", "LULU", "LUMN", "LUV", "LVS", "LW", "LYB", "LYG", "LYV", "MA", "MAA", "MANH", "MAR", "MAS", "MASI", "MAT", "MCD", "MCHP", "MCK", "MCO", "MDB", "MDLZ", "MDT", "MELI", "MET", "MFC", "MFG", "MGA", "MGM", "MHK", "MIDD", "MKC", "MKL", "MKTX", "MLM", "MMC", "MMM", "MNST", "MO", "MOH", "MORN", "MOS", "MPC", "MPLX", "MPW", "MPWR", "MRK", "MRNA", "MRVL", "MS", "MSCI", "MSFT", "MSI", "MT", "MTB", "MTCH", "MTD", "MTN", "MU", "MUFG", "NBIX", "NDAQ", "NDSN", "NEE", "NEM", "NET", "NFE", "NFLX", "NGG", "NI", "NICE", "NIO", "NKE", "NLY", "NMR", "NNN", "NOC", "NOK", "NOW", "NRG", "NSC", "NTAP", "NTES", "NTR", "NTRS", "NU", "NUE", "NVCR", "NVDA", "NVO", "NVR", "NVS", "NWG", "NWL", "NWSA", "NXPI", "NXST", "O", "OC", "ODFL", "OGE", "OGN", "OHI", "OKE", "OKTA", "OLN", "OLPX", "OMC", "ON", "ORCL", "ORI", "ORLY", "OTEX", "OTIS", "OVV", "OXY", "PAA", "PAC", "PAG", "PANW", "PATH", "PAYC", "PAYX", "PBA", "PBR", "PCAR", "PCG", "PCTY", "PDD", "PEG", "PEP", "PFE", "PFG", "PFGC", "PG", "PGR", "PH", "PHG", "PHM", "PINS", "PKG", "PKX", "PLD", "PLTR", "PLUG", "PM", "PNC", "PNR", "PNW", "PODD", "POOL", "PPC", "PPG", "PPL", "PRU", "PSA", "PSO", "PSTG", "PSX", "PTC", "PUK", "PWR", "PYPL", "QCOM", "QGEN", "QRVO", "QSR", "RACE", "RBA", "RBLX", "RCI", "RCL", "RDY", "REG", "REGN", "RELX", "REXR", "RF", "RGA", "RGEN", "RGLD", "RHI", "RIO", "RIVN", "RJF", "RKT", "RMD", "RNR", "ROK", "ROKU", "ROL", "ROP", "ROST", "RPM", "RPRX", "RRX", "RS", "RSG", "RTNTF", "RTX", "RY", "RYAAY", "SAN", "SAP", "SBAC", "SBSW", "SCCO", "SCHW", "SCI", "SE", "SEDG", "SEE", "SEIC", "SEMPRA", "SHG", "SHW", "SIRI", "SJM", "SKM", "SLB", "SLF", "SMFG", "SNA", "SNAP", "SNN", "SNOW", "SNPS", "SNX", "SNY", "SO", "SONY", "SPG", "SPGI", "SPOT", "SQM", "SSL", "SSNC", "STE", "STLA", "STLD", "STM", "STT", "STX", "STZ", "SU", "SUI", "SUZ", "SWK", "SWKS", "SYF", "SYK", "SYY", "T", "TAK", "TAP", "TCOM", "TD", "TDG", "TDY", "TEAM", "TECH", "TECK", "TEF", "TEL", "TER", "TEVA", "TFC", "TFII", "TFX", "TGT", "TJX", "TLK", "TM", "TME", "TMO", "TMUS", "TOST", "TPL", "TPR", "TRGP", "TRI", "TRMB", "TROW", "TRP", "TRU", "TRV", "TS", "TSCO", "TSLA", "TSM", "TSN", "TT", "TTC", "TTD", "TTE", "TTEK", "TTWO", "TU", "TW", "TWLO", "TX", "TXN", "TXT", "TYL", "U", "UAL", "UBER", "UBS", "UDR", "UELMO", "UGI", "UHAL", "UHS", "UI", "UL", "ULTA", "UMC", "UNH", "UNM", "UNP", "UPS", "URI", "USB", "USFD", "USO", "UTHR", "VALE", "VEEV", "VFC", "VICI", "VIPS", "VIV", "VLO", "VMC", "VOD", "VRSK", "VRSN", "VRTX", "VST", "VTR", "VTRS", "VZ", "WAB", "WAL", "WAT", "WBA", "WCN", "WDAY", "WDC", "WEC", "WELL", "WES", "WEX", "WFC", "WFG", "WHR", "WIT", "WLK", "WM", "WMB", "WMG", "WMT", "WOLF", "WPC", "WPM", "WPP", "WRB", "WSC", "WSM", "WSO", "WST", "WTRG", "WY", "XEL", "XOM", "XP", "XPEV", "XRAY", "XYL", "YMM", "YUM", "YUMC", "ZBH", "ZBRA", "ZG", "ZION", "ZM", "ZS", "ZTO", "ZTS"
];