最近一用户在使用BeetleX.HttpClient组件并发访问延时比较高的https服务时引起了卡死现像。由于组件更多使用场景是内部服务和非https,一直没有这情况出现;但用户提供测试场景下这情况必现,所以翻查了一些相关代码。
protected virtual void OnSslAuthenticate(SslStream sslStream){Task task;if (SslProtocols == null)SslProtocols = System.Security.Authentication.SslProtocols.Tls | System.Security.Authentication.SslProtocols.Tls11 |System.Security.Authentication.SslProtocols.Tls12;task = sslStream.AuthenticateAsClientAsync(SslServiceName, CertificateCollection.Count > 0 ? CertificateCollection : null, SslProtocols.Value, false);task.Wait();}
BeetleX的tcpclient中使了无限超时来等待ssl验证环节,正常局域网环境这个等待是不会卡死的,毕竟网内环境良好要么成功要因异常触发取消等。但在网络差的环境下就存在问题了,socket的异步receive无法触发异常引起这个Wait把线程永久挂起了,当一段时间太多这种情况出现那非常的事情就是大量线程被这个Wait抽干导致服务不能正常工作。
为了解决这些比较特殊的情况加上超时时间就好了
protected virtual void OnSslAuthenticate(SslStream sslStream){Task task;if (SslProtocols == null)SslProtocols = System.Security.Authentication.SslProtocols.Tls | System.Security.Authentication.SslProtocols.Tls11 |System.Security.Authentication.SslProtocols.Tls12;task = sslStream.AuthenticateAsClientAsync(SslServiceName, CertificateCollection.Count > 0 ? CertificateCollection : null, SslProtocols.Value, false);if (!task.Wait(5000)){throw new BeetleX.BXException($"connect {mIPAddress}:{mPort} SSL Authenticate timeout!");}}
把超时设置成5秒,在创建连接5秒后ssl还没有握手成功就直接超时关闭连接重新创建。
基础的问题解决了,但httpclientpool层面还有一个问题需要处理。当连接池一开始处于满负载请求,同时创建大量的Client会引起大量线程等待导致服务不太稳定(毕竟在不很多稳定的网络环境下ssl握手时间有些长)。为了解决问题使用一个指定的线程数的队列来创建ssl握手处理机制,这样就可以避免同时创建大量连接线程引起短暂卡壳现象。
public Task<HttpClientHandler> Pop(){HttpClientHandler result;TaskCompletionSource<HttpClientHandler> completionSource;lock (this){if (mPools.Count > 0){result = mPools.Pop();result.Using = true;result.TimeOut = BeetleX.TimeWatch.GetElapsedMilliseconds() + TimeOut;return Task.FromResult(result);}if (Clients.Count > MaxConnections){if (mWaitQueue.Count < MaxWaitLength){completionSource = new TaskCompletionSource<HttpClientHandler>();mWaitQueue.Enqueue(completionSource);return completionSource.Task;}else{throw new HttpClientException($"Request {Host} connections limit");}}}completionSource = new TaskCompletionSource<HttpClientHandler>();mCreateDispatchCenter.Next().Enqueue(new CreateClientTask { ClientHandlerPool = this, CompletionSource = completionSource });return completionSource.Task;}struct CreateClientTask{public TaskCompletionSource<HttpClientHandler> CompletionSource;public HttpClientHandlerPool ClientHandlerPool;}private static BeetleX.Dispatchs.DispatchCenter<CreateClientTask> mCreateDispatchCenter= new Dispatchs.DispatchCenter<CreateClientTask>(OnProcessCreateClient, 20);private static void OnProcessCreateClient(CreateClientTask e){try{var result = e.ClientHandlerPool.Create();result.Using = true;result.TimeOut = BeetleX.TimeWatch.GetElapsedMilliseconds() + e.ClientHandlerPool.TimeOut;e.CompletionSource.TrySetResult(result);}catch (Exception e_){e.CompletionSource.TrySetException(e_);}}
简单修改一下连接池Pop方法的代码就可以了。
BeetleX
开源跨平台通讯框架(支持TLS)
提供高性能服务和大数据处理解决方案
https://beetlex.io