咨询区
Grief Coder:
我的项目中有下面一段代码:
// let's say there is a list of 1000+ URLs
string[] urls = { "http://google.com", "http://yahoo.com", ... };// now let's send HTTP requests to each of these URLs in parallel
urls.AsParallel().ForAll(async (url) => {var client = new HttpClient();var html = await client.GetStringAsync(url);
});
这段代码有一个问题,当我开启了 1000+
的并发请求,是否有一种简便的方式限制这些 异步http请求
并发量,比如说实现同一时刻不会超过 20 个下载,请问我该如何去实现?
回答区
Jay Shah:
可以使用 SemaphoreSlim
,它可以非常完美的搞定,下面是我实现的扩展方法。
public static async Task ForEachAsyncConcurrent<T>(this IEnumerable<T> enumerable,Func<T, Task> action,int? maxActionsToRunInParallel = null){if (maxActionsToRunInParallel.HasValue){using (var semaphoreSlim = new SemaphoreSlim(maxActionsToRunInParallel.Value, maxActionsToRunInParallel.Value)){var tasksWithThrottler = new List<Task>();foreach (var item in enumerable){// Increment the number of currently running tasks and wait if they are more than limit.await semaphoreSlim.WaitAsync();tasksWithThrottler.Add(Task.Run(async () =>{await action(item).ContinueWith(res =>{// action is completed, so decrement the number of currently running taskssemaphoreSlim.Release();});}));}// Wait for all of the provided tasks to complete.await Task.WhenAll(tasksWithThrottler.ToArray());}}else{await Task.WhenAll(enumerable.Select(item => action(item)));}}
然后像下面这样使用。
await enumerable.ForEachAsyncConcurrent(async item =>{await SomeAsyncMethod(item);},5);
Serge Semenov:
其实直接用 semaphore
稍不注意就会遇到很多的坑,而且排查起来还特别棘手,我建议你使用 AsyncEnumerator NuGet Package
,参考地址:https://www.nuget.org/packages/AsyncEnumerator/1.1.0 ,这样也不需要再造什么轮子了,参考如下代码:
using System.Linq;
using System.Buffers;
using Dasync.Collections;// let's say there is a list of 1000+ URLs
string[] urls = { "http://google.com", "http://yahoo.com"};// now let's send HTTP requests to each of these URLs in parallel
await urls.ParallelForEachAsync(async (url) => {var client = new HttpClient();var html = await client.GetStringAsync(url);
},maxDegreeOfParallelism: 20);
点评区
在异步上做并发限制要比同步复杂的多,不过也是有一些可选方式,比如本篇的这两种,学习了。