I'd like to spawn some threads and in each thread sequentially make calls to an API and aggregate the results (some sort of stress testing). Here is my attempt:
private async Task DoWork()
{
var allResponses = new List<int>();
for (int i = 0; i < 10; i++)
{
await Task.Run(() =>
{
var responses = Enumerable.Range(0, 50).Select(i => CallApiAndGetStatusCode());
allResponses.AddRange(responses);
});
}
// do more work
}
private int CallApiAndGetStatusCode()
{
try
{
var request = new HttpRequestMessage(httpMethod.Get, "some url");
var responseResult = httpClient.SendAsync(request).Result;
return (int)responseResult.StatusCode;
}
catch (Exception e)
{
logger.LogError(e, "Calling API failed");
}
}
However, this code always ends up the catch with the inner exception being {"A task was canceled."}. What am I doing wrong here?
There is no benefit to using either Enumerable.Range or .AddRange in your example, since you do not need the seeded number. Your code must be converted to async/await to avoid deadlocks and in doing so, you can simply loop inside of each task and avoid any odd interactions between Enumerable.Select and await:
private async Task DoWork()
{
var allTasks = new List<Task>(10);
var allResponses = new List<int>();
for (int i = 0; i < 10; i++)
{
allTasks.Add(Task.Run(async () =>
{
var tempResults = new List<int>();
for (int i = 0; i < 50; i++)
{
var result = await CallApiAndGetStatusCode();
if (result > 0) tempResults.Add(result);
}
if (tempResults.Count > 0)
{
lock (allResponses)
{
allResponses.AddRange(tempResults);
}
}
}));
}
await Task.WhenAll(allTasks);
// do more work
}
private async Task<int> CallApiAndGetStatusCode()
{
try
{
var request = new HttpRequestMessage(HttpMethod.Get, "some url");
var responseResult = await httpClient.SendAsync(request);
return (int)responseResult.StatusCode;
}
catch (Exception e)
{
logger.LogError(e, "Calling API failed");
}
return -1;
}
Note that this code is overly protective, locking the overall batch before adding the temp results.
I changed your code to this and work
async Task DoWork()
{
var allResponses = new List<int>();
for (int i = 0; i < 10; i++)
{
await Task.Run(() =>
{
var responses = Enumerable.Range(0, 3).Select(i => CallApiAndGetStatusCodeAsync());
allResponses.AddRange(responses.Select(x => x.Result));
});
}
// do more work
}
async Task<int> CallApiAndGetStatusCodeAsync()
{
try
{
var request = new HttpRequestMessage(HttpMethod.Get, "http://www.google.com");
var responseResult = await httpClient.SendAsync(request);
return (int)responseResult.StatusCode;
}
catch (Exception e)
{
logger.LogError(e, "Calling API failed");
return -1;
}
}
Related
I'm a bit new to using Semaphores, and I'm a bit confused by this. All I'm doing is grabbing the html code from a page, and writing it to a file, over and over. When I use it in a foreach, it works perfectly. When I change it over to a for loop, I get a the file is used by another process exception, and it references the file that is after the max count. What is the difference between these two? They should be processing the same files.
Works perfectly:
private async Task GetFiles()
{
string strDir = ConfigurationManager.AppSettings["location"];
var maxThreads = 4;
var allTasks = new List<Task>();
SemaphoreSlim throttler = new SemaphoreSlim(initialCount: maxThreads);
foreach(FileToProcess ftpFile in lstFiles)
{
await throttler.WaitAsync();
allTasks.Add(
Task.Run(async () =>
{
try
{
await GetHtmlFromUrlAsync(ftpFile, strDir);
}
finally
{
throttler.Release();
}
}));
}
await Task.WhenAll(allTasks);
}
Gets an error after on the 5th item in the list (max thread + 1). If I adjusted the max thread count, the error would change as well:
private async Task GetFiles()
{
string strDir = ConfigurationManager.AppSettings["location"];
var maxThreads = 4;
var allTasks = new List<Task>();
SemaphoreSlim throttler = new SemaphoreSlim(initialCount: maxThreads);
for (int x = 0; x < lstFiles.Count; x++)
{
await throttler.WaitAsync();
allTasks.Add(
Task.Run(async () =>
{
try
{
await GetHtmlFromUrlAsync(lstFiles[x], strDir);
}
finally
{
throttler.Release();
}
}));
}
await Task.WhenAll(allTasks);
}
I have some code that looks like this:
async void MyMethod()
{
var client = new HttpClient();
var tasks = new List<Task<string>>();
for (int id = 1; id <= 10; id++)
{
tasks.Add(GetValue(client, id));
}
await Task.WhenAll(tasks);
}
async Task<string> GetValue(HttpClient client, int id)
{
var response = await client.GetAsync($"http://example.com?id={id}");
// Do something else here so that this task can't be condensed to a
// single line of code
System.Diagnostics.Debug.WriteLine("something");
return await response.Content.ReadAsStringAsync();
}
What is the syntax to add anonymous tasks to tasks without the use of the GetValue method? I want to do something like the code below, but I can't figure out the syntax to return a Task<string> from my anonymous method (this code gives me a compilation error Cannot convert async lambda expression to delegate type Func<string>):
for (int id = 1; id <= 10; id++)
{
tasks.Add(new Task<string>(async () =>
{
var response = await client.GetAsync($"http://example.com?id={id}");
// Do something else here so that this task can't be condensed to a
// single line of code
System.Diagnostics.Debug.WriteLine("something");
return await response.Content.ReadAsStringAsync();
}));
}
The way you execute a delegate is pretty simple, you just invoke it.
public static T Execute<T>(Func<T> func)
{
return func();
}
for (int id = 1; id <= 10; id++)
{
tasks.Add(Execute(async () =>
{
var response = await client.GetAsync($"http://example.com?id={id}");
return await response.Content.ReadAsStringAsync();
}));
}
Well, I'm trying to run a task 100 times on each run (with paralellism) but I can't manage this to work.
I'm trying to bruteforce an API, for this the API allows me to concatenate as many IDS as possible (without exceeding the timeout)
// consts:
// idsNum = 1000
// maxTasks = 100
// We prepare the ids that we will process (in that case 100,000)
var ids = PrepareIds(i * idsNum * maxTasks, idsNum, maxTasks);
// This was my old approach (didn't work)
//var result = await Task.WhenAll(ids.AsParallel().Select(async x => (await client.GetItems(x)).ToArray()));
// This is my new approach (also this didn't worked...)
var items = new List<item[]>();
ids.AsParallel().Select(x => client.GetItems(x).GetAwaiter().GetResult()).ForAll(item =>
{
//Console.WriteLine("processed!");
items.Add(item.ToArray());
});
var result = items.ToArray();
As you can see I put Console.WriteLine("processed!"); statment, in order to check if anything worked... But I can't manage this to work.
Those are my other methods:
private static IEnumerable<ulong[]> PrepareIds(int startingId, int idsNum = 1000, int maxTasks = 100)
{
for (int i = 0; i < maxTasks; i++)
yield return Range((ulong)startingId, (ulong)(startingId + idsNum)).ToArray();
}
And...
public static async Task<IEnumerable<item>> GetItems(this HttpClient client, ulong[] ids, Action notFoundCallback = null)
{
var keys = PrepareDataInKeys(type, ids); // This prepares the data to be sent to the API server
var content = new FormUrlEncodedContent(keys);
content.Headers.ContentType =
new MediaTypeHeaderValue("application/x-www-form-urlencoded") { CharSet = "UTF-8" };
client.DefaultRequestHeaders.ExpectContinue = false;
// We create a post request
var response = await client.PostAsync(new Uri(FILE_URL), content);
string contents = null;
JObject jObject;
try
{
contents = await response.Content.ReadAsStringAsync();
jObject = JsonConvert.DeserializeObject<JObject>(contents);
}
catch (Exception ex)
{
Console.WriteLine(ex);
Console.WriteLine(contents);
return null;
}
// Then we read the items from the parsed JObject
JArray items;
try
{
items = jObject
.GetValue("...")
.ToObject<JObject>()
.GetValue("...").ToObject<JArray>();
}
catch (Exception ex)
{
Console.WriteLine(ex);
return null;
}
int notFoundItems = 0;
int nonxxxItems = 0;
int xxxItems = 0;
var all = items.BuildEnumerable(notFoundCallback, item =>
{
if (item.Result != 1)
++notFoundItems;
else if (item.id != 5)
++nonxxxItems;
else
++xxxItems;
});
CrawledItems += ids.Length;
NotFoundItems += notFoundItems;
NonXXXItems += nonxxxItems;
XXXItems += xxxItems;
return all;
}
private static IEnumerable<item> BuildEnumerable(this JArray items, Action notFoundCallback, Action<item> callback = null)
{
foreach (var item in items)
{
item _item;
try
{
_item = new item(item.ToObject<JObject>());
callback?.Invoke(_item);
}
catch (Exception ex)
{
if (notFoundCallback == null)
Console.WriteLine(ex, Color.Red);
else
notFoundCallback();
continue;
}
yield return _item;
}
}
So as you can see I create 100 parallel post requests using an HttpClient. But I can't manage it to work.
So the thing that I want to achieve is to retrieve as many items as possible because I need to crawl +2,000,000,000 items.
But any breakpoint is triggered, neither any caption is updated on Console (I'm using Konsole project in order to print values at a fixed position on console), so any advice can be given there?
I am calling WCF service in ASP.NET Core and everything is working fine, but whenever end of using gets executed, I get an error:
This OperationContextScope is being disposed out of order
I believe I am using wrong pattern to call WCF service using async/await but I am not sure what I am doing wrong.
Below is the code I am using to call a service.
[HttpPost]
public async Task<IActionResult> Runcase(IFormCollection formCollection)
{
if (ModelState.IsValid)
{
var runnumber = formCollection["Run number"];
await CallServiceasync();
return RedirectToAction("", "");
}
else
{
return View(formCollection);
}
}
public async Task CallServiceasync()
{
var product = p1.Value;
var a = product.first;
foreach (int Age in a.age)
{
foreach (int Gender in a.sex)
{
foreach (int Healthclass in a.uclass)
{
RequestData requestData = new RequestData()
{
ProductID = 534,
STATE = "CO",
AGE1 = Age,
SEX1 = Gender,
UND_CLASS1 = Healthclass,
};
RecieveResponseasync(requestData);
}
}
}
}
public async Task RecieveResponseasync(InputValues inputValues)
{
string reqedata = "";
string apikey = "001010iZno7001010L";
QuoteEngineService.MarketingSoftwareClient Service = new QuoteEngineService.MarketingSoftwareClient();
await Service.OpenAsync();
try
{
using (OperationContextScope scope = new OperationContextScope(Service.InnerChannel))
{
HttpRequestMessageProperty httpRequestMessage = new HttpRequestMessageProperty();
httpRequestMessage.Headers.Add("apikey", apikey);
OperationContext.Current.OutgoingMessageProperties[HttpRequestMessageProperty.Name] = httpRequestMessage;
reqedata = inputValues.XmlSerializetoString();
var result = await Service.ProcessRequestAsync(reqedata, "4fa2-ae27");
var outputvalues = new OutputvaluesViewModel();
outputvalues = result.DeserializeToObject();
List<OutputValue> outputs = new List<OutputValue>();
if (outputvalues.InitialPremium != null)
outputs.Add(new OutputValue { Name = "InitialPremium", Result = outputvalues.InitialPremium});
if (outputvalues.TargetPremium != null)
outputs.Add(new OutputValue { Name = "TargetPremium", Result = outputvalues.TargetPremium });
foreach (var output in outputs)
{
await _context.outputValues.AddAsync(output);
await _context.SaveChangesAsync();
}
await Task.Delay(500);
}
}// **At this point I am getting error**
catch (Exception ex)
{
throw;
}
finally
{
if (Service.State == System.ServiceModel.CommunicationState.Opened)
{
await Service.CloseAsync();
}
}
}
From the docs:
Warning
Do not use the asynchronous "await" pattern within a OperationContextScope block. When the continuation occurs, it may run on a different thread and OperationContextScope is thread specific. If you need to call "await" for an async call, use it outside of the OperationContextScope block.
I Created several Task in the way below. But it seems WaitAll is not working. It is sending response without wait. Anything goes wrong here?
private void GetItemsPrice(IEnumerable<Item> items, int customerNumber)
{
try
{
var tasks = new List<Task>();
for (var i = 0; i < items.Count(); i += 50)
{
var newTask = DoGetItemsPrice(items.Skip(i).Take(50), customerNumber);
tasks.Add(newTask);
}
Task.WaitAll(tasks.ToArray());
}
catch (Exception ex)
{
ErrorLog.WriteLog(GetType().Name, "GetItemsPrice", string.Format("customerNumber={0}", customerNumber), ex.Message);
}
}
private static Task DoGetItemsPrice(IEnumerable<Item> items, int customerNumber)
{
return Task.Factory.StartNew(() =>
{
var sxApiObj = new SxApiService();
var request = new OEPricingMultipleRequest();
request.customerNumber = customerNumber;
request.arrayProduct =
items.Select(
itemCode =>
new OEPricingMultipleinputProduct
{
productCode = itemCode.ItmNum,
quantity = itemCode.Quantity,
warehouse = ConfigurationVariables.DefaultWareHouse
}).ToArray();
var response = sxApiObj.OEPricingMultiple(ConfigurationVariables.SfAppServer,
ConfigurationVariables.SfUserId,
ConfigurationVariables.SfPassword,
request);
if (response.arrayPrice != null)
{
foreach (var priceData in response.arrayPrice)
{
var productCode = priceData.productCode;
var item = items.FirstOrDefault(itm => itm.ItmNum == productCode);
if (item == null) continue;
item.ItmListPrice1 = priceData.price.ToString("c", ConfigurationVariables.UsCulture);
item.ItmListPrice2 = priceData.discountAmount.ToString("c", ConfigurationVariables.UsCulture);
item.ItmListPrice3 = priceData.extendedAmount.ToString("c", ConfigurationVariables.UsCulture);
item.Quantity = priceData.netAvailable;
}
}
});
}
There is nothing wrong with my question. WaitAll works fine and the code also correct.