I've never attempted to use WaitAll() or WhenAll() when running async functionality. After looking at many documentations, SO posts, and tutorials, I haven't found enough information for this, so here I am.
I'm trying to figure out the best/proper way(s) to do the following:
Using EF6, get data as List<Entity>.
Iterate through each Entity and call an external API to perform some action.
External API returns data per Entity which I need to store on the same Entity.
Currently I have built (not tested) the following (without the error handling code):
public IEnumerable<Entity> Process() {
bool hasChanged = false;
var data = _db.Entity.Where(x => !x.IsRegistered);
foreach (var entity in data) {
var result = await CallExternalApi(entity.Id, entity.Name);
entity.RegistrationId = result.RegistrationId;
entity.IsRegistered = true;
_db.Entry(entity).State = EntityState.Modified;
hasChanges = true;
}
if (hasChanges) {
uow.Commit();
}
return data;
}
I feel like I may be able to take advantage of some other functionality/feature in async, but if I can I'm not sure how to implement it here.
Any guidance is really appreciated.
Update
The API I'm calling is the Zoom Api to add Registrants. While they do have an route to batch add Registrants, it does not return the RegistrantId and the Join Url I need.
First, figure out if your external API might have a way to get all the items you want in a batch. If it does, use that instead of sending a whole bunch of requests.
If you need to send a separate request for each item, but want to do it concurrently, you could do this:
public async Task<IReadOnlyCollection<Entity>> Process() {
var data = _db.Entity.Where(x => !x.IsRegistered).ToList();
if(!data.Any()) { return data; }
var entityResultTasks = data
.Select(async entity => new { entity, result = await CallExternalApi(entity.Id, entity.Name) })
.ToList();
var entityResults = await Task.WhenAll(entityResultTasks);
foreach (var entityResult in entityResults) {
var entity = entityResult.entity;
var result = entityResult.result;
entity.RegistrationId = result.RegistrationId;
entity.IsRegistered = true;
_db.Entry(entity).State = EntityState.Modified;
}
uow.Commit();
return data;
}
You will want to watch out for possible concurrency limits on the target source. Consider using Chunk to break your work into batches of acceptable sizes, or leveraging a semaphore or something to throttle the number of calls you're making.
Related
I am trying to filter out all errors except one from a list of errors I send back to the front end. I realize this operation should be an async operation as my request is giving a 500 internal server error. I am new to C# and am having a hard time figuring out how to do so.
My code that gets invoked on the route request looks like:
public async Task<ActionResult> Index(ProfileParams profileParameters)
{
// ...... //
var user = await GenerateUser(Request.RequestContext);
var userState = await _userStateFactory.CreateAsync(user);
var stateTree = new BusinessProfileStateTreeModel
{
Global = await _globalStateFactory.CreateAsync(user),
Header = await _headerStateFactory.CreateAsync(user, null),
User = userState,
Modals = _modalsStateFactory.Create(),
Page = CreatePageState(),
BusinessProfile = _businessProfileReviewsStateFactory.Create(viewModel, customerReviewModel),
Analytics = await _analyticsStateFactory.CreateAsync(user, CreateDtmData(viewModel?.Categories?.PrimaryCategoryName, profileBbbInfo?.BbbName, viewModel), userState)
};
// trying to filter out errors here from the state tree alerts
var errors = filterErrorsAsync(stateTree.BusinessProfile.Display.Alerts.AllAlerts);
var metaData =
GenerateProfileMetaData(customerReviewModel.NumFound, viewModel.ProfileUrl.ToUrlString(), viewModel);
var serverSideModel =
GenerateServerSideModel(
viewModel,
metaData,
profileBbbInfo,
stateTree.Analytics.DtmData,
user);
return await ReduxViewAsync(stateTree.ToList(), serverSideModel);
}
}
The filterErrorsAsync method looks like:
private List<BPAlert> filterErrorsAsync(List<BPAlert> allAlerts)
{
foreach (BPAlert alert in allAlerts)
{
if (alert.AlertTypeId == (int)BusinessReportCustomTextType.CustomerReviews)
{
allAlerts.Clear();
allAlerts.Add(alert);
}
}
return allAlerts;
}
Can someone tell me how to achieve this correctly?
You can't loop a list and modify it at the same time. This is probably what is causing your 500 error.
It looks like you only want filter out certain errors from a list. If you want to keep your method as a loop you can do:
private List<BPAlert> filterErrorsAsync(List<BPAlert> allAlerts)
{
List<BPAlert> temp = new List<BPAlert>(); //copy into new list
foreach (BPAlert alert in allAlerts)
{
if (alert.AlertTypeId == (int)BusinessReportCustomTextType.CustomerReviews)
{
temp.Add(alert);
}
}
return temp;
}
If you want to be a little more modern you can also just use LINQ
private List<BPAlert> filterErrorsAsync(List<BPAlert> allAlerts)
{
return allAlerts.Where(alert => alert.AlertTypeId == (int)BusinessReportCustomTextType.CustomerReviews).ToList();
}
You're attempting to modify a list while enumerating it which won't work. Since you already know which kind of error you want to filter to, you can utilize LINQ's Where method to filter out the other errors, then use Take to get the first one.
private List<BPAlert> filterErrors(List<BPAlert> allAlerts)
=> allAlerts.Where(alert => alert.AlertTypeID == (int)BusinessReportCustomTextType.CustomerReviews)
.Take(1)
.ToList();
There isn't anything asynchronous happening in this method, so no need to mark it async.
I have a an aync method that is looped:
private Task<HttpResponseMessage> GetResponseMessage(Region region, DateTime startDate, DateTime endDate)
{
var longLatString = $"q={region.LongLat.Lat},{region.LongLat.Long}";
var startDateString = $"{startDateQueryParam}={ConvertDateTimeToApixuQueryString(startDate)}";
var endDateString = $"{endDateQueryParam}={ConvertDateTimeToApixuQueryString(endDate)}";
var url = $"http://api?key={Config.Key}&{longLatString}&{startDateString}&{endDateString}";
return Client.GetAsync(url);
}
I then take the response and save it to my ef core database, however in some instances I get this Exception message: The Operaiton was canceled
I really dont understand that. This is a TCP handshake issue?
Edit:
For context I am making many of these calls, passing response to the method that writes to db (which is also so slow Its unbelievable):
private async Task<int> WriteResult(Response apiResponse, Region region)
{
// since context is not thread safe we ensure we have a new one for each insert
// since a .net core app can insert data at the same time from different users different instances of context
// must be thread safe
using (var context = new DalContext(ContextOptions))
{
var batch = new List<HistoricalWeather>();
foreach (var forecast in apiResponse.Forecast.Forecastday)
{
// avoid inserting duplicates
var existingRecord = context.HistoricalWeather
.FirstOrDefault(x => x.RegionId == region.Id &&
IsOnSameDate(x.Date.UtcDateTime, forecast.Date));
if (existingRecord != null)
{
continue;
}
var newHistoricalWeather = new HistoricalWeather
{
RegionId = region.Id,
CelsiusMin = forecast.Day.Mintemp_c,
CelsiusMax = forecast.Day.Maxtemp_c,
CelsiusAverage = forecast.Day.Avgtemp_c,
MaxWindMph = forecast.Day.Maxwind_mph,
PrecipitationMillimeters = forecast.Day.Totalprecip_mm,
AverageHumidity = forecast.Day.Avghumidity,
AverageVisibilityMph = forecast.Day.Avgvis_miles,
UvIndex = forecast.Day.Uv,
Date = new DateTimeOffset(forecast.Date),
Condition = forecast.Day.Condition.Text
};
batch.Add(newHistoricalWeather);
}
context.HistoricalWeather.AddRange(batch);
var inserts = await context.SaveChangesAsync();
return inserts;
}
Edit: I am making 150,000 calls. I know this is questionable since It all goes in memory I guess before even doing a save but this is where I got to in trying to make this run faster... only I guess my actual writing code is blocking :/
var dbInserts = await Task.WhenAll(
getTasks // the list of all api get requests
.Select(async x => {
// parsed can be null if get failed
var parsed = await ParseApixuResponse(x.Item1); // readcontentasync and just return the deserialized json
return new Tuple<ApiResult, Region>(parsed, x.Item2);
})
.Select(async x => {
var finishedGet = await x;
if(finishedGet.Item1 == null)
{
return 0;
}
return await writeResult(finishedGet.Item1, finishedGet.Item2);
})
);
.net core has a DefaultConnectionLimit setting as answered in comments.
this limits outgoing connections to specific domains to ensure all ports are not taken etc.
i did my parallel work incorrectly causing it to go over the limit - which everything i read says should not be 2 on .net core but it was - and that caused connections to close before receiving responses.
I made it greater, did parallel work correctly, lowered it again.
I am facing an issue with two different endpoints in my single asp.net app. Basically, the issue is that one of the endpoints does not allow asynchronous methods in the page and the other endpoint does. If I run the app one endpoint will ask me to have an asynchronous asp.net page, but the other one crashes and vice versa.
public async Task<AirtableListRecordsResponse> RetrieveRecord()
{
string MyProductID = ProductID;
string baseId = "00000000000xxxx";
string appKey = "00000000000xxxx";
var records = new List<AirtableRecord>();
using (AirtableBase airtableBase = new AirtableBase(appKey, baseId))
{
Task<AirtableListRecordsResponse> task = airtableBase.ListRecords(tableName: "efls", filterByFormula: ProductID);
AirtableListRecordsResponse response = await task;
if (!response.Success)
{
string errorMessage = null;
if (response.AirtableApiError is AirtableApiException)
{
errorMessage = response.AirtableApiError.ErrorMessage;
}
else
{
errorMessage = "Unknown error";
}
// Report errorMessage
}
else
{
records.AddRange(response.Records.ToList());
var record = response.Records;
//offset = response.Offset;
//var record = response.Record;
foreach (var item in record)
{
foreach (var Fields in item.Fields)
{
if (Fields.Key == "pdfUrl")
{
string link = Fields.Value.ToString();
MyLink = Fields.Value.ToString();
}
}
}
// Do something with your retrieved record.
// Such as getting the attachmentList of the record if you
// know the Attachment field name
//var attachmentList = response.Record.GetAttachmentField(YOUR_ATTACHMENT_FIELD_NAME);
}
return response;
}
}
This is the asynchronous method which asks for an asynchronous page, the other contains a strong structure and it cannot be changed for any reason. Is there any way to make them work together?
I am using airtable.com api by the way.
Thanks in advance.
I solved by my own,
The solution I found is the following:
When a page works with two different endpoints and one of them obligates the page to be asynchronous the best solution is to split the procedures into two different sections and/or pages, one of them will call the asynchronous methods and retrieves the info and other works without being asynchronous.
How can I pass the information between the sites?
Using session variables, there are endpoints which only needs to display simple data as in this case, so the session variables will be called in the page #2 which is the non-asynchronous page.
It is a simple solution but effective.
Thank you very much to all for you answers.
Using Wait on Task, you can use synchronous method
Task<AirtableListRecordsResponse> task = Task.Run(() => airtableBase.ListRecords(tableName: "efls", filterByFormula: ProductID));
task.Wait();
AirtableListRecordsResponse response = task.Result;
Use it only when you cannot use async method.
This method is completely deadlock free as mentioned on msdn blog-
https://blogs.msdn.microsoft.com/jpsanders/2017/08/28/asp-net-do-not-use-task-result-in-main-context/
Within an ASP.NET MVC Application I'm recieving the following error message for one of my controller methods that uses my Entity Framework context.
A second operation started on this context before a previous asynchronous operation completed. Use 'await' to ensure that any asynchronous operations have completed before calling another method on this context. Any instance members are not guaranteed to be thread safe.
I'm aware that you cannot run queries in parallel, and everything appears to be awaited properly. If I debug the program and step and inspect some of the data returned from EF then it works, probably because this forces the queries to complete.
EDIT If I place a breakpoint at the null check in the controller method and inspect the data of shipmentDetail the exception is NOT thrown.
Here's a snippit of the code:
Controller Method:
[Route("{id:int}/Deliveries")]
public async Task<ActionResult> DeliveryInfo(int id)
{
var shipmentDetail = await db.ShipmentDetails.SingleOrDefaultAsync(s => s.Id == id);
if (shipmentDetail == null)
return HttpNotFound(string.Format("No shipment detail found with id {0}", id));
var model = await DeliveryInfoModel.CreateModel(db, shipmentDetail);
return View("DeliveryInfo", model);
}
CreateModel Method:
public static async Task<DeliveryInfoModel> CreateModel(Context db, ShipmentDetail shipment)
{
DeliveryInfoModel model = new DeliveryInfoModel()
{
ShipmentInfo = shipment
};
//initialize processing dictionary
Dictionary<int, bool> boxesProcessed = new Dictionary<int, bool>();
List<DeliveryBoxStatus> statuses = new List<DeliveryBoxStatus>();
for (int i = 1; i <= shipment.BoxCount; i++ )
{
boxesProcessed.Add(i, false);
}
//work backwards through process
//check for dispositions from this shipment
if(shipment.Dispositions.Count > 0)
{
foreach (var d in shipment.Dispositions)
{
DeliveryBoxStatus status = new DeliveryBoxStatus()
{
BoxNumber = d.BoxNumber,
LastUpdated = d.Date,
Status = d.Type.GetDescription().ToUpper()
};
statuses.Add(status);
boxesProcessed[d.BoxNumber] = true;
}
}
//return if all boxes have been accounted for
if (boxesProcessed.Count(kv => kv.Value) == shipment.BoxCount)
{
model.BoxStatuses = statuses;
return model;
}
//check for deliveries
if(shipment.Job_Detail.Count > 0)
{
foreach (var j in shipment.Job_Detail.SelectMany(d => d.DeliveryInfos))
{
DeliveryBoxStatus status = new DeliveryBoxStatus()
{
BoxNumber = j.BoxNumber,
LastUpdated = j.Job_Detail.To_Client.GetValueOrDefault(),
Status = "DELIVERED"
};
statuses.Add(status);
boxesProcessed[j.BoxNumber] = true;
}
}
//check for items still in processing & where
foreach (int boxNum in boxesProcessed.Where(kv => !kv.Value).Select(kv => kv.Key))
{
//THIS LINE THROWS THE EXCEPTION
var processInfo = await db.Processes.Where(p => p.Jobs__.Equals(shipment.Job.Job__, StringComparison.InvariantCultureIgnoreCase) && p.Shipment == shipment.ShipmentNum && p.Box == boxNum)
.OrderByDescending(p => p.date)
.FirstOrDefaultAsync();
//process returned data
//...
}
model.BoxStatuses = statuses;
return model;
}
I'm not completely sure if it's because of the query made in the controller, or because of the queries made in the loop that aren't completing causing this behavior. Is there something I'm not understanding about when the queries are actually made/returned due to EF's laziness, or how async/await works in this situation? I have a lot of other methods & controllers that make async EF calls and haven't run into this previously.
EDIT
My context is injected into my controller using Ninject as my IoC container. Here's its config inside of NinjectWebCommon's RegisterServices method:
kernel.Bind<Context>().ToSelf().InRequestScope();
Avoid lazy loading when using async with Entity Framework. Instead, either load the data you need first, or use Include()'s to ensure the data you need is loaded with the query.
https://msdn.microsoft.com/en-gb/magazine/dn802603.aspx
Current State of Async Support
... Async
support was added to Entity Framework (in the EntityFramework NuGet
package) in version 6. You do have to be careful to avoid lazy
loading when working asynchronously, though, because lazy loading is
always performed synchronously. ...
(Emphasis mine)
Also:
https://entityframework.codeplex.com/wikipage?title=Task-based%20Asynchronous%20Pattern%20support%20in%20EF.#ThreadSafety
I am not sure what to use in this scenario.
I have an asp.net web api method that basically does this
Finds points of interests from foursquare near user.
Uses the foursquare locations to do queries in my database to find unique data about point of interest near user.
However since I need to store some of the foursquare information to link to my unique data to that location I decided to store all the information in my database and have my database act as my caching system.
This means anything new point of interest that comes in I have to insert into my database, check if it exists and if so then skip it or if it exists check the last refresh date(foursquare policy states all data must be refreshed after 30 day) and if it out past the refresh date I have to update the data.
I want to slow the user down and have to wait for the above to happen. I want my code to do step 1 and then do what I just mentioned while at the same time doing step 2.
Once step 2 finishes I want to return the data and let the user get on their way. If my caching system is not finished then it should keep going but not bog down the user.
I won't use any of these new results in step 2 as if I am inserting it then I won't have any data on that location at this time.
Not sure if I need to make a thread or use the async/await to achieve this.
Edit
Here is what I am trying to do
public HttpResponseMessage Get()
{
// this will do a foursquare lookup to find all stores near the user
// I want to insert them into my database and link it to my unquie data.
// stores pulled from foursquare will
// a) Be new and not in my database
// b) exist in my database but have been refreshed lately
// c) have not been refreshed in timeframe of foursquare policy
// THIS SHOULD WORK IN THE BACKGROUND
storeService.PointsOfInterestNearUser(80, -130); //As you can see it is
//void. Not sure where to put the async/await stuff
// find this product. Should be happening at the same time as above line.
var product = productService.FindProduct("Noodles");
//This will get returned to the user.
// the new stores taht are being added in StoreNearUser
//won't effect this search as I will have not data on this new store
// if existing store is being refreshed it is possible old
//address might be picked up...
//I can live with that as I doubt the address will change much.
// this should happen after product
var allStores = storeService.FindStoresThatHaveItem(product);
// this should be returned as soon as above line is finished.
//If StoreNearUser is not done, it should keep going but not hold up user.
return allStores;
}
public void StoresNearUser(double latitude, double longitude)
{
// get all categories I can about in foursquare.
//First time from db otherwise cached.
List<StoreCategory> storeCategories = GetStoreCategories();
// do a request and get everything in near the user
//(provided it is also in a category I care about)
var request = CreateFoursquareStoreRequest
(latitude, longitude, storeCategories);
// do the actual call.
var response = client.Execute<VenueSearch>(request);
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
// start going through the results, add or update or skip of entry will happen
AddUpdateStores(storeCategories, response);
}
else
{
ErrorSignal.FromCurrentContext().Raise(response.ErrorException);
}
}
Edit 2
public async Task StoresNearUser(double latitude, double longitude)
{
// get all categories I can about in foursquare. First time from db otherwise cached.
List<StoreCategory> storeCategories = GetStoreCategories();
// do a request and get everything in near the user(provided it is also in a category I care about)
var request = CreateFoursquareStoreRequest(latitude, longitude, storeCategories);
await client.ExecuteAsync<VenueSearch>
( request
, response =>
{
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
AddUpdateStores(storeCategories, response);
}
else
{
ErrorSignal.FromCurrentContext()
.Raise(response.ErrorException);
}
}
);
}
gives me this error
Cannot await 'RestSharp.RestRequestAsyncHandle'
I also don't get the difference between Task and void. From what I read if you just use Task it means you are sending nothing back of meaning, then why not just use void?
Edit 2
I found this post to show me how to make the wrapper for Restsharp. It is not 100% what I want but that is a separate issue.
public async Task StoresNearUser(double latitude, double longitude)
{
List<StoreCategory> storeCategories = GetStoreCategories();
var request = CreateFoursquareStoreRequest
(latitude, longitude, maxRadius, returnLimit, storeCategories);
var response = await client.GetResponseAsync(request);
if (response.StatusCode == HttpStatusCode.OK)
{
// had to use json.net right now as the wrapper does not expose restsharps deserilizer
var venue = JsonConvert
.DeserializeObject<VenueSearch>(response.Content);
AddUpdateStores(storeCategories, venue);
}
else
{
ErrorSignal.FromCurrentContext()
.Raise(response.ErrorException);
}
}
public async Task<HttpResponseMessage>Get()
{
await storeService.PointsOfInterestNearUser(80, -130);
var product = productService.FindProduct("Noodles");
var allStores = storeService.FindStoresThatHaveItem(product);
return allStores;
}
When I watch from the debugger it looks like it is still all going in order. I think product and allStores need to be since I need the product before I can find the stores but PointsOfInterestNearUser should be going at the same time as FindProduct.
Edit 3
Here is my FindProduct Method. Not sure what to make async to me it looks like everything needs to wait.
public ResponseResult<Product> FindProduct(string barcode)
{
ResponseResult<Product> responseResult = new ResponseResult<Product>();
Product product = null;
try
{
var findBarCode = context.Barcodes.Where(x => x.Code == barcode).Select(x => x.Product).FirstOrDefault();
responseResult.Response = product;
if (product == null)
{
responseResult.Status.Code = HttpStatusCode.NotFound;
}
else
{
responseResult.Status.Code = HttpStatusCode.OK;
}
}
catch (SqlException ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
responseResult.Status.Code = HttpStatusCode.InternalServerError;
responseResult.Status.Message = GenericErrors.InternalError;
}
return responseResult;
}
Edit 4
Still not sure how to do the Task.WhenAll()
public async Task<HttpResponseMessage>Get()
{
Task[] tasks = new Task[2];
tasks[0] = storeService.PointsOfInterestNearUser(80, -130);
tasks[1] = productService.FindProduct("Noodles");
await Task.WhenAll(tasks);
// not sure how to get product back out. I looked in the debugger and saw a "Result" that has it but when I do tasks[1].Result inetllisene cannot find .Result
var allStores = storeService.FindStoresThatHaveItem(product);
return allStores;
}
I would recommend using async/await for this. Updating a cache is one of the rare situations where returning early from an ASP.NET request is acceptable. You can see my blog post on the subject for some helpful code.
So, something like this (simplified to just look up one "interesting place" per location):
public async Task<PlaceWithData> FindPlaceAsync(Location myLocation)
{
Place place = await GetPlaceFromFoursquareAsync(myLocation);
PlaceWithData ret = await GetExtraDataFromDatabaseAsync(place);
if (ret.NeedsRefresh)
BackgroundTaskManager.Run(() => UpdateDatabaseAsync(place, ret));
return ret;
}
You may also want to consider extending the ASP.NET caching system rather than doing a "roll your own" cache.