Develop Reference

AngleSharp error while making the async calls

I am trying to load many pages using the AngleSharp. The idea is that it loads a page, and if this page has a link to the next, loads the next page and so forth, the methods are described like bellow. But I am getting the inner exception:
Specified argument was out of the range of valid values.
Parameter name: index"
I believe is something related with Thread and syncrhronization.
public static bool ContainsNextPage(IDocument document)
{
String href = document.QuerySelectorAll(".prevnext a")[0].GetAttribute("href");
if (href == String.Empty)
return false;
else
return true;
}
public static string GetNextPageUrl(IDocument document)
{
return document.QuerySelectorAll(".prevnext a")[0].GetAttribute("href");
}
public static async Task<IDocument> ParseUrlSynch(string Url)
{
var config = new Configuration().WithDefaultLoader();
IDocument document = await BrowsingContext.New(config).OpenAsync(Url);
return document;
}
public static async Task<ConcurrentBag<IDocument>> GetAllPagesDOMs(IDocument initialDocument)
{
ConcurrentBag< IDocument> AllPagesDOM = new ConcurrentBag< IDocument>();
IDocument nextPageDOM;
IDocument currentDocument = initialDocument;
if (initialDocument != null)
{
AllPagesDOM.Add(initialDocument);
}
while (ContainsNextPage(currentDocument))
{
String nextPageUrl = GetNextPageUrl(currentDocument);
nextPageDOM = ParseUrlSynch(nextPageUrl).Result;
if (nextPageDOM != null)
AllPagesDOM.Add(nextPageDOM);
currentDocument = nextPageDOM;
}
return AllPagesDOM;
}
static void Main(string[] args)
{
List<IDocument> allPageDOMs = new List<IDocument>();
IDocument initialDocument = ParseUrlSynch(InitialUrl).Result;
List<String> urls = new List<string>();
List<Subject> subjects = new List<Subject>();
IHtmlCollection<IElement> subjectAnchors = initialDocument.QuerySelectorAll(".course_title a");
String[] TitleAndCode;
String Title;
String Code;
String Description;
IDocument currentDocument = initialDocument;
ConcurrentBag<IDocument> documents =
GetAllPagesDOMs(initialDocument).Result; //Exception in here
...
}

Error message is caused by this code:
document.QuerySelectorAll(".prevnext a")[0]
One of your documents doesn't have any anchors inside prevnext. Maybe it's first page, maybe the last, either way you need to check the array for it's length.
Also blocking call on async method is a bad practice and should be avoided. You'll get the deadlock in any UI app. The only reason you don't get it now is that you're in console app.

Your instincts are correct, if you are using this from an application with a non-default SynchronizationContext such as WPF, Win Forms, or ASP.NET then you will have a deadlock because you are synchronously blocking on an async Task returning function (this is bad and should be avoided). When the first await is reaching inside of the blocking call, it will try to post the continuation to the current SyncronizationContext, which will be already locked by the blocking call (if you use .ConfigureAwait(false) you avoid this, but that is a hack in this case).
A quick fix would be to use async all the way through by changing:
nextPageDOM = ParseUrlSynch(nextPageUrl).Result;
with:
nextPageDOM = await ParseUrlSynch(nextPageUrl);
After you get stung by this a few times, you'll learn to have alarm bells go off in your head every time you block an asynchronous method.

Correct pattern to call a service containing an async call from an MVC controller

I am new to TAP and async/await in practice in C# so I may have some bad code smells here, so be gentle. :-)
I have a service method that looks as follows:
public OzCpAddOrUpdateEmailAddressToListOutput AddOrUpdateEmailAddressToList(
OzCpAddOrUpdateEmailAddressToListInput aParams)
{
var result = new OzCpAddOrUpdateEmailAddressToListOutput();
try
{
var mailChimManager = new MailChimpManager(aParams.MailChimpApiKey);
Task<Member> mailChimpResult =
mailChimManager.Members.AddOrUpdateAsync(
aParams.Listid,
new Member
{
EmailAddress = aParams.EmailAddress
});
//Poll async task until it completes.
//Give it at most 8 seconds to do what it needs to do
var outOfTime = DateTime.Now.AddSeconds(8);
while (!mailChimpResult.IsCompleted)
{
if (DateTime.Now > outOfTime)
{
throw new Exception("Timed out waiting for MailChimp API.");
}
}
//Should there have been a problem with the call then we raise an exception
if (mailChimpResult.IsFaulted)
{
throw new Exception(
mailChimpResult.Exception?.Message ??
"Unknown mail chimp library error.",
mailChimpResult.Exception);
}
else
{
//Call to api returned without failing but unless we have
//the email address subscribed we have an issue
if (mailChimpResult.Result.Status != Status.Subscribed)
{
throw new Exception(
$"There was a problem subscribing the email address
{aParams.EmailAddress} to the mailchimp list id
{aParams.Listid}");
}
}
}
catch (Exception ex)
{
result.ResultErrors.AddFatalError(PlatformErrors.UNKNOWN, ex.Message);
}
return result;
}
But when I call in from MVC Controller action mailChimpResult.IsCompleted always returns false and eventually I hit the timeout.
I realise this is because I am not chaining the async calls as per HttpClient IsComplete always return false and because of different threads this behaviour is "expected".
However I want my service method to hide the complexity of the async nature of what it is doing and merely do what appears to be a synchronous call in my action method namely:
var mailChimpResult =
_PlatformMailChimpService.AddOrUpdateEmailAddressToList(
new OzCpAddOrUpdateEmailAddressToListInput
{
EmailAddress = aFormCollection["aEmailAddress"],
Listid = ApplicationSettings.Newsletter.MailChimpListId.Value,
MailChimpApiKey = ApplicationSettings.Newsletter.MailChimpApiKey.Value
});
if (mailChimpResult.Result == true)
{
//So something
}

Ideally you should avoid the .Result and .IsFaulted properties of the Task and Task<T> objects, that was code smell number one. When you're using these objects you should use async and await through the entire stack. Consider your service written this way instead:
public async Task<OzCpAddOrUpdateEmailAddressToListOutput>
AddOrUpdateEmailAddressToList(
OzCpAddOrUpdateEmailAddressToListInput aParams)
{
var result = new OzCpAddOrUpdateEmailAddressToListOutput();
try
{
var mailChimManager = new MailChimpManager(aParams.MailChimpApiKey);
Member mailChimpResult =
await mailChimManager.Members.AddOrUpdateAsync(
aParams.Listid,
new Member
{
EmailAddress = aParams.EmailAddress
});
}
catch (Exception ex)
{
result.ResultErrors.AddFatalError(PlatformErrors.UNKNOWN, ex.Message);
}
return result;
}
Notice that I was able to remove all of that unnecessary polling and examining of properties. We mark the method as Task<OzCpAddOrUpdateEmailAddressToListOutput> returning and decorate it with the async keyword. This allows us to use the await keyword in the method body. We await the .AddOrUpdateAsync which yields the Member.
The consuming call into the service looks similar, following the same paradigm of async and await keywords with Task or Task<T> return types:
var mailChimpResult =
await _PlatformMailChimpService.AddOrUpdateEmailAddressToList(
new OzCpAddOrUpdateEmailAddressToListInput
{
EmailAddress = aFormCollection["aEmailAddress"],
Listid = ApplicationSettings.Newsletter.MailChimpListId.Value,
MailChimpApiKey = ApplicationSettings.Newsletter.MailChimpApiKey.Value
});
if (mailChimpResult.Result == true)
{
//So something
}
It is considered best practice to postfix the "Async" word to the method, to signify that it is asynchronous, i.e.; AddOrUpdateEmailAddressToListAsync.

limit # of web service requests simultaneously

I have an Excel Add-In written in C#, .NET 4.5. It will send many web service requests to a web server to get data. E.g. it sends 30,000 requests to web service server. When data of a request comes back, the addin will plot the data in Excel.
Originally I did all the requests asynchronously, but sometime I will get OutOfMemoryException
So I changed, sent the requests one by one, but it is too slow, takes long time to finish all requests.
I wonder if there is a way that I can do 100 requests at a time asynchronously, once the data of all the 100 requests come back and plot in Excel, then send the next 100 requests.
Thanks
Edit
On my addin, there is a ribbon button "Refresh", when it is clicked, refresh process starts.
On main UI thread, ribbon/button is clicked, it will call web service BuildMetaData,
once it is returned back, in its callback MetaDataCompleteCallback, another web service call is sent
Once it is returned back, in its callback DataRequestJobFinished, it will call plot to plot data on Excel. see below
RefreshBtn_Click()
{
if (cells == null) return;
Range firstOccurence = null;
firstOccurence = cells.Find(functionPattern, null,
null, null,
XlSearchOrder.xlByRows,
XlSearchDirection.xlNext,
null, null, null);
DataRequest request = null;
_reportObj = null;
Range currentOccurence = null;
while (!Helper.RefreshCancelled)
{
if(firstOccurence == null ||IsRangeEqual(firstOccurence, currentOccurence)) break;
found = true;
currentOccurence = cells.FindNext(currentOccurence ?? firstOccurence);
try
{
var excelFormulaCell = new ExcelFormulaCell(currentOccurence);
if (excelFormulaCell.HasValidFormulaCell)
{
request = new DataRequest(_unityContainer, XLApp, excelFormulaCell);
request.IsRefreshClicked = true;
request.Workbook = Workbook;
request.Worksheets = Worksheets;
_reportObj = new ReportBuilder(_unityContainer, XLApp, request, index, false);
_reportObj.ParseParameters();
_reportObj.GenerateReport();
//this is necessary b/c error message is wrapped in valid object DataResponse
//if (!string.IsNullOrEmpty(_reportObj.ErrorMessage)) //Clear previous error message
{
ErrorMessage = _reportObj.ErrorMessage;
Errors.Add(ErrorMessage);
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
}
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
Errors.Add(ErrorMessage);
_reportObj.ErrorMessage = ErrorMessage;
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
Helper.LogError(ex);
}
}
}
on Class to GenerateReport
public void GenerateReport()
{
Request.ParseFunction();
Request.MetacompleteCallBack = MetaDataCompleteCallback;
Request.BuildMetaData();
}
public void MetaDataCompleteCallback(int id)
{
try
{
if (Request.IsRequestCancelled)
{
Request.FormulaCell.Dispose();
return;
}
ErrorMessage = Request.ErrorMessage;
if (string.IsNullOrEmpty(Request.ErrorMessage))
{
_queryJob = new DataQueryJob(UnityContainer, Request.BuildQueryString(), DataRequestJobFinished, Request);
}
else
{
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
finally
{
Request.MetacompleteCallBack = null;
}
}
public void DataRequestJobFinished(DataRequestResponse response)
{
Dispatcher.Invoke(new Action<DataRequestResponse>(DataRequestJobFinishedUI), response);
}
public void DataRequestJobFinished(DataRequestResponse response)
{
try
{
if (Request.IsRequestCancelled)
{
return;
}
if (response.status != Status.COMPLETE)
{
ErrorMessage = ManipulateStatusMsg(response);
}
else // COMPLETE
{
var tmpReq = Request as DataRequest;
if (tmpReq == null) return;
new VerticalTemplate(tmpReq, response).Plot();
}
}
catch (Exception e)
{
ErrorMessage = e.Message;
Helper.LogError(e);
}
finally
{
//if (token != null)
// this.UnityContainer.Resolve<IEventAggregator>().GetEvent<DataQueryJobComplete>().Unsubscribe(token);
ModifyCommentOnFormulaCellPublishRefreshEvent();
Request.FormulaCell.Dispose();
}
}
on plot class
public void Plot()
{
...
attributeRange.Value2 = headerArray;
DataRange.Value2 = ....
DataRange.NumberFormat = ...
}

OutOfMemoryException is not about the too many requests sent simultaneously. It is about freeing your resources right way. In my practice there are two main problems when you are getting such exception:
Wrong working with immutable structures or System.String class
Not disposing your disposable resources, especially graphic objects and WCF requests.
In case of reporting, for my opinion, you got a second one type of a problem. DataRequest and DataRequestResponse are good point to start the investigation for the such objects.
If this doesn't help, try to use the Tasks library with async/await pattern, you can find good examples here:
// Signature specifies Task<TResult>
async Task<int> TaskOfTResult_MethodAsync()
{
int hours;
// . . .
// Return statement specifies an integer result.
return hours;
}
// Calls to TaskOfTResult_MethodAsync
Task<int> returnedTaskTResult = TaskOfTResult_MethodAsync();
int intResult = await returnedTaskTResult;
// or, in a single statement
int intResult = await TaskOfTResult_MethodAsync();
// Signature specifies Task
async Task Task_MethodAsync()
{
// . . .
// The method has no return statement.
}
// Calls to Task_MethodAsync
Task returnedTask = Task_MethodAsync();
await returnedTask;
// or, in a single statement
await Task_MethodAsync();
In your code I see a while loop, in which you can store your Task[] of size of 100, for which you can use the WaitAll method, and the problem should be solved. Sorry, but your code is huge enough, and I can't provide you a more straight example.

I'm having a lot of trouble parsing your code to figure out is being iterated for your request but the basic template for batching asynchronously is going to be something like this:
static const int batchSize = 100;
public async Task<IEnumerable<Results>> GetDataInBatches(IEnumerable<RequestParameters> parameters) {
if(!parameters.Any())
return Enumerable.Empty<Result>();
var batchResults = await Task.WhenAll(parameters.Take(batchSize).Select(doQuery));
return batchResults.Concat(await GetDataInBatches(parameters.Skip(batchSize));
}
where doQuery is something with the signature
Task<Results> async doQuery(RequestParameters parameters) {
//.. however you do the query
}
I wouldn't use this for a million requests since its recursive, but your case should would generate a callstack only 300 deep so you'll be fine.
Note that this also assumes that your data request stuff is done asynchronously and returns a Task. Most libraries have been updated to do this (look for methods with the Async suffix). If it doesn't expose that api you might want to create a separate question for how to specifically get your library to play nice with the TPL.

async value not returned after lock released while using sqlite-net in winrt app

I recently encountered this very strange problem.
Initially I have this block of code
public async Task<string> Fetch(string module, string input)
{
if (module != this._moduleName)
{
return null;
}
try
{
var db = new SQLiteAsyncConnection(_dbPath);
ResponsePage storedResponse = new ResponsePage();
Action<SQLiteConnection> trans = connect =>
{
storedResponse = connect.Get<ResponsePage>(input);
};
await db.RunInTransactionAsync(trans);
string storedResponseString = storedResponse.Response;
return storedResponseString;
}
catch (Exception e)
{
return null;
}
}
However control will never be handed back to my code after the transaction finishes running. I traced the program and it seems that after the lock is release, the flow of program stops. Then I switched to using the GetAsync method from SQLiteAsyncConnection class. Basically it did the same thing so I was still blocked at await. Then I removed the async calls and used the synchronous api like below:
public async Task<string> Fetch(string module, string input)
{
if (module != this._moduleName)
{
return null;
}
try
{
var db = new SQLiteConnection(_dbPath);
ResponsePage storedResponse = new ResponsePage();
lock (_dbLock)
{
storedResponse = db.Get<ResponsePage>(input);
}
string storedResponseString = storedResponse.Response;
return storedResponseString;
}
catch (Exception e)
{
return null;
}
}
Only then can the logic flows back to my code. However I can't figure out why is this so.
Another problem is that for this kind of simple query is there any gain in terms of query time if I use aysnc api instead of sync api? If not I'll stick to the sync version then.

You are most likely calling Result (or Wait) further up the call stack from Fetch. This will cause a deadlock, as I explain on my blog and in a recent MSDN article.
For your second question, there is some overhead from async, so for extremely fast asynchronous operations, the synchronous version will be faster. There is no way to tell whether this is the case in your code unless you do profiling.

Access is Denied when Saving a Object in in local storage Windows 8 Metro App

I'm attempting to store a List (where Show is my class that implements IXmlSerializable) to the local isolated storage. I'm using the code from this page:
http://metrostoragehelper.codeplex.com/
I've implemented the change suggested in the Issues section.
I am using the following code to add a Show object when it is clicked from an item list.
private async void addShowButton_Click_1(object sender, RoutedEventArgs e)
{
var isoStorage = new StorageHelper<List<Show>>(StorageType.Local);
List<Show> currentShows = await isoStorage.LoadASync("myShowsEx");
if(currentShows == null) {
currentShows = new List<Show>();
}
currentShows.Add(currentShow);
isoStorage.SaveASync(currentShows, "myShowsEx");
//Read it back, for debugging to check if it has been added properly.
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx"); //Exception here
}
The first show is added perfectly fine and it shows up in the currentShowsRB List. When a second item is clicked and the method above invoked an exception occurs on the last LoadAsync call: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
How can I get around this to access the local data store for multiple calls?
Below is also the relevant code from the StorageHelper:
public async void SaveASync(T Obj, string FileName)
{
FileName = FileName + ".xml";
try
{
if (Obj != null)
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.CreateFileAsync(FileName, CreationCollisionOption.ReplaceExisting);
using (var writeStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
Stream outStream = Task.Run(() => writeStream.AsStreamForWrite()).Result;
serializer.Serialize(outStream, Obj);
//writeStream.Dispose(); //Added and we get UnauthorizedAccessException
// outStream.Dispose(); //Added ObjectDisposedException caught in catch statement below
}
}
}
catch (Exception)
{
throw;
}
}
public async Task<T> LoadASync(string FileName)
{
FileName = FileName + ".xml";
try
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.GetFileAsync(FileName);
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
Stream inStream = Task.Run(() => readStream.AsStreamForRead()).Result;
inStream.Position = 0;
return (T)serializer.Deserialize(inStream);
}
}
catch (FileNotFoundException)
{
//file not existing is perfectly valid so simply return the default
return default(T);
//throw;
}
catch (Exception)
{
//Unable to load contents of file
throw;
}
}
The writeStream.Dispose() line I added in, but even when this is included I get the same error message of Access is Denied. If I also include the outStream.Dispose() line then I get a ObjectDisposedException being caught in the catch statement right below. Is there something else I should be doing?

You are not waiting for the SaveAsync to finish, trying to Load when the Save is still in progress. Change it to:
//isoStorage.SaveASync(currentShows, "myShowsEx");
await isoStorage.SaveASync(currentShows, "myShowsEx");
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx");
Edit, awaiting on void is a standard problem.
The quick fix is :
await TaskEx.Run(() => isoStorage.SaveASync(currentShows, "myShowsEx"));
But you can also move the TaskEx.Run() inside SaveASync(). And given a name that ends with Async, it should not be void but:
Task SaveASyncT Obj, string FileName)
{
return TaskEx.Run() => { .... }
}
I don't believe there is an async version of Serialize, so it stays at TaskEx.Run() .

This block can be shortened to:
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
return (T)serializer.Deserialize(readStream);
}
as you can directly read/write from the stream in the XmlSerializer class.
You may also find that CreationCollisionOption.ReplaceExisting can cause issues and OpenIfExists will cause the file to be overwritten by the serializer as required.