Develop Reference

Async Web Service call not consistently asynchronous

I'm having issues creating an asynchronous web service using the Task Parallel Library with ASP.NET Web API 2. I make an asynchronous call to a method StartAsyncTest and create a cancellation token to abort the method. I store the token globally and then retrieve it and call it from a second method CancelAsyncTest. Here is the code:
// Private Global Dictionary to hold text search tokens
private static Dictionary<string, CancellationTokenSource> TextSearchLookup
= new Dictionary<string, CancellationTokenSource>();
/// <summary>
/// Performs an asynchronous test using a Cancellation Token
/// </summary>
[Route("StartAsyncTest")]
[HttpGet]
public async Task<WsResult<long>> StartAsyncTest(string sSearchId)
{
Log.Debug("Method: StartAsyncTest; ID: " + sSearchId + "; Message: Entering...");
WsResult<long> rWsResult = new WsResult<long>
{
Records = -1
};
try
{
var rCancellationTokenSource = new CancellationTokenSource();
{
var rCancellationToken = rCancellationTokenSource.Token;
// Set token right away in TextSearchLookup
TextSearchLookup.Add("SyncTest-" + sSearchId, rCancellationTokenSource);
HttpContext.Current.Session["SyncTest-" + sSearchId] =
rCancellationTokenSource;
try
{
// Start a New Task which has the ability to be cancelled
var rHttpContext = (HttpContext)HttpContext.Current;
await Task.Factory.StartNew(() =>
{
HttpContext.Current = rHttpContext;
int? nCurrentId = Task.CurrentId;
StartSyncTest(sSearchId, rCancellationToken);
}, TaskCreationOptions.LongRunning);
}
catch (OperationCanceledException e)
{
Log.Debug("Method: StartAsyncText; ID: " + sSearchId
+ "; Message: Cancelled!");
}
}
}
catch (Exception ex)
{
rWsResult.Result = "ERROR";
if (string.IsNullOrEmpty(ex.Message) == false)
{
rWsResult.Message = ex.Message;
}
}
// Remove token from Dictionary
TextSearchLookup.Remove(sSearchId);
HttpContext.Current.Session[sSearchId] = null;
return rWsResult;
}
private void StartSyncTest(string sSearchId, CancellationToken rCancellationToken)
{
// Spin for 1100 seconds
for (var i = 0; i < 1100; i++)
{
if (rCancellationToken.IsCancellationRequested)
{
rCancellationToken.ThrowIfCancellationRequested();
}
Log.Debug("Method: StartSyncTest; ID: " + sSearchId
+ "; Message: Wait Pass #" + i + ";");
Thread.Sleep(1000);
}
TextSearchLookup.Remove("SyncTest-" + sSearchId);
HttpContext.Current.Session.Remove("SyncTest-" + sSearchId);
}
[Route("CancelAsyncTest")]
[HttpGet]
public WsResult<bool> CancelAsyncTest(string sSearchId)
{
Log.Debug("Method: CancelAsyncTest; ID: " + sSearchId
+ "; Message: Cancelling...");
WsResult<bool> rWsResult = new WsResult<bool>
{
Records = false
};
CancellationTokenSource rCancellationTokenSource =
(CancellationTokenSource)HttpContext.Current.Session["SyncTest-" + sSearchId];
// Session doesn't always persist values. Use TextSearchLookup as backup
if (rCancellationTokenSource == null)
{
rCancellationTokenSource = TextSearchLookup["SyncTest-" + sSearchId];
}
if (rCancellationTokenSource != null)
{
rCancellationTokenSource.Cancel();
TextSearchLookup.Remove("SyncTest-" + sSearchId);
HttpContext.Current.Session.Remove("SyncTest-" + sSearchId);
rWsResult.Result = "OK";
rWsResult.Message = "Cancel delivered successfully!";
}
else
{
rWsResult.Result = "ERROR";
rWsResult.Message = "Reference unavailable to cancel task"
+ " (if it is still running)";
}
return rWsResult;
}
After I deploy this to IIS, the first time I call StartAsyncTest and then CancelAsyncTest (via the REST endpoints), both requests go through and it cancels as expected. However, the second time, the CancelAsyncTest request just hangs and the method is only called after StartAsyncTest completes (after 1100 seconds). I don't know why this occurs. StartAsyncTest seems to highjack all threads after it's called once. I appreciate any help anyone can provide!

I store the token globally and then retrieve it and call it from a second method CancelAsyncTest.
This is probably not a great idea. You can store these tokens "globally", but that's only "global" to a single server. This approach would break as soon as a second server enters the picture.
That said, HttpContext.Current shouldn't be assigned to, ever. This is most likely the cause of the odd behavior you're seeing. Also, if your real code is more complex than ThrowIfCancellationRequested - i.e., if it's actually listening to the CancellationToken - then the call to Cancel can execute the remainder of StartSyncTest from within the call to Cancel, which would cause considerable confusion over the value of HttpContext.Current.
To summarize:
I recommend doing away with this approach completely; it won't work at all on web farms. Instead, keep your "task state" in an external storage system like a database.
Don't pass HttpContext across threads.

A colleague offered a alternative call to Task.Factory.StartNew (within StartAsyncTest):
await Task.Factory.StartNew(() =>
{
StartSyncTest(sSearchId, rCancellationToken);
},
rCancellationToken,
TaskCreationOptions.LongRunning,
TaskScheduler.FromCurrentSynchronizationContext());
This implementation seemed to solve the asynchronous issue. Now future calls to CancelAsyncTest succeed and cancel the task as intended.

How to know when all my threads have finished executing when in recursive method?

I have been working on a webscraping project.
I am having two issues, one being presenting the number of urls processed as percentage but a far larger issue is that I can not figure out how I know when all the threads i am creating are totaly finished.
NOTE: I am aware of that the a parallel foreach once done moves on BUT this is within a recursive method.
My code below:
public async Task Scrape(string url)
{
var page = string.Empty;
try
{
page = await _service.Get(url);
if (page != string.Empty)
{
if (regex.IsMatch(page))
{
Parallel.For(0, regex.Matches(page).Count,
index =>
{
try
{
if (regex.Matches(page)[index].Groups[1].Value.StartsWith("/"))
{
var match = regex.Matches(page)[index].Groups[1].Value.ToLower();
if (!links.Contains(BaseUrl + match) && !Visitedlinks.Contains(BaseUrl + match))
{
Uri ValidUri = WebPageValidator.GetUrl(match);
if (ValidUri != null && HostUrls.Contains(ValidUri.Host))
links.Enqueue(match.Replace(".html", ""));
else
links.Enqueue(BaseUrl + match.Replace(".html", ""));
}
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details."); ;
}
});
WebPageInternalHandler.SavePage(page, url);
var context = CustomSynchronizationContext.GetSynchronizationContext();
Parallel.ForEach(links, new ParallelOptions { MaxDegreeOfParallelism = 25 },
webpage =>
{
try
{
if (WebPageValidator.ValidUrl(webpage))
{
string linkToProcess = webpage;
if (links.TryDequeue(out linkToProcess) && !Visitedlinks.Contains(linkToProcess))
{
ShowPercentProgress();
Thread.Sleep(15);
Visitedlinks.Enqueue(linkToProcess);
Task d = Scrape(linkToProcess);
Console.Clear();
}
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details.");
}
});
Console.WriteLine("parallel finished");
}
}
catch (Exception e)
{
log.Error("Error occured: " + e.Message);
Console.WriteLine("Error occured, check log for further details.");
}
}
NOTE that Scrape gets called multiple times(recursive)
call the method like this:
public Task ExecuteScrape()
{
var context = CustomSynchronizationContext.GetSynchronizationContext();
Scrape(BaseUrl).ContinueWith(x => {
Visitedlinks.Enqueue(BaseUrl);
}, context).Wait();
return null;
}
which in turn gets called like so:
static void Main(string[] args)
{
RunScrapper();
Console.ReadLine();
}
public static void RunScrapper()
{
try
{
_scrapper.ExecuteScrape();
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
my result:
How do I solve this?

(Is it ethical for me to answer a question about web page scraping?)
Don't call Scrape recursively. Place the list of urls you want to scrape in a ConcurrentQueue and begin processing that queue. As the process of scraping a page returns more urls, just add them into the same queue.
I wouldn't use just a string, either. I recommend creating a class like
public class UrlToScrape //because naming things is hard
{
public string Url { get; set; }
public int Depth { get; set; }
}
Regardless of how you execute this it's recursive, so you have to somehow keep track of how many levels deep you are. A website could deliberately generate URLs that send you into infinite recursion. (If they did this then they don't want you scraping their site. Does anybody want people scraping their site?)
When your queue is empty that doesn't mean you're done. The queue could be empty, but the process of scraping the last url dequeued could still add more items back into that queue, so you need a way to account for that.
You could use a thread safe counter (int using Interlocked.Increment/Decrement) that you increment when you start processing a url and decrement when you finish. You're done when the queue is empty and the count of in-process urls is zero.
This is a very rough model to illustrate the concept, not what I'd call a refined solution. For example, you still need to account for exception handling, and I have no idea where the results go, etc.
public class UrlScraper
{
private readonly ConcurrentQueue<UrlToScrape> _queue = new ConcurrentQueue<UrlToScrape>();
private int _inProcessUrlCounter;
private readonly List<string> _processedUrls = new List<string>();
public UrlScraper(IEnumerable<string> urls)
{
foreach (var url in urls)
{
_queue.Enqueue(new UrlToScrape {Url = url, Depth = 1});
}
}
public void ScrapeUrls()
{
while (_queue.TryDequeue(out var dequeuedUrl) || _inProcessUrlCounter > 0)
{
if (dequeuedUrl != null)
{
// Make sure you don't go more levels deep than you want to.
if (dequeuedUrl.Depth > 5) continue;
if (_processedUrls.Contains(dequeuedUrl.Url)) continue;
_processedUrls.Add(dequeuedUrl.Url);
Interlocked.Increment(ref _inProcessUrlCounter);
var url = dequeuedUrl;
Task.Run(() => ProcessUrl(url));
}
}
}
private void ProcessUrl(UrlToScrape url)
{
try
{
// As the process discovers more urls to scrape,
// pretend that this is one of those new urls.
var someNewUrl = "http://discovered";
_queue.Enqueue(new UrlToScrape { Url = someNewUrl, Depth = url.Depth + 1 });
}
catch (Exception ex)
{
// whatever you want to do with this
}
finally
{
Interlocked.Decrement(ref _inProcessUrlCounter);
}
}
}
If I was doing this for real the ProcessUrl method would be its own class, and it would take HTML, not a URL. In this form it's difficult to unit test. If it were in a separate class then you could pass in HTML, verify that it outputs results somewhere, and that it calls a method to enqueue new URLs it finds.
It's also not a bad idea to maintain the queue as a database table instead. Otherwise if you're processing a bunch of urls and you have to stop, you'd have start all over again.

Can't you add all tasks Task d to some type of concurrent collection you thread through all recursive calls (via method argument) and then simply call Task.WhenAll(tasks).Wait()?
You'd need an intermediate method (makes it cleaner) that launches the base Scrape call and passes in the empty task collection. When the base call returns you have in hand all tasks and you simply wait them out.
public async Task Scrape (
string url) {
var tasks = new ConcurrentQueue<Task>();
//call your implementation but
//change it so that you add
//all launched tasks d to tasks
Scrape(url, tasks);
//1st option: Wait().
//This will block caller
//until all tasks finish
Task.WhenAll(tasks).Wait();
//or 2nd option: await
//this won't block and will return to caller.
//Once all tasks are finished method
//will resume in WriteLine
await Task.WhenAll(tasks);
Console.WriteLine("Finished!"); }
Simple rule: if you want to know when something finishes, the first step is to keep track of it. In your current implementation you are essentially firing and forgetting all launched tasks...

Exceptions are just ignored in async code block

Before I use Nito.MVVM, I used plain async/await and it was throwing me an aggregate exception and I could read into it and know what I have. But since Nito, my exceptions are ignored and the program jumps from async code block and continue executes. I know that it catch exceptions because when I put a breakpoint on catch(Exception ex) line it breaks here but with ex = null. I know that NotifyTask has properties to check if an exception was thrown but where I put it, it checks when Task is uncompleted, not when I need it.
View model:
public FileExplorerPageViewModel(INavigationService navigationService)
{
_navigationService = navigationService;
_manager = new FileExplorerManager();
Files = NotifyTask.Create(GetFilesAsync("UniorDev", "GitRemote/GitRemote"));
}
Private method:
private async Task<ObservableCollection<FileExplorerModel>> GetFilesAsync(string login, string reposName)
{
return new ObservableCollection<FileExplorerModel>(await _manager.GetFilesAsync(login, reposName));
}
Manager method(where exception throws):
public async Task<List<FileExplorerModel>> GetFilesAsync(string login, string reposName)
{
//try
//{
var gitHubFiles = await GetGitHubFilesAsync(login, reposName);
var gitRemoteFiles = new List<FileExplorerModel>();
foreach ( var file in gitHubFiles )
{
if ( file.Type == ContentType.Symlink || file.Type == ContentType.Submodule ) continue;
var model = new FileExplorerModel
{
Name = file.Name,
FileType = file.Type.ToString()
};
if ( model.IsFolder )
{
var nextFiles = await GetGitHubFilesAsync(login, reposName);
var count = nextFiles.Count;
}
model.FileSize = file.Size.ToString();
gitRemoteFiles.Add(model);
}
return gitRemoteFiles;
//}
//catch ( WebException ex )
//{
// throw new Exception("Something wrong with internet connection, try to On Internet " + ex.Message);
//}
//catch ( Exception ex )
//{
// throw new Exception("Getting ExplorerFiles from github failed! " + ex.Message);
//}
}
With try/catch or without it has the same effect. This behavior is anywhere where I have NotifyTask.
Update
There is no event, that fires when exception occurred, but there is Property Changed event, so I used it and added this code:
private void FilesOnPropertyChanged(object sender, PropertyChangedEventArgs propertyChangedEventArgs)
{
throw new Exception("EXCEPTION");
bool failed;
if ( Files.IsFaulted )
failed = true;
}
And exception not fires.
I added throw exception in App class (main class) and it fired. And when I have exceptions that come from XAML, it also fires. So maybe it not fires when it comes from a view model, or something else. I have no idea. Will be very happy for some help with it.
Update
We deal with exception = null, but the question is still alive. What I wanna add, that I rarely this issue, when the app is starting to launch on the physic device. I read some info about it, and it doesn't seem to be related, but maybe it is:

I'm not entirely sure what your desired behavior is, but here's some information I hope you find useful.
NotifyTask is a data-bindable wrapper around Task. That's really all it does. So, if its Task faults with an exception, then it will update its own data-bindable properties regarding that exception. NotifyTask is intended for use when you want the UI to respond to a task completing, e.g., show a spinner while the task is in progress, an error message if the task faults, and data if the task completes successfully.
If you want your application to respond to the task faulting (with code, not just a UI update), then you should use try/catch like you have commented out in GetFilesAsync. NotifyTask doesn't change how those exceptions work; they should work just fine.
I know that it catch exceptions because when I put a breakpoint on catch(Exception ex) line it breaks here but with ex = null.
That's not possible. I suggest you try it again.
I know that NotifyTask has properties to check if an exception was thrown but where I put it, it checks when Task is uncompleted, not when I need it.
If you really want to (asynchronously) wait for the task to complete and then check for exceptions, then you can do so like this:
await Files.TaskCompleted;
var ex = Files.InnerException;
Or, if you just want to re-raise the exception:
await Files.Task;
Though I must say this usage is extremely unusual. The much more proper thing to do is to have a try/catch within your GetFilesAsync.

Code after await runs twice

i have been developing a Windows form application and for DB end i am using Dapper.
The problem i am trying to solve is to execute a parametered stored procedure asynchronously so that my the execution of the application is not stopped and once the execution is done i should be able to show user a message
This is the function which calls the stored procedure (ExceptionData is not the usual meaning of the word. it has some other meaning in my business logic)
public async Task<bool> Load_ExceptionData(DateTime t)
{
IDbTransaction trans = null;
try
{
trans = _connection.BeginTransaction();
DynamicParameters prms = new DynamicParameters();
prms.Add("Today", t ,DbType.Date);
await _connection.ExecuteAsync("usp_CC_Load_ExceptionTable",
prms,
trans,
commandTimeout: 0,
commandType: CommandType.StoredProcedure
);
trans.Commit();
return true;
}
catch (Exception)
{
trans.Rollback();
return false;
}
}
And this function is called in trigger to a click
private async void todayToolStripMenuItem_Click(object sender, EventArgs e)
{
try
{
DateTime today;
today = Convert.ToDateTime("3/16/2016");//DateTime.Today;
CycleCountRepository _rep = new CycleCountRepository(); // Repository
todayToolStripMenuItem.Click -= todayToolStripMenuItem_Click; // Disabling the button so while in execution not clicked again
if (Enabled && _rep.Check_CycleCountToday(today)) // Checks if the data is already present for the date
{
todayToolStripMenuItem.Image = Properties.Resources.progress;
bool result = await _rep.Load_ExceptionData(today); // the async function
if (result == true)
{
ShowMessage("Data Load Successfully! Date: " + today, "Success", MessageBoxIcon.None); // This message is shown twice
}
else
{
ShowMessage("Data Load Error! Date: " + today, "Warning", MessageBoxIcon.Warning);
}
CheckTodayLoad();
}
else
{
ShowMessage("Nothing to Load! No locations scanned today!", "Warning", MessageBoxIcon.Warning);
}
}
catch (Exception ex)
{
ShowMessage(ex.Message, "Warning", MessageBoxIcon.Warning);
}
}
When i debug the code. the first time execution hits
await _rep.Load_ExceptionData(today);
it goes back again to the click event start and comes back to the await line.
After that the execution starts and once await is done the message is shown.
But after showing debugger points to
if (result == true)
line and when i step forward it goes inside it and again Shows the message
I am new to Async/await and this all may just be a dumb mistake i did. But still i will appreciate the answer

Alright guys i have figured out the problem. I will share so that no one else does the same mistake like i did.
So i had a function which was called on form load and was responsible to check whether the data which i am going to load is already present in the table. If "Yes" then remove the click event from the ToolStripMenuItem and if "No" then Add it.
What i was doing wrong was
todayToolStripMenuItem.Click += todayToolStripMenuItem_Click;
i was adding it twice to the Click
One time implicitly it was added in start of the form
When you click on the control from the design view. It adds its click even in your code and the assign it in the designer of the form
Second time i was explicitly adding it my self
check the line in the code
Solution was to remove the event before adding it to click so that it doesnt add twice
private void CheckTodayLoad()
{
DateTime today;
today = Convert.ToDateTime(DateTime.Today);
CycleCountRepository _rep = new CycleCountRepository();
if (_rep.CheckTodayLoaded(today))
{
todayToolStripMenuItem.Image = Properties.Resources.checkmark;
todayToolStripMenuItem.Text = "Today " + "[" + today.ToShortDateString() + "]" + " : Loaded";
todayToolStripMenuItem.Click -= todayToolStripMenuItem_Click;
}
else
{
todayToolStripMenuItem.Image = SystemIcons.Warning.ToBitmap();
todayToolStripMenuItem.Text = "Today " + "[" + today.ToShortDateString() + "]" + " : Not Loaded";
todayToolStripMenuItem.Click -= todayToolStripMenuItem_Click; // Just to make sure it is not registered twice. Previously this line wasnt there
todayToolStripMenuItem.Click += todayToolStripMenuItem_Click;
}
}
Cheers!

When await _connection.ExecuteAsync(...) is executed (because it, most certainly, doesn't complete synchronously), control is returned to the caller (bool result = await _rep.Load_ExceptionData(today);) which also returns control to the caller.
When connection.ExecuteAsync(...) completes, the continuation of the Load_ExceptionData method is posted to the synchronization context.
When _rep.Load_ExceptionData(today) completes, the continuation of the todayToolStripMenuItem_Click method is posted to the synchronization context.
I'm sure that's what you're seeing.

Do something like that
todayToolStripMenuItem.Click -= todayToolStripMenuItem_Click;
Thanks

Why TextBlock.Text doesn't get updated the first time (of two) I set its content? and how to resolve/work-around it?

Clipboard.SetText(...) very often creates considerable delay in my GUI. Set aside for now "correct-programming/design", I want to tell the user to wait a moment (line #01), and then let him/her know if succeeded or failed (lines #06 and #10). The thing is that I never see the "Copying..." text (line #01):
01| CopiedToClipboardTextBlock.Text = "Copying...";
02| try
03| {
04| // May cause considerable delay and GUI-freeze for a while...
05| Clipboard.SetText(textBlockContent);
06| CopiedToClipboardTextBlock.Text = "Copied to clipboard:\n" + textBlockContent;
07| }
08| catch (Exception e)
09| {
10| CopiedToClipboardTextBlock.Text = "Failed to copy to clipboard!";
11| }
So I thought - OK, maybe the GUI gets updated only when the event handler returns (which doesn't make sense to me...), so I tried many things, among others:
Using PreviewMouseDoubleClick to do CopiedToClipboardTextBlock.Text = "Copying...". Didn't help...
Using new thread in which Dispatcher.Invoke in new thread (don't event start... I know it's quite stupid... I was desperate)
(BTW, the delay in Clipboard.SetText(...) happens when it's about to fail. The failure is COMException: OpenClipboard Failed (Exception from HRESULT: 0x800401D0 (CLIPBRD_E_CANT_OPEN)), which is almost inevitable in many cases, but that's not the main issue here...)
Update:
This is my workaround:
CopiedToClipboardTextBlock.Text = "Copying...";
Exception resultException = null;
await Task.Run(() =>
{
var t = new Thread(obj =>
{
try { Clipboard.SetText(textBlockContent); }
catch (COMException e) { resultException = e; }
});
t.SetApartmentState(ApartmentState.STA);
t.Start();
t.Join();
});
if (resultException == null)
CopiedToClipboardTextBlock.Text = "Copied to clipboard:\n" + textBlockContent;
else
CopiedToClipboardTextBlock.Text = "Failed to copy to clipboard!";
The inner Thread is due to the requirement to have STA by Clipboard.SetText(...) function.
I don't know, seems very verbose to me. I suspect there's a simpler solution.

Yes, GUI updates only when the event handler returns (because event handler are called inside UI thread which is responsible for GUI updating).
To update the text before long operation you should invoke the operation in background thread. So event handler will be freed and GUI will be refreshed.
Try to set clipboard text like this:
await Task.Run(() => Clipboard.SetText(textBlockContent));