Develop Reference

Problem with multithreading and File options c#

my goal is to create a output file which parse all url in result.
My problem :
I use Paralelism and he can't be associated with File because it's secured,
i try to put a lock, and i think i don't put it correctly
message error is :
The process cannot access because it is being used by another process the file
What i want :
result is saved in a txt file
Code :
//VARIABLE
List<string> dorklist = File.ReadAllLines("dorks.txt").ToList();
int numdork = File.ReadAllLines("dorks.txt").Length;
int totalurls = File.ReadAllLines("Queue.txt").Length;
int chekeddork = 0;
int errors = 0;
File.Create("Queue.txt");
//TITLE
Task.Factory.StartNew(() =>
{
while (true) {
Console.Title = ("PARSER | " + chekeddork + "/" + numdork + " URLS : " + totalurls);
Thread.Sleep(100);
}
});
//BOUCLE
Parallel.ForEach(dorklist, new ParallelOptions { MaxDegreeOfParallelism = 100 }, dorklist => {
try
{
using (HttpRequest req = new HttpRequest())
{
//HEADERS
req.AddHeader(HttpHeader.UserAgent, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0");
//REQUESTS
var content = req.Get("https://www.google.com/search?q="+dorklist);
if (Convert.ToString(content.StatusCode) == "OK")
{
chekeddork++;
Console.WriteLine("Dork used : " + dorklist);
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(content.ToString());
var links = doc.DocumentNode.SelectNodes("//div[#class='r']//a");
foreach (var link in links)
{
var href = link.Attributes["href"];
var result = href.Value.ToString();
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
}
}
}
else
{ Console.WriteLine("Bad");
errors++;
}
}
}
catch (HttpException)
{
Console.WriteLine("Banned");
errors++;
}
});
}
}
}
AND the problem is here : `
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
Because there is Parallel.ForEach and the maxdegree
Thanks to all help me

you cannot be writing to one file from several threads at once via separate opens.
Open separate files per thread and merge afterwards.
Or open the file once before the parallel foreach and , under lock, write to the one open file.
Or put that using inside a lock
lock(dorkList) // or create a lock object
{
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
}
}
or use a specifc lock objecct like this
int chekeddork = 0;
int errors = 0;
File.Create("Queue.txt");
object locker = new Object();
....
lock(locker) // or create a lock object
{
using (StreamWriter streamaze = new StreamWriter("Queue.txt", true))
{
streamaze.WriteLine(results);
streamaze.Close();
}
}

Loading collection view with task async method

I am trying to load thumbnails with async task method with depency service :
In my pcl page I have this :
protected override void OnAppearing()
{
Device.BeginInvokeOnMainThread(() => UserDialogs.Instance.ShowLoading("Loading...", MaskType.Black));
Task.Run(async () =>
{
directoryPath = await getThumbnails.GetBitmaps(fileInfo.FullName);
List<ThumbnailsModel> thumbnailsModels = new List<ThumbnailsModel>();
int i = 1;
Directory.GetFiles(directoryPath).ToList<string>().ForEach(delegate (string thumbnailsEmplacement)
{
thumbnailsModels.Add(new ThumbnailsModel(i, thumbnailsEmplacement));
i++;
});
CollectionViewThumbnails.ItemsSource = thumbnailsModels;
}).ContinueWith(result => Device.BeginInvokeOnMainThread(() =>
{
UserDialogs.Instance.HideLoading();
}
)
);
}
My method to get the thumbnails :
public async Task<string> GetBitmaps(string filePath)
{
//TODO-- WORK ON THIS
var appDirectory = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyDocuments);
string fileName = System.IO.Path.GetFileNameWithoutExtension(filePath);
string directoryPath = System.IO.Path.Combine(appDirectory, "thumbnailsTemp", System.IO.Path.GetFileNameWithoutExtension(fileName));
var stream = new MemoryStream();
using (Stream resourceStream = new FileStream(filePath, FileMode.Open))
{
resourceStream.CopyTo(stream);
}
Document document = new Document(stream);
int count = document.Pages.Count;
for(int i = 0; i<= count; i++) {
TallComponents.PDF.Rasterizer.Page page = document.Pages[0];
using (var outputStream = new FileStream(System.IO.Path.Combine(directoryPath, fileName + "Thumbnails" + i + ".png"), FileMode.Create, FileAccess.Write))
{
await Task.Run(() =>
{
page.SaveAsBitmap(outputStream, CompressFormat.Png, 5);
});
}
}
return directoryPath;
}
The problem is that my application is going in my Dependency service method then going back in my pcl OnAppearing method before the thumbnails are done and going at this line
UserDialogs.Instance.HideLoading();

Seems like you have an unhandled exception. That continuation will run even if an exception is thrown on the Task you're continuing.
This can be changed using something like TaskContinuationOptions.OnlyOnRanToCompleted (and others) in the overload for ContinueWith. The default is TaskContinuationOptions.None if not specified.
Alternatively, you can access result.Exception in your continuation if you want it to run on failure and handle it.

I have code in C# that looks like this that I need to extract the same fields, but I am uncertain how to do it

In python, I have code that looks like this using libxml
parser = etree.HTMLParser()
id = 0
nodes = node.findall(r'.//div[#id="flexBox_flex_calendar_mainCal"]//table/tr[#class]')
for x in nodes:
if x.attrib['class'].startswith('calendar'):
item = GetARow(x, id)
newsitems.addRow(item)
id = id + 1
for id in range(0, newsitems.getLength()):
rowDict = newsitems.getRow(id)
if rowDict is not None:
rowItems = QStringList([rowDict['Time'], rowDict['Currency'], rowDict['Impact'], rowDict['Event'], rowDict['Actual'], rowDict['Forecast'], rowDict['Previous']] )
#newsItems[rowDict['Time']].append(rowItems)
newsTable.addrow(rowItems)
I have code in C# that looks like this that I need to extract the same fields, but I am uncertain how to do it. The whatNodesToFind string is problematical.
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Linq;
using System.Windows.Forms;
using HtmlAgilityPack;
namespace ConsoleApplication276
{
// a container for a url and a parser Action
public class Link
{
public string link { get; set; }
public Action<string> parser { get; set; }
}
public class Program
{
static string[] monthstrings = new string[] { "", "jan", "feb", "mar", "apr", "may", "jun", "july", "aug", "sep", "oct", "nov", "dec" };
public static string GetDateInFOREXFactoryFormat()
{
var today = System.DateTime.Now;
var dayStr = today.Day.ToString();
var monthStr = monthstrings[today.Month];
var yearStr = today.Year.ToString();
return dayStr + monthStr + '.' + yearStr;
}
// Entry Point of the console app
public static void Main(string[] args)
{
try
{
// download each page and dump the content
// you can add more links here, associate each link with a parser action, as for what data should the parser generate create a property for that in the Link container
var task = MessageLoopWorker.Run(DoWorkAsync, new Link()
{
link = "http://www.forexfactory.com/calendar.php?day=" + GetDateInFOREXFactoryFormat(),
parser = (string html) =>
{
//do what ever you need with hap here
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
string whatNodesToFind = ".//div";
//string whatNodesToFind = "table";
//var someNodes = doc.DocumentNode.SelectSingleNode(whatNodesToFind);
var someNodes = doc.DocumentNode.SelectNodes(whatNodesToFind);
foreach (var node in someNodes)
{
Console.WriteLine(node);
}
}
});
task.Wait();
Console.WriteLine("DoWorkAsync completed.");
}
catch (Exception ex)
{
Console.WriteLine("DoWorkAsync failed: " + ex.Message);
}
Console.WriteLine("Press Enter to exit.");
Console.ReadLine();
}
// navigate WebBrowser to the list of urls in a loop
public static async Task<Link> DoWorkAsync(Link[] args)
{
Console.WriteLine("Start working.");
using (var wb = new WebBrowser())
{
wb.ScriptErrorsSuppressed = true;
TaskCompletionSource<bool> tcs = null;
WebBrowserDocumentCompletedEventHandler documentCompletedHandler = (s, e) =>
tcs.TrySetResult(true);
// navigate to each URL in the list
foreach (var arg in args)
{
tcs = new TaskCompletionSource<bool>();
wb.DocumentCompleted += documentCompletedHandler;
try
{
wb.Navigate(arg.link.ToString());
// await for DocumentCompleted
await tcs.Task;
// after the page loads pass the html to the parser
arg.parser(wb.DocumentText);
}
finally
{
wb.DocumentCompleted -= documentCompletedHandler;
}
// the DOM is ready
Console.WriteLine(arg.link.ToString());
Console.WriteLine(wb.Document.Body.OuterHtml);
}
}
Console.WriteLine("End working.");
return null;
}
}
// a helper class to start the message loop and execute an asynchronous task
public static class MessageLoopWorker
{
public static async Task<Object> Run(Func<Link[], Task<Link>> worker, params Link[] args)
{
var tcs = new TaskCompletionSource<object>();
var thread = new Thread(() =>
{
EventHandler idleHandler = null;
idleHandler = async (s, e) =>
{
// handle Application.Idle just once
Application.Idle -= idleHandler;
// return to the message loop
await Task.Yield();
// and continue asynchronously
// propogate the result or exception
try
{
var result = await worker(args);
tcs.SetResult(result);
}
catch (Exception ex)
{
tcs.SetException(ex);
}
// signal to exit the message loop
// Application.Run will exit at this point
Application.ExitThread();
};
// handle Application.Idle just once
// to make sure we're inside the message loop
// and SynchronizationContext has been correctly installed
Application.Idle += idleHandler;
Application.Run();
});
// set STA model for the new thread
thread.SetApartmentState(ApartmentState.STA);
// start the thread and await for the task
thread.Start();
try
{
return await tcs.Task;
}
finally
{
thread.Join();
}
}
}
}
I tried this but it doesn't work, meaning it returns no nodes. Yet, I can see those nodes using Google Chrome inspect element:
var findclasses = doc.DocumentNode.Descendants("div").Where(d =>
d.Attributes.Contains("class") && d.Attributes["id"].Value.Contains("flex"));
foreach (var d in findclasses)
{
Console.WriteLine(d);
}

Regarding the Edit 1 section, I'd recommend to use d.GetAttributeValue("id", "") to replace d.Attributes["id"].Value, because the latter will throw exception in case current d element doesn't have the attribute id (and it did happen when parsing HTML page retrieved from the URL in this sample) :
var link = "http://www.forexfactory.com/calendar.php?day=aug7.2015";
var doc = new HtmlWeb().Load(link);
var findclasses = doc.DocumentNode
.Descendants("div")
.Where(d => d.Attributes.Contains("class")
&&
d.GetAttributeValue("id", "").Contains("flex")
);
foreach (var d in findclasses)
{
Console.WriteLine("{0}, {1}", d.Name, d.GetAttributeValue("id", ""));
}
Dotnetfiddle Demo
output :
div, flexBox_flex_minicalendar_
div, flexBox_flex_calendar_mainCal
div, flexDatePicker_Calendar_mainCal_begindate
div, flexDatePicker_Calendar_mainCal_enddate

The answer is really subtle. It turns out there is a section missing from the html! Instantiating a "correct" Webclient with the right headers works:
using (WebClient wb = new WebClient())
{
wb.Headers["User-Agent"] =
"User-Agent" + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3 Gecko/2008092417 Firefox/3.0.3";

How to create async http requests in a loop?

Yesterday I've found out how to create several async http requests without async/await. But today I need to do it in a loop: if some of responses don't satisfy some condition - I need to change a request for them and send these requests again. It may be repeated several times.
I've tried this code:
do
{
var loadingCoordinatesTasks = new List<Task<Terminal>>();
var totalCountOfTerminals = terminalPresetNode.ChildNodes.Count;
var uiTaskScheduler = TaskScheduler.FromCurrentSynchronizationContext();
foreach (var terminal in terminals.Except(_terminalsWithCoordinates))
{
var address = terminal.GetNextAddress();
var webRequest = (HttpWebRequest)WebRequest.Create(GeoCoder.GeoCodeUrl + address);
var webRequestTask = Task.Factory.FromAsync<WebResponse>(webRequest.BeginGetResponse,
webRequest.EndGetResponse,
terminal);
var parsingTask = webRequestTask.ContinueWith(antecedent =>
{
// Parse the response
});
loadingCoordinatesTasks.Add(parsingTask);
}
Task.Factory.ContinueWhenAll(loadingCoordinatesTasks.ToArray(), antecedents =>
{
foreach (var antecedent in antecedents)
{
var terminalWithCoordinates = antecedent.Result;
if (antecedent.Status == TaskStatus.RanToCompletion &&
!terminalWithCoordinates.Coordinates.AreUnknown)
{
_terminalsWithCoordinates.Add(terminalWithCoordinates);
_countOfProcessedTerminals++;
}
}
});
} while (_countOfProcessedTerminals < totalCountOfTerminals);
but is it possible to check the condition in while just after every single set of requests executed?

You can perform the check after increasing the count:
_countOfProcessedTerminals++;
if (_countOfProcessedTerminals >= totalCountOfTerminals)
{
break;
}
Is _countOfProcessedTerminals thread-safe though?

I manage to do it using recursion:
public void RunAgainFailedTasks(IEnumerable<Task<Terminal>> tasks)
{
Task.Factory.ContinueWhenAll(tasks.ToArray(), antecedents =>
{
var failedTasks = new List<Task<Terminal>>();
foreach (var antecedent in antecedents)
{
var terminal = antecedent.Result;
// Previous request was failed
if (terminal.Coordinates.AreUnknown)
{
string address;
try
{
address = terminal.GetNextAddress();
}
catch (FormatException) // No versions more
{
continue;
}
var getCoordinatesTask = CreateGetCoordinatesTask(terminal, address);
failedTasks.Add(getCoordinatesTask);
}
else
{
_terminalsWithCoordinates.Add(terminal);
}
}
if (failedTasks.Any())
{
RunAgainFailedTasks(failedTasks);
}
else
{
// Display a map
}
}, CancellationToken.None,
TaskContinuationOptions.None,
TaskScheduler.FromCurrentSynchronizationContext());
}
private Task<Terminal> CreateGetCoordinatesTask(Terminal terminal, string address)
{
var webRequest = (HttpWebRequest)WebRequest.Create(GeoCoder.GeoCodeUrl + address);
webRequest.KeepAlive = false;
webRequest.ProtocolVersion = HttpVersion.Version10;
var webRequestTask = Task.Factory.FromAsync<WebResponse>(webRequest.BeginGetResponse,
webRequest.EndGetResponse,
terminal);
var parsingTask = webRequestTask.ContinueWith(webReqTask =>
{
// Parse the response
});
return parsingTask;
}

Task.WaitAll not waiting on other async methods

I'm asynchronously retrieving some rss articles with my Portable Class Library that uses the Microsoft.Bcl library (which doesn't have Task.WhenAll). Each article has a url to rss comments that I need to asynchronously retrieve as well.
The code below is my library. I call GetArticles() but it does not return any of the which creates a list of tasks that call GetComments() to asynchronously get the comments.
I've tried using Task.WaitAll in GetArticles to wait for the comments but it does not block the thread. Any help would be appreciated.
private const string ArticlesUri = "";
public async Task<List<ArticleBrief>> GetArticles()
{
var results = new List<ArticleBrief>();
try
{
var wfw = XNamespace.Get("http://wellformedweb.org/CommentAPI/");
var media = XNamespace.Get("http://search.yahoo.com/mrss/");
var dc = XNamespace.Get("http://purl.org/dc/elements/1.1/");
var t = await WebHttpRequestAsync(ArticlesUri);
StringReader stringReader = new StringReader(t);
using (var xmlReader = System.Xml.XmlReader.Create(stringReader))
{
var doc = System.Xml.Linq.XDocument.Load(xmlReader);
results = (from e in doc.Element("rss").Element("channel").Elements("item")
select
new ArticleBrief()
{
Title = e.Element("title").Value,
Description = e.Element("description").Value,
Published = Convert.ToDateTime(e.Element("pubDate").Value),
Url = e.Element("link").Value,
CommentUri = e.Element(wfw + "commentRss").Value,
ThumbnailUri = e.Element(media + "thumbnail").FirstAttribute.Value,
Categories = GetCategoryElements(e.Elements("category")),
Creator = e.Element(dc + "creator").Value
}).ToList();
}
var tasks = new Queue<Task>();
foreach (var result in results)
{
tasks.Enqueue(
Task.Factory.StartNew(async ()=>
{
result.Comments = await GetComments(result.CommentUri);
}
));
}
Task.WaitAll(tasks.ToArray());
}
catch (Exception ex)
{
// should do some other
// logging here. for now pass off
// exception to callback on UI
throw ex;
}
return results;
}
public async Task<List<Comment>> GetComments(string uri)
{
var results = new List<Comment>();
try
{
var wfw = XNamespace.Get("http://wellformedweb.org/CommentAPI/");
var media = XNamespace.Get("http://search.yahoo.com/mrss/");
var dc = XNamespace.Get("http://purl.org/dc/elements/1.1/");
var t = await WebHttpRequestAsync(uri);
StringReader stringReader = new StringReader(t);
using (var xmlReader = System.Xml.XmlReader.Create(stringReader))
{
var doc = System.Xml.Linq.XDocument.Load(xmlReader);
results = (from e in doc.Element("rss").Element("channel").Elements("item")
select
new Comment()
{
Description = e.Element("description").Value,
Published = Convert.ToDateTime(e.Element("pubDate").Value),
Url = e.Element("link").Value,
Creator = e.Element(dc + "creator").Value
}).ToList();
}
}
catch (Exception ex)
{
// should do some other
// logging here. for now pass off
// exception to callback on UI
throw ex;
}
return results;
}
private static async Task<string> WebHttpRequestAsync(string url)
{
//TODO: look into getting
var request = WebRequest.Create(url);
request.Method = "GET";
var response = await request.GetResponseAsync();
return ReadStreamFromResponse(response);
}
private static string ReadStreamFromResponse(WebResponse response)
{
using (Stream responseStream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(responseStream))
{
string strContent = sr.ReadToEnd();
return strContent;
}
}
private List<string> GetCategoryElements(IEnumerable<XElement> categories)
{
var listOfCategories = new List<string>();
foreach (var category in categories)
{
listOfCategories.Add(category.Value);
}
return listOfCategories;
}
Updated Code from Solution, just added .UnWrap() on the Enqueue method:
var tasks = new Queue<Task>();
foreach (var result in results)
{
tasks.Enqueue(
Task.Factory.StartNew(async ()=>
{
result.Comments = await GetComments(result.CommentUri);
}
).Unwrap());
}
Task.WaitAll(tasks.ToArray());

It is waiting appropriately. The problem is that you are creating a Task which creates another task (i.e. StartNew is returning a Task<Task> and you are only waiting on the outer Task which completes rather quickly (it completes before the inner Task is complete)).
The questions will be:
Do you really want that inner task?
If yes, then you can use Task.Unwrap to get a proxy task that represents the completion of both the inner and outer Task and use that to Wait on.
If no, then you could remove the use of async/await in StartNew so that there is not an inner task (I think this would be prefered, it's not clear why you need the inner task).
Do you really need to do a synchronous Wait on an asynchronous Task? Read some of Stephen Cleary's blog: http://blog.stephencleary.com/2012/02/async-unit-tests-part-1-wrong-way.html
As an aside, if you are not using C# 5, then watch out for closing over the foreach variable result See
Has foreach's use of variables been changed in C# 5?, and
http://blogs.msdn.com/b/ericlippert/archive/2009/11/12/closing-over-the-loop-variable-considered-harmful.aspx)

In Microsoft.Bcl.Async we couldn't add any static methods to Task. However, you can find most of the methods on TaskEx, for example, TaskEx.WhenAll() does exist.