I'm currently working with the Async CTP and need to convert this code into code where I can use Task.WhenAll().
What I did until now was using the UserState object and put my identifier (AID) into it and then use it in the completed event.
However the wc.DownloadFileTaskAsync methode doesn't have an overload with UserState. What can I do?
for (int i = 0; i < SortedRecommendations.Count; i++)
{
string tempfilepath = filepath + SortedRecommendations[i].Aid + ".jpg";
if (File.Exists(tempfilepath))
continue;
WebClient wc = new WebClient();
wc.DownloadFileCompleted += (s, e) =>
{
var q = SortedRecommendations.Where(x => x.Aid == (int)e.UserState);
if (q.Count() > 0)
q.First().Image = tempfilepath;
};
wc.DownloadFileAsync(new Uri(SortedRecommendations[i].Image.Replace("t.jpg", ".jpg")), tempfilepath, SortedRecommendations[i].Aid);
}
This is basically with what I came up with, however I'm getting a out ouf bounds exception at y.Aid == SortedRecommendations[i].Aid because i is now obvioulsy something else then it was when the download started. Only other possibility I see is using something like TaskEx.Run( () => { // download data synchronously }; but I don't like this approach.
for (int i = 0; i < SortedRecommendations.Count; i++)
{
string tempfilepath = filepath + SortedRecommendations[i].Aid + ".jpg";
if (File.Exists(tempfilepath))
continue;
WebClient wc = new WebClient();
wc.DownloadFileCompleted += (s, e) =>
{
var q = SortedRecommendations.Where(x => x.Aid == SortedRecommendations[i].Aid);
if (q.Count() > 0)
q.First().Image = tempfilepath;
};
tasks.Add(wc.DownloadFileTaskAsync(new Uri(SortedRecommendations[i].Image.Replace("t.jpg", ".jpg")), tempfilepath));
}
await TaskEx.WhenAll(tasks);
//Everything finished
First, I think you shouldn't base your logic on ids (unless you really have to). You should use references to the objects in the SortedRecommendations collection.
Now, if you wanted to download only one file at a time, you could simply use await:
for (int i = 0; i < SortedRecommendations.Count; i++)
{
string tempfilepath = filepath + SortedRecommendations[i].Aid + ".jpg";
if (File.Exists(tempfilepath))
continue;
WebClient wc = new WebClient();
var recommendation = SortedRecommendations[i];
await wc.DownloadFileTaskAsync(new Uri(recommendation.Image.Replace("t.jpg", ".jpg")), tempfilepath);
recommendation.Image = tempfilepath;
}
But, if you wanted to start all of the downloads at the same time, like your DownloadFileAsync() code does, you could use ContinueWith() instead. And you don't need user state, that's what closures are for:
for (int i = 0; i < SortedRecommendations.Count; i++)
{
string tempfilepath = filepath + SortedRecommendations[i].Aid + ".jpg";
if (File.Exists(tempfilepath))
continue;
WebClient wc = new WebClient();
var recommendation = SortedRecommendations[i];
var downloadTask = wc.DownloadFileTaskAsync(new Uri(recommendation.Image.Replace("t.jpg", ".jpg")), tempfilepath);
var continuation = downloadTask.ContinueWith(t => recommendation.Image = tempfilepath);
tasks.Add(continuation);
}
await Task.WhenAll(tasks);
The best solution would probably be to download a limited number of files at once, not one or all of them. Doing that is more complicated and one solution would be to use ActionBlock from TPL Dataflow with MaxDegreeOfParallelism set.
Related
I have some nested async methods calling each other and it is confusing. I am trying to convert a project which downloads the files in an async download.
On the click of the download button this is the method triggered:
private async void enableOfflineModeToolStripMenuItem_Click(object sender, EventArgs e)
{
for(int i = 0; i < _playlists.Count; i++)
{
DoubleDimList.Add(new List<String>());
for(int j = 0; j < 5; j++)
{
string sMp3 = IniReadValue(_playlists[i], "Track " + j);
DoubleDimList[i].Add(sMp3);
}
await Task.Run(() => _InetGetHTMLSearchAsyncs(DoubleDimList[i]));
}
}
It creates a 2d List which at the end looks like this DoubleDimList[3][20].
At the end of each sublist I am doing an async download as you can see. The method looks like this
private async Task _InetGetHTMLSearchAsyncs(List<string> urlList)
{
foreach (var url in urlList)
{
await Task.Run(() => _InetGetHTMLSearchAsync(url));
}
}
the _InetGetHTMLSearchAsync method looks like this and here is where it gets tricky
private async Task _InetGetHTMLSearchAsync(string sTitle)
{
Runs++;
if (AudioDumpQuery == string.Empty)
{
//return string.Empty;
}
string sResearchURL = "http://www.audiodump.biz/music.html?" + AudioDumpQuery + sTitle.Replace(" ", "+");
try
{
using (var client = new WebClient())
{
client.Headers.Add("Referer", #"http://www.audiodump.com/");
client.Headers.Add("user-agent", "Mozilla / 5.0(Macintosh; Intel Mac OS X 10_9_3) AppleWebKit / 537.75.14(KHTML, like Gecko) Version / 7.0.3 Safari / 7046A194A");
client.DownloadStringCompleted += Client_DownloadStringCompleted;
await Task.Run(() => client.DownloadStringAsync(new Uri(sResearchURL)));
}
}
catch (Exception ex)
{
Console.WriteLine("Debug message: " + ex.Message + "InnerEx: " + ex.StackTrace);
Console.WriteLine("Runs: " + Runs);
return;
}
}
On Client_DownloadStringCompleted there is another async method called. Here it is
private async void Client_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
string[] sStringArray;
string aRet = e.Result;
string[] aTable = _StringBetween(aRet, "<BR><table", "table><BR>", RegexOptions.Singleline);
if (aTable != null)
{
string[] aInfos = _StringBetween(aTable[0], ". <a href=\"", "<a href=\"");
if (aInfos != null)
{
for (int i = 0; i < 1; i++)
{
sStringArray = aInfos[i].Split('*');
sStringArray[0] = sStringArray[0].Replace("'", "'");
aLinks.Add(sStringArray[0]);
}
await Task.Run(() => DownloadFile(aLinks[FilesDownloaded]));
}
}
}
From there, surprise! Another async call.
private async Task DownloadFile(string url)
{
try
{
using (var client = new WebClient())
{
client.Headers.Add("Referer", #"http://www.audiodump.biz/");
client.Headers.Add("user-agent", "Mozilla / 5.0(Macintosh; Intel Mac OS X 10_9_3) AppleWebKit / 537.75.14(KHTML, like Gecko) Version / 7.0.3 Safari / 7046A194A");
client.DownloadFileCompleted += Client_DownloadFileCompleted;
await Task.Run(() => client.DownloadFileTaskAsync(url, mp3Path + "\\" + count + ".mp3"));
}
}
catch (Exception Ex)
{
Console.WriteLine("File download error: " + Ex.StackTrace);
}
}
Now the first part after the creation of the 2d List is to retrieve the download links of the mp3s. The second part is to download the mp3 as soon as a valid URL was provided. It works but in a bizarre way. Instead of downloading the file normally(1st, 2nd, 3rd...), it will download randomly the files(1st, 5th, 8th...).
It is my first go for async download and boy, I am already far from my limits.
Where am I messing this up? And the main question, will this ever work the way it is supposed to work?
Your code looks pretty good, except for two things:
You shouldn't be using Task.Run. The primary use case for Task.Run is for moving CPU-bound work off a GUI thread so it doesn't block the UI. I have a series on the proper use of Task.Run that goes into this in detail.
You should use a consistent asynchronous pattern, ideally TAP. Your code is currently using TAP everywhere except in _InetGetHTMLSearchAsync, which is using EAP. This is what is causing the odd behavior you're seeing.
A fixed _InetGetHTMLSearchAsync would look something like this:
private async Task _InetGetHTMLSearchAsync(string sTitle)
{
Runs++;
string sResearchURL = "http://www.audiodump.biz/music.html?" + AudioDumpQuery + sTitle.Replace(" ", "+");
try
{
using (var client = new WebClient())
{
client.Headers.Add("Referer", #"http://www.audiodump.com/");
client.Headers.Add("user-agent", "Mozilla / 5.0(Macintosh; Intel Mac OS X 10_9_3) AppleWebKit / 537.75.14(KHTML, like Gecko) Version / 7.0.3 Safari / 7046A194A");
string[] sStringArray;
string aRet = await client.DownloadStringTaskAsync(new Uri(sResearchURL));
string[] aTable = _StringBetween(aRet, "<BR><table", "table><BR>", RegexOptions.Singleline);
if (aTable != null)
{
string[] aInfos = _StringBetween(aTable[0], ". <a href=\"", "<a href=\"");
if (aInfos != null)
{
for (int i = 0; i < 1; i++)
{
sStringArray = aInfos[i].Split('*');
sStringArray[0] = sStringArray[0].Replace("'", "'");
aLinks.Add(sStringArray[0]);
}
await DownloadFile(aLinks[FilesDownloaded]); // Should really be called "DownloadFileAsync"
}
}
}
}
catch (Exception ex)
{
Console.WriteLine("Debug message: " + ex.Message + "InnerEx: " + ex.StackTrace);
Console.WriteLine("Runs: " + Runs);
return;
}
}
I'm trying to scroll through images in my app, but I'm having trouble figuring out how to populate my list. The images are named using numbers from 1.jpg upwards. If anyone could help it would be great.
async private void Exec()
{
// Get the file location.
StorageFolder appFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;
string myImageFolder = (appFolder.Path + "\\Assets\\Images");
int imageNumber = 1;
List<Uri> fileList = new List<Uri>();
foreach (var fileItem in fileList)
{
string imageFileName = imageNumber + ".jpg";
Uri uri = new Uri(myImageFolder + "/" + imageFileName);
fileList.Add(uri);
image.Source = new BitmapImage(new Uri(uri.ToString()));
await Task.Delay(TimeSpan.FromSeconds(1));
imageNumber++;
}
}
UPDATE
I have tried to create a workaround and do this without the foreach statement but its crashing when testing if the next file exists: :(
async private void Exec()
{
// Get the file location.
string root = Windows.ApplicationModel.Package.Current.InstalledLocation.Path;
string path = root + #"\Assets\Images";
StorageFolder appFolder = await StorageFolder.GetFolderFromPathAsync(path);
int imageNumber = 1;
int test = imageNumber;
do
{
string imageFileName = imageNumber + ".jpg";
Uri uri = new Uri(path + "\\" + imageFileName);
image.Source = new BitmapImage(new Uri(uri.ToString()));
await Task.Delay(TimeSpan.FromSeconds(1));
test = imageNumber + 1;
imageNumber++;
string testFile = test + ".jpg";
Uri uri1 = new Uri(path + "\\" + testFile);
if (await appFolder.TryGetItemAsync(uri1.ToString()) != null)
{
test = 99999;
}
}
while (test != 99999);
}
Your list does not contain any items. Your foreach will never run, as there will be no entries in your list.
You need to go through all paths in myImageFolder-root and add those uris to the list, then you can just use them in a foreach to create images and set their source, for every uri in the list.
Also imageNumber is un-needed then as you will have the URIs.
Prep the list of URIs first, by traversing the folder. Then modify the existing foreach to use those to build image objects.
Also, refrain from adding to a collection WHILE iterating it...
I have this working, and not a single foreach was required :D Thanks #Richard Eriksson
async private void Exec()
{
// Get the file location.
string root = Windows.ApplicationModel.Package.Current.InstalledLocation.Path;
string path = root + #"\Assets\Images";
StorageFolder appFolder = await StorageFolder.GetFolderFromPathAsync(path);
int imageNumber = 1;
int test = imageNumber;
do
{
string imageFileName = imageNumber + ".jpg";
Uri uri = new Uri(path + "\\" + imageFileName);
image.Source = new BitmapImage(new Uri(uri.ToString()));
await Task.Delay(TimeSpan.FromSeconds(1));
test = imageNumber + 1;
imageNumber++;
string testFile = test + ".jpg";
if (await appFolder.TryGetItemAsync(testFile) != null)
{
test = 99999;
}
else
{
test = 1;
}
}
while (test == 99999);
}
I want to download more than one file by using webclient method and many threads running at the same time. My url structure depends on a variable 'int i', so i use a for loop to generate urls and filepaths. The problem is until the started thread is brought upon, the url and filepath values are changed. The timeline occurs as below:
In main loop, url = "url1" and path = "filepath1".
Thread1 is called with value "url1" and "filepath1".
In main loop, url = "url2" and path = "filepath2".
Thread2 is called with value "url2" and "filepath2".
Thread1 started with value "url2" and "filepath2".
Thread2 started with value "url2" and "filepath2".
I couldn't find any elegant solutions. What would you suggest?
string path = "";
string url = "";
string baseURL = "http://www.somewebsite.com/12/";
for (int i = 10; i <= DateTime.Now.Month; i++)
{
path = "C:\\folder\\" + i.ToString() + ".html";
url = baseURL + i.ToString();
Thread webThread = new Thread(delegate()
{
downloadScheduleFile(url,path);
});
webThread.Start()
}
private void downloadScheduleFile(string url, string filepath)
{
var client = new WebClient();
try
{
client.DownloadFile(url, filepath);
}
catch(WebException e) {
Console.WriteLine(System.Threading.Thread.CurrentThread.Name+e.Message);
}
}
Its because by the time your thread starts, path and url have changed. You have to create closer local copies.
string baseURL = "http://www.somewebsite.com/12/";
for (int i = 10; i <= DateTime.Now.Month; i++)
{
string path = "C:\\folder\\" + i.ToString() + ".html"; // path declared here
string url = baseURL + i.ToString(); // url declared here
Thread webThread = new Thread(delegate()
{
downloadScheduleFile(url,path);
});
webThread.Start()
}
The way you code is written all threads calling the downloadScheduleFile are referencing the same 2 variables defined in the encompassing block. What you should do is to give every thread its own set of variables.
You need to capture the variables in the outer scope within the delegate, I'm pretty sure you can do this:
string path = "";
string url = "";
string baseURL = "http://www.somewebsite.com/12/";
for (int i = 10; i <= DateTime.Now.Month; i++)
{
path = "C:\\folder\\" + i.ToString() + ".html";
url = baseURL + i.ToString();
Thread webThread = new Thread(delegate()
{
string innerPath = path;
string innerUrl = url
downloadScheduleFile(innerUrl,innerPath);
});
webThread.Start()
}
But give it a try as you might end up with the same issue...
I have a listview which I show video results from YouTube. Everything works fine but one thing I noticed is that the way it works seems to be a bit slow and it might be due to my code. Are there any suggestions on how I can make this better? Maybe loading the images directly from the url instead of using a webclient?
I am adding the listview items in a loop from video feeds returned from a query using the YouTube API. The piece of code which I think is slowing it down is this:
Feed<Video> videoFeed = request.Get<Video>(query);
int i = 0;
foreach (Video entry in videoFeed.Entries)
{
string[] info = printVideoEntry(entry).Split(',');
WebClient wc = new WebClient();
wc.DownloadFile(#"http://img.youtube.com/vi/" + info[0].ToString() + "/hqdefault.jpg", info[0].ToString() + ".jpg");
string[] row1 = { "", info[0].ToString(), info[1].ToString() };
ListViewItem item = new ListViewItem(row1, i);
YoutubeList.Items.Add(item);
imageListSmall.Images.Add(Bitmap.FromFile(info[0].ToString() + #".jpg"));
imageListLarge.Images.Add(Bitmap.FromFile(info[0].ToString() + #".jpg"));
}
public static string printVideoEntry(Video video)
{
return video.VideoId + "," + video.Title;
}
As you can see I use a Webclient which downloads the images so then I can use them as image in my listview. It works but what I'm concerned about is speed..any suggestions? maybe a different control all together?
Ok, I hope this is the code you were looking for, I can't test it since I don't have dll that you are using but I think it's OK.
Feed<Video> videoFeed = request.Get<Video>(query);
Thread th = new Thread(new ParameterizedThreadStart( GetImages));
th.Start(videoFeed);
int i = 0;
foreach (Video entry in videoFeed.Entries)
{
string[] info = printVideoEntry(entry).Split(',');
string[] row1 = { "", info[0].ToString(), info[1].ToString() };
ListViewItem item = new ListViewItem(row1, i++);
YoutubeList.Items.Add(item);
}
}
void GetImages(object arg)
{
Feed<Video> videoFeed = Feed<Video> arg;
foreach (Video entry in videoFeed.Entries)
{
string[] info = printVideoEntry(entry).Split(',');
WebClient wc = new WebClient();
wc.DownloadFile(#"http://img.youtube.com/vi/" + info[0].ToString() + "/hqdefault.jpg", info[0].ToString() + ".jpg");
ImageAdd(info[0]+".jpg");
}
}
delegate void imageAdder(string imgName);
void AddImage(string imgName)
{
imageListSmall.Images.Add(Bitmap.FromFile(imgName + #".jpg"));
imageListLarge.Images.Add(Bitmap.FromFile(imgName + #".jpg"));
listView1.Refresh();
}
void ImageAdd(string imgName)
{
this.Invoke(new imageAdder(AddImage), new object[] { imgName });
}
Try it and I will comment it if you have some problems.
I want to go to multiple pages using ASP.NET 4.0, copy all HTML and then finally paste it in a text box. From there I would like to run my parsing function, what is the best way to handle this?
protected void goButton_Click(object sender, EventArgs e)
{
if (datacenterCombo.Text == "BL2")
{
fwURL = "http://website1.com/index.html";
l2URL = "http://website2.com/index.html";
lbURL = "http://website3.com/index.html";
l3URL = "http://website4.com/index.html";
coreURL = "http://website5.com/index.html";
WebRequest objRequest = HttpWebRequest.Create(fwURL);
WebRequest layer2 = HttpWebRequest.Create(l2URL);
objRequest.Credentials = CredentialCache.DefaultCredentials;
using (StreamReader layer2 = new StreamReader(layer2.GetResponse().GetResponseStream()))
using (StreamReader objReader = new StreamReader(objRequest.GetResponse().GetResponseStream()))
{
originalBox.Text = objReader.ReadToEnd();
}
objRequest = HttpWebRequest.Create(l2URL);
//Read all lines of file
String[] crString = { "<BR> " };
String[] aLines = originalBox.Text.Split(crString, StringSplitOptions.RemoveEmptyEntries);
String noHtml = String.Empty;
for (int x = 0; x < aLines.Length; x++)
{
if (aLines[x].Contains(ipaddressBox.Text))
{
noHtml += (RemoveHTML(aLines[x]) + "\r\n");
}
}
//Print results to textbox
resultsBox.Text = String.Join(Environment.NewLine, noHtml);
}
}
public static string RemoveHTML(string text)
{
text = text.Replace(" ", " ").Replace("<br>", "\n");
var oRegEx = new System.Text.RegularExpressions.Regex("<[^>]+>");
return oRegEx.Replace(text, string.Empty);
}
Instead of doing all this manually you should probably use HtmlAgilityPack instead then you could do something like this:
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("http://google.com");
var targetNodes = doc.DocumentNode
.Descendants()
.Where(x=> x.ChildNodes.Count == 0
&& x.InnerText.Contains(someIpAddress));
foreach (var node in targetNodes)
{
//do something
}
If HtmlAgilityPack is not an option for you, simplify at least the download portion of your code and use a WebClient:
using (WebClient wc = new WebClient())
{
string html = wc.DownloadString("http://google.com");
}