Get html from website - c#

I can get an html code from web site this way:
public void Test()
{
WebClient client = new WebClient();
client.DownloadStringCompleted +=
new DownloadStringCompletedEventHandler(client_DownloadStringCompleted);
client.DownloadStringAsync(new Uri("http://testUrl.xml"));
}
void client_DownloadStringCompleted(object sender,
DownloadStringCompletedEventArgs e)
{
string html = e.Result;
//Now do something with the string...
}
But I need to get updated html each 30 seconds, so I wrote:
public void TestMain()
{
DispatcherTimer Timer = new DispatcherTimer()
{
Interval = TimeSpan.FromSeconds(30)
};
Timer.Tick += (s, t) =>
{
Test();
};
Timer.Start();
}
I changed the xml but I get the same html, what is wrong?

There's a cache included in the WebClient. If you request the same URI twice, the second time it will fetch the whole content directly from the cache.
There's no way to disable cache on WebClient, so you have two workarounds:
Use HttpWebRequest instead of WebClient
Add a random parameter to the URI:
client.DownloadStringAsync(new Uri("http://testUrl.xml?nocache=" + Guid.NewGuid()));

Related

Detect when Webclient.DownloadFileAsync hangs because of change in dynamic Ip Address

I am using webclient for large size file download.
In my country, every 12 hours, the Ip Changes, so if user is downloading file and his ipChanges , the download hangs, but I get not feedback that it hanged..
I would like to detect this in windows form , c#.
public void DownloadFile(string urlAddress, string location)
{
try
{
// The stopwatch which we will be using to calculate the download speed
using (webClient = new WebClient())
{
webClient.DownloadFileCompleted += new AsyncCompletedEventHandler(Completed);
webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(ProgressChanged);
// The variable that will be holding the url address (making sure it starts with http://)
Uri URL = urlAddress.StartsWith("http://", StringComparison.OrdinalIgnoreCase) ? new Uri(urlAddress) : new Uri("http://" + urlAddress);
// Start the stopwatch which we will be using to calculate the download speed
sw.Start();
// Start downloading the file
webClient.DownloadFileAsync(new Uri(urlAddress), location);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
I also tried calling this for detecting Ip Address change but it is not being fired when manually resetting router.
public Form1()
{
NetworkChange.NetworkAddressChanged += new NetworkAddressChangedEventHandler(AddressChangedCallback);
}
private void AddressChangedCallback(object sender, EventArgs e)
{
//Code here...
}

Pause loop until download event is complete

How can I pause the loop in someRandomMethod() until the code in DownloadCompleted() have been executed? This code below only unpacks the latest version in the versions array. It's like the loop is faster than the first download and m_CurrentlyDownloading have the latest value the first time DownloadCompleted() is beeing executed.
private void someRandomMethod() {
for (int i = 0; i < versions.Count; i++)
{
//ClearInstallFolder();
m_CurrentlyDownloading = versions.ElementAt(i);
Download(versions.ElementAt(i));
LocalUpdate(versions.ElementAt(i));
System.Threading.Thread.Sleep(500);
}
}
private void Download(string p_Version)
{
string file = p_Version + ".zip";
string url = #"http://192.168.56.5/" + file;
//client is global in the class
client = new WebClient();
client.DownloadFileCompleted += new AsyncCompletedEventHandler(DownloadCompleted);
client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(DownloadProgressChanged);
client.DownloadFileAsync(new Uri(url), #"C:\tmp\" + file);
}
private void DownloadCompleted(object sender, AsyncCompletedEventArgs e)
{
if (e.Error == null)
{
Unpack(m_CurrentlyDownloading);
if (GetInstalledVersion() == GetLatestVersion())
ClearZipFiles();
}
else
MessageBox.Show(e.Error.ToString());
}
The easiest way would be to not use the *async methods. The normal DownloadFile will pause execution until it completes.
But if you've got access to the Await keyword, try this.
private async Task Download(string p_Version)
{
string file = p_Version + ".zip";
string url = #"http://192.168.56.5/" + file;
//client is global in the class
client = new WebClient();
client.DownloadFileCompleted += new AsyncCompletedEventHandler(DownloadCompleted);
client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(DownloadProgressChanged);
await client.DownloadFileAsync(new Uri(url), #"C:\tmp\" + file);
}
something like this can be used to wait
make it class property
bool IsDownloadCompleted=false;
Add this in DownloadCompletedEvent
IsDownloadCompleted=true;
and this where you want to stop loop
while(DownloadCompleted!=true)
{
Application.DoEvents();
}
Create some boolean variable, create a delegate and get\set methods for this variable.
Then just in loop made smth like :
while(!isDownLoadCompleted)Thread.Sleep(1024);
You Can use Paralel.ForEach. this loop will wait until all threads done.
check Here for how to use :
http://msdn.microsoft.com/tr-tr/library/dd460720(v=vs.110).aspx
or
http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx

Wanna read a RSS file into an organized ADT in WP8 C#

I'm trying to read a RSS feed from the web, and put it's result into an array.
Unfortuneatly, I've been for 2 days on the web, searching for every solution, without success.
In the example at windows phone developers site :
private void loadFeedButton_Click(object sender, System.Windows.RoutedEventArgs e)
{
WebClient webClient = new WebClient();
webClient.DownloadStringCompleted += new DownloadStringCompletedEventHandler(webClient_DownloadStringCompleted);
webClient.DownloadStringAsync(new System.Uri("http://windowsteamblog.com/windows_phone/b/windowsphone/rss.aspx"));
}
// Event handler which runs after the feed is fully downloaded.
private void webClient_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
if (e.Error != null)
{
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
MessageBox.Show(e.Error.Message);
});
}
else
{
this.State["feed"] = e.Result;
UpdateFeedList(e.Result);
}
}
// This method sets up the feed and binds it to our ListBox.
private void UpdateFeedList(string feedXML)
{
StringReader stringReader = new StringReader(feedXML);
XmlReader xmlReader = XmlReader.Create(stringReader);
SyndicationFeed feed = SyndicationFeed.Load(xmlReader);
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
feedListBox.ItemsSource = feed.Items;
loadFeedButton.Content = "Refresh Feed";
});
}
u can see that the "button_click" event ended, when it continues into the FinishDownload event and then it puts the result into the feedListBox Item Source.
Now, I don't want it.
I wanna put the results into a list or any ADT like this
public static [ADT] Execute (string link)
{ .... }
so that this functions returns me the results from the feed.
Now I was trying anything, threading and XDocument, but it doesn't work.
So if I understand you correctly you want to download the feed and then just have the items in an List, array or similair instead of in the listbox? Then you can simpy add the line:
var feedItems = new List<SyndicationItem>(feed.Items);
I also get the feeling you want it all in one function instead, then you can do it this way using Tasks:
public static Task<List<SyndicationItem>> Execute(string link)
{
WebClient wc = new WebClient();
TaskCompletionSource<List<SyndicationItem>> tcs = new TaskCompletionSource<List<SyndicationItem>>();
wc.DownloadStringCompleted += (s, e) =>
{
if (e.Error == null)
{
StringReader stringReader = new StringReader(e.Result);
XmlReader xmlReader = XmlReader.Create(stringReader);
SyndicationFeed feed = SyndicationFeed.Load(xmlReader);
tcs.SetResult(new List<SyndicationItem>(feed.Items));
}
else
{
tcs.SetResult(new List<SyndicationItem>());
}
};
wc.DownloadStringAsync(new Uri(link, UriKind.Absolute));
return tcs.Task;
}
And then your event handler for the button would be:
private async void loadFeedButton_Click(object sender, System.Windows.RoutedEventArgs e)
{
var rssItems = await Execute("http://windowsteamblog.com/windows_phone/b/windowsphone/rss.aspx");
MessageBox.Show("Number of RSS items: " + rssItems.Count);
}
Note the async and await keywords, you can read more about those in the documentation.

Wait for asynchronous methods in a loop to complete

I have a C# program that currently downloads data from several sites synchronously after which the code does some work on the data I've downloaded. I am trying to move this to do my downloads asynchronously and then process the data I've downloaded. I am having some trouble with this sequencing. Below is a snapshot of code I am using:
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Started URL downloader");
UrlDownloader d = new UrlDownloader();
d.Process();
Console.WriteLine("Finished URL downloader");
Console.ReadLine();
}
}
class UrlDownloader
{
public void Process()
{
List<string> urls = new List<string>() {
"http://www.stackoverflow.com",
"http://www.microsoft.com",
"http://www.apple.com",
"http://www.google.com"
};
foreach (var url in urls)
{
WebClient Wc = new WebClient();
Wc.OpenReadCompleted += new OpenReadCompletedEventHandler(DownloadDataAsync);
Uri varUri = new Uri(url);
Wc.OpenReadAsync(varUri, url);
}
}
void DownloadDataAsync(object sender, OpenReadCompletedEventArgs e)
{
StreamReader k = new StreamReader(e.Result);
string temp = k.ReadToEnd();
PrintWebsiteTitle(temp, e.UserState as string);
}
void PrintWebsiteTitle(string temp, string source)
{
Regex reg = new Regex(#"<title[^>]*>(.*)</title[^>]*>");
string title = reg.Match(temp).Groups[1].Value;
Console.WriteLine(new string('*', 10));
Console.WriteLine("Source: {0}, Title: {1}", source, title);
Console.WriteLine(new string('*', 10));
}
}
Essentially, my problem is this. My output from above is:
Started URL downloader
Finished URL downloader
"Results of d.Process()"
What I want to do is complete the d.Process() method and then return to the "Main" method in my Program class. So, the output I am looking for is:
Started URL downloader
"Results of d.Process()"
Finished URL downloader
My d.Process() method runs asynchronously, but I can't figure out how to wait for all of my processing to complete before returning to my Main method. Any ideas on how to do this in C#4.0? I am not sure how I'd go about 'telling' my Process() method to wait until all it's asynchronous activity is complete before returning to the Main method.
If you are on .NET>=4.0 you can use TPL
Parallel.ForEach(urls, url =>
{
WebClient Wc = new WebClient();
string page = Wc.DownloadString(url);
PrintWebsiteTitle(page);
});
I would also use HtmlAgilityPack to parse the page instead of regex.
void PrintWebsiteTitle(string page)
{
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(page);
Console.WriteLine(doc.DocumentNode.Descendants("title").First().InnerText);
}
I would recommend using WebClient.DownloadDataAsync instead of writing your own. You could then use the Task Parallel Library to wrap the call to DownloadDataAsync in a TaskCompletionSource to get multiple Task objects you can wait on or continue with:
webClient.DownloadDataAsync(myUri);
webClient.DownloadDataCompleted += (s, e) =>
{
tcs.TrySetResult(e.Result);
};
if (wait)
{
tcs.Task.Wait();
Console.WriteLine("got {0} bytes", tcs.Task.Result.Length);
}
else
{
tcs.Task.ContinueWith(t => Console.WriteLine("got {0} bytes", t.Result.Length));
}
To handle error conditions, you can expand the use of the TaskCompletionSource:
webClient.DownloadDataCompleted += (s, e) =>
{
if(e.Error != null) tcs.SetException(e.Error);
else if(e.Cancelled) tcs.SetCanceled();
else tcs.TrySetResult(e.Result);
};
To do similar with multiple tasks:
Task.WaitAll(tcs.Task, tcs2.Task);
or
Task.Factory.ContinueWhenAll(new Task[] {tcs.Task, tcs2.Task}, ts =>
{
/* do something with all the results */
});

WebClient DownloadProgressChangedEventHandler Not Firing

I'm having trouble getting DownloadProgressChangedEventHandler to fire. I understand that, worst case, the event handler should fire every 64Kb. The URL I'm trying to download data from creates 680Kb of XML data on the fly, but the handler does not fire at all.
Here's test code that demonstrates the problem. Unfortunately I cannot share the specific URL as it contains proprietary data.
static void Main(string[] args)
{
Console.WriteLine("Downloading data");
string url = "https://MyUrlThatRespondsWith680KbOfData";
string outputPath = #"C:\Temp\DeleteMe.xml";
using (WebClient webClient = new WebClient())
{
webClient.DownloadProgressChanged += new DownloadProgressChangedEventHandler(webClient_DownloadProgressChanged);
webClient.DownloadFile(url, outputPath);
}
Console.Write("Done");
Console.ReadKey(true);
}
static void webClient_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
Console.WriteLine("Download progress: " + e.BytesReceived);
}
Your code looks ok, but documentation says "This event is raised each time an asynchronous download makes progress" while you are using synchronous version of the download. Switch to use DownloadFileAsync.
My code was structured so that the WebClient was already used on a non-UI thread, so I extended WebClient to allow for a synchronous call to get events. I also extended it to allow for a custom connection timeout (I was calling a web service that could take quite a while to respond). The new method DownloadFileWithEvents internally calls DownloadFileAsync and properly blocks until the appropriate complete event is received.
Here's the code in case it's useful to anyone:
public class MyWebClient : WebClient
{
//time in milliseconds
private int timeout;
public int Timeout
{
get
{
return timeout;
}
set
{
timeout = value;
}
}
public MyWebClient(int timeout = 60000)
{
this.timeout = timeout;
this.DownloadFileCompleted += new System.ComponentModel.AsyncCompletedEventHandler(MyWebClient_DownloadFileCompleted);
}
EventWaitHandle asyncWait = new ManualResetEvent(false);
public void DownloadFileWithEvents(string url, string outputPath)
{
asyncWait.Reset();
Uri uri = new Uri(url);
this.DownloadFileAsync(uri, outputPath);
asyncWait.WaitOne();
}
void MyWebClient_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
asyncWait.Set();
}
protected override WebRequest GetWebRequest(Uri address)
{
var result = base.GetWebRequest(address);
result.Timeout = this.timeout;
return result;
}
}

Categories

Resources