Force HttpClient to get new version of source - c#

I'm currently working on something, and it requires me to get a new version of the source each time the function is ran. Here is the code so far.
static class DataManager
{
public static async void CollectData()
{
HttpClient client = new HttpClient();
// Call asynchronous network methods in a try/catch block to handle exceptions
try
{
string responseBody = await client.GetStringAsync("http://www.lipsum.com");
ParseHTML(responseBody);
}
catch (HttpRequestException e)
{
Console.WriteLine("Error: {0}", e.Message);
}
// Need to call dispose on the HttpClient object
// when done using it, so the app doesn't leak resources
client.Dispose();
}
public static void ParseHTML(string _responseString)
{
HtmlAgilityPack.HtmlDocument document = new HtmlAgilityPack.HtmlDocument();
document.LoadHtml(_responseString);
HtmlAgilityPack.HtmlNode contrib = document.DocumentNode.SelectSingleNode("//*[contains(#class,'randomclass')]");
Console.WriteLine(contrib.InnerText.Replace(",", ""));
}
public static void UpdateLabels(string _timer, string _participants)
{
}
}
I'm wondering if there is a way to make this get a new version of the website each time I run the function.
I'm running it by typing
DataManager.CollectData();

Have you tried:
var client = new HttpClient(new WebRequestHandler() {
CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore)
});
try
{
string responseBody = await client.GetStringAsync("http://www.lipsum.com");
ParseHTML(responseBody);
}
catch (HttpRequestException e)
{
Console.WriteLine("Error: {0}", e.Message);
}
Have a look at the HttpRequestCacheLevel enum - there are quite a few options that may help you.

Related

Async function freezes UI thread

I have an async function which still freezes / lags the UI thread for me when I execute it. This is my function calling it.
private void TcpListenerLogic(object sender, string e)
{
Application.Current.Dispatcher.BeginInvoke((Action)async delegate {
try
{
dynamic results = JsonConvert.DeserializeObject<dynamic>(e);
if (results.test_id != null)
{
// Get properties for new anchor
string testInformation = await CommunicationCommands.getJsonFromURL(
"http://" + ServerIP + ":" + ServerPort + "/api/" + results.test_id);
}
}
catch (Exception exception)
{
// Writing some Trace.WriteLine()'s
}
});
}
And this is the async function that freezes my UI Thread
public static async Task<string> getJsonFromURL(string url)
{
try
{
string returnString = null;
using (System.Net.WebClient client = new System.Net.WebClient())
{
returnString = await client.DownloadStringTaskAsync(url);
}
return returnString;
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
return null;
}
}
I already tried to make everything in TcpListenerLogic run in a new Thread:
new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
}).Start();
Which resulted in the whole UI completely freezing. And I tried to make TcpListenerLogic async and await the dispatcher, which also made everything freeze permanently. I also tried to make TcpListenerLogic async and leave the dispatcher. The dispatcher is only there because I normally have some UI code in there, which I left out for my tests.
I have ventured far through the internet, but no BackgroundWorker, ThreadPool or other methods helped me in my endeavour.
If anyone has help for this particular problem, or a resource that would improve my understanding of async functions in C#, I would much appreciate it.
Edit
As requested a deeper insight in how this event handler is called.
I have System.Net.Websocket, which is connected to the Backend API I am working with and triggers an event, everytime he receives new Data. To guarantee the socket listens as longs as it is open, there is a while loop which checks for the client state:
public event EventHandler<string> TcpReceived;
public async void StartListener(string ip, int port, string path)
{
try
{
using (client = new ClientWebSocket())
{
try
{ // Connect to backend
Uri serverUri = new Uri("ws://" + ip + ":" + port.ToString() + path );
await client.ConnectAsync(serverUri, CancellationToken.None);
}
catch (Exception ex)
{
BackendSettings.IsConnected = false;
Debug.WriteLine("Error connecting TCP Socket: " + ex.ToString());
}
state = client.State;
// Grab packages send in backend
while (client.State == WebSocketState.Open || client.State == WebSocketState.CloseSent)
{
try
{
// **Just formatting the received data until here and writing it into the "message" variable**//
TcpReceived(this, message);
// Close connection on command
if (result.MessageType == WebSocketMessageType.Close)
{
Debug.WriteLine("Closing TCP Socket.");
shouldstayclosed = true;
await client.CloseAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None);
break;
}
state = client.State;
}
catch
{
BackendSettings.IsConnected = false;
state = client.State;
}
}
state = client.State;
}
}
catch (Exception ex)
{
// Some error messages and settings handling
}
}
The Event has a handler attached:
TcpReceived += TcpListener_TcpReceived;
And this is the Handler, which calls the previously seen "TcpListenereLogic".
private void TcpListener_TcpReceived(object sender, string e)
{
TcpListenerLogic(sender, e);
//App.Current.Dispatcher.BeginInvoke(new Action(() => {
// TcpListenerLogic(sender, e);
//}));
//new Thread(() =>
//{
// Thread.CurrentThread.IsBackground = true;
// TcpListenerLogic(sender, e);
//}).Start();
}
I previously had the "TcpListenereLogic" as the handler, but I wanted to try different methods to call it. I also left in the commented out part, to show how the call of "TcpListenereLogic" looked already. All my attempts were with all mentioned setups and sadly lead to nothing.
Thank you very much #TheodorZoulias for helping me to find the solution to my problem.
It turns out it wasn't the async function itself, but rather how often it gets called. It got called roughly ~120 times every second.
My solution starts by calling the Listener method over a new Thread:
new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
MainWindow.tcpListener.StartListener(ip, portNumber, "/api/");
}).Start();
To limit the amount of calls that happen every second I added a dispatcher timer, that resets a bool after it has been used for a call, by my Event.
readonly System.Windows.Threading.DispatcherTimer packageIntervallTimer =
new System.Windows.Threading.DispatcherTimer();
bool readyForNewPackage = true;
private void ReadyForPackage(object sender, EventArgs e)
{
readyForNewPackage = true;
}
public async void StartListener(string ip, int port, string path)
{
packageIntervallTimer.Interval = TimeSpan.FromMilliseconds(50);
packageIntervallTimer.Tick += (s, e) => { Task.Run(() => ReadyForPackage(s, e)); };
packageIntervallTimer.Start();
Then I wrapped everything inside the while loop into an if condition based on the bool, the most important part was to have my "event EventHandler TcpReceived" in there:
// Grab packages sent in backend
while (client.State == WebSocketState.Open || client.State == WebSocketState.CloseSent)
{
if (readyForNewPackage == true)
{
readyForNewPackage = false;
try
{
....
TcpReceived(this, message);
....
}
catch
{
...
}
}
}
I added my TcpListenerLogic to the Eventhandler:
TcpReceived += TcpListenerLogic;
And my TcpListenerLogic now looked like this (names have been changed):
private async void TcpListenerLogic(object sender, string e)
{
try
{
dynamic results = JsonConvert.DeserializeObject<dynamic>(e);
if (results.test_id != null)
{
string testID = "";
if (results.test_id is JValue jValueTestId)
{
testID = jValueTestId.Value.ToString();
}
else if (results.test_id is string)
{
testID = results.test_id;
}
// Get properties for new object
string information = await CommunicationCommands.getJsonFromURL(
"http://" + ServerIP + ":" + ServerPort + "/api/" + testID );
if (information != null)
{
await App.Current.Dispatcher.BeginInvoke(new Action(() =>
{
// Create object out of the json string
TestStatus testStatus = new TestStatus();
testStatus.Deserialize(information);
if (CommunicationCommands.isNameAlreadyInCollection(testStatus.name) == false)
{
// Add new object to the list
CommunicationCommands.allFoundTests.Add(testStatus);
}
}));
{
}
catch (Exception exception)
{
....
}
}
Adding a new Thread to execute any step results in problems, so keep in mind that all this uses the thread created at the beginning for "StartListener"

Windows Service memory usage steady increase

I have simple Windows service which sends HTTP requests on few timer's. I installed Release version of this service and after few days running of it I noticed memory usage increase from ~5MB to ~32MB. This made me worried, I rarely do anything for desktop so I really lack knowledge to figure out where leak is happening...
Maybe log4net object is growing?
Here I initialize my service and it's configuration:
private static readonly log4net.ILog logger = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
private Historian historian;
private EndpointsConfiguration endpointsConfiguration;
private readonly string endpointsJson;
public Service()
{
InitializeComponent();
historian = new Historian(ConfigurationManager.ConnectionStrings["Historian"].ConnectionString);
try
{
string path = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase).Substring(6);
endpointsJson = File.ReadAllText($#"{path}\EndpointsConfiguration.json");
var jsonOptions = new JsonSerializerOptions
{
Converters =
{
new JsonStringEnumConverter(JsonNamingPolicy.CamelCase)
}
};
endpointsConfiguration = JsonSerializer.Deserialize<EndpointsConfiguration>(endpointsJson, jsonOptions);
}
catch (Exception ex)
{
logger.Fatal("Nepavyko nuskaityti konfigūracijos", ex);
return;
}
}
OnStart method:
protected override void OnStart(string[] args)
{
logger.Info("Servisas startavo");
try
{
historian.Ping();
logger.Info("Pavyko prisijungti prie Historian duomenų bazės");
foreach (var endpoint in endpointsConfiguration.Endpoints)
{
var timer = new TimerWithContext(endpoint, endpoint.Interval)
{
Enabled = true
};
timer.Elapsed += async (sender, e) => await Timer_Elapsed(sender, e);
}
logger.Info("Suformuotos užduočių gijos");
}
catch (Exception ex)
{
logger.Fatal("Nepavyko prisijungti prie Historian duomenų bazės", ex);
OnStop();
}
}
Timer Elapsed (primary work of this service):
private async Task Timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
Endpoint endpoint = ((TimerWithContext)sender).Endpoint;
var endpointTags = endpoint.Tags;
try
{
historian.RetrieveLiveValues(ref endpointTags);
}
catch (Exception ex)
{
logger.Fatal($"Nepavyko nuskaityti naujausių tag'o reikšmių, Gavėjas:${endpoint.UrlAddress}", ex);
}
finally
{
using (HttpClient httpClient = new HttpClient())
{
if (endpoint.AuthScheme != null && endpoint.AuthKey != null)
{
httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue(endpoint.AuthScheme, endpoint.AuthKey);
}
object data = new { };
switch (endpoint.SerializationOption)
{
case SerializationOption.PECustom:
data = new DeviceComponents
{
SerialNr = endpoint.SerialNr,
MeterCode = endpoint.MeterCode,
ReadDate = $"{DateTime.Now:yyyy-MM-dd HH:mm:ss}",
DevCompValues = endpointTags.ToDictionary(tag => tag.DestinationName, tag => tag.Value.ToString())
};
break;
default:
data = endpointTags.ToDictionary(tag => tag.DestinationName, tag => tag.Value.ToString());
break;
}
var content = new StringContent(JsonSerializer.Serialize(data), Encoding.UTF8, "application/json");
try
{
var response = await httpClient.PostAsync(endpoint.UrlAddress, content);
response.EnsureSuccessStatusCode();
logger.Info($"Duomenys perduoti gavėjui:{endpoint.Name}");
}
catch (Exception ex)
{
logger.Warn($"Nepavyko perduoti duomenų šaltiniui: {endpoint.Name}; Adresas{endpoint.UrlAddress}; Duomenų kiekis:{endpoint.Tags.Count}", ex);
}
}
}
}

How to avoid unreasonable long response time by using .net webclient with proxy?

I am trying to just request the webpage by using webclient DownloadString method with proxy.
The following is my code:
using (WebClient wc = new WebClient())
{
try
{
wc.Encoding = Encoding.UTF8;
wc.Proxy = new WebProxy(myProxy);
string result = wc.DownloadString(url);
}
catch (Exception ex)
{
//log exception
}
}
I have few proxies and almost all of them work well with above code.
However, sometimes the response return time is very very long (over an hour) and I believe it's due to my proxy slow issue.
But I don't understand that why it didn't throw exception since the webclient class should have an default timeout (I search default timeout that it should be 100 seconds).
Therefore I want to ask how to avoid this case?
Thanks everyone.
It will be complicated as you need two threads - one for download and one for cancellation on timeout.
wc.Encoding = Encoding.UTF8;
wc.Proxy = new WebProxy(myProxy);
string result = null;
var waitCancel = new CancellationTokenSource();
wc.DownloadStringCompleted += (sender, e) =>
{
if (e.Cancelled) return;
waitCancel.Cancel();
result = e.Result;
};
wc.DownloadStringAsync(url);
waitCancel.Token.WaitHandle.WaitOne(30 * 1000);
if (waitCancel.IsCancellationRequested == false)
{
wc.CancelAsync();
}
Console.WriteLine("Result: " + result);
First you need to make use of the "new" HttpClient for .Net, below I will illustrate in two steps
HttpClient has handlers you can place in a chain and then control how the whole client behaves, so in this case we will add a progress handler that will dictate behavior in regard to the speed of the transfer
if the transfer is too slow then cancel it and start a new one with different proxy in ur case ...
Create the progress handler (to tell us how the progress is going .. If too slow then throw an exception)
Type :
ProgressMessageHandler
"https://msdn.microsoft.com/en-us/library/system.net.http.handlers.progressmessagehandler(v=vs.118).aspx"
Note : Package (System.Net.Http.Formatting.Extension) required
We will wrap it with an own object to erase testing and configuring to certain expected transfer rate
Code :
public class MyOwnProgressHandlerContainer
{
private readonly long _expectedBytesTransferred;
private long _lastRecoredBytesTransferred = 0;
public MyOwnProgressHandlerContainer(long expectedBytesTransferred)
{
_expectedBytesTransferred = expectedBytesTransferred;
}
public void ReceivedProgressHandler(object sender, HttpProgressEventArgs e)
{
// you can uses e.ProgressPercentage but this is calculated based on content length
// http header which is very much ignored nowadays
if (_lastRecoredBytesTransferred != 0 && e.BytesTransferred < (_lastRecoredBytesTransferred + _expectedBytesTransferred))
throw new Exception("Too Slow, Abort !");
_lastRecoredBytesTransferred = e.BytesTransferred;
}
public void SendProgressHandler(object sender, HttpProgressEventArgs e)
{
// write stuff to handle here when sending data (mainly for post or uploads)
Console.WriteLine("Sent data ...");
}
}
Create the HttpClient and inject the handlers that we wrapped :)
Type :
HttpClient
https://msdn.microsoft.com/en-us/library/system.net.http.httpclient(v=vs.118).aspx
Code :
// set expected rate
int expectedTransferRate = 25000; // AKA 2.5Mbps
// create own handler instance
MyOwnProgressHandlerContainer myHandler = new MyOwnProgressHandlerContainer(expectedTransferRate);
// create "ProgressMessageHandler"
ProgressMessageHandler progresshandler = new ProgressMessageHandler();
// Hookup event (send)
progresshandler.HttpSendProgress += myHandler.SendProgressHandler; // these are delegates so they can be a part of a stat-full class
// Hookup event (Receive)
progresshandler.HttpReceiveProgress += myHandler.ReceivedProgressHandler; // these are delegates so they can be a part of a stat-full class
// Create proxy handler
HttpClientHandler httpClientProxyHandler =
new HttpClientHandler
{
Proxy = new WebProxy("http://localhost:8888", false),
UseProxy = true
};
// Create client from factory with progress and "proxy" in your case
using (HttpClient httpClient = HttpClientFactory.Create(progresshandler, httpClientProxyHandler))
{
try
{
string downloadResult =
httpClient
// get result (progress handlers are notified based on sent / received data)
.GetAsync("https://httpbin.org/image/svg")
// could be a stream to read file content
.Result.Content.ReadAsStringAsync().Result;
Console.WriteLine(downloadResult);
}
catch (Exception)
{
// inspected if the exception is the same as the on u throw in MyOwnProgressHandlerContainer
throw;
}
}
WebClient doesnt have Timeout. You must extend it like this:
using System;
using System.Net;
namespace ConsoleApplication1
{
internal class Program
{
private static void Main(string[] args)
{
using (var webClient = new WebClientTimeOut())
{
webClient.Encoding = Encoding.UTF8;
webClient.Proxy = new WebProxy(myProxy);
try
{
var response = webClient.DownloadString("http://www.go1ogle.com");
Console.WriteLine(response);
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
}
}
public class WebClientTimeOut : WebClient
{
protected override WebRequest GetWebRequest(Uri uri)
{
var w = base.GetWebRequest(uri);
w.Timeout = 1000; // Timeout 1 second
return w;
}
}
}

How to use deserialized object on navigation to another page?

So basically I have a button. when I click it, two things must occur.
1) a web request to get the data
2) navigate to the other page and populate the data
the problem is that when the app navigates to page2, App.mydirectories gives a nullreferenceException...
How can I make sure App.mydirectories isn't null and wait before populating data to the new page.
private void Button_Click(object sender, RoutedEventArgs e)
{
makeEventwebRequest(number.Text,date.Text);
NavigationService.Navigate(new Uri("/page2.xaml", UriKind.Relative));
}
public void makeEventwebRequest(string numb, string date)
{
string requesturi = string.Format(baseUri, numb, date);
try
{
WebClient client = new WebClient();
client.DownloadStringCompleted +=
new DownloadStringCompletedEventHandler(directories_DownloadStringCallback);
client.DownloadStringAsync(new Uri(requesturi));
}
catch (Exception e)
{
}
}
private void directories_DownloadStringCallback(object sender, DownloadStringCompletedEventArgs e)
{
App.mydirectories = JsonConvert.DeserializeObject<directoriescs>(e.Result);
}
This error occurs because your code to do the WebRequest is async. When you navigate to page2.xaml your data isn't downloaded yet.
This is an example on how you could do your code:
private async void Button_Click(object sender, RoutedEventArgs e)
{
await makeEventwebRequest(number.Text,date.Text);
NavigationService.Navigate(new Uri("/page2.xaml", UriKind.Relative));
}
public async void makeEventwebRequest(string numb, string date)
{
string requesturi = string.Format(baseUri, numb, date);
var request = (HttpWebRequest)WebRequest.Create(requesturi );
var result = await GetHttpResponse(request);
App.mydirectories = JsonConvert.DeserializeObject<directoriescs>(result);
}
// Method helper to Http async request
public static async Task<String> GetHttpResponse(HttpWebRequest request)
{
String received = null;
try
{
using (var response = (HttpWebResponse)(await Task<WebResponse>.Factory.FromAsync(request.BeginGetResponse, request.EndGetResponse, null)))
{
using (var responseStream = response.GetResponseStream())
{
using (var sr = new StreamReader(responseStream))
{
received = await sr.ReadToEndAsync();
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine(ex);
}
return received;
}
You should read an article on async/await methods like http://msdn.microsoft.com/en-us/library/hh191443.aspx so you can better understand what is different from your code to mine.
The baseline is that you navigated to page2 while the webrequest was still being made, while in the code I posted above, the execution waits for the web request to complete and then navigates to page2.

WebException not caught in try/catch

I saw several posts with similar problem but no solution works :/
I debug a windows service by using a console application. It executes tasks on website and must be able to collect http code status for create logs. As you can see, sensitive code is in try/catch.
When I debug (F5), I have a WebException that is not caught. When I run (CTRL + F5), the exception's message is write in my console and stops my program.
This is my code :
public partial class Schedulor : ServiceBase
{
void RunTasks()
{
schedulor.Start();
List<Task> task = new List<Task>();
foreach (TaskPlanner tp in listTp)
{
if (tp.CountDown == 0 && tp.IsRunning == false)
{
// Initialisation lors de GetTasks()
tp.IsRunning = true;
try
{
task.Add(Task.Factory.StartNew(() => tr = tp.ExecuteBot.Execute())); // WEBEXECPTION HERE (cannot find 404)
}
catch (Exception e)
{
if (e is WebException)
{
// treatment
}
}
}
}
Task.WaitAll(task.ToArray());
CreateLogs();
}
}
public class Bot : IBot
{
public TaskResult Execute()
{
TaskResult tr = new TaskResult();
int codeResponse, timeout;
string credentials;
try
{
WebRequest wrSettings = WebRequest.Create(settings.Url);
// treatment
}
catch (Exception e)
{
//Console.WriteLine(e.Message);
if (e is WebException)
{
var code = ((HttpWebResponse)((WebException)e).Response).StatusCode;
if ((int)code != settings.HttpResponse)
{
tr.MyResult = TaskResult.Result.nok;
goto next;
}
else tr.MyResult = TaskResult.Result.ok;
}
}
next:
return tr;
}
}
I do not understand why my catch does not work. I need to treat this information because the task can test if a website return 404 or anything else.
Thanks in advance
EDIT : -----------
I reduce code as it requests because deleted code does not the real problem
You should catch that exception in task. Add another method, and create your tasks similar to:
task.Add(Task.Factory.StartNew(() => Process(tp)));
void Process(TaskPlanner tp)
{
try
{
tp.ExecuteBot.Execute();
}
catch (WebException wex)
{
}
}

Categories

Resources