i am asking the general question here. i really don't know what to do here. i have developed a windows phone 8 mobile app with back end as the web service which is a web API service. it has around 4 to 5 screens.
the problem:
when i first load my app and let everything load(not INTERRUPTING the fetching from the webapi operation by going to the second window or the third window or the 4th windows by the application bar and starting another fetching of the records). it works fine.
but if i once let the first loading run and go for the second fetching of records. it creates huge problems. (delay , returning null etc). any idea how to overcome this problem with the second fetching while the first fetching is running. is this a general problem? or only i am having this problem?
this is my code which i am using
private static readonly HttpClient client;
public static Uri ServerBaseUri
{
get { return new Uri("http://169.254.80.80:30134/api/"); }
}
static PhoneClient()
{
client =new HttpClient();
client.MaxResponseContentBufferSize = 256000;
client.Timeout = TimeSpan.FromSeconds(100);
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
}
public async static Task<List<Categories>> GetDefaultCategories()
{
HttpResponseMessage getresponse = await client.GetAsync(ServerBaseUri + "Categorys");
string json = await getresponse.Content.ReadAsStringAsync();
json = json.Replace("<br>", Environment.NewLine);
var categories = JsonConvert.DeserializeObject<List<Categories>>(json);
return categories.ToList();
}
There are two main problems as Vladimir points out in his comment. You need to create a new instance of HttpClient for each request and I'd also recommend not using statics.
public class DataService
{
public HttpClient CreateHttpClient()
{
var client = new HttpClient();
client.MaxResponseContentBufferSize = 256000;
client.Timeout = TimeSpan.FromSeconds(100);
// I'm not sure why you're adding this, I wouldn't
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
return client;
}
public async Task<List<Categories>> GetDefaultCategories()
{
var client = CreateHttpClient();
HttpResponseMessage getresponse = await client.GetAsync(ServerBaseUri + "Categorys");
string json = await getresponse.Content.ReadAsStringAsync();
json = json.Replace("<br>", Environment.NewLine);
var categories = JsonConvert.DeserializeObject<List<Categories>>(json);
return categories.ToList();
}
}
If you absolutely must you statics, and I'm not recommending this technically, but, you can have an instance of this service statically accessible off your app class to get it up and running easily. I'd prefer a dependency injection technique. The important part is that you limit your static instances. If I have any in my code I tend to hang them off of the main App class.
public class App
{
public static DataService DataService { get; set; }
static App()
{
DataService = new DataService();
}
// other app.xaml.cs stuff
}
Then anywhere in your code you could call:
var categories = await App.DataService.GetDefaultCategories();
Related
I'm trying to call a endpoint to get some info in my website but when I run the getAsync it's only works the first time in the current process.
static async Task Main(string[] args)
{
await GetUserInfo("srmilton"); //Working
await GetUserInfo("srmilton"); //Not Working
while (true)
{
var res = await GetUserInfo("srmilton");
}
}
public static async Task<(string,string)> GetUserInfo(string username)
{
string url = "https://www.mywebsite.com/api/user/detail";
var baseAddress = new Uri(url);
using (var handler2 = new HttpClientHandler { UseCookies = false })
using (var client2 = new HttpClient(handler2) { BaseAddress = baseAddress })
{
client2.DefaultRequestHeaders.Clear();
client2.DefaultRequestHeaders.Add("Cookie", "session=zgbEqIjfSC7M7QdTTpHDkpWLt");
client2.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
client2.DefaultRequestHeaders.Add("x-requested-with", "XMLHttpRequest");
client2.DefaultRequestHeaders.Add("referer", "https://www.mywebsite.com/");
var result = await client2.GetAsync("");
string responseString = await result.Content.ReadAsStringAsync();
dynamic jsonresponse = JObject.Parse(responseString);
id = jsonresponse.userInfo.user.id;
sec_id = jsonresponse.userInfo.user.secUid;
return (id, sec_id);
}
}
The first time the fuction GetUserInfo is called it's return the correct json response from the api but the second loop gets stuck in GetAsync. I have already tried .ConfigureAwait(false), .Result and even creating HttpClient just once and reusing but it's aways hang on the second loop.
I don't know what i'm doing wrong, if someone can explain and show the right way to make this works i'll be thankful.
Solved. I was missing a Accept header on my getAsync headers.
client2.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
This question already has answers here:
Test if a website is alive from a C# application
(7 answers)
Closed 2 years ago.
I do realize that there are different wat to check the existence of a website in C# and other languages. However, I have come to understand that some of them don't work as good. For example, this code:
WebClient client = new WebClient();
try
{
string response = client.DownloadString(url);
return true; // Successful -> the website does exist.
}
catch
{
return false; // Unsuccessful -> the website doesn't exist.
}
doesn't detect www.bb.com, although this website clearly does exist. What code should I use to be absolutely on the safe side here?
Given that:
The site you linked expects one of the main browsers to be set in the User-Agent string
HttpClient is more recommended and newer than WebClient
you could do it like this:
using System;
using System.Net.Http;
using System.Threading.Tasks;
namespace resourcechecker
{
class Program
{
static HttpClient client = new HttpClient();
public async static Task<bool> ResourceExists(HttpClient client, string url)
{
HttpResponseMessage response;
response = await client.GetAsync(url);
if (response.IsSuccessStatusCode)
{
return true;
}
else
{
return false;
}
}
static void Main(string[] args)
{
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0");
string url = "http://www.bb.com/";
Console.WriteLine($"It is {ResourceExists(client, url).Result} that {url} exists");
}
}
}
However, this could still give the wrong answer if, for example, the IP address of the testing computer is blocked from accessing that web server
So I am trying to call a api in my C# method and am not getting the response that I would be getting, if I called the api using postman or insomia.
public static async Task GetObject()
{
var httpClient = new HttpClient();
//var t = await httpClient.GetStringAsync("https://ipapi.co/json/");
var y = await httpClient.GetAsync("https://ipapi.co/json/");
var o = await y.Content.ReadAsStringAsync();
var j = y.Content.ReadAsStringAsync();
}
The api is getting my request but its not returning the right result. You should be getting this result. Click on this https://ipapi.co/json/
What am getting is this
{
"error":true,
"reason":"RateLimited",
"message":"Sign up for IP Address Location API # https://ipapi.co"
}
But I don't get it, when am using postman
You need to add the user-agent header.
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1");
Better to Use xNet.dll or Leaf.xNet.dll libraries
Or restsharp one
Google them
I'm also new If ur interesting to contact me
Discord Cyx.#4260
I am developing an C# console application for testing whether a URL is valid or not. It works well for most of URLs. But we found that there are some cases the application always got 404 response from target site but the URLs actually work in the browser. And those URLs also works when I tried them in the tools such as DHC (Dev HTTP Client).
In the beginning, I though that this could be the reason of not adding right headers. But after tried using Fiddler to compose a http request with same headers, it works in Fiddler.
So what's wrong with my code? Is there any bug in .NET HttpClient?
Here are the simplified code of my test application:
class Program
{
static void Main(string[] args)
{
var urlTester = new UrlTester("http://www.hffa.it/short-master-programs/fashion-photography");
Console.WriteLine("Test is started");
Task.WhenAll(urlTester.RunTestAsync());
Console.WriteLine("Test is stoped");
Console.ReadKey();
}
public class UrlTester
{
private HttpClient _httpClient;
private string _url;
public UrlTester(string url)
{
_httpClient = new HttpClient
{
Timeout = TimeSpan.FromMinutes(1)
};
// Add headers
_httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36");
_httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip,deflate,sdch");
_httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
_httpClient.DefaultRequestHeaders.Add("Accept-Language", "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4");
_url = url;
}
public async Task RunTestAsync()
{
var httpRequestMsg = new HttpRequestMessage(HttpMethod.Get, _url);
try
{
using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead))
{
Console.WriteLine("Response: {0}", response.StatusCode);
}
}
catch (HttpRequestException e)
{
Console.WriteLine(e.InnerException.Message);
}
}
}
}
This appears to be an issue with the accepted languages. I got a 200 response when using the following Accept-Language header value
_httpClient.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ru;q=0.4");
p.s. I assume you know in your example _client should read _httpClient in the urlTester constructor or it wont build.
Another possible cause of this problem is if the url you are sending is over approx 2048 bytes long. At that point the content (almost certainly the query string) can become truncated and this in turn means that it may not be matched correctly with a server side route.
Although these urls were processed correctly in the browser, they also failed using the get command in power shell.
This issue was resolved by using a POST with key value pairs instead of using a GET with a long query string.
Is there a way to spoof a web request from C# code so it doesn't look like a bot or spam hitting the site? I am trying to web scrape my website, but keep getting blocked after a certain amount of calls. I want to act like a real browser. I am using this code, from HTML Agility Pack.
var web = new HtmlWeb();
web.UserAgent =
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11";
I do way too much web scraping, but here are the options:
I have a default list of headers I add as all of these are expected from a browser:
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11";
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
wc.Headers[HttpRequestHeader.Accept] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-GB,en-US;q=0.8,en;q=0.6";
wc.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
(WC is my WebClient).
As a further help - here is my webclient class that keeps cookies stored - which is also a massive help:
public class CookieWebClient : WebClient
{
public CookieContainer m_container = new CookieContainer();
public WebProxy proxy = null;
protected override WebRequest GetWebRequest(Uri address)
{
try
{
ServicePointManager.DefaultConnectionLimit = 1000000;
WebRequest request = base.GetWebRequest(address);
request.Proxy = proxy;
HttpWebRequest webRequest = request as HttpWebRequest;
webRequest.Pipelined = true;
webRequest.KeepAlive = true;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
catch
{
return null;
}
}
}
Here is my usual use for it. Add a static copy to your base site class with all your parsing functions you likely have:
protected static CookieWebClient wc = new CookieWebClient();
And call it as such:
public HtmlDocument Download(string url)
{
HtmlDocument hdoc = new HtmlDocument();
HtmlNode.ElementsFlags.Remove("option");
HtmlNode.ElementsFlags.Remove("select");
Stream read = null;
try
{
read = wc.OpenRead(url);
}
catch (ArgumentException)
{
read = wc.OpenRead(HttpHelper.HTTPEncode(url));
}
hdoc.Load(read, true);
return hdoc;
}
The other main reason you may be crashing out is the connection is being closed by the server as you have had an open connection for too long. You can prove this by adding a try catch around the download part as above and if it fails, reset the webclient and try to download again:
HtmlDocument d = new HtmlDocument();
try
{
d = this.Download(prp.PropertyUrl);
}
catch (WebException e)
{
this.Msg(Site.ErrorSeverity.Severe, "Error connecting to " + this.URL + " : Resubmitting..");
wc = new CookieWebClient();
d = this.Download(prp.PropertyUrl);
}
This saves my ass all the time, even if it was the server rejecting you, this can re-jig the lot. Cookies are cleared and your free to roam again. If worse truly comes to worse - add proxy support and get a new proxy applied per 50-ish requests.
That should be more than enough for you to kick your own and any other sites arse.
RATE ME!
Use a regular browser and fiddler (if the developer tools are not up to scratch) and take a look at the request and response headers.
Build up your requests and request headers to match what the browser sends (you can use a couple of different browsers to asses if this makes a difference).
In regards to "getting blocked after a certain amount of calls" - throttle your calls. Only make one call every x seconds. Behave nicely to the site and it will behave nicely to you.
Chances are good that they simply look at the number of calls from your IP address per second and if it passes a threshold, the IP address gets blocked.