Develop Reference

How to get json from Secured API

I did an api, for the exemple i will call it: https://testapp.azurewebsites.net.
I did the Authentication / Authorization in Azure for google Facebook and Miscrosoft account.
Then i want to consume it in xamarin for my Android/iOS app, so i did the login button etc but when I'm authentify i can't get the json of my api URL: "https://testapp.azurewebsites.net/api/Test/allCoordinates".
It work's perfectly in my browser and in postman... This is my code in C#:
var requestCoord = new OAuth2Request("GET", new Uri(URL),null , e.Account);
var responseCoord = await requestCoord.GetResponseAsync(); //it works for google userinfo but not for my Api...
string coordJson = await responseCoord.GetResponseTextAsync();
var mapTest = JsonConvert.DeserializeObject<List<CustomPin>>(coordJson);
In Postman it works and i can see this code for C# from postman:
var client = new RestClient("https://testapp.azurewebsites.net/api/Test/allCoordinates?access_token=ya29.a0AfH6SMCEGy4tP_zngNEhAcpf31d3O_ZYl7NE9QJjbKrW0KPh-dC7PjNmz-KOCbkySRtuwDJdg2ckhiTaTdIEsONxVhFhK3NpnUk9iITyCB1BnwpWJwNEEivxg0pL93UPP9r4UYf1dHEiTVd63eydfV7HoKlxExMFtS8");
client.Timeout = -1;
var request = new RestRequest(Method.GET);
request.AddHeader("User-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36");
request.AddHeader("Cookie", "ARRAffinity=d2e047f134af60dd8e0802593ad5206002e99e56a6231fee0e85747cfa96ea6f; AppServiceAuthSession=Dh0GnGQjaNoBXKv8r4lM9BoJkAA1UFSLuoDDAVP1qGrPP3ICauM1Glsb+Q7NhU+4m+IuPh5ZqGv2bzU6FtvEqri4Io88RuP6ZzKPayXSJKn4WbkzteU59if76yVY/KSjmwjbdUTC47yO+XO2snKygYlGZ9+pVlgaF/UdmW6OLWDlqPvJ069oSXkkZb/gGV5m6dHzYvfn3PcJ4HJmfEPQDclsRvRYUmpIY11hWcRUiiVx26o/SE+IaytRfWxkGk4g/thMFW3IOFtw09DdGXma/Qik8ANybClwXZ7G/3i1VyHQLM9TnU3UGcjtArLUFVj4T3jNkdaVioxtNQWJcDvwN54OL24eNFMM4Ov7Rbo7t2QtQrW73KxOrG/RyJHvBTHTyhjmAw6Hb7wg7VwcJvpKwcJKFBWH5ntvouFhj/DmCrzBuG/Cz6K+81ocEnHBLsHcx9qHrEBXCU3FlMQbogDcRRo1om78IwK+OxKoY+CzDWAJW3taJLl+jVO6QgFtbyqZKErzxEX1jeVcHTWTBdTImYaiA6zs1KKCSgo+rR3G0GWxvyWt9XCZwZD/5E+MYK3pxWFduKmmsEjSYrCgQ7Yhwe2bQg2bvX2HPScfo+yKVoIzQHNArqDr2NVTaWRUt2zN3GoLzSDxe5YgDjHXyo0ES6mEbEKsKy4dYDD7uRS/rdHRTTHUdih5i169sHvJlj0UFyaU8MV+J/dxbuMmNysqOmzVUU18oWQntE48RN35/Js=");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);
But how to find the cookie data in Xamarin ? and call this url ?
If you have any idea to make this Http Get from protected webapi i will help me so much !!!
Thanks a lot
If you need more precision let me know;)

How can I call a web api in c# using HttpClient?

So I am trying to call a api in my C# method and am not getting the response that I would be getting, if I called the api using postman or insomia.
public static async Task GetObject()
{
var httpClient = new HttpClient();
//var t = await httpClient.GetStringAsync("https://ipapi.co/json/");
var y = await httpClient.GetAsync("https://ipapi.co/json/");
var o = await y.Content.ReadAsStringAsync();
var j = y.Content.ReadAsStringAsync();
}
The api is getting my request but its not returning the right result. You should be getting this result. Click on this https://ipapi.co/json/
What am getting is this
{
"error":true,
"reason":"RateLimited",
"message":"Sign up for IP Address Location API # https://ipapi.co"
}
But I don't get it, when am using postman

You need to add the user-agent header.
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1");

Better to Use xNet.dll or Leaf.xNet.dll libraries
Or restsharp one
Google them
I'm also new If ur interesting to contact me
Discord Cyx.#4260

C# HttpClient.SendAsync always returns 404 but URL works in browser

I am developing an C# console application for testing whether a URL is valid or not. It works well for most of URLs. But we found that there are some cases the application always got 404 response from target site but the URLs actually work in the browser. And those URLs also works when I tried them in the tools such as DHC (Dev HTTP Client).
In the beginning, I though that this could be the reason of not adding right headers. But after tried using Fiddler to compose a http request with same headers, it works in Fiddler.
So what's wrong with my code? Is there any bug in .NET HttpClient?
Here are the simplified code of my test application:
class Program
{
static void Main(string[] args)
{
var urlTester = new UrlTester("http://www.hffa.it/short-master-programs/fashion-photography");
Console.WriteLine("Test is started");
Task.WhenAll(urlTester.RunTestAsync());
Console.WriteLine("Test is stoped");
Console.ReadKey();
}
public class UrlTester
{
private HttpClient _httpClient;
private string _url;
public UrlTester(string url)
{
_httpClient = new HttpClient
{
Timeout = TimeSpan.FromMinutes(1)
};
// Add headers
_httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36");
_httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip,deflate,sdch");
_httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
_httpClient.DefaultRequestHeaders.Add("Accept-Language", "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4");
_url = url;
}
public async Task RunTestAsync()
{
var httpRequestMsg = new HttpRequestMessage(HttpMethod.Get, _url);
try
{
using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead))
{
Console.WriteLine("Response: {0}", response.StatusCode);
}
}
catch (HttpRequestException e)
{
Console.WriteLine(e.InnerException.Message);
}
}
}
}

This appears to be an issue with the accepted languages. I got a 200 response when using the following Accept-Language header value
_httpClient.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ru;q=0.4");
p.s. I assume you know in your example _client should read _httpClient in the urlTester constructor or it wont build.

Another possible cause of this problem is if the url you are sending is over approx 2048 bytes long. At that point the content (almost certainly the query string) can become truncated and this in turn means that it may not be matched correctly with a server side route.
Although these urls were processed correctly in the browser, they also failed using the get command in power shell.
This issue was resolved by using a POST with key value pairs instead of using a GET with a long query string.

windows phone 8 (connectivity with web api) issue

i am asking the general question here. i really don't know what to do here. i have developed a windows phone 8 mobile app with back end as the web service which is a web API service. it has around 4 to 5 screens.
the problem:
when i first load my app and let everything load(not INTERRUPTING the fetching from the webapi operation by going to the second window or the third window or the 4th windows by the application bar and starting another fetching of the records). it works fine.
but if i once let the first loading run and go for the second fetching of records. it creates huge problems. (delay , returning null etc). any idea how to overcome this problem with the second fetching while the first fetching is running. is this a general problem? or only i am having this problem?
this is my code which i am using
private static readonly HttpClient client;
public static Uri ServerBaseUri
{
get { return new Uri("http://169.254.80.80:30134/api/"); }
}
static PhoneClient()
{
client =new HttpClient();
client.MaxResponseContentBufferSize = 256000;
client.Timeout = TimeSpan.FromSeconds(100);
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
}
public async static Task<List<Categories>> GetDefaultCategories()
{
HttpResponseMessage getresponse = await client.GetAsync(ServerBaseUri + "Categorys");
string json = await getresponse.Content.ReadAsStringAsync();
json = json.Replace("<br>", Environment.NewLine);
var categories = JsonConvert.DeserializeObject<List<Categories>>(json);
return categories.ToList();
}

There are two main problems as Vladimir points out in his comment. You need to create a new instance of HttpClient for each request and I'd also recommend not using statics.
public class DataService
{
public HttpClient CreateHttpClient()
{
var client = new HttpClient();
client.MaxResponseContentBufferSize = 256000;
client.Timeout = TimeSpan.FromSeconds(100);
// I'm not sure why you're adding this, I wouldn't
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
return client;
}
public async Task<List<Categories>> GetDefaultCategories()
{
var client = CreateHttpClient();
HttpResponseMessage getresponse = await client.GetAsync(ServerBaseUri + "Categorys");
string json = await getresponse.Content.ReadAsStringAsync();
json = json.Replace("<br>", Environment.NewLine);
var categories = JsonConvert.DeserializeObject<List<Categories>>(json);
return categories.ToList();
}
}
If you absolutely must you statics, and I'm not recommending this technically, but, you can have an instance of this service statically accessible off your app class to get it up and running easily. I'd prefer a dependency injection technique. The important part is that you limit your static instances. If I have any in my code I tend to hang them off of the main App class.
public class App
{
public static DataService DataService { get; set; }
static App()
{
DataService = new DataService();
}
// other app.xaml.cs stuff
}
Then anywhere in your code you could call:
var categories = await App.DataService.GetDefaultCategories();

Html Agility Pack, Web Scraping, and spoofing in C#

Is there a way to spoof a web request from C# code so it doesn't look like a bot or spam hitting the site? I am trying to web scrape my website, but keep getting blocked after a certain amount of calls. I want to act like a real browser. I am using this code, from HTML Agility Pack.
var web = new HtmlWeb();
web.UserAgent =
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11";

I do way too much web scraping, but here are the options:
I have a default list of headers I add as all of these are expected from a browser:
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11";
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
wc.Headers[HttpRequestHeader.Accept] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-GB,en-US;q=0.8,en;q=0.6";
wc.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
(WC is my WebClient).
As a further help - here is my webclient class that keeps cookies stored - which is also a massive help:
public class CookieWebClient : WebClient
{
public CookieContainer m_container = new CookieContainer();
public WebProxy proxy = null;
protected override WebRequest GetWebRequest(Uri address)
{
try
{
ServicePointManager.DefaultConnectionLimit = 1000000;
WebRequest request = base.GetWebRequest(address);
request.Proxy = proxy;
HttpWebRequest webRequest = request as HttpWebRequest;
webRequest.Pipelined = true;
webRequest.KeepAlive = true;
if (webRequest != null)
{
webRequest.CookieContainer = m_container;
}
return request;
}
catch
{
return null;
}
}
}
Here is my usual use for it. Add a static copy to your base site class with all your parsing functions you likely have:
protected static CookieWebClient wc = new CookieWebClient();
And call it as such:
public HtmlDocument Download(string url)
{
HtmlDocument hdoc = new HtmlDocument();
HtmlNode.ElementsFlags.Remove("option");
HtmlNode.ElementsFlags.Remove("select");
Stream read = null;
try
{
read = wc.OpenRead(url);
}
catch (ArgumentException)
{
read = wc.OpenRead(HttpHelper.HTTPEncode(url));
}
hdoc.Load(read, true);
return hdoc;
}
The other main reason you may be crashing out is the connection is being closed by the server as you have had an open connection for too long. You can prove this by adding a try catch around the download part as above and if it fails, reset the webclient and try to download again:
HtmlDocument d = new HtmlDocument();
try
{
d = this.Download(prp.PropertyUrl);
}
catch (WebException e)
{
this.Msg(Site.ErrorSeverity.Severe, "Error connecting to " + this.URL + " : Resubmitting..");
wc = new CookieWebClient();
d = this.Download(prp.PropertyUrl);
}
This saves my ass all the time, even if it was the server rejecting you, this can re-jig the lot. Cookies are cleared and your free to roam again. If worse truly comes to worse - add proxy support and get a new proxy applied per 50-ish requests.
That should be more than enough for you to kick your own and any other sites arse.
RATE ME!

Use a regular browser and fiddler (if the developer tools are not up to scratch) and take a look at the request and response headers.
Build up your requests and request headers to match what the browser sends (you can use a couple of different browsers to asses if this makes a difference).
In regards to "getting blocked after a certain amount of calls" - throttle your calls. Only make one call every x seconds. Behave nicely to the site and it will behave nicely to you.
Chances are good that they simply look at the number of calls from your IP address per second and if it passes a threshold, the IP address gets blocked.