Trying to use the HttpClient to get a json response from an API but keep getting html response. In the browser and in Postman I get the result in json just by typing in the url. When using RestSharp I also get the response in json. What do I need to add to get the response in json? The variable responseString in a html string, not a json string.
I use .net core 3.1.
Here's the code:
class Program
{
static async Task Main(string[] args)
{
var response = await GetResponse();
System.Console.ReadKey();
}
public static async Task<string> GetResponse()
{
var client = new HttpClient();
client.BaseAddress = new Uri("https://musicbrainz.org/ws/2/");
client.DefaultRequestHeaders.Add("Accept",
"application/json");
using var response = await client.GetAsync((
"/artist/5b11f4ce-a62d-471e-81fc-a69a8278c7da?fmt=json&inc=url-rels+release-groups"));
response.EnsureSuccessStatusCode();
var responseString = await response.Content.ReadAsStringAsync();
return responseString;
}
}
I think the API is looking for a User-Agent.
Try to Add
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36")
As a side note, you might want to consider declaring your HttpClient as static. It is recommended in its documentation. Also if you want json, you need to use DeserializeObject method:
var result = JsonConvert.DeserializeObject<RootObject>(responseString);
You just need to make sure that you have already installed the Newtonsoft.Json from Nuget packages and also added the following to your using directives:
using Newtonsoft.Json;
Related
I'm trying to call a endpoint to get some info in my website but when I run the getAsync it's only works the first time in the current process.
static async Task Main(string[] args)
{
await GetUserInfo("srmilton"); //Working
await GetUserInfo("srmilton"); //Not Working
while (true)
{
var res = await GetUserInfo("srmilton");
}
}
public static async Task<(string,string)> GetUserInfo(string username)
{
string url = "https://www.mywebsite.com/api/user/detail";
var baseAddress = new Uri(url);
using (var handler2 = new HttpClientHandler { UseCookies = false })
using (var client2 = new HttpClient(handler2) { BaseAddress = baseAddress })
{
client2.DefaultRequestHeaders.Clear();
client2.DefaultRequestHeaders.Add("Cookie", "session=zgbEqIjfSC7M7QdTTpHDkpWLt");
client2.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
client2.DefaultRequestHeaders.Add("x-requested-with", "XMLHttpRequest");
client2.DefaultRequestHeaders.Add("referer", "https://www.mywebsite.com/");
var result = await client2.GetAsync("");
string responseString = await result.Content.ReadAsStringAsync();
dynamic jsonresponse = JObject.Parse(responseString);
id = jsonresponse.userInfo.user.id;
sec_id = jsonresponse.userInfo.user.secUid;
return (id, sec_id);
}
}
The first time the fuction GetUserInfo is called it's return the correct json response from the api but the second loop gets stuck in GetAsync. I have already tried .ConfigureAwait(false), .Result and even creating HttpClient just once and reusing but it's aways hang on the second loop.
I don't know what i'm doing wrong, if someone can explain and show the right way to make this works i'll be thankful.
Solved. I was missing a Accept header on my getAsync headers.
client2.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
I'm using HttpRequestMessage from HttpClient in a couple of methods and currently, I'm repeating the following piece of code all over my code:
This code was converted by https://curl.olsh.me/ so I'm not sure if best practices were used here.
// using System.Net.Http;
using (var httpClient = new HttpClient(handler))
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://www.url.com/"))
{
request.Headers.TryAddWithoutValidation("authority", "www.url.com");
request.Headers.TryAddWithoutValidation("pragma", "no-cache");
request.Headers.TryAddWithoutValidation("cache-control", "no-cache");
request.Headers.TryAddWithoutValidation("dnt", "1");
request.Headers.TryAddWithoutValidation("x-requested-with", "XMLHttpRequest");
request.Headers.TryAddWithoutValidation("x-odesk-csrf-token", "19b91748869456a4ae700ffb69077745");
request.Headers.TryAddWithoutValidation("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36");
request.Headers.TryAddWithoutValidation("accept", "*/*");
request.Headers.TryAddWithoutValidation("origin", "https://www.url.com");
request.Headers.TryAddWithoutValidation("sec-fetch-site", "same-origin");
request.Headers.TryAddWithoutValidation("sec-fetch-mode", "cors");
request.Headers.TryAddWithoutValidation("sec-fetch-dest", "empty");
request.Headers.TryAddWithoutValidation("referer", "https://www.url.com/");
request.Headers.TryAddWithoutValidation("accept-language", "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,fr-FR;q=0.6,fr;q=0.5");
request.Headers.TryAddWithoutValidation("cookie", "G_AUTHUSER_H=1; AccountSecurity_cat=fc4d14f1.oauth2v2_812293");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
var response = await httpClient.SendAsync(request);
IEnumerable<string> cookies = new List<string>();
response.Headers.TryGetValues("Set-Cookie", out cookies);
I assume you are trying to understand how to extract the code in question into a helper method given its use of using blocks. If not, please clarify and I will adjust my answer. Here is what I would do in this case:
Task<HttpResponseMessage> PostAsyncWithHeaders(Uri uri)
{
var response = new Task<HttpResponseMessage>();
using (var httpClient = new HttpClient(handler))
using (var request = new HttpRequestMessage(HttpMethod.Post, uri))
{
//Add your headers as before
response = await httpClient.SendAsync(request);
}
return response;
}
You can then call the method from anywhere it is accessible like this:
response = await PostAsyncWithHeaders(new Uri("https://www.url.com/"));
IEnumerable<string> cookies = new List<string>();
response.Headers.TryGetValues("Set-Cookie", out cookies);
//Presumably consume cookies (yum!)
I made this test in .NET Core 3 for reference.
If you want to be able to reuse your method throughout the code-base, you can try something like the following:
static async Task Main(string[] args)
{
var client = new HttpClient();
// .NET core setting the content type.
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json; charset=utf-8");
// this is wrapped in the using statement.
using var requestMessage = GetRequestMessage("https://www.url.com/", HttpMethod.Post);
await client.SendAsync(requestMessage);
}
// Get the request message for reuse. You can then reuse this for maybe different end-points and method types.
// the "referer" or "origin" values can be passed in as parameters too.
static HttpRequestMessage GetRequestMessage(string url, HttpMethod method)
{
var request = new HttpRequestMessage(method, url);
request.Headers.TryAddWithoutValidation("authority", "www.url.com");
request.Headers.TryAddWithoutValidation("pragma", "no-cache");
request.Headers.TryAddWithoutValidation("cache-control", "no-cache");
request.Headers.TryAddWithoutValidation("dnt", "1");
request.Headers.TryAddWithoutValidation("x-requested-with", "XMLHttpRequest");
request.Headers.TryAddWithoutValidation("x-odesk-csrf-token", "19b91748869456a4ae700ffb69077745");
request.Headers.TryAddWithoutValidation("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36");
request.Headers.TryAddWithoutValidation("accept", "*/*");
request.Headers.TryAddWithoutValidation("origin", "https://www.url.com");
request.Headers.TryAddWithoutValidation("sec-fetch-site", "same-origin");
request.Headers.TryAddWithoutValidation("sec-fetch-mode", "cors");
request.Headers.TryAddWithoutValidation("sec-fetch-dest", "empty");
request.Headers.TryAddWithoutValidation("referer", "https://www.url.com/");
request.Headers.TryAddWithoutValidation("accept-language", "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,fr-FR;q=0.6,fr;q=0.5");
request.Headers.TryAddWithoutValidation("cookie", "G_AUTHUSER_H=1; AccountSecurity_cat=fc4d14f1.oauth2v2_812293");
return request;
}
If you are using this in a console program, you can be fine with creating HttpClient in a using statement. However, if you are intending to use the client in something like a web app in .net core, you want to use the services.AddHttpClient method and dependency injection. This is because the HttpClient isn't intended to be disposed of after every use, if you are using it multiple times in a given call especially.
really appreciate the answers from #Jaquez and #Kuroiyatsu, from both I get to the following
public async Task<string> postAsync(param1, param2)
{
using (var httpClient = new HttpClient(handler))
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://www.url.com/"))
{
....
var response = await PostAsync(param1, param2);
var variable = JsonSerializer.Deserialize<Obj>(response);
WebScrapFunc(response);
...
Although it seems weird to return a Task<string> from PostAsync it fits fine my intents.
I am experiencing a known bug in the HttpClient. Anytime the server response contains "UTF-8" (including quotes), an exception is triggered:
The character set provided in ContentType is invalid. Cannot read content as string using an invalid character set. ---> System.ArgumentException: '"utf-8"' is not a supported encoding name.
Example code:
HttpClient _client = new HttpClient();
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, "https://www.facebook.com");
requestMessage.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.4044.55 Safari/537.36");
HttpResponseMessage response = _client.SendAsync(requestMessage).GetAwaiter().GetResult();
What is the usual workaroud? I am using .NETFramework 4.6.1.
To workaround the referenced issue:
using (var client = new HttpClient())
{
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get,
"https://www.facebook.com");
HttpResponseMessage response = await client.SendAsync(requestMessage);
byte[] buf = await response.Content.ReadAsByteArrayAsync();
string content = Encoding.UTF8.GetString(buf);
}
So I am trying to call a api in my C# method and am not getting the response that I would be getting, if I called the api using postman or insomia.
public static async Task GetObject()
{
var httpClient = new HttpClient();
//var t = await httpClient.GetStringAsync("https://ipapi.co/json/");
var y = await httpClient.GetAsync("https://ipapi.co/json/");
var o = await y.Content.ReadAsStringAsync();
var j = y.Content.ReadAsStringAsync();
}
The api is getting my request but its not returning the right result. You should be getting this result. Click on this https://ipapi.co/json/
What am getting is this
{
"error":true,
"reason":"RateLimited",
"message":"Sign up for IP Address Location API # https://ipapi.co"
}
But I don't get it, when am using postman
You need to add the user-agent header.
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1");
Better to Use xNet.dll or Leaf.xNet.dll libraries
Or restsharp one
Google them
I'm also new If ur interesting to contact me
Discord Cyx.#4260
I'm trying to deserialize some JSON in C#, but when I run my program I'm getting this error message:
I've looked through all my code, and I can't find a "<" anywhere there shouldn't be one, and I went to the web address that the json is coming from:
http://forecast.weather.gov/MapClick.php?lat=47.1211&lon=-88.5694&FcstType=json,
and there isn't a "<" character. I used json2csharp.com to translate to C# classes, and everything there seems fine as well. Any thoughts? Here is the part of my code where I try to do all of this:
var http = new HttpClient();
var url = "http://forecast.weather.gov/MapClick.php?lat=47.1211&lon=-88.5694&FcstType=json";
var response = await http.GetAsync(url);
var result = await response.Content.ReadAsStringAsync();
var serializer = new DataContractJsonSerializer(typeof(RootObject2));
var ms = new MemoryStream(Encoding.UTF8.GetBytes(result));
var data = (RootObject2)serializer.ReadObject(ms);
return data;
Your call is failing because you are not setting a header the API is expecting. Add a user agent and check for success prior to attempting to read the response.
var http = new HttpClient();
var url = "http://forecast.weather.gov/MapClick.php?lat=47.1211&lon=-88.5694&FcstType=json";
//Supply the same header as chrome
http.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36");
var response = await http.GetAsync(url);
if (response.IsSuccessStatusCode)
{
var result = await response.Content.ReadAsStringAsync();
var ms = new MemoryStream(Encoding.UTF8.GetBytes(result));
var serializer = new DataContractJsonSerializer(typeof(RootObject2));
var data = (RootObject2)serializer.ReadObject(ms);
}
check that answer, it says some issue with the connection, that he was not receiving the full response from the API
Unexpected character encountered while parsing value: