I'm scraping data from some sites and one of theses use headers for some information. I'm using Smart Proxy Manager from Zyte and having problems with a request with headers. When I use the zyte proxy, I receive a response, but the site that I'm scraping return an unsuccess message. If I not use the proxy, the request works like a charm. Can anyone help me?
Here is my code
string zyteKey = _configuration["ZyteKey"];
var proxy = new WebProxy("myproxy");
proxy.Credentials = new NetworkCredential(zyteKey, "");
var clientHandler = new HttpClientHandler()
{
Proxy = proxy,
UseProxy = true,
PreAuthenticate = true,
AllowAutoRedirect = false
};
using (var client = new HttpClient(clientHandler))
{
string json = JsonConvert.SerializeObject(jsonObject);
var content = new StringContent(json, Encoding.UTF8, "application/json");
client.Timeout = new TimeSpan(0, 1, 0);
client.DefaultRequestHeaders.Add("cookie", BuildCookieRequest(latitude, longitude, restaurantName));
client.DefaultRequestHeaders.Add("Accept", "application/json, text/plain, */*");
client.DefaultRequestHeaders.Add("x-csrf-token", "x");
return await client.PostAsync(url, content);
}
Related
I tried to follow an answer and this is what I have now:
var response = string.Empty;
HttpContent content = new StringContent(poststring);
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
//const string contentType = "multipart/form-data; boundary=<calculated when request is sent>";
HttpRequestMessage request = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = u,
Content = content,
};
var handler = new TemporaryRedirectHandler()
{
InnerHandler = new HttpClientHandler()
{
AllowAutoRedirect = false
}
};
HttpClient client = new HttpClient(handler);
HttpResponseMessage result = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
return result;
Problem is that I still get a 200 and when I look at my request in Fiddler I don't see the AllowAutoRdirect set to false anywhere. WHere should i capture that field to include it in the request?
I am trying to call Rest API with content and headers in c#. Actually I am trying to convert to c# from Python code which is:
import requests
url = 'http://url.../token'
payload = 'grant_type=password&username=username&password=password'
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
response = requests.request('POST', url, headers = headers, data = payload, allow_redirects=False)
print(response.text)
So far I am trying with:
HttpClient client = new HttpClient();
client.BaseAddress = new Uri(Url);
var tmp = new HttpRequestMessage
{
Method = HttpMethod.Post,
Content =
{
}
};
var result = client.PostAsync(Url, tmp.Content).Result;
}
I have no idea how to put from Python code Headers (Content-Type) and additional string (payload).
If you use RestSharp, you should be able to call your service with the following code snipped
var client = new RestClient("http://url.../token");
var request = new RestRequest(Method.POST);
request.AddHeader("content-type", "application/x-www-form-urlencoded");
request.AddParameter("application/x-www-form-urlencoded", "grant_type=password&username=username&password=password", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
var result = response.Content;
I based my answer on the anwser of this answer.
Here a sample I use in one of my apps:
_client = new HttpClient { BaseAddress = new Uri(ConfigManager.Api.BaseUrl),
Timeout = new TimeSpan(0, 0, 0, 0, -1) };
_client.DefaultRequestHeaders.Accept.Clear();
_client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
_client.DefaultRequestHeaders.Add("Bearer", "some token goes here");
using System.Net.Http;
var content = new StringContent("grant_type=password&username=username&password=password");
content.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
client.PostAsync(Url, content);
Or use FormUrlEncodedContent without set header
var data = new Dictionary<string, string>
{
{"grant_type", "password"},
{"username", "username"},
{"password", "password"}
};
var content = new FormUrlEncodedContent(data);
client.PostAsync(Url, content);
If you write UWP application, use HttpStringContent or HttpFormUrlEncodedContent instead in Windows.Web.Http.dll.
using Windows.Web.Http;
var content = new HttpStringContent("grant_type=password&username=username&password=password");
content.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
client.PostAsync(Url, content);
var data = new Dictionary<string, string>
{
{"grant_type", "password"},
{"username", "username"},
{"password", "password"}
};
var content = new FormUrlEncodedContent(data);
client.PostAsync(Url, content);
I'm trying to set the content-type of my HttpClient Post request, and use the profile parameter, but when I change the content type I get an exception thrown:
"The format of value 'application/json; profile={URL HERE}' is
invalid."
For reference, I found this Q&A: Zoopla Sandbox with cURL http header error
X509Certificate2 cert = new X509Certificate2("cert.pfx", "PASSWORD");
WebRequestHandler handler = new WebRequestHandler();
handler.ClientCertificates.Add(cert);
var client = new HttpClient(handler);
client.BaseAddress = new Uri("https://realtime-listings-api.webservices.zpg.co.uk");
var stringContent = new StringContent(propertyData, Encoding.UTF8, "application/json; profile=http://realtime-listings.webservices.zpg.co.uk/docs/v1.1/schemas/listing/list.json");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var response = await client.PostAsync("/sandbox/v1/listing/list", stringContent);
return _resultFactory.Create(true, await response.Content.ReadAsStringAsync());
If you create a HttpRequestMessage and use client.SendAsync(), you can add the parameters to request.Content.Headers.ContentType.Parameters
var client = new HttpClient();
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, "https://realtime-listings-api.webservices.zpg.co.uk/sandbox/v1/listing/list"))
{
request.Content = new StringContent("propertyData", Encoding.UTF8, "application/json");
request.Content.Headers.ContentType.Parameters.Add(
new NameValueHeaderValue("profile", "http://realtime-listings.webservices.zpg.co.uk/docs/v1.1/schemas/listing/list.json")
);
var response = await client.SendAsync(request);
//Handle response..
}
You don't need to use HttpRequestMessage but you do need to add the profile value as a quoted string via the NameValueHeaderValue parameter:
var content = new StringContent(request.ToJson(), Encoding.UTF8);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
content.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("profile", "\"https://realtime-listings.webservices.zpg.co.uk/docs/v1.2/schemas/listing/update.json\""))
httpClient.PostAsync("listing/update", content);
This will get round the FormatException. Otherwise you will run into this dotnet bug.
I am trying to use the Trustpilot API, to post invitations to review products.
I have successfully gone through the authentication step as you can see in the code below, however I am unable to successfully post data to the Trustpilot Invitations API. The PostAsnyc method appears to be stuck with an WaitingForActivation status. I wonder if there is anything you can suggest to help.
Here is my code for this (the API credentials here aren't genuine!):
using (HttpClient httpClient = new HttpClient())
{
string trustPilotAccessTokenUrl = "https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken";
httpClient.BaseAddress = new Uri(trustPilotAccessTokenUrl);
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded"));
var authString = "MyApiKey:MyApiSecret";
httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Base64Encode(authString));
var stringPayload = "grant_type=password&username=MyUserEmail&password=MyPassword";
var httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/x-www-form-urlencoded");
HttpResponseMessage httpResponseMessage = httpClient.PostAsync(trustPilotAccessTokenUrl, httpContent).Result;
var accessTokenResponseString = httpResponseMessage.Content.ReadAsStringAsync().Result;
var accessTokenResponseObject = JsonConvert.DeserializeObject<AccessTokenResponse>(accessTokenResponseString);
// Create invitation object
var invitation = new ReviewInvitation
{
ReferenceID = "inv001",
RecipientName = "Jon Doe",
RecipientEmail = "Jon.Doe#comp.com",
Locale = "en-US"
};
var jsonInvitation = JsonConvert.SerializeObject(invitation);
var client = new HttpClient();
client.DefaultRequestHeaders.Add("token", accessTokenResponseObject.AccessToken);
var invitationsUri = new Uri("https://invitations-api.trustpilot.com/v1/private/business-units/{MyBusinessID}/invitations");
// This here as a status of WaitingForActivation!
var a = client.PostAsync(invitationsUri, new StringContent(jsonInvitation)).ContinueWith((postTask) => postTask.Result.EnsureSuccessStatusCode());
}
This is how I solved the issue:
// Serialize our concrete class into a JSON String
var jsonInvitation = JsonConvert.SerializeObject(invitationObject);
// Wrap our JSON inside a StringContent which then can be used by the HttpClient class
var stringContent = new StringContent(jsonInvitation);
// Get the access token
var token = GetAccessToken().AccessToken;
// Create a Uri
var postUri = new Uri("https://invitations-api.trustpilot.com/v1/private/business-units/{BusinessUnitID}/invitations");
// Set up the request
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, postUri);
request.Content = stringContent;
request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
request.Content.Headers.Add("token", token);
// Set up the HttpClient
var httpClient = new HttpClient();
//httpClient.DefaultRequestHeaders.Accept.Clear();
//httpClient.BaseAddress = postUri;
//httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
//httpClient.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en-US"));
var task = httpClient.SendAsync(request);
task.Wait();
This question here on SO was helpful:
How do you set the Content-Type header for an HttpClient request?
I am trying to update a SharePoint 2013 list title using the REST api and HttpClient. I have the following code but the StatusCode is returning BadRequest.
private async void UpdateTitle()
{
string webUrl = http://server;
Uri uri = new Uri(webUrl + "/_api/web/lists/GetByTitle('Old')");
//start replacement
HttpClientHandler httpClientHandler = new HttpClientHandler();
httpClientHandler.UseDefaultCredentials = true;
HttpClient client = new HttpClient(httpClientHandler);
client.DefaultRequestHeaders.Add("ContentType", "application/json;odata=verbose");
string digest = await GetFormDigest("http://devsp13.dev.local/_api/contextinfo");
client.DefaultRequestHeaders.Add("X-RequestDigest", digest);
client.DefaultRequestHeaders.Add("X-HTTP-Method", "Merge");
client.DefaultRequestHeaders.Add("IF-MATCH", "*");
HttpContent content = new StringContent("{ '__metadata': { 'type': 'SP.List' }, 'Title': 'NewTitle' }");
HttpResponseMessage response = await client.PostAsync(uri, content);
response.EnsureSuccessStatusCode();}
I'm guessing I've missed something in forming the HttpClient but I'm struggling to see what. Any help is much appreciated.
Cheers,
Geoff
Try adding Accept to the HttpClient DefaultRequestHeaders and set the ContentType on the HttpContent Headers instead. This worked for me.
HttpClientHandler handler = new HttpClientHandler();
handler.UseDefaultCredentials = true;
HttpClient client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
client.DefaultRequestHeaders.Add("X-RequestDigest", GetFormDigest());
client.DefaultRequestHeaders.Add("X-HTTP-Method", "MERGE");
client.DefaultRequestHeaders.Add("IF-MATCH", "*");
HttpContent content = new StringContent(metadataString);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
content.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("odata", "verbose"));
HttpResponseMessage response = await client.PostAsync(uri, content);
response.EnsureSuccessStatusCode();