so I have this website
After inspecting the network traffic for the download button I got the below curl post request
curl "https://flood-map-for-planning.service.gov.uk/pdf" -X POST -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" -H "Accept-Encoding: gzip, deflate, br" -H "Content-Type: application/x-www-form-urlencoded" -H "Origin: https://flood-map-for-planning.service.gov.uk" -H "Connection: keep-alive" -H "Referer: https://flood-map-for-planning.service.gov.uk/flood-zone-results?easting=429240&northing=431613&location=LS118TR" -H "Upgrade-Insecure-Requests: 1" -H "Sec-Fetch-Dest: document" -H "Sec-Fetch-Mode: navigate" -H "Sec-Fetch-Site: same-origin" -H "Sec-Fetch-User: ?1" -H "TE: trailers" --data-raw "id=1660136366038&polygon=¢er="%"5B429240"%"2C431613"%"5D&reference=&scale=2500"
I went over to this website in order to convert the curl to c#
This is what I got
using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://flood-map-for-planning.service.gov.uk/pdf"))
{
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0");
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8");
request.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.5");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
request.Headers.TryAddWithoutValidation("Origin", "https://flood-map-for-planning.service.gov.uk");
request.Headers.TryAddWithoutValidation("Connection", "keep-alive");
request.Headers.TryAddWithoutValidation("Referer", "https://flood-map-for-planning.service.gov.uk/flood-zone-results?easting=429240&northing=431613&location=LS118TR");
request.Headers.TryAddWithoutValidation("Upgrade-Insecure-Requests", "1");
request.Headers.TryAddWithoutValidation("Sec-Fetch-Dest", "document");
request.Headers.TryAddWithoutValidation("Sec-Fetch-Mode", "navigate");
request.Headers.TryAddWithoutValidation("Sec-Fetch-Site", "same-origin");
request.Headers.TryAddWithoutValidation("Sec-Fetch-User", "?1");
request.Headers.TryAddWithoutValidation("TE", "trailers");
request.Content = new StringContent("id=1660136366038&polygon=¢er=");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/x-www-form-urlencoded");
var response = await httpClient.SendAsync(request);
}
}
I changed it to:
var httpClient = new HttpClient();
var request =
new HttpRequestMessage(new HttpMethod("POST"), "https://flood-map-for-planning.service.gov.uk/pdf");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0");
request.Headers.TryAddWithoutValidation("Accept", "application/pdf");
request.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.5");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
request.Headers.TryAddWithoutValidation("Origin", "https://flood-map-for-planning.service.gov.uk");
request.Headers.TryAddWithoutValidation("Connection", "keep-alive");
request.Headers.TryAddWithoutValidation("Referer", "https://flood-map-for-planning.service.gov.uk/flood-zone-results?easting=429240&northing=431613&location=LS118TR");
request.Headers.TryAddWithoutValidation("Upgrade-Insecure-Requests", "1");
request.Headers.TryAddWithoutValidation("Sec-Fetch-Dest", "document");
request.Headers.TryAddWithoutValidation("Sec-Fetch-Mode", "navigate");
request.Headers.TryAddWithoutValidation("Sec-Fetch-Site", "same-origin");
request.Headers.TryAddWithoutValidation("Sec-Fetch-User", "?1");
request.Headers.TryAddWithoutValidation("TE", "trailers");
request.Content = new StringContent("center=&scale=2500");
var response = httpClient.Send(request);
response.Content.Headers.Add("Content-Disposition", "inline;filename=\"Testpdf.pdf\"");
response.Content.Headers.Add("Content-Name", "Testpdf.PDF");
response.Content.Headers.Add("Content-Type", "application/pdf;charset=UTF-8");
if (response.IsSuccessStatusCode)
{
using (FileStream fs = new FileStream("somepdf.pdf", FileMode.CreateNew))
{
using (StreamWriter writer = new StreamWriter(fs))
{
var contentStream = response.Content.ReadAsStream(); // get the actual content stream
writer.Write(contentStream);
}
}
}
This is the issue.
My goal is to download the pdf locally.
I usually get a file which is 1KB or 6KB.
The curl command with an output parameter works without an issue. I'm just not sure what the above c# http post request is missing.
As you can see I've added the filestream and streamwriter usages.
I've also tried to play with the response in order to nagivate it to an application/pdf response.
Any ideas why I am doing wrong?
=======================================================
EDIT
Thanks to #thehennyy,
here is the working solution:
var unixTimestamp = (long)DateTime.UtcNow.Subtract(DateTime.UnixEpoch).TotalSeconds;
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (var httpClient = new HttpClient(handler))
{
using (var request =
new HttpRequestMessage(new HttpMethod("POST"), "https://flood-map-for-planning.service.gov.uk/pdf"))
{
request.Headers.TryAddWithoutValidation("Referer",
"https://flood-map-for-planning.service.gov.uk/flood-zone-results?easting=429240&northing=431613&location=LS118TR");
request.Content =
new StringContent($"id={unixTimestamp}&polygon=¢er=[429240,431613]&reference=&scale=2500");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/x-www-form-urlencoded");
var response = await httpClient.SendAsync(request);
if (response.IsSuccessStatusCode)
{
using (FileStream fs = new FileStream("somepdf.pdf", FileMode.Create))
{
var contentStream = await response.Content.ReadAsStreamAsync();
await contentStream.CopyToAsync(fs);
}
}
}
}
There are a few things to consider here:
It seems like the curl to httpclient converter had a problem converting the post content. The following works for me:
request.Content = new StringContent("id=1&polygon=¢er=[429240,431613]&reference=&scale=2500");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/x-www-form-urlencoded");
The parameter id has to be provided, otherwise the request will fail. The website uses the current unix timestamp as value for the id parameter.
Adding headers to the response response.Content.Headers.Add([...]) is not meaningful, just delete these lines.
Writing the content to disk can be done simpler:
using (FileStream fs = new FileStream("somepdf.pdf", FileMode.Create))
{
var contentStream = await response.Content.ReadAsStreamAsync();
await contentStream.CopyToAsync(fs);
}
While testing i got the same "wrong" files, these are usual just html responses, sometimes containing an error message. View them as html. Maybe they seem like gibberish, then you have to turn on automatic decompression:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var httpClient = new HttpClient(handler);
The automatic decompression values should match this headers values:
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
Current versions of dotnet also support "br" - DecompressionMethods.Brotli.
Using automatic decompression is helpful in nearly every case.
Related
I made http post request with content type application/x-www-form-urlencoded UTF-8 using Httpclient.PostRequest. I need to convert application/x-www-form-urlencoded request result to json string. How can I application/x-www-form-urlencoded to json in C#?
I did next post request.
HttpResponseMessage response;
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://sv.ki.court.gov.ua/new.php"))
{
request.Headers.TryAddWithoutValidation("Connection", "keep-alive");
request.Headers.TryAddWithoutValidation("Accept", "application/json, text/javascript, */*; q=0.01");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
request.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.9");
request.Headers.TryAddWithoutValidation("Referer", "https://sv.ki.court.gov.ua/sud2608/gromadyanam/csz");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36");
request.Headers.TryAddWithoutValidation("Origin", "https://sv.ki.court.gov.ua");
request.Headers.TryAddWithoutValidation("X-Requested-With", "XMLHttpRequest");
request.Headers.TryAddWithoutValidation("Cookie", "_ga=GA1.3.1754947237.1562858299; PHPSESSID=1nosm5dhdljsv8tpsu97bl8dn5; cookiesession1=257F9822DPOYFGLLUDVTJCMDYYR98AB6; _gid=GA1.3.1599643655.1563891940; _gat=1");
request.Content = new StringContent("q_court_id=2608", Encoding.UTF8, "application/x-www-form-urlencoded");
response = await _client.SendAsync(request);
}
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
var json = await response.Content.ReadAsStringAsync();
As StringContent type I used application/x-www-form-urlencoded. application/json doesn't work. Post request return result string "\u001f<....". I need to convert response result application/x-www-form-urlencoded to application/json
This should give you some idea. In the example below inside PostAsync you pass in a new instance of StringContent where you set the type of content i.e. in this case Json.
public static async Task MainAsync()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost:1234");
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("", "Joe Bloggs")
});
var result = await client.PostAsync("/api/Customer/exists", new StringContent(content.ToString(), Encoding.UTF8, "application/json"));
string resultContent = await result.Content.ReadAsStringAsync();
Console.WriteLine(resultContent);
}
}
I am trying to extract data from a government site which is rendered on display of a pop up. I checked the network console and got the POST request URL and able to replicate the request-response on Postman. Now I am trying to make the call programmatically. I tried using the default code generated by Postman but it did not work.
I am writing the code in C# and I am able to get the response but I am not able to get the correct encoding to extract the response.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://saubhagya.gov.in/dashboard/data/dashboard_saubhagya");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "application/json, text/javascript, */*; q=0.01";
request.Host = "saubhagya.gov.in";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept-Language: en-US,en;q=0.9");
request.Headers.Add("Cache-Control: no-cache");
request.Headers.Add("Pragma: no-cache");
request.Headers.Add("Origin: http://saubhagya.gov.in");
request.Referer = "http://saubhagya.gov.in/";
String CookieStr = "_ga=GA1.3.590075225.1533991967; _gid=GA1.3.790472263.1533991967; saubhagyasession=BHxX%2FFfttUfxM7JhoIruGVzdq0m%2F4sGeTn95c%2BUB%2BGvJok9PkS3g9pR8vLVfeEJ1XB8UULGNThvbAeN5HfAu%2FE6qt%2F5X3qL8Yla4my0qmxSmz6Q9ztpLztCD0PyY17uWDnJgkSjSt%2BSF0B5Xh32SUsxBXHH%2BeFGwtIXdAnzSLcxC0MO8KZSiE2io4ksZO6AZ31YSxnGei6CluQzg4fCFgXvVwR4%2F00%2FKAbf0MnhLwaTtXxD0jngmDv3Rjy8enD87c20vwObHGTgcLC3KQoh2lw5L1WRF1lVLlpjzLrUoeJV3cD8o0c15bT5SA%2FV1Y8OqFPhqhpr0%2BzzG%2FbAVs6OKMmLiokl7hHrPx5NECDsmY3KzmCkNHka%2B1ueEWTv%2FTOUqH2hll2A8485gFhqFgnrh%2FKkhOb6I8lChI2QQoyHr%2B9U%3D92add88ce105d8b3ec1dd72efa1dd7ec9b9f1e52";
CookieContainer cookiecontainer = new CookieContainer();
string[] cookies = CookieStr.Split(';');
foreach (string cookie in cookies)
cookiecontainer.SetCookies(new Uri("http://saubhagya.gov.in/dashboard/data/dashboard_saubhagya"), cookie);
request.CookieContainer = cookiecontainer;
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
string json = "ci_csrf_token=&state=35&district=638&village=645115&vtype=&discom=&search_text=&uuid=&maptype=states&kyroargs=&drilldownkey=&kyroclickid=&kyrorefreshid=&page=dashboard_saubhagya";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream(), System.Text.Encoding.GetEncoding(response.CharacterSet)))
{
var result = sr.ReadToEnd();
}
I get an encoded/junk string as output.
Requesting for help!
In the header you state that you accept gzip, but the response is not decompressed on your end, so just add:
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
Then you can remove this line because headers will be added automatically:
//request.Headers.Add("Accept-Encoding: gzip, deflate");
I am trying to send HttpClient PostAsync() request to company's internal sharepoint site but its returning response with forbidden error. I have all necessary access permission for site to load and have also passed required headers to the HttpClient object.
Here is code snippet.
HttpClient client = new System.Net.Http.HttpClient (new HttpClientHandler { UseDefaultCredentials = true });
client.BaseAddress = new Uri (string.Format (API_URL, p_siteNumber));
client.DefaultRequestHeaders.Accept.Add (new MediaTypeWithQualityHeaderValue (#"application/atom+xml"));
client.DefaultRequestHeaders.TryAddWithoutValidation ("Accept-Encoding", "gzip, deflate");
client.DefaultRequestHeaders.TryAddWithoutValidation ("Accept-Language", "en-US, en;q=0.8, hi;q=0.6");
client.DefaultRequestHeaders.TryAddWithoutValidation ("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
client.DefaultRequestHeaders.TryAddWithoutValidation ("Accept-Charset", "ISO-8859-1");
HttpResponseMessage httpResponse = await client.PostAsync (urlHttpPost, new StringContent (string.Empty));
string response = await httpResponse.Content.ReadAsStringAsync ();
Can anyone help me with this?
Thanks in advance.
I ran into the same problem I wanted to send the file and some string contents with it.
so below code helped me!!
using (var client = new HttpClient())
{
//client.DefaultRequestHeaders.Add("User-Agent", "CBS Brightcove API Service");
string authorization = GenerateBase64();
client.DefaultRequestHeaders.Add("Authorization", authorization);
using (var content = new MultipartFormDataContent())
{
string fileName = Path.GetFileName(textBox1.Text);
//Content-Disposition: form-data; name="json"
var stringContent = new StringContent(InstancePropertyObject);
stringContent.Headers.Remove("Content-Type");
stringContent.Headers.Add("Content-Type", "application/json");
stringContent.Headers.Add("Content-Disposition", "form-data; name=\"instance\"");
content.Add(stringContent, "instance");
var fileContent = new ByteArrayContent(filecontent);
fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = fileName
};
content.Add(fileContent);
var result = client.PostAsync(targetURL, content).Result;
}
}
I am doing a get request in c# but i am getting some invalid response than the original content
code
using System;
using System.Net;
using System.Text;
using System.IO;
public class Test
{
// Specify the URL to receive the request.
public static void Main(string[] args)
{
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create("http://www1.bloomingdales.com/api/store/v2/stores/367,363,6113,364,4946?upcNumber=849262004629");
// Set some reasonable limits on resources used by this request
myHttpWebRequest.MaximumAutomaticRedirections = 4;
myHttpWebRequest.MaximumResponseHeadersLength = 4;
// Set credentials to use for this request.
myHttpWebRequest.Credentials = CredentialCache.DefaultCredentials;
myHttpWebRequest.Method = "GET";
myHttpWebRequest.AllowAutoRedirect = false;
myHttpWebRequest.ContentLength = 0;
myHttpWebRequest.Accept = "application/json, text/javascript, */*; q=0.01";
myHttpWebRequest.Headers.Add("Cookie", "ShippingCountry=US;");
myHttpWebRequest.UserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/49.0.2623.108 Chrome/49.0.2623.108 Safari/537.36";
myHttpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate, sdch");
myHttpWebRequest.Headers.Add("Accept-Language", "en-US,en;q=0.8");
myHttpWebRequest.Headers.Add("X-Macys-ClientId", "NavApp");
var response = (HttpWebResponse)myHttpWebRequest.GetResponse();
var rmyResponseHeaders = response.Headers;
Console.WriteLine("Content length is {0}", response.ContentLength);
Console.WriteLine("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
Console.WriteLine("Response stream received.");
Console.WriteLine(readStream.ReadToEnd());
Console.ReadLine();
response.Close();
readStream.Close();
}
}
The above getting invalid character like ???????? and some other characters
You can check original response here
curl 'http://www1.bloomingdales.com/api/store/v2/stores/367,363,6113,364,4946?upcNumber=849262004629' -H 'Cookie:shippingCountry=US;' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/49.0.2623.108 Chrome/49.0.2623.108 Safari/537.36' -H 'Accept-Language: en-US,en;q=0.8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'X-Macys-ClientId: NavApp' --compressed
How to fix this and get exact response(orginal response)?
You're not decompressing the response stream. Set the Accept-Encoding header to empty string until you're ready to work out which compression was used and decompress it.
My current job is to consume a RESTful API with OAuth2. Currently I worked out how to get the access token and it is working ok while I use the chrome extension Rest Console, but when I try to do it from my application I always get the error that I am sending an invalid OAuth request. Below you can see three of the ways I tried to consume the API, but to no success. The page always returns error 500. Any help will be appreciated, if I had missed something crucial.
var auth = "Bearer " + item.access_token;
/* First Attempt */
var client = new RestClient("http://<link>");
var request = new RestRequest("sample", Method.GET);
request.AddHeader("Authorization", auth);
request.AddHeader("Content-Type", "application/json;charset=UTF-8");
request.AddHeader("Pragma", "no-cache");
request.AddHeader("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
request.AddHeader("Accept", "application/json");
request.RequestFormat = DataFormat.Json;
var response = client.Execute(request);
var content = response.Content;
/* Second Attempt */
string sURL = "http://<link>/sample";
string result = "";
using (WebClient client = new WebClient())
{
client.Headers["Authorization"] = auth;
client.Headers["Content-Type"] = "application/json;charset=UTF-8";
client.Headers["Pragma"] = "no-cache";
client.Headers["User-Agent"] = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36";
client.Headers["Accept"] = "application/json";
byte[] byteArray = Encoding.UTF8.GetBytes(parameters);
var result1 = client.DownloadString(sURL);
}
/* Third Attempt */
var request = (HttpWebRequest)WebRequest.Create(sURL);
request.Method = "GET";
request.ContentType = "application/json;charset=UTF-8";
request.Accept = "application/json";
request.Headers["Authorization"] = auth;
request.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36";
string content;
HttpStatusCode statusCode;
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
var contentType = response.ContentType;
Encoding encoding = null;
if (contentType != null)
{
var match = Regex.Match(contentType, #"(?<=charset\=).*");
if (match.Success)
encoding = Encoding.GetEncoding(match.ToString());
}
encoding = encoding ?? Encoding.UTF8;
statusCode = ((HttpWebResponse)response).StatusCode;
using (var reader = new StreamReader(stream, encoding))
content = reader.ReadToEnd();
}
--------EDIT--------
For the first attempt I also tried to add the authentication to the client variable client.Authenticator = Authenticate; where OAuth2AuthorizationRequestHeaderAuthenticator Authenticate = new OAuth2AuthorizationRequestHeaderAuthenticator(item.access_token, item.token_type);
The code seems right. The fail attempts you did suggest that the issue is with the token and not the code. Bearer tokens have expiration time. So semms like your token expired between the first time you got it using chrome REST Console extension and when you wrote your code. But the strange situation here is the 500 error code you got. 401 is standard response when you token expired or not exist. 500 error code always mean server error.