I'm trying to simulate HTTP post file exactly like the browser (chrome) does:
HTML:
<form name="fnup" method="post" action="action.shtml" enctype="multipart/form-data">
<input name="firmfile" id="firmfile" type="file">
<input type="submit" id="firmbutton" value="Upgrade" class="othsave">
</form>
C#:
public async Task<string> UploadFile(string actionUrl, string filePath)
{
var httpClient = new HttpClient();
//Should I need to add all those headers??? it will help me? what is necessary?
httpClient.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
httpClient.DefaultRequestHeaders.Add("Origin", "http://" + ip);
httpClient.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
httpClient.DefaultRequestHeaders.Add("Pragma", "no-cache");
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36");
httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
httpClient.DefaultRequestHeaders.Add("Referer", "http://" + ip + "/");
httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate");
httpClient.DefaultRequestHeaders.Add("Accept-Language", "he,en-US;q=0.9,en;q=0.8");
httpClient.DefaultRequestHeaders.Add("Cookie", "page=DevSet");
FileStream paramFileStream = File.OpenRead(filePath);
HttpContent fileStreamContent = new StreamContent(paramFileStream);
using (var content = new MultipartFormDataContent())
{
content.Add(fileStreamContent, "firmfile", Path.GetFileName(filePath));
HttpResponseMessage response = await httpClient.PostAsync(actionUrl, content);
string res = await response.Content.ReadAsStringAsync();
return await Task.Run(() => res);
}
}
The C# example is not working in my server, and I cannot understand why...
After looking on Wireshark, I've seen some differents:
Chrome:
C#
The questions:
1) How can I delete the "filename*=utf-8''VP445_all_V116.bin" from Content-Disposition
2) Why I cannot see the second line Content-Type: application/octet-stream like in Chrome?
3) Also the Content-length is not the same (even it's the same file)
4) The bottom line is that the C# is not working and chrome is working, and the server side is black-box for me, so I need to guess here what I'm doing wrong in C# post request.
Found the solution.
The problem indeed was the 'Content-Type: application/octet-stream'
Why the HttpClient StreamContent is not added it automatically? I really don't know.
The workaround is added it manually
The main idea is from here:
https://github.com/paulcbetts/ModernHttpClient/issues/92
Here is the correct C# code:
public async Task<string> UploadFile(string actionUrl, string filePath)
{
var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromMilliseconds(FileReqTimeout);
FileStream fileStream = File.OpenRead(filePath);
var streamContent = new StreamContent(fileStream);
streamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data");
streamContent.Headers.ContentDisposition.Name = "\"firmfile\"";
streamContent.Headers.ContentDisposition.FileName = "\"" + Path.GetFileName(filePath) + "\"";
streamContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
string boundary = Guid.NewGuid().ToString();
var content = new MultipartFormDataContent(boundary);
content.Headers.Remove("Content-Type");
content.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
content.Add(streamContent);
HttpResponseMessage response = null;
try
{
response = await httpClient.PostAsync(actionUrl, content, cts.Token);
}
catch (WebException ex)
{
// handle web exception
return null;
}
catch (TaskCanceledException ex)
{
if (ex.CancellationToken == cts.Token)
{
// a real cancellation, triggered by the caller
return null;
}
else
{
// a web request timeout (possibly other things!?)
return null;
}
}
try
{
response.EnsureSuccessStatusCode();
}
catch (Exception)
{
return null;
};
string res = await response.Content.ReadAsStringAsync();
return await Task.Run(() => res);
}
}
Related
I am trying to make web scraper in C# for NSE. The code works with other sites but when ran on https://www.nseindia.com/ it gives error - An error occurred while sending the request. Unable to read data from the transport connection: Operation timed out.
I have tried with two different approaches Try1() & Try2().
Can anyone please tell what I am missing in my code?
class Program
{
public void Try1() {
HtmlWeb web = new HtmlWeb();
HttpStatusCode statusCode = HttpStatusCode.OK;
web.UserAgent = GetUserAgent();
web.PostResponse = (request, response) =>
{
if (response != null)
{
statusCode = response.StatusCode;
Console.WriteLine("Status Code: " + statusCode);
}
};
Task<HtmlDocument> task = web.LoadFromWebAsync(GetURL());
HtmlDocument document = task.Result;
}
public void Try2() {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(GetURL());
request.UserAgent = GetUserAgent();
request.Accept= "*/*;";
using (var response = (HttpWebResponse)(request.GetResponse()))
{
HttpStatusCode code = response.StatusCode;
if (code == HttpStatusCode.OK)
{
using (StreamReader streamReader = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
HtmlDocument htmlDoc = new HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
htmlDoc.Load(streamReader);
Console.WriteLine("Document Loaded.");
}
}
}
}
private string GetURL() {
// return "https://html-agility-pack.net/";
return "https://www.nseindia.com/";
}
private string GetUserAgent() {
return "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36";
}
}
Your are lack of headers towards Accept and others so it couldn't response back.
Besides that, I would recommend you using HttpClient instead of HttpWebRequest
public static async Task GetHtmlData(string url)
{
HttpClient httpClient = new HttpClient();
using (var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url)))
{
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml, charset=UTF-8, text/javascript, */*; q=0.01");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36 OPR/67.0.3575.137");
request.Headers.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");
request.Headers.TryAddWithoutValidation("X-Requested-With", "XMLHttpRequest");
using (var response = await httpClient.SendAsync(request).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode();
using (var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
using (var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress))
using (var streamReader = new StreamReader(decompressedStream))
{
var result = await streamReader.ReadToEndAsync().ConfigureAwait(false);
HtmlDocument htmlDoc = new HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
htmlDoc.LoadHtml(result);
Console.WriteLine(result);
Console.WriteLine("Document Loaded.");
}
}
}
Use it by
await GetHtmlData("https://www.nseindia.com/");
I have the below code, working as expected (given correct URL etc) as a POST request. Seems I have a problem reading the Status Code (I receive a successful 201, and based on that number I need to continue processing). Any idea how to get the status code?
static async Task CreateConsentAsync(Uri HTTPaddress, ConsentHeaders cconsentHeaders, ConsentBody cconsent)
{
HttpClient client = new HttpClient();
try
{
client.BaseAddress = HTTPaddress;
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("*/*"));
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Cache-Control", "no-cache");
client.DefaultRequestHeaders.Add("otherHeader", myValue);
//etc. more headers added, as needed...
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, client.BaseAddress);
request.Content = new StringContent(JsonConvert.SerializeObject(cconsent, Formatting.Indented), System.Text.Encoding.UTF8, "application/json");
Console.WriteLine("\r\n" + "POST Request:\r\n" + client.DefaultRequestHeaders + "\r\nBody:\r\n" + JsonConvert.SerializeObject(cconsent, Formatting.Indented) + "\r\n");
await client.SendAsync(request).ContinueWith
(
responseTask =>
{
Console.WriteLine("Response: {0}", responseTask.Result + "\r\nBody:\r\n" + responseTask.Result.Content.ReadAsStringAsync().Result);
}
);
Console.ReadLine();
}
catch (Exception e)
{
Console.WriteLine("Error in " + e.TargetSite + "\r\n" + e.Message);
Console.ReadLine();
}
}
There is a Status code in your Result.
responseTask.Result.StatusCode
Or even better
var response = await client.SendAsync(request);
var statusCode = response.StatusCode;
It helps to avoid using ContinueWith if you're already inside an async function because you can use the (much cleaner) await keyword.
If you await the SendAsync call you'll get a HttpResponseMessage object you can get the status code from:
Also, wrap your IDisposable objects in using() blocks (except HttpClient - which should be a static singleton or better yet, use IHttpClientFactory).
Don't use HttpClient.DefaultRequestHeaders for request-specific headers, use HttpRequestMessage.Headers instead.
The Connection: Keep-alive header will be sent by HttpClientHandler automatically for you.
Are you sure you need to send Cache-control: no-cache in the request? If you're using HTTPS then it's almost guaranteed that there won't be any proxy-caches causing any issues - and HttpClient does not use the Windows Internet Cache either.
Don't use Encoding.UTF8 because it adds a leading byte-order-mark. Use a private UTF8Encoding instance instead.
Always use .ConfigureAwait(false) with every await on code that does not run in a thread-sensitive context (such as WinForms and WPF).
private static readonly HttpClient _httpClient = new HttpClient();
private static readonly UTF8Encoding _utf8 = new UTF8Encoding( encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true );
static async Task CreateConsentAsync( Uri uri, ConsentHeaders cconsentHeaders, ConsentBody cconsent )
{
using( HttpRequestMessage req = new HttpRequestMessage( HttpMethod.Post, uri ) )
{
req.Headers.Accept.Add( new MediaTypeWithQualityHeaderValue("*/*") );
req.Headers.Add("Cache-Control", "no-cache");
req.Headers.Add("otherHeader", myValue);
//etc. more headers added, as needed...
String jsonObject = JsonConvert.SerializeObject( cconsent, Formatting.Indented );
request.Content = new StringContent( jsonObject, _utf8, "application/json");
using( HttpResponseMessage response = await _httpClient.SendAsync( request ).ConfigureAwait(false) )
{
Int32 responseHttpStatusCode = (Int32)response.StatusCode;
Console.WriteLine( "Got response: HTTP status: {0} ({1})", response.StatusCode, responseHttpStatusCode );
}
}
}
You could simply check the StatusCode property of the response:
https://learn.microsoft.com/en-us/previous-versions/visualstudio/hh159080(v=vs.118)?redirectedfrom=MSDN
static async void dotest(string url)
{
using (HttpClient client = new HttpClient())
{
HttpResponseMessage response = await client.GetAsync(url);
if (response.IsSuccessStatusCode)
{
Console.WriteLine(response.StatusCode.ToString());
}
else
{
// problems handling here
Console.WriteLine(
"Error occurred, the status code is: {0}",
response.StatusCode
);
}
}
}
#AthanasiosKataras is correct for returning the status code itself but if you would also like to return the status code value (ie 200, 404). You can do the following:
var response = await client.SendAsync(request);
int statusCode = (int)response.StatusCode
The above will give you the int 200.
EDIT:
Is there no reason why you cannot do the following?
using (HttpResponseMessage response = await client.SendAsync(request))
{
// code
int code = (int)response.StatusCode;
}
I'm trying to call a URL that should return an authentication token.
Data is posted to the URL and after a number of redirects returns a JSON object with a token.
I'm using C# and WPF.
Here is the excerpt from what I am doing:
HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create(action);
request1.Method = "POST";
StringBuilder sb = new StringBuilder();
String boundary = "-----------------------------1721856231228";
foreach (elem in elems)
{
String nameStr = elem.GetAttribute("name");
if (nameStr != null && nameStr.Length != 0)
{
String valueStr = elem.GetAttribute("value");
sb.Append("\r\n" + boundary + "\r\n");
sb.Append("Content-Disposition: form-data; name=\"" + nameStr + "\"" + "\r\n");
sb.Append("\r\n");
sb.Append(valueStr);
}
}
sb.Append("\r\n--" + boundary + "--" + "\r\n");
String postData1 = sb.ToString();
request1.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
request1.UserAgent = "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36";
request1.ContentType = "application/x-www-form-urlencoded; boundary=" + boundary;
request1.ContentLength = postData1.Length;
request1.KeepAlive = true;
request1.AllowAutoRedirect = true;
StreamWriter w = new StreamWriter(request1.GetRequestStream());
w.Write(postData1);
w.Close();
HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse();
StreamReader reader1 = new StreamReader(response1.GetResponseStream());
String responseText1 = reader1.ReadToEnd();
reader1.Close();
response1.Close();
But the response doesn't contain the JSON with a token.
I am using Fiddler and can pause at the end of the above code and the URI that should have the JSON hasn't been called. I can continue executing other code in the debugger, and then later, Fiddler will show the URI as having been called and a File Download popup lets me then download a JSON file that contains the token.
I don't want the popup and I want to be able to capture the JSON data programmatically.
I found by adding the following line to the end of the code above, and just executing that line in the debugger, that Fiddler will report that the token URL has been called (and I can see in Fiddler the correct JSON response):
System.Windows.Forms.Application.DoEvents();
But I don't know how to access this response or how to short-circuit the file download popup from not happening.
Maybe something in the KeepAlive setting would help?
try newtonsoft to read
using Newtonsoft.Json
TokenModel tokenModel;
StreamReader reader1 = new StreamReader(response1.GetResponseStream());
using (JsonTextReader reader = new JsonTextReader(reader1))
{
tokenModel = serializer.Deserialize<TokenModel>(reader);
}
Reference: https://www.newtonsoft.com/json/help/html/ReadJson.htm
Or you can use following complete request with response with HttpClient
var client = new HttpClient();
client.BaseAddress = new Uri("your url");
int _TimeoutSec = 90;
client.Timeout = new TimeSpan(0, 0, _TimeoutSec);
string _ContentType = "application/x-www-form-urlencoded";
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(_ContentType));
//if you have any content to send use following keyValuePair
var kv = new List<KeyValuePair<string, string>>();
kv.Add(new KeyValuePair<string, string>("key1", "value"));
kv.Add(new KeyValuePair<string, string>("key2", "value"));
var req = new HttpRequestMessage(System.Net.Http.HttpMethod.Post, "your url") { Content = new FormUrlEncodedContent(kv) };
var responseAsyn = client.SendAsync(req);
var response = responseAsyn.GetAwaiter().GetResult();
TokenModel tokenResponse = new TokenModel();
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
var responseString = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
tokenResponse = JsonConvert.DeserializeObject<TokenModel>(responseString);
}
string boundray = "---------------------------" + DateTime.Now.Ticks.ToString("x");
string url = HttpDomainHandling(currentUser.DomainURL) + API + "&is_multi_part_upload=true";
string header = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\"\r\n" + "Content-Type: {3}\r\n\r\n", boundray, "file", file.Name, "application/octet-stream");
string footer = string.Format("\r\n--{0}--\r\n", boundray);
Stream headerStream = GenerateStreamFromString(header);
Stream footerStream = GenerateStreamFromString(footer);
Stream dataStream = await sfile.OpenStreamForReadAsync();
MemoryStream fileDataStream = new MemoryStream();
await headerStream.CopyToAsync(fileDataStream);
await dataStream.CopyToAsync(fileDataStream);
await footerStream.CopyToAsync(fileDataStream);
fileDataStream.Position = 0;
IInputStream stream = fileDataStream.AsInputStream();
BackgroundUploader backgroundUploader = new BackgroundUploader();
backgroundUploader.SetRequestHeader("Content-Type", "multipart/form-data; boundary=" + boundray);
backgroundUploader.SetRequestHeader("Cookie", Constants.FELIXSESSIONID + "=" + currentUser.SessionID);
backgroundUploader.Method = "POST";
UploadOperation uploadOpration = await backgroundUploader.CreateUploadFromStreamAsync(new Uri(url), stream);
await Task.Factory.StartNew(() => CheckUploadStatus(uploadOpration, progressEvent, cts));
var result = await uploadOpration.StartAsync();
ResponseInformation info = uploadOpration.GetResponseInformation();
return info;
Unable to find json response in result and response information..where can i get this response...
I am trying to upload file to my server.. and its return upload data in json format..
Finally got the Answer
BackgroundUploader is used when you want to upload something on IP or some address and you don't want the response string from server...
BackgroundUploader can return just status code and successfully or error message info
if you want to upload something and you want you response string example JSON or XML...
need to use UWP HTTPCLIENT
//Create an HTTP client object
Windows.Web.Http.HttpClient httpClient = new Windows.Web.Http.HttpClient();
//Add a user-agent header to the GET request.
var headers = httpClient.DefaultRequestHeaders;
//The safe way to add a header value is to use the TryParseAdd method and verify the return value is true,
//especially if the header value is coming from user input.
string header = "ie";
if (!headers.UserAgent.TryParseAdd(header))
{
throw new Exception("Invalid header value: " + header);
}
header = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)";
if (!headers.UserAgent.TryParseAdd(header))
{
throw new Exception("Invalid header value: " + header);
}
Uri requestUri = new Uri("http://www.contoso.com");
//Send the GET request asynchronously and retrieve the response as a string.
Windows.Web.Http.HttpResponseMessage httpResponse = new Windows.Web.Http.HttpResponseMessage();
string httpResponseBody = "";
try
{
//Send the GET request
httpResponse = await httpClient.GetAsync(requestUri);
httpResponse.EnsureSuccessStatusCode();
httpResponseBody = await httpResponse.Content.ReadAsStringAsync();
}
catch (Exception ex)
{
httpResponseBody = "Error: " + ex.HResult.ToString("X") + " Message: " + ex.Message;
}
I am trying to make a login POST request with json to this website
Link,and follow the redirect.My current program works fine if the login details are wrong.If the details are wrong I get the '(401) Unauthorized.' message,which means the Post request was succesful.
However,my problem is that,if the login details are correct,I get the '(400) Bad Request'. I have no idea why this happens and I am currently stuck at this point.
Here is my code,and I hope someone can help me out:
static string url = "https://auth.riotgames.com/authz/auth";
static string uriString = "";
static void Main(string[] args)
{
var request_check = (HttpWebRequest)HttpWebRequest.Create("https://auth.riotgames.com/authz/auth");
request_check.Host = "auth.riotgames.com";
request_check.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0";
request_check.Accept = "application/json, text/javascript, */*; q=0.01";
request_check.Headers.Add("Accept-Language", "en-US,en;q=0.5");
request_check.Headers.Add("Accept-Encoding", "gzip, deflate, br");
request_check.ContentType = "application/json";
request_check.Headers.Add("X-Requested-With", "XMLHttpRequest");
request_check.Referer = "https://auth.riotgames.com/authorize?response_type=code&scope=openid%20email&client_id=merch-store-client&ui_locales=de-DE&login_hint=euw&redirect_uri=https://euw.merch.riotgames.com/de/riot_sso/auth/redirect/";
var cookieContainer = new CookieContainer();
request_check.CookieContainer = cookieContainer;
request_check.Method = "POST";
request_check.KeepAlive = true;
request_check.AllowAutoRedirect = false;
// Account details Senturia:a12365478
using (var streamWriter = new StreamWriter(request_check.GetRequestStream()))
{
string json = "{\"username\":\"Senturia\",\"password\":\"a12365478\",\"remember\":false,\"region\":\"EUW1\",\"language\":\"de_DE\",\"lang\":\"de_DE\"}";
streamWriter.Write(json);
}
try
{
// Get the response ...
using (var webResponse = (HttpWebResponse)request_check.GetResponse())
{
// Now look to see if it's a redirect
if ((int)webResponse.StatusCode >= 300 && (int)webResponse.StatusCode <= 399)
{
uriString = webResponse.Headers["Location"];
Console.WriteLine("Redirect to " + uriString ?? "NULL");
}
}
}
catch(Exception e)
{
Console.WriteLine(e.Message);
}
Console.ReadKey();
}
When an HTTP request fails you can catch a WebException, and read the response from the server as it might contain useful information about the reason why the request failed:
catch (WebException e)
{
using (var stream = e.Response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
Console.WriteLine(reader.ReadToEnd());
}
}
In your case this prints:
{"error":"invalid_session_id","error_description":"Missing session id."}
So I guess that the server requires some session id parameter to be sent along with the request. Consult the documentation of the endpoint you are trying to invoke for more details on how to do that.