Our environment has multiple web servers running behind a load balancer. If I modify my host file to go directly to a web server which will by-pass the load balancer, the HttpWebRequest works fine. If I remove the entry in my host file and let the request go through the load balancer it returns 404. However, if I perform the same request using a simple .html file on the same computer I am running the HttpWebRequest from it all works fine whether there is a entry in my host file or not.
Any ideas why it would work through the browser(Chrome) with the html file, but not an HttpWebRequest?
The request is a POST request which uploads a file and 3 other parameters.
Thanks in advance for any ideas.
EDIT: I tried setting my UserAgent to Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 which is what Chrome sends when I use the html file.
Here is the function I'm using and currently works when no load balancer is in the mix:
private void PostFileToWebServer(byte[] file)
{
string boundary = "----------------------------" + DateTime.Now.Ticks.ToString("x");
var request = (HttpWebRequest)WebRequest.Create(_url);
request.ContentType = "multipart/form-data; boundary=" + boundary;
request.Method = "POST";
request.KeepAlive = true;
//Time in milliseconds
request.Timeout = 1200000;
request.Host = "ourdomain.com";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1";
request.CookieContainer = new CookieContainer(100000);
var memStream = new MemoryStream();
byte[] boundarybytes = Encoding.ASCII.GetBytes("\r\n--" + boundary + "\r\n");
string formDataTemplate = "\r\n--" + boundary + "\r\nContent-Disposition: form-data; name=\"{0}\"\r\n\r\n{1}";
foreach(var key in _parameters.AllKeys)
{
string formData = string.Format(formDataTemplate, key, _parameters[key]);
byte[] formDataBytes = Encoding.UTF8.GetBytes(formData);
memStream.Write(formDataBytes, 0, formDataBytes.Length);
memStream.Write(boundarybytes, 0, boundarybytes.Length);
}
string header = string.Format("Content-Disposition: form-data; name=\"file\"; filename=\"{0}\"\r\n Content-Type: application/octet-stream\r\n\r\n", _fileName);
byte[] headerBytes = Encoding.UTF8.GetBytes(header);
memStream.Write(headerBytes, 0, headerBytes.Length);
memStream.Write(file, 0, file.Length);
memStream.Write(boundarybytes, 0, boundarybytes.Length);
request.ContentLength = memStream.Length;
Stream requestStream = request.GetRequestStream();
memStream.Position = 0;
var tempBuffer = new byte[memStream.Length];
memStream.Read(tempBuffer, 0, tempBuffer.Length);
memStream.Close();
requestStream.Write(tempBuffer, 0, tempBuffer.Length);
requestStream.Close();
var response = request.GetResponse();
Stream stream2 = response.GetResponseStream();
var reader2 = new StreamReader(stream2);
System.Diagnostics.Debug.WriteLine(reader2.ReadToEnd());
response.Close();
}
EDIT: I commented out request.ContentType = "multipart/form-data; boundary=" + boundary; and it works with the load balancer. Anyone have an explanation for this?
Ok, me and Fiddler had a long conversation on the differences between the successful call and the unsuccessful call. Fiddler says the last boundary of the successful called sent from the browser has an extra "--" at the end of it. When I put that, all is well with the world and I can keep my Content-Type set as it is.
Related
so im working on an instagram tool. i need to Create an account with HttpWebRequest but somethings wrong with my code and it doesnt work here is the Code:
username = "something";
password = "something";
mail = " something#mail.com";
name = "something";
string postData = "email=" + mail + "&password=" + password + "&enc_password=%23PWD_INSTAGRAM_BROWSER%3A6%3A1583227313%3AAXFQANTXAGE5jwEKNbpJbvot0SGTp%2Bq7a0ckELnuYQnQLIJa2Th6UwIqdknx%2FDa8R7q1%2F2Bt4scBUrh%2B1aFDKL0H%2Fut5cyi3w1sIEPa1keAuNcNj9nAXo1oFzUYMhCGVB7qerse5hQfMjTVjNa4%3D&username=" + username + "&first_name=" + name + "&seamless_login_enabled=1&tos_version=row&opt_into_one_tap=false";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://www.instagram.com/accounts/emailsignup/");
byte[] postBytes = Encoding.ASCII.GetBytes(postData);
req.ContentLength = postBytes.Length;
req.CookieContainer = new CookieContainer();
req.Method = "POST";
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36";
req.Accept = "*/*";
req.ContentType = "application/x-www-form-urlencoded";
req.Referer = "https://www.instagram.com/accounts/emailsignup/";
req.Headers["Accept-Language"] = "de,en-US;q=0.7,en;q=0.3";
req.Headers["Accept-Encoding"] = "gzip, deflate";
req.Headers["X-Requested-With"] = "XMLHttpRequest";
Stream dataStream = req.GetRequestStream();
dataStream.Write(postBytes, 0, postBytes.Length);
dataStream.Flush();
dataStream.Close();
HttpWebResponse webResp = (HttpWebResponse)req.GetResponse(); //error in this line---> The remote server returned an error: (403) Forbidden.
Stream datastream = webResp.GetResponseStream();
StreamReader reader = new StreamReader(datastream);
string s = reader.ReadToEnd();
Console.WriteLine(s);
I'm trying to get a captcha from a 3rd party website and store it into a picturebox.
Here is the captcha url: http://www.lbj00.com/ValidationControls.aspx?
Using Google chrome I could view the captcha.
Sample image:
Sample image from chrome
But getting it to work in C#.net seems there is a problem. So far I have tried to use another captcha url: http://www.xl18.biz/general/captcha.aspx from a different 3rd party website and it could store it in a picturebox without a problem.
Here is a sample code what I have from a class:
public static Stream GetStream(string url,out string refcookies)
{
HttpWebResponse httpWebResponse = null;
HttpWebRequest httpWebRequest;
httpWebRequest = WebRequest.Create(url) as HttpWebRequest;
httpWebRequest.Method = "GET";
httpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
httpWebRequest.Headers["Cache-Control"] = "max-age=0";
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.Referer = url;
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36";
httpWebRequest.Proxy = null;
httpWebRequest.AllowAutoRedirect = true;
httpWebRequest.Credentials = CredentialCache.DefaultCredentials;
try
{
httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
}
catch (WebException wex)
{
Console.WriteLine(wex.Message);
}
refcookies = ""; //out string parameter
foreach (Cookie ck in httpWebResponse.Cookies)
{
refcookies += ck.Name + "=" + ck.Value + "; ";
}
try
{
using (Stream inStream = httpWebResponse.GetResponseStream())
{
MemoryStream outStream = new MemoryStream();
byte[] buffer = new byte[1024];
int size = 0;
while ((size = inStream.Read(buffer, 0, (int)buffer.Length)) > 0)
{
outStream.Write(buffer, 0, size);
}
return outStream;
}
}
catch
{
if (httpWebResponse != null)
{
httpWebResponse.Close();
httpWebResponse = null;
}
return null;
//throw;
}
finally
{
httpWebRequest = null;
}
}
Here is a sample code I have from a button to store it in the picturebox:
string cookies;
Stream s;
s = Http.GetStream(codeUrl, out cooikes);
Image image = Image.FromStream(s); //returns an error here - System.ArgumentException: 'Parameter is not valid.'
byte[] data;
using (MemoryStream ms = new MemoryStream())
{
image.Save(ms, image.RawFormat);
ms.Position = 0;
data = new byte[ms.Length];
ms.Read(data, 0, Convert.ToInt32(ms.Length));
pictureBox2.Image = Image.FromStream(ms);
ms.Flush();
}
[Update]
Could this be a problem from the server of the 3rd party website?
- not a server issue, seems that the response value is a _incapsula_resource.
Thanks in advance.
The stream GetStream returns is not in the required format.
Check the httpWebResponse properties upon arrival, particularly the ContentType. It should be image/png or similar. Most probably the stream you receive from it is not ONLY the image binary but other values as well.
For unknown reason the server changed it response based from what I have set in the request headers.
These are the only request headers I managed to set to make it work the rest in the code you will find in the above question is removed:
httpWebRequest.Method = "GET";
httpWebRequest.Referer = url;
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36";
So far now the response is successfully converted into stream and managed to store it within the picturebox. Both urls mentioned above in my question are working using only those headers.
I'm having a problem when sending HttpWebRequest to a particular url.
The response is generated but it is not that I want.
It is giving a response like "Session Timeout! Please Login to continue".
Whereas,
When I'm sending the same request through Python 3.4 it is giving me the required response.
The response is like "--some html is here--" which I want.
I don't know why this is happening because I wan't it to do throught C#.
The CodeBehind (giving the wrong response) is:
string postData = "somequerystring";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("someurl");
request.KeepAlive = true;
request.ProtocolVersion = HttpVersion.Version10;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36";
request.Method = "POST";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
using (var dataStream = request.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string responseFromServer = new StreamReader(response.GetResponseStream()).ReadToEnd();
response.Close();
Context.Response.Write(responseFromServer);
The Python Script (giving the correct response) is:
import requests
url = "someurl"
payload = "somequerystring"
headers = {
'content-type': "application/x-www-form-urlencoded"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
Help me out?
I am building one auto login application for one of my website. I have developed the auto login tool in C#. I have used httpwebrequest to make post call on the login page. I'm facing issue with the compression of the webpage. In some of the computers I'm not getting compressed response while in others I'm getting the response. Can anyone suggest how I can guaranteed get compressed response. The following is the code I have written for my httpwebrequest:
httpRequest.Connection = "keepalive";
httpRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
httpRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
httpRequest.UserAgent = "Mozilla/5.0 (Windows T 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36";
httpRequest.UnsafeAuthenticatedConnectionSharing = true;
httpRequest.Headers.Set("Cache-Control", "no-cache");
httpRequest.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(postdata);
httpRequest.ContentLength = bytes.Length;
requestStream = httpRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Flush();
requestStream.Close();
response = (HttpWebResponse)httpRequest.GetResponse();
HttpWebResponse response2 = response;
switch (response2.StatusCode)
{
case HttpStatusCode.OK:
responseStream = response2.GetResponseStream();
if (response2.ContentEncoding.ToLower().Contains("gzip"))
{
str = new StreamReader(new GZipStream(responseStream, CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else if (response2.ContentEncoding.ToLower().Contains("deflate"))
{
str = new StreamReader(new DeflateStream(responseStream, CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else
{
str = new StreamReader(responseStream, Encoding.UTF8).ReadToEnd();
}
responseStream.Close();
responseStream = null;
break;
}
I am using this code below to authenticate and upload file to a website, but I am getting an error "Corrupt form data: no leading boundary".
In the Fiddler I see this, ( I see the exact same thing when I upload the file using browser), not sure why I am getting an error when I am doing using C# HttpWebRequest/WebClient
Fiddler Info:
------WebKitFormBoundary9ewWOMyBmk0YAhTL
Content-Disposition: form-data; name="FileSubmitted"; filename="SAMPLE_0001.xls"
Content-Type: application/vnd.ms-excel
------WebKitFormBoundary9ewWOMyBmk0YAhTL
Content-Disposition: form-data; name="FileSubmittedValue"
C:\path\SAMPLE_0001.xls
------WebKitFormBoundary9ewWOMyBmk0YAhTL--
Assumptions:
I hard coded the 'boundary' value, because I am not sure how to get the 'boundary' value after login.
Code: Copied from Stackoverflow
static void Main()
{
NameValueCollection nvCollection = new NameValueCollection();
CookieAwareWebClient webClient = new CookieAwareWebClient();
nvCollection.Clear();
nvCollection["Name"] = "ABC";
nvCollection["Password"] = "XYZ";
//Login to the Site
byte[] responseBytes = webClient.UploadValues("https://www.somesite.com/login.cfm", "POST", nvCollection);
string resultAuthTicket = Encoding.UTF8.GetString(responseBytes);
//Get Cookies
CookieCollection cookies = webClient.CookieContainer.GetCookies(new Uri("https://www.somesite.com/login.cfm"));
//
string URL = "https://www.somesite.com/app/request.cfm";
string boundary = "----WebKitFormBoundary9ewWOMyBmk0YAhTL";
string FilePath = "C:\\Users\\user.name\\Desktop\\SAMPLE_0001.xls";
byte[] fileData = GetMultipartFormData(new Dictionary<string, object>() { { "FileSubmitted", FilePath } }, boundary);
PostForm(URL, "", "", fileData, boundary, cookies);
}
private static byte[] GetMultipartFormData(Dictionary<string, object> postParameters, string boundary)
{
Encoding encoding = Encoding.UTF8;
Stream formDataStream = new System.IO.MemoryStream();
bool needsCLRF = false;
foreach (var param in postParameters)
{
// Thanks to feedback from commenters, add a CRLF to allow multiple parameters to be added.
// Skip it on the first parameter, add it to subsequent parameters.
if (needsCLRF)
formDataStream.Write(encoding.GetBytes("\r\n"), 0, encoding.GetByteCount("\r\n"));
needsCLRF = true;
{
string postData = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\"\r\nContent-Type: {3}\r\n\r\n",
boundary,
param.Key,
param.Value,
"application/vnd.ms-excel");
formDataStream.Write(encoding.GetBytes(postData), 0, encoding.GetByteCount(postData));
string postData2 = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\" \r\n\r\n{2}",
boundary,
"FileSubmittedValue",
"C:\\path\\SAMPLE_0001.xls");
formDataStream.Write(encoding.GetBytes(postData2), 0, encoding.GetByteCount(postData2));
}
}
// Add the end of the request. Start with a newline
string footer = "\r\n--" + boundary + "--\r\n";
formDataStream.Write(encoding.GetBytes(footer), 0, encoding.GetByteCount(footer));
// Dump the Stream into a byte[]
formDataStream.Position = 0;
byte[] formData = new byte[formDataStream.Length];
formDataStream.Read(formData, 0, formData.Length);
formDataStream.Close();
return formData;
}
private static HttpWebResponse PostForm(string postUrl, string userAgent, string contentType, byte[] formData, string boundary, CookieCollection cookies)
{
HttpWebRequest request = WebRequest.Create(postUrl) as HttpWebRequest;
if (request == null)
{
throw new NullReferenceException("request is not a http request");
}
// Set up the request properties.
request.UserAgent = userAgent;
request.ContentLength = formData.Length;
request.ContentType = "multipart/form-data; boundary=" + boundary;
request.Method = "POST";
request.KeepAlive = true;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
request.Host = "secure.somesite.com";
request.Referer = String.Format("https://secure.somesite.com/app/request.cfm?CFID={0}&CFTOKEN={1}", cookies[0].Value, cookies[1].Value);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36";
// Send the form data to the request.
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(formData, 0, formData.Length);
requestStream.Close();
}
try
{
return request.GetResponse() as HttpWebResponse;
}
catch (WebException wex)
{
var pageContent = new StreamReader(wex.Response.GetResponseStream())
.ReadToEnd();
return null;
}
return null;
}
Error:
<head><title>JRun Servlet Error</title></head><h1>500 </h1><body>
<pre>
Corrupt form data: no leading boundary: != ------WebKitFormBoundary9ewWOMyBmk0YAhTL</pre><br><pre>
java.io.IOException: Corrupt form data: no leading boundary: != ------WebKitFormBoundary9ewWOMyBmk0YAhTL
at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:176)
at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:95)
at coldfusion.filter.FormScope.fillMultipart(FormScope.java:170)
at coldfusion.filter.FusionContext.SymTab_initForRequest(FusionContext.java:435)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:33)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
at coldfusion.CfmServlet.service(CfmServlet.java:175)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
</pre></body>
Don't Knows: Since I copied this source code from Stackoverflow, I don't know if the file really getting uploaded to C:\Path folder on server (look at the Fiddler data).
Update: When I try to upload file using browser I see Content-Length: 49504, but when I try using C# program Content-Length: 385 (Even when I use the same file in both browser and C# program)
Also, when I try now I am getting "No data was received in the uploaded file SAMPLE_0001.xls"
The code is missing this line of code, which is the actual file content.
formDataStream.Write(fileToUpload.File, 0, fileToUpload.File.Length);
Thanks all for your help!