Develop Reference

Unable To Instantiate StreamReader with HttpWebResponse Stream - System.ArgumentException: Stream was not readable

I haven't posted here in a long time so please forgive me if I am not formatting this question properly. I am trying to login to a website(omitted) via the .NET objects HttpWebRequest and HttpWebResponse. Using Wireshark, I can see that my POST request is identical to Chrome's POST request when I login to this website through my application. I am having issues getting the full response back though.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
This response object has all the appropriate response headers, but there is still data I would like to see that is being sent through HTTP chunked responses. I can verify this in Wireshark as well. My understanding is that I need to instantiate a StreamReader object to read this remaining data. My code is blowing up at this line:
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
The stack trace is showing this error:
System.ArgumentException: Stream was not readable.
How can I use this StreamReader object to read the entire response after my POST request is sent? Below is my code that sends the POST request for logging into the website. Please let me know if you have any questions and I will be happy to clarify any confusion.
public bool Login()
{
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(#"http://WEBSITE-REMOVED/CheckAccess");
request.CookieContainer = CookieContainer;
//Set POST data.
string postData = "institution=AAA";
postData += "&ismobile=false";
postData += "&id=BBB";
postData += "&password=CCC";
byte[] data = Encoding.ASCII.GetBytes(postData);
//Configure HTTP POST request.
request.Headers.Clear();
request.Method = "POST";
request.Accept = #"text/html, application/xhtml+xml, image/jxr, */*";
request.Referer = #"http://WEBSITE-REMOVED/entry.html";
request.Headers.Add("Accept-Language", "en-US");
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko";
request.ContentType = #"application/x-www-form-urlencoded";
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Host = "op.responsive.net";
request.ContentLength = data.Length;
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "no-cache");
using (Stream stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}//using
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
//TO-DO
}
}//try
catch(Exception ex)
{
File.AppendAllText(ErrorLogDirectory + "Errors.txt", "Login() Exception\r\n" + System.DateTime.Now + "\r\n" + ex.ToString() + Environment.NewLine);
}//catch
return false;
}//Login

HttpWebRequest dosent get response

When i try to load "https://web.com/forum" i dont get response, this url dosent need login to load. With the browser i only have to copy the url and it works fine without previous session.
When i try to load other url like "https://web.com" it works but "https://web.com/forum" dosent work. why?
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://web.com/forum");
CookieContainer cookieJar = new CookieContainer();
req.CookieContainer = cookieJar;
req.Referer = "https://web.com/forum";
req.Method = "GET";
req.Host = "web.com";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.ContentType = "application/x-www-form-urlencoded";
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
req.Proxy = null;
using (HttpWebResponse response = (HttpWebResponse)req.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (StreamReader sr = new StreamReader(responseStream))
{
respuesta = sr.ReadToEnd();
}
}
}
MessageBox.Show(respuesta);
What's the problem here?.
I have tried also to remove referer, contentype, useragent... and other headers without success.
Thanks.

Reading ajax response from Youtube in C#

I'm trying to post a comment on Youtube and get the xml response (so I can get the comment ID, or if I need to enter captcha or something), but I'm only able to post the comment. For some reason I am not able to read the xml response by using response.GetResponseStream(). When I try to output the response to the console, I get nothing. And, I've sniffed the requests and response my program sends and receives using Wireshark and I can see that the xml is in the response.
Here is the code I'm using to read the response:
using (HttpWebResponse response = MakeRequest(request, cookies, post))
{
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), UTF8Encoding.UTF8))
{
string xml = reader.ReadToEnd();
Console.WriteLine(xml);
}
}
and the MakeRequest function
private static HttpWebResponse MakeRequest(HttpWebRequest request, CookieContainer SessionCookieContainer, Dictionary<string, string> parameters = null)
{
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,text/xml,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
request.KeepAlive = true;
if (proxy != "") request.Proxy = myproxy;
if (parameters != null) request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string postData = string.Empty;
if (parameters != null)
{
postData = getPostData(parameters);
byte[] postBuffer = UTF8Encoding.UTF8.GetBytes(postData);
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBuffer, 0, postBuffer.Length);
}
}
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
SessionCookieContainer.Add(response.Cookies);
while (response.StatusCode == HttpStatusCode.Found)
{
response.Close();
request = GetNewRequest(response.Headers["Location"], SessionCookieContainer);
response = (HttpWebResponse)request.GetResponse();
SessionCookieContainer.Add(response.Cookies);
}
return response;
}
Any ideas on why this isn't working and how to solve this issue?

I think switching to HttpClient would solve the problem.

Cannot login website with httprequest POST in C#?

I'm trying to write a bit of code to login to a website. But it's not working. Please can you give me some advice. This is my a bit of code:
static void Main(string[] args)
{
CookieContainer container = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://pagehype.com/login.php");
request.Method = "POST";
request.Timeout = 10000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) (Prevx 3.0.5)";
request.CookieContainer = container;
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "username=user&password=password&processlogin=1&return=";
byte[] data = encoding.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream newStream = request.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string htmldoc = reader.ReadToEnd();
response.Close();
Console.Write(htmldoc);
}
Many thanks,

Use http://www.fiddler2.com/fiddler2/ to view the http request sent went you login in using a browser and ensure that the request you build in code is the same.

PHP logins use PHPSESSID cookie. You'll need to capture this and pass it back in the CookieContainer. This is how the server will recognise you as an authenticated user.
The cookie is set in the Set-Cookie header in the initial response. You'll need to parse it to recreate the cookie in your container (don't forget the path (and domain?)
var setCookie = response.GetResponseHeader("Set-Cookie");
response.Close();
container = new CookieContainer();
foreach (var cookie in setCookie.Split(','))
{
var split = cookie.Split(';');
string name = split[0].Split('=')[0];
string value = split[0].Split('=')[1];
var c = new Cookie(name, value);
if (cookie.Contains(" Domain="))
c.Domain = split.Where(x => x.StartsWith(" Domain")).First().Split('=')[1];
else
{
c.Domain = ".pagehype.com";
}
if (cookie.Contains(" Path="))
c.Path = split.Where(x => x.StartsWith(" Path")).First().Split('=')[1];
container.Add(c);
}
Then add this container to your request.

getResponse in c# not working. No response coming back

I have this code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
request.Timeout = 30000;
request.Method = "POST";
request.KeepAlive = true;
request.AllowAutoRedirect = false;
Stream newStream = request.GetRequestStream();
newStream.Write(bPostData, 0, bPostData.Length);
byte[] buf = new byte[1025]; int read = 0; string sResp = "";
HttpWebResponse wResp = (HttpWebResponse)request.GetResponse();
Stream resp = wResp.GetResponseStream();
The line HttpWebResponse wResp =... just hangs (as in no response from the URL). I'm not sure where exactly its crashing (cause i dont even get an exception error). I tested the URL in IE and it works fine. I also checked the bPostData and that one has data in it.
Where is it going wrong?

Try closing the request stream in variable newStream. Maybe the API waits for it to be done.

You have to increase the limit:
ServicePointManager.DefaultConnectionLimit = 10; // Max number of requests

Try simplifying your code and faking a user agent. Maybe the site is blocking/throttling scrapers/bots. Also ensure your application/x-www-form-urlencoded HTTP POST values are properly encoded. For this I would recommend you WebClient:
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
var values = new NameValueCollection
{
{ "param1", "value1" },
{ "param2", "value2" },
};
byte[] result = client.UploadValues(url, values);
}

When I commented earlier, I had run your code at my office (heavily firewalled) I got the same result you did. Came home, tried again (less firewalled) it worked fine... I'm guessing you have a barrier there. I believe you are facing a firewall issue.

Use a content-length=0
Example:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.Method = "POST";
request.ContentLength = 0;
var requestStream = request.GetRequestStream();
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
res.Close();