Develop Reference

Web Response 'an error occured while sending this request' while trying to login into Rockstar social club page

I want to login to Rockstar Social Club page https://pl.socialclub.rockstargames.com
I have this script
public static void Login()
{
string firstUrl = "https://pl.socialclub.rockstargames.com/profile/signin";
string formParams = string.Format("login-field={0}&password-field={1}", "mynickname", "mypassword");
string cookieHeader;
WebRequest req = WebRequest.Create(firstUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string pageSource;
string getUrl = "https://pl.socialclub.rockstargames.com/games/gtav/pc/career/overview/gtaonline";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse(); //Here returns me this error: System.Net.WebException: 'An error occurred while sending the request"
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
}
Error occures in WebResponse getResponse = getRequest.GetResponse();
System.Net.WebException: 'An error occurred while sending the request'
I don't know how to repair this, and succesfully login to this website.

I have accomplished what you are attempting to do, but on a different website.
Basically - a few years ago, I wanted to create a website that would track my Guild/Company details on Final Fantasy XIV.
They didn't have an API, so I made one.
In order to get the information I required, I needed to use a mix of HtmlAgilityPack along with the C# WebBrowser control.
In order to pass the verification token stage above, you need to run the page source in a Web Browser control.
This will allow dynamic fields and data to be generated.
You then need to take that data, and submit it with your post data.
This is to fool it into thinking the request is coming from the page.
Be warned, when doing your posts - you may need to allow for redirects and you may need to mirror the referrer and host fields to match the website you are emulating.
The specific process I followed was:
Navigate to login page in WebBrowser control
Get page source
Load into HtmlAgilityPack HtmlDocument class
Use XPath to scrape the login form.
Take _verification tokens, csrf tokens etc make note of them.
Post a web-request with the necessary data to the form target destination url.
Read the response
Be aware - sometimes the response will actually be html code that tells it to do a Javascript redirect - in my case with Final Fantasy XIV - it was loading up another form and performing an autopost on page load.
You will also want to use
LoggedInCookies = new CookieContainer();
In your first HttpWebRequest
followed by:
request.CookieContainer = LoggedInCookies;
for each subsequent request.
The cookie container will trap and persist the authentication related cookies, while the WebBrowser control and HtmlAgilityPack will allow you to scrape the fields from the web forms that you need to break through.
Adding some code from wayback when I solved this for Final Fantasy XIV's lodestone website.
This code is very old and may not work anymore, but the process it follows could be adapted for sites that do not use Javascript as part of the login process.
Pay attention to the areas where it allows the request to be redirected, this is because the Server endpoint you are calling may do Action redirects etc
If your request does not allow those redirects, then it will not be emulating the login process.
class LoggedInClient
{
public static CookieContainer LoginCookie(string user, string pass)
{
string sStored = "";
string url = "http://eu.finalfantasyxiv.com/lodestone/account/login/";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
CookieContainer cookies = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36";
request.CookieContainer = cookies;
HttpWebResponse response1 = (HttpWebResponse)request.GetResponse();
Console.WriteLine(cookies.Count.ToString());
string sPage = "";
using (var vPage = new StreamReader(response1.GetResponseStream()))
{
sPage = vPage.ReadToEnd();
}
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(sPage);
sStored = doc.DocumentNode.SelectSingleNode("//input[#type='hidden' and #name='_STORED_']").Attributes["value"].Value;
string param = "sqexid="+user+"8&password="+pass+"&_STORED_=" + sStored;
string postURL = doc.DocumentNode.SelectSingleNode("//form[#name='mainForm']").Attributes["action"].Value;
//Console.WriteLine(sStored);
postURL = "https://secure.square-enix.com/oauth/oa/" + postURL;
request.Method = "POST";
byte[] paramAsBytes = Encoding.Default.GetBytes(param);
request = (HttpWebRequest)WebRequest.Create(postURL);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36";
request.CookieContainer = cookies;
request.AllowAutoRedirect = false;
try
{
using (Stream stream = request.GetRequestStream())
{
stream.Write(paramAsBytes, 0, paramAsBytes.Length);
}
}
catch (Exception ee)
{
Console.WriteLine(ee.ToString());
}
string sGETPage = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (var vPage = new StreamReader(response.GetResponseStream()))
{
sPage = vPage.ReadToEnd();
sGETPage = response.Headers["Location"];
}
}
// Console.WriteLine(sPage);
request = (HttpWebRequest)WebRequest.Create(sGETPage);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36";
request.CookieContainer = cookies;
HttpWebResponse response2 = (HttpWebResponse)request.GetResponse();
Console.WriteLine(cookies.Count.ToString());
sPage = "";
using (var vPage = new StreamReader(response2.GetResponseStream()))
{
sPage = vPage.ReadToEnd();
}
// Console.WriteLine(sPage);
doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(sPage);
string _c = doc.DocumentNode.SelectSingleNode("//input[#type='hidden' and #name='_c']").Attributes["value"].Value;
string cis_sessid = doc.DocumentNode.SelectSingleNode("//input[#type='hidden' and #name='cis_sessid']").Attributes["value"].Value;
string action = doc.DocumentNode.SelectSingleNode("//form[#name='mainForm']").Attributes["action"].Value;
string sParams = "_c=" + _c + "&cis_sessid=" + cis_sessid;
byte[] bData = Encoding.Default.GetBytes(sParams);
// Console.WriteLine(sStored);
request = (HttpWebRequest)WebRequest.Create(action);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36";
request.CookieContainer = cookies;
request.AllowAutoRedirect = true;
try
{
using (Stream stream = request.GetRequestStream())
{
stream.Write(bData, 0, bData.Length);
}
}
catch (Exception ee)
{
Console.WriteLine(ee.ToString());
}
string nextPage = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (var vPage = new StreamReader(response.GetResponseStream()))
{
nextPage = vPage.ReadToEnd();
}
}
// Console.WriteLine(nextPage);
doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(nextPage);
string csrf_token = doc.DocumentNode.SelectSingleNode("//input[#type='hidden' and #name='csrf_token']").Attributes["value"].Value;
string cicuid = "51624738";
string timestamp = Convert.ToInt32(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1)).TotalSeconds).ToString() + "100";
action = "http://eu.finalfantasyxiv.com/lodestone/api/account/select_character/";
sParams = "csrf_token=" + csrf_token + "&cicuid=" + cicuid + "&timestamp=" + timestamp;
bData = Encoding.Default.GetBytes(sParams);
request = (HttpWebRequest)WebRequest.Create(action);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36";
request.CookieContainer = cookies;
request.AllowAutoRedirect = true;
try
{
using (Stream stream = request.GetRequestStream())
{
stream.Write(bData, 0, bData.Length);
}
}
catch (Exception ee)
{
Console.WriteLine(ee.ToString());
}
nextPage = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (var vPage = new StreamReader(response.GetResponseStream()))
{
nextPage = vPage.ReadToEnd();
}
}
return cookies;
}
}

Reading data from url returns empty value

I have this code that creates a request and reads the data but it is always empty
static string uri = "http://yiimp.ccminer.org/api/wallet?address=DshDF3zmCX9PUhafTAzxyQidwgdfLYJkBrd";
static void Main(string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
Console.WriteLine("Response stream received.");
Console.WriteLine(readStream.ReadToEnd());
response.Close();
readStream.Close();
}
When i try to access the link from browser i get this json:
{"currency": "DCR", "unsold": 0.030825917365192, "balance": 0.02007306, "unpaid": 0.05089898, "paid24h": 0.05796425, "total": 0.10886323}
what am I missing?

When you perform the request from a browser there are a lot of headers that get sent to the web service. Apparently this web service validates the UserAgent. This is a decision on the part of the web service implementation, they might not want you to programmatically access it.
var client = (HttpWebRequest)HttpWebRequest.Create(new Uri("http://yiimp.ccminer.org/api/wallet?address=DshDF3zmCX9PUhafTAzxyQidwgdfLYJkBrd"));
client.AutomaticDecompression = DecompressionMethods.GZip;
client.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063";
client.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
client.Host = "yiimp.ccminer.org";
client.KeepAlive = true;
using (var s = client.GetResponse().GetResponseStream())
using (var sw = new StreamReader(s))
{
var ss = sw.ReadToEnd();
Console.WriteLine(ss);
}
Sending the headers seems to make it work.

Reading ajax response from Youtube in C#

I'm trying to post a comment on Youtube and get the xml response (so I can get the comment ID, or if I need to enter captcha or something), but I'm only able to post the comment. For some reason I am not able to read the xml response by using response.GetResponseStream(). When I try to output the response to the console, I get nothing. And, I've sniffed the requests and response my program sends and receives using Wireshark and I can see that the xml is in the response.
Here is the code I'm using to read the response:
using (HttpWebResponse response = MakeRequest(request, cookies, post))
{
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), UTF8Encoding.UTF8))
{
string xml = reader.ReadToEnd();
Console.WriteLine(xml);
}
}
and the MakeRequest function
private static HttpWebResponse MakeRequest(HttpWebRequest request, CookieContainer SessionCookieContainer, Dictionary<string, string> parameters = null)
{
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,text/xml,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
request.KeepAlive = true;
if (proxy != "") request.Proxy = myproxy;
if (parameters != null) request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string postData = string.Empty;
if (parameters != null)
{
postData = getPostData(parameters);
byte[] postBuffer = UTF8Encoding.UTF8.GetBytes(postData);
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBuffer, 0, postBuffer.Length);
}
}
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
SessionCookieContainer.Add(response.Cookies);
while (response.StatusCode == HttpStatusCode.Found)
{
response.Close();
request = GetNewRequest(response.Headers["Location"], SessionCookieContainer);
response = (HttpWebResponse)request.GetResponse();
SessionCookieContainer.Add(response.Cookies);
}
return response;
}
Any ideas on why this isn't working and how to solve this issue?

I think switching to HttpClient would solve the problem.

missing post data by using HttpWebRequest

I got a problem on posting data by using HttpWebRequest.
There is a string(ie. key1=value1&key2=value2&key3=value3) and I have post it to a site (ie. www.*.com/edit), but ,I don't know why that sometimes it's nothing wrong , but sometimes ,the first key=value1 will be missing, only key2=value&key3=value3 that can find in HttpAnalyzer.
public static string SubmitData(string Url, string FormData, CookieContainer _Cc, string ContentType)
{
Stream RequestStream = null, ResponseStream = null; StreamReader Sr = null;
HttpWebRequest HRequest = (HttpWebRequest)WebRequest.Create(Url);
try
{
HRequest.CookieContainer = _Cc;
HRequest.Method = "POST";
HRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)";
HRequest.ContentType = ContentType;
HRequest.ContentLength = FormData.Length;
//byte[] BFromData = new ASCIIEncoding().GetBytes(FormData);
byte[] BFromData = Encoding.ASCII.GetBytes(FormData);
BFromData = Encoding.Convert(Encoding.ASCII, Encoding.UTF8, BFromData);//ascii → utf8
RequestStream = HRequest.GetRequestStream();
RequestStream.Write(BFromData, 0, BFromData.Length);
//RequestStream.Write(utf8Bytes,0,utf8Bytes.Length );
HttpWebResponse HResponse = (HttpWebResponse)HRequest.GetResponse();
ResponseStream = HResponse.GetResponseStream();
Sr = new StreamReader(ResponseStream, Encoding.UTF8);
return Sr.ReadToEnd();
}
catch
{
return "";
}
finally
{
if (null != RequestStream) RequestStream.Close();
if (null != ResponseStream) ResponseStream.Close();
if (null != Sr) Sr.Close();
}
}

Use Fiddler to see how the request looks like when you click on the form then try using this approach and modify what you need for your request.
public static void PostDataAndDoSomething()
{
string URI = "http://www.something.com";
//make your request payload
string requestBody = String.Format("{{'param1': {0}, 'param2': {1}}}",value1, value2); //json format
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URI); //make request
// set request headers as you need
request.ContentType = "application/json; charset=UTF-8";
request.Accept = "application/json, text/javascript;
request.Method = "POST";
request.UserAgent = "";
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
using (StreamWriter writer = new StreamWriter(request.GetRequestStream()))
{
writer.Write(requestBody); //write your request payload
}
WebResponse response = request.GetResponse();
string jsonData = String.Empty;
using (var reader = new StreamReader(response.GetResponseStream()))
{
jsonData = reader.ReadToEnd();
}
response.Close();
//do something with your data, deserialize, Regex etc....
}

encoding problems with HttpWebResponse

I have problems with characters encoding received from http web response, I receive ? instead é.
I set the encoding to according Content-Type of web page that's text/javascript; charset=ISO-8859;
My code is:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(..);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.Referer = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream, Encoding.GetEncoding("iso-8859-1"));
char[] buf = new char[256];
int count;
StringBuilder buffer = new StringBuilder();
while ((count = sr.Read(buf, 0, 256)) > 0)
{
buffer.Append(buf, 0, count);
}
string responseStr = buffer.ToString();
Console.WriteLine(responseStr);
response.Close();
stream.Close();
sr.Close();
Can you tell me what is wrong with it?

Try adding the following before you make your request:
request.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1");
Btw, you should keep your StreamReader with ISO-8859-1 (instead of UTF8) if you want to try my proposed solution. Good luck!

Have you tried setting it at UTF-8?
Further more you send a referrer which I think you tried to set the UserAgent. The code below is the same as yours, but then does not go over the byte array and sets the useragent and utf8 encoding.
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
using(var response = (HttpWebResponse)request.GetResponse())
using(var stream = response.GetResponseStream())
using (var sr = new StreamReader(stream, Encoding.UTF8))
{
string responseStr = sr.ReadToEnd();
Console.WriteLine(responseStr);
response.Close();
if (stream != null)
stream.Close();
sr.Close();
}