Develop Reference

Not getting the correct encoding to extract body from HttpWebResponse

I am trying to extract data from a government site which is rendered on display of a pop up. I checked the network console and got the POST request URL and able to replicate the request-response on Postman. Now I am trying to make the call programmatically. I tried using the default code generated by Postman but it did not work.
I am writing the code in C# and I am able to get the response but I am not able to get the correct encoding to extract the response.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://saubhagya.gov.in/dashboard/data/dashboard_saubhagya");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "application/json, text/javascript, */*; q=0.01";
request.Host = "saubhagya.gov.in";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept-Language: en-US,en;q=0.9");
request.Headers.Add("Cache-Control: no-cache");
request.Headers.Add("Pragma: no-cache");
request.Headers.Add("Origin: http://saubhagya.gov.in");
request.Referer = "http://saubhagya.gov.in/";
String CookieStr = "_ga=GA1.3.590075225.1533991967; _gid=GA1.3.790472263.1533991967; saubhagyasession=BHxX%2FFfttUfxM7JhoIruGVzdq0m%2F4sGeTn95c%2BUB%2BGvJok9PkS3g9pR8vLVfeEJ1XB8UULGNThvbAeN5HfAu%2FE6qt%2F5X3qL8Yla4my0qmxSmz6Q9ztpLztCD0PyY17uWDnJgkSjSt%2BSF0B5Xh32SUsxBXHH%2BeFGwtIXdAnzSLcxC0MO8KZSiE2io4ksZO6AZ31YSxnGei6CluQzg4fCFgXvVwR4%2F00%2FKAbf0MnhLwaTtXxD0jngmDv3Rjy8enD87c20vwObHGTgcLC3KQoh2lw5L1WRF1lVLlpjzLrUoeJV3cD8o0c15bT5SA%2FV1Y8OqFPhqhpr0%2BzzG%2FbAVs6OKMmLiokl7hHrPx5NECDsmY3KzmCkNHka%2B1ueEWTv%2FTOUqH2hll2A8485gFhqFgnrh%2FKkhOb6I8lChI2QQoyHr%2B9U%3D92add88ce105d8b3ec1dd72efa1dd7ec9b9f1e52";
CookieContainer cookiecontainer = new CookieContainer();
string[] cookies = CookieStr.Split(';');
foreach (string cookie in cookies)
cookiecontainer.SetCookies(new Uri("http://saubhagya.gov.in/dashboard/data/dashboard_saubhagya"), cookie);
request.CookieContainer = cookiecontainer;
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
string json = "ci_csrf_token=&state=35&district=638&village=645115&vtype=&discom=&search_text=&uuid=&maptype=states&kyroargs=&drilldownkey=&kyroclickid=&kyrorefreshid=&page=dashboard_saubhagya";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream(), System.Text.Encoding.GetEncoding(response.CharacterSet)))
{
var result = sr.ReadToEnd();
}
I get an encoded/junk string as output.
Requesting for help!

In the header you state that you accept gzip, but the response is not decompressed on your end, so just add:
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
Then you can remove this line because headers will be added automatically:
//request.Headers.Add("Accept-Encoding: gzip, deflate");

C# POST sending Request Payload JSON data - login to nike.com

I'm trying login to nike.com with my C# windows application.
var request = (HttpWebRequest)WebRequest.Create("https://unite.nike.com/login?appVersion=376&experienceVersion=318&uxid=com.nike.commerce.nikedotcom.web&locale=en_US&backendEnvironment=identity&browser=Google%20Inc.&os=undefined&mobile=true&native=false&visit=1&visitor=048a11da-9ba2-4a4b-91c3-e3cc2774694c");
string stringData = "appVersion=376&experienceVersion=318&uxid=com.nike.commerce.nikedotcom.web&locale=en_US&backendEnvironment=identity&browser=Google%20Inc.&os=undefined&mobile=true&native=false&visit=1&visitor=048a11da-9ba2-4a4b-91c3-e3cc2774694c";
var data = Encoding.ASCII.GetBytes(stringData);
request.Method = "POST";
request.ContentType = "text/plain";
request.ContentLength = data.Length;
request.UserAgent = "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Mobile Safari/537.36";
request.Referer = "https://m.nike.com/us/en_us/?l=shop,login_register";
request.Accept = "*/*";
request.Headers.Add("origin", "https://m.nike.com");
request.Headers.Add("cookie", "AnalysisUserId=104.96.91.109.102041520967791940; exp.swoosh.user=%7B%22granted%22%3A0%7D; nike_locale=us/en_us; NIKE_COMMERCE_COUNTRY=US; NIKE_COMMERCE_LANG_LOCALE=en_US; guidU=7d702b13-ba36-4c8a-d743-f726b99d8807; neo.swimlane=52; dreams_sample=44; cicIntercept=1; cicGuestIntercept=1; cto_lwid=a72ef9b0-3206-45ff-91c7-cf41bf80c118; RES_TRACKINGID=303445732649467; ResonanceSegment=1; AMCVS_F0935E09512D2C270A490D4D%40AdobeOrg=1; AKNIKE=CT-2; lls=3; NIKE_COMMERCE_CCR=1520971547200; CONSUMERCHOICE_SESSION=t; CONSUMERCHOICE=us/en_us; bm_sz=BB6980129B3DFBC73F801C679C8953FB~QAAQbVtgaHQolR5iAQAAceRoI+xF8PxoczE2kGatzaDU90mSX4sJNBTWHwhFh7uDbTBpEQHu9AfqzSCrZKBDtteV5n2LhACXIuzQPezvcsAV4beXFH8tNbnpCs2PEvR7OU9em9lBpGFcKorAL8+S5xz0pTfCxfFlBuI/jXzeXOToMSbg4sbv9Z8Cktgy; guidS=eb252880-ba16-468d-f497-070593e9031c; RES_SESSIONID=718716042916119; AMCV_F0935E09512D2C270A490D4D%40AdobeOrg=2121618341%7CMCIDTS%7C17604%7CMCMID%7C50402020119854507093682697974758826217%7CMCAID%7CNONE%7CMCOPTOUT-1521019750s%7CNONE; ak_bmsc=A51447336BE456E9AF9F08FFF8188EAA68605B6DDC27000013CFA85AEF1E1F19~plBWC7o30uP95yZXaGw7lBKJ4dm6/EyPeO9FAJdZdKsRmLIsW/rBraVYu2IUQMMTzazdq8S4a9PTCJGxaaNGIXxedF/Z1NA1A/EdQFeCCKL9nnFtlavdjlwi2eFiKcBQhJ+bcS8k0tuQKJ1cNS2ng6Me92rK7RrBqtg85tNGZytGfTHxJUd7jWWF43tskP9lnsbmau+myiG1dhgpLSG+40zS02iXa3U0uSZv6gLlNxssDgQEJC/orrdIs/XKOxdsHXuqVzq3RsYGyWzaz05xqUrQ==; neo.experiments=%7B%22main%22%3A%7B%223039-interceptor%22%3A%22a%22%2C%223250-interceptor%22%3A%22a%22%7D%2C%22plus%22%3A%7B%7D%2C%22mobile%22%3A%7B%223199-interceptor%22%3A%22a%22%2C%223246-interceptor%22%3A%22a%22%7D%7D; USID=8049283F7E6AF657B21C7DEB135F66F6.sin-239-app-us-0; _abck=71900787CAC90E301D55611832D59B6268605B6DDC2700000720A85A5D67BF01~0~gnbkwgPHLhkJw9twiqxqN8vDHgFrBO4hdMKVy2uNJLY=~-1~-1; s_sess=%20c51%3Dvertical%3B%20s_cc%3Dtrue%3B%20tp%3D5325%3B%20s_ppv%3Dnikecom%25253Ehomepage%252C12%252C12%252C640%3B%20prevList2%3D%3B; slCheck=N; llCheck=N; sls=0; utag_main=_st:1521016412752$ses_id:1521012619059%3Bexp-session; s_pers=%20c58%3Dno%2520value%7C1521016402289%3B%20s_dfa%3Dnikecomprod%7C1521016414639%3B; guidSTimestamp=1521012520794|1521014615096; bm_sv=86FAB72505FE57713A77C0CA3407044F~o/nQM0bVBprEbVN+2VGk4ea4sVZcujD42BoLvCRxpu5NKwgMTO3oYvgHyrOfOp6h8hv1tI0+3tH/0i3tM3eX8PCQLcibu6uf3iTtPDKnHFTtfDkN+8R4UN0nD0p7L9RGwrkRLuzauSMSonmJHxzCD1HCzVft9Pjx0r02SuViioM=; ppd=profile%7Cnikecom%3Eprofile%3Elogin%20start; _uetsid=_uetef349e8d; bc_nike_triggermail=%7B%22distinct_id%22%3A%20%2216220f6b312234-0cca71d41deec1-b353461-1fa400-16220f6b3134a6%22%7D");
request.Headers.Add("accept-language", "en,hr;q=0.9");
request.Headers.Add("accept-encoding", "gzip, deflate, br");
request.Headers.Add("authority", "unite.nike.com");
request.Headers.Add("path", "/login?appVersion=376&experienceVersion=318&uxid=com.nike.commerce.nikedotcom.web&locale=en_US&backendEnvironment=identity&browser=Google%20Inc.&os=undefined&mobile=true&native=false&visit=1&visitor=048a11da-9ba2-4a4b-91c3-e3cc2774694c");
request.Headers.Add("scheme", "https");
var newStream = request.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
WebResponse response = request.GetResponse();
//MessageBox.Show((((HttpWebResponse)response).StatusCode.ToString()));
StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
string responseFromServer = reader.ReadToEnd();
MessageBox.Show(responseFromServer);
reader.Close();
response.Close();
I don't know how to include Request Payload data in my request
With current code I receive response from server:
The remote server returned an error: (401) Unauthorized.
How to include JSON data in my POST request?

Can't get response from HttpWebRequest in C#

The error says:
"The remote server returned an error:
(http://www.tgv.com.my/movies/man-city-v-arsenal-HO00005174)
Forbidden"
below is my code:
string url = https://translate.google.com/translate_a/single?client=t&sl=en&tl=vi&hl=vi&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t&dt=at&ie=UTF-8&oe=UTF-8&otf=1&srcrom=1&ssel=0&tsel=0&kc=5&tk=520987|10880&q=" + keyword;
var request = (HttpWebRequest)WebRequest.Create(url);
WebProxy proxy = (WebProxy)WebProxy.GetDefaultProxy();
if (proxy.Address != null)
{
proxy.Credentials = proxy.Credentials = new NetworkCredential("username", "pw");
WebRequest.DefaultWebProxy = new System.Net.WebProxy(proxy.Address, proxy.BypassProxyOnLocal, proxy.BypassList, proxy.Credentials);
}
request.Proxy = proxy;
var postData = "";
var data = Encoding.ASCII.GetBytes(postData);
request.Method = "POST";
request.ContentType = "text/html; charset=UTF-8";
request.ContentLength = data.Length;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
return responseString;
Thanks!

Using your Url, it's not a POST request, it's a GET request and could be done like this:
string url = "https://translate.google.com/translate_a/single?client=t&sl=de&tl=en&hl=de&dt=at&dt=bd&dt=ex&dt=ld&dt=md&dt=qca&dt=rw&dt=rm&dt=ss&dt=t&ie=UTF-8&oe=UTF-8&otf=2&ssel=0&tsel=0&kc=4&tk=767774.885916&q=hallo%20du";
var request = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
But if your q should have another value, other values must be changed too, otherwise you will have the Error 403 whitch tells you, that you do not have permission to do your request.
Using Google Translate, consider having a look at the Google Translate API.
There you can do your request like that:
https://www.googleapis.com/language/translate/v2?key=YOUR_API_KEY&q=hello%20world&source=en&target=de
But this is a payd service...

Website Login And Scraping HTML

I'm a little bit stuck here. I am busy making an Windows application that reads data from a website. However de website requires a login first and i don't seem to be able to get passes that. I'm fairly new to programming, so i hope someone know a solution.
This is the code i use to login:
private void btnLogin2_Click(object sender, EventArgs e)
{
HttpWebRequest request = WebRequest.Create(LoginPageURL) as HttpWebRequest;
request.KeepAlive = true;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36";
string postData = "j_username=" + number + "&j_password=" + password;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request.ContentLength = dataBytes.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request.GetResponse() as HttpWebResponse;
request = WebRequest.Create(Page2URL) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(httpResponse.Cookies);
request.Method = "GET";
HttpWebResponse httpResponse2 = request.GetResponse() as HttpWebResponse;
StreamReader stream = new StreamReader(httpResponse2.GetResponseStream(), System.Text.Encoding.UTF8);
string result = stream.ReadToEnd();
stream.Close();
tbOutput2.Text = result;
}
The point is that i get the HTML of the page after the login (Page2URL). But i keep getting the HTML from the Login Page.

You are adding cookies to the request from first response:
request.CookieContainer.Add(httpResponse.Cookies);
Probably the cookies in response are null! To cope up with this issue, read cookie values from response header and add them to the next request like this:
string response_header_cookies = response.Headers.Get("Set-Cookie")
req.Headers.Add("Cookie",response_header_cookies);
In most of the cases this is the more efficient way. Hope this helps!
Source: msdn

C# Send POST request and receive 303 statuscode

I'm writing simple application which connects to webpage and receives specific data from document's body.
I'm sending HttpWebRequest in this way:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://webpage.com/page1.php");
request.Method = "POST";
string postData = "some_post_data";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.Referer = "http://webpage.com/";
request.ContentLength = byteArray.Length;
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31";
request.Headers.Add("Cookie", string.Format("sess_id={0}", UserInfo.SessionId));
request.Headers.Add("Accept-Charset", "ISO-8859-2,utf-8;q=0.7,*;q=0.3");
request.Headers.Add("Accept-Language", "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4");
request.Headers.Add("Accept-Encoding", "gzip");
request.AutomaticDecompression = DecompressionMethods.GZip;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close(); // *** [1] ***
HttpWebResponse response = (HttpWebResponse)request.GetResponse(); // *** [2] ***
dataStream = response.GetResponseStream();
response.Close();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
Fine! The request is ok, there is nothing magic. The point is, after dataStream.Close(); (line [1]) I receive 303 status code, which means there will be redirect to another page. When I read response at point [2] I see new (redirected) page.
All I want is to read headers before redirect. Is it possible anyway?

Set the AllowAutoRedirect property of request to false prior to calling GetResponse.
request.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
More information.