I have problems with characters encoding received from http web response, I receive ? instead é.
I set the encoding to according Content-Type of web page that's text/javascript; charset=ISO-8859;
My code is:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(..);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.Referer = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream, Encoding.GetEncoding("iso-8859-1"));
char[] buf = new char[256];
int count;
StringBuilder buffer = new StringBuilder();
while ((count = sr.Read(buf, 0, 256)) > 0)
{
buffer.Append(buf, 0, count);
}
string responseStr = buffer.ToString();
Console.WriteLine(responseStr);
response.Close();
stream.Close();
sr.Close();
Can you tell me what is wrong with it?
Try adding the following before you make your request:
request.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1");
Btw, you should keep your StreamReader with ISO-8859-1 (instead of UTF8) if you want to try my proposed solution. Good luck!
Have you tried setting it at UTF-8?
Further more you send a referrer which I think you tried to set the UserAgent. The code below is the same as yours, but then does not go over the byte array and sets the useragent and utf8 encoding.
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
using(var response = (HttpWebResponse)request.GetResponse())
using(var stream = response.GetResponseStream())
using (var sr = new StreamReader(stream, Encoding.UTF8))
{
string responseStr = sr.ReadToEnd();
Console.WriteLine(responseStr);
response.Close();
if (stream != null)
stream.Close();
sr.Close();
}
Related
I am trying post some data via HttpWebRequest.
Here is the data:
string data = string.Format("username={0}&password={1}", username, password);
byte[] bytes = Encoding.UTF8.GetBytes(data);
There isn't any difference bewtwen UTF8 and ASCII in this case. The string is pure ASCII chars.
The code below will throw an exception:
using (Stream stream = request.GetRequestStream())
{
writer = new StreamWriter(stream, Encoding.UTF8);
string a = data.Substring(0, 1);
string b = data.Replace(a, string.Empty);
writer.Write(a);
writer.Flush();
writer.Write(b);
writer.Flush();
}//---->Last line with no code but right curly braces. Here's EXACTLY where the ex.StackTrace suggests.
This works perfectly:
using (Stream stream = request.GetRequestStream())
{
writer = new StreamWriter(stream, Encoding.ACSII); //---> from UTF8 to ACSII
// ... the rest is same as before
}
This also works perfectly:
using (Stream stream = request.GetRequestStream())
{
stream.Write(bytes, 0, bytes.Length);
}
The exception is this:
The request was aborted: The request was canceled.
StackTrace:
at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
at System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
at System.Net.ConnectStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at my function at the forementioned line.
The request:
request = (HttpWebRequest)HttpWebRequest.Create("https://-.-/takelogin.php");
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Language", "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2");
request.Headers.Add("Accept-Encoding", "gzip, deflate, br");
request.Referer = "https://-.-/login.php";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = Encoding.UTF8.GetByteCount(data);
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", "1");
I want to know the internal reason in this situation...Any help will be appreciated. Thank you.
Encoding.UTF8 passes a “byte order mark” when used with StreamWriter, often called a BOM.
If you use new UTF8Encoding(false), that will not send the BOM, and things should work.
The Encoding.UTF8 is equivalent to new UTF8Encoding(true), where true is “write the BOM”.
I am building one auto login application for one of my website. I have developed the auto login tool in C#. I have used httpwebrequest to make post call on the login page. I'm facing issue with the compression of the webpage. In some of the computers I'm not getting compressed response while in others I'm getting the response. Can anyone suggest how I can guaranteed get compressed response. The following is the code I have written for my httpwebrequest:
httpRequest.Connection = "keepalive";
httpRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
httpRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
httpRequest.UserAgent = "Mozilla/5.0 (Windows T 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36";
httpRequest.UnsafeAuthenticatedConnectionSharing = true;
httpRequest.Headers.Set("Cache-Control", "no-cache");
httpRequest.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(postdata);
httpRequest.ContentLength = bytes.Length;
requestStream = httpRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Flush();
requestStream.Close();
response = (HttpWebResponse)httpRequest.GetResponse();
HttpWebResponse response2 = response;
switch (response2.StatusCode)
{
case HttpStatusCode.OK:
responseStream = response2.GetResponseStream();
if (response2.ContentEncoding.ToLower().Contains("gzip"))
{
str = new StreamReader(new GZipStream(responseStream, CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else if (response2.ContentEncoding.ToLower().Contains("deflate"))
{
str = new StreamReader(new DeflateStream(responseStream, CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else
{
str = new StreamReader(responseStream, Encoding.UTF8).ReadToEnd();
}
responseStream.Close();
responseStream = null;
break;
}
I'm trying to download this json file to read out some data:
http://ddragon.leagueoflegends.com/cdn/5.7.2/data/en_US/item.json
My problem is now that somehow always the values in the tags attribute is missing.
I already used different methods to retrieve them used different encodings but nothing helped. Also the content length is less than the one which I receive in the browser.
Below are some codes I already tried. As you can see I used the WebClient to download this as a string. Also tried to download it as a file, same result.
Used HttpWebRequest with all header attributes.
Used the HEAD method to receive the content length which is less than in the browser.
I also tried using the MemoryStream and the StreamReader to get the correct data. Nothing worked.
String url = "http://ddragon.leagueoflegends.com/cdn/" + version + "/data/en_US/item.json";
//String json = new WebClient().DownloadString(url);
String json = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
request.ContentType = "text/plain; charset=utf-8 ";
request.Accept = "*/*";
request.Method = "GET";
//Get the headers associated with the request.
WebHeaderCollection myWebHeaderCollection = request.Headers;
myWebHeaderCollection.Add("DNT", "1");
myWebHeaderCollection.Add("Accept-Encoding", "gzip, deflate, sdch");
myWebHeaderCollection.Add("Accept-Language", "de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4");
HttpWebRequest requestHeader = ((HttpWebRequest) WebRequest.Create(url));
requestHeader.Method = "HEAD";
using (WebResponse resp = requestHeader.GetResponse())
{
Console.WriteLine(resp.ContentLength);
}
Stream responseStream;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (responseStream = response.GetResponseStream())
{
if (responseStream != null)
{
Console.WriteLine(response.ContentLength);
Console.WriteLine(response.ContentType);
StreamReader readStream = null;
if (string.IsNullOrEmpty(response.CharacterSet))
{
readStream = new StreamReader(responseStream);
}
else
{
readStream = new StreamReader(responseStream, Encoding.GetEncoding(response.CharacterSet));
}
json = readStream.ReadToEnd();
readStream.Close();
//using (var memoryStream = new MemoryStream())
//{
// responseStream.CopyTo(memoryStream);
// json = Encoding.UTF8.GetString(memoryStream.ToArray());
//}
}
}
Edit:
Sample Outputs:
Browser: http://pastebin.com/nQBj64L0
C# Client: http://pastebin.com/7a0Gxvcg
i using HttpWebRequest/HttpWebResponse to get html document, the code follow was running but i can not encode received stream to html string:
string uri = "https://myfavoritesite.come";
HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(uri);
webrequest.KeepAlive = true;
webrequest.Method = "GET";
webrequest.ContentType = "text/html";
webrequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
//webrequest.Connection = "keep-alive";
webrequest.Host = "cat.sabresonicweb.com";
webrequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webrequest.Headers.Add("Accept-Language", "en-US,en;q=0.5");
webrequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 Firefox/18.0";
HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse();
Console.Write(webresponse.StatusCode);
Stream receiveStream = webresponse.GetResponseStream();
Encoding enc = System.Text.Encoding.GetEncoding(1252);//1252
StreamReader loResponseStream = new
StreamReader(receiveStream, enc);
string Response = loResponseStream.ReadToEnd();
loResponseStream.Close();
webresponse.Close();
Console.Write(Response);
So, i use below code line to test is there successful request.
Console.Write(webresponse.StatusCode);
The result on the screen was OK, it's mean the request was sent but the Response string expose on screen was not html format, it's something so strange like this: #32u%&$&(#*#Eeeuw
By using webrequest.Headers.Add("Accept-Encoding", "gzip, deflate"); you are telling the server that you understand compressed responses. Remove that header and use a normal UTF8 encoding instead of 1252 that you are using. You should then get the proper string. You can just use System.Text.Encoding.UTF8.
I got a problem on posting data by using HttpWebRequest.
There is a string(ie. key1=value1&key2=value2&key3=value3) and I have post it to a site (ie. www.*.com/edit), but ,I don't know why that sometimes it's nothing wrong , but sometimes ,the first key=value1 will be missing, only key2=value&key3=value3 that can find in HttpAnalyzer.
public static string SubmitData(string Url, string FormData, CookieContainer _Cc, string ContentType)
{
Stream RequestStream = null, ResponseStream = null; StreamReader Sr = null;
HttpWebRequest HRequest = (HttpWebRequest)WebRequest.Create(Url);
try
{
HRequest.CookieContainer = _Cc;
HRequest.Method = "POST";
HRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)";
HRequest.ContentType = ContentType;
HRequest.ContentLength = FormData.Length;
//byte[] BFromData = new ASCIIEncoding().GetBytes(FormData);
byte[] BFromData = Encoding.ASCII.GetBytes(FormData);
BFromData = Encoding.Convert(Encoding.ASCII, Encoding.UTF8, BFromData);//ascii → utf8
RequestStream = HRequest.GetRequestStream();
RequestStream.Write(BFromData, 0, BFromData.Length);
//RequestStream.Write(utf8Bytes,0,utf8Bytes.Length );
HttpWebResponse HResponse = (HttpWebResponse)HRequest.GetResponse();
ResponseStream = HResponse.GetResponseStream();
Sr = new StreamReader(ResponseStream, Encoding.UTF8);
return Sr.ReadToEnd();
}
catch
{
return "";
}
finally
{
if (null != RequestStream) RequestStream.Close();
if (null != ResponseStream) ResponseStream.Close();
if (null != Sr) Sr.Close();
}
}
Use Fiddler to see how the request looks like when you click on the form then try using this approach and modify what you need for your request.
public static void PostDataAndDoSomething()
{
string URI = "http://www.something.com";
//make your request payload
string requestBody = String.Format("{{'param1': {0}, 'param2': {1}}}",value1, value2); //json format
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URI); //make request
// set request headers as you need
request.ContentType = "application/json; charset=UTF-8";
request.Accept = "application/json, text/javascript;
request.Method = "POST";
request.UserAgent = "";
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
using (StreamWriter writer = new StreamWriter(request.GetRequestStream()))
{
writer.Write(requestBody); //write your request payload
}
WebResponse response = request.GetResponse();
string jsonData = String.Empty;
using (var reader = new StreamReader(response.GetResponseStream()))
{
jsonData = reader.ReadToEnd();
}
response.Close();
//do something with your data, deserialize, Regex etc....
}