Develop Reference

One specific site which Http Response (hebrew) characters do not come property encoded

The following has been amusing me for a while now.
First of all, I have been scraping sites for a couple of months. Among them hebrew sites as well, and had no problem whatsoever in receiving hebrew characters from the http server.
For some reason I am very curious to sort out, the following site is an exception. I can't get the characters properly encoded. I tried emulating the working requests I do via Fiddler, but to no avail. My c# request headers look exactly the same, but still the characters will not be readable.
What I do not understand is why I have always been able to retrieve hebrew characters from other sites, while from this one specifically I am not. What is this setting that is causing this.
Try the following sample out.
HttpClient httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0");
//httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html;q=0.9");
//httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.5");
//httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
var getTask = httpClient.GetStringAsync("http://winedepot.co.il/Default.asp?Page=Sale");
//doing it like this for the sake of the example
var contents = getTask.Result;
//add a breakpoint at the following line to check the contents of "contents"
Console.WriteLine();
As mentioned, such code works for any other israeli site I try - say, Ynet news site, for instance.
Update: I figured out while "debugging" with Fiddler that the response object, for the ynet site (one which works), returns the header
Content-Type: text/html; charset=UTF-8
while this header is absent in the response from winedepot.co.il
I tried adding it, but still made no difference.
var getTask = httpClient.GetAsync("http://www.winedepot.co.il");
var response = getTask.Result;
var contentObj = response.Content;
contentObj.Headers.Remove("Content-Type");
contentObj.Headers.Add("Content-Type", "text/html; charset=UTF-8");
var readTask = response.Content.ReadAsStringAsync();
var contents = readTask.Result;
Console.WriteLine();

The problem you're encountering is that the webserver is lying about its content-type, or rather, not being specific enough.
The first site responds with this header:
Content-Type: text/html; charset=UTF-8
The second one with this header:
Content-Type: text/html
This means that in the second case, your client will have to make assumptions about what encoding the text is actually in. To learn more about text encodings, please read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).
And the built-in HTTP clients for .NET don't really do a great job at this, which is understandable, because it is a Hard Problem. Read the linked article for the trouble a web browser will have to go through in order to guess the encoding, and then try to understand why you don't want this logic in a programmable web client.
Now the sites do provide you with a <meta http-equiv="Content-Type" content="actual encoding here" /> tag, which is a nasty workaround for not having to properly configure a web server. When a browser encounters such a tag, it will have to restart parsing the document with the specified content-type, and then hope it is correct.
The steps roughly are, assuming an HTML payload:
Perform web request, keep the response document in a binary buffer.
Inspect the content-type header, if present, and if it isn't present or doesn't provide a charset, do some assumption about the encoding.
Read the response by decoding the buffer, and parsing the resulting HTML.
When encountering a <meta http-equiv="Content-Type" /> header, discard all decoded text, and start again by interpreting the binary buffer as text encoded in the specified encoding.
The C# HTTP clients stop at step 2, and rightfully so. They are HTTP clients, not HTML-displaying browsers. They don't care that your payload is HTML, JSON, XML, or any other textual format.
When no charset is given in the content-type response header, the .NET HTTP clients default to the ISO-8859-1 encoding, which cannot display the characters from the character set Windows-1255 (Hebrew) that the page actually is encoded in (or rather, it has different characters at the same code points).
Some C# implementations that try to do encoding detection from the meta HTML element are provided in Encoding trouble with HttpWebResponse. I cannot vouch for their correctness, so you'll have to try it at your own risk. I do know that the currently highest-voted answer actually re-issues the request when it encounters the meta tag, which is quite silly, because there is no guarantee that the second response will be the same as the first, and it's just a waste of bandwidth.
You can also do some assumption about that you know the encoding being used for a certain site or page, and then force the encoding to that:
using (Stream resStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(resStream, YourFixedEncoding);
string content = reader.ReadToEnd();
}
Or, for HttpClient:
using (var client = new HttpClient())
{
var response = await client.GetAsync(url);
var responseStream = await client.ReadAsStreamAsync();
using (var fixedEncodingReader = new StreamReader(responseStream, Encoding.GetEncoding(1255)))
{
string responseString = fixedEncodingReader.ReadToEnd();
}
}
But assuming an encoding for a particular response, or URL, or site, is entirely unsafe altogether. It is in no way guaranteed that this assumption will be correct every time.

Asp .net server support for RFC 6266 and filename*

I am using ASP.NET server on .NET 4.5 and client is C# HttpClient on WinRT platform. I want to upload files using the HttpClient and used System.Net.Http.MultipartFormDataContent class to construct a valid http request. Everything worked fine until I had a filename with DBCS characters.
MultiPartFormDataContent class correctly encodes characters in the uploaded filename and sends both filename and filename* keys as per RFC 6266 in the content disposition header.
However, ASP.NET server ignores the filename* and read filename only and hence the file gets saved on the server with weird characters.
Has someone else faced the same problem? How can I get filename* at the server end and ignore filename key from the HttpRequest? [This would be my preferred solution. ]
Alternatively, how can I force MultiPartFormDataContent to send filename key only and force set UTF-8 encoded string?

Add a reference to System.Net.Http and do something like below...
string suggestedFileName;
string dispositionString = response.GetResponseHeader("Content-Disposition");
if (dispositionString.StartsWith("attachment")) {
System.Net.Http.Headers.ContentDispositionHeaderValue contentDisposition = System.Net.Http.Headers.ContentDispositionHeaderValue.Parse(dispositionString);
if (!string.IsNullOrEmpty(contentDisposition.FileNameStar))
{
suggestedFileName = contentDisposition.FileNameStar;
}
else
{
suggestedFileName = contentDisposition.FileName.Trim('"');
}
}
ContentDispositionHeaderValue From Microsoft

Late to the party..
With control over both the client and server, my (dirty) workaround was simply to always Base64-encode the filename in HttpClient when creating the content, and decode it again on the server side.
This way you avoid having to deal with the aptly named FileNameStar.
You could also try manually detecting the FileName encoding and decode it on the server.
Related thread: System.Net.Mail and =?utf-8?B?XXXXX.... Headers

HttpWebRequest not being received correctly in MVC ASP.NET

This is me publicly documenting my mistake so that if I or anyone does it again, they don't have to spend 3 hours tearing their hair out trying to fix such a simple thing.
Context
I was sending an HttpRequest from one C# MVC ASP.NET application to another.
The applications require an HTTPS connection, and we are using URLRewrite to redirect an HTTP request to an HTTPS url.
One application was sending a POST request with some JSON data in the body, pretty standard stuff. The other application was set up to receive this data with an MVC controller class (CollectionAction and Insert methods for GET and POST respectively).
Symptoms of the problem
The receiving application was running the GET method (CollectionAction) instead of the POST action (ItemAction). The reason for this was that the request coming in to the application was in fact a GET request, and to top it off the JSON data was missing too.
I sent the header "x-http-method" to override the request method from GET to POST (I was already setting the request httpmethod to POST but this was being ignored). This worked but still I had no data being sent.
So now I am stuck pulling my hair out, because I can see a POST request with content-length and data being sent out and I have a GET request with no data or content-length coming in (but the headers were preserved)

Turns out I was using UriBuilder to take a base URL and apply a resource path to it. For example I would have "google.com" in my web.config and then the UriBuilder would take a resource like Pages and construct the url "google.com/Pages". Unfortunately, I was not initializing the UriBuilder with the base URL, and instead was using a second UriBuilder to extract the host and add that to the path like so:
public Uri GetResourceUri(string resourceName)
{
var domain = new UriBuilder(GetBaseUrl());
var uribuilder = new UriBuilder()
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
Host = domain.Host
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
The problem with this code is that the scheme is ignored (HTTP:// vs HTTPS://) and it defaults to HTTP. So my client was sending out the request to an HTTP url instead of the required HTTPS url. This is the interesting part, URLRewrite was kicking in and saying that we needed to go to an HTTPS url instead so it redirected us there. But in doing so, it ignored the Http-Method and the POST data, which just got set to defaults GET and null. This is what the 2nd application could see at the receiving end.
So the function had to be rewritten to this which fixed the problem:
public Uri GetResourceUri(string resourceName)
{
var baseUrl = GetBaseUrl();
var domain = new UriBuilder(baseUrl);
var uribuilder = new UriBuilder(baseUrl)
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}

C#: Sending HTTP GET request without UTF-8 encoding

I need to send HTTP GET request from C# to CLASSIC ASP service.
The service is built in a way that it decodes the data from the QueryString using Windows-1255 encoding, rather than the standard UTF-8.
It seems that HttpWebRequest class always encodes GET data with UTF-8 and it doesn't work for me. Is there any way to send HTTP GET request from C#, while GET data is encoded with different than UTF-8 encoding?
Thanks.

You need to set a header on your get request:
Content-Type:text/xml; Charset=windows-1255
HttpRequest r = new HttpRequest(.....);
r.Headers.Add("Content-Type", "text/xml; Charset=windows-1255");
Maybe this post will be of some use too:
Read non-english characters from http get request

Ok, I finally got the answer.
First of all, specifying ContentType in the header doesn't work.
If destination URL is containing none-English letters, the HttpWebRequest will always use UTF-8 + URLEncode to build the final URI the request is sent to.
To use encoding different from UTF-8 I needed to encode URL values by myself (instead of providing necessary encoding to HttpWebRequest as I expected).
Following function that builds HTTP GET URL, while values are encoded with any requested encoding (and not with the default UTF-8):
string BuildData(NameValueCollection getData, Encoding enc)
{
StringBuilder urldata = new StringBuilder();
for (int i = 0; i < getData.Count; i++)
{
if (i > 0) urldata.Append("&");
urldata.Append(getData.Keys[i] + "=" + HttpUtility.UrlEncode(enc.GetBytes(getData[i])));
}
return urldata.ToString();
}
The HttpWebRequest can be used with something like
"http://get-destination.com/submit?" + BuildData(keysAndValues, Encoding.GetEncoding(1255));
In this case HttpWebRequest gets already encoded URL which doesn't contain none-English letters and it keeps it as is.

Why is my S3 pre-signed request invalid when I set a response header override that contains a "+"?

I'm using the Amazon .NET SDK to generate a pre-signed URL like this:
public System.Web.Mvc.ActionResult AsActionResult(string contentType, string contentDisposition)
{
ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides();
headerOverrides.ContentType = contentType;
if (!string.IsNullOrWhiteSpace(contentDisposition))
{
headerOverrides.ContentDisposition = contentDisposition;
}
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(objectKey)
.WithProtocol(Protocol.HTTPS)
.WithExpires(DateTime.Now.AddMinutes(6))
.WithResponseHeaderOverrides(headerOverrides);
string url = S3Client.GetPreSignedURL(request);
return new RedirectResult(url, permanent: false);
}
This works perfectly, except if my contentType contains a + in it. This happens when I try to get an SVG file, for example, which gets a content type of image/svg+xml. In this case, S3 throws a SignatureDoesNotMatch error.
The error message shows the StringToSign like this:
GET 1234567890 /blah/blabh/blah.svg?response-content-disposition=filename="blah.svg"&response-content-type=image/svg xml
Notice there's a space in the response-content-type, where it now says image/svg xml instead of image/svg+xml. It seems to me like that's what is causing the problem, but what's the right way to fix it?
Should I be encoding my content type? Enclose it within quotes or something? The documentation doesn't say anything about this.

Update
This bug has been fixed as of Version 1.4.1.0 of the SDK.
Workaround
This is a confirmed bug in the AWS SDK, so until they issue a fix I'm going with this hack to make things work:
Specify the content type exactly how you want it to look like in the response header. So, if you want S3 to return a content type of image/svg+xml, set it exactly like this:
ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides();
headerOverrides.ContentType = "image/svg+xml";
Now, go ahead and generate the pre signed request as usual:
GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
.WithBucketName(bucketName)
.WithKey(objectKey)
.WithProtocol(Protocol.HTTPS)
.WithExpires(DateTime.Now.AddMinutes(6))
.WithResponseHeaderOverrides(headerOverrides);
string url = S3Client.GetPreSignedURL(request);
Finally, "fix" the resulting URL with the properly URL encoded value for your content type:
url = url.Replace(contentType, HttpUtility.UrlEncode(contentType));
Yes, it's a dirty workaround but, hey, it works for me! :)

Strange indeed - I've been able reproduce this easily, with the following observed behavior:
replacing + in the the URL generated by GetPreSignedURL() with its encoded form %2B yields a working URL/signature
this holds true, no matter whether / is replaced with its encoded form %2F or not
encoding the contentType upfront before calling GetPreSignedURL(), e.g. via the HttpUtility.UrlEncode Method, yields invalid signatures regardless of any variation of the generated URL
Given how long this functionality is available already, this is somewhat surprising, but I'd still consider it to be a bug - accordingly it might be best to inquiry about this in the Amazon Simple Storage Service forum.
Update: I just realized you asked the same question there already and the bug got confirmed indeed, so the correct answer can be figured out over time by monitoring the AWS team response ;)
Update: This bug has been fixed as of Version 1.4.1.0 of the SDK.