Develop Reference

HttpClient decoding my encoded URI causes request to fail

HttpClient seems to not respect encoded strings.
var id = Uri.EscapeDataString("\"MyId\"");
var uri = new Uri($#"https://www.example.com/{id}").AbsoluteUri;
var req = new HttpRequestMessage(HttpMethod.Post, uri);
My uri object contains https://www.example.com/%22MyId%22 which is correct.
Inspecting req.RequestUri shows the correct value for every property except for LocalPath, everything else seems to have the correct %22 encoded quote.
I then send the request:
await httpClient.SendAsync(req)
and it makes a request to https://www.example.com/"MyId" (no longer encoded?) and the request fails.
uri.UserEscaped shows false after creating. I tried to escape the entire URI using Uri.EscapeUriString, but that function is deprecated and suggests using Uri.EscapeDataString for the query (which I'm already doing).
I followed the solution from the GitHub issue in this question, however it did not fix my issue (using AbsoluteUri).
This question is incorrect as it is not just the ToString() method, the actual request includes the decoded values.
This question is just a dead link.
How can I actually send my request with the encoded %22 quote values?

Don't do that Uri.EscapeDataString, just include the id as-is in the Uri.
var id = "\"MyId\"";
var uri = new Uri($#"https://www.example.com/{id}").AbsoluteUri;
var req = new HttpRequestMessage(HttpMethod.Post, uri);
Fiddler shows below request was made.
POST https://www.example.com/%22MyId%22

you can use them (namespace System.Web)
HttpUtility.UrlDecode("")
HttpUtility.UrlEncode("")

HttpWebRequest maximum length for Get request url

I've been tasked with building a service which pulls information from a 3rd party API into our internal datawarehouse. They have a get request to pull the data I want where you specify the parameters you want via query strings. E.g.
http://www.api.com?parameter=firstname&parameter=surname
In my code the length of the URL is over 3600 characters long as the requirement is for 116 parameters.
My web request is generated using this code:
private HttpWebRequest GetWebRequest(string url, string type, int timeout)
{
var httpWebRequest = (HttpWebRequest) WebRequest.Create(_baseUrl + url);
httpWebRequest.Method = type;
httpWebRequest.Timeout = timeout;
httpWebRequest.ContentType = "application/json";
httpWebRequest.Headers.Add("Authorization", "Bearer " + _token.access_token);
httpWebRequest.ContentLength = 0;
return httpWebRequest;
}
When I run the code I am getting back a web exception with the message "Unable to connect to the remote server" with an internal exception message of "No connection could be made because the target machine actively refused it IP Address"
I have not included the entire URL in this post as I have found that if I copy and paste the url into Postman and run the request I get the response I expect so I know that the URL is formatted correctly. I have also discovered that if I cut down the length of the url to around 2100 characters the request then works.
I have been searching but have not found any definitive documentation to suggest that there is a limit to the length of the URL, but I can not explain why the whole url works in Postman but not in a c# web request and that if I cut the length of the URL it then works in the web request!
If anyone has any ideas about this I'd be greatfull.

An old post suggests that depending on the server and client the maximum request length is somewhere between 2 - 4 and 8 KB, which is consistent with your observations.
Postman is 'another' client, so it is well possible that it works there while it doesn't in your program. Bottom-line is: you should not GET such long requests. You should POST them instead. If you have control over the service, you could change it so it supports POST too, if not already (documented or not).

C#: Sending HTTP GET request without UTF-8 encoding

I need to send HTTP GET request from C# to CLASSIC ASP service.
The service is built in a way that it decodes the data from the QueryString using Windows-1255 encoding, rather than the standard UTF-8.
It seems that HttpWebRequest class always encodes GET data with UTF-8 and it doesn't work for me. Is there any way to send HTTP GET request from C#, while GET data is encoded with different than UTF-8 encoding?
Thanks.

You need to set a header on your get request:
Content-Type:text/xml; Charset=windows-1255
HttpRequest r = new HttpRequest(.....);
r.Headers.Add("Content-Type", "text/xml; Charset=windows-1255");
Maybe this post will be of some use too:
Read non-english characters from http get request

Ok, I finally got the answer.
First of all, specifying ContentType in the header doesn't work.
If destination URL is containing none-English letters, the HttpWebRequest will always use UTF-8 + URLEncode to build the final URI the request is sent to.
To use encoding different from UTF-8 I needed to encode URL values by myself (instead of providing necessary encoding to HttpWebRequest as I expected).
Following function that builds HTTP GET URL, while values are encoded with any requested encoding (and not with the default UTF-8):
string BuildData(NameValueCollection getData, Encoding enc)
{
StringBuilder urldata = new StringBuilder();
for (int i = 0; i < getData.Count; i++)
{
if (i > 0) urldata.Append("&");
urldata.Append(getData.Keys[i] + "=" + HttpUtility.UrlEncode(enc.GetBytes(getData[i])));
}
return urldata.ToString();
}
The HttpWebRequest can be used with something like
"http://get-destination.com/submit?" + BuildData(keysAndValues, Encoding.GetEncoding(1255));
In this case HttpWebRequest gets already encoded URL which doesn't contain none-English letters and it keeps it as is.

how to read the response from a web site?

I have a website url which gives corresponding city names by taking zip code as input parameter. Now I want to know how to read the response from the site.
This is the link I am using http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680

You'll have to Use the HTTPWebRequest object to connect to the site and scrape the information from the response.
Look for html tags or class names that wrap the content you are trying to find, then use either regexes or string functions to get the required data.
Good example here:

try this (you'll need to include System.text and System.net)
WebClient client = new WebClient();
string url = "http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680";
Byte[] requestedHTML;
requestedHTML = client.DownloadData(url);
UTF8Encoding objUTF8 = new UTF8Encoding();
string html = objUTF8.GetString(requestedHTML);
Response.Write(html);

The simplest way it to use the light-weight WebClient classes in System.Net namespace. The following example code will just download the entire response as a string:
using (WebClient wc = new WebClient())
{
string response = wc.DownloadString("http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680");
}
However, if you require more control over the response and request process then you can use the more heavy-weight HttpWebRequest Class. For instance, you may want to deal with different status codes or headers. There's an example of using HttpWebRequest this in the article How to use HttpWebRequest and HttpWebResponse in .NET on CodeProject.

Used the WebClient Class (http://msdn.microsoft.com/en-us/library/system.net.webclient%28v=VS.100%29.aspx) to request the page and get the response as a string.
WebClient wc = new WebClient();
String s = wc.DownloadString(DestinationUrl);
You can search the response for specific HTML using String.IndexOf, SubString, etc, regular expressions, or try something like the HTML Agility Pack (http://htmlagilitypack.codeplex.com/) which was created specifically to help parse HTML.

first of all, you better find a good Web Service for this purpose.
and this is an HttpWebRequest example:
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680");
httpRequest.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream dataStream = httpResponse.GetResponseStream();

You need to use HttpWebRequest for receiving content and some tools for parsing html and finding what you need. One of the most popular libs for working with html in c# is HtmlAgilityPack, you can see simple example here: http://www.fairnet.com/post/2010/08/28/Html-screen-scraping-with-HtmlAgilityPack-Library.aspx

you can use a WebClient object, and an easy way to scrape the data is with xpath.

Cannot handle redirect from HTTP/HTTPS protocols to other dissimilar ones

Basically, I'm trying to grab an EXE from CNet's Download.com
So i created web parser and so far all is going well.
Here is a sample link pulled directly from their site:
http://dw.com.com/redir?edId=3&siteId=4&oId=3001-20_4-10308491&ontId=20_4&spi=e6323e8d83a8b4374d43d519f1bd6757&lop=txt&tag=idl2&pid=10566981&mfgId=6250549&merId=6250549&pguid=PlvcGQoPjAEAAH5rQL0AAABv&destUrl=ftp%3A%2F%2F202.190.201.108%2Fpub%2Fryl2%2Fclient%2Finstaller-ryl2_v1673.exe
Here is the problem: When you attempt to download, it begins with HTTP, then redirects to an FTP site. I have tried .NET's WebClient and HttpWebRequest Objects, and it looks like Neither can support Redirects.
This Code Fails at GetResponse();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://dw.com.com/redir");
WebResponse response = req.GetResponse();
Now, I also tried this:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://dw.com.com/redir");
req.AllowAutoRedirect = false;
WebResponse response = req.GetResponse();
string s = new StreamReader(response.GetResponseStream()).ReadToEnd();
And it does not throw the error anymore, however variable s turns out to be an empty string.
I'm at a loss! Can anyone help out?

You can get the value of the "Location" header from the response.headers, and then create a new FtpWebRequest to download that resource.

in your first code snippet you will be redirected to a link using a different protocol (i.e it's no longer Http as in HttpWebRequest) so it fails du to a malformed http response.
In the second part you're no longer redirected and hence you don't receive a FTP response (which is not malform when interpreted as HTTP response).
You need to acquire FTP link,as ferozo wrote you can do this by getting the value of the header "location", and use a FtpWebRequest to access the file