Develop Reference

Strange behavior in Uri-class (.NET)

Why does the Uri class urldecode my url that I send to its contructor and how can I prevent this?
Example (look at the querystring value "options"):
string url = "http://www.example.com/default.aspx?id=1&name=andreas&options=one%3d1%26two%3d2%26three%3d3";
Uri uri = new Uri(url); // http://www.example.com/default.aspx?id=1&name=andreas&options=one=1&two=2&three=3
Update:
// ?id=1&name=andreas&options=one%3d1%26two%3d2%26three%3d3
Request.QueryString["options"] = one=1&two=2&three=3
// ?id=1&name=andreas&options=one=1&two=2&three=3
Request.QueryString["options"] = one=1
This is my problem :)

why exactly?
you can get to the encoded version using url.AbsoluteUri
EDIT
Console.WriteLine("1) " + uri.AbsoluteUri);
Console.WriteLine("2) " + uri.Query);
OUT:
1) http://www.example.com/default.aspx?id=1&name=andreas&options=one%3d1%26two%3d2%26three%3d3
2) ?id=1&name=andreas&options=one%3d1%26two%3d2%26three%3d3

I would expect that from a Uri class. I am quite sure that it still gets you in a good place if you use it with e.g. WebClient class (i.e. WebClient.OpenRead (Uri uri)). What's the problem in your case?

This is how the internal code of .NET behaves - in previous versions you could use another constructor of Uri that accepted boolean value telling if to escape or not, but it has been deprecated.
The only way around it is hackish: accessing some private method directly by means of reflection:
string url = "http://www.example.com/default.aspx?id=1&name=andreas&options=one%3d1%26two%3d2%26three%3d3";
Uri uri = new Uri(url);
MethodInfo mi = uri.GetType().GetMethod("CreateThis", BindingFlags.NonPublic | BindingFlags.Instance);
if (mi != null)
mi.Invoke(uri, new object[] { url, true, UriKind.RelativeOrAbsolute });
This worked for me in quick test, but not ideal as you "hack" into .NET internal code.

HttpWebRequest Url escaping

I know, the title sounds like this question has been addressed many times. But I am struggling with a specific case and I am very confused over it. Hopefully a seasoned C#'er could point me in the correct direction.
I have the code:
string serviceURL = "https://www.domain.com/service/tables/bucketname%2Ftables%2Ftesttable/imports";
HttpWebRequest dataRequest = (HttpWebRequest)WebRequest.Create(serviceURL);
Now when I quickwatch dataRequest, I see that:
RequestUri: {https://www.domain.com/service/tables/bucketname/tables/testtable/imports}
And it looks like the HttpWebRequest has changed both the %2F to /. However, the server needs the requested Uri to be exactly as serviceURL is written, containing the %2F.
Is there any way to get the HttpWebRequest class to call the Url:
https://www.domain.com/service/tables/bucketname%2Ftables%2Ftesttable/imports
Many thanks! I am at a complete loss here...
-Brett

Kyle posted the answer in a comment, so to make it official:
GETting a URL with an url-encoded slash
It's a weird work around, but nevertheless gets the job done.

As long as the problem lies in %2F being unescaped to "/" there are solutions out there. One involving a hack and for newer versions of .Net, an app.config setting. Check here: How to make System.Uri not to unescape %2f (slash) in path?
However I have still to figure out how to prevent it unescaping some specifically escaped characters, like '(' and ')' (%28 and %29). I have tried all the settings and hacks that I found out there to prevent the Uri class from delivering a partially unescaped path for the WebRequest. The solutions will happily prevent %2F being unescaped, but not %28 and %29 and possible most of the other chars being specifically escaped.
It seems like the WebRequest is specifically asking for 1 value from the Uri object to create the "GET /path HTTP/1.1" syntax: Uri.PathAndQuery which again calls its UriParser.GetComponents.
If you want to download from mediafire and it contains the chars %28 and %29 you will get into a infinite redirect loop as .Net keeps changing %28 and %29 to '(' and ')' and following the redirect (exception: "Too many automatic redirections were attempted").
So this is a solution for those who are stuck and have not been able to find a way to prevent the unescape of some characters.
The only way I have found to override this (currenly using .Net 4.6) and deliver my own PathAndQuery has been a combination of inherting UriParser and hacking its use.
public sealed class MyUriParser : System.UriParser
{
private UriParser _originalParser;
private MethodInfo _getComponentsMethod;
public MyUriParser(UriParser originalParser) : base()
{
if (_originalParser == null)
{
_originalParser = originalParser;
_getComponentsMethod = typeof(UriParser).GetMethod("GetComponents", BindingFlags.NonPublic | BindingFlags.Instance);
if (_getComponentsMethod == null)
{
throw new MissingMethodException("UriParser", "GetComponents");
}
}
}
private static Regex rx = new Regex(#"^(?<Scheme>[^:]+):(?://((?<User>[^#/]+)#)?(?<Host>[^#:/?#]+)(:(?<Port>\d+))?)?(?<Path>([^?#]*)?)?(\?(?<Query>[^#]*))?(#(?<Fragment>.*))?$",RegexOptions.Compiled | RegexOptions.ExplicitCapture | RegexOptions.Singleline);
private Match m = null;
protected override string GetComponents(Uri uri, UriComponents components, UriFormat format)
{
var original = (string)_getComponentsMethod.Invoke(_originalParser, BindingFlags.InvokeMethod, null, new object[] { uri, components, format }, null);
if (components == UriComponents.PathAndQuery)
{
var reg = rx.Match(uri.OriginalString);
var path = reg.Groups["Path"]?.Value;
var query = reg.Groups["Query"]?.Value;
if (path != null && query != null) return $"{path}?{query}";
if (query == null) return $"{path}";
return $"{path}";
}
return original;
}
}
And then hacking it into the Uri instance by replacing its UriParser with this one.
public static Uri CreateUri(string url)
{
var uri = new Uri(url);
if (url.Contains("%28") || url.Contains("%29"))
{
var originalParser = ReflectionHelper.GetValueByReflection(uri, "m_Syntax") as UriParser;
var parser = new MyUriParser(originalParser);
ReflectionHelper.SetValueByReflection(parser, "m_Scheme", "http");
ReflectionHelper.SetValueByReflection(parser, "m_Port", 80);
ReflectionHelper.SetValueByReflection(uri, "m_Syntax", parser);
}
return uri;
}
Due to the way UriParser works, it normally needs to register to have its port and scheme name set, so these 2 values has to be set by reflection as we are not registering it the correct way. I have not found a way to register "http" as it already exist. The ReflectionHelper is just a class I have but can be quickly replaced with normal reflection code.
Then call it like this:
HttpWebRequest dataRequest = (HttpWebRequest)WebRequest.Create(CreateUri(serviceURL));

string serviceURL = Uri.EscapeUriString("https://www.domain.com/service/tables/bucketname%2Ftables%2Ftesttable/imports");

HttpModule to add headers to request

This seems like a simple operation.
We have a need in our development environment (running on XP/IIS 5) to add some headers into each HttpRequest arriving at our application. (This is to simulate a production environment that we don't have available in dev). At first blush, this seemed like a simple HttpModule, along the lines of:
public class Dev_Sim: IHttpModule
{
public void Init(HttpApplication app)
{
app.BeginRequest += delegate { app.Context.Request.Headers.Add("UserName", "XYZZY"); };
}
public void Dispose(){}
}
But on trying to do that, I find that the Headers collection of the Request is read-only, and the Add method fails with an OperationNotSupported exception.
Spending a couple hours researching this on Google, I've come up with no easy answer to what should be a relatively straight-forward problem.
Does anyone have any pointers?

Okay, with the assistance of a co-worker and some experimentation, I found that this can be done with the assistance of some protected properties and methods accessed through reflection:
var headers = app.Context.Request.Headers;
Type hdr = headers.GetType();
PropertyInfo ro = hdr.GetProperty("IsReadOnly",
BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.IgnoreCase | BindingFlags.FlattenHierarchy);
// Remove the ReadOnly property
ro.SetValue(headers, false, null);
// Invoke the protected InvalidateCachedArrays method
hdr.InvokeMember("InvalidateCachedArrays",
BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Instance,
null, headers, null);
// Now invoke the protected "BaseAdd" method of the base class to add the
// headers you need. The header content needs to be an ArrayList or the
// the web application will choke on it.
hdr.InvokeMember("BaseAdd",
BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Instance,
null, headers,
new object[] { "CustomHeaderKey", new ArrayList {"CustomHeaderContent"}} );
// repeat BaseAdd invocation for any other headers to be added
// Then set the collection back to ReadOnly
ro.SetValue(headers, true, null);
This works for me, at least.

You can add to the Header this way. This is a way to add credential information to the request before it enter the authentication sequence.
string cred = "UN:PW";
System.Web.HttpContext.Current.Request.Headers.Add("Authorization", "Basic " +Convert.ToBase64String(Encoding.ASCII.GetBytes(cred)));

How to validate xml using a .dtd via a proxy and NOT using system.net.defaultproxy

Someone else has already asked a somewhat similar question: Validate an Xml file against a DTD with a proxy. C# 2.0
Here's my problem: We have a website application that needs to use both internal and external resources.
We have a bunch of internal
webservices. Requests to the CANNOT
go through the proxy. If we try to, we get 404 errors since the proxy DNS doesn't know about our internal webservice domains.
We generate a
few xml files that have to be valid.
I'd like to use the provided dtd
documents to validate the xml. The
dtd urls are outside our network and
MUST go through the proxy.
Is there any way to validate via dtd through a proxy without using system.net.defaultproxy? If we use defaultproxy, the internal webservices are busted, but the dtd validation works.#
Here is what I'm doing to validate the xml right now:
public static XDocument ValidateXmlUsingDtd(string xml)
{
var xrSettings = new XmlReaderSettings {
ValidationType = ValidationType.DTD,
ProhibitDtd = false
};
var sr = new StringReader(xml.Trim());
XmlReader xRead = XmlReader.Create(sr, xrSettings);
return XDocument.Load(xRead);
}
Ideally, there would be some way to assign a proxy to the XmlReader much like you can assign a proxy to the HttpWebRequest object. Or perhaps there is a way to programatically turn defaultproxy on or off? So that I can just turn it on for the call to Load the Xdocument, then turn it off again?
FYI - I'm open to ideas on how to tackle this - note that the proxy is located in another domain, and they don't want to have to set up a dns lookup to our dns server for our internal webservice addresses.
Cheers,
Lance

Yes, you can fix this.
One option is to create your own resolver that handles the DTD resolution. It can use whatever mechanism it likes, including employing a non-default proxy for outbound communications.
var xmlReaderSettings = new XmlReaderSettings
{
ProhibitDtd = false,
ValidationType = ValidationType.DTD,
XmlResolver = new MyCustomDtdResolver()
};
In the code for MyCustomDtdResolver, you'd specify your desired proxy setting. It could vary depending on the DTD.
You didn't specify, but if the DTDs you are resolving against are fixed and unchanging, then Silverlight and .NET 4.0 have a built-in resolver that does not hit the network (no proxy, no http comms whatsoever). It's called XmlPreloadedResolver. Out of the box it knows how to resolve RSS091 and XHTML1.0. If you have other DTDs, including your own custom DTDs, and they are fixed or unchanging, you can load them into this resolver and use it at runtime, and completely avoid HTTP comms and the proxy complication.
More on that.
If you are not using .NET 4.0, then you can build a "no network" resolver yourself. To avoid the W3C traffic limit, I built a custom resolver myself, for XHTML, maybe you can re-use it.
See also, a related link.
For illustration, here's the code for ResolveUri in a custom Uri resolver.
/// <summary>
/// Resolves URIs.
/// </summary>
/// <remarks>
/// <para>
/// The only Uri's supported are those for W3C XHTML 1.0.
/// </para>
/// </remarks>
public override Uri ResolveUri(Uri baseUri, string relativeUri)
{
if (baseUri == null)
{
if (relativeUri.StartsWith("http://"))
{
Trace(" returning {0}", relativeUri);
return new Uri(relativeUri);
}
// throw if Uri scheme is unknown/unhandled
throw new ArgumentException();
}
if (relativeUri == null)
return baseUri;
// both are non-null
var uri = baseUri.AbsoluteUri;
foreach (var key in knownDtds.Keys)
{
// look up the URI in the table of known URIs
var dtdUriRoot = knownDtds[key];
if (uri.StartsWith(dtdUriRoot))
{
string newUri = uri.Substring(0,dtdUriRoot.Length) + relativeUri;
return new Uri(newUri);
}
}
// must throw if Uri is unknown/unhandled
throw new ArgumentException();
}
here's the code for GetEntity
/// <summary>
/// Gets the entity associated to the given Uri, role, and
/// Type.
/// </summary>
/// <remarks>
/// <para>
/// The only Type that is supported is the System.IO.Stream.
/// </para>
/// <para>
/// The only Uri's supported are those for W3C XHTML 1.0.
/// </para>
/// </remarks>
public override object GetEntity(Uri absoluteUri, string role, Type t)
{
// only handle streams
if (t != typeof(System.IO.Stream))
throw new ArgumentException();
if (absoluteUri == null)
throw new ArgumentException();
var uri = absoluteUri.AbsoluteUri;
foreach (var key in knownDtds.Keys)
{
if (uri.StartsWith(knownDtds[key]))
{
// Return the stream containing the requested DTD.
// This can be a FileStream, HttpResponseStream, MemoryStream,
// or whatever other stream you like. I used a Resource stream
// myself. If you retrieve the DTDs via HTTP, you could use your
// own IWebProxy here.
var resourceName = GetResourceName(key, uri.Substring(knownDtds[key].Length));
return GetStreamForNamedResource(resourceName);
}
}
throw new ArgumentException();
}
The full working code for my custom resolver is available.
If your resolver does network comms, then for a general solution you may want to override the Credentials property.
public override System.Net.ICredentials Credentials
{
set { ... }
}
Also, you may want to expose a Proxy property. Or not. As I said above, you may want to automatically determine the proxy to use, from the DTD URI.

Alternative to .NET's Uri implementation?

I have a problem with the .NET's Uri implementation. It seems that if the scheme is "ftp", the query part is not parsed as a Query, but as a part of the path instead.
Take the following code for example:
Uri testuri = new Uri("ftp://user:pass#localhost/?passive=true");
Console.WriteLine(testuri.Query); // Outputs an empty string
Console.WriteLine(testuri.AbsolutePath); // Outputs "/%3Fpassive=true"
It seems to me that the Uri class wrongfully parses the query part as a part of the path. However changing the scheme to http, the result is as expected:
Uri testuri = new Uri("http://user:pass#localhost/?passive=true");
Console.WriteLine(testuri.Query); // Outputs "?passive=true"
Console.WriteLine(testuri.AbsolutePath); // Outputs "/"
Does anyone have a solution to this, or know of an alternative Uri class that works as expected?

Well, the problem is not that I am unable to create a FTP connection, but that URI's are not parsed accoding to RFC 2396.
What I actually intended to do was to create a Factory that provides implementations of a generic File transfer interface (containing get and put methods), based on a given connection URI. The URI defines the protocol, user info, host and path, and any properties needed to be passed should be passed through the Query part of the URI (such as the Passive mode option for the FTP connection).
However this proved difficult using the .NET Uri implementation, because it seems to parse the Query part of URI's differently based on the schema.
So I was hoping that someone knew a workaround to this, or of an alternative to the seemingly broken .NET Uri implementation. Would be nice to know before spending hours implementing my own.

I have been struggling with the same issue for a while. Attempting to replace the existing UriParser for the "ftp" scheme using UriParser.Register throws an InvalidOperationException because the scheme is already registered.
The solution I have come up with involves using reflection to modify the existing ftp parser so that it allows the query string. This is based on a workaround to another UriParser bug.
MethodInfo getSyntax = typeof(UriParser).GetMethod("GetSyntax", System.Reflection.BindingFlags.Static
| System.Reflection.BindingFlags.NonPublic);
FieldInfo flagsField = typeof(UriParser).GetField("m_Flags", System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.NonPublic);
if (getSyntax != null && flagsField != null)
{
UriParser parser = (UriParser)getSyntax.Invoke(null, new object[] { "ftp"});
if (parser != null)
{
int flagsValue = (int)flagsField.GetValue(parser);
// Set the MayHaveQuery attribute
int MayHaveQuery = 0x20;
if ((flagsValue & MayHaveQuery) == 0) flagsField.SetValue(parser, flagsValue | MayHaveQuery);
}
}
Run that somewhere in your initialization, and your ftp Uris will have the query string go into the Query parameter, as you would expect, instead of Path.

You should use the FtpWebRequest and FtpWebResponse classes unless you have a specific reason not to.
FtpWebRequest.fwr = (FtpWebRequest)FtpWebRequest.Create(new Uri("ftp://uri"));
fwr.ftpRequest.Method = WebRequestMethods.Ftp.UploadFile;
fwr.ftpRequest.Credentials = new NetworkCredential("user", "pass");
FileInfo ff = new FileInfo("localpath");
byte[] fileContents = new byte[ff.Length];
using (FileStream fr = ff.OpenRead())
{
fr.Read(fileContents, 0, Convert.ToInt32(ff.Length));
}
using (Stream writer = fwr.GetRequestStream())
{
writer.Write(fileContents, 0, fileContents.Length);
}
FtpWebResponse frp = (FtpWebResponse)fwr.GetResponse();
Response.Write(frp.ftpResponse.StatusDescription);
Ref1 Ref2

You have to use a specific class for FTP protocol like FtpWebRequest that has a Uri property like RequestUri.
You should search in thoses classes for a Uri parser I think.