Develop Reference

Why Uri.TryCreate throws NRE when url contains Turkish character?

I have encountered an interesting situation where I get NRE from Uri.TryCreate method when it's supposed to return false.
You can reproduce the issue like below:
Uri url;
if (Uri.TryCreate("http:Ç", UriKind.RelativeOrAbsolute, out url))
{
Console.WriteLine("success");
}
I guess it's failing during the parse, but when I try "http:A" for example, it returns true and parses it as relative url. Even if fails on parse it should just return false as I understand, what could be the problem here? This seems like a bug in the implementation cause documentation doesn't mention about any exception on this method.
The error occurs in .NET 4.6.1 but not 4.0

This is a bug in the .NET framework. You can open a ticket on MicrosoftConnect.
The exception will be raised in this method
void Systen.Uri.CreateUriInfo(System.Uri.Flags cF)
on line 2290 (inspect the reference source) executing following statement:
// This is NOT an ImplicitFile uri
idx = (ushort)m_Syntax.SchemeName.Length;
At this time, the m_Syntax object will be null, because during parsing, it will be discarded.
Method
void InitializeUri(ParsingError err, UriKind uriKind, out UriFormatException e)
line 121:
if (m_Syntax.IsSimple)
{
if ((err = PrivateParseMinimal()) != ParsingError.None)
{
if (uriKind != UriKind.Absolute && err <= ParsingError.LastRelativeUriOkErrIndex)
{
// RFC 3986 Section 5.4.2 - http:(relativeUri) may be considered a valid relative Uri.
m_Syntax = null; // convert to relative uri
e = null;
m_Flags &= Flags.UserEscaped; // the only flag that makes sense for a relative uri
}
// ...
}
// ...
}
The PrivateParseMinimal() method returns ParsingError.BadAuthority and uriKind == UriKind.RelativeOrAbsolute by your specification.
The PrivateParseMinimal() method looks for any of the following character sequences: "//", "\", "/\", "/". And since there are no such sequences in your input string, a ParsingError.BadAuthority code will be returned.

NBitcoin throws InvalidOperationException with the message: "Mac HMACSHA256 not recognised."

I got the message when I try to sign a transaction:
Transaction payment = new Transaction();
BitcoinSecret PaymentSecret = new BitcoinSecret("1sXCvdpXz...UqkXW9mvT");
...
payment.Sign(Container.PaymentSecret, false);
I dig into the opensource NBitcoin API and figured out these lines give me the error message. What can I do? (https://github.com/NicolasDorier/NBitcoin/blob/master/NBitcoin/Crypto/DeterministicECDSA.cs)
try
{
hmac = MacUtilities.GetMac(macName);
}
catch(SecurityUtilityException nsae)
{
throw new InvalidOperationException(nsae.Message, nsae);
}

If someone would like to figure out what is happening exactly, here's a code snippet that'll cause the bug:
string mechanism = "HMACSHA256";
if (mechanism.StartsWith("HMAC"))
{
Console.WriteLine("good");
}
else
{
Console.WriteLine("bad");
}
Console.ReadLine();
If you set mechanism = "HMAC-SHA256", then the bug won't happen.
If you use mechanism.StartsWith("HMAC",StringComparison.InvariantCulture), then bug won't happen.
I've also fixed the bug in github and created a pull request to the NBitcoin API, so hopefully it won't happen with others in the future.

The problem was here:
NBitcoin/NBitcoin.BouncyCastle/security/MacUtilities.cs
public static IMac GetMac(string algorithm)
{
...
if(mechanism.StartsWith("HMAC"))
{
...
}
...
}
The mechanism string is "HMACSHA256" and the if statement never evaluates true for me, because my language is Hungarian and "CS" is a letter in Hungarian. So according to the StartsWith function "HMACSHA256" doesn't start with "HMAC" in Hungarian.
The issue has been fixed in the NBitcoin API by Nicolas Dorier as adding StringComparison.OrdinalIgnoreCase setting to the StartWith function.
In case someone wants to test it, here's an email from Nicolas:
Ok, for history.
Thread.CurrentThread.CurrentCulture = new CultureInfo("hu");
string mechanism = "HMACSHA256";
var v1 = mechanism.StartsWith("HMAC");
mechanism = "HMAC-SHA256";
var v2 = mechanism.StartsWith("HMAC");
In hungarian, v1 is false and v2 true.

There is a hyphen after HMAC. HMAC-SHA256

I just tried the code on my machine, and it works perfectly.
public void CanSignSimple()
{
Key bob = new Key();
Transaction tx = new Transaction();
tx.Inputs.Add(new TxIn()
{
ScriptSig = bob.ScriptPubKey
});
tx.Sign(bob, false);
}
It should be noted that I am not using Bouncy castle library now. I copied the needed part of bouncycastle INSIDE NBitcoin.
Your bug seems like you are using NBitcoin along with the official BouncyCastle, which can happen only if you recompiled everything yourself with using the official BouncyCastle lib, OR you are using an old version of NBitcoin.
What version are you using ?

Get specific subdomain from URL in foo.bar.car.com

Given a URL as follows:
foo.bar.car.com.au
I need to extract foo.bar.
I came across the following code :
private static string GetSubDomain(Uri url)
{
if (url.HostNameType == UriHostNameType.Dns)
{
string host = url.Host;
if (host.Split('.').Length > 2)
{
int lastIndex = host.LastIndexOf(".");
int index = host.LastIndexOf(".", lastIndex - 1);
return host.Substring(0, index);
}
}
return null;
}
This gives me like foo.bar.car. I want foo.bar. Should i just use split and take 0 and 1?
But then there is possible wwww.
Is there an easy way for this?

Given your requirement (you want the 1st two levels, not including 'www.') I'd approach it something like this:
private static string GetSubDomain(Uri url)
{
if (url.HostNameType == UriHostNameType.Dns)
{
string host = url.Host;
var nodes = host.Split('.');
int startNode = 0;
if(nodes[0] == "www") startNode = 1;
return string.Format("{0}.{1}", nodes[startNode], nodes[startNode + 1]);
}
return null;
}

I faced a similar problem and, based on the preceding answers, wrote this extension method. Most importantly, it takes a parameter that defines the "root" domain, i.e. whatever the consumer of the method considers to be the root. In the OP's case, the call would be
Uri uri = "foo.bar.car.com.au";
uri.DnsSafeHost.GetSubdomain("car.com.au"); // returns foo.bar
uri.DnsSafeHost.GetSubdomain(); // returns foo.bar.car
Here's the extension method:
/// <summary>Gets the subdomain portion of a url, given a known "root" domain</summary>
public static string GetSubdomain(this string url, string domain = null)
{
var subdomain = url;
if(subdomain != null)
{
if(domain == null)
{
// Since we were not provided with a known domain, assume that second-to-last period divides the subdomain from the domain.
var nodes = url.Split('.');
var lastNodeIndex = nodes.Length - 1;
if(lastNodeIndex > 0)
domain = nodes[lastNodeIndex-1] + "." + nodes[lastNodeIndex];
}
// Verify that what we think is the domain is truly the ending of the hostname... otherwise we're hooped.
if (!subdomain.EndsWith(domain))
throw new ArgumentException("Site was not loaded from the expected domain");
// Quash the domain portion, which should leave us with the subdomain and a trailing dot IF there is a subdomain.
subdomain = subdomain.Replace(domain, "");
// Check if we have anything left. If we don't, there was no subdomain, the request was directly to the root domain:
if (string.IsNullOrWhiteSpace(subdomain))
return null;
// Quash any trailing periods
subdomain = subdomain.TrimEnd(new[] {'.'});
}
return subdomain;
}

You can use the following nuget package Nager.PublicSuffix. It uses the PUBLIC SUFFIX LIST from Mozilla to split the domain.
PM> Install-Package Nager.PublicSuffix
Example
var domainParser = new DomainParser();
var data = await domainParser.LoadDataAsync();
var tldRules = domainParser.ParseRules(data);
domainParser.AddRules(tldRules);
var domainName = domainParser.Get("sub.test.co.uk");
//domainName.Domain = "test";
//domainName.Hostname = "sub.test.co.uk";
//domainName.RegistrableDomain = "test.co.uk";
//domainName.SubDomain = "sub";
//domainName.TLD = "co.uk";

private static string GetSubDomain(Uri url)
{
if (url.HostNameType == UriHostNameType.Dns)
{
string host = url.Host;
String[] subDomains = host.Split('.');
return subDomains[0] + "." + subDomains[1];
}
return null;
}

OK, first. Are you specifically looking in 'com.au', or are these general Internet domain names? Because if it's the latter, there is simply no automatic way to determine how much of the domain is a "site" or "zone" or whatever and how much is an individual "host" or other record within that zone.
If you need to be able to figure that out from an arbitrary domain name, you will want to grab the list of TLDs from the Mozilla Public Suffix project (http://publicsuffix.org) and use their algorithm to find the TLD in your domain name. Then you can assume that the portion you want ends with the last label immediately before the TLD.

I would recommend using Regular Expression. The following code snippet should extract what you are looking for...
string input = "foo.bar.car.com.au";
var match = Regex.Match(input, #"^\w*\.\w*\.\w*");
var output = match.Value;

In addition to the NuGet Nager.PubilcSuffix package specified in this answer, there is also the NuGet Louw.PublicSuffix package, which according to its GitHub project page is a .Net Core Library that parses Public Suffix, and is based on the Nager.PublicSuffix project, with the following changes:
Ported to .NET Core Library.
Fixed library so it passes ALL the comprehensive tests.
Refactored classes to split functionality into smaller focused classes.
Made classes immutable. Thus DomainParser can be used as singleton and is thread safe.
Added WebTldRuleProvider and FileTldRuleProvider.
Added functionality to know if Rule was a ICANN or Private domain rule.
Use async programming model
The page also states that many of above changes were submitted back to original Nager.PublicSuffix project.

HttpWebRequest Url escaping

I know, the title sounds like this question has been addressed many times. But I am struggling with a specific case and I am very confused over it. Hopefully a seasoned C#'er could point me in the correct direction.
I have the code:
string serviceURL = "https://www.domain.com/service/tables/bucketname%2Ftables%2Ftesttable/imports";
HttpWebRequest dataRequest = (HttpWebRequest)WebRequest.Create(serviceURL);
Now when I quickwatch dataRequest, I see that:
RequestUri: {https://www.domain.com/service/tables/bucketname/tables/testtable/imports}
And it looks like the HttpWebRequest has changed both the %2F to /. However, the server needs the requested Uri to be exactly as serviceURL is written, containing the %2F.
Is there any way to get the HttpWebRequest class to call the Url:
https://www.domain.com/service/tables/bucketname%2Ftables%2Ftesttable/imports
Many thanks! I am at a complete loss here...
-Brett

Kyle posted the answer in a comment, so to make it official:
GETting a URL with an url-encoded slash
It's a weird work around, but nevertheless gets the job done.

As long as the problem lies in %2F being unescaped to "/" there are solutions out there. One involving a hack and for newer versions of .Net, an app.config setting. Check here: How to make System.Uri not to unescape %2f (slash) in path?
However I have still to figure out how to prevent it unescaping some specifically escaped characters, like '(' and ')' (%28 and %29). I have tried all the settings and hacks that I found out there to prevent the Uri class from delivering a partially unescaped path for the WebRequest. The solutions will happily prevent %2F being unescaped, but not %28 and %29 and possible most of the other chars being specifically escaped.
It seems like the WebRequest is specifically asking for 1 value from the Uri object to create the "GET /path HTTP/1.1" syntax: Uri.PathAndQuery which again calls its UriParser.GetComponents.
If you want to download from mediafire and it contains the chars %28 and %29 you will get into a infinite redirect loop as .Net keeps changing %28 and %29 to '(' and ')' and following the redirect (exception: "Too many automatic redirections were attempted").
So this is a solution for those who are stuck and have not been able to find a way to prevent the unescape of some characters.
The only way I have found to override this (currenly using .Net 4.6) and deliver my own PathAndQuery has been a combination of inherting UriParser and hacking its use.
public sealed class MyUriParser : System.UriParser
{
private UriParser _originalParser;
private MethodInfo _getComponentsMethod;
public MyUriParser(UriParser originalParser) : base()
{
if (_originalParser == null)
{
_originalParser = originalParser;
_getComponentsMethod = typeof(UriParser).GetMethod("GetComponents", BindingFlags.NonPublic | BindingFlags.Instance);
if (_getComponentsMethod == null)
{
throw new MissingMethodException("UriParser", "GetComponents");
}
}
}
private static Regex rx = new Regex(#"^(?<Scheme>[^:]+):(?://((?<User>[^#/]+)#)?(?<Host>[^#:/?#]+)(:(?<Port>\d+))?)?(?<Path>([^?#]*)?)?(\?(?<Query>[^#]*))?(#(?<Fragment>.*))?$",RegexOptions.Compiled | RegexOptions.ExplicitCapture | RegexOptions.Singleline);
private Match m = null;
protected override string GetComponents(Uri uri, UriComponents components, UriFormat format)
{
var original = (string)_getComponentsMethod.Invoke(_originalParser, BindingFlags.InvokeMethod, null, new object[] { uri, components, format }, null);
if (components == UriComponents.PathAndQuery)
{
var reg = rx.Match(uri.OriginalString);
var path = reg.Groups["Path"]?.Value;
var query = reg.Groups["Query"]?.Value;
if (path != null && query != null) return $"{path}?{query}";
if (query == null) return $"{path}";
return $"{path}";
}
return original;
}
}
And then hacking it into the Uri instance by replacing its UriParser with this one.
public static Uri CreateUri(string url)
{
var uri = new Uri(url);
if (url.Contains("%28") || url.Contains("%29"))
{
var originalParser = ReflectionHelper.GetValueByReflection(uri, "m_Syntax") as UriParser;
var parser = new MyUriParser(originalParser);
ReflectionHelper.SetValueByReflection(parser, "m_Scheme", "http");
ReflectionHelper.SetValueByReflection(parser, "m_Port", 80);
ReflectionHelper.SetValueByReflection(uri, "m_Syntax", parser);
}
return uri;
}
Due to the way UriParser works, it normally needs to register to have its port and scheme name set, so these 2 values has to be set by reflection as we are not registering it the correct way. I have not found a way to register "http" as it already exist. The ReflectionHelper is just a class I have but can be quickly replaced with normal reflection code.
Then call it like this:
HttpWebRequest dataRequest = (HttpWebRequest)WebRequest.Create(CreateUri(serviceURL));

string serviceURL = Uri.EscapeUriString("https://www.domain.com/service/tables/bucketname%2Ftables%2Ftesttable/imports");

Alternatives to .NET provided apis regarding uris and urls

I've recently come to the realization that the .NET apis working with URLs and URIs frequently come up short in achieving even basic functionality (atleast easily) including things such as: generating a FQDN url from a relative path, forcing https or back to http, getting the root of the site, combining relative urls properly and so forth.
Are there any alternative libraries out there that have put all of these type of functionality in a simple and reliable project?

I've certainly found myself doing much the same URI-manipulation code more than once, in .NET, but I don't see your cases as places it lacks.
Full URI from relative Uri:
new Uri(base, relative) // (works whether relative is a string or a Uri).
Obtaining the actual FQDN:
string host = uri.Host;
string fqdn = hostEndsWith(".") ? host : host + ".";
Forcing https or back to http:
UriBuilder toHttp = new UriBuilder(someUri);
toHttp.Scheme = "http";
toHttp.Port = 80;
return toHttp.Uri;
UriBuilder toHttps = new UriBuilder(someUri);
toHttps.Scheme = "https";
toHttps.Port = 443;
return toHttps.Uri;
Getting the root of the site:
new Uri(startingUri, "/");
Combining relative urls properly:
new Uri(baseUri, relUri); // We had this one already.
Only two of these are more than a single method call, and of those obtaining the FQDN is pretty obscure (unless rather than wanting the dot-ended FQDN you just wanted the absolute URI, in which case we're back to a single method call).
There is a single method version of the HTTPS/HTTP switching, though it's actually more cumbersome since it calls several properties of the Uri object. I can live with it taking a few lines to do this switch.
Still, to provide a new API one need only supply:
public static Uri SetHttpPrivacy(this Uri uri, bool privacy)
{
UriBuilder ub = new UriBuilder(uri);
if(privacy)
{
ub.Scheme = "https";
ub.Port = 443;
}
else
{
ub.Scheme = "http";
ub.Port = 80;
}
return ub.Uri;
}
I really can't see how an API could possibly be any more concise in the other cases.

XUri is a nice class that is part of the open source project from MindTouch
http://developer.mindtouch.com/en/ref/dream/MindTouch.Dream/XUri?highlight=XUri
This article includes a quick sample on how to use it.
http://blog.developer.mindtouch.com/2009/05/18/consuming-rest-services-and-tdd-with-plug/
I am a fan of it. A little overkill assembly wise if you are going to just use the XUri portion, but there are other really nice things in the library too.

I use a combination of extensions with 'System.IO.Path' object as well.
These are just blurbs for example.
public static Uri SecureIfRemote(this Uri uri){
if(!System.Web.HttpContext.Current.Request.IsSecureConnection &&
!System.Web.HttpContext.Current.Request.IsLocal){
return new Uri......(build secure uri here)
}
return uri;
}
public static NameValueCollection ParseQueryString(Uri uri){
return uri.Query.ParseQueryString();
}
public static NameValueCollection ParseQueryString(this string s)
{
//return
return HttpUtility.ParseQueryString(s);
}