ObjectCache not caching json data - c#

I'm trying to cache some data returned as Json, problem is my cache does not seem to be caching the data and I do not know why, can anyone please have a look at my code and let a newbie know where he have gone wrong.
public class GetTravelAdvice : ITravelAdvice
{
private static readonly string Key = ConfigurationManager.AppSettings["FCO_URL"];
public async Task<IEnumerable<TravelAdvice>> GetFcoTravelAdvice(string country)
{
using (var ta = new HttpClient())
{
string cacheDataTravelAdvice = "FCOLocation" + RegexHelpers.RegexRemoveAllInvalidCharactersKeepWhiteSpace(country.ToLower());
ObjectCache travelCache = MemoryCache.Default;
var objectInCache = travelCache.Get(cacheDataTravelAdvice) as IEnumerable<TravelAdvice>;
if (objectInCache != null)
return objectInCache.AsEnumerable();
string ensureCountryStringIsValid = RegexHelpers.RegexRemoveAllInvalidCharactersKeepWhiteSpace(country);
var url = await ta.GetStringAsync(string.Format("{0}{1}{2}", Key, HttpUtility.HtmlEncode(ensureCountryStringIsValid), ".json")).ConfigureAwait(false);
var policy = new CacheItemPolicy { AbsoluteExpiration = DateTime.Now.AddMinutes(15) };
travelCache.Add(cacheDataTravelAdvice, url, policy);
return new[] { JsonConvert.DeserializeObject<TravelAdvice>(url) };
}
}
}
Thanks for any help


On this line the url variable is a string instance, right?
var url = await ta.GetStringAsync(string.Format("{0}{1}{2}", Key, HttpUtility.HtmlEncode(ensureCountryStringIsValid), ".json")).ConfigureAwait(false);
This is what you have stored in the cache: a string value:
travelCache.Add(cacheDataTravelAdvice, url, policy);
and here:
var objectInCache = travelCache.Get(cacheDataTravelAdvice) as IEnumerable<TravelAdvice>;
you are attempting to cast it to IEnumerable<TravelAdvice>. It's normal that if you have stored a string in your cache you cannot expect to get an IEnumerable<TravelAdvice> out from it.
So you need to read it as string:
string objectInCache = travelCache.Get(cacheDataTravelAdvice) as string;
and then deserialize it to the underlying object:
if (objectInCache != null)
{
return new[] { JsonConvert.DeserializeObject<TravelAdvice>(objectInCache) };
}
The other possibility is to directly store your IEnumerable<TravelAdvice> instance into the cache:
var url = await ta.GetStringAsync(string.Format("{0}{1}{2}", Key, HttpUtility.HtmlEncode(ensureCountryStringIsValid), ".json")).ConfigureAwait(false);
var data = new[] { JsonConvert.DeserializeObject<TravelAdvice>(url) };
travelCache.Add(cacheDataTravelAdvice, data, policy);
return data;

Related

Sending a Get request in C# with options doesn't work [duplicate]

If I wish to submit a http get request using System.Net.HttpClient there seems to be no api to add parameters, is this correct?
Is there any simple api available to build the query string that doesn't involve building a name value collection and url encoding those and then finally concatenating them?
I was hoping to use something like RestSharp's api (i.e AddParameter(..))

If I wish to submit a http get request using System.Net.HttpClient
there seems to be no api to add parameters, is this correct?
Yes.
Is there any simple api available to build the query string that
doesn't involve building a name value collection and url encoding
those and then finally concatenating them?
Sure:
var query = HttpUtility.ParseQueryString(string.Empty);
query["foo"] = "bar<>&-baz";
query["bar"] = "bazinga";
string queryString = query.ToString();
will give you the expected result:
foo=bar%3c%3e%26-baz&bar=bazinga
You might also find the UriBuilder class useful:
var builder = new UriBuilder("http://example.com");
builder.Port = -1;
var query = HttpUtility.ParseQueryString(builder.Query);
query["foo"] = "bar<>&-baz";
query["bar"] = "bazinga";
builder.Query = query.ToString();
string url = builder.ToString();
will give you the expected result:
http://example.com/?foo=bar%3c%3e%26-baz&bar=bazinga
that you could more than safely feed to your HttpClient.GetAsync method.

For those who do not want to include System.Web in projects that don't already use it, you can use FormUrlEncodedContent from System.Net.Http and do something like the following:
keyvaluepair version
string query;
using(var content = new FormUrlEncodedContent(new KeyValuePair<string, string>[]{
new KeyValuePair<string, string>("ham", "Glazed?"),
new KeyValuePair<string, string>("x-men", "Wolverine + Logan"),
new KeyValuePair<string, string>("Time", DateTime.UtcNow.ToString()),
})) {
query = content.ReadAsStringAsync().Result;
}
dictionary version
string query;
using(var content = new FormUrlEncodedContent(new Dictionary<string, string>()
{
{ "ham", "Glaced?"},
{ "x-men", "Wolverine + Logan"},
{ "Time", DateTime.UtcNow.ToString() },
})) {
query = content.ReadAsStringAsync().Result;
}

In a ASP.NET Core project you can use the QueryHelpers class, available in the Microsoft.AspNetCore.WebUtilities namespace for ASP.NET Core, or the .NET Standard 2.0 NuGet package for other consumers:
// using Microsoft.AspNetCore.WebUtilities;
var query = new Dictionary<string, string>
{
["foo"] = "bar",
["foo2"] = "bar2",
// ...
};
var response = await client.GetAsync(QueryHelpers.AddQueryString("/api/", query));

TL;DR: do not use accepted version as It's completely broken in relation to handling unicode characters, and never use internal API
I've actually found weird double encoding issue with the accepted solution:
So, If you're dealing with characters which need to be encoded, accepted solution leads to double encoding:
query parameters are auto encoded by using NameValueCollection indexer (and this uses UrlEncodeUnicode, not regular expected UrlEncode(!))
Then, when you call uriBuilder.Uri it creates new Uri using constructor which does encoding one more time (normal url encoding)
That cannot be avoided by doing uriBuilder.ToString() (even though this returns correct Uri which IMO is at least inconsistency, maybe a bug, but that's another question) and then using HttpClient method accepting string - client still creates Uri out of your passed string like this: new Uri(uri, UriKind.RelativeOrAbsolute)
Small, but full repro:
var builder = new UriBuilder
{
Scheme = Uri.UriSchemeHttps,
Port = -1,
Host = "127.0.0.1",
Path = "app"
};
NameValueCollection query = HttpUtility.ParseQueryString(builder.Query);
query["cyrillic"] = "кирилиця";
builder.Query = query.ToString();
Console.WriteLine(builder.Query); //query with cyrillic stuff UrlEncodedUnicode, and that's not what you want
var uri = builder.Uri; // creates new Uri using constructor which does encode and messes cyrillic parameter even more
Console.WriteLine(uri);
// this is still wrong:
var stringUri = builder.ToString(); // returns more 'correct' (still `UrlEncodedUnicode`, but at least once, not twice)
new HttpClient().GetStringAsync(stringUri); // this creates Uri object out of 'stringUri' so we still end up sending double encoded cyrillic text to server. Ouch!
Output:
?cyrillic=%u043a%u0438%u0440%u0438%u043b%u0438%u0446%u044f
https://127.0.0.1/app?cyrillic=%25u043a%25u0438%25u0440%25u0438%25u043b%25u0438%25u0446%25u044f
As you may see, no matter if you do uribuilder.ToString() + httpClient.GetStringAsync(string) or uriBuilder.Uri + httpClient.GetStringAsync(Uri) you end up sending double encoded parameter
Fixed example could be:
var uri = new Uri(builder.ToString(), dontEscape: true);
new HttpClient().GetStringAsync(uri);
But this uses obsolete Uri constructor
P.S on my latest .NET on Windows Server, Uri constructor with bool doc comment says "obsolete, dontEscape is always false", but actually works as expected (skips escaping)
So It looks like another bug...
And even this is plain wrong - it send UrlEncodedUnicode to server, not just UrlEncoded what server expects
Update: one more thing is, NameValueCollection actually does UrlEncodeUnicode, which is not supposed to be used anymore and is incompatible with regular url.encode/decode (see NameValueCollection to URL Query?).
So the bottom line is: never use this hack with NameValueCollection query = HttpUtility.ParseQueryString(builder.Query); as it will mess your unicode query parameters. Just build query manually and assign it to UriBuilder.Query which will do necessary encoding and then get Uri using UriBuilder.Uri.
Prime example of hurting yourself by using code which is not supposed to be used like this

You might want to check out Flurl [disclosure: I'm the author], a fluent URL builder with optional companion lib that extends it into a full-blown REST client.
var result = await "https://api.com"
// basic URL building:
.AppendPathSegment("endpoint")
.SetQueryParams(new {
api_key = ConfigurationManager.AppSettings["SomeApiKey"],
max_results = 20,
q = "Don't worry, I'll get encoded!"
})
.SetQueryParams(myDictionary)
.SetQueryParam("q", "overwrite q!")
// extensions provided by Flurl.Http:
.WithOAuthBearerToken("token")
.GetJsonAsync<TResult>();
Check out the docs for more details. The full package is available on NuGet:
PM> Install-Package Flurl.Http
or just the stand-alone URL builder:
PM> Install-Package Flurl

Along the same lines as Rostov's post, if you do not want to include a reference to System.Web in your project, you can use FormDataCollection from System.Net.Http.Formatting and do something like the following:
Using System.Net.Http.Formatting.FormDataCollection
var parameters = new Dictionary<string, string>()
{
{ "ham", "Glaced?" },
{ "x-men", "Wolverine + Logan" },
{ "Time", DateTime.UtcNow.ToString() },
};
var query = new FormDataCollection(parameters).ReadAsNameValueCollection().ToString();

Since I have to reuse this few time, I came up with this class that simply help to abstract how the query string is composed.
public class UriBuilderExt
{
private NameValueCollection collection;
private UriBuilder builder;
public UriBuilderExt(string uri)
{
builder = new UriBuilder(uri);
collection = System.Web.HttpUtility.ParseQueryString(string.Empty);
}
public void AddParameter(string key, string value) {
collection.Add(key, value);
}
public Uri Uri{
get
{
builder.Query = collection.ToString();
return builder.Uri;
}
}
}
The use will be simplify to something like this:
var builder = new UriBuilderExt("http://example.com/");
builder.AddParameter("foo", "bar<>&-baz");
builder.AddParameter("bar", "second");
var uri = builder.Uri;
that will return the uri:
http://example.com/?foo=bar%3c%3e%26-baz&bar=second

Good part of accepted answer, modified to use UriBuilder.Uri.ParseQueryString() instead of HttpUtility.ParseQueryString():
var builder = new UriBuilder("http://example.com");
var query = builder.Uri.ParseQueryString();
query["foo"] = "bar<>&-baz";
query["bar"] = "bazinga";
builder.Query = query.ToString();
string url = builder.ToString();

Darin offered an interesting and clever solution, and here is something that may be another option:
public class ParameterCollection
{
private Dictionary<string, string> _parms = new Dictionary<string, string>();
public void Add(string key, string val)
{
if (_parms.ContainsKey(key))
{
throw new InvalidOperationException(string.Format("The key {0} already exists.", key));
}
_parms.Add(key, val);
}
public override string ToString()
{
var server = HttpContext.Current.Server;
var sb = new StringBuilder();
foreach (var kvp in _parms)
{
if (sb.Length > 0) { sb.Append("&"); }
sb.AppendFormat("{0}={1}",
server.UrlEncode(kvp.Key),
server.UrlEncode(kvp.Value));
}
return sb.ToString();
}
}
and so when using it, you might do this:
var parms = new ParameterCollection();
parms.Add("key", "value");
var url = ...
url += "?" + parms;

The RFC 6570 URI Template library I'm developing is capable of performing this operation. All encoding is handled for you in accordance with that RFC. At the time of this writing, a beta release is available and the only reason it's not considered a stable 1.0 release is the documentation doesn't fully meet my expectations (see issues #17, #18, #32, #43).
You could either build a query string alone:
UriTemplate template = new UriTemplate("{?params*}");
var parameters = new Dictionary<string, string>
{
{ "param1", "value1" },
{ "param2", "value2" },
};
Uri relativeUri = template.BindByName(parameters);
Or you could build a complete URI:
UriTemplate template = new UriTemplate("path/to/item{?params*}");
var parameters = new Dictionary<string, string>
{
{ "param1", "value1" },
{ "param2", "value2" },
};
Uri baseAddress = new Uri("http://www.example.com");
Uri relativeUri = template.BindByName(baseAddress, parameters);

Or simply using my Uri extension
Code
public static Uri AttachParameters(this Uri uri, NameValueCollection parameters)
{
var stringBuilder = new StringBuilder();
string str = "?";
for (int index = 0; index < parameters.Count; ++index)
{
stringBuilder.Append(str + parameters.AllKeys[index] + "=" + parameters[index]);
str = "&";
}
return new Uri(uri + stringBuilder.ToString());
}
Usage
Uri uri = new Uri("http://www.example.com/index.php").AttachParameters(new NameValueCollection
{
{"Bill", "Gates"},
{"Steve", "Jobs"}
});
Result
http://www.example.com/index.php?Bill=Gates&Steve=Jobs

To avoid double encoding issue described in taras.roshko's answer and to keep possibility to easily work with query parameters, you can use uriBuilder.Uri.ParseQueryString() instead of HttpUtility.ParseQueryString().

Thanks to "Darin Dimitrov", This is the extension methods.
public static partial class Ext
{
public static Uri GetUriWithparameters(this Uri uri,Dictionary<string,string> queryParams = null,int port = -1)
{
var builder = new UriBuilder(uri);
builder.Port = port;
if(null != queryParams && 0 < queryParams.Count)
{
var query = HttpUtility.ParseQueryString(builder.Query);
foreach(var item in queryParams)
{
query[item.Key] = item.Value;
}
builder.Query = query.ToString();
}
return builder.Uri;
}
public static string GetUriWithparameters(string uri,Dictionary<string,string> queryParams = null,int port = -1)
{
var builder = new UriBuilder(uri);
builder.Port = port;
if(null != queryParams && 0 < queryParams.Count)
{
var query = HttpUtility.ParseQueryString(builder.Query);
foreach(var item in queryParams)
{
query[item.Key] = item.Value;
}
builder.Query = query.ToString();
}
return builder.Uri.ToString();
}
}

HttpClient client = new HttpClient();
var uri = Environment.GetEnvironmentVariable("URL of Api");
var requesturi = QueryHelpers.AddQueryString(uri, "parameter_name",parameter_value);
client.BaseAddress = new Uri(requesturi);
And then you can add request headers also eg:
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("x-api-key", secretValue);
response syntax eg:
HttpResponseMessage response = client.GetAsync(requesturi).Result;
Hope it will work for you.

My answer doesn't globally differ from the accepted/other answers. I just tried to create an extension method for the Uri type, which takes variable number of parameters.
public static class UriExtensions
{
public static Uri AddParameter(this Uri url, params (string Name, string Value)[] #params)
{
if (!#params.Any())
{
return url;
}
UriBuilder uriBuilder = new(url);
NameValueCollection query = HttpUtility.ParseQueryString(uriBuilder.Query);
foreach (var param in #params)
{
query[param.Name] = param.Value.Trim();
}
uriBuilder.Query = query.ToString();
return uriBuilder.Uri;
}
}
Usage example:
var uri = new Uri("http://someuri.com")
.AddParameter(
("p1.name", "p1.value"),
("p2.name", "p2.value"),
("p3.name", "p3.value"));

I couldn't find a better solution than creating a extension method to convert a Dictionary to QueryStringFormat. The solution proposed by Waleed A.K. is good as well.
Follow my solution:
Create the extension method:
public static class DictionaryExt
{
public static string ToQueryString<TKey, TValue>(this Dictionary<TKey, TValue> dictionary)
{
return ToQueryString<TKey, TValue>(dictionary, "?");
}
public static string ToQueryString<TKey, TValue>(this Dictionary<TKey, TValue> dictionary, string startupDelimiter)
{
string result = string.Empty;
foreach (var item in dictionary)
{
if (string.IsNullOrEmpty(result))
result += startupDelimiter; // "?";
else
result += "&";
result += string.Format("{0}={1}", item.Key, item.Value);
}
return result;
}
}
And them:
var param = new Dictionary<string, string>
{
{ "param1", "value1" },
{ "param2", "value2" },
};
param.ToQueryString(); //By default will add (?) question mark at begining
//"?param1=value1&param2=value2"
param.ToQueryString("&"); //Will add (&)
//"&param1=value1&param2=value2"
param.ToQueryString(""); //Won't add anything
//"param1=value1&param2=value2"

C# .net core 3.1 accessing to Json elements

I have below code that works, and the response is Json data as below:
{
"code": 0,
"data": {
"BCH": { # BCH account
"available": "13.60109", # Available BCH
"frozen": "0.00000" # Frozen BCH
},
"BTC": { # BTC account
"available": "32590.16", # Available BTC
"frozen": "7000.00" # Frozen BTC
},
"ETH": { # ETH account
"available": "5.06000", # Available ETH
"frozen": "0.00000" # Frozen ETH
}
},
"message": "Ok"
}
How can i access to Json elements in this code? adding Json data to array or variable, i read other sample codes in stackoverflow but i coudn't do.
private JsonElement Post(string path, Dictionary<string, object> args, string signature = null)
{
var json = JsonSerializer.Serialize(args);
var content = new StringContent(json);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var req = new HttpRequestMessage(HttpMethod.Post, this.baseUrl + path);
req.Content = content;
if (signature != null)
{
req.Headers.Add("authorization", signature);
}
var res = this.client.SendAsync(req).Result;
var result = res.Content.ReadAsStringAsync().Result;
return JsonSerializer.Deserialize<JsonElement>(result);
}
private JsonElement Get(string path, Dictionary<string, object> args = null, string signature = null)
{
var url = this.baseUrl + path;
if (args != null)
{
var param = string.Join("&", args.Select(p => $"{p.Key}={p.Value}"));
url = url + "?" + param;
}
string result;
if (signature != null)
{
var req = new HttpRequestMessage(HttpMethod.Get, url);
req.Headers.Add("authorization", signature);
var res = client.SendAsync(req).Result;
result = res.Content.ReadAsStringAsync().Result;
}
else
{
result = this.client.GetStringAsync(url).Result;
}
return JsonSerializer.Deserialize<JsonElement>(result);
}

I'm not quite sure if I understand your question correctly, however if I understand you're trying to access values inside the JsonElement type you have deserialized the response to?
If so, you can lean on the GetProperty and TryGetProperty methods, and the Item[Int32] (if the object is an array) property of the JsonElement struct. For example myJsonElement.GetProperty("data").GetProperty("BCH").GetProperty("available").ToString() would return the string "13.60109" from your example JSON.
However this isn't particularly practical as it could quickly get messy if you want to extract more than one or two values. As Jonathan Alfaro has suggested your best bet would be to define a strongly typed model and deserialize to that type instead.

C# Web API method returns 403 Forbidden

Solved!!! - See last edit.
In my MVC app I make calls out to a Web API service with HMAC Authentication Filterign. My Get (GetMultipleItemsRequest) works, but my Post does not. If I turn off HMAC authentication filtering all of them work. I'm not sure why the POSTS do not work, but the GETs do.
I make the GET call from my code like this (this one works):
var productsClient = new RestClient<Role>(System.Configuration.ConfigurationManager.AppSettings["WebApiUrl"],
"xxxxxxxxxxxxxxx", true);
var getManyResult = productsClient.GetMultipleItemsRequest("api/Role").Result;
I make the POST call from my code like this (this one only works when I turn off HMAC):
private RestClient<Profile> profileClient = new RestClient<Profile>(System.Configuration.ConfigurationManager.AppSettings["WebApiUrl"],
"xxxxxxxxxxxxxxx", true);
[HttpPost]
public ActionResult ProfileImport(IEnumerable<HttpPostedFileBase> files)
{
//...
var postResult = profileClient.PostRequest("api/Profile", newProfile).Result;
}
My RestClient builds like this:
public class RestClient<T> where T : class
{
//...
private void SetupClient(HttpClient client, string methodName, string apiUrl, T content = null)
{
const string secretTokenName = "SecretToken";
client.BaseAddress = new Uri(_baseAddress);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
if (_hmacSecret)
{
client.DefaultRequestHeaders.Date = DateTime.UtcNow;
var datePart = client.DefaultRequestHeaders.Date.Value.UtcDateTime.ToString(CultureInfo.InvariantCulture);
var fullUri = _baseAddress + apiUrl;
var contentMD5 = "";
if (content != null)
{
var json = new JavaScriptSerializer().Serialize(content);
contentMD5 = Hashing.GetHashMD5OfString(json); // <--- Javascript serialized version is hashed
}
var messageRepresentation =
methodName + "\n" +
contentMD5 + "\n" +
datePart + "\n" +
fullUri;
var sharedSecretValue = ConfigurationManager.AppSettings[_sharedSecretName];
var hmac = Hashing.GetHashHMACSHA256OfString(messageRepresentation, sharedSecretValue);
client.DefaultRequestHeaders.Add(secretTokenName, hmac);
}
else if (!string.IsNullOrWhiteSpace(_sharedSecretName))
{
var sharedSecretValue = ConfigurationManager.AppSettings[_sharedSecretName];
client.DefaultRequestHeaders.Add(secretTokenName, sharedSecretValue);
}
}
public async Task<T[]> GetMultipleItemsRequest(string apiUrl)
{
T[] result = null;
try
{
using (var client = new HttpClient())
{
SetupClient(client, "GET", apiUrl);
var response = await client.GetAsync(apiUrl).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
await response.Content.ReadAsStringAsync().ContinueWith((Task<string> x) =>
{
if (x.IsFaulted)
throw x.Exception;
result = JsonConvert.DeserializeObject<T[]>(x.Result);
});
}
}
catch (HttpRequestException exception)
{
if (exception.Message.Contains("401 (Unauthorized)"))
{
}
else if (exception.Message.Contains("403 (Forbidden)"))
{
}
}
catch (Exception)
{
}
return result;
}
public async Task<T> PostRequest(string apiUrl, T postObject)
{
T result = null;
try
{
using (var client = new HttpClient())
{
SetupClient(client, "POST", apiUrl, postObject);
var response = await client.PostAsync(apiUrl, postObject, new JsonMediaTypeFormatter()).ConfigureAwait(false); //<--- not javascript formatted
response.EnsureSuccessStatusCode();
await response.Content.ReadAsStringAsync().ContinueWith((Task<string> x) =>
{
if (x.IsFaulted)
throw x.Exception;
result = JsonConvert.DeserializeObject<T>(x.Result);
});
}
}
catch (HttpRequestException exception)
{
if (exception.Message.Contains("401 (Unauthorized)"))
{
}
else if (exception.Message.Contains("403 (Forbidden)"))
{
}
}
catch (Exception)
{
}
return result;
}
//...
}
My Web API Controller is defined like this:
[SecretAuthenticationFilter(SharedSecretName = "xxxxxxxxxxxxxxx", HmacSecret = true)]
public class ProfileController : ApiController
{
[HttpPost]
[ResponseType(typeof(Profile))]
public IHttpActionResult PostProfile(Profile Profile)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
GuidValue = Guid.NewGuid();
Resource res = new Resource();
res.ResourceId = GuidValue;
var data23 = Resourceservices.Insert(res);
Profile.ProfileId = data23.ResourceId;
_profileservices.Insert(Profile);
return CreatedAtRoute("DefaultApi", new { id = Profile.ProfileId }, Profile);
}
}
Here is some of what SecretAuthenticationFilter does:
//now try to read the content as string
string content = actionContext.Request.Content.ReadAsStringAsync().Result;
var contentMD5 = content == "" ? "" : Hashing.GetHashMD5OfString(content); //<-- Hashing the non-JavaScriptSerialized
var datePart = "";
var requestDate = DateTime.Now.AddDays(-2);
if (actionContext.Request.Headers.Date != null)
{
requestDate = actionContext.Request.Headers.Date.Value.UtcDateTime;
datePart = requestDate.ToString(CultureInfo.InvariantCulture);
}
var methodName = actionContext.Request.Method.Method;
var fullUri = actionContext.Request.RequestUri.ToString();
var messageRepresentation =
methodName + "\n" +
contentMD5 + "\n" +
datePart + "\n" +
fullUri;
var expectedValue = Hashing.GetHashHMACSHA256OfString(messageRepresentation, sharedSecretValue);
// Are the hmacs the same, and have we received it within +/- 5 mins (sending and
// receiving servers may not have exactly the same time)
if (messageSecretValue == expectedValue
&& requestDate > DateTime.UtcNow.AddMinutes(-5)
&& requestDate < DateTime.UtcNow.AddMinutes(5))
goodRequest = true;
Any idea why HMAC doesn't work for the POST?
EDIT:
When SecretAuthenticationFilter tries to compare the HMAC sent, with what it thinks the HMAC should be they don't match. The reason is the MD5Hash of the content doesn't match the MD5Hash of the received content. The RestClient hashes the content using a JavaScriptSerializer.Serialized version of the content, but then the PostRequest passes the object as JsonMediaTypeFormatted.
These two types don't get formatted the same. For instance, the JavaScriptSerializer give's us dates like this:
\"EnteredDate\":\"\/Date(1434642998639)\/\"
The passed content has dates like this:
\"EnteredDate\":\"2015-06-18T11:56:38.6390407-04:00\"
I guess I need the hash to use the same data that's passed, so the Filter on the other end can confirm it correctly. Thoughts?
EDIT:
Found the answer, I needed to change the SetupClient code from using this line:
var json = new JavaScriptSerializer().Serialize(content);
contentMD5 = Hashing.GetHashMD5OfString(json);
To using this:
var json = JsonConvert.SerializeObject(content);
contentMD5 = Hashing.GetHashMD5OfString(json);
Now the sent content (formatted via JSON) will match the hashed content.
I was not the person who wrote this code originally. :)

Found the answer, I needed to change the SetupClient code from using this line:
var json = new JavaScriptSerializer().Serialize(content);
contentMD5 = Hashing.GetHashMD5OfString(json);
To using this:
var json = JsonConvert.SerializeObject(content);
contentMD5 = Hashing.GetHashMD5OfString(json);
Now the content used for the hash will be formatted as JSON and will match the sent content (which is also formatted via JSON).

Build query string for System.Net.HttpClient get

If I wish to submit a http get request using System.Net.HttpClient there seems to be no api to add parameters, is this correct?
Is there any simple api available to build the query string that doesn't involve building a name value collection and url encoding those and then finally concatenating them?
I was hoping to use something like RestSharp's api (i.e AddParameter(..))

If I wish to submit a http get request using System.Net.HttpClient
there seems to be no api to add parameters, is this correct?
Yes.
Is there any simple api available to build the query string that
doesn't involve building a name value collection and url encoding
those and then finally concatenating them?
Sure:
var query = HttpUtility.ParseQueryString(string.Empty);
query["foo"] = "bar<>&-baz";
query["bar"] = "bazinga";
string queryString = query.ToString();
will give you the expected result:
foo=bar%3c%3e%26-baz&bar=bazinga
You might also find the UriBuilder class useful:
var builder = new UriBuilder("http://example.com");
builder.Port = -1;
var query = HttpUtility.ParseQueryString(builder.Query);
query["foo"] = "bar<>&-baz";
query["bar"] = "bazinga";
builder.Query = query.ToString();
string url = builder.ToString();
will give you the expected result:
http://example.com/?foo=bar%3c%3e%26-baz&bar=bazinga
that you could more than safely feed to your HttpClient.GetAsync method.

For those who do not want to include System.Web in projects that don't already use it, you can use FormUrlEncodedContent from System.Net.Http and do something like the following:
keyvaluepair version
string query;
using(var content = new FormUrlEncodedContent(new KeyValuePair<string, string>[]{
new KeyValuePair<string, string>("ham", "Glazed?"),
new KeyValuePair<string, string>("x-men", "Wolverine + Logan"),
new KeyValuePair<string, string>("Time", DateTime.UtcNow.ToString()),
})) {
query = content.ReadAsStringAsync().Result;
}
dictionary version
string query;
using(var content = new FormUrlEncodedContent(new Dictionary<string, string>()
{
{ "ham", "Glaced?"},
{ "x-men", "Wolverine + Logan"},
{ "Time", DateTime.UtcNow.ToString() },
})) {
query = content.ReadAsStringAsync().Result;
}

In a ASP.NET Core project you can use the QueryHelpers class, available in the Microsoft.AspNetCore.WebUtilities namespace for ASP.NET Core, or the .NET Standard 2.0 NuGet package for other consumers:
// using Microsoft.AspNetCore.WebUtilities;
var query = new Dictionary<string, string>
{
["foo"] = "bar",
["foo2"] = "bar2",
// ...
};
var response = await client.GetAsync(QueryHelpers.AddQueryString("/api/", query));

TL;DR: do not use accepted version as It's completely broken in relation to handling unicode characters, and never use internal API
I've actually found weird double encoding issue with the accepted solution:
So, If you're dealing with characters which need to be encoded, accepted solution leads to double encoding:
query parameters are auto encoded by using NameValueCollection indexer (and this uses UrlEncodeUnicode, not regular expected UrlEncode(!))
Then, when you call uriBuilder.Uri it creates new Uri using constructor which does encoding one more time (normal url encoding)
That cannot be avoided by doing uriBuilder.ToString() (even though this returns correct Uri which IMO is at least inconsistency, maybe a bug, but that's another question) and then using HttpClient method accepting string - client still creates Uri out of your passed string like this: new Uri(uri, UriKind.RelativeOrAbsolute)
Small, but full repro:
var builder = new UriBuilder
{
Scheme = Uri.UriSchemeHttps,
Port = -1,
Host = "127.0.0.1",
Path = "app"
};
NameValueCollection query = HttpUtility.ParseQueryString(builder.Query);
query["cyrillic"] = "кирилиця";
builder.Query = query.ToString();
Console.WriteLine(builder.Query); //query with cyrillic stuff UrlEncodedUnicode, and that's not what you want
var uri = builder.Uri; // creates new Uri using constructor which does encode and messes cyrillic parameter even more
Console.WriteLine(uri);
// this is still wrong:
var stringUri = builder.ToString(); // returns more 'correct' (still `UrlEncodedUnicode`, but at least once, not twice)
new HttpClient().GetStringAsync(stringUri); // this creates Uri object out of 'stringUri' so we still end up sending double encoded cyrillic text to server. Ouch!
Output:
?cyrillic=%u043a%u0438%u0440%u0438%u043b%u0438%u0446%u044f
https://127.0.0.1/app?cyrillic=%25u043a%25u0438%25u0440%25u0438%25u043b%25u0438%25u0446%25u044f
As you may see, no matter if you do uribuilder.ToString() + httpClient.GetStringAsync(string) or uriBuilder.Uri + httpClient.GetStringAsync(Uri) you end up sending double encoded parameter
Fixed example could be:
var uri = new Uri(builder.ToString(), dontEscape: true);
new HttpClient().GetStringAsync(uri);
But this uses obsolete Uri constructor
P.S on my latest .NET on Windows Server, Uri constructor with bool doc comment says "obsolete, dontEscape is always false", but actually works as expected (skips escaping)
So It looks like another bug...
And even this is plain wrong - it send UrlEncodedUnicode to server, not just UrlEncoded what server expects
Update: one more thing is, NameValueCollection actually does UrlEncodeUnicode, which is not supposed to be used anymore and is incompatible with regular url.encode/decode (see NameValueCollection to URL Query?).
So the bottom line is: never use this hack with NameValueCollection query = HttpUtility.ParseQueryString(builder.Query); as it will mess your unicode query parameters. Just build query manually and assign it to UriBuilder.Query which will do necessary encoding and then get Uri using UriBuilder.Uri.
Prime example of hurting yourself by using code which is not supposed to be used like this

You might want to check out Flurl [disclosure: I'm the author], a fluent URL builder with optional companion lib that extends it into a full-blown REST client.
var result = await "https://api.com"
// basic URL building:
.AppendPathSegment("endpoint")
.SetQueryParams(new {
api_key = ConfigurationManager.AppSettings["SomeApiKey"],
max_results = 20,
q = "Don't worry, I'll get encoded!"
})
.SetQueryParams(myDictionary)
.SetQueryParam("q", "overwrite q!")
// extensions provided by Flurl.Http:
.WithOAuthBearerToken("token")
.GetJsonAsync<TResult>();
Check out the docs for more details. The full package is available on NuGet:
PM> Install-Package Flurl.Http
or just the stand-alone URL builder:
PM> Install-Package Flurl

Along the same lines as Rostov's post, if you do not want to include a reference to System.Web in your project, you can use FormDataCollection from System.Net.Http.Formatting and do something like the following:
Using System.Net.Http.Formatting.FormDataCollection
var parameters = new Dictionary<string, string>()
{
{ "ham", "Glaced?" },
{ "x-men", "Wolverine + Logan" },
{ "Time", DateTime.UtcNow.ToString() },
};
var query = new FormDataCollection(parameters).ReadAsNameValueCollection().ToString();

Since I have to reuse this few time, I came up with this class that simply help to abstract how the query string is composed.
public class UriBuilderExt
{
private NameValueCollection collection;
private UriBuilder builder;
public UriBuilderExt(string uri)
{
builder = new UriBuilder(uri);
collection = System.Web.HttpUtility.ParseQueryString(string.Empty);
}
public void AddParameter(string key, string value) {
collection.Add(key, value);
}
public Uri Uri{
get
{
builder.Query = collection.ToString();
return builder.Uri;
}
}
}
The use will be simplify to something like this:
var builder = new UriBuilderExt("http://example.com/");
builder.AddParameter("foo", "bar<>&-baz");
builder.AddParameter("bar", "second");
var uri = builder.Uri;
that will return the uri:
http://example.com/?foo=bar%3c%3e%26-baz&bar=second

Good part of accepted answer, modified to use UriBuilder.Uri.ParseQueryString() instead of HttpUtility.ParseQueryString():
var builder = new UriBuilder("http://example.com");
var query = builder.Uri.ParseQueryString();
query["foo"] = "bar<>&-baz";
query["bar"] = "bazinga";
builder.Query = query.ToString();
string url = builder.ToString();

Darin offered an interesting and clever solution, and here is something that may be another option:
public class ParameterCollection
{
private Dictionary<string, string> _parms = new Dictionary<string, string>();
public void Add(string key, string val)
{
if (_parms.ContainsKey(key))
{
throw new InvalidOperationException(string.Format("The key {0} already exists.", key));
}
_parms.Add(key, val);
}
public override string ToString()
{
var server = HttpContext.Current.Server;
var sb = new StringBuilder();
foreach (var kvp in _parms)
{
if (sb.Length > 0) { sb.Append("&"); }
sb.AppendFormat("{0}={1}",
server.UrlEncode(kvp.Key),
server.UrlEncode(kvp.Value));
}
return sb.ToString();
}
}
and so when using it, you might do this:
var parms = new ParameterCollection();
parms.Add("key", "value");
var url = ...
url += "?" + parms;

The RFC 6570 URI Template library I'm developing is capable of performing this operation. All encoding is handled for you in accordance with that RFC. At the time of this writing, a beta release is available and the only reason it's not considered a stable 1.0 release is the documentation doesn't fully meet my expectations (see issues #17, #18, #32, #43).
You could either build a query string alone:
UriTemplate template = new UriTemplate("{?params*}");
var parameters = new Dictionary<string, string>
{
{ "param1", "value1" },
{ "param2", "value2" },
};
Uri relativeUri = template.BindByName(parameters);
Or you could build a complete URI:
UriTemplate template = new UriTemplate("path/to/item{?params*}");
var parameters = new Dictionary<string, string>
{
{ "param1", "value1" },
{ "param2", "value2" },
};
Uri baseAddress = new Uri("http://www.example.com");
Uri relativeUri = template.BindByName(baseAddress, parameters);

Or simply using my Uri extension
Code
public static Uri AttachParameters(this Uri uri, NameValueCollection parameters)
{
var stringBuilder = new StringBuilder();
string str = "?";
for (int index = 0; index < parameters.Count; ++index)
{
stringBuilder.Append(str + parameters.AllKeys[index] + "=" + parameters[index]);
str = "&";
}
return new Uri(uri + stringBuilder.ToString());
}
Usage
Uri uri = new Uri("http://www.example.com/index.php").AttachParameters(new NameValueCollection
{
{"Bill", "Gates"},
{"Steve", "Jobs"}
});
Result
http://www.example.com/index.php?Bill=Gates&Steve=Jobs

To avoid double encoding issue described in taras.roshko's answer and to keep possibility to easily work with query parameters, you can use uriBuilder.Uri.ParseQueryString() instead of HttpUtility.ParseQueryString().

Thanks to "Darin Dimitrov", This is the extension methods.
public static partial class Ext
{
public static Uri GetUriWithparameters(this Uri uri,Dictionary<string,string> queryParams = null,int port = -1)
{
var builder = new UriBuilder(uri);
builder.Port = port;
if(null != queryParams && 0 < queryParams.Count)
{
var query = HttpUtility.ParseQueryString(builder.Query);
foreach(var item in queryParams)
{
query[item.Key] = item.Value;
}
builder.Query = query.ToString();
}
return builder.Uri;
}
public static string GetUriWithparameters(string uri,Dictionary<string,string> queryParams = null,int port = -1)
{
var builder = new UriBuilder(uri);
builder.Port = port;
if(null != queryParams && 0 < queryParams.Count)
{
var query = HttpUtility.ParseQueryString(builder.Query);
foreach(var item in queryParams)
{
query[item.Key] = item.Value;
}
builder.Query = query.ToString();
}
return builder.Uri.ToString();
}
}

HttpClient client = new HttpClient();
var uri = Environment.GetEnvironmentVariable("URL of Api");
var requesturi = QueryHelpers.AddQueryString(uri, "parameter_name",parameter_value);
client.BaseAddress = new Uri(requesturi);
And then you can add request headers also eg:
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("x-api-key", secretValue);
response syntax eg:
HttpResponseMessage response = client.GetAsync(requesturi).Result;
Hope it will work for you.

My answer doesn't globally differ from the accepted/other answers. I just tried to create an extension method for the Uri type, which takes variable number of parameters.
public static class UriExtensions
{
public static Uri AddParameter(this Uri url, params (string Name, string Value)[] #params)
{
if (!#params.Any())
{
return url;
}
UriBuilder uriBuilder = new(url);
NameValueCollection query = HttpUtility.ParseQueryString(uriBuilder.Query);
foreach (var param in #params)
{
query[param.Name] = param.Value.Trim();
}
uriBuilder.Query = query.ToString();
return uriBuilder.Uri;
}
}
Usage example:
var uri = new Uri("http://someuri.com")
.AddParameter(
("p1.name", "p1.value"),
("p2.name", "p2.value"),
("p3.name", "p3.value"));

I couldn't find a better solution than creating a extension method to convert a Dictionary to QueryStringFormat. The solution proposed by Waleed A.K. is good as well.
Follow my solution:
Create the extension method:
public static class DictionaryExt
{
public static string ToQueryString<TKey, TValue>(this Dictionary<TKey, TValue> dictionary)
{
return ToQueryString<TKey, TValue>(dictionary, "?");
}
public static string ToQueryString<TKey, TValue>(this Dictionary<TKey, TValue> dictionary, string startupDelimiter)
{
string result = string.Empty;
foreach (var item in dictionary)
{
if (string.IsNullOrEmpty(result))
result += startupDelimiter; // "?";
else
result += "&";
result += string.Format("{0}={1}", item.Key, item.Value);
}
return result;
}
}
And them:
var param = new Dictionary<string, string>
{
{ "param1", "value1" },
{ "param2", "value2" },
};
param.ToQueryString(); //By default will add (?) question mark at begining
//"?param1=value1&param2=value2"
param.ToQueryString("&"); //Will add (&)
//"&param1=value1&param2=value2"
param.ToQueryString(""); //Won't add anything
//"param1=value1&param2=value2"

Synchronous VirusTotal code to Rx enabled asynchronous (and maybe some JSON parsing)

last time I posted a question on here everyone provided some great guidance on getting my problem solved. Move forward in time and here is another. I'm attempting to redo a small helper tool I have that checks URL's and Files against VirusTotal to get some basic information. The code below works quite well but locks up the UI. I was told that I should look into Rx and am enjoying reading up on it but cannot seem to get my head wrapped around it. So now here is where the question comes in, what is the best way to design the following code to make it utilize Rx so that it is asynchronous and leaves my UI alone while it does it's thing. VirusTotal also utilizes multilevel JSON for responses so if anyone has a nice way of integrating that into this that would even be better.
class Virustotal
{
private string APIKey = "REMOVED";
private string FileReportURL = "https://www.virustotal.com/vtapi/v2/file/report";
private string URLReportURL = "http://www.virustotal.com/vtapi/v2/url/report";
private string URLSubmitURL = "https://www.virustotal.com/vtapi/v2/url/scan";
WebRequest theRequest;
HttpWebResponse theResponse;
ArrayList theQueryData;
public string GetFileReport(string checksum) // Gets latest report of file from VT using a hash (MD5 / SHA1 / SHA256)
{
this.WebPostRequest(this.FileReportURL);
this.Add("resource", checksum);
return this.GetResponse();
}
public string GetURLReport(string url) // Gets latest report of URL from VT
{
this.WebPostRequest(this.URLReportURL);
this.Add("resource", url);
this.Add("scan", "1"); //Automatically submits to VT if no result found
return this.GetResponse();
}
public string SubmitURL(string url) // Submits URL to VT for insertion to scanning queue
{
this.WebPostRequest(this.URLSubmitURL);
this.Add("url", url);
return this.GetResponse();
}
public string SubmitFile() // Submits File to VT for insertion to scanning queue
{
// File Upload code needed
return this.GetResponse();
}
private void WebPostRequest(string url)
{
theRequest = WebRequest.Create(url);
theRequest.Method = "POST";
theQueryData = new ArrayList();
this.Add("apikey", APIKey);
}
private void Add(string key, string value)
{
theQueryData.Add(String.Format("{0}={1}", key, Uri.EscapeDataString(value)));
}
private string GetResponse()
{
// Set the encoding type
theRequest.ContentType="application/x-www-form-urlencoded";
// Build a string containing all the parameters
string Parameters = String.Join("&",(String[]) theQueryData.ToArray(typeof(string)));
theRequest.ContentLength = Parameters.Length;
// We write the parameters into the request
StreamWriter sw = new StreamWriter(theRequest.GetRequestStream());
sw.Write(Parameters);
sw.Close();
// Execute the query
theResponse = (HttpWebResponse)theRequest.GetResponse();
StreamReader sr = new StreamReader(theResponse.GetResponseStream());
return sr.ReadToEnd();
}
}

Your code is poorly written which makes it more difficult to make it asynchronous - primarily the three class-level variables. When coding in Rx you want to think "functional programming" and not "OOP" - so no class-level variables.
So, what I've done is this - I've recoded the GetResponse method to encapsulate all of the state into a single call - and I've made it return IObservable<string> rather than just string.
The public functions can now be written like this:
public IObservable<string> GetFileReport(string checksum)
{
return this.GetResponse(this.FileReportURL,
new Dictionary<string, string>() { { "resource", checksum }, });
}
public IObservable<string> GetURLReport(string url)
{
return this.GetResponse(this.URLReportURL,
new Dictionary<string, string>()
{ { "resource", url }, { "scan", "1" }, });
}
public IObservable<string> SubmitURL(string url)
{
return this.GetResponse(this.URLSubmitURL,
new Dictionary<string, string>() { { "url", url }, });
}
public IObservable<string> SubmitFile()
{
return this.GetResponse("UNKNOWNURL", new Dictionary<string, string>());
}
And GetResponse looks like this:
private IObservable<string> GetResponse(
string url,
Dictionary<string, string> theQueryData)
{
return Observable.Start(() =>
{
var theRequest = WebRequest.Create(url);
theRequest.Method = "POST";
theRequest.ContentType="application/x-www-form-urlencoded";
theQueryData.Add("apikey", APIKey);
string Parameters = String.Join("&",
theQueryData.Select(x =>
String.Format("{0}={1}", x.Key, x.Value)));
theRequest.ContentLength = Parameters.Length;
using (var sw = new StreamWriter(theRequest.GetRequestStream()))
{
sw.Write(Parameters);
sw.Close();
}
using (var theResponse = (HttpWebResponse)theRequest.GetResponse())
{
using (var sr = new StreamReader(theResponse.GetResponseStream()))
{
return sr.ReadToEnd();
}
}
});
}
I haven't actually tested this - I don't have the APIKEY for starters - but it should work OK. Let me know how you go.

Categories

Resources