I have a C# class as follows:
public class TestObj
{
private int intval;
private string stringval;
private int[] intarray;
private string[] stringarray;
//... public properties not shown here
}
I would like to serialize an instance of this class into a string.
In addition:
I will be appending this string as a QueryString param to a URL. So I would like to take some effort to ensure that the string cannot be tampered with easily.
Also, I would like the serialization method to be efficient so the size of the string is minmal.
Any suggestions of specific .NET Framework classes/methods I should use?
Sign the stream and add the signature to your query. Use a HMAC signing algorithm, like HMACSHA1. You will need to have a secret between your client and your server to sign and validate the signature.
1) To serialize:
public String SerializeObject(TestObj object)
{
String Serialized = String.Empty;
MemoryStream memoryStream = new MemoryStream ( );
XmlSerializer xs = new XmlSerializer(typeof(TestObj));
XmlTextWriter xmlTextWriter = new XmlTextWriter ( memoryStream, Encoding.UTF8 );
xs.Serialize (xmlTextWriter, object);
memoryStream = (MemoryStream) xmlTextWriter.BaseStream;
Serialized = UTF8Encoding.GetString(memoryStream.ToArray());
return Serialized;
}
2) To prevent tampering:
Come up with a secret string, e.g. "MySecretWord".
Take your serialized object instance as a string, and append the secret word to it.
Hash the string (e.g. SHA or use HMAC (as suggested by Remus) )
Append the hash to the query string
On the receiving side (which also knows your "MySecretWord" secret string) you strip away the hash, take the original serialized instance, append the known secret string and hash it again. Then compare the two hashes for equality. If they are equal, your string was not modified.
You may need to Url/Base64 Encode your string so it works as a query string. This is also important as you need the query string to arrive exactly as sent.
Related
I wanted to do a simple message encrypter to dip my toes into the matter but I can't make it to work. The problem is that whatever input I start with, sometimes it encrypts it but when I try to decrypt it, it just doesn't return the original string. It would be really helpful if you could tell me what I'm doing wrong or guide in the right direction.
Complete code
This are the sections in charge of encrypting and decrypting.
void Decrypt()
{
using var crypt = Aes.Create();
string[] input = ClipboardService.GetText()?.Split(SEPARATOR) ?? Array.Empty<string>();
byte[] key = input[0].ToBytes();
byte[] IV = input[^1].ToBytes();
byte[] value = string.Join(string.Empty, input[1..^1]).ToBytes();
crypt.IV = IV;
crypt.Key = key;
var decryptedValue = crypt.DecryptCbc(value, IV, PaddingMode.Zeros);
string decryptedValueInText = decryptedValue.ToUnicodeString();
ClipboardService.SetText(decryptedValueInText);
LogInfoMessage($"{decryptedValueInText}: {decryptedValue.Length}");
crypt.Clear();
}
void Encrypt()
{
using var crypt = Aes.Create();
crypt.GenerateKey();
string value = ClipboardService.GetText() ?? string.Empty;
var encryptedValue = crypt.EncryptCbc(value.ToBytes(), crypt.IV, PaddingMode.Zeros);
string encryptedValueInText = $"{crypt.Key.ToUnicodeString()}{SEPARATOR}{encryptedValue.ToUnicodeString()}{SEPARATOR}{crypt.IV.ToUnicodeString()}";
ClipboardService.SetText(encryptedValueInText);
LogInfoMessage($"{encryptedValueInText}: {encryptedValue.Length}");
crypt.Clear();
}
There are two extension methods:
public static string ToUnicodeString(this byte[] bytes) => Encoding.Unicode.GetString(bytes);
public static byte[] ToBytes(this string str) => Encoding.Unicode.GetBytes(str);
Example
The input links were:
https://www.youtube.com/
https://www.youtube.com/watch?v=bSA91XTzeuA
I don't think it matters because the key and IV are autogenerated everytime anyways but still.
Per our discussion...
Using the clipboard to store binary data as Unicode text will fail due to invalid UTF-16 codepoints. UTF-16 uses some multi-word encoding for certain Unicode characters, using 32 bits in surrogate pairs to encode Unicode code points from the supplementary planes. There are plenty of primers on the UTF-16 encoding, but basically you have a pair of 16-bit values where the first is in the range 0xD800-0xDBFF and the second must be in the range 0xDC00-0xDFFF. Odds on your encrypted data will break this rule.
As noted, if your encrypted binary data must be sent through a text-only transport you should encode the bytes in the encrypted block using Base64 or similar.
I'd also like to stress that writing methods that can be called with parameters rather than directly accessing the clipboard for I/O makes it much simpler to do testing, including round-trip tests on the various parts of the problem. Proving that the codec is working without reference to the clipboard is a good test and separation of concerns helps to more readily identify the source of problems in the future.
I need to convert a big XElement to a byte array so that it can be uploaded later to a fileshare. What is the correct method to call to do that?
Below you see the signature of a method fileShare.PutFile that is internal:
void PutFile(string folder, string fileName, byte[] content);
Then given an XElement xml, I tried converting it to a byte array by encoding its XElement.Value using Encoding.Default.GetBytes() as follows:
byte[] bytes = Encoding.Default.GetBytes(xml.Value);
fileShare.PutFile(folderName, blobName, bytes);
I am not so sure xml.Value (XElement.Value) is really what GetBytes method is really needing though. Is this correct?
To test this, I spun up a console app and put in some fake data. I did this for the XElement:
XElement root = new XElement("Root",
new XElement("Child1", 1),
new XElement("Child2", 2),
new XElement("Child3", 3),
new XElement("Child4", 4),
new XElement("Child5", 5),
new XElement("Child6", 6)
);
Then I tried that line of code putting to a byte array
byte[] bytes = Encoding.Default.GetBytes(root.Value);
Well I guess I forgot that when I step over and see Autos that bytes variable is btye[6] and when I expand - I see that [0] = 49 and so on
Now this may not mean it is not working ... or does it mean that? How can I interpret the contents of the bytes array, to check whether it is correct?
Firstly, using Encoding.Default is not recommended. From the docs:
Warning
Different computers can use different encodings as the default, and the default encoding can change on a single computer. If you use the Default encoding to encode and decode data streamed between computers or retrieved at different times on the same computer, it may translate that data incorrectly. In addition, the encoding returned by the Default property uses best-fit fallback to map unsupported characters to characters supported by the code page. For these reasons, using the default encoding is not recommended. To ensure that encoded bytes are decoded properly, you should use a Unicode encoding, such as UTF8Encoding or UnicodeEncoding. You could also use a higher-level protocol to ensure that the same format is used for encoding and decoding.
Secondly, XElement.Value returns
A String that contains all of the text content of this element. If there are multiple text nodes, they will be concatenated.
Thus if you upload the Value you will be stripping away the entire XML markup structure from your file leaving only the plain text. While you might want to do that, it seems very unlikely. If you compare the value with the string returned by XElement.ToString() the difference should be clear.
Instead, to convert the XML contents of your XElement (including both markup and text) to a byte array, it would be better to write your XElement directly to a MemoryStream using an appropriately configured XmlWriterSettings and return the byte array thereby created. The following extension method does the job:
public static partial class XNodeExtensions
{
static Encoding DefaultEncoding { get; } = new UTF8Encoding(false); // Disable the BOM because XElement.ToString() does not include it.
public static byte [] ToByteArray(this XNode node, SaveOptions options = default, Encoding encoding = default)
{
// Emulate the settings of XElement.ToString() and XDocument.ToString()
// https://referencesource.microsoft.com/#System.Xml.Linq/System/Xml/Linq/XLinq.cs,2004
// I omitted the XML declaration because XElement.ToString() omits it, but you might want to include it, depending upon your needs.
var settings = new XmlWriterSettings { OmitXmlDeclaration = true, Indent = (options & SaveOptions.DisableFormatting) == 0, Encoding = encoding ?? DefaultEncoding };
if ((options & SaveOptions.OmitDuplicateNamespaces) != 0)
settings.NamespaceHandling |= NamespaceHandling.OmitDuplicates;
return node.ToByteArray(settings);
}
public static byte [] ToByteArray(this XNode node, XmlWriterSettings settings)
{
using var ms = new MemoryStream();
using (var writer = XmlWriter.Create(ms, settings))
node.WriteTo(writer);
return ms.ToArray();
}
}
Now you can format your XElement to a UTF8-encoded byte array by doing:
var bytes = root.ToByteArray();
The extension method has the added advantage that, if you really need to use some encoding other than UTF8, unsupported Unicode characters will be escaped rather than replaced with a fallback as explained in this answer to XmlDocument with Kanji text content is not encoded correctly to ISO-8859-1 using XmlTextWriter.
var bytes = root.ToByteArray(encoding : Encoding.Default);
To check for correctness, you could examine the contents of the byte array in the debugger or your console app by decoding it to a string as follows:
var resultString = Encoding.UTF8.GetString(bytes);
Console.WriteLine(resultString);
Or with the default encoding:
var resultString = Encoding.Default.GetString(bytes);
You could also assert that the contents of the byte array are correct by parsing the contents back to a new XElement and checking that the result is semantically identical to the original by using XNode.DeepEquals():
Assert.IsTrue(
XNode.DeepEquals(root,
XElement.Load(new StreamReader(new MemoryStream(bytes), encoding))));
Demo fiddle here.
Sometimes the byte array b64 is UTF-8, and other times is UTF-16. I keep reading online that C# strings are always UTF-16, but that is not the case for me here. Why is this happening, and how do I fix it? I have a simple method for converting a base64 string to a normal string:
public static string FromBase64(this string input)
{
String corrected = new string(input.ToCharArray());
byte[] b64 = Convert.FromBase64String(corrected);
if (b64[1] == 0)
{
return System.Text.Encoding.Unicode.GetString(b64);
}
else
{
return System.Text.Encoding.UTF8.GetString(b64);
}
}
The same thing is happening to my base 64 encoder:
public static string ToBase64(this string input)
{
String b64 = Convert.ToBase64String(input.GetBytes());
return b64;
}
public static byte[] GetBytes(this string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
}
Example:
On my computer, "cABhAHMAcwB3AG8AcgBkADEA" decodes to:
'p','\0','a','\0','s','\0','s','\0','w','\0','o','\0','r','\0','d','\0','1','\0'
But on my coworkers computer it is:
'p','a','s','s','w','o','r','d','1'
Edit:
I know that the string I create comes from a textbox, and that the file where I am saving it to is always going to be UTF-8, so everything is pointing to the Convert method causing my encoding switch.
Update:
After digging in further, it appears that my coworker had a very important line commented in his version of the code, the one that saves the value read from file to the hashtable. The default value I was using is a UTF-8 base64 value, so I am going to correct the default, to a utf-16 value, then I can clean up the code removing any UTF8 references.
Also, I had been naively using the UTF-8 base64 encoding I had retrieved from a website, not realizing what I was getting myself into. The funny part is I would never have found that fact if my coworker hadn't commented the line that saves the values from the file.
Final version of the code:
public static string FromBase64(this string input)
{
byte[] b64 = Convert.FromBase64String(input);
return System.Text.Encoding.Unicode.GetString(b64);
}
public static string ToBase64(this string input)
{
String b64 = Convert.ToBase64String(input.GetBytes());
return b64;
}
public static byte[] GetBytes(this string str)
{
return System.Text.Encoding.Unicode.GetBytes(str);
}
First of all, I want to debunk the title of the question:
Convert.FromBase64String() returns Unicode sometimes, or UTF-8
That is not the case. Give then same input, valid base64 encoded text, Convert.FromBase64String() always returns the same output.
Moving on, you cannot determine definitively, just by examining the payload, the encoding used for a string. You attempt to do this with
if (b64[1] == 0)
// encoding must be UTF-16
This is not the case. The overwhelming majority of UTF-16 character elements fail that test. It does not matter how you try to write this test it is doomed to fail. And that is because there exist byte arrays that are well-defined strings when interpreted as different encodings. In other words it is possible, for instance, to construct byte arrays that are valid when considered as either UTF-8 or UTF-16.
So, you have to know a priori whether the payload is encoded as UTF-16, UTF-8 or indeed some other encoding.
The solution will be to keep track of the original encoding, before the base64 encoding. Pass that information along with the base64 encoded payload. Then when you decode, you can determine which Encoding to use to decode back to a string.
It looks to me very much that your strings are all coming from UTF-16 .net strings. In which case you won't have UTF-8 strings ever, and should always decode with UTF-16. That is you use Encoding.Unicode.GetString().
Also, the GetBytes method in your code is poor. It should be:
public static byte[] GetBytes(this string str)
{
return Encoding.Unicode.GetBytes(str);
}
Another oddity:
String corrected = new string(input.ToCharArray());
This is a no-op.
Finally, it is quite likely that your text will be more compact when encoded as UTF-8. So perhaps you should consider doing that before applying the base64 encoding.
Regarding your update, what you state is incorrect. This code:
string str = Encoding.Unicode.GetString(
Convert.FromBase64String("cABhAHMAcwB3AG8AcgBkADEA"));
assigns password1 to str wherever it is run.
Try revising the code to make it a little more readable/accurate. As mentioned in my comment and David Hefferman's answer you're trying to do things that either:
A) do nothing
or
B) demonstrate flawed logic
The following code based upon yours works fine:
class Program
{
static void Main(string[] args)
{
string original = "password1";
string encoded = original.ToBase64();
string decoded = encoded.FromBase64();
Console.WriteLine("Original: {0}", original);
Console.WriteLine("Encoded: {0}", encoded);
Console.WriteLine("Decoded: {0}", decoded);
}
}
public static class Extensions
{
public static string FromBase64(this string input)
{
return System.Text.Encoding.Unicode.GetString(Convert.FromBase64String(input));
}
public static string ToBase64(this string input)
{
return Convert.ToBase64String(input.GetBytes());
}
public static byte[] GetBytes(this string str)
{
return System.Text.Encoding.Unicode.GetBytes(str);
}
}
What you are doing is no different than encoding data in either EBCDIC or ASCII, then trying to figure out which was used during the decode. As you have already discovered, this is not going to work very well.
The only way to get this to work correctly is to have a single encoding format used by all participants. This is a fundamental concept of communications.
Pick an encoding - let's say UTF-8 - and use it for all transformations between String and byte[]. This will ensure that you have accurate knowledge of the format of the payload and how to deal with it, as David Tanner has been telling you.
Here's the basic form:
public static string ToBase64(this string self)
{
byte[] bytes = Encoding.UTF8.GetBytes(self);
return Convert.ToBase64String(bytes);
}
public static string FromBase64(this string self)
{
byte[] bytes = Convert.FromBase64String(self);
return Encoding.UTF8.GetString(bytes);
}
Regardless of whatever weirdness might be happening between your computers, this code will produce the same encoded strings.
I'm working on implementing a hosted checkout, and the hosted checkout is supposed to redirect the user back to my website so that I can show a custom receipt page.
This is a sample querystring that I'd get back:
trnApproved=0&trnId=10000000&messageId=71&messageText=Declined&authCode=000000&responseType=T&trnAmount=20.00&trnDate=9%2f23%2f2011+9%3a30%3a56+AM&trnOrderNumber=1000000&trnLanguage=eng&trnCustomerName=FirstName+LastName&trnEmailAddress=something_something%40gmail.com&trnPhoneNumber=1235550123&avsProcessed=0&avsId=0&avsResult=0&avsAddrMatch=0&avsPostalMatch=0&avsMessage=Address+Verification+not+performed+for+this+transaction.&cvdId=3&cardType=VI&trnType=P&paymentMethod=CC&ref1=9dae6af7-7c22-4697-b23a-413d8a129a75&ref2=&ref3=&ref4=&ref5=&hashValue=33dacf84682470f267b2cc6d528b1594
To validate the request, I'm supposed to remove &hashValue=f3cf58ef0fd363e0c2241938b04f1068 from the end of the querystring, and then append a key. I then perform an MD5 hash of the entire string, and the result should be 33dacf84682470f267b2cc6d528b1594, same as the original.
This is easy, except that a few of the fields are causing a problem for me. This is the code I use (taken from a dummy application, so you can ignore some of the bad coding):
// Split up the query string parameters
string[] parameters = GetQueryString().Split(new[] { "&" }, StringSplitOptions.None);
var querySet = new List<string>();
// Rebuild the query string, encoding the values.
foreach (string s in parameters)
{
// Every field that contains a "." will need to be encoded except for trnAmount
querySet.Add(param.Contains("trnAmount") ? param : UrlEncodeToUpper(param));
}
// Create the querystring without the hashValue, we need to calculate our hash without it.
string qs = string.Join("&", querySet.ToArray());
qs = qs.Substring(0, qs.IndexOf("&hashValue"));
qs = qs + "fb76124fea73488fa11995dfa4cbe89b";
var encoding = new UTF8Encoding();
var md5 = new MD5CryptoServiceProvider();
var hash = md5.ComputeHash(encoding.GetBytes(qs));
var calculatedHash = BitConverter.ToString(hash).Replace("-", String.Empty).ToLower();
This is the UrlEncode method I use.
private static string UrlEncodeToUpper(string value)
{
// Convert their encoding into uppercase so we can do our hash
value = Regex.Replace(value, "(%[0-9af][0-9a-f])", c => c.Value.ToUpper());
// Encode the characters that they missed
value = value.Replace("-", "%2D").Replace(".", "%2E").Replace("_", "%5F");
return value;
}
This all works (until someone enters a character I haven't accounted for), except this seems more complicated than it should be. I know I'm not the only one who has to implement this HCO into an ASP.NET application, so I don't think the simple validation should be so complicated.
Am I missing an easier way to do this? Having to loop through the fields, encoding some of them while skipping others, converting their encoding to uppercase and then selectively replacing characters seems a little... odd.
Here's a better way to work with query strings:
var queryString = "trnApproved=0&trnId=10000000&messageId=71&messageText=Declined&authCode=000000&responseType=T&trnAmount=20.00&trnDate=9%2f23%2f2011+9%3a30%3a56+AM&trnOrderNumber=1000000&trnLanguage=eng&trnCustomerName=FirstName+LastName&trnEmailAddress=something_something%40gmail.com&trnPhoneNumber=1235550123&avsProcessed=0&avsId=0&avsResult=0&avsAddrMatch=0&avsPostalMatch=0&avsMessage=Address+Verification+not+performed+for+this+transaction.&cvdId=3&cardType=VI&trnType=P&paymentMethod=CC&ref1=9dae6af7-7c22-4697-b23a-413d8a129a75&ref2=&ref3=&ref4=&ref5=&hashValue=33dacf84682470f267b2cc6d528b1594";
var values = HttpUtility.ParseQueryString(queryString);
// remove the hashValue parameter
values.Remove("hashValue");
var result = values.ToString();
// At this stage result = trnApproved=0&trnId=10000000&messageId=71&messageText=Declined&authCode=000000&responseType=T&trnAmount=20.00&trnDate=9%2f23%2f2011+9%3a30%3a56+AM&trnOrderNumber=1000000&trnLanguage=eng&trnCustomerName=FirstName+LastName&trnEmailAddress=something_something%40gmail.com&trnPhoneNumber=1235550123&avsProcessed=0&avsId=0&avsResult=0&avsAddrMatch=0&avsPostalMatch=0&avsMessage=Address+Verification+not+performed+for+this+transaction.&cvdId=3&cardType=VI&trnType=P&paymentMethod=CC&ref1=9dae6af7-7c22-4697-b23a-413d8a129a75&ref2=&ref3=&ref4=&ref5=
// now add some other query string value
values["foo"] = "bar"; // you can stuff whatever you want it will be properly url encoded
Then I didn't quite understand what you wanted to do. You want to calculate an MD5 on the result? You could do that and then append to the query string.
I need to express a collection of about 10-15 short strings (and maybe some ints) as a fairly compact alphanumeric string - one which I can send as a parameter in a get request.
Basically, I'm thinking that my collection will be a hashtable, and I'd like to serialize it so it looks sort of like a viewstate string (only hopefully not so long!).
eg.
testpage.aspx?code=rO0ABXNyAAlTb21lQ2xhc3PSHbLk6OgfswIAA0kAAWl
and then testpage.aspx can deserialize this back to the original collection.
Is this possible?
One option here is to pick a delimiter, for example ¤; join the strings, encode them (perhaps UTF8), and pack the bytes as base-64...
string[] data = {"abc","123", "def"};
string s = string.Join("¤", data);
byte[] raw = Encoding.UTF8.GetBytes(s);
string alphaNumeric = Convert.ToBase64String(raw); // send this
(you may need to handle the few non-alphanumeric characters that base-64 uses).
And to reverse it:
raw = Convert.FromBase64String(alphaNumeric);
s = Encoding.UTF8.GetString(raw);
data = s.Split('¤');
If you want to send key/value pairs... well, the obvious choice would be query-string parameters themselves, since they are designed for this. But if you need it as a byte-stream:
var data = new DbConnectionStringBuilder();
data["foo"] = "abc";
data["bar"] = "123 + ;la";
string s = data.ConnectionString;
byte[] raw = Encoding.UTF8.GetBytes(s);
string alphaNumeric = Convert.ToBase64String(raw); // send this
raw = Convert.FromBase64String(alphaNumeric);
s = Encoding.UTF8.GetString(raw);
data.ConnectionString = s;
foreach (string key in data.Keys)
{
Console.WriteLine(key + "=" + data[key]);
}
Why don't you just serialize the data using protobuf-net and pass it through the Session? Or, if it has to be a string, just use XmlSerializer?
Personally, passing serialized data through the URL seems really bad to me!
you can serialize your dictionary/hashtable to JSON,
and then change it to base64 (just to make it a tad less visible and resolve possible usage of URL characters etc).
or you can just URLEncode it.
You can use standard .Net serialization and serialize your object to a MemoryStream. You can then read out the contents of the MemoryStream as a byte array and use Convert.ToBase64String on the array to get a string representation of it.
To deserialize it you can do the opposite.
If you are worried that the serialized object is too large, you can wrap the MemoryStream in a System.IO.Compression.DeflateStream to compress it.
Thank you all, putting it together I wrote a test console app. I was a bit disappointed that the resulting string was so long.
My example implements a DeflateStream, which for my datasize probably introduces more overhead than it saves with compression. But even without the compression it was still pretty big.
What I was hoping to achieve was to make something slightly more compact (obfuscation for the user was a plus, but not critical) - I suspect that it'd actually be better for me to use a plain old parameterized string. Maybe JSON might be ok, but I'm using ASP.net 2.0, and I don't think that I get a readybaked json serializer there.
Nonetheless, I learnt something new and interesting, so thanks for that!
static void Main(string[] args)
{
Hashtable ht1 = new Hashtable(1);
ht1.Add("name", "bob");
Console.WriteLine(ToCompactString(ht1));
Console.WriteLine();
string str = "name:bob";
Console.WriteLine(ToCompactString(str));
Console.ReadLine();
}
private static string ToCompactString(object obj)
{
var ms = new MemoryStream();
var ds = new DeflateStream(ms, CompressionMode.Compress);
var bf = new BinaryFormatter();
bf.Serialize(ds, obj);
byte[] bytes = ms.ToArray();
ds.Close();
ms.Close();
string result = Convert.ToBase64String(bytes);
return result;
}