Develop Reference

Convert from hex string (from UCS-2) into UTF-8

I'm using a third party SMS provider and have hit an issue with converting from UCS-2 messages back into readable text.
Their API documentation has this code sample which converts UCS-2 messges into what I'm picking up on the API.
string message = "Это тестовое сообщение юникода";
byte[] ba = Encoding.BigEndianUnicode.GetBytes (message);
var hexString = BitConverter.ToString (ba);
Console.WriteLine ("#U" + hexString.Replace("-",""));
Which converts the message string into
#U042D0442043E00200442043504410442043E0432043E043500200441043E043E043104490435043D043804350020044E043D0438043A043E04340430
This looks like the UCS-2 messages I'm picking up from their API.
Unfortunately they don't give any code samples of how to convert the messages back into a readable form.
I'm sure its not there in the docs because its something simple - but I just seem to figure out how to do it.

To reverse what you have (the string of hex prefixed with #U)
var message = "Это тестовое сообщение юникода";
var ba = Encoding.BigEndianUnicode.GetBytes(message);
var hexString = BitConverter.ToString(ba);
var encoded = "#U" + hexString.Replace("-", "");
Console.WriteLine(encoded);
// reverse
var bytes = Enumerable.Range(2, encoded.Length-2)
.Where(x => x % 2 == 0)
.Select(x => Convert.ToByte(encoded.Substring(x, 2), 16))
.ToArray();
var result = Encoding.BigEndianUnicode.GetString(bytes);
Console.WriteLine(result);
Output
#U042D0442043E00200442043504410442043E0432043E043500200441043E043E043104490435043D043804350020044E043D0438043A043E04340430
Это тестовое сообщение юникода
Demo here

looks like this would be the reverse:
string message = Encoding.BigEndianUnicode.GetString(ba);
The extraction of bytes could be done by such a method:
private IEnumerable<byte> GetTheBytes(string uc2Message)
{
string bytesOnly = uc2Message.Trim('#', 'U');
for (int i = 0; i < bytesOnly.Length-2; i+=2)
{
yield return Convert.ToByte($"{bytesOnly[i]}{bytesOnly[i+1]}", 16);
}
}
Console.WriteLine(Encoding.BigEndianUnicode.GetString(GetTheBytes(uc2Message).ToArray()));

How should I decode a UTF-8 string

I have a string like:
About \xee\x80\x80John F Kennedy\xee\x80\x81\xe2\x80\x99s Assassination . unsolved mystery \xe2\x80\x93 45 years later. Over the last decade, a lot of individuals have speculated on conspiracy theories that ...
I understand that \xe2\x80\x93 is a dash character. But how should I decode the above string in C#?

If you have a string like that, then you have used the wrong encoding when you decoded it in the first place. There is no "UTF-8 string", the UTF-8 data is whent the text is encoded into binary data (bytes). When it's decoded into a string, then it's not UTF-8 any more.
You should use the UTF-8 encoding when you create the string from binary data, once the string is created using the wrong encoding, you can't reliably fix it.
If there is no other alternative, you could try to fix the string by encoding it again using the same wrong encoding that was used to create it, and then decode it using the corrent encoding. There is however no guarantee that this will work for all strings, some characters will simply be lost during the wrong decoding. Example:
// wrong use of encoding, to try to fix wrong decoding
str = Encoding.UTF8.GetString(Encoding.Default.GetBytes(str));

Scan the input string char-by-char and convert values starting with \x (string to byte[] and back to string using UTF8 decoder), leaving all other characters unchanged:
static string Decode(string input)
{
var sb = new StringBuilder();
int position = 0;
var bytes = new List<byte>();
while(position < input.Length)
{
char c = input[position++];
if(c == '\\')
{
if(position < input.Length)
{
c = input[position++];
if(c == 'x' && position <= input.Length - 2)
{
var b = Convert.ToByte(input.Substring(position, 2), 16);
position += 2;
bytes.Add(b);
}
else
{
AppendBytes(sb, bytes);
sb.Append('\\');
sb.Append(c);
}
continue;
}
}
AppendBytes(sb, bytes);
sb.Append(c);
}
AppendBytes(sb, bytes);
return sb.ToString();
}
private static void AppendBytes(StringBuilder sb, List<byte> bytes)
{
if(bytes.Count != 0)
{
var str = System.Text.Encoding.UTF8.GetString(bytes.ToArray());
sb.Append(str);
bytes.Clear();
}
}
Output:
About John F Kennedy’s Assassination . unsolved mystery – 45 years later. Over the last decade, a lot of individuals have speculated on conspiracy theories that ...

Finally I've used something like this:
public static string UnescapeHex(string data)
{
return Encoding.UTF8.GetString(Array.ConvertAll(Regex.Unescape(data).ToCharArray(), c => (byte) c));
}

How to fix this Base 64 array issue

I have a base64 string in the view side. If I pass the whole base64 array at a time I can convert that in to bytes like this
byte[] myBinary = Convert.FromBase64String(data);
where data represents the data that is coming form the view page. But I am having huge data. So, I am splitting the data in the view page like
var arr = [];
for (var i = 0; i < data.length - 1; i += 1000000) {
arr.push(data.substr(i, 1000000));
}
And now I am passing the data to the controller
for (var x = 0; x < arr.length; x++) {
if (x = 0) {
r = "first";
}
else if (x = arr.length - 1) {
r = "last";
}
else {
r = "next";
}
$.post('/Home/Content', { content: e, data: r }, function (d) {
});
}
And in the controller side I have written code like:
public JsonResult Content(string content, string data)
{
datavalueincont += content;
if (data == "last")
{
byte[] myBinary = Convert.FromBase64String(datavalueincont);
var fname = "D://sri//data.mp4";
FileStream stream = new FileStream(fname, FileMode.Create, FileAccess.Write);
System.IO.BinaryWriter br = new System.IO.BinaryWriter(stream);
br.Write(myBinary);
br.Close();
read.Close();
stream.Close();
}
return Json("suc", JsonRequestBehavior.AllowGet);
}
But I am getting error at:
byte[] myBinary = Convert.FromBase64String(datavalueincont);
and that error is
The input is not a valid Base-64 string as it contains a non-base 64
character, more than two padding characters, or an illegal character
among the padding characters.
How can I rectify this. If I pass the data at a time I am able to get the bytes in the
myBinary array. Hope you understand my question.

I have an idea.
As you are sending your data using Ajax, nothing ensures you that your chunks will be sent sequentially.
So maybe when you aggregate your data your chunks are not in a good order.
Try to make your Ajax call sequentially to confirm this point.
[Edit]
something like this (not tested):
var data = [];//your data
var sendMoreData = function (firstTime) {
if (data.length == 0)
return;//no more data to send
var content = data.shift();
var r = firstTime ? "first" :
data.length == 0 ? "last":
"next";
$.post('/Home/Content', { content: content, data: r }, function (d) {
sendMoreData();
});
};
sendMoreData(true);

You can't use byte[] myBinary = Convert.FromBase64String(datavalueincont); until you have the fully encrypted string.
The problem is that you're splitting the Base64 data into chunks after which you send those chunks to the server -> on the server you're trying to convert back from base64 on each individual chunk rather than the whole collection of chunks.
The way I see it, you have 2 options:
Encrypt each individually split chunk of data to base64 (rather than the whole thing before hand) and decrypt it on the server.
Encrypt the whole thing, then split it into pieces (like you're doing now) -> send it to the server -> cache each result (any way you
want -> session, db etc.) till you get the last one -> decrypt at
once
As a side note:
if (x = 0) {
r = "first";
}
else if (x = arr.length - 1) {
r = "last";
}
should really be:
if (x == 0) {
r = "first";
}
else if (x == arr.length - 1) {
r = "last";
}
Not sure if typo, just sayin'.

I think your concept is fine... from what I understand you are doing the following...
View converts binary data to Base64String
View splits string into chunks and sends to controller
Controller waits for all chunks and concatenates them
Controller converts from Base64String
The problem is in how you are splitting your data in the view... I am assuming the splitting code has some extra padding characters on the end maybe?
var arr = [];
for (var i = 0; i < data.length - 1; i += 1000000) {
arr.push(data.substr(i, 1000000));
}
I can't build a test rig to check the code but certainly on your last section of text you can't get 1000000 characters from .substr because there aren't that many characters in the string. I don't know what .substr will return but I would troubleshoot the splitting section of code to find the problem.

Are you sure that datavalueincont += content; is really aggregating all your data. How do you store datavalueincont after each http request?
Maybe you are only missing that.
Have you debugged when data == "last" to see if you have all your data in datavalueincont ?

string values to byte array without converting

I'm trying to put the values of a string into a byte array with out changing the characters. This is because the string is in fact a byte representation of the data.
The goal is to move the input string into a byte array and then convert the byte array using:
string result = System.Text.Encoding.UTF8.GetString(data);
I hope someone can help me although I know it´s not a very good description.
EDIT:
And maybe I should explain that what I´m working on is a simple windows form with a textbox where users can copy the encoded data into it and then click preview to see the decoded data.
EDIT:
A little more code:
(inputText is a textbox)
private void button1_Click(object sender, EventArgs e)
{
string inputString = this.inputText.Text;
byte[] input = new byte[inputString.Length];
for (int i = 0; i < inputString.Length; i++)
{
input[i] = inputString[i];
}
string output = base64Decode(input);
this.inputText.Text = "";
this.inputText.Text = output;
}
This is a part of a windows form and it includes a rich text box. This code doesn´t work because it won´t let me convert type char to byte.
But if I change the line to :
private void button1_Click(object sender, EventArgs e)
{
string inputString = this.inputText.Text;
byte[] input = new byte[inputString.Length];
for (int i = 0; i < inputString.Length; i++)
{
input[i] = (byte)inputString[i];
}
string output = base64Decode(input);
this.inputText.Text = "";
this.inputText.Text = output;
}
It encodes the value and I don´t want that. I hope this explains a little bit better what I´m trying to do.
EDIT: The base64Decode function:
public string base64Decode(byte[] data)
{
try
{
string result = System.Text.Encoding.UTF8.GetString(data);
return result;
}
catch (Exception e)
{
throw new Exception("Error in base64Decode" + e.Message);
}
}
The string is not encoded using base64 just to be clear. This is just bad naming on my behalf.
Note this is just one line of input.
I've got it. The problem was I was always trying to decode the wrong format. I feel very stupid because when I posted the example input I saw this had to be hex and it was so from then on it was easy. I used this site for reference:
http://msdn.microsoft.com/en-us/library/bb311038.aspx
My code:
public string[] getHexValues(string s)
{
int j = 0;
string[] hex = new String[s.Length/2];
for (int i = 0; i < s.Length-2; i += 2)
{
string temp = s.Substring(i, 2);
this.inputText.Text = temp;
if (temp.Equals("0x")) ;
else
{
hex[j] = temp;
j++;
}
}
return hex;
}
public string convertFromHex(string[] hex)
{
string result = null;
for (int i = 0; i < hex.Length; i++)
{
int value = Convert.ToInt32(hex[i], 16);
result += Char.ConvertFromUtf32(value);
}
return result;
}
I feel quite dumb right now but thanks to everyone who helped, especially #Jon Skeet.

Are you saying you have something like this:
string s = "48656c6c6f2c20776f726c6421";
and you want these values as a byte array? Then:
public IEnumerable<byte> GetBytesFromByteString(string s) {
for (int index = 0; index < s.Length; index += 2) {
yield return Convert.ToByte(s.Substring(index, 2), 16);
}
}
Usage:
string s = "48656c6c6f2c20776f726c6421";
var bytes = GetBytesFromByteString(s).ToArray();
Note that the output of
Console.WriteLine(System.Text.ASCIIEncoding.ASCII.GetString(bytes));
is
Hello, world!
You obviously need to make the above method a lot safer.

Encoding has the reverse method:
byte[] data = System.Text.Encoding.UTF8.GetBytes(originalString);
string result = System.Text.Encoding.UTF8.GetString(data);
Debug.Assert(result == originalString);
But what you mean 'without converting' is unclear.

One way to do it would be to write:
string s = new string(bytes.Select(x => (char)c).ToArray());
That will give you a string that has one character for every single byte in the array.
Another way is to use an 8-bit character encoding. For example:
var MyEncoding = Encoding.GetEncoding("windows-1252");
string s = MyEncoding.GetString(bytes);
I'm think that Windows-1252 defines all 256 characters, although I'm not certain. If it doesn't, you're going to end up with converted characters. You should be able to find an 8-bit encoding that will do this without any conversion. But you're probably better off using the byte-to-character loop above.

If anyone still needs it this worked for me:
byte[] result = Convert.FromBase64String(str);

Have you tried:
string s = "....";
System.Text.UTF8Encoding.UTF8.GetBytes(s);

Converting string to byte array in C# (CSV)

I wrote a function to convert byte[] to string, and I add ";" after each byte. Now I want to convert this string to byte[] by splitting the string (similar to a CSV string).
public string ByteArrayToString(byte[] byteArray,string s)
{
for (int i = 0; i < byteArray.Length; i++)
{
s += byteArray[i].ToString() + ";";
}
s = s.Substring(0, s.Length - 1);
return s;
}
How could I write a function to convert this string to that byte array again?

try this
var byteArray = new byte[] {123, 11, 111};
var stringBytes = string.Join(";", byteArray.Select(b => b.ToString()));
var newByteArray = stringBytes.Split(';').Select(s => byte.Parse(s)).ToArray();

I guess that you want to get rid of the ; when converting also. I think you want to do something like this:
byte[] result = Encoding.UTF8.GetBytes(s.Replace(";",""));
This will fail if the original byte array actually contains a ;that is valid data, but in that case you will have lots of problems anyway since your "CSV" file will be wrongly formatted.

Consider using Split String

StringBuilder will be useful instead of String (Performance wise).
With StringBuilder:
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(objStringBuilder.ToString());
with String:
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(objString);

System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
return encoding.GetBytes(yourStringVariable);

I know you already know the answer by now... but this code solves the problem, i hope it helps someone else.
int counter= 0;
string cadena = "8,5,6,3,4,6,3"
string[] foto = cadena.Split(',');
byte[] fotoFinal = new byte[foto.Length];
foreach (string s in foto)
{
fotoFinal[contador] = Convert.ToByte(s);
counter++;
}

str.Split(new char[]{';'},
StringSplitOptions.RemoveEmptyEntries).Select(s => byte.Parse(s)).ToArray();

Simply :)
public static byte[] Bytes ( this string Key )
{
return Enumerable.Range(0, Key.Binary().Length / 8 )
.Select(Index => Convert.ToByte(
Key.Binary().Substring(Index * 8, 8), 2))
.ToArray();
}

string[] sbytes = sl.Split(',');
byte[] b = new byte[sbytes.Length];
for (int j = 0; j < sbytes.Length; j++)
{
byte newByte = byte.Parse(sbytes[j], System.Globalization.NumberStyles.HexNumber);
b[j] = newByte;
}
I like using number styles hex number.