I was given a url that, in the documentation says the following:
The response is a binary encoded octet stream of the raw 52 byte
archive records.
I created this code to test it out:
public static void GetHistory(string url)
{
var response = WebRequestClasses.GetResponse(url);
var nres = DeserializeObject(response);
Console.WriteLine(nres);
}
public static object DeserializeObject(string str)
{
byte[] bytes = Convert.FromBase64String(str);
using (MemoryStream stream = new MemoryStream(bytes))
{
return new BinaryFormatter().Deserialize(stream);
}
}
It is returning the following error:
An unhandled exception of type 'System.FormatException' occurred in
mscorlib.dll
Additional information: The input is not a valid Base-64 string as it
contains a non-base 64 character, more than two padding characters, or
an illegal character among the padding characters.
I am getting a response as you can see from the print screen:
I've tried searching on octet parsing but I'm not getting anything. I'm not very familiar with encoding but I would assume it's a problem with the Base=64 vs octet stream?
Related
I am using C# and .NET to encode and decode base64 string. The following are snippets of my code:
Base64 encoding:
using (var stream = new MemoryStream())
…...
return Convert.ToBase64String(stream.ToArray());
}
Base64 decoding
byte[] bytes = Convert.FromBase64String(messageBody);
My code fails 99% of the time with 1% chance to succeed though. The stack trace is as follows:
5xx Error Returned:System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. at System.Convert.FromBase64_ComputeResultLength(Char inputPtr, Int32 inputLength) at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength) at System.Convert.FromBase64String(String s)*
Does anyone know what can cause base64 decoding to fail? My encoding and decoding methods are symmetric and I am really confused about what can be the root cause for this issue?
Thanks for all your replies.
It turned out there were still some old messages in Json format that previously failed in getting delivered and kept retrying in our system; however the new code change of our receiving side got deployed and our receiving side starts to expect messages in protobuf format which results in Deserialization failure when receiving old Json format messages.
In order to debug an issue like this I usually write some tests or create a console app to watch the variables as they change from function to function.
One of the possible scenario's for base64 decoding to fail is if the decoder input is HTMLEncoded. This is common when you pass an encrypted string into a URL for example. It will automatically be HTML encoded and then it sometimes can and sometimes can't be decoded depending on the characters that the encoded output has.
Here's a simple console app to demonstrate this.
class Program
{
static void Main(string[] args)
{
string input = "testaa";
TestEncodeDecode("test");
TestEncodeDecode("testa");
TestEncodeDecode("testaa");
Console.ReadLine();
}
private static void TestEncodeDecode(string input)
{
string encoded = Encode(input);
Console.WriteLine($"Encoded: {encoded}");
string htmlEncoded = WebUtility.UrlEncode(encoded);
Console.WriteLine($"htmlEncoded: {htmlEncoded}");
string decodedString = Decode(htmlEncoded);
Console.WriteLine($"Decoded: {decodedString}");
Console.WriteLine();
}
private static string Decode(string htmlEncoded)
{
try
{
byte[] decoded = Convert.FromBase64String(htmlEncoded);
return Encoding.ASCII.GetString(decoded);
}
catch(Exception)
{
return "Decoding failed";
}
}
private static string Encode(string input)
{
byte[] bytes = Encoding.ASCII.GetBytes(input);
using (var stream = new MemoryStream())
{
stream.Write(bytes);
return Convert.ToBase64String(stream.ToArray());
}
}
}
You'll see that the first two arguments ("test" and "testa") fail to decode, but the third ("testaa") will succeed.
In order to "fix" this, change the Decode method as follows:
private static string Decode(string htmlEncoded)
{
try
{
string regularEncodedString = WebUtility.UrlDecode(htmlEncoded);
byte[] decoded = Convert.FromBase64String(regularEncodedString);
return Encoding.ASCII.GetString(decoded);
}
catch(Exception)
{
return "Decoding failed";
}
}
I encode data of images in base64 using python before sending it to the server which is written in C#. The data that is received is identical to the data that is being sent. However, when I decode the encoded string I get a different result.
Here is the code that takes a screenshot and encodes it in base64:
screen_shot_string_io = StringIO.StringIO()
ImageGrab.grab().save(screen_shot_string_io, "PNG")
screen_shot_string_io.seek(0)
return base64.b64encode(screen_shot_string_io.getvalue())
it is sent as is to the server and the server receives the encoded string correcetly with no data corruption.
Here is the c# code that decodes the string:
byte[] decodedImg = new byte[bytesReceived];
FromBase64Transform transfer = new FromBase64Transform();
transfer.TransformBlock(encodedImg, 0, bytesReceived, decodedImg, 0);
So does anyone know why when the data is decoded the result is incorrect?
If it was me I would simply use the Convert.FromBase64String() and not mess with the FromBase64Transform. You don't have all the details here, so I had to improvise.
In Python I took a screen shot, encoded it, and wrote to file:
# this example converts a png file to base64 and saves to file
from PIL import ImageGrab
from io import BytesIO
import base64
screen_shot_string_io = BytesIO()
ImageGrab.grab().save(screen_shot_string_io, "PNG")
screen_shot_string_io.seek(0)
encoded_string = base64.b64encode(screen_shot_string_io.read())
with open("example.b64", "wb") as text_file:
text_file.write(encoded_string)
And in C# I decoded the file contents are wrote the binary:
using System;
using System.IO;
namespace Base64Decode
{
class Program
{
static void Main(string[] args)
{
byte[] imagedata = Convert.FromBase64String(File.ReadAllText("example.b64"));
File.WriteAllBytes("output.png",imagedata);
}
}
}
If you have a properly encoded byte array, then convert the array to string and then decode the string.
public static void ConvertByteExample()
{
byte[] imageData = File.ReadAllBytes("example.b64");
string encodedString = System.Text.Encoding.UTF8.GetString(imageData); //<-- do this
byte[] convertedData = Convert.FromBase64String(encodedString);
File.WriteAllBytes("output2.png", convertedData);
}
Here is my method for sending data:
// this method serializes an object and is returned as a string (Base64 encoded)
public void Send(Packet packetData)
{
try
{
StreamWriter w = new StreamWriter(cli.GetStream());
string s = SerializeObject(packetData);
w.WriteLine(s + "\n");
w.Flush();
}
catch (ObjectDisposedException) { ShutdownClient(-2); }
}
cli is TcpClient object
Packet is a serializable object
Here's my receive method:
private string GetMessage(StreamReader r)
{
try
{
string s = r.ReadLine();
s = s.Replace(" ", "");
// this string is returned and then deserialized
return s;
}
catch (Exception e) { System.Windows.Forms.MessageBox.Show(e.Message); return null; }
}
When I use this, 50% of the time it works. If it doesn't work, it's because of this:
"The input stream is not a valid binary format. The starting contents (in bytes) are: 6D-2E-44-72-61-77-69-6E-67-2E-43-6F-6C-6F-72-0F-00 ..."
I have tried using Encoding.Default.GetString/GetBytes in replacement of Base64, but then I get this:
"Binary stream '0' does not contain a valid BinaryHeader. Possible causes are invalid stream or object version change between serialization and deserialization."
If I print out the length of this (Default encoded) string, it is 183. But if I print out the actual string, nothing is printed.
How else can I send a byte[] as a string over StreamWriter?
How else can I send a byte[] as a string
Not the way you do it now, the byte[] content will get corrupted when the string is normalized. A string should only ever be used to store Unicode characters. Not every Unicode codepoint is valid.
If using a string is paramount then you should use Convert.ToBase64String() at the transmitter, Convert.FromBase64String() at the receiving end.
Do keep in mind that TCP is entirely capable of transferring binary data. You possibly fell into this hole because TCP implements a stream, it doesn't do anything to help you transmit messages. The simple way to transfer a binary 'message' is to first write the Length of the byte[]. The receiver first read that length, then knows what it should pass to the Read() call to recover the byte[] back from the TCP stream.
I'm trying to send a string containing special characters through a TcpClient (byte[]). Here's an example:
Client enters "amé" in a textbox
Client converts string to byte[] using a certain encoding (I've tried all the predefined ones plus some like "iso-8859-1")
Client sends byte[] through TCP
Server receives and outputs the string reconverted with the same encoding (to a listbox)
Edit :
I forgot to mention that the resulting string was "am?".
Edit-2 (as requested, here's some code):
#DJKRAZE here's a bit of code :
byte[] buffer = Encoding.ASCII.GetBytes("amé");
(TcpClient)server.Client.Send(buffer);
On the server side:
byte[] buffer = new byte[1024];
Client.Recieve(buffer);
string message = Encoding.ASCII.GetString(buffer);
ListBox1.Items.Add(message);
The string that appears in the listbox is "am?"
=== Solution ===
Encoding encoding = Encoding.GetEncoding("iso-8859-1");
byte[] message = encoding.GetBytes("babé");
Update:
Simply using Encoding.Utf8.GetBytes("ééé"); works like a charm.
Never too late to answer a question I think, hope someone will find answers here.
C# uses 16 bit chars, and ASCII truncates them to 8 bit, to fit in a byte. After some research, I found UTF-8 to be the best encoding for special characters.
//data to send via TCP or any stream/file
byte[] string_to_send = UTF8Encoding.UTF8.GetBytes("amé");
//when receiving, pass the array in this to get the string back
string received_string = UTF8Encoding.UTF8.GetString(message_to_send);
Your problem appears to be the Encoding.ASCII.GetBytes("amé"); and Encoding.ASCII.GetString(buffer); calls, as hinted at by '500 - Internal Server Error' in his comments.
The é character is a multi-byte character which is encoded in UTF-8 with the byte sequence C3 A9. When you use the Encoding.ASCII class to encode and decode, the é character is converted to a question mark since it does not have a direct ASCII encoding. This is true of any character that has no direct coding in ASCII.
Change your code to use Encoding.UTF8.GetBytes() and Encoding.UTF8.GetString() and it should work for you.
Your question and your error is not clear to me but using Base64String may solve the problem
Something like this
static public string EncodeTo64(string toEncode)
{
byte[] toEncodeAsBytes
= System.Text.ASCIIEncoding.ASCII.GetBytes(toEncode);
string returnValue
= System.Convert.ToBase64String(toEncodeAsBytes);
return returnValue;
}
static public string DecodeFrom64(string encodedData)
{
byte[] encodedDataAsBytes
= System.Convert.FromBase64String(encodedData);
string returnValue =
System.Text.ASCIIEncoding.ASCII.GetString(encodedDataAsBytes);
return returnValue;
}
I am having a problem with reading my sms through putty, Its beacuse I type AT+CMGL="ALL" but the message(text) and number are just numbers, I read that my gms modem nokia s10 uses UCS2, but I dont know what to do here? how can I read my message intead of just seeing numbers?? help please
Also I am using this code from codeproject and I changed this line but It is the same result as putty just number in ucs2
public ShortMessageCollection ReadSMS(SerialPort port, string p_strCommand)
{
// Set up the phone and read the messages
ShortMessageCollection messages = null;
try
{
#region Execute Command
// Check connection
ExecCommand(port,"AT", 300, "No phone connected");
// Use message format "Text mode"
ExecCommand(port,"AT+CMGF=1", 300, "Failed to set message format.");
// Use character set "PCCP437"
**ExecCommand(port, "AT+CSCS=\"UCS2\"", 300, "Failed to set character set.")**;
// Select SIM storage
ExecCommand(port,"AT+CPMS=\"SM\"", 300, "Failed to select message storage.");
// Read the messages
string input = ExecCommand(port, p_strCommand, 5000, "Failed to read the messages.");
#endregion
#region Parse messages
messages = ParseMessages(input);
#endregion
}
catch (Exception ex)
{
throw ex;
}
if (messages != null)
return messages;
else
return null;
}
Notice that AT+CSCS only affects string parameters to commands and responses. In the case of AT+CMGL the content of the message is not a string, but a <data> format. See the 27.005 specification for more details on that format, it is a bit complicated (only pay attention to the first In the case of SMS part, ignore the second In the case of CBS part).
But the short version of it is that for UCS-2 you will get the data hex encoded (e.g. two characters '2' and 'A' represents one byte with value 0x2A (ASCII/UTF-8 character '*')). So you should decode 4 and 4 received bytes as the hex encoding of the 16 bits in a UCS-2 character.
So decode into a byte array and then convert to string, see Appleman1234's answer for that (his answer does not address the core issue, namely the hex decoding).
To convert from the UCS-2 encoding store the result (input) in a byte array instead of a string and then call
System.Text.Encoding enc = Encoding.Unicode;
string myString = enc.GetString(myByteArray);
If the UCS-2 encoding is Big Endian then change System.Text.Encoding enc = Encoding.Unicode; to
System.Text.Encoding enc = Encoding.BigEndianUnicode;.
Related resources include:
Unicode and .NET
C# big-endian UCS-2