Develop Reference

Convert.ToBase64String throws 'System.OutOfMemoryException' for byte [] (file: large size)

I am trying to convert byte[] to base64 string format so that i can send that information to third party. My code as below:
byte[] ByteArray = System.IO.File.ReadAllBytes(path);
string base64Encoded = System.Convert.ToBase64String(ByteArray);
I am getting below error:
Exception of type 'System.OutOfMemoryException' was thrown. Can you
help me please ?

Update
I just spotted #PanagiotisKanavos' comment pointing to Is there a Base64Stream for .NET?. This does essentially the same thing as my code below attempts to achieve (i.e. allows you to process the file without having to hold the whole thing in memory in one go), but without the overhead/risk of self-rolled code / rather using a standard .Net library method for the job.
Original
The below code will create a new temporary file containing the Base64 encoded version of your input file.
This should have a lower memory footprint, since rather than doing all data at once, we handle it several bytes at a time.
To avoid holding the output in memory, I've pushed that back to a temp file, which is returned. When you later need to use that data for some other process, you'd need to stream it (i.e. so that again you're not consuming all of this data at once).
You'll also notice that I've used WriteLine instead of Write; which will introduce non base64 encoded characters (i.e. the line breaks). That's deliberate, so that if you consume the temp file with a text reader you can easily process it line by line.
However, you can amend per your needs.
void Main()
{
var inputFilePath = #"c:\temp\bigfile.zip";
var convertedDataPath = ConvertToBase64TempFile(inputFilePath);
Console.WriteLine($"Take a look in {convertedDataPath} for your converted data");
}
//inputFilePath = where your source file can be found. This is not impacted by the below code
//bufferSizeInBytesDiv3 = how many bytes to read at a time (divided by 3); the larger this value the more memory is required, but the better you'll find performance. The Div3 part is because we later multiple this by 3 / this ensures we never have to deal with remainders (i.e. since 3 bytes = 4 base64 chars)
public string ConvertToBase64TempFile(string inputFilePath, int bufferSizeInBytesDiv3 = 1024)
{
var tempFilePath = System.IO.Path.GetTempFileName();
using (var fileStream = File.Open(inputFilePath,FileMode.Open))
{
using (var reader = new BinaryReader(fileStream))
{
using (var writer = new StreamWriter(tempFilePath))
{
byte[] data;
while ((data = reader.ReadBytes(bufferSizeInBytesDiv3 * 3)).Length > 0)
{
writer.WriteLine(System.Convert.ToBase64String(data)); //NB: using WriteLine rather than Write; so when consuming this content consider removing line breaks (I've used this instead of write so you can easily stream the data in chunks later)
}
}
}
}
return tempFilePath;
}

How to send large size of the caption on telegram bot using c#?

Hi i am working on telegram bot sync with my app and sending the image with caption. i am getting success for sending image on the telegram bot.but when i am sending long caption with image half of it is not getting passed.i don't know if any limit of the caption or text send on telegram i have no idea about it. please any one know then please tell me.i want to send full caption with image. my caption is long then i want to make 2 part of the caption and send it. here below i have try like this code.
This my api =>
[System.Web.Http.AcceptVerbs("POST")]
public void SendCasesOnTelegramBot()
{
try
{
DataSet ds = DataAccess.ExecuteDataset(Setting.ConnectionString(), "GetPostForTelegramBot");
if (ds != null && ds.Tables.Count > 0)
{
if (ds.Tables[0] != null && ds.Tables[0].Rows.Count > 0)
{
var Image = ds.Tables[0].Rows[0]["Url"].ToString();
byte[] imageData = null;
WebClient webClient = new WebClient();
imageData = webClient.DownloadData(Image);
MemoryStream ms = new MemoryStream(imageData);
var sb = new StringBuilder();
sb.Append(Environment.NewLine);
sb.Append(".");
sb.Append(Environment.NewLine);
sb.Append(".");
sb.Append(Environment.NewLine);
if (ds.Tables[0].Rows[0]["Description"].ToString().Length <= 133)
{
sb.AppendLine(ds.Tables[0].Rows[0]["Description"].ToString());
}
else
{
sb.AppendLine(ds.Tables[0].Rows[0]["Description"].ToString().Substring(0, 133));
sb.AppendLine(ds.Tables[0].Rows[0]["Description"].ToString().Substring(134, 0));
}
sb.Append(Environment.NewLine);
sb.Append("join our team...");
Bot.SendPhotoAsync("#abc", new FileToSend(ds.Tables[0].Rows[0]["Url"].ToString(), ms), sb.ToString());
}
}
}
catch (Exception ex)
{
}
}
This is my caption =>
**it is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using 'Content here, content here', making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for 'lorem ipsum' will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like).**
any one please help me i have not idea how can do that.

As #tashakori said, you can send only 1024 char in caption.
There is a way to dirty pass it, use sendMessage instead of sendPhoto, and use HTML or Markdown to put photo file link in first char (can be emoji).
Your photo will show as preview, and you can put up to 4096 char as caption.
BTW, you can't use bold/italic in photo caption, but you can use it now. :)

Retrieving entire line from a socket in C#?

I have a simple client-server system sending plain text - though only commands that have been approved. The server is a Python system - and I've confirmed proper connections.
However, the client is C# - in Unity. Searching for examples, I stumbled across this bit of code. It does seem to do what I want, however, only partially:
public String readSocket()
{
if (!socketReady)
return "";
if (theStream.DataAvailable)
return theReader.ReadLine();
return "";
}
The strings I am sending end with \n, but I'm only getting half the message like this:
Message A:
claim_2
Message B:
_20_case
claim_1
I know this probably has to do with how I'm directly reading the line but I cannot find any better examples - strangely enough, everyone seems to point back at this snippet even when multiple people point out the problems.
Can anything be done to fix this bit of code properly?
In case it helps, I'm sending the information (from my Python server) out like this:
action = str(command) + "_" + str(x) + "_" + str(userid) + "_" + str(user)
cfg.GameSendConnection.sendall((action + "\n").encode("utf-8"))

When you do sockets programming, it is important to note that data might not be
available in one piece. In fact, this is exactly what you are seeing. Your
messages are being broken up.
So why does ReadLine not wait until there's a line to read?.
Here's some simple sample code:
var stream = new MemoryStream();
var reader = new StreamReader(stream);
var writer = new StreamWriter(stream) { AutoFlush = true };
writer.Write("foo");
stream.Seek(0, SeekOrigin.Begin);
Console.WriteLine(reader.ReadLine());
Note that there is no newline at the end. Still, the output of this little
snippet is foo.
ReadLine returns the string up to the first line break or until there is no
more data to read. The exception being reading from a stream that has no more
data to read, then it returns null.
When a NetworkStream has its DataAvailable property return true, it has
data. But as mentioned before, there is no guarantee whatsoever about what that
data is. It might be a single byte. Or a part of a message. Or a full message
plus part of the next message. Note that depending on the encoding, it could
even be possible to receive only part of a character. Not all character
encodings have all characters be at most a single byte. This includes UTF-8, which cfg.GameSendConnection.sendall((action + "\n").encode("utf-8")) sends.
How to solve this? Read bytes, not lines. Put them in some buffer. After every
read, check if the buffer contains a newline. If it does, you now have a full
message to handle. Remove the message up to and including the newline from the
buffer and keep appending new data to it until the next newline is received. And
so on.

This is how I process the entire line in my similar application, which is a very simple code, and your code may be different, but you can get the idea.
private string incompleteRecord = "";
public void ReadSocket()
{
if (_networkStream.DataAvailable)
{
var buffer = new byte[8192];
var receivedString = new StringBuilder();
do
{
int numberOfBytesRead = _networkStream.Read(buffer, 0, buffer.Length);
receivedString.AppendFormat("{0}", Encoding.UTF8.GetString(buffer, 0, numberOfBytesRead));
} while (_networkStream.DataAvailable);
var bulkMsg = receivedString.ToString();
// When you receive data from the socket, you can receive any number of messages at a time
// with no guarantee that the last message you receive will be complete.
// You can receive only part of a complete message, with next part coming
// with the next call. So, we need to save any partial messages and add
// them to the beginning of the data next time.
bulkMsg = incompleteRecord + bulkMsg;
// clear incomplete record so it doesn't get processed next time too.
incompleteRecord = "";
// loop though the data breaking it apart into lines by delimiter ("\n")
while (bulkMsg.Length > 0)
{
var newLinePos = bulkMsg.IndexOf("\n");
if (newLinePos > 0)
{
var line = bulkMsg.Substring(0, newLinePos);
// Do whatever you want with your line here ...
// ProcessYourLine(line)
// Move to the next message.
bulkMsg = bulkMsg.Substring(line.Length + 1);
}
else
{
// there are no more newline delimiters
// so we save the rest of the message (if any) for processing with the next batch of received data.
incompleteRecord = bulkMsg;
bulkMsg = "";
}
}
}
}

base64 encoded image (.png) to byte[] insert into xml (open xml sdk) - ' ', hexadecimal value 0x0B, is an invalid character

This is a tricky one. I have a base64 encoded image passed from a client side app, to a c# API server side.
Upon receiving Server-side, I convert the image to a byte[], whereby I try to insert it into my Powerpoint Presentation Slide making use of the Open Xml Sdk.
From my searches on SO and the net, what I could find so far is that it is a problem with XML that don't allow certain characters, in this case 0x0B (a vertical tab if I'm not mistaken).
Now, I don't really know my options at this point, but the only other thing I can think of, is to save the image to disk first, then include it in my powerpoint programitacally but this is not ideal, as I want to do as little as possible directly on the disk, I don't even save the powerpoint on the disk, I email it directly from a Stream.
I can't "sanitize" the image for illegal chars, cause that would break the image, the same way as removing even just one of the characters in the raw base64 encoded string breaks it.
Is there any other alternative? Maybe someone who has had the same issue with normal XML and a byte[] of an image that somehow got a workaround? I include my code for my method below. (The code breaks on the last line, newSlidePart.Slide.Save(); with the following message:
' ', hexadecimal value 0x0B, is an invalid character. or "'\v', hexadecimal value 0x0B, is an invalid character."
Can anyone help please?
//assign imageID
const int imageIdStartId = 1;
string imgId = "ImgID" + imageIdStartId;
//Add new image part to slide
ImagePart imgPart = newSlidePart.AddImagePart(ImagePartType.Png, imgId);
//Feed the data to the new imgPart
imgPart.FeedData(new MemoryStream(imageAsBytes.ToArray()));
//find blip
Blip blip = newSlidePart.Slide.Descendants<Blip>().First();
//IMPORTANT - property embed refers to ppt image, point this to correct image ID, i.e. one just created.
blip.Embed = imgId;
//Finally save the slide
newSlidePart.Slide.Save();
Method for creating an image from my base64 string:
private byte[] CreateImageFromBase64String(string modelSceneImageContent)
{
try
{
string[] validImage = modelSceneImageContent.Split(',');
byte[] bytes = Convert.FromBase64String(validImage[1]);
#region if we want to save to an image
//Image image;
//using (MemoryStream ms = new MemoryStream(bytes))
//{
// image = Image.FromStream(ms);
// //FileStream fs = new FileStream(HttpContext.Current.ApplicationInstance.Server.MapPath(ConfigurationManager.AppSettings["TempImagePath"]), FileMode.Create);
// File.WriteAllBytes(HttpContext.Current.ApplicationInstance.Server.MapPath(ConfigurationManager.AppSettings["TempImagePath"]), bytes);
//}
#endregion
//return raw bytes for image replac in ppt (see above for image save to bmp then returning image as alternative)
return bytes;
}
catch (Exception ex)
{
Log.Exception(ex, "Error in powerpoint template retrieval.");
throw;
}
}

How to tell if a file is text-readable in C#

Part of a list of projects I'm doing is a little text-editor.
At one point, you can load all the sub directories and files in a given directory. The program will add each as a node in a TreeView.
What I want the functionality to be is to only add the files that are readable by a normal text reader.
This code currently adds it to the tree:
TreeNode navNode = new TreeNode();
navNode.Text = file.Name;
navNode.Tag = file.FullName;
directoryNode.Nodes.Add(navNode);
I know I could easily create an if statement with something like:
if(file.extension.equals(".txt"))
but I would have to expand that statement to contain every single extension that it could possibly be.
Is there an easier way to do this? I'm thinking it may have something to do with the mime types or file encoding.

There is no general way of figuring type of information stored in the file.
Even if you know in advance that it is some sort of text if you don't know what encoding was used to create file you may not be able to load it properly.
Note that HTTP give you some hints on type of file by content-type header, but there is no such information on file system.

There are a few methods you could use to "best guess" whether or not the file is a text file. Of course, the more encodings you support, the harder this becomes, especially if plan to support CJK (Chinese, Japanese, Korean) scripts. Let's just start with Encoding.Ascii and Encoding.UTF-8 for now.
Fortunately, most non-text files (executables, images, and the like) have a lot of non-parsable characters in their first couple of kilobytes.
What you could do is take a file and scan the first 1-4KB (up to you) and see if any "non-printable" characters come up. This operation shouldn't take much time and will at least give you some certainty of the contents of the file.
public static async Task<bool> IsValidTextFileAsync(string path,
int scanLength = 4096)
{
using(var stream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read))
using(var reader = new StreamReader(stream, Encoding.UTF8))
{
var bufferLength = (int)Math.Min(scanLength, stream.Length);
var buffer = new char[bufferLength];
var bytesRead = await reader.ReadBlockAsync(buffer, 0, bufferLength);
reader.Close();
if(bytesRead != bufferLength)
throw new IOException("There was an error reading from the file.");
for(int i = 0; i < bytesRead; i++)
{
var c = buffer[i];
if(char.IsControl(c))
return false;
}
return true;
}
}

My approach based on #Rubenisme's comment and #Erik's answer.
public static bool IsValidTextFile(string path)
{
using (var stream = System.IO.File.Open(path, System.IO.FileMode.Open, System.IO.FileAccess.Read, System.IO.FileShare.Read))
using (var reader = new System.IO.StreamReader(stream, System.Text.Encoding.UTF8))
{
var bytesRead = reader.ReadToEnd();
reader.Close();
return bytesRead.All(c => // Are all the characters either a:
c == (char)10 // New line
|| c == (char)13 // Carriage Return
|| c == (char)11 // Tab
|| !char.IsControl(c) // Non-control (regular) character
);
}
}

A hacky way to do it would be to see if the file contains any of the lower control characters (0-31) that aren't forms of white space (carriage return, tab, vertical tab, line feed, and just to be safe null and end of text). If it does, then it is probably binary. If it does not, it probably isn't. I haven't done any testing or anything to see what happens when applying this rule to non ASCII encodings, so you'd have to investigate further yourself :)