I have a simple client-server system sending plain text - though only commands that have been approved. The server is a Python system - and I've confirmed proper connections.
However, the client is C# - in Unity. Searching for examples, I stumbled across this bit of code. It does seem to do what I want, however, only partially:
public String readSocket()
{
if (!socketReady)
return "";
if (theStream.DataAvailable)
return theReader.ReadLine();
return "";
}
The strings I am sending end with \n, but I'm only getting half the message like this:
Message A:
claim_2
Message B:
_20_case
claim_1
I know this probably has to do with how I'm directly reading the line but I cannot find any better examples - strangely enough, everyone seems to point back at this snippet even when multiple people point out the problems.
Can anything be done to fix this bit of code properly?
In case it helps, I'm sending the information (from my Python server) out like this:
action = str(command) + "_" + str(x) + "_" + str(userid) + "_" + str(user)
cfg.GameSendConnection.sendall((action + "\n").encode("utf-8"))
When you do sockets programming, it is important to note that data might not be
available in one piece. In fact, this is exactly what you are seeing. Your
messages are being broken up.
So why does ReadLine not wait until there's a line to read?.
Here's some simple sample code:
var stream = new MemoryStream();
var reader = new StreamReader(stream);
var writer = new StreamWriter(stream) { AutoFlush = true };
writer.Write("foo");
stream.Seek(0, SeekOrigin.Begin);
Console.WriteLine(reader.ReadLine());
Note that there is no newline at the end. Still, the output of this little
snippet is foo.
ReadLine returns the string up to the first line break or until there is no
more data to read. The exception being reading from a stream that has no more
data to read, then it returns null.
When a NetworkStream has its DataAvailable property return true, it has
data. But as mentioned before, there is no guarantee whatsoever about what that
data is. It might be a single byte. Or a part of a message. Or a full message
plus part of the next message. Note that depending on the encoding, it could
even be possible to receive only part of a character. Not all character
encodings have all characters be at most a single byte. This includes UTF-8, which cfg.GameSendConnection.sendall((action + "\n").encode("utf-8")) sends.
How to solve this? Read bytes, not lines. Put them in some buffer. After every
read, check if the buffer contains a newline. If it does, you now have a full
message to handle. Remove the message up to and including the newline from the
buffer and keep appending new data to it until the next newline is received. And
so on.
This is how I process the entire line in my similar application, which is a very simple code, and your code may be different, but you can get the idea.
private string incompleteRecord = "";
public void ReadSocket()
{
if (_networkStream.DataAvailable)
{
var buffer = new byte[8192];
var receivedString = new StringBuilder();
do
{
int numberOfBytesRead = _networkStream.Read(buffer, 0, buffer.Length);
receivedString.AppendFormat("{0}", Encoding.UTF8.GetString(buffer, 0, numberOfBytesRead));
} while (_networkStream.DataAvailable);
var bulkMsg = receivedString.ToString();
// When you receive data from the socket, you can receive any number of messages at a time
// with no guarantee that the last message you receive will be complete.
// You can receive only part of a complete message, with next part coming
// with the next call. So, we need to save any partial messages and add
// them to the beginning of the data next time.
bulkMsg = incompleteRecord + bulkMsg;
// clear incomplete record so it doesn't get processed next time too.
incompleteRecord = "";
// loop though the data breaking it apart into lines by delimiter ("\n")
while (bulkMsg.Length > 0)
{
var newLinePos = bulkMsg.IndexOf("\n");
if (newLinePos > 0)
{
var line = bulkMsg.Substring(0, newLinePos);
// Do whatever you want with your line here ...
// ProcessYourLine(line)
// Move to the next message.
bulkMsg = bulkMsg.Substring(line.Length + 1);
}
else
{
// there are no more newline delimiters
// so we save the rest of the message (if any) for processing with the next batch of received data.
incompleteRecord = bulkMsg;
bulkMsg = "";
}
}
}
}
Related
I am trying to convert byte[] to base64 string format so that i can send that information to third party. My code as below:
byte[] ByteArray = System.IO.File.ReadAllBytes(path);
string base64Encoded = System.Convert.ToBase64String(ByteArray);
I am getting below error:
Exception of type 'System.OutOfMemoryException' was thrown. Can you
help me please ?
Update
I just spotted #PanagiotisKanavos' comment pointing to Is there a Base64Stream for .NET?. This does essentially the same thing as my code below attempts to achieve (i.e. allows you to process the file without having to hold the whole thing in memory in one go), but without the overhead/risk of self-rolled code / rather using a standard .Net library method for the job.
Original
The below code will create a new temporary file containing the Base64 encoded version of your input file.
This should have a lower memory footprint, since rather than doing all data at once, we handle it several bytes at a time.
To avoid holding the output in memory, I've pushed that back to a temp file, which is returned. When you later need to use that data for some other process, you'd need to stream it (i.e. so that again you're not consuming all of this data at once).
You'll also notice that I've used WriteLine instead of Write; which will introduce non base64 encoded characters (i.e. the line breaks). That's deliberate, so that if you consume the temp file with a text reader you can easily process it line by line.
However, you can amend per your needs.
void Main()
{
var inputFilePath = #"c:\temp\bigfile.zip";
var convertedDataPath = ConvertToBase64TempFile(inputFilePath);
Console.WriteLine($"Take a look in {convertedDataPath} for your converted data");
}
//inputFilePath = where your source file can be found. This is not impacted by the below code
//bufferSizeInBytesDiv3 = how many bytes to read at a time (divided by 3); the larger this value the more memory is required, but the better you'll find performance. The Div3 part is because we later multiple this by 3 / this ensures we never have to deal with remainders (i.e. since 3 bytes = 4 base64 chars)
public string ConvertToBase64TempFile(string inputFilePath, int bufferSizeInBytesDiv3 = 1024)
{
var tempFilePath = System.IO.Path.GetTempFileName();
using (var fileStream = File.Open(inputFilePath,FileMode.Open))
{
using (var reader = new BinaryReader(fileStream))
{
using (var writer = new StreamWriter(tempFilePath))
{
byte[] data;
while ((data = reader.ReadBytes(bufferSizeInBytesDiv3 * 3)).Length > 0)
{
writer.WriteLine(System.Convert.ToBase64String(data)); //NB: using WriteLine rather than Write; so when consuming this content consider removing line breaks (I've used this instead of write so you can easily stream the data in chunks later)
}
}
}
}
return tempFilePath;
}
I have little test program
public class Test
{
public string Response { get; set; }
}
My console simply call Test class
class Program
{
static void Main(string[] args)
{
Test t = new Test();
using (StreamReader reader = new StreamReader("C:\\Test.txt"))
{
t.Response = reader.ReadToEnd();
}
t.Response = t.Response.Substring(0, 5);
Console.WriteLine(t.Response);
Console.Read();
}
}
I have appox 60 MB data in my Test.txt file. When the program get executes, it is taking lot of memory because string is immutable. What is the better way handle this kind of scenario using string.
I know that i can use string builder. but i have created this program to replicate a scenario in one of my production application which uses string.
when i tried with GC.Collect(), memory is released immediately. I am not sure whether i can call GC in code.
Please help. Thanks.
UPDATE:
I think i did not explain it clearly. sorry for the confusion.
I am just reading data from file to get huge data as don't want create 60MB of data in code.
My pain point is below line of code where i have huge data in Response field.
t.Response = t.Response.Substring(0, 5);
You could limit your reads to a block of bytes (buffer). Loop through and read the next block into your buffer and write that buffer out. This will prevent a large chunk of data being stored in memory.
using (StreamReader reader = new StreamReader(#"C:\Test.txt", true))
{
char[] buffer = new char[1024];
int idx = 0;
while (reader.ReadBlock(buffer, idx, buffer.Length) > 0)
{
idx += buffer.Length;
Console.Write(buffer);
}
}
Can you read your file line by line? If so, I would recommend calling:
IEnumerable<string> lines = File.ReadLines(path)
When you iterate this collection using
foreach(string line in lines)
{
// do something with line
}
the collection will be iterated using lazy evaluation. That means the entire contents of the file won't need to be kept in memory while you do something with each line.
StreamReader provides just version of Read that you looking for - Read(Char[], Int32, Int32) - which lets you pick out first characters of the stream. Alternatively you can read char-by-char with regular StreamReader.Read till you decided that you have enough.
var textBuffer = new char[5];
reader.ReadToEnd(textBuffer, 0, 5); // TODO: check if it actually read engough
t.Response = new string(textBuffer);
Note that if you know encoding of the stream you may use lower level reading as byte array and use System.Text.Encoding classes to construct strings with encoding yourself instead of relaying on StreamReader.
Guys i'm using SslStream as a server to test my app, but i have issues reading from the stream. I'm using the following code:
while (true)
{
int read = sslStream.Read(buffer, 0, buffer.Length);
string bufferString = System.Text.Encoding.Default.GetString(buffer);
// Check for End?
if (bufferString.IndexOf("\n\r\n", System.StringComparison.Ordinal) != -1)
{
break;
}
}
The problem is that the first loop returns:
G\0\0\0\0\0
and the second run returns:
ET /whateverman
while the result should be
GET /whateverman
What is the issue and is there a better way to read from an SslStream?
Result is exactly as expected (and not directly related to SSL stream) - you are converting bytes that you did not read.
If you want to manually read strings from Stream you must respect result of Read call that tells you home many bytes actually read from the stream.
string partialString = System.Text.Encoding.Default.GetString(buffer, 0, read);
And than don't forget to concatenate strings.
Note: using some sort of reader (StreamReader/BinaryReader) may be better approach.
I'm trying to send the Base64 string of a screenshot to the server via NetworkStream and it appears I'm receiving the full string, problem is it's scrambled...
I assume this has something to do with it being fragmented and put back together? What would be the appropriate way to go about this...
Client Code
byte[] ImageBytes = Generics.Imaging.ImageToByte(Generics.Imaging.Get_ScreenShot_In_Bitmap());
string StreamData = "REMOTEDATA|***|" + Convert.ToBase64String(ImageBytes);
SW.WriteLine(StreamData);
SW.Flush();
Server Code
char[] ByteData = new char[350208];
SR.Read(ByteData, 0, 350208);
string Data = new string(ByteData);
File.WriteAllText("C:\\RecievedText", Data);
Also the size of the sent message and the char array are exactly the same.\
EDIT:
After messing around with it some more I realized the text isnt scrambled but the proper text is trailing the previous stream.. How can I ensure the stream is clear or gets the entire text
It's likely that you're not reading all of the previous response. You have to read in a loop until you get no data, like this:
char[] ByteData = new char[350208];
int totalChars = 0;
int charsRead;
while ((charsRead = SR.Read(ByteData, totalChars, ByteData.Length - totalChars) != 0)
{
totalChars += charsRead;
}
string Data = new string(ByteData, 0, totalChars);
File.WriteAllText("C:\\RecievedText", Data);
The key here is that StreamReader.Read reads up to the maximum number of characters you told it to. If there aren't that many characters immediately available, it will read what's available and return those. The return value tells you how many it read. You have to keep reading until you get the number of characters you want, or until Read returns 0.
im trying to send an image via network stream, i have a sendData and Getdata functions
and i always get an invalid parameter when using the Image.FromStream function
this is my code :
I am Getting the pic from the screen, then converting it to a byte[]
Inserting it to a Memory stream that i send via a networkStream.
private void SendData()
{
StreamWriter swWriter = new StreamWriter(this._nsClient);
// BinaryFormatter bfFormater = new BinaryFormatter();
// this method
lock (this._secLocker)
{
while (this._bShareScreen)
{
// Check if you need to send the screen
if (this._bShareScreen)
{
MemoryStream msStream = new MemoryStream();
this._imgScreenSend = new Bitmap(this._imgScreenSend.Width, this._imgScreenSend.Height);
// Send an image code
swWriter.WriteLine(General.IMAGE);
swWriter.Flush();
// Copy image from screen
this._grGraphics.CopyFromScreen(0, 0, 0, 0, this._sizScreenSize);
this._imgScreenSend.Save(msStream, System.Drawing.Imaging.ImageFormat.Jpeg);
msStream.Seek(0, SeekOrigin.Begin);
// Create the pakage
byte[] btPackage = msStream.ToArray();
// Send its langth
swWriter.WriteLine(btPackage.Length.ToString());
swWriter.Flush();
// Send the package
_nsClient.Write(btPackage, 0, btPackage.Length);
_nsClient.Flush();
}
}
}
}
private void ReciveData()
{
StreamReader srReader = new StreamReader(this._nsClient);
string strMsgCode = String.Empty;
bool bContinue = true;
//BinaryFormatter bfFormater = new BinaryFormatter();
DataContractSerializer x = new DataContractSerializer(typeof(Image));
// Lock this method
lock (this._objLocker)
{
while (bContinue)
{
// Get the next msg
strMsgCode = srReader.ReadLine();
// Check code
switch (strMsgCode)
{
case (General.IMAGE):
{
// Read bytearray
int nSize = int.Parse(srReader.ReadLine().ToString());
byte[] btImageStream = new byte[nSize];
this._nsClient.Read(btImageStream, 0, nSize);
// Get the Stream
MemoryStream msImageStream = new MemoryStream(btImageStream, 0, btImageStream.Length);
// Set seek, so we read the image from the begining of the stream
msImageStream.Position = 0;
// Build the image from the stream
this._imgScreenImg = Image.FromStream(msImageStream); // Error Here
Part of the problem is that you're using WriteLine() which adds Environment.NewLine at the end of the write. When you just call Read() on the other end, you're not dealing with that newline properly.
What you want to do is just Write() to the stream and then read it back on the other end.
The conversion to a string is strange.
What you're doing, when transferring an image, is sending an array of bytes. All you need to do is send the length of the expected stream and then the image itself, and then read the length and the byte array on the other side.
The most basic and naive way of transferring a byte array over the wire is to first send an integer that represents the length of the array, and read that length on the receiving end.
Once you now know how much data to send/receive, you then send the array as a raw array of bytes on the wire and read the length that you previously determined on the other side.
Now that you have the raw bytes and a size, you can reconstruct the array from your buffer into a valid image object (or whatever other binary format you've just sent).
Also, I'm not sure why that DataContractSerializer is there. It's raw binary data, and you're already manually serializing it to bytes anyway, so that thing isn't useful.
One of the fundamental problems of network programming using sockets and streams is defining your protocol, because the receiving end can't otherwise know what to expect or when the stream will end. That's why every common protocol out there either has a very strictly defined packet size and layout or else does something like sending length/data pairs, so that the receiving end knows what to do.
If you implement a very simple protocol such as sending an integer which represents array length and reading an integer on the receiving end, you've accomplished half the goal. Then, both sender and receiver are in agreement as to what happens next. Then, the sender sends exactly that number of bytes on the wire and the receiver reads exactly that number of bytes on the wire and considers the read to be finished. What you now have is an exact copy of the original byte array on the receiving side and you can then do with it as you please, since you know what that data was in the first place.
If you need a code example, I can provide a simple one or else there are numerous examples available on the net.
Trying to keep it short:
the Stream.Read function (which you use) returns an int that states how many bytes were read, this is return to you so you could verify that all the bytes you need are received.
something like:
int byteCount=0;
while(byteCount < nSize)
{
int read = this._nsClient.Read(btImageStream, byteCount, nSize-byteCount);
byteCount += read;
}
this is not the best code for the job