My app reads bytes from a TCP socket and needs to buffer them up, so that I can extract messages from them later. Due to the nature of TCP I may get partial or multiple messages in one read, so after each read I would like to inspect the buffer and extract as many full messages as are available.
Therefore I want a class that allows me to do the following:
append arbitrary byte[] data to it
inspect the content without consuming it, in particular checking the amount of content and also searching for the existence of a certain byte or bytes
extract and consume part of the data as a byte[], while leaving the rest in there for a future read
I expect that what I want can be done with 1 or more existing classes in the .NET library, but I'm not sure which ones. System.IO.MemoryStream looks close to what I want, but (a) it isn't clear whether it's suited to being used as a buffer (does the read data get removed from the capacity?) and (b) reads and writes seem to happen at the same place - "The current position of a stream is the position at which the next read or write operation could take place." - which is not what I want. I need to be writing to the end and reading from the front.
I suggest you use MemoryStream under the hood, but encapsulate it in another class which stores:
The MemoryStream
The current "read" position
The current "consumed" position
It will then expose:
Write: set the stream's position to the end, write data, set the stream's position back to the read position
Read: read data, set the read position to the stream's position
Consume: update the consumed position (details based on how you're trying to consume); if the consume position is above a certain threshold, copy the existing buffered data into a new MemoryStream and update all the variables. (You probably don't want to copy the buffer on every consume request.)
Note that none of this will be thread-safe without extra synchronization.
Just use a big byte-array and Array.Copy - it should do the trick.
If not, use List<byte>.
If you use the array you have to implement an index to it (where you copy additional data) yourself (same for checking the content-size), but it's straightforward.
If you are interested: here is a simple implementation of a "cyclic buffer". The test should run (I threw a couple unit test at it, but it didn't check all critical path):
public class ReadWriteBuffer
{
private readonly byte[] _buffer;
private int _startIndex, _endIndex;
public ReadWriteBuffer(int capacity)
{
_buffer = new byte[capacity];
}
public int Count
{
get
{
if (_endIndex > _startIndex)
return _endIndex - _startIndex;
if (_endIndex < _startIndex)
return (_buffer.Length - _startIndex) + _endIndex;
return 0;
}
}
public void Write(byte[] data)
{
if (Count + data.Length > _buffer.Length)
throw new Exception("buffer overflow");
if (_endIndex + data.Length >= _buffer.Length)
{
var endLen = _buffer.Length - _endIndex;
var remainingLen = data.Length - endLen;
Array.Copy(data, 0, _buffer, _endIndex, endLen);
Array.Copy(data, endLen, _buffer, 0, remainingLen);
_endIndex = remainingLen;
}
else
{
Array.Copy(data, 0, _buffer, _endIndex, data.Length);
_endIndex += data.Length;
}
}
public byte[] Read(int len, bool keepData = false)
{
if (len > Count)
throw new Exception("not enough data in buffer");
var result = new byte[len];
if (_startIndex + len < _buffer.Length)
{
Array.Copy(_buffer, _startIndex, result, 0, len);
if (!keepData)
_startIndex += len;
return result;
}
else
{
var endLen = _buffer.Length - _startIndex;
var remainingLen = len - endLen;
Array.Copy(_buffer, _startIndex, result, 0, endLen);
Array.Copy(_buffer, 0, result, endLen, remainingLen);
if (!keepData)
_startIndex = remainingLen;
return result;
}
}
public byte this[int index]
{
get
{
if (index >= Count)
throw new ArgumentOutOfRangeException();
return _buffer[(_startIndex + index) % _buffer.Length];
}
}
public IEnumerable<byte> Bytes
{
get
{
for (var i = 0; i < Count; i++)
yield return _buffer[(_startIndex + i) % _buffer.Length];
}
}
}
Please note: the code "consumes" on read - if you don't want that just remove the "_startIndex = ..." parts (or make a overload optional parameter and check or whatever).
I think BufferedStream is the solution to the problem. Also it is possible to go unread len byte of data by calling Seek.
BufferdStream buffer = new BufferedStream(tcpStream, size); // we have a buffer of size
...
...
while(...)
{
buffer.Read(...);
// do my staff
// I read too much, I want to put back len bytes
buffer.Seek(-len, SeekOrigin.End);
// I shall come back and read later
}
Growing memory
Contrary to BufferedStream where the size is specified initially, MemoryStream can grow.
Remembering streaming data
MemoryStream holds all the dara-read all the time, while BufferedStream only holds a segment of stream data.
Source stream vs byte array
MemoryStream allows to add input-bytes in Write() method which can be Read() in future. While BufferedSteam takes input-bytes from another source-stream specified in the constructor.
Here's another implementation of a buffer I wrote a while ago:
Resizeable: allowing to queue up data and not throw buffer overflow exception;
Efficient: uses a single buffer and Buffer.Copy operations to enqueue/dequeue data
Coming to this late, but for posterity:
When I've done this in the past I've take a slight different approach. If your messages have a fixed header size (that tells you how many bytes in the body), and bearing in mind the network stream is already buffering, I perform the operation in two phases:
a read on the stream for the bytes for the header
a subsequent read on the stream for the bytes for the body, based on the header
repeat
This leverages the fact that - for a stream - when you ask for 'n' bytes you'll never get more back, so you can ignore many of the 'opps I read too many, let me put these aside till next time' issues.
Now this isn't the whole story, to be fair. I had an underlying wrapper class over the stream to handle fragmentation issues (ie if asked for 4 bytes, don't return until 4 bytes received, or stream closed). But that bit is fairly easy.
In my mind the key is to decouple the message handling with the stream mechanics, and if you stop attempting to consume the message as a single ReadBytes() from a stream, life becomes much simpler.
[all of this is true whether your reads are blocking, or async (APM/await)]
It sounds like you want to read from the socket into a MemoryStream buffer, and then 'pop' data out of the buffer and reset it each time a certain byte is encountered. It would look something like this:
void ReceiveAllMessages(Action<byte[]> messageReceived, Socket socket)
{
var currentMessage = new MemoryStream();
var buffer = new byte[128];
while (true)
{
var read = socket.Receive(buffer, 0, buffer.Length);
if (read == 0)
break; // Connection closed
for (var i = 0; i < read; i++)
{
var currentByte = buffer[i];
if (currentByte == END_OF_MESSAGE)
{
var message = currentMessage.ToByteArray();
messageReceived(message);
currentMessage = new MemoryStream();
}
else
{
currentMessage.Write(currentByte);
}
}
}
}
You could do this with a Stream wrapping a ConcurrentQueue<ArraySegment<byte>> (keep in mind this makes it forward only). However I really dislike the idea of keeping data in memory before doing something with it; it opens you up to a bunch of attacks (intentional or not) regarding the size of the message. You might also want to Google "circular buffer".
You should actually be writing code that does something meaningful with the data as soon as it is received: 'Push Parsing' (this is what, for example, SAX supports). As an example of how you would do this with text:
private Encoding _encoding;
private Decoder _decoder;
private char[] _charData = new char[4];
public PushTextReader(Encoding encoding)
{
_encoding = encoding;
_decoder = _encoding.GetDecoder();
}
// A single connection requires its own decoder
// and charData. That connection should never
// call this method from multiple threads
// simultaneously.
// If you are using the ReadAsyncLoop you
// don't need to worry about it.
public void ReceiveData(ArraySegment<byte> data)
{
// The two false parameters cause the decoder
// to accept 'partial' characters.
var charCount = _decoder.GetCharCount(data.Array, data.Offset, data.Count, false);
charCount = _decoder.GetChars(data.Array, data.Offset, data.Count, _charData, 0, false);
OnCharacterData(new ArraySegment<char>(_charData, 0, charCount));
}
If you must be able to accept complete messages before deserializing them, you can use a MemoryMappedFile, which has the advantage that the sending entity won't be able to out-of-memory your server. What gets tricky is resetting the file back to zero; because that can bunch of issues. One way to tackle this is by:
TCP Receiver End
Write to the current stream.
If the stream exceeds a certain length move to a new one.
Deserialization End
Read from the current stream.
Once you have emptied the stream destroy it.
The TCP receiver end is very simple. The deserializer end will need some elementary buffer stitching logic (remember to use Buffer.BlockCopy and not Array.Copy).
Side note: Sounds like a fun project, if I have the time and remember I might go ahead and implement this system.
There are only three answers here which provide code.
One of them is clumsy and the others do not answer the question.
Here a class that you can just copy and paste:
/// <summary>
/// This class is a very fast and threadsafe FIFO buffer
/// </summary>
public class FastFifo
{
private List<Byte> mi_FifoData = new List<Byte>();
/// <summary>
/// Get the count of bytes in the Fifo buffer
/// </summary>
public int Count
{
get
{
lock (mi_FifoData)
{
return mi_FifoData.Count;
}
}
}
/// <summary>
/// Clears the Fifo buffer
/// </summary>
public void Clear()
{
lock (mi_FifoData)
{
mi_FifoData.Clear();
}
}
/// <summary>
/// Append data to the end of the fifo
/// </summary>
public void Push(Byte[] u8_Data)
{
lock (mi_FifoData)
{
// Internally the .NET framework uses Array.Copy() which is extremely fast
mi_FifoData.AddRange(u8_Data);
}
}
/// <summary>
/// Get data from the beginning of the fifo.
/// returns null if s32_Count bytes are not yet available.
/// </summary>
public Byte[] Pop(int s32_Count)
{
lock (mi_FifoData)
{
if (mi_FifoData.Count < s32_Count)
return null;
// Internally the .NET framework uses Array.Copy() which is extremely fast
Byte[] u8_PopData = new Byte[s32_Count];
mi_FifoData.CopyTo(0, u8_PopData, 0, s32_Count);
mi_FifoData.RemoveRange(0, s32_Count);
return u8_PopData;
}
}
/// <summary>
/// Gets a byte without removing it from the Fifo buffer
/// returns -1 if the index is invalid
/// </summary>
public int PeekAt(int s32_Index)
{
lock (mi_FifoData)
{
if (s32_Index < 0 || s32_Index >= mi_FifoData.Count)
return -1;
return mi_FifoData[s32_Index];
}
}
}
Related
I'm a novice programmer. I'm creating a library to process binary files of a certain type -- like a codec (though without a need to process a progressive stream coming over a wire). I'm looking for an efficient way to read the file into memory and then parse portions of the data as needed. In particular, I'd like to avoid large memory copies, hopefully without a lot of added complexity to avoid that.
In some situations, I want to do sequential reading of values in the data. For this, a MemoryStream works well.
FileStream fs = new FileStream(_fileName, FileMode.Open, FileAccess.Read);
byte[] bytes = new byte[fs.Length];
fs.Read(bytes, 0, bytes.Length);
_ms = new MemoryStream(bytes, 0, bytes.Length, false, true);
fs.Close();
(That involved a copy from the bytes array into the memory stream; that's one time, and I don't know of a way to avoid it.)
With the memory stream, it's easy to seek to arbitrary positions and then start reading structure members. E.g.,
_ms.Seek(_tableRecord.Offset, SeekOrigin.Begin);
byte[] ab32 = new byte[4];
_version = ConvertToUint(_ms.Read(ab32));
_numRecords = ConvertToUint(_ms.Read(ab32));
// etc.
But there may also be times when I want to take a slice out of the memory corresponding to some large structure and then pass into a method for certain processing. MemoryStream doesn't support that. I could always pass the MemoryStream plus offset and length, though that might not always be the most convenient.
Instead of MemoryStream, I could store the data in memory using Memory. That supports slicing, but not sequential reading.
If for some situation I want to get a slice (rather than pass stream & offset/length), I could construct an ArraySegment from MemoryStream.GetBuffer.
ArraySegment<byte> as = new ArraySegment<byte>(ms.GetBuffer(), offset, length);
It's not clear to me, though, if that will result in a (potentially large) copy, or if that uses a reference into the same memory held by the MemoryStream. I gather that GetBuffer exposes the underlying memory rather than providing a copy; and that ArraySegment will point into the same memory?
There will be times when I need to get a slice that is a copy as I'll need to modify some elements and then process that, but without changing the original. If ArraySegment gets a reference rather than a copy, I gather I could use ArraySegment<byte>.ToArray()?
So, my questions are:
Is MemoryStream the best approach? Is there any other type that allows sequential reading like MemoryStream but also allows slicing like Memory?
If I want a slice without copying memory, will ArraySegment<byte>(ms.GetBuffer(), offset, length) do that?
Then if I need a copy that can be modified without affecting the original, use ArraySegment<byte>.ToArray()?
Is there a way to read the data from a file directly into a new MemoryStream without creating a temporary byte array that gets copied?
Am I approaching all this the best way?
To get the initial MemoryStream from reading the file, the following works:
byte[] bytes;
try
{
// File.ReadAllBytes opens a filestream and then ensures it is closed
bytes = File.ReadAllBytes(_fi.FullName);
_ms = new MemoryStream(bytes, 0, bytes.Length, false, true);
}
catch (IOException e)
{
throw e;
}
File.ReadAllBytes() copies the file content into memory. It uses using, which means that it ensures the file gets closed. So no Finally statement is needed.
I can read individual values from the MemoryStream using MemoryStream.Read. These calls involve copies of those values, which is fine.
In one situation, I needed to read a table out of the file, change a value, and then calculate a checksum of the entire file with that change in place. Instead of copying the entire file so that I could edit one part, I was able to calculate the checksum in progressive steps: first on the initial, unchanged segment of the file, then continue with the middle segment that was changed, then continue with the remainder.
For this I could process the first and final segments using the MemoryStream. This involved lots of reads, with each read copying; but those copies were transient variables, so no significant working set increase.
For the middle segment, that needed to be copied since it had to be changed (but the original version needed to be kept intact). The following worked:
// get ref (not copy!) to the byte array underlying the MemoryStream
byte[] fileData = _ms.GetBuffer();
// determine the required length
int length = _tableRecord.Length;
// create array to hold the copy
byte[] segmentCopy = new byte[length];
// get the copy
Array.ConstrainedCopy(fileData, _tableRecord.Offset, segmentCopy, 0, length);
After modifying values in segmentCopy, I then needed to pass this to my static method for calculating checksums, which expected a MemoryStream (for sequential reading). This worked:
// new MemoryStream will hold a ref to the segmentCopy array (no new copy!)
MemoryStream ms = new MemoryStream(segmentCopy, 0, segmentCopy.Length);
What I haven't needed to do yet, but will want to do, is to get a slice of the MemoryStream that doesn't involve copying. This works:
MemoryStream sliceFromMS = new MemoryStream(fileData, offset, length);
From above, fileData was a ref to the array underlying the original MemoryStream. Now sliceFromMS will have a ref to a segment within that same array.
You can use FileStream.Seek, as I understand it, there is no need to load data into memory, then to use this method of MemoryStream
In the following example, str1 and str2 are equal:
using (var fs = new FileStream(#"C:\Users\bar_v\OneDrive\Desktop\js_balancer.txt", FileMode.Open))
{
var buffer = new byte[20];
fs.Read(buffer, 0, 20);
var str1= Encoding.ASCII.GetString(buffer);
fs.Seek(0, SeekOrigin.Begin);
fs.Read(buffer, 0, 20);
var str2 = Encoding.ASCII.GetString(buffer);
}
By the way, when you create a new MemoryStream object, you don’t copy the byte array, you just keep a reference to it:
public MemoryStream(byte[] buffer, bool writable)
{
if (buffer == null)
throw new ArgumentNullException(nameof(buffer), SR.ArgumentNull_Buffer);
_buffer = buffer;
_length = _capacity = buffer.Length;
_writable = writable;
_exposable = false;
_origin = 0;
_isOpen = true;
}
But when reading, as we can see, copying occurs:
public override int Read(byte[] buffer, int offset, int count)
{
if (buffer == null)
throw new ArgumentNullException(nameof(buffer), SR.ArgumentNull_Buffer);
if (offset < 0)
throw new ArgumentOutOfRangeException(nameof(offset), SR.ArgumentOutOfRange_NeedNonNegNum);
if (count < 0)
throw new ArgumentOutOfRangeException(nameof(count), SR.ArgumentOutOfRange_NeedNonNegNum);
if (buffer.Length - offset < count)
throw new ArgumentException(SR.Argument_InvalidOffLen);
EnsureNotClosed();
int n = _length - _position;
if (n > count)
n = count;
if (n <= 0)
return 0;
Debug.Assert(_position + n >= 0, "_position + n >= 0"); // len is less than 2^31 -1.
if (n <= 8)
{
int byteCount = n;
while (--byteCount >= 0)
buffer[offset + byteCount] = _buffer[_position + byteCount];
}
else
Buffer.BlockCopy(_buffer, _position, buffer, offset, n);
_position += n;
return n;
}
I have a problem with my school project, i use Protobuf library but i have the following error:
Google.Protobuf.InvalidProtocolBufferException" Protocol message contained an invalid tag (zero).
My protocol message wrapper is:
syntax = "proto3";
package CardGameGUI.Network.Protocol.Message;
message WrapperMessage {
enum MessageType {
HELLO_MESSAGE = 0;
JOIN_ROOM_MESSAGE = 1;
JOIN_ROOM_RESPONSE_MESSAGE = 2;
}
MessageType type = 1;
bytes payload = 2;
}
I use this to send a message:
public void SendObject<T>(Protocol.Message.WrapperMessage.Types.MessageType type, T messageObject)
{
byte[] message;
// Serialize message
using (var stream = new MemoryStream())
{
((IMessage)messageObject).WriteTo(stream);
message = stream.GetBuffer();
}
byte[] wrapper = new Protocol.Message.WrapperMessage{Type = type, Payload = Google.Protobuf.ByteString.CopyFrom(message)}.ToByteArray();
Connection.SendObject<byte[]>("ByteMessage", wrapper);
}
And my server handler:
private void IncommingMessageHandler(PacketHeader header, Connection connection, byte[] message)
{
Protocol.Message.WrapperMessage wrapper = Protocol.Message.WrapperMessage.Parser.ParseFrom(message);
switch (wrapper.Type)
{
case Protocol.Message.WrapperMessage.Types.MessageType.HelloMessage:
GetClient(connection.ConnectionInfo.NetworkIdentifier).MessageHandler(Protocol.Message.HelloMessage.Parser.ParseFrom(wrapper.Payload.ToByteArray()));
break;
}
}
The wrapper message is perfectly unserialized, and type is correctly matched, but at the treatment of my Payload, the exception pops.
Do i do something bad?
Edit: a small screen of the message Payload
The problem is probably that you used GetBuffer without making use of the known length. GetBuffer returns the oversized backing array. The data after the stream's .Length is garbage and should not be consumed - it will typically (but not always) be zeros, which is what you are seeing.
Either use ToArray() instead of GetBuffer(), or track the .Length of the stream and only consume that much of the oversized buffer.
Another possibility is "framing" - it looks like you're handling packets, but if this is TCP there is no guarantee that the chunks you receive are the same sizes as the chunks you send. If you are sending multiple messages over TCP you need to implement your own framing (typically via a length prefix, since you're talking binary data).
Incidentally, this isn't protobuf-net.
If neither of those is the problem: check the data you receive is exactly (byte for byte) the data you send (including lengths). It is easy for data to get corrupted or mis-chunked by IO code.
i encounter this problem in this case
because my serialize bytestream loss the varint lenth
such as if i serialize a "Person.proto" message which have 672 bytes
if i deserialize the 672 bytes will encounter the error
the solve strategy is that add the varint len in the 672bytes so youcan get a 674bytes stream
the extra amount data is the "varint" code for 672, which is 160,5
you can get the varint bytes by the function
public static byte[] VarInt(int value)
{
//data len
List<byte> varIntBuffer = new List<byte>();
int index = 0;
while (true)
{
if ((value & ~0x7f) == 0)
{
varIntBuffer.Add((byte)(value & 0x7f));
break;
}
else
{
varIntBuffer.Add((byte)((value & 0x7f) | 0x80));
value = value >> 7;
}
index++;
}
return varIntBuffer.ToArray();
}
I had this same issue when attempting to deserialize a byte array which had been initialized to a fixed size but there was a bug which meant I was not populating the array with proto bytes (so the byte array was populated with zeros when I was attempting to deserialize).
It turns out that I was reading bytes from a JMS BytesMessage twice in a test case but was not calling BytesMessage.reset() before the second read.
I'm guessing you could get a similar bug if attempting to read from an InputStream twice without calling reset()
As above really, I'm trying to create a tcp linux server in c to accept data, perform some processing then send it back.
The code I'm trying to use on the client side to send the data and then read it back:
TcpClient tcpclnt = new TcpClient();
tcpclnt.Connect("192.168.0.14", 1235);
NetworkStream stm = tcpclnt.GetStream();
_signal.WaitOne();
Image<Bgr, Byte> frame = null;
while (_queue.TryDequeue(out frame))
{
if (frame != null)
{
resizedBMPFrame = frame.Resize(0.5, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR).ToBitmap();
using (MemoryStream ms = new MemoryStream())
{
resizedBMPFrame.Save(ms, ImageFormat.Bmp);
byte[] byteFrame = ms.ToArray();
l = byteFrame.Length;
byte[] buf = Encoding.UTF8.GetBytes(l.ToString());
stm.Write(buf, 0, buf.Length);
stm.Write(byteFrame, 0, byteFrame.Length);
}
}
else
{
Reading = false;
}
int i;
Bitmap receivedBMPFrame;
byte[] receivedFramesize = new byte[4];
int j = stm.Read(receivedFramesize, 0, receivedFramesize.Length);
int receivedFramesizeint = BitConverter.ToInt32(receivedFramesize, 0);
byte[] receivedFrame = new byte[receivedFramesizeint];
j = stm.Read(receivedFrame, 0, receivedFrame.Length);
using (MemoryStream ms = new MemoryStream(receivedFrame))
{
receivedBMPFrame = new Bitmap(ms);
if (receivedBMPFrame != null)
{
outputVideoPlayer.Image = receivedBMPFrame;
}
else
{
Reading = false;
}
}
}
}
stm.Close();
tcpclnt.Close();
So the idea is it waits for the display thread to send the current frame it's displaying using a concurrentqueue, it then takes it, and makes it a quarter of the size, converts it to a byte array and then sends its length and then it itself over the tcp socket.
In theory the server gets it, performs some processing then sends it back, so it reads the length of it then the new frame itself.
The server code is below:
while (1)
{
int incomingframesize;
int n;
n = read(conn_desc, framesizebuff, 6);
if ( n > 0)
{
printf("Length of incoming frame is %s\n", framesizebuff);
}
else
{
printf("Failed receiving length\n");
return -1;
}
char framebuff[atoi(framesizebuff)];
n = read(conn_desc, framebuff, atoi(framesizebuff));
if ( n > 0)
{
printf("Received frame\n");
}
else
{
printf("Failed receiving frame\n");
return -1;
}
printf("Ready to write\n");
int k = sizeof(framebuff);
n = write(conn_desc, &k, sizeof(int));
if (n <0)
{
printf("ERROR writing to socket\n");
}
else
{
printf("Return frame size is %d\n", k);
}
n = write(conn_desc, &framebuff, sizeof(char)*k);
if (n <0)
{
printf("ERROR writing to socket\n");
}
frameno++;
printf("Frames sent: %d\n", frameno);
}
So it reads the length, then the actual frame, which seems to work, and at the moment then just sends it straight back without doing any processing. However it only works for one loop seemingly, if I step through the client code line by line the server code runs through once, but on the 2nd read by the client, receiving the frame from the server, the server then runs the two reads of the loop straight away, without waiting for another write. Failing on the 2nd having seemingly read in nothing as it outputs:
Length of incoming frame is
Failed receiving frame
With no number, which to me makes sense as I haven't sent another write with the length of the next frame. I'm just wondering what I'm missing/why it's acting like this? As on the first loop it waits until the write commands from the client. I'm wondering if it means there is left over data in the write stream, so when it goes back to the top it immediately reads it again? Although it then doesn't print any form of number which to me implies there's nothing there...
Any help would be greatly appreciated.
EDIT/UPDATE:
Changed the read/write sections on the server to do a single byte at a time like this:
while (ntotal != incomingframesize)
{
n = read(conn_desc, &framebuff[ntotal], sizeof(char));
ntotal = ntotal + n;
while (i < k)
{
m = write(conn_desc, &framebuff[i], sizeof(char));
Which seems to have solved the problems I was having and now the correct data is being transferred :)
When the client writes the frame size it uses the length of some object, but when the server reads it always tries to read 6 characters. You need to use a fixed length for the frame size!
When reading, you cannot assume that you get as many bytes as you ask for. The return value, if >0, is the number of bytes actually read. If you get less than you asked for, you need to keep reading until you have received the number of bytes you expect.
First read until you've got 6 bytes (frame size).
Next read until you've got the number of bytes indicated by the frame size.
Make sure you use the same number of bytes for the frame size in all places.
Edit:
I also noted a bug in the call to write in the server:
n = write(conn_desc, &framebuff, sizeof(char)*k);
framebuff is a pointer to the data, so you probably mean:
n = write(conn_desc, &framebuff[0], k);
I'm trying to send data back and forth between only two computers using a Socket. The data is in the form of serialized Packet objects.
When testing the program on another computer on my local network, I'm getting random SerializationExceptions so that no data goes through.
The program consistently sends different data, so when it makes another pass at sending it again, it will sometimes go through and sometimes hit the same SerializationException again. If I catch the exception and leave it running, all data eventually makes it through, but it takes several tries.
The exception says: "The input stream is not a valid binary format. The starting contents (in bytes) are [byte data]"
Not sure exactly where my problem lies. The larger amounts of data I send (~100kb max) always go through. The smaller ones (50-70 bytes) have trouble. Here's everything to do with my serialization and reading/writing data.
Socket defined as such:
SocketMain = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
Send & Read methods. I'm aware this is probably a horrible way to do so and might end up being my issue. Suggestions?:
public void SendPacket(Packet P)
{
using (MemoryStream MS = new MemoryStream())
{
BinaryFormatter BF = new BinaryFormatter();
BF.Serialize(MS, P);
SocketMain.Send(MS.ToArray());
}
}
public void ReadPacket()
{
byte[] BufferArray = new byte[131072];
int BytesReceived = SocketMain.Receive(BufferArray);
byte[] ActualData = new byte[BytesReceived];
Buffer.BlockCopy(BufferArray, 0, ActualData, 0, BytesReceived);
using (MemoryStream MS = new MemoryStream(ActualData))
{
BinaryFormatter BF = new BinaryFormatter();
HandlePacket((Packet)BF.Deserialize(MS));
}
}
Example Packet object. This is one of my smaller ones. I think this might be the one that is causing the issue, but I don't know how I could tell.
[Serializable()]
public class Packet4BlockVerify : Packet, ISerializable
{
public byte Index;
public string MD5Hash;
public Packet4BlockVerify(int Index, string MD5Hash): base(4)
{
this.Index = (byte)Index;
this.MD5Hash = MD5Hash;
}
protected Packet4BlockVerify(SerializationInfo info, StreamingContext context)
{
this.ID = info.GetByte("ID");
this.Index = info.GetByte("Index");
this.MD5Hash = info.GetString("MD5Hash");
}
public override void GetObjectData(SerializationInfo info, StreamingContext context)
{
info.AddValue("ID", this.ID);
info.AddValue("Index", this.Index);
info.AddValue("MD5Hash", this.MD5Hash);
}
}
Does anyone see anything wrong?
You are not reading all the bytes you sent. Your receive call:
int BytesReceived = SocketMain.Receive(BufferArray);
returns any number of bytes. You will need to pre-pend the bytes you send with the size of the remaining bytes, read that then continue reading till you have all your bytes before trying to deserialize.
TCP sends a continuous byte stream so your receive call reads arbitrary sized chunks. One of the overloads you can specify the number of bytes you want to receive so after reading the number bytes you are expecting you could use that. e.g.
// Warning untested! (but you get the idea)
// when sending
var payload = MS.ToArray();
var payloadSize = payload.Length;
mySocket.Send(BitConverter.GetBytes(payloadSize));
mySocket.Send(payload);
// when recieving
mySocket.Recieve(myBuffer, sizeof(int), SocketFlags.None);
var payloadSize = BitConverter.ToInt32(myBuffer, 0);
mySocket.Recieve(myBuffer, payloadSize, SocketFlags.None);
// now myBuffer from index 0 - payloadSize contains the payload you sent
im trying to send an image via network stream, i have a sendData and Getdata functions
and i always get an invalid parameter when using the Image.FromStream function
this is my code :
I am Getting the pic from the screen, then converting it to a byte[]
Inserting it to a Memory stream that i send via a networkStream.
private void SendData()
{
StreamWriter swWriter = new StreamWriter(this._nsClient);
// BinaryFormatter bfFormater = new BinaryFormatter();
// this method
lock (this._secLocker)
{
while (this._bShareScreen)
{
// Check if you need to send the screen
if (this._bShareScreen)
{
MemoryStream msStream = new MemoryStream();
this._imgScreenSend = new Bitmap(this._imgScreenSend.Width, this._imgScreenSend.Height);
// Send an image code
swWriter.WriteLine(General.IMAGE);
swWriter.Flush();
// Copy image from screen
this._grGraphics.CopyFromScreen(0, 0, 0, 0, this._sizScreenSize);
this._imgScreenSend.Save(msStream, System.Drawing.Imaging.ImageFormat.Jpeg);
msStream.Seek(0, SeekOrigin.Begin);
// Create the pakage
byte[] btPackage = msStream.ToArray();
// Send its langth
swWriter.WriteLine(btPackage.Length.ToString());
swWriter.Flush();
// Send the package
_nsClient.Write(btPackage, 0, btPackage.Length);
_nsClient.Flush();
}
}
}
}
private void ReciveData()
{
StreamReader srReader = new StreamReader(this._nsClient);
string strMsgCode = String.Empty;
bool bContinue = true;
//BinaryFormatter bfFormater = new BinaryFormatter();
DataContractSerializer x = new DataContractSerializer(typeof(Image));
// Lock this method
lock (this._objLocker)
{
while (bContinue)
{
// Get the next msg
strMsgCode = srReader.ReadLine();
// Check code
switch (strMsgCode)
{
case (General.IMAGE):
{
// Read bytearray
int nSize = int.Parse(srReader.ReadLine().ToString());
byte[] btImageStream = new byte[nSize];
this._nsClient.Read(btImageStream, 0, nSize);
// Get the Stream
MemoryStream msImageStream = new MemoryStream(btImageStream, 0, btImageStream.Length);
// Set seek, so we read the image from the begining of the stream
msImageStream.Position = 0;
// Build the image from the stream
this._imgScreenImg = Image.FromStream(msImageStream); // Error Here
Part of the problem is that you're using WriteLine() which adds Environment.NewLine at the end of the write. When you just call Read() on the other end, you're not dealing with that newline properly.
What you want to do is just Write() to the stream and then read it back on the other end.
The conversion to a string is strange.
What you're doing, when transferring an image, is sending an array of bytes. All you need to do is send the length of the expected stream and then the image itself, and then read the length and the byte array on the other side.
The most basic and naive way of transferring a byte array over the wire is to first send an integer that represents the length of the array, and read that length on the receiving end.
Once you now know how much data to send/receive, you then send the array as a raw array of bytes on the wire and read the length that you previously determined on the other side.
Now that you have the raw bytes and a size, you can reconstruct the array from your buffer into a valid image object (or whatever other binary format you've just sent).
Also, I'm not sure why that DataContractSerializer is there. It's raw binary data, and you're already manually serializing it to bytes anyway, so that thing isn't useful.
One of the fundamental problems of network programming using sockets and streams is defining your protocol, because the receiving end can't otherwise know what to expect or when the stream will end. That's why every common protocol out there either has a very strictly defined packet size and layout or else does something like sending length/data pairs, so that the receiving end knows what to do.
If you implement a very simple protocol such as sending an integer which represents array length and reading an integer on the receiving end, you've accomplished half the goal. Then, both sender and receiver are in agreement as to what happens next. Then, the sender sends exactly that number of bytes on the wire and the receiver reads exactly that number of bytes on the wire and considers the read to be finished. What you now have is an exact copy of the original byte array on the receiving side and you can then do with it as you please, since you know what that data was in the first place.
If you need a code example, I can provide a simple one or else there are numerous examples available on the net.
Trying to keep it short:
the Stream.Read function (which you use) returns an int that states how many bytes were read, this is return to you so you could verify that all the bytes you need are received.
something like:
int byteCount=0;
while(byteCount < nSize)
{
int read = this._nsClient.Read(btImageStream, byteCount, nSize-byteCount);
byteCount += read;
}
this is not the best code for the job