What I am Trying to do
I am the author of this project which grabs the currently playing song from Spotify's local instance via their local API and sets Discord's 'Now Playing' message to reflect it.
Pretty simple stuff, but for various reasons I want to switch to C#, and that doesn't have the same level of support for Spotify.
To cut a long story shorter, Cross-Platform + Working Useful API + Spotify Playlist = Clementine. So I decided to create a similar Discord Integration but for Clementine.
But there's a problem
I can create a socket that connects to 127.0.0.1:5500.
I can send a ConnectRequest message successfully through this socket.
I can receive these message types with no problems whatsoever:
KEEP_ALIVE,
PLAY,
PAUSE,
STOP,
REPEAT,
SHUFFLE,
VOLUME_UPDATE,
TRACK_POSITION_UPDATE
But if I try to Play a song from a Stopped State, 20 exceptions are thrown and then the "PLAY" message is parsed.
I believe this is what should be the CURRENT_METAINFO message.
Similar exceptions are thrown if I try to add a new song to the playlist.
The mechanism I am using to retrieve messages is
Message.Parser.ParseDelimitedFrom(Client.GetStream());
Where:
Message = Class defined in .proto file from Repo
Parser = Protobuf built-in Object Parser
ParseDelimitedFrom = Protobuf built-in method which takes a [Length:PayloadOfLength'Length'] message from a Stream and parses the Payload as an object of the provided type (Message).
Client = System.Net.Sockets.TcpClient
GetStream = System.Net.Sockets.NetworkStream
Aside from detecting roughly 3 'Unknown' messages for every keepalive (The NetworkStream seems to send 0's every quarter of a second without data?) when Clementine is Idle, this method works just fine for Messages that are not too complex (IE: The ones without a SongMetadata class in the Response Object, and the ones without a Response Object at all).
My Suspicions
Due to Proto3 not being backwards-compatible with Proto2, I had to modify the provided .proto file slightly to remove all the 'optional' keywords, and the default values for things, and re-set all the enums to start from 0.
This may have introduced a subtle bug in the Parser which tries to read a value that doesn't exist, and moves on to the next instead of realising there's a default, or something like that.
Some of the exceptions seem to indicate that the parser is not handling the length of the data properly.
This could mean that the Parser is reading the Stream to the end, before the message is written completely, and is not waiting for the rest before stopping and trying to parse it. Then not being able to read the first part of the incomplete message, but removing it from the Stream anyway, rendering all the following chunks also illegible.
It could also mean that due to the way Clementine nests messages, the parser is detecting the size of the outer message and not catering for the optional nested object properly (or the actual Clementine message is not setting the length appropriately).
It could also be a problem with the album art being a sequence of bytes of indeterminate length.
The walls I have hit
I have completely re-written the Socket logic at least 3 times. The first two with bare Socket's and using byte[] buffers to read messages.
This seemed promising but would never actually return from the recursive function that was intended to retrieve the remainder of the bytes for a particular message, as there were always more bytes to read. It didn't occur to me until writing this to try to build the buffer from the front instead and tryParse it each iteration until it understood the message, then remove the message from the buffer.
I have attempted to implement a byte[] buffer with the TcpClient in a similar manner to the above with my current implementation. But again, trouble with the Socket writing 0's instead of remaining idle cause a problem with leading 0's being treated as an invalid tag and throwing exceptions.
I have attempted to use a BufferedStream to wrap the NetworkStream, but was unsure of exactly how to go about implementing a re-read of previously read, exception-causing data, once more data arrived.
Saucy
The actual code as a whole. Should be drop-in ready with any VSCode installation with the C# plugin and dotnet core properly installed and configured.
https://bitbucket.org/roberestarkk/discordclementineremotedotnet
A cry for Help
I would be immensely grateful for literally ANY assistance anyone can offer! I am at my wit's end with this project.
I've tried everything short of messaging the developers (which I intend to do shortly anwyays, I just need to install an IRC client), and am no-closer to getting to the bottom of things.
If I only knew WHY those exceptions were being thrown...
The solution to this problem was to roll my own message buffering to cut out the first 4 bytes (int) into a byte array, reverse it and parse it into an int. If the int is non-zero (Clementine sends an int representing 0 every [interval] when it's not sending a message), I can then grab the actual message (buffer[3..length]), and feed it into the non delimited version of protobuf's message parser.
Working code uploaded to the repo.
Related
I need to send messages from a C#.Net application to a C++ application on Windows. They'll be running on the same PC. After doing some research, it sounds like using a named pipe might work. But I'm still confused about several details. So if anyone can fill me in, I'd appreciate it.
It sounds like a named pipe is basically a type of file. If my .Net application keeps writing to the file, will it keep getting larger? Or will whatever I'm writing go away as soon as the C++ application has read it?
If I send a message with a single write() call, am I guaranteed that it will be read together, or could it be broken up? For example, if I send "hello", is it possible the timing will be such that I'll read "hel" and then "lo"?
Am I correct that if I send several messages before trying to read them, they just sit there and I can read several at once? Will it take multiple read() calls to get every message, or will they all come concatenated together?
Is there any way for the C++ application to know that a message is waiting? Or should I just have a loop going that tries to read a message, sleeps, then tries to read again?
It sounds like a named pipe is basically a type of file. If my .Net application keeps writing to the file, will it keep getting larger? Or will whatever I'm writing go away as soon as the C++ application has read it?
A pipe doesn't really have a size. There may be some number of bytes in it, and you could call that the size of the pipe. This would be cosmetic. Why do you care? If your concern is that pipes may be implemented terribly on your platform, then you should switch platforms.
If I send a message with a single write() call, am I guaranteed that it will be read together, or could it be broken up? For example, if I send "hello", is it possible the timing will be such that I'll read "hel" and then "lo"?
Pipes are streams of bytes. There is no such thing as a message on a pipe. (At least, as far as the pipe knows.)
Am I correct that if I send several messages before trying to read them, they just sit there and I can read several at once? Will it take multiple read() calls to get every message, or will they all come concatenated together?
There aren't messages. Pipes are streams of bytes. If you try to read 100 bytes, you will get 100 bytes, unless there are fewer than that number available.
Is there any way for the C++ application to know that a message is waiting? Or should I just have a loop going that tries to read a message, sleeps, then tries to read again?
You can have a thread block on reading the pipe. That thread can exist just to permit a simple way for you to query whether a message is waiting, for example, by feeding bytes read from the pipe into a thread-safe queue of some kind. It could include the application-level message protocol logic, so the queue would consist of complete application-level messages.
A pipe pretty much acts like a TCP connection as far as the read and write semantics go.
I'm writing a simple chat program using sockets. When I'm sending a long message, flush the stream and a short message afterwards, the end of the long message gets appended to the short message. It looks like this:
Send "aaasdsd"
Recieve "aaasdsd"
Send "bb"
Recieve "bbasdsd"
Through debugging I've found that the Flush method, that's supposed to clear all data from the stream, does not do that. According to mdsn, it is the expected behaviour, because NetworkStream is not bufferized. How do I clear the stream in that case? I could just follow every message with an empty (consisting of \0 chars) one of the same length, but I don't think it's correct to do that, also, it would screw up some features I need.
TCP doesn't work this way. It's as simple as that.
TCP is a stream-based protocol. That means that you shouldn't ever treat it as a message-based protocol (unlike, say, UDP). If you need to send messages over TCP, you have to add your own messaging protocol on top of TCP.
What you're trying to do here is send two separate messages, and receive two separate messages on the other side. This would work fine on UDP (which is message-based), but it will not work on TCP, because TCP is a stream with no organisation.
So yeah, Flush works just fine. It's just that no matter how many times you call Flush on one side, and how many times you call individual Sends, each Receive on the other end will get as much data as can fit in its buffer, with no respect to the Sends on the other side.
The solution you've devised (almost - just separate the strings with a single \0) is actually one of the proper ways to handle this. By doing that, you're working with messages on top of the stream again. This is called message framing - it allows you to tell individual messages apart. In your case, you've added delimiters between the messages. Think about writing the same data in a file - again, you'll need some way of your own to separate the individual messages (for example, using end lines).
Another way to handle message framing is using a length prefix - before you send the string itself, send it's length. Then, when you read on the other side, you know that between the strings, there should always be a length prefix, so the reader knows when the message ends.
Yet another way isn't probably very useful for your case - you can work with fixed-length data. So a message will always be exactly 100 bytes, for example. This is very powerful when combined with pre-defined message types - so message type 1 would contain exactly two integers, representing some coördinates, for example.
In either case, though, you'll need your own buffering on the receiving end. This is because (as you've already seen) a single receive can read multiple messages at once, and at the same time, it's not guaranteed to read the whole message in a single read. Writing your own networking is actually pretty tricky - unless you're doing this to actually learn network programming, I'd recommend using some ready technology - for example, Lindgren (a nice networking library, optimized for games but works fine for general networking as well) or WCF. For a chat system, simple HTTP (especially with the bi-directional WebSockets) might be just fine as well.
EDIT:
As Damien correctly noted, there seems to be another problem with your code - you seem to be ignoring the return value of Read. The return value tells you the amount of bytes you've actually read. Since you have a fixed-size persistent buffer on the receiving side (apparently), it means that every byte after the amount you've just read will still contain the old data. To fix this, just make sure you're only working with as much bytes as Read returned. Also, since this seems to indicate you're ignoring the Read return value altogether, make sure to properly handle the case when Read returns 0 - that means the other side has gracefully shutdown its connection - and the receiving side should do the same.
Recently I started working with sockets. I realized that when reading from a network stream, you can not know how much data is coming in. So either you know in advance how many bytes have to be recieved or you know which bytes.
Since I am currently trying to implement a C# WebSocket server I need to process HTTP requests. A HTTP request can have arbitrary length, so knowing in advance how many bytes is out of the question. But a HTTP request always has a certain format. It starts with the request-line, followed by zero or more headers, etc. So with all this information it should be simple, right?
Nope.
One approach I came up with was reading all data until a specific sequence of bytes was recognized. The StreamReader class has the ReadLine method which, I believe, works like this. For HTTP a reasonable delimiter would be the empty line separating the message head from the body.
The obvious problem here is the requirement of a (preferrably short) termination sequence, like a line break. Even the HTTP specification suggests that these two adjacent CRLFs are not a good choice, since they could also occur at the beginning of the message. And after all, two CRLFs are not a simple delimiter anyways.
So expanding the method to arbitrary type-3 grammars, I concluded the best choice for parsing the data is a finite state machine. I can feed the data to the machine byte after byte, just as I am reading it from the network stream. And as soon as the machine accepts the input I can stop reading data. Also, the FSM could immediately capture the significant tokens.
But is this really the best solution? Reading byte after byte and validating it with a custom parser seems tedious and expensive. And the FSM would be either slow or quite ugly. So...
How do you process data from a network stream when the form is known but not the size?
How can classes like the HttpListener parse the messages and be fast at it too?
Did I miss something here? How would this usually be done?
HttpListener and other such components can parse the messages because the format is deterministic. The Request is well documented. The request header is a series of CRLF-terminated lines, followed by a blank line (two CRLF in a row).
The message body can be difficult to parse, but it's deterministic in that the header tells you what encoding is used, whether it's compressed, etc. Even multi-part messages are not terribly difficult to parse.
Yes, you do need a state machine to parse HTTP messages. And yes you have to parse it byte-by-byte. It's somewhat involved, but it's very fast. Typically you read a bunch of data from the stream into a buffer and then process that buffer byte-by-byte. You don't read the stream one byte at a time because the overhead will kill performance.
You should take a look at the HttpListener source code to see how it all works. Go to http://referencesource.microsoft.com/netframework.aspx and download the .NET 4.5 Update 1 source.
Be prepared to spend a lot of time digging through that and through the HTTP spec.
By the way, it's not difficult to create a program that handles a small subset of HTTP requests. But I wonder why you'd want to do that when you can just use HttpListener and have all the details handled for you.
Update
You are talking about two different protocols. HTTP and WebSocket are two entirely different things. As the Wikipedia article says:
The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.
With HTTP, you know that the server will send the stream and then close the connection; it's a stream of bytes with a defined end. WebSocket is a message-based protocol; it enables a stream of messages. Those messages have to be delineated in some way; the sender has to tell the receiver where the end of the message is. That can be implicit or explicit. There are several different ways this is done:
The sender includes the length of message in the first few bytes of the message. For example, the first four bytes are a binary integer that says how many bytes follow in that message. So the receiver reads the first four bytes, converts that to an integer, and then reads that many bytes.
The length of the message is implicit. For example, sender and receiver agree that all messages are 80 bytes long.
The first byte of the message is a message type, and each message type has a defined length. For example, message type 1 is 40 bytes, message type 2 is 27 bytes, etc.
Messages have some terminator. In a line-oriented message system, for example, messages are terminated by CRLF. The sender sends the text and then CRLF. The receiver reads bytes until it receives CRLF.
Whatever the case, sender and receiver must agree on how messages are structured. Otherwise the case that you're worried about does crop up: the receiver is left waiting for bytes that will never be received.
In order to handle possible communications problems you set the ReceiveTimeout property on the socket, so that a Read will throw SocketException if it takes too long to receive a complete message. That way, your program won't be left waiting indefinitely for data that is not forthcoming. But this should only happen in the case of communications problems. Any reasonable message format will include a way to determine the length of a message; either you know how much data is coming, or you know when you've reached the end of a message.
If you want to send a message you can just pre-pend the size of the message to it. Get the number of bytes in the message, pre-pend a ulong to it. At the receiver, read the size of a ulong, parse it, then read that amount of bytes from the stream and then close it.
In a HTTP header you can read: Content-Length The length of the request body in octets (8-bit bytes)
I wasn't quite sure how to explain my problem in the title, but I'll try to elaborate on my problem.
Basically I'm coding a chat that is not P2P, but where all users connect to a central server, similar to IRC. The connections are asynchronous and it almost works flawlessly. The main issue is that, when a lot of data is sent to one user (or to the server from one user) at once, the bytes may merge, resulting in errors. I've approached this by adding a header of 4 bytes containing the length of the data in front of the rest of the data. Still, the bytes seem to merge. I've also tried setting NoDelay to true and DontFragment to false; still, it doesn't work.
I'm guessing the problem is that when the bytes merge, I only handle the first bytes and then do nothing with the remaining. What would be the best way to approach this issue?
Receive callback code: http://pastebin.com/f0MvjHag
That's why they call it a stream. You put bytes in at one end and TCP guarantees they come out in the same order, none missing or duplicated, at the far end. Anything bigger than a byte is your problem.
You have to accumulate enough bytes in a buffer to have your header. Then interpret it and start processing additional bytes. You may have a few left over that start the next header.
This is normal behavior. When your application is not receiving data the system will be buffering it for you. It will try to hand off the available data the next time you make a request. On the other side, a large write may travel over connections that do not support an adequate frame size. They will be split as needed and arrive eventually in dribs and drabs.
This usually happens when two or more packets of data are sent at close intervals.
I recently had this problem myself, and the way I resolved it was to a separating key. You can then tokenize each message. For example, you could add the ASCII character #4 (the End-of-Transmission character) to the end of each message being sent like I did.
Write("Message1" + ((char)4).ToString())
Write("Message2" + ((char)4).ToString())
Then, when the client receives the data, you can iterate through the received data. When it finds that special character, it knows it's the end of one message, and (maybe) the beginning of a new one.
"Message1(EOT char)Message2(EOT char)"
\n may be easier to work with than using ASCII characters.
Background:
The application I am programming uses async sockets (using BeginSend, EndSend, BeginReceive, EndReceive) to send data between each other. The sockets are TCP, no socket flags, on IPV4.
It uses the system where it sends a 4-byte (int) message, followed by a message with the length specified in the previous message. I use function helpers that handle the MessageLength, and the MessageBody. The flow is something like this
BeginReceive()
EndReceive()
MessageLengthReceived()
BeginReceive()
MessageBodyReceived()
Issue:
The issue arrives when I send file data, in chunks of 16kb (with an additional small overhead: offset, pieceIndex, etc). Occasionally, when receiving the MessageLength, it receives a data from a random part in the previous message, instead of the actual message length. Part of this issue is that it doesn't always happen at a set offset (eg beginning or end of file / piece / 16 kb chunk) and can happen with any file, but happens more if I send a lot more files / larger files.
There are internal messages that are sent (eg RequestMessages) that never experience this problem. All the internal messages are < 100 bytes.
I've tried waiting for the file chunk to save completely before requesting another chunk, but it still fails. I've also tried limiting how many chunks to send at a time, but this only resolves the issue when using 127.0.0.1 (local clients), and not cross network (LAN).
I've spent hours going through my application to see if there's any issues, but I have yet to see any where it would be sending the wrong data as a header. The issue always seems to inbetween the send and the receive of the two clients. Is there settings for socket / method of sending that I should be using? Or could it be some sort of race condition (I thought about race condition, but the fact that the data can be anywhere randomly in a file made me rethink this).
From the question, i guess the problem you are dealing with is inside the MonoTorrent library.
I myself has never encountered such problem. and by looking at the codes, i think the receive part is already ordered because the network IO will not try to receive a second message until the first one has been handled. PieceMessages' write requests are queued in DiskIO also so that should not be the problem.
however, in sending procedure, the ProcessQueue function can be called from several places. and the EnqueueSendMessage called by ProcessQueue indirectly doesn't actually enqueue the message to any queue. it just simply call the Socket.BeginSend. I don't know if Socket.BeginSend() has any queue mechanism inside. If there is not, this may bring some problem when multiple threads are trying to make the same socket "BeginSend" different data.