I need to write a program that will communicate with other .NET programs ... but also a legacy VFP program over TCP. I need to choose a fairly simple TCP message format that the VFP programmer can use. It could be as simple as a sequence of small XML blobs delimited by... I dunno, a null character? Whatever.
I need to choose between TcpListener/TcpClient and WCF. I started researching WCF but its architecture seems opaque and built-in Visual Studio templates are heavily biased toward making "web services" that act like a sort of RPC mechanism, but require a special "host" or web server that is external to the application. And Microsoft's 6-stage tutorial makes WCF sound pretty cumbersome (involving code generators, command-line and XML crap just to remotely subtract or multiply two numbers).
I want a self-contained app (no "host"), I want control of the wire protocol, and I want to understand how it works. WCF doesn't seem to facilitate these things, so I abandoned it in factor of TcpListener/TcpClient.
However, the program is to serve as an intermediary between a single (VFP) server and many (.NET) clients, and there will be communication in both directions and across different connections. Using TcpListener and TcpClient, the work of juggling the connections and threads is getting a bit messy, I have no experience with IAsyncResult, and I'm not just not confident in my code quality.
So I would like to solicit opinions again: should I still consider WCF instead? If yes, can you point me toward answers to the following questions?
Where in the web is a good explanation of WCF's architecture? Or do I need a book?
How is bi-directional communication done in WCF, where either side (of a single TCP connection) can send a message at any time?
How can I get past all the web-services and RPC mumbo-jumbo, and control the wire protocol?
In WCF, how do I shut down the app cleanly, closing all connections in parallel without hacky Thread.Abort() commands?
If no, how can I set up my code (that uses TcpListener/TcpClient/NetworkStream) so that I can read a message from a NetworkStream, but also accept requests from other connections, shut down cleanly at any time, and avoid wasting CPU time to poll queues and NetworkStreams that are inactive?
The short answer: go with WCF. While there's a good amount of tooling and code-generation and other bells and whistles around it, there's nothing that is preventing you from setting up everything in code (you can define your contracts, set the endpoints up, etc. all in code).
For your specific questions:
WCF Architecture - This is pretty basic, and it should get you up and running relatively quickly.
What you are looking for is duplex services. The NetTcpBinding allows for duplex services out-of-the-box (although you can do it with HTTP, you need a specific binding).
If you want to control the wire format, you will want to create a custom encoder. However, I have to strongly recommend against it. You want to create an XML file with null character to delineate separate messages? There's no need for that, the nature of XML is that you can create child elements to perform the appropriate grouping; there's no limit to how many elements you can nest. There's really no need for this.
Simply shutting down the ServiceHost by calling Close, this will allow all outstanding requests to complete, and then shut down gracefully. If you really want to tear down without concern, then call Abort.
In the end, I'd strongly recommend that you not use the NetTcpBinding; VFP will have a difficult time consuming the protocol. However, if you use an HTTP-based protocol, there are always tools that VFP can easily use to make the call and consume the contents (assuming you stick with XML).
Just to tack on about a common on using DCOM, VFP can utilize DCOM, but needs to be done with CreateObjectEx()... the only big difference is you need to know the GUID of the class instance you are connecting to on whatever server it is connecting to, AND the machine name its going to connect to.
Then the remote object does its work via exposed functions, but VFP calling it from some other machine on the network treats it as if the function was being performed locally and gets whatever the return values are.
I've done DCOM with VFP even as far back as 10 yrs ago for an insurance company...
Related
I am attempting to send player information from my Game to my network client to then be sent off to the server.
Currently the ClientNetwork -> ClientGame relationship is held with XML files. They read/write back and forth at very high speeds. If you use just one XML file for this trade, one will "hog" the file at times, making a kind of lag when one cannot read because the other is viciously writing and rewriting.
To fix this I have 2 of each of my XML files. If it cannot read one, it will read the other. In theory they should be using both of them, since it'd be a tradeoff from one to another. Not working up to par.
But my main problem is just the usage in general of XML is very sloppy, dozens of try-catch statements to make sure they're all happy (and my personal favorite, try catches within try catches -- WE HAVE TO GO DEEPER)
I am just curious of if there is a better way to be doing this. I need a static point of variables that can be accessed by both client side programs. I'm afraid someone is going to say databases...
I'd like to state for anyone who is looking into this as well and stumbled across this page that Shared Memory is awesome. Though I have to convert all strings to characters and then to bytes and read them one by one, in the whole it's ALOT better than dealing with things that cannot read/write the same file at the same time. If you wish to further understand it rather than just use it, go to this link, it explains a lot of the messaging varieties and how to use them.
Yes there is!
The term you are looking for is interprocess communication - communication between two processes on the same machine.
There are various methods which allow two processes on the same machine to communicate with each other, including:
Named pipes
Shared memory
Sockets
HTTP
Fortunately C# applications can simply use the WCF framework to perform IPC (interprocess communication) using one of the above, and let the WCF framework take care of the difficult bits! Here are a couple of guides to get you started (there are many more):
WCF Tutorial - Basic Interprocess Communication
Many to One Local IPC using WCF and NetNamedPipeBinding
Also, one of the neat things about WCF is that you can also use it to communicate between different machines simply by changing the "Transport" (i.e. the communication method) to one which works over a network, (e.g. HTTP).
If you are targetting .Net 2.0 then you should look into either .Net remoting or web services instead.
A simple TCP stream jumps out at me. Have the network client open a listening TCP socket, and have the game connect to the network client. You could continue to send the same XML data you're already writing, if you like.
I agree with the tcp/ip socket answer proposed by David. I would simply submit the data to a socket on the local pc and have the other application listen to the socket. You can transmit data easily and quickly using this method and it will work no matter what version of the .net framework you are targeting.
I suppose similar questions were already asked, but I was unable to find any. Please feel free to point me to an existing solutions.
I'll explain my scenario. I'd like to create a server application. There are many clients (currently only a few dozens, but it should scale up to 1000+) that connect to the server (which is running on a single machine).
Each client periodically sends a small amount of data to the server to process (processing is quick). The server can also send small amounts of data to each client on a regular basis. The response time should be low (<100 ms), but realtime or anything like that is not required.
My first idea was back from when I was still programming in VB6: Create a server socket to listen to incoming requests, then create a client socket for each possible client (singlethreaded). I doubt this scales well. It is also difficult to implement the communication.
So I figured I'd create a listener thread to accept new client connections and a different thread to actually read the incoming data by the clients. Since there are going to be many clients, I don't want to create a thread for each client. Instead, I'd prefer to use a single thread to read all incoming data in a loop, then either processing these data directly or creating work items for a different thread to process. I guess this approach would scale well enough. Any comments on this idea are most welcome.
The remaining problem I'm worried about is easy of communication. The above solution seems to require a manual protocol, possibly sending ASCII commands via TCP. While this would work, I think there should be a better way nowadays.
Some interface/proxyish way seems reasonable. I worked a bit with Java RMI before. From my point of understanding, .NET Remoting serves a similar purpose. Is Remoting a feasible solution to the scenario I described (many clients)? Is there an even better way I don't know of yet?
Edit:
This is not in LAN, but internet, if that matters.
If possible, it should also run under Linux.
As AresnMkrt pointed out, you should try WCF.
Just take it as is (with netTcpBinding, but don't forget to switch security off) and create a Tracer Bullet - measure if performance meets your requirements.
If not, you can try to tune WCF - WCF is very extensible, and you can modify message serialization to send ASCII messages as you want.
Are you sure you need a binary protocol? Rather, are you sure you need to invent a whole new protocol where plain RESTful service with JSON/XML will suffice? WCF can help you in this regard a lot.
Say I need to design an in-memory service because of a very high load read/write system. I want to dump the results of the objects every 2 minutes. How would I access the in-memory objects/data from within a web application?
(I was thinking a Windows service would be running in the background handling the in-memory service etc.)
I want the fastest possible solution, and I would guess most people would say use a web service? What other options would I have? I just don't understand how I could hook into the Windows service's objects etc.
(Please don't ask why I would want to do this, maybe you're right and it's a bad idea but I am also curious if this type of architecture is possible.)
Update
I was looking at this site swoopo.com that I would think has a lot of hits near the end of auctions, but since the auction keeps resetting the hits to the database would be just crazy so I was thinking if they did it in memory then dumped to db every x minutes...
What you're describing is called a cache, with a facade front-end.
You write a facade to which you commit your changes and acquire your datasets. The facade queues up reads and writes and commits when the queue is full or after a certain amount of time has passed. Your web application has a single point of access to the data (the facade), and the facade is structured in such a way to avoid writing and reading from storage too often.
Most relational database management systems do this for you. They do this kind of optimization and queuing internally so writing another layer on top of it would only slow things down. So don't write a cache if you're using an RDBMS.
Regarding the specifics of accessing such a facade, you can treat it as just an object, and implement it however you want (its own thread, a thread pool, a Web service, a Windows service, whatever).
Any remoting technology would work such as sockets, pipes and the like.
Check out: www.remobjects.com
You could use a Windows Message Queues or a Service Bus, or even .NET remoting.
See http://www.nservicebus.com/, or http://code.google.com/p/masstransit/.
You could hook into the Windows Services objects by using Remoting or WCF, both offer very fast interprocess communication. Sockets are fast too but are more cumbersome to program compared to WCF. There is a ton of WCF documentation and support online.
Databases provide a level of caching for you. The advantage of an in memory golden copy such as the one you propose is that it never has to read from disk when a request comes in and if you host it on the same machine as your IIS (provided you have enough RAM for both) there is no extra network hop, making it much faster that querying a db. However, the downside to this approach is that it does not scale as well if you need to add machines to load balance.
Third party messaging providers such as TIBCO are also worth looking at.
We're developing a .NET app that must make up to tens of thousands of small webservice calls to a 3rd party webservice. We would prefer a more 'chunky' call, but the 3rd party does not support it. We've designed the client to use a configurable number of worker threads, and through testing have code that is fairly well optimized for one multicore machine. However, we still want to improve the speed, and are looking at spreading the work accross multiple machines. We're well versed in typical client/server/database apps, but new to designing for multiple machines. So, a few questions related to that:
Is there any other client-side optimization, besides multithreading, that we should look at that could improve speed of a http request/response? (I should note this is a non-standard webservice, so is implemented using WebClient, not a WCF or SOAP client)
Our current thinking is to use WCF to publish chunks of work to MSMQ, and run clients on one or more machines to pull work off of the queue. We have experience with WCF + MSMQ, but want to be sure we're not missing better options. Are there other, better ways to do this today?
I've seen some 3rd party tools like DigiPede and Microsoft's HPC offerings, but these seem like overkill. Any experience with those products or reasons we should consider them over roll-our-own?
Sounds like your goal is to execute all these web service calls as quickly as you can, and get the results tabulated. Given that, your greatest efficiency control is going to be through scaling the number of concurrent requests you can make.
Be sure to look at your client-side connection limits. By default, I think the system default is 2 connections. I haven't tried this myself, but by upping the number of connections with this property, you should theoretically see a multiplier effect in terms of generating more requests by generating more connections from a single machine. There's more info on MS forums.
The MSMQ option works well. I'm running that configuration myself. ActiveMQ is also a fine solution, but MSMQ is already on the server.
You have a good starting point. Get that in operation, then move on to performance and throughput.
At CodeMash this year, Wesley Faler did an interesting presentation on this sort of problem. His solution was to store "jobs" in a DB, then use clients to pull down work and mark status when complete.
He then pushed the whole infrastructure up to Amazon's EC2.
Here's his slides from the presentation - they should give you the basic idea:
I've done something similar w/ multiple PC's locally - the basics of managing the workload were similar to Faler's approach.
If you have optimized the code, you could look into optimizing the network side to minimize the number of packets sent:
reuse HTTP sessions (i.e.: multiple transactions into one session by keeping the connection open, reduces TCP overhead)
reduce the number of HTTP headers to the minimum in the request to save bandwidth
if supported by server, use gzip to compress the body of the request (need to balance CPU usage to do the compression, and the bandwidth you save)
You might want to consider Rhino Service Bus instead of MSMQ. The source is available here.
I have two unrelated processes that use .NET assemblies as plugins. However, either process can be started/stopped at any time. I can't rely on a particular process being the server. In fact, there may be multiple copies running of one of the processes, but only one of the other.
I initially implemented a solution based off of this article. However, this requires the one implementing the server to be running before the client.
Whats the best way to implement some kind of notification to the server when the client(s) were running first?
Using shared memory is tougher because you'll have to manage the size of the shared memory buffer (or just pre-allocate enough). You'll also have to manually manage the data structures that you put in there. Once you have it tested and working though, it will be easier to use and test because of its simplicity.
If you go the remoting route, you can use the IpcChannel instead of the TCP or HTTP channels for a single system communication using Named Pipes. http://msdn.microsoft.com/en-us/library/4b3scst2.aspx. The problem with this solution is that you'll need to come up with a registry type solution (either in shared memory or some other persistent store) that processes can register their endpoints with. That way, when you're looking for them, you can find a way to query for all the endpoints that are running on the system and you can find what you're looking for. The benefits of going with Remoting are that the serialization and method calling are all pretty straightforward. Also, if you decide to move to multiple machines on a network, you could just flip the switch to use the networking channels instead. The cons are that Remoting can get frustrating unless you clearly separate what are "Remote" calls from what are "Local" calls.
I don't know much about WCF, but that also might be worth looking into. Spider sense says that it probably has a more elegant solution to this problem... maybe.
Alternatively, you can create a "server" process that is separate from all the other processes and that gets launched (use a system Mutex to make sure more than one isn't launched) to act as a go-between and registration hub for all the other processes.
One more thing to look into the Publish-Subscribe model for events (Pub/Sub). This technique helps when you have a listener that is launched before the event source is available, but you don't want to wait to register for the event. The "server" process will handle the event registry to link up the publishers and subscribers.
Why not host the server and the client on both sides, and whoever comes up first gets to be the server? And if the server drops out, the client that is still active switches roles.
There are many ways to handle IPC (.net or not) and via a TCP/HTTP tunnel is one way...but can be a very bad choice (depending on circumstances and enviornment).
Shared memory and named pipes are two ways (and yes they can be done in .Net) that might be better solutions for you. There is also the IPC class in the .Net Framework...but I personally don't like them due to some AppDomain issues...
I agree with Garo.
Using a pub/sub service would be a great solution. This obviously means that this service would need to be up and running before either of the other two.
If you want to skip the pub/sub you can just implement the service in both applications with different end points. When either of the applications is launched it tries to access the other known object via the IPC proxy. If the proxy fails, the other object isn't up.
-Scott
I've spent 2 days meandering through all the options available for IPC while looking for a reliable, simple, and fast way to do full-duplex IPC. IPCLibrary, which I found on Codeplex.com, is so far working perfectly out of all the options that I tried. All with only 7 lines of code. :D If anyone stumbles across this trying to find a full-duplex IPC, save yourself a ton of time and give this library a try. Grab the source code, compile the data.dll and follow the examples given.
HTH,
Circ