I'm working on an "application system" , where I also need to make a server application. I'm working in C# (.NET 4.0). The server will mainly collect data from different POS applications / clients (which should be around 50-100, but the server should be capable of handling also around 200-300 clients). From a single client a server will probably receive around 1KB about 100x times a day. The server mainly needs to accept the data, decrypt it and store it to disk. It should also check for changes in specific directory in order to send new configurations to clients, which shouldn't be very often.
I'm quite new to C# and server programming so please bear with me. I thought about using threadpooling and async methods (there is a nice example using that in a book "C# in a nutshell"). But I spend quite some time looking for best solution and I found this. But multithreading brings more problems than benefits in my case. Thus I thought about even driven server. "A single process, handle every event (accepted connection, data available to read, can write to client, ...) on a callback." from " what is event driven web server". I find that the best solution to my problem.
But I have no idea on how to code it, I couldn't find any examples about event driven servers. As far as I understand it I should make one thread (+ 1 for GUI), then create a TCP listener and then somehow create events so that when TCP listener could accept a client the event would fire and wake up the server, also when data to read from clients would be available it would wake up the server.
Please help me out to code this, I'm totally lost. I know how I could do this using
while(true)
{
check if client wants to connect
accept client and add it to client list
iterate through client list and check if anyone is sending data ...
accept data and store it
...
}
But that is not event driven and is wasting CPU. Server will not be very active, so I'd like to make it as efficient as possible.
Some examples would really help.
Thank you for your time and answers.
p.s. Can I use just one port for all the clients?
EDIT: To clarify, I want to code an event driven server, but I don't know how to, thus I just made an example of what I know (client polling).
First, if you're new to C# and multithreading and sockets, that is a lot to bite off for your first project. I recommend learning these individually.
That said, you may find Nito.Async.Sockets helpful; it includes an event-driven server socket and handles the multithreading concerns for you.
Here is a server skeleton for what you might need. No exceptions are handled.
class Program
{
public static ManualResetEvent connected = new ManualResetEvent(false);
static void Main(string[] args)
{
string ip = "127.0.0.1";
int port = 14500;
TcpListener server = new TcpListener(IPAddress.Parse(ip), port);
server.Start();
Console.WriteLine("Server started...");
while (true)
{
connected.Reset();
server.BeginAcceptTcpClient(new AsyncCallback(AcceptCallback), server);
connected.WaitOne();
}
}
public static void AcceptCallback(IAsyncResult ar)
{
TcpListener listener = (TcpListener)ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
byte[] buffer = new byte[1024];
NetworkStream ns = client.GetStream();
if (ns.CanRead)
{
ns.BeginRead(buffer, 0, buffer.Length, new AsyncCallback(ReadCallback), new object[] { ns, buffer });
}
connected.Set();
}
public static void ReadCallback(IAsyncResult ar)
{
NetworkStream ns = (NetworkStream)((ar.AsyncState as object[])[0]);
byte[] buffer = (byte[])((ar.AsyncState as object[])[1]);
int n = ns.EndRead(ar);
if (n > 0)
{
Console.WriteLine(Encoding.ASCII.GetString(buffer, 0, n));
}
ns.BeginRead(buffer, 0, buffer.Length, new AsyncCallback(ReadCallback), new object[] { ns, buffer });
}
}
I don't know about C# frameworks, but you can have a look at Twisted, an event driven Python framework. You can find examples of servers and clients code, similar to your needs.
Here is a sample of a very simple server that echoes back to client whatever it received from it:
#!/usr/bin/env python
# Copyright (c) 2001-2009 Twisted Matrix Laboratories.
# See LICENSE for details.
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
### Protocol Implementation
# This is just about the simplest possible protocol
class Echo(Protocol):
def dataReceived(self, data):
"""
As soon as any data is received, write it back.
"""
self.transport.write(data)
def main():
f = Factory()
f.protocol = Echo
reactor.listenTCP(8000, f)
reactor.run()
if __name__ == '__main__':
main()
Regarding you last question:
P.S. Can I use just one port for all the clients?
Yes, you can (whatever language/framework you use).
I would suggest to use WCF hosted as a windows service, which provide a scalable and multithread platform. It can be tailored as per protocol requirement as well. Here is an example which can be used as a reference:
http://msdn.microsoft.com/en-us/library/ms733069.aspx
Event driven doesn't exactly describe what you expect your system to work like because you describe polling your clients for data to process instead of your clients pushing their data to your service (triggering an event).
If you have very little idea about how to code a system like this you could look into existing solutions/products for your scenario.
I'd recommend checking out EAI tools like BlueIntegrator or BizTalk for integrating POS clients.
For your requirement concerning rolling out client updates you could look into BITS.
I'm a bit late to the party, but I've recently started a new job developing software to integrate with scientific instruments and am currently going through the pain of learning threading, comms, async processing, etc. I've found Joe Albahari's Threading in C# to be an excellent resource for learning about the threading side of things.
Related
I'm pretty new to Android programming that's why I need your advice.
Current Situation:
I built an Android application (C#) aswell as a regular Server application (C++) which runs on a Raspberry Pi. Both programs communicate via UDP. At the moment that the Server application receives a signal it sends out a broadcast message which the Android application is listening for. Everything works just fine to the moment that the Android device falls asleep/goes idle which leads to my question.
Question:
How can I accomplish that the Android applications' listener still works, when the device falls asleep? I do not expect any solutions but any kind of advice so I don't waste time with wrong approaches.
Research:
- I read about and tried services that will keep running in the background but the service also stopped as the device went to sleep.
- I read about Broadcast Receivers which allow the application/service to get further information of the system.
- I read about WAKELOCK which allows me to keep the CPU alive, but for my purpose it should be up 'all the time' and that would drain to much energy.
Code that I would like to run in the background:
public void AsyncReceive()
{
// ...
Task.Run(() =>
{
while (this.isActive)
{
byte[] buffer = new byte[1];
DatagramPacket incoming = new DatagramPacket(buffer,
buffer.Length);
try
{
sock.Receive(incoming);
}
catch (...)
{
// Exception handling goes here...
}
// Communicate with the Android application
this.FireBroadCastReceivedEvent();
}
});
}
Edit
I also need to notice the application about incoming messages (#the 'FireBroadCastReceivedEvent()' part of the code). What would be a good way to do that?
I think you must read this link : https://developer.android.com/training/run-background-service/index.html
Hope you find what are you looking for.
I'm writing my first application with NetMQ (ZeroMQ implementation for .NET).
I also need to listen to information sent from a client using a traditional TCP socket (a.k.a a non-0MQ socket).
I've seen references to the availability of this socket type in the official ZeroMQ documentation here, (look for ZMQ_STREAM), but there's very few details on how to use it (and that doesn't help much either, the .NET API is quite a bit different from the C++ API).
The offical NetMQ documentation also makes no mention of the Streaming socket type.
Finally I had a look over to the Test suite for NetMQ on Github, and found a partial answer to my question in the method RawSocket.
The following snippet works:
using (NetMQContext context = NetMQContext.Create())
{
using (var routerSocket = context.CreateRouterSocket())
{
routerSocket.Options.RouterRawSocket = true;
routerSocket.Bind("tcp://127.0.0.1:5599");
byte[] id = routerSocket.Receive();
byte[] message = routerSocket.Receive();
Console.WriteLine(Encoding.ASCII.GetString(id));
Console.WriteLine(Encoding.ASCII.GetString(message));
}
}
When using standard TCP/IP test-tools, the byte[] message is printed out nicely, e.g. like this:
Hello World!
but the byte[] id is printed out like this:
???♥
In other words, I have no clue what's up with the id part. Why is routerSocket.Receive called twice? What is contained within the id? Is this something ZeroMQ/NetMQ specific, or is something TCP/IP specific information being extracted here?
Thanks to #Mangist for pointing this out.
The answer is in the RouterSocket documentation:
An identity, sometimes called an address, is just a binary string
with no meaning except "this is a unique handle to the connection".
Then, when you send a message via a ROUTER socket, you first send an
identity frame.
When receiving messages a ZMQ_ROUTER socket shall prepend a message
part containing the identity of the originating peer to the message
before passing it to the application. Messages received are
fair-queued from among all connected peers. When sending messages a
ZMQ_ROUTER socket shall remove the first part of the message and use
it to determine the identity of the peer the message shall be routed
to.
Identities are a difficult concept to understand, but it's essential
if you want to become a ZeroMQ expert. The ROUTER socket invents a
random identity for each connection with which it works. If there are
three REQ sockets connected to a ROUTER socket, it will invent three
random identities, one for each REQ socket.
This image illustrates the core concept of the ID frames:
I am working on a C# client for a server that wraps Netty. It is a TCP/IP server and I have tried using C# class TcpClient, but could not write anything onto the server or receive a printed response.
The Netty socket classes include the following: http://docs.jboss.org/netty/3.2/api/org/jboss/netty/channel/socket/nio/NioClientSocketChannelFactory.html http://docs.jboss.org/netty/3.2/api/org/jboss/netty/bootstrap/ClientBootstrap.html
The message is encoded as a byte[] in Java. Part of class PingSerializer, in the server code, reads as follows:
public byte[] requestToBytes(Ping message) {
return NorbertExampleProtos.Ping.newBuilder().setTimestamp(message.timestamp).build().toByteArray();
}
public Ping requestFromBytes(byte[] bytes) {
try {
return new Ping(NorbertExampleProtos.Ping.newBuilder().mergeFrom(bytes).build().getTimestamp());
} catch (InvalidProtocolBufferException e) {
System.out.println("Invalid protocol buffer exception " + e.getMessage());
throw new IllegalArgumentException(e);
}
}
I would like to know whether it is possible for a client written in C# to connect to the socket, ping the server and print out the server's response, without modifying the server code or using a cross-language development tool such as Apache Thrift or IKVM to handle the messages. Thanks, I would appreciate any help.
Judging by the code sample you've given, it looks like the data is encoded using Protocol Buffers, Google's serialization format.
Fortunately, there are at least two libraries implementing Protocol Buffers for .NET:
protobuf-net: Written for .NET from the ground up, this is a good choice if you don't particularly need the C# code to look like the equivalent Java/C++ code.
protobuf-csharp-port: This is a port from the Java client code, with some .NET idioms added - so if you're working with Protocol Buffers on multiple platforms, this may be more appropriate. (Disclaimer: I did most of the coding for this port.)
The good news is that the wire format for the two is the same, because it's the standard Protocol Buffer wire format. So if you decide later on that you've made the wrong choice, you don't need to worry about the data format changing.
In terms of communicating with the server, TcpClient should be absolutely fine. You'll need to find out exactly what the protocol is - for example, whether it's Protocol Buffers over HTTP, or something similar. (If it is over HTTP, WebClient would be a simpler approach.) However, beyond that it's straight TCP/IP: you write the bytes to the server, and it should write a reply. You can use Wireshark to look at the traffic between the client and the server, if you need to trace where problems are occurring.
There is an application called CS2J that will convert all of your C# code directly over to Java. However, you cannot expect it to be perfect and you will have a bit of debugging to do. It is supposed to very accurate.
I just started to learn ZeroMQ and want to build a distributed webcrawler as an example while learing.
My idea is to have a "server", written in PHP, which accepts a url where the crawling should start.
Workers (C# cli) will have to crawl that url, extract links, and push them back into a stack on the server. The server keeps sending urls in the stack to workers.
Perhaps a redis will keep track of all crawled urls, so we dont crawl sites multiple times and have the ability to extract statistics of the current process.
I would like to have the server to distribute tasks evenly, be aware of new/missing workers and redistribute urls when a worker doesnt respond.
Why PHP for the server: i'm just very comfortable with PHP, that is all. I dont want to make the example/testing project more complicated.
Why C# for the minions: because it runs on most windows machines. I can give the executable to various friends which can just execute it and help me test my project.
The crawling process and redis functionality are not part of my question.
My first approach was the PUSH/PULL pattern, which generally works for my scenario, but isnt aware of it's minions. I think i need a DEALER/ROUTER broker in the middle and have to handle the worker-awareness for myself.
I found this question but i'm not really sure if i understand the answer...
I'm asking for some hints how to impement the zmq stuff. Is the dealer approach correct? Is there any way to get an automatic worker-awareness? I think I need some resources/examples, or do you think that i just need to dig deeper in the zmq guide?
However, some hints towards the right direction would be great :)
Cheers
I'm building a job/task distributor that works the same as your crawler, in principal, at least. Here's a few things I've learned:
Define All Events
Communication between server and crawlers will be based on different things happening in your system, such as dispatching work from server to crawler, or a crawler sending a heartbeat message to the server. Define the system's event types; they are the use cases:
DISPATCH_WORK_TO_CRAWLER_EVENT
CRAWLER_NODE_STATUS_EVENT
...
Define a Message Standard
All communication between server and crawlers should be done using ZMsg's, so define a standard that organizes your frames, something like this:
Frame1: "Crawler v1.0" //this is a static header
Frame2: <event type> //ex: "CRAWLER_NODE_STATUS_EVENT"
Frame3: <content xml/json/binary> //content that applies to this event (if any)
Now you can create message validators to validate ZMsgs received between peers since you have a standard convention all messages must follow.
Server
Use a single ROUTER on the server for asynchrounous and bidirectional communication with the crawlers. Also, use a PUB socket for broadcasting heartbeat messages.
Don't block on the ROUTER socket, use a POLLER to loop every 5s or whatever, this allows the server to do other things periodically, like broadcast heartbeat events to the crawlers; something like this:
Socket rtr = .. //ZMQ.ROUTER
Socket pub = .. //ZMQ.PUB
ZMQ.Poller poller = new ZMQ.Poller(2)
poller.register( rtr, ZMQ.Poller.POLLIN)
poller.register( pub, ZMQ.Poller.POLLIN)
while (true) {
ZMsg msg = null
poller.poll(5000)
if( poller.pollin(0)){
//messages from crawlers
msg = ZMsg.recvMsg(rtr)
}
//send heartbeat messages
ZMsg hearbeatMsg = ...
//create message content here,
//publish to all crawlers
heartbeatMsg.send(pub)
}
To address your question about worker awareness, a simple and effective method uses a FIFO stack along with the heartbeat messages; something like this:
server maintains a simple FIFO stack in memory
server sends out heartbeats; crawlers respond with their node name; the ROUTER automatically puts the address of the node in the message as well (read up on message enveloping)
push 1 object onto the stack containing the node name and node address
when the server wants to dispatch work to a crawler, just pop the next object from the stack, create the message and address is properly (using the node address), and off it goes to that worker
dispatch more work to other crawlers the same way; when a crawler responds back to the server, just push another object with node name/address back on the stack; the other workers won't be available until they respond, so we don't bother them.
This is a simple but effective method of distributing work based on worker availability instead of blindly sending out work. Check lbbroker.php example, the concept is the same.
Crawler (Worker)
The worker should use a single DEALER socket along with a SUB. The DEALER is the main socket for async communication, and the SUB subscribes to heartbeat messages from the server. When the worker receives a heartbeat messages, it responds to the server on the DEALER socket.
Socket dlr = .. //ZMQ.DEALER
Socket sub = .. //ZMQ.SUB
ZMQ.Poller poller = new ZMQ.Poller(2)
poller.register( dlr, ZMQ.Poller.POLLIN)
poller.register( sub, ZMQ.Poller.POLLIN)
while (true) {
ZMsg msg = null
poller.poll(5000)
if( poller.pollin(0)){
//message from server
msg = ZMsg.recvMsg(dlr)
}
if( poller.pollin(1)){
//heartbeat message from server
msg = ZMsg.recvMsg(sub)
//reply back with status
ZMsg statusMsg = ...
statusMsg.send(dlr)
}
The rest you can figure out on your own. Work through the PHP examples, build stuff, break it, build more, it's the only way you'll learn!
Have fun, hope it helps!
I am designing a webservice interface for use between a Windows CE device and a PC. The Windows CE device is server and the PC is client.
I have decided to use the gSOAP library to implement the server and I am using .NET/C# for the client. I have followed the approach described here and everything is working well.
My question is about how to best implement an asynchronous callback/event from the server to the client. I can think of two methods:
Continuously polling the server for active events
A blocking method that keeps the connection open until an event occurs
I have currently chosen option 2 and it seems to be working well. I use an asynchronous method in the client and therefore get a callback when the method completes, i.e. when an event occurs on the Windows CE device. I then immediately call the same method again so it is ready for the next event.
Example server method (no error handling):
int ns__WaitForEvent(struct soap* soap, int *eventId)
{
WaitForSingleObject(hMyServerEvent, INFINITE);
*eventId = GetCurrentEventId();
return SOAP_OK;
}
Example client (no error handling):
private void SubscribeToServerEvents()
{
var server = new MyMethods.ServicePortTypeClient(
new BasicHttpBinding(),
new EndpointAddress(myIpAddress));
AsyncCallback cb = this.Callback;
server.BeginWaitForEvent(cb, server);
}
private void Callback(IAsyncResult ar)
{
var server = (MyMethods.ServicePortType)ar.AsyncState;
var result = server.EndWaitForEvent(ar);
// Do stuff with result
}
The server must be multi-threaded for this approach to work, and the number of clients should be limited so the server does not have a large number of threads hanging with blocking methods. In my case none of these issues are a problem - it is simple to setup a multi-threaded server using gSOAP and there will only ever be one client (which I control) attached to each server.
Are there any significant disadvantages to this approach? Can you suggest a better solution?
I suggest to turn the WinCE device into a webclient instead of a webserver and the PC into a server, that will be notified on something happens on the client. It is more natural this approach, you can still use gSoap for a soap client. On the PC you should have a web-server like Apache or IIS installed, or you could make a Windows server that will host an embedded light webserver.