Develop Reference

C# BigData Queue

We´ve got an REST server which runs on a seperate machine than the main application server. Now we want to shift the data amongst other things from the REST server to the main-application server, also we want to send some messages from main-server to the REST server. Therefor we evaluated MQRabbit, for the message porpose it seems very suitable. But we now wondering whether MQRabbit can proceed about 1~2 GB of data in its queues.
I´ve followed the RabbitMQ tutorials
And now we have the following code:
public class QueueController<T> : IDisposable
{
private IModel channel;
private IConnection connection;
private ConnectionFactory factory = new ConnectionFactory() { HostName = "localhost" };
public string Topic { get; private set; }
public string LastMessage { get; private set; }
public QueueController()
{
connection = factory.CreateConnection();
channel = connection.CreateModel();
Topic = nameof(T);
}
public void Publish(List<T> data)
{
var body = Encoding.UTF8.GetBytes(LastMessage = data.SerializeJson());
var properties = channel.CreateBasicProperties();
properties.Persistent = true;
channel.BasicPublish(exchange: "",
routingKey: $"{Topic}_queue",
basicProperties: properties,
body: body);
}
public void Dispose()
{
channel.Dispose();
connection.Dispose();
}
}
Als MQRabbit´s tutorials show one producer and many consumer but our case is the other way around. Many producer and one consumer. Are there some best practices for those cases?

Let's first consider what a message queue does: sending messages -- small bits of data which communicate something to another computer system. The operative word here is small. Messages typically contain one of three things: 1. commands (go do something), 2. events (something happened), 3. requests (give me some data), and 4. responses (here is your data). A full discussion on these is beyond the scope, but suffice it to say that each of these can generally be composed of a small message less than 100kB.
Indeed, the AMQP protocol, which underlies RabbitMQ, is a fairly chatty protocol. It requires large messages be divided into multiple segments of no more than 131kB. This can add a significant amount of overhead to a large file transfer, especially when compared to other file transfer mechanisms (FTP, for instance). Secondly, the message has to be fully processed by the broker before it is made available in a queue, and it ties up valuable resources on the broker while this is being done. For one, the whole message must fit into RAM on the broker due to its architecture. This solution may work for one client and one broker, but it will break quickly when scaling out is attempted.

Firebase Xamarin.Forms - too many concurrent connections

currently I'm developing an App with a small team using Xamarin.Forms. We need to communicate with a database to get some locations , order details and so on. We are using Google's Firebase (Realtime DB) for this purpose. Everything is working fine when we are writing and reading data. However in the Firebase Console, in the usage tab, there are over 50 concurrent connections. This is weird since we are currently developing and didn't release any version of our app. There should be at most 5 concurrent connections (we are a team of 5).
We are using the NuGet-Package FirebaseDatabase.net (4.0.4) https://www.nuget.org/packages/FirebaseDatabase.net/ to read and write to the database.
Multiple Listeners are used to be able to react to changes in the database (so far it seems that each listener is taking up one connection which doesn't seem to be correct).
The code below shows the initialization of the FirebaseClient which is called once in the constructor.
private FirebaseClient InitDbClient()
{
var dbClient = new FirebaseClient(Constants.Values.FIREBASE_DATABASE_URL, new FirebaseOptions()
{
AuthTokenAsyncFactory = () => Task.FromResult(_authToken)
});
return dbClient;
}
Each listener is implemented in a similar way to the following code:
public IDisposable SubscribeToChatMessages(string orderID)
{
var observer = _dbClient.Child($"orders/{orderID}/Chat/Messages").AsObservable<JObject>();
var subscribe = observer.Subscribe(t =>
{
if (t.EventType == Firebase.Database.Streaming.FirebaseEventType.Delete)
{
return;
}
ChatMessage msg;
try
{
msg = JsonConvert.DeserializeObject<ChatMessage>(t.Object.ToString());
}catch(Exception e)
{
Debug.WriteLine(e.Message);
msg = new ChatMessage() { C = null, T = new DateTime(), U = null };
}
//...do something with the chat message
});
return subscribe;
}
Since I'm not sure what the problem is I just put some of our code in here. It would be awesome if anyone has a solution for this problem or has any idea what we might try.

I found the answer myself. As I already suspected each listener uses one connection. The reason for that is simple: the package is built on top of the rest api of firebase. In some other question it was mentioned that each listener is basically a streaming web socket (or something like that). Each of these consume one of the concurrent connections.
As a workaround (to reduce the amount of concurrent connections) I replaced all listeners that are not actually necessary with a combination of a timer and a simple db request. To be able to query for new data I added timestamps to the data itself. This allows me to use a db query similar to
dbClient.Child($"orders/{orderID}/Chat/Messages").orderBy("Time").startAt(lastRead).OnceAsync<>()...

C# How to detect internet connection / disconnection (with an event)?

I wrote this code that works perfectly, but I fear that ping every 2 seconds consumes too many resources or can create some problems with internet connection.
new Thread(() =>
{
if (CheckInternetConnection() == false)
{
Dispatcher.Invoke(new Action(delegate
{
//internet access lost
}));
}
else
{
Dispatcher.Invoke(new Action(delegate
{
//internet access
}));
}
Thread.Sleep(2000);
}).Start();
[DllImport("wininet.dll")]
private extern static bool InternetGetConnectedState(out int Description, int ReservedValue);
public static bool CheckInternetConnection()
{
int output = 0;
return InternetGetConnectedState(out output, 0);
}
These are two events that don't work in all occasions (only when IP or network card changes)
NetworkChange.NetworkAvailabilityChanged += NetworkChange_NetworkAvailabilityChanged
NetworkChange.NetworkAddressChanged += NetworkChange_NetworkAddressChanged;
Can someone help me ?

Note : In regaurds to your original solution
NetworkChange.NetworkAvailabilityChanged works fine, but
there are a couple of caveats: 1) it doesn't tell you if you have
Internet access, it just tells you whether there's at least one
non-loopback network adapter working, and 2) there are often extra
network adapters installed for various reasons that leave the system
in a "network is available" state, even when your main
Internet-connected adapter is disabled/unavailable - thanks to Peter Duniho
Since networking is more than just your routers or network card, and is really every hop to where ever it is you are trying to connect to at any time. The easiest and most reliable way is just ping a well known source like google, or use some sort of heart beat to one of your internet services.
The reasons this is the only reliable way is that any number of connectivity issues can occur in between you and the outside world. Even major service providers can go down.
So an IMCP ping to a known server like Google, or calling OpenRead on a WebClient are 2 valid approaches. These calls are not expensive comparatively and can be put into a light weight timer or continual task.
As for your comments you can probably signal a custom event to denote the loss of network after a certain amount of fails to be safe
To answer your question
But I fear that ping every 2 seconds consumes too many resources or
can create some problems with internet connection.
Both methods are very inexpensive in regards to CPU and network traffic, any resources used should be very minimal
Note : Just make sure you are pinging or connecting to a server with high availability, this will
allow such shenanigans and not just block you
Ping Example
using System.Net.NetworkInformation;
// Implementation
using (var ping = new Ping())
{
var reply = ping.Send("www.google.com");
if (reply != null && reply.Status != IPStatus.Success)
{
// Raise an event
// you might want to check for consistent failures
// before signalling a the Internet is down
}
}
// Or if you wanted to get fancy ping multiple sources
private async Task<List<PingReply>> PingAsync(List<string> listOfIPs)
{
Ping pingSender = new Ping();
var tasks = listOfIPs.Select(ip => pingSender.SendPingAsync(ip, 2000));
var results = await Task.WhenAll(tasks);
return results.ToList();
}
Connection Example
using System.Net;
// Implementation
try
{
using (WebClient client = new WebClient())
{
using (client.OpenRead("http://www.google.com/"))
{
// success
}
}
}
catch
{
// Raise an event
// you might want to check for consistent failures
// before signalling the Internet is down
}
Note : Both these methods have an async variant that will return a
Task and can be awaited for an Asynchronous programming pattern better suited for IO bound tasks
Resources
Ping.Send Method
Ping.SendAsync Method
WebClient.OpenRead Method
WebClient.OpenReadAsync Method

NetworkInterface.GetIsNetworkAvailable() is unreliable... since it would return true even if all the networks are not connected to internet. The best approach to check for connectivity, in my opinion, is to ping a well known and fast online resource. For example:
public static Boolean InternetAvailable()
{
try
{
using (WebClient client = new WebClient())
{
using (client.OpenRead("http://www.google.com/"))
{
return true;
}
}
}
catch
{
return false;
}
}
Anyway, those two events you are subscribing don't work the way you think... actually they check for the hardware status of your network adapters... not whether they are connected to internet or not. They have the same drawback as NetworkInterface.GetIsNetworkAvailable(). Keep on checking for connectivity into a separate thread that pings a safe source and act accordingly. Your Interop solution is excellent too.

Doing ping to public resources brings extra calls to your app and adds a dependency on that website or whatever you would use in the loop.
What if you use this method: NetworkInterface.GetIsNetworkAvailable() ?
Would it be enough for your app's purposes?
I found it here https://learn.microsoft.com/en-us/dotnet/api/system.net.networkinformation.networkinterface.getisnetworkavailable?view=netframework-4.7.1#System_Net_NetworkInformation_NetworkInterface_GetIsNetworkAvailable

c# save tcpclients which are connected to server

I am new to c# and I am building server / client application.
I have created both the server and the client successfully , but when any client connects to the server... I need to save that client because the server is supposed to send them a message after 10 minutes .
private void Form1_Load(object sender, EventArgs e) {
TcpListener myList;
myList = new TcpListener(IPAddress.Any, 8001);
while (true)
{
TcpClient client = myList.AcceptTcpClient();
MessageBox.Show("Connection accepted from " + client.Client.LocalEndPoint.ToString());
}
}
Now , my problem is how to save "client" id or anything about this client which is connected, to send message after 10 minutes from server to this client.
Can anyone help please?

Form onLoad is rather bad place to accept clients. Instead use ,for example, Background Worker. Also you might want to avoid using while(true) without any way to break the loop.
Object storage must be outside of method (event handler) to preserve connections from evil Garbage Collector. There are many ways to store object, it might be array (bothersome) or some collection, which although heavier to compute are pleasant to use. You can even use Concurent collections which will take care of thread synchronization on their own.
Dictionary<string,TcpClient> clientDict;
List<TcpClient> clientList;
...
void acceptClients()
{
TcpListener myList;
myList = new TcpListener(IPAddress.Any, 8001);
while (true)
{
TcpClient client = myList.AcceptTcpClient();
clientDict.Add("client nickname, id etc.",client);
clientList.Add(client);
MessageBox.Show("Connection accepted from " + client.Client.LocalEndPoint.ToString());
if (clientList.count>=8 || clientDict.count>=8)
{
break; // I want to break freeeeee!!!!
}
}
}
...
void sendToClient(string nick)
{
if (clientDict.ContainsKey(nick))
{
TcpClient client = clientDict[nick];
//and use selected client.
}
}
void broadcast()
{
foreach(TcpClient client in clientList) //clientList can be replaced with clientDict.Values
{
//and use selected client.
}
}

you would have to store the TCPClient in some list/dictionary. Regarding identifying connections you can use IP/Port in the TCPClient to differentiate between different connections.
Below is one such article i have posted to create a multi-threaded TCP chat application. it might be helpful.
http://shoaibsheikh.blogspot.com/2012/05/multi-threaded-tcp-socket-chat-server.html

The only possible way is to keep the connection open. Because many clients connect from behind NAT devices (corporate access points, home routers etc) is not possible to ask the client for a 'call back' address (IP:port).
What that mean in C# code, you need a reference to the client object you created in AcceptTcpClient. When you want to send something, you must retrieve this object and Write something into the client's stream (obtained via client.GetStream()). How exactly this is accomplished, it depends entirely on your code. A Dictionary perhaps? Do expect the connection to had closed by then for various reasons, even if KeepAlive is set.
Note that having a large number of accepted connections is not feasible (for many reasons).

How to do a UDP multicast across the local network in c#?

I am trying to get some simple UDP communication working on my local network.
All i want to do is do a multicast to all machines on the network
Here is my sending code
public void SendMessage(string message)
{
var data = Encoding.Default.GetBytes(message);
using (var udpClient = new UdpClient(AddressFamily.InterNetwork))
{
var address = IPAddress.Parse("224.100.0.1");
var ipEndPoint = new IPEndPoint(address, 8088);
udpClient.JoinMulticastGroup(address);
udpClient.Send(data, data.Length, ipEndPoint);
udpClient.Close();
}
}
and here is my receiving code
public void Start()
{
udpClient = new UdpClient(8088);
udpClient.JoinMulticastGroup(IPAddress.Parse("224.100.0.1"), 50);
receiveThread = new Thread(Receive);
receiveThread.Start();
}
public void Receive()
{
while (true)
{
var ipEndPoint = new IPEndPoint(IPAddress.Any, 0);
var data = udpClient.Receive(ref ipEndPoint);
Message = Encoding.Default.GetString(data);
// Raise the AfterReceive event
if (AfterReceive != null)
{
AfterReceive(this, new EventArgs());
}
}
}
It works perfectly on my local machine but not across the network.
-Does not seem to be the firewall. I disabled it on both machines and it still did not work.
-It works if i do a direct send to the hard coded IP address of the client machine (ie not multicast).
Any help would be appreciated.

Does your local network hardware support IGMP?
It's possible that your switch is multicast aware, but if IGMP is disabled it won't notice if any attached hardware subscribes to a particular multicast group so it wouldn't forward those packets.
To test this, temporarily connect two machines directly together with a cross-over cable. That should (AFAICR) always work.
Also, it should be the server half of the code that has the TTL argument supplied to JoinMulticastGroup(), not the client half.

I've just spent 4 hours on something similar (I think), the solution for me was:
client.Client.Bind(new IPEndPoint(IPAddress.Any, SSDP_PORT));
client.JoinMulticastGroup(SSDP_IP,IP.ExternalIPAddresses.First());
client.MulticastLoopback = true;
Using a specific (first external) IP address on the multicast group.

I can't see a TTL specified anywhere in the code. Remember that TTL was originally meant to be in unit seconds, but is has become unit hops. This means that by using a clever TTL you could eliminate passing through the router. The default TTL on my machine is 32 - I think that should be more than adequate; but yours may actually be different (UdpClient.Ttl) if your system has been through any form of a security lockdown.
I can't recommend the TTL you need - as I personally need to do a lot of experimentation.
If that doesn't work, you could have a look at these articles:
OSIX Article
CodeProject Article
All-in-all it looks like there has been success with using Sockets and not UdpClients.
Your chosen multicast group could also be local-only. Try another one.
Your physical network layer could also be causing issues. I would venture to question switches and direct (x-over) connections. Hubs and all more intelligent should handle them fine. I don't have any literature to back that, however.