Develop Reference

Firebase Xamarin.Forms - too many concurrent connections

currently I'm developing an App with a small team using Xamarin.Forms. We need to communicate with a database to get some locations , order details and so on. We are using Google's Firebase (Realtime DB) for this purpose. Everything is working fine when we are writing and reading data. However in the Firebase Console, in the usage tab, there are over 50 concurrent connections. This is weird since we are currently developing and didn't release any version of our app. There should be at most 5 concurrent connections (we are a team of 5).
We are using the NuGet-Package FirebaseDatabase.net (4.0.4) https://www.nuget.org/packages/FirebaseDatabase.net/ to read and write to the database.
Multiple Listeners are used to be able to react to changes in the database (so far it seems that each listener is taking up one connection which doesn't seem to be correct).
The code below shows the initialization of the FirebaseClient which is called once in the constructor.
private FirebaseClient InitDbClient()
{
var dbClient = new FirebaseClient(Constants.Values.FIREBASE_DATABASE_URL, new FirebaseOptions()
{
AuthTokenAsyncFactory = () => Task.FromResult(_authToken)
});
return dbClient;
}
Each listener is implemented in a similar way to the following code:
public IDisposable SubscribeToChatMessages(string orderID)
{
var observer = _dbClient.Child($"orders/{orderID}/Chat/Messages").AsObservable<JObject>();
var subscribe = observer.Subscribe(t =>
{
if (t.EventType == Firebase.Database.Streaming.FirebaseEventType.Delete)
{
return;
}
ChatMessage msg;
try
{
msg = JsonConvert.DeserializeObject<ChatMessage>(t.Object.ToString());
}catch(Exception e)
{
Debug.WriteLine(e.Message);
msg = new ChatMessage() { C = null, T = new DateTime(), U = null };
}
//...do something with the chat message
});
return subscribe;
}
Since I'm not sure what the problem is I just put some of our code in here. It would be awesome if anyone has a solution for this problem or has any idea what we might try.

I found the answer myself. As I already suspected each listener uses one connection. The reason for that is simple: the package is built on top of the rest api of firebase. In some other question it was mentioned that each listener is basically a streaming web socket (or something like that). Each of these consume one of the concurrent connections.
As a workaround (to reduce the amount of concurrent connections) I replaced all listeners that are not actually necessary with a combination of a timer and a simple db request. To be able to query for new data I added timestamps to the data itself. This allows me to use a db query similar to
dbClient.Child($"orders/{orderID}/Chat/Messages").orderBy("Time").startAt(lastRead).OnceAsync<>()...

SqlDependency in a Windows Service not firing

I am trying to monitor a database table for changes using the SqlDependency class. Though I must be missing something. I have followed all of the examples that I see online and I have reviewed all the questions on this site. I just don't see what I am missing. Here are the initial commands that I ran on the database to enable the service broker and create the queue and service.
CREATE QUEUE ScheduleChangeQueue
GO
CREATE SERVICE ScheduleChangeService ON QUEUE ScheduleChangeQueue ([http://schemas.microsoft.com/SQL/Notifications/PostQueryNotification])
GO
ALTER DATABASE [database] SET ENABLE_BROKER
On the C# side, I have created a class that has a single static Setup method that is called to initiate the process. Here is the code for that:
public class SqlDependencyManager
{
private static bool DoesUserHavePermission()
{
var success = false;
try
{
Program.Log.Info("Retrieving SqlPermission to establish dependency...");
var clientPermission = new SqlClientPermission(System.Security.Permissions.PermissionState.Unrestricted);
// this will throw an error if the user does not have the permissions
clientPermission.Demand();
success = true;
Program.Log.Info("SqlPermission established. Continue setting up dependency.");
}
catch (Exception ex)
{
Program.Log.Error(ex, "SqlPermission not able to be established.");
}
return success;
}
public static void Setup()
{
if (!DoesUserHavePermission())
{
return;
}
var connectionString = ConfigurationManager.ConnectionStrings["ShowMakerPro"].ConnectionString;
// You must stop the dependency before starting a new one.
// You must start the dependency when creating a new one.
SqlDependency.Stop(connectionString);
SqlDependency.Start(connectionString);
using (var cn = new SqlConnection(connectionString))
{
using (var cmd = cn.CreateCommand())
{
cmd.CommandType = CommandType.Text;
//cmd.CommandText = "SELECT MAX(LastChangeTime) FROM Schedule WHERE ChannelID IN ( SELECT ID FROM Channels WHERE Type = 1 ) AND StartTime BETWEEN (GETDATE() - 7) AND (GETDATE() + 30)";
cmd.CommandText = "SELECT LastChangeTime FROM dbo.Schedule";
cmd.Notification = null;
// Creates a new dependency for the SqlCommand. Then creates attaches handler for the notification of data changes
new SqlDependency(cmd).OnChange += SqlDependency_OnChange;
cn.Open();
cmd.ExecuteReader();
}
}
Program.Log.Info("SQL Dependency set. Now monitoring schedule table for changes.");
}
private static void SqlDependency_OnChange(object sender, SqlNotificationEventArgs e)
{
if (e.Type == SqlNotificationType.Change)
{
// this will remove the event handler since the dependency is only for a single notification
((SqlDependency)sender).OnChange -= SqlDependency_OnChange;
ScheduleOutputterService.BuildSchedules();
Program.Log.Info("SQL Dependency triggered schedule rebuild. Resetting SqlDependency to monitor for changes.");
Setup();
}
}
}
I see the code get setup ok and the OnChange method is fired once for the Subscribe but then I never see it fire after that. I manually go into the database and change the LastChangeTime field hoping that it will force the firing of the event but nothing happens.
Can someone please shed some light on where I am screwing up? I see some people saying on line that this works fine in a windows form but they are also having some problems while in a service.

So I finally figured out the answer to my question and I thought I should list all the steps I took to get to this point so someone else coming along behind me will also have another place to look for answers since I seemed unable to find all of my answers in one place.
First off, I noticed in my situation that as soon as the subscription was set the OnChange event would fire right away. That is why I put in the check for change type so I could ignore those events. It turns out that ignoring those events was not a good thing because those events were actually trying to tell me something. Doing a search on my values directed me here:
http://msmvps.com/blogs/siva/archive/2011/11/22/subtle-sqldependency-notification-issue.aspx
This was very valuable because it helped me to see that there must have been a problem with some of my options in the database. Upon further inspection I noticed that my database was set to SQL Server 2000 compatibility. That is obviously my first problem because this is a 2005 and greater feature. So I tried to change my settings to the high version. This worked ok but then I still noticed that I was receiving the same event. So then I checked my database settings and I found that they were not set to match the options required to run service broker. You can see all the required option settings here:
http://msdn.microsoft.com/en-us/library/ms181122(v=SQL.100).aspx
After I inspected all these and tried to do some workarounds to get the right settings all squared away the setup was still failing. But this time it was failing because the updates would not save. It turned out that the client had triggers on the table in question and it's database settings that were being used to execute the trigger were in conflict with the needed settings to run QueryNotifications.
So long story short, the client decided that they did not want to change all the triggers that were being used so they abandoned the effort. But I learned a lot about how to troubleshoot SqlDependency and ServiceBroker. Hopefully these few links I provided will be helpful for someone else. A couple more links that were very helpful were mentioned in the comments but I am going to repost them in this answer so you can have some other items to review.
http://rusanu.com/2006/06/17/the-mysterious-notification/
http://rusanu.com/2005/12/20/troubleshooting-dialogs/

alternate options for infinite loop

We are running one third party application which uses C#, SQL SERVER. We have created one other application which prints the pass.
Basically it does continuous checking of new entry from third party application in one of the table on remote database. If new entry is present then it prints pass. Accessing network database in such way is not good way and also sometimes application hang.
Instead of continuous loop, I am searching for some other way like: As the new entry comes, it trigger my application for print. Or any other good way to implement.

What you are looking for is the SqlDependency that can help you in
listening to the OnChange event.
Example from msdn:
void Initialization()
{
// Create a dependency connection.
SqlDependency.Start(connectionString, queueName);
}
void SomeMethod()
{
// Assume connection is an open SqlConnection.
// Create a new SqlCommand object.
using (SqlCommand command=new SqlCommand(
"SELECT ShipperID, CompanyName, Phone FROM dbo.Shippers",
connection))
{
// Create a dependency and associate it with the SqlCommand.
SqlDependency dependency=new SqlDependency(command);
// Maintain the refence in a class member.
// Subscribe to the SqlDependency event.
dependency.OnChange+=new
OnChangeEventHandler(OnDependencyChange);
// Execute the command.
using (SqlDataReader reader = command.ExecuteReader())
{
// Process the DataReader.
}
}
}
// Handler method
void OnDependencyChange(object sender,
SqlNotificationEventArgs e )
{
// Handle the event (for example, invalidate this cache entry).
}
void Termination()
{
// Release the dependency.
SqlDependency.Stop(connectionString, queueName);
}
Check it out: http://msdn.microsoft.com/en-us/library/62xk7953.aspx
If any user subsequently changes the underlying data, Microsoft SQL
Server detects that there is a notification pending for such a change,
and posts a notification that is processed and forwarded to the client
through the underlying SqlConnection that was created by calling
SqlDependency.Start. The client listener receives the invalidation
message. The client listener then locates the associated SqlDependency
object and fires the OnChange event.

let me just sum up the data:
you need to give us more information for a more intelligent help here bu basically you can:
Use a timer: that way you are not in an infinite loop and you don't check every few ticks the 3rd party. you can be more sophisticated even and make the time interval grow and shrink according to data changes found or not
if you can change the 3rd party so it'll send events or signal when it has new data then you can use that and register to the events and save a lot of process time
if you can change the DB add triggers and use them to know then new data come and thus you don't need nothing more then that

SQL Server Notifications - My OnChange does not fire

I would like to make use of SQL Server notifications to capture insert events at my database within a winforms app. I am attempting to use the SQLDependency object. The MSDN articles make this seem pretty straight forward. So I have created a little example application to give it a try. The event only seems to fire as I enter my application the first time(MessageBox appears). Inserting data into the table does not raise the OnChange event it would seem. Can someone tell me what I'm missing? Thanks!
public Main()
{
InitializeComponent();
var check = EnoughPermission();
SqlDependency.Stop(constr);
SqlDependency.Start(constr);
if(connection == null)
{
connection = new SqlConnection(constr);
}
if(command == null)
{
command = new SqlCommand("Select ID, ChatMessage FROM dbo.Chat",connection);
}
connection.Open();
command.Notification = null;
SqlDependency dependency = new SqlDependency(command);
dependency.OnChange += new OnChangeEventHandler(dependency_OnChange);
command.ExecuteReader();
}
private void dependency_OnChange(object sender, SqlNotificationEventArgs e)
{
MessageBox.Show("Change!");
}

While I was working in the implementation of query notification, I got the exact problem. I checked all configurations, code pieces, and even TCP settings, but nothing helped. Then, I figured out the following query to run on database and it solved my problem. Maybe you can try it.
ALTER AUTHORIZATION ON DATABASE::[Your DB] TO sa;

Your first notification is the only notification you'll get. Query Notifications are not a subscription for changes, once a notification is fired it is also invalidate. You are supposed to re-submit a new notification subscription.
If your query is notified immedeatly it means you did not get a notification for a change, but one for an invalid query. Check the values of the SqlNotificationEventArgs argument you receive. Check the Info to be Insert/Update/Delete, check the Source to be Data, check the Type to be Change.
Have a look at the Watcher Application example to better understand how you are supposed to re-subscribe when notified. For a better understanding of how the Query Notifications work, see The Mysterious Notification.

Do you know a Bulked/Batched Flows Library for C#

I am working on a project with peek performance requirements, so we need to bulk (batch?) several operations (for example persisting the data to a database) for efficiency.
However, I want our code to maintain an easy to understand flow, like:
input = Read();
parsed = Parse(input);
if (parsed.Count > 10)
{
status = Persist(parsed);
ReportSuccess(status);
return;
}
ReportFailure();
The feature I'm looking for here is automatically have Persist() happen in bulks (and ergo asynchronously), but behave to its user as if it's synchronous (user should block until the bulk action completes). I want the implementor to be able to implement Persist(ICollection).
I looked into flow-based programming, with which I am not highly familiar. I saw one library for fbp in C# here, and played a bit with Microsoft's Workflow Foundation, but my impression is that both are overkill for what I need. What would you use to implement a bulked flow behavior?
Note that I would like to get code that is exactly like what I wrote (simple to understand & debug), so solutions that involve yield or configuration in order to connect flows to one another are inadequate for my purpose. Also, chaining
is not what I'm looking for - I don't want to first build a chain and then run it, I want code that looks as if it is a simple flow ("Do A, Do B, if C then do D").

Common problem - instead of calling Persist I usually load up commands (or smt along those lines) into a Persistor class then after the loop is finished I call Persistor.Persist to persist the batch.
Just a few pointers - If you're generating sql the commands you add to the persistor can represent your queries somehow (with built-in objects, custom objects or just query strings). If you're calling stored procedures you can use the commands to append stuff to a piece of xml tha will be passed down to the SP when you call the persist method.
hope it helps - Pretty sure there's a pattern for this but dunno the name :)

I don't know if this is what you need, because it's sqlserver based, but have you tried taking a look to SSIS and or DTS?

One simple thing that you can do is to create a MemoryBuffer where you push the messages which simply add them to a list and returns. This MemoryBuffer has a System.Timers.Timer which gets invoked periodically and do the "actual" updates.
One such implementation can be found in a Syslog Server (C#) at http://www.fantail.net.nz/wordpress/?p=5 in which the syslog messages gets logged to a SQL Server periodically in a batch.
This approach might not be good if the info being pushed to database is important, as if something goes wrong, you will lose the messages in MemoryBuffer.

How about using the BackgroundWorker class to persist each item asynchronously on a separate thread? For example:
using System;
using System.Collections;
using System.Collections.Generic;
using System.ComponentModel;
using System.Threading;
class PersistenceManager
{
public void Persist(ICollection persistable)
{
// initialize a list of background workers
var backgroundWorkers = new List<BackgroundWorker>();
// launch each persistable item in a background worker on a separate thread
foreach (var persistableItem in persistable)
{
var worker = new BackgroundWorker();
worker.DoWork += new DoWorkEventHandler(worker_DoWork);
backgroundWorkers.Add(worker);
worker.RunWorkerAsync(persistableItem);
}
// wait for all the workers to finish
while (true)
{
// sleep a little bit to give the workers a chance to finish
Thread.Sleep(100);
// continue looping until all workers are done processing
if (backgroundWorkers.Exists(w => w.IsBusy)) continue;
break;
}
// dispose all the workers
foreach (var w in backgroundWorkers) w.Dispose();
}
void worker_DoWork(object sender, DoWorkEventArgs e)
{
var persistableItem = e.Argument;
// TODO: add logic here to save the persistableItem to the database
}
}