Develop Reference

How to join results on a firebase query the right way

I am using Firebase real-time database for my app build with Unity. In order to build a "friend" leaderboard, the database will keep track of users their friends and their scores.
The database has the following structure:
scores{
user id : score
}
Users:{
Id: {
ageRange
email
name
friendlist : {
multiple friends user ids
}
}
}
The problem is in order to get the scores and the names of every friend the app has to make alot of api calls. Atleast if I correctly understand firebase. If the user has 10 friends it will take 21 calls before the leaderboard is filled.
I came up with the following code written in c#:
List<UserScore> leaderBoard = new List<UserScore>();
db.Child("users").Child(uid).Child("friendList").GetValueAsync().ContinueWith(task => {
if (task.IsCompleted)
{
//foreach friend
foreach(DataSnapshot h in task.Result.Children)
{
string curName ="";
int curScore = 0;
//get his name in the user table
db.Child("users").Child(h.Key).GetValueAsync().ContinueWith(t => {
if (t.IsCompleted)
{
DataSnapshot s = t.Result;
curName = s.Child("name").Value.ToString();
//get his score from the scores table
db.Child("scores").Child(h.Key).GetValueAsync().ContinueWith(q => {
if (q.IsCompleted)
{
DataSnapshot b = q.Result;
curScore = int.Parse(b.Value.ToString());
//make new userscore and add to leaderboard
leaderBoard.Add(new UserScore(curName, curScore));
Debug.Log(curName);
Debug.Log(curScore.ToString());
}
});
}
});
}
}
});
Is there any other way to do this? I've read multiple stack overflow questions watched firebase tutorials but i didnt found any simpler or more efficient way to get the job done.

There's not a way of reducing the number of API calls without duplicating data/restructuring your database. However, reducing API calls doesn't necessarily mean the overall read strategy is faster/better. My suggestion would be to optimize your reading strategy to reduce overall data read and to make reads concurrently when possible.
Solution 1: Optimize reading strategy
This is my recommended solution, because it doesn't include extra unnecessary data nor does it include managing consistency of your data.
Your code should look something like below:
(DISCLAIMER: I'm not a C# programmer, so there might be some errors)
List<UserScore> leaderBoard = new List<UserScore>();
db.Child("users").Child(uid).Child("friendList").GetValueAsync().ContinueWith(task => {
if (task.IsCompleted)
{
//foreach friend
foreach(DataSnapshot h in task.Result.Children)
{
// kick off the task to retrieve friend's name
Task nameTask = db.Child("users").Child(h.Key).Child("name").GetValueAsync();
// kick off the task to retrieve friend's score
Task scoreTask = db.Child("scores").Child(h.Key).GetValueAsync();
// join tasks into one final task
Task finalTask = Task.Factory.ContinueWhenAll((new[] {nameTask, scoreTask}), tasks => {
if (nameTask.IsCompleted && scoreTask.IsCompleted) {
// both tasks are complete; add new record to leaderboard
string name = nameTask.Result.Value.ToString();
int score = int.Parse(scoreTask.Result.Value.ToString());
leaderBoard.Add(new UserScore(name, score));
Debug.Log(name);
Debug.Log(score.ToString());
}
})
}
}
});
The above code improves the overall read strategy by not pulling all of a friend's user data (i.e. name, email, friendlist, etc.) and by pulling the name concurrently with score.
Solution 2: Duplicate name to scores table
If this still isn't optimal enough, you can always duplicate the friend's name in their score table. Something like below:
scores: {
<user_id>: {
name: <user_name>,
score: <user_score>
}
}
This would then allow you to only make one call per friend instead of two. However, you will still be reading the same amount of data, and you will have to manage the consistency of the data (either use a Firebase Function to propagate user name changes or write to both places).
Solution 3: Combine scores table into users table
If you don't want to manage the consistency issue, you can just combine the scores table into the users table.
Your structure would be something like:
users: {
<user_id>: {
name: <user_name>,
...,
score: <user_score>
}
}
However, in this instance, you will be reading more data (email, friendlist, etc.)
I hope this helps.

While the following will not reduce the number of calls, it will create tasks for the retrieval of the data and run them all simultaneously, returning the list of desired user scores.
var friendList = await db.Child("users").Child(uid).Child("friendList").GetValueAsync();
List<Task<UserScore>> tasks = new List<Task<UserScore>>();
//foreach friend
foreach(DataSnapshot friend in friendList.Children) {
var task = Task.Run( async () => {
var friendKey = friend.Key;
//get his name in the user table
var getName = db.Child("users").Child(friendKey).Child("name").GetValueAsync();
//get his score from the scores table
var getScore = db.Child("scores").Child(friendKey).GetValueAsync();
await Task.WhenAll(getName, getScore);
var name = getName.Result.Value.ToString();
var score = int.Parse(getScore.Result.Value.ToString());
//make new userscore to add to leader board
var userScore = new UserScore(name, score);
Debug.Log($"{name} : {score}");
return userScore;
});
tasks.Add(task);
}
var scores = await Task.WhenAll(tasks);
List<UserScore> leaderBoard = new List<UserScore>(scores);

This is mostly database structure issue.
First, you need leaderboard table.
leaderboard: {
<user_id>: <score>
}
Second, you need users table.
users: {
<user_id>: {
name: <user_name>,
...,
score: <user_score>,
friendlist: {
multiple friends user ids
}
}
}
And you have to update leaderboard's score and users' score at the same time.
If you want to avoid Callback hell.
You can also try like this. (This code is JAVA)
// Create a new ThreadPoolExecutor with 2 threads for each processor on the
// device and a 60 second keep-alive time.
int numCores = Runtime.getRuntime().availableProcessors();
ExecutorService executor = new ThreadPoolExecutor(
numCores * 2,
numCores * 2,
60L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>()
);
Tasks.call(executor, (Callable<Void>) () -> {
Task<Token> getStableTokenTask = NotificationUtil.getStableToken(token);
Token stableToken;
stableToken = Tasks.await(getStableTokenTask);
if (stableToken != null) {
Task<Void> updateStableTokenTask = NotificationUtil.updateStableToken(stableToken.getId(), versionCode, versionName);
Tasks.await(updateStableTokenTask);
}
if (stableToken == null) {
Token newToken = new Token(token, versionCode, versionName);
Task<Void> insertStableTokenTask = NotificationUtil.insertStableToken(newToken);
Tasks.await(insertStableTokenTask);
}
return null;
}).continueWith((Continuation<Void, Void>) task -> {
if (!task.isSuccessful()) {
// You will catch every exceptions from here.
Log.w(TAG, task.getException());
return null;
}
return null;
});

SignalR message being sent to every opened page instance. How to send to only exact one?

This question has been answered multiple times, but I'm still not getting it.
I am using an example from this page: https://www.codeproject.com/Articles/1124691/SignalR-Progress-Bar-Simple-Example-Sending-Live-D
However, I've modified the SendProgress method a bit, to send the message only to the specific connection, not all clients:
public static void SendProgress(string connectionId, string progressMessage, int progressCount, int totalItems)
{
var hubContext = GlobalHost.ConnectionManager.GetHubContext<RealTimeProgressBar.ProgressHub>();
var percentage = (progressCount * 100) / totalItems;
hubContext.Clients.Client(connectionId).AddProgress(progressMessage, percentage + "%");
}
I am getting connection Id from overrided OnConnected method:
public override Task OnConnected()
{
Environments.ConnectionId = Context.ConnectionId;
return base.OnConnected();
}
I'm currently storing it in a static class Environments, I guess there is a better way for doing so, but I don't know it yet :)
The question is, when I open for example two instances of my web page in browser, the progress bar is still being shown for all of these instances, though the connectionIds on these pages are different (checked it in SendProgress function).
What is wrong here?
Perhaps, I should change something in js code?
$(function () {
// Reference the auto-generated proxy for the hub.
var progress = $.connection.progressHub;
console.log(progress);
// Create a function that the hub can call back to display messages.
progress.client.AddProgress = function (message, percentage) {
ProgressBarModal("show", message + " " + percentage);
$('#ProgressMessage').width(percentage);
if (percentage == "100%") {
ProgressBarModal();
}
};
$.connection.hub.start().done(function () {
var connectionId = $.connection.hub.id;
console.log(connectionId);
});
});
As i'm not really sure if js funcs really uses Environments.ConnectionId or smthing else..
Thanks!
Edit: What I want to achive is to send unique progress bars for every opened page. e.g. when I press the button on a first page, I'm starting to collect data and progress bar reflects that. I also open another page in a browser, and collect data from another source, so the progress bar should reflect that for this page as well.
my controller method:
public ActionResult Generate(InputDataViewModel viewModel, string connectionId)
{
...
var Tasks = getResultsWithProgressBar(viewModel.jobId, connectionId);
...
return RedirectToAction("Details", new { id = job.JobDataId });
}
and somewhere inside this "getResult" is a function that actually calls progress bar(SendProgress)

Your change should work fine.
You are using a static ConnectionId, so only your last browser, the browser you opened last(or connected last) will see the progress bar as well as updates.
This is the only one i made apart apart from adding in the static Environment.ConnectionId.
//PUSHING DATA TO ALL CLIENTS
// hubContext.Clients.All.AddProgress(progressMessage, percentage + "%");
hubContext.Clients.Client(Environments.ConnectionId).AddProgress(progressMessage, percentage + "%");
So if you open 10 browser windows and click on the button on all those, one by one, only the browser you initially opened last will see the the progress bar as well as updates, but it will see updates from all the 10 browsers shown on that single progress bar!
It is fun to see though!
When I tried your code, I saw the behavior you observed initially. But when I opened two separate Incognito browsers, I see the correct behavior. Close the IIS express site, rebuild and retry it.
Edit:
If you need each tab to have its own progress bar, then you will need one of the two approaches below:
A way to identify which rest request came from which connectionId. If you want to continue using the current code. Just pass in the connection Id to the LongRunningProcess. Client code below:
var progress = $.connection.progressHub;
var connectionId = $.connection.hub.id;
$.getJSON("/Home/LongRunningProcess",
{"connectionId":connectionId},
function (data) {
if (!data) {
alert("Success");
}
else
{
alert(data);
}
});
Server code would be:
public JsonResult LongRunningProcess(string connectionId)
{
//THIS COULD BE SOME LIST OF DATA
int itemsCount = 100;
for (int i = 0; i <= itemsCount; i++)
{
//SIMULATING SOME TASK
Thread.Sleep(500);
//CALLING A FUNCTION THAT CALCULATES PERCENTAGE AND SENDS THE DATA TO THE CLIENT
Functions.SendProgress(connectionId, "Process in progress...", i , itemsCount);
}
return Json("", JsonRequestBehavior.AllowGet);
}
The other option is to call a signalr method on the server instead of a rest method. Create a new hub method in the ProgressHub like below:
public void LongRunningHubMethod()
{
//THIS COULD BE SOME LIST OF DATA
int itemsCount = 100;
for (int i = 0; i <= itemsCount; i++)
{
//SIMULATING SOME TASK
Thread.Sleep(500);
//CALLING A FUNCTION THAT CALCULATES PERCENTAGE AND SENDS THE DATA TO THE CLIENT
Functions.SendProgress(Context.ConnectionId, "Process in progress...", i, itemsCount);
}
}
The client would then just call in the hub method on click like below:
var progress = $.connection.progressHub;
var connectionId = $.connection.hub.id;
progress.invoke('LongRunningHubMethod');

Making a Discord Loop

I am currently making a bot for my server and I'm trying to make a function to when the bot receives .raid it spams messages I've tried loops but it keeps messing up my code. This is what I have for my C#. I just cant seem to get a loop working specifically in discord.
//RAID
private void RegisterRaidCommand()
{
commands.CreateCommand("raid")
.Do(async (e) =>
{
await e.Channel.SendMessage("READY SIR!");
//here is where the loop needs to start
//here is where the spam message goes
});
}

i made a spam/raid command using a normal for loop and it works fine for me. This command just takes in 2 parameters, one for the word or phrase that will be spammed and the int parameter for the amount of times you want it to be spammed.
[Command("spam")]
[RequireUserPermission(GuildPermission.Administrator)]
public async Task CMD14(int times, [Remainder] string Word)
{
for (int i = 0; i < times; i++)
{
await Context.Channel.SendMessageAsync($"`{Word}`");
}
}
Hope this helps in some way, shape or form :)

SignalR - Send message OnConnected

I've been experimenting with SignalR today and It's really neat. Basically what I wanted to achieve is the following:
As soon as a device connects it should send a message to the first one. If there are more devices than 1 connected I would like to send two messages. One to all except the last connected client. And one message to only the last connected client.
The code I've been using works perfect when I place it in a custom API controller and basically call the action, but that's not what I want.
I would like to send the messages as soon as a device connects within OnConnected without any user interaction, but when I place my code inside the OnConnected override it stops working. It doesn't send to the specific clients anymore (first connected and last connected).
I hope someone is able to help me out with this, because I've been banging my head for a few hours now.
public override System.Threading.Tasks.Task OnConnected()
{
UserHandler.ConnectedIds.Add(Context.ConnectionId, UserHandler.ConnectedIds.Count + 1);
int amountOfConnections = UserHandler.ConnectedIds.Count;
var lastConnection = UserHandler.ConnectedIds.OrderBy(x => x.Value).LastOrDefault();
var allExceptLast = UserHandler.ConnectedIds.Take(amountOfConnections - 1).Select(x => x.Key).ToList();
if (amountOfConnections == 1)
{
Clients.Client(UserHandler.ConnectedIds.First().Key).hello("Send to only(also first) one");
}
else
{
Clients.Clients(allExceptLast).hello("Send to everyone except last");
Clients.Client(lastConnection.Key).hello("Send to only the last one");
}
return base.OnConnected();
}

Unless I miss something from your question, the solution looks pretty simple to me, you just need to switch to using
Clients.Caller.hello("Send to only the last one");
instead of trying to understand yourself who's the last connected id. Same for the other ones, you can use:
Clients.Others.hello("Send to everyone except last");
You do not need all the logic you setup, those 2 lines do what you need, and they work inside OnConnected.

Thanks for all the help (upvoted you guys). Actually found the problem.. it was inside my client. I first subscribed to the 'hello' function and after that I started the HubConnection. As soon as I changed this order everything worked fine.
It worked with the following client code:
private async Task ConnectToSignalR()
{
var hubConnection = new HubConnection("url");
hubConnection.Headers["x-zumo-application"] = "clientapikey";
IHubProxy proxy = hubConnection.CreateHubProxy("ChatHub");
proxy.On<string>("hello", async (msg) =>
{
Console.WriteLine(msg);
});
await hubConnection.Start();
}

Since you haven't established a connection yet, trying to call your client .hello() function within OnConnected is not possible at this point. However, we can define a server hub method and immediately call that upon our connection .done callback. Then, in our new server method we can reallocate the logic you currently have in OnConnected.
This will change our setup quite a bit and introduce some additional steps, but observe the following example...
// WhateverHub
public override Task OnConnected()
{
return base.OnConnected()
}
public void AfterConnected()
{
// if(stuff) -- whatever if/else first user/last user logic
// {
Clients.Caller.hello("message")
// }
}
var proxy= $.connection.whateverHub;
proxy.client.hello = function(message) {
// last step in event chain
}
$.connection.hub.start().done(function () {
proxy.server.afterConnected() // call AfterConnected() on hub
});
So the basic idea here is to first
Connect => .done(function() { ... });
Call server.afterConnected()
Execute logic within that method
If we're satisfied with conditions call our .hello() client function
Note - this implementation is for a JavaScript client - but the same idea can be translated to a .net client. This is mostly an architectural issue.

well... you are returning a task... so i think that may be the issue...
you should first execute your code and then return the task... or put a ContinueWith... like...
public override Task OnConnected()
{
Task task = new Task(() =>
{
UserHandler.ConnectedIds.Add(Context.ConnectionId, UserHandler.ConnectedIds.Count + 1);
int amountOfConnections = UserHandler.ConnectedIds.Count;
var lastConnection = UserHandler.ConnectedIds.OrderBy(x => x.Value).LastOrDefault();
var allExceptLast = UserHandler.ConnectedIds.Take(amountOfConnections - 1).Select(x => x.Key).ToList();
if (amountOfConnections == 1)
{
Clients.Client(UserHandler.ConnectedIds.First().Key).hello("Send to only(also first) one");
}
else
{
Clients.Clients(allExceptLast).hello("Send to everyone except last");
Clients.Client(lastConnection.Key).hello("Send to only the last one");
}
});
task.ContinueWith(base.OnConnected());
return task;
}
I haven't tested that... its just a guess..

Closing WCF Service from Async method?

I have a service layer project on an MVC 5 ASP.NET application I am creating on .NET 4.5.2 which calls out to an External 3rd Party WCF Service to Get Information asynchronously. An original method to call external service was as below (there are 3 of these all similar in total which I call in order from my GetInfoFromExternalService method (note it isnt actually called that - just naming it for illustration)
private async Task<string> GetTokenIdForCarsAsync(Car[] cars)
{
try
{
if (_externalpServiceClient == null)
{
_externalpServiceClient = new ExternalServiceClient("WSHttpBinding_IExternalService");
}
string tokenId= await _externalpServiceClient .GetInfoForCarsAsync(cars).ConfigureAwait(false);
return tokenId;
}
catch (Exception ex)
{
//TODO plug in log 4 net
throw new Exception("Failed" + ex.Message);
}
finally
{
CloseExternalServiceClient(_externalpServiceClient);
_externalpServiceClient= null;
}
}
So that meant that when each async call had completed the finally block ran - the WCF client was closed and set to null and then newed up when another request was made. This was working fine until a change needed to be made whereby if the number of cars passed in by User exceeds 1000 I create a Split Function and then call my GetInfoFromExternalService method in a WhenAll with each 1000 - as below:
if (cars.Count > 1000)
{
const int packageSize = 1000;
var packages = SplitCarss(cars, packageSize);
//kick off the number of split packages we got above in Parallel and await until they all complete
await Task.WhenAll(packages.Select(GetInfoFromExternalService));
}
However this now falls over as if I have 3000 cars the method call to GetTokenId news up the WCF service but the finally blocks closes it so the second batch of 1000 that is attempting to be run throws an exception. If I remove the finally block the code works ok - but it is obviously not good practice to not be closing this WCF client.
I had tried putting it after my if else block where the cars.count is evaluated - but if a User uploads for e.g 2000 cars and that completes and runs in say 1 min - in the meantime as the user had control in the Webpage they could upload another 2000 or another User could upload and again it falls over with an Exception.
Is there a good way anyone can see to correctly close the External Service Client?

Based on the related question of yours, your "split" logic doesn't seem to give you what you're trying to achieve. WhenAll still executes requests in parallel, so you may end up running more than 1000 requests at any given moment of time. Use SemaphoreSlim to throttle the number of simultaneously active requests and limit that number to 1000. This way, you don't need to do any splits.
Another issue might be in how you handle the creation/disposal of ExternalServiceClient client. I suspect there might a race condition there.
Lastly, when you re-throw from the catch block, you should at least include a reference to the original exception.
Here's how to address these issues (untested, but should give you the idea):
const int MAX_PARALLEL = 1000;
SemaphoreSlim _semaphoreSlim = new SemaphoreSlim(MAX_PARALLEL);
volatile int _activeClients = 0;
readonly object _lock = new Object();
ExternalServiceClient _externalpServiceClient = null;
ExternalServiceClient GetClient()
{
lock (_lock)
{
if (_activeClients == 0)
_externalpServiceClient = new ExternalServiceClient("WSHttpBinding_IExternalService");
_activeClients++;
return _externalpServiceClient;
}
}
void ReleaseClient()
{
lock (_lock)
{
_activeClients--;
if (_activeClients == 0)
{
_externalpServiceClient.Close();
_externalpServiceClient = null;
}
}
}
private async Task<string> GetTokenIdForCarsAsync(Car[] cars)
{
var client = GetClient();
try
{
await _semaphoreSlim.WaitAsync().ConfigureAwait(false);
try
{
string tokenId = await client.GetInfoForCarsAsync(cars).ConfigureAwait(false);
return tokenId;
}
catch (Exception ex)
{
//TODO plug in log 4 net
throw new Exception("Failed" + ex.Message, ex);
}
finally
{
_semaphoreSlim.Release();
}
}
finally
{
ReleaseClient();
}
}
Updated based on the comment:
the External WebService company can accept me passing up to 5000 car
objects in one call - though they recommend splitting into batches of
1000 and run up to 5 in parallel at one time - so when I mention 7000
- I dont mean GetTokenIdForCarAsync would be called 7000 times - with my code currently it should be called 7 times - i.e giving me back 7
token ids - I am wondering can I use your semaphore slim to run first
5 in parallel and then 2
The changes are minimal (but untested). First:
const int MAX_PARALLEL = 5;
Then, using Marc Gravell's ChunkExtension.Chunkify, we introduce GetAllTokenIdForCarsAsync, which in turn will be calling GetTokenIdForCarsAsync from above:
private async Task<string[]> GetAllTokenIdForCarsAsync(Car[] cars)
{
var results = new List<string>();
var chunks = cars.Chunkify(1000);
var tasks = chunks.Select(chunk => GetTokenIdForCarsAsync(chunk)).ToArray();
await Task.WhenAll(tasks);
return tasks.Select(task => task.Result).ToArray();
}
Now you can pass all 7000 cars into GetAllTokenIdForCarsAsync. This is a skeleton, it can be improved with some retry logic if any of the batch requests has failed (I'm leaving that up to you).