I do a query on a path then add new data on the same path then read again with the same query and the new data is not in the result. I can see the new data in my FB console and if I restart my app, it will show. It's like I'm reading from cached data. What is wrong?
public static void GetScores(string readDbPath)
{
FirebaseDatabase.DefaultInstance.GetReference(readDbPath).OrderByChild("score")
.LimitToLast(Constants.FIREBASE_QUERY_ITEM_LIMIT)
.GetValueAsync().ContinueWith(task =>
{
if (task.IsFaulted)
{
// Handle the error...
Debug.LogError("FirebaseDatabase task.IsFaulted" + task.Exception.ToString());
}
else if (task.IsCompleted)
{
DataSnapshot snapshot = task.Result;
// Do something with snapshot...
List<Score> currentScoreList = new List<Score>();
foreach (var rank in snapshot.Children)
{
var highscoreobject = rank.Value as Dictionary<string, System.Object>;
string userID = highscoreobject["userID"].ToString();
int score = int.Parse(highscoreobject["score"].ToString());
currentScoreList.Add(new Score(score, userID));
}
OnStatsDataQueryReceived.Invoke(currentScoreList); // catched by leaderboard
}
});
}
It's very likely that you're using Firebase's disk persistence, which doesn't work well with Get calls. For a longer explanation of why that is, see my answer here: Firebase Offline Capabilities and addListenerForSingleValueEvent
So you'll have to choose: either use disk persistence, or use Get calls. The alternative to calling Get would be to monitor the ValueChanged event. In this case your callback will be invoked immediately when you change the value in the database. For more on this, see the Firebase documentation on listening for events.
This post was deleted stating it was just additional infos on my question. If fact it is the solution which is to use GoOffline() / GoOnline().
Thanks Frank for the detailed answer.
I tried with
FirebaseDatabase.DefaultInstance.SetPersistenceEnabled(false)
but the problem stayed the same. Using listeners is not what I want since every time a player would send a score, every player on the same path would receive refreshed data so I'm worried about b/w costs and performance.
The best solution I just found is to call before using a get
FirebaseDatabase.DefaultInstance.GoOnline();
then right after I get a response I set
FirebaseDatabase.DefaultInstance.GoOffline();
So far no performance hit I can notice and I get what I want, fresh data on each get. Plus persistence if working off line then going back.
Related
hope you guys are fine?
OK.. i am using MySQL.Data client/library to access and use MySQL database. I was using happily it for sometimes on quite a few project. But suddenly facing a new issue that causing me hold on my current project. :(
Because current project makes some (looks like it's a lot) db queries. and i am facing following exception :
Can't create more than max_prepared_stmt_count statements (current value: 16382)
i am closing and disposing the db engine/connection every time i am done with it. But getting damn confused why i am still getting this error.
here is the sample code just to give you idea.. (trimmed out unnecessary parts)
//this loop call an API with pagination and get API response
while(ContinueSalesOrderPage(apiClient, ref pageNum, days, out string response, window) == true)
{
//this handle the API date for the current page, it's normally 500 entry per page, and it throws the error on 4th page
KeyValueTag error = HandleSalesOrderPageData(response, pageNum, out int numOrders, window);
}
private KeyValueTag HandleSalesOrderPageData(string response, int pageNum, out int numOrders, WaitWindow window)
{
numOrders = json.ArrayOf("List").Size;
//init db
DatabaseWriter dbEngine = new DatabaseWriter()
{
Host = dbHost,
Name = dbName,
User = dbUser,
Password = dbPass,
};
//connecting to database
bool pass = dbEngine.Connect();
//loop through all the entry for the page, generally it's 500 entries
for(int orderLoop = 0; orderLoop < numOrders; orderLoop++)
{
//this actually handle the queries, and per loop there could be 3 to 10+ insert/update query using prepared statements
KeyValueTag error = InsertOrUpdateSalesOrder(dbEngine, item, config, pageNum, orderLoop, numOrders, window);
}
//here as you can see, i disconnect from db engine, and following method also close the db connection before hand
dbEngine.Disconnect();
}
//code from DatabaseWriter class, as you see this method close and dispose the database properly
public void Disconnect()
{
_CMD.Dispose();
_engine.Close();
_engine.Dispose();
}
so, as you can see i close/dispose the database connection on each page processing, but still it shows me that error on 4th page. FYI, 4th page data is not the matter i checked that. If i skip the page and only process the 4th page, it process successfully.
and after some digging more in google, i found prepare statement is saved in database server and that needs to be close/deallocate. But i can't find any way to do that using MySQL.Data Client :(
following page says:
https://dev.mysql.com/doc/refman/8.0/en/sql-prepared-statements.html
A prepared statement is specific to the session in which it was created. If you terminate a session without deallocating a previously prepared statement, the server deallocates it automatically.
but that seems incorrect, as i facing the error even after closing connection on each loop
so, i am at dead end and looking for some help here?
thanks in advance
best regards
From the official docs, the role of max_prepared_stmt_count is
This variable limits the total number of prepared statements in the server.
Therefore, you need to increase the value of the above variable, so as to increase the maximum number of allowed prepared statements in your MySQL server's configuration
Open the my.cnf file
Under the mysqld section, there is a variable max_prepared_stmt_count. Edit the value accordingly(remember the upper end of this value is 1048576)
Save and close the file. Restart MySQL service for changes to take place.
You're probably running into bug 77421 in MySql.Data: by default, it doesn't reset connections.
This means that temporary tables, user-declared variables, and prepared statements are never cleared on the server.
You can fix this by adding Connection Reset = True; to your connection string.
Another fix would be to switch to MySqlConnector, an alternative ADO.NET provider for MySQL that fixes this and other bugs. (Disclaimer: I'm the lead author.)
I have inherited a Windows Forms Application. We are having some performance issues, when trying to save a new record to the SQL Table. It hangs.
For the most part, I have ruled out the database or Table Structure as the problem. I don't believe that is the case here.
I am very new to this, and am having trouble stepping through and finding the root of the problem.
The form basics: (Only included what I thought was relevant code, but can add more if needed)
public partial class frmBadgeCreate : daBaseForm
{
private boBadger_History oBadger_History;
Form mainFormHandler;
private string whome;
}
public frmBadgeCreate(String cutomItem)
{
InitializeComponent();
if (daAppDesktop.IsRunning)
{
oBadger_History = new boBadger_History();
oBadger_History.GetAll(); /// This line right here seems to have some importance
whome = customItem;
}
}
public void saveitall() /// Save Action that hangs
{
// listing of form variables to be saved to table columns
var vlast = textbox_H_lname.Text;
var vfirst = textbox_H_fname.Text;
. . .and on and on . . .
var badger_History = new Badger_History() {hlastname = vlast, vfirstname = vfirst . . . and on and on . . };
oBadger_History.Add(badger_History);
oBadger_History.Save(); /// This is where things just hang forever.
Because this is a 'Lone Ranger App' that was handed to me, I am struggling to grasp it. What really confuses me is that when I comment out the 'oBadger_History.GetAll()' line, the save works very fast! Instantly. When I add the line back in, it hangs. I only know this, because I have spent days commenting out each line, one by one, and testing results.
The oBadger_History.GetAll(); looks like it is somehow used for the auto complete feature, so it is needed.
What has me totally scratching my head is that I can't see the connection between the 'GetAll' and the save. Why would the getAll impact the save function at all?
Here is the GetAll code, if that sheds any light:
public daBindingList<Badger_History> GetAll()
{
BadgerContext cn = (BadgerContext)this.context;
List<Badger_History> ents = cn.Badger_History.ToList();
this.EntityList = new daBindingList<Badger_History>(ents);
return this.EntityList;
}
Again, I don't believe that the SQL database/tables are the problem, seeing that I can get the save to work properly by removing the one line of code. I just can't seem to find a way to resolve
This line:
List<Badger_History> ents = cn.Badger_History.ToList();
Will load EVERY row in the Badger_History table into memory. How many records are in this table?
If there are lots of rows, then when you call Save(); (which I assume is some sort of wrapper around SaveChanges(), then EF will look through every row for anything that has changed. In your case, there may be 0 rows changed, as all you are interested in, is the row you are adding.
To speed things, you could change loading the data into a 'non-tracking' query
List<Badger_History> ents = cn.Badger_History.AsNoTracking().ToList();
This will still load in all the records, but they will no longer be counted when trying to save.
I am using Firebase Realtime Database to save some some links under a parent called - Links.
Let's say there are 5 links and my app fetched them. And after few days I added 5 more. How can I make my app only check the latest 5 ones added and not the previous ones ?
I have use OnValueChanged, ChildAdded, Child Removed etc. Nothing works the way I want. It checks all the children under Links which is wastage of bandwidth.
I specifically want my app to only fetch the updated or new ones.
I am using it with Unity. Any suggestions, please ?
Generate a reference per link and push them individually
Firebase's Push method to generate a time based key for each link as follows:
var linksParent = Firebase.Database.FirebaseDatabase.GetReference("linksParent");
var newLinks = new List<string>(){"somelink","otherLink","anotherLink","yetAnother", "lastLink"};
foreach(var link in links)
{
var linkRef = linksParent.Push(); // Generates a time based key for this reference.
linkRef.SetValueAsync(link);
}
To retrieve the latest 5 links, since they're stored lexicographically in the database by the time they were inserted.
var linksParent = Firebase.Database.FirebaseDatabase.GetReference("linksParent");
// Retrieve the most recent 5 links, if there's less than 5 in the list, it'll fetch them all.
linksParent.LimitToLast(5).GetValueAsync().ContinueWith(task=>
{
if(task.IsCompleted)
{
// Yes there are links and we fetched the latest.
if(task.Result.Exists)
{
// Do something with your links at task.Result.
}
}
});
You can also listen to the links list and fetch the latest 5 every time the list changes
var linksParent = Firebase.Database.FirebaseDatabase.GetReference("linksParent");
linksParent.LimitToLast(5).ValueChanged += OnLinksChanged;
void HandleValueChanged(object sender, ValueChangedEventArgs args)
{
if (args.DatabaseError != null)
{
Debug.LogError(args.DatabaseError.Message);
return;
}
// Do something with the links in args.Snapshot
}
I'm using the OnEntryWritten event in order to get events from the event log when they are fired.
The problem I started to see today is that some messages come with parameters unset.
For example:
The Windows Filtering Platform has permitted a bind to a local port. Application Information: Process ID:9852 Application Name:\device\harddiskvolume7\program files (x86)\google\chrome\application\chrome.exe Network Information: Source Address::: Source Port:51714 Protocol:17 Filter Information: Filter Run-Time ID:0 Layer Name:%%14608 Layer Run-Time ID:38
You can see the %%14608 parameter. This comes with a value if I see the same log in the Event Viewer.
I'm running a windows service as LocalSystem, so I don't know if this is a permission issue or this technology is not useful at all.
I have tried the rest offered on C# and they also don't meet my requirements.
UPDATE: this is the code I'm using to read the events.
1) First I subscribe to the corresponding Event log:
private void Subscribe()
{
_eventLog.EnableRaisingEvents = true;
var callbackFunction = new EntryWrittenEventHandler(OnEntryWritten);
_eventLog.EntryWritten += callbackFunction;
// Save a reference for callbackFunction
_eventHandler = callbackFunction;
}
2) Then on the callback method, I read data from the message:
public void OnEntryWritten(Object source, EntryWrittenEventArgs entryArgs)
{
// When overwrite policy is enabled, this will trigger for all elements when it starts writing new ones
try
{
var entry = entryArgs.Entry;
var timeWritten = entry.TimeWritten;
// This comes with %% values depending on the log
string message = entry.Message;
}
catch(Exception ex)
{
...
}
}
3) The event log variable is simply initialized as:
var eventLog = EventLog.GetEventLogs().FirstOrDefault(el => el.Log.Equals(logName, StringComparison.OrdinalIgnoreCase));
I need some help on this, honestly I don't know what else to try.
UPDATE
I'm adding some images here so everybody can understand the situation a little bit better. To be honest, it looks like there's no solution but to implement a dictionary and replace manually the required values, which appear to be always constants.
This is what I see on the Event Viewer for a given Event ID:
This is what I see on my program when I read that entry:
You can clearly see that the following values:
"Machine key." (Key type)
"Read persisted key from file." (Operation)
Are coming unmapped in the ReplacementStrings and the Message properties as: %%2499 and %%2458
This is the message value I get on the program:
"Key file operation.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tMyAccount$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t6644\r\n\tProcess Creation Time:\t2019-04-03T12:17:24.587994400Z\r\n\r\nCryptographic Parameters:\r\n\tProvider Name:\tMicrosoft Software Key Storage Provider\r\n\tAlgorithm Name:\tUNKNOWN\r\n\tKey Name:\t816339d2-c476-4f1e-bc40-954f0aa0f851\r\n\tKey Type:\t%%2499\r\n\r\nKey File Operation Information:\r\n\tFile Path:\tC:\ProgramData\Microsoft\Crypto\Keys\6d55a45fd69327293e9ed3e7930f4565_5663a8bb-2d1d-4c0d-90c1-624beddabe9c\r\n\tOperation:\t%%2458\r\n\tReturn Code:\t0x0"
What can be done here? There also nothing in entry.Data that might help me out to obtain both values.
No, I believe you're mistaken, sorry that this answer is too late. I found a similar event that was raised by chrome and evaluated if there's anything missed by the event handler. There wasn't anything missed. The message I got in my console output exactly matched what I saw in my Event Viewer.
A better solution would be to
Use entry.Data to get the complete data instead of entry.Message
entry.Data will return a byte[] which you can convert to a string. Here's the link to all properties that an entry will have.
I'm using the TraceEvent library to capture ETW traces, but I'm not able to determine the name of the process that caused an event.
Here is what I have so far:
var session = new TraceEventSession(sessionName, null);
session.EnableProvider(MyEventSource.Log.Guid, TraceEventLevel.Informational,
options: TraceEventOptions.Stacks);
Task.Delay(1000).ContinueWith(t => session.Stop()); // for testing, deal with it (⌐■_■)
var src = new ETWTraceEventSource(sessionName, TraceEventSourceType.Session);
TraceLog.CreateFromSource(src, etlxFile, null);
var log = TraceLog.OpenOrConvert(etlxFile);
var process = log.Events.First().ProcessName;
// breakpoint
When the breakpoint at the end is hit, process is "". ProcessID is a proper PID, but that's the only useful information I could find from the processes in the log.
I expected process names to be captured by the log. Am I doing something wrong, or is this API just not available on my OS (Windows 7)?
I truly believe that process name is not being captured by the ETW log. Etw system event contains only process ID field. Although TraceEvent library declares this one as a part of TraceEvent, this one actually is being populated based on executable image filename and process ID, which is implemented differently for all 4 TraceEventSource implementations.
Another observation is that I was never able to have this one populated (my OS is Windows 8.1).
The simple repro is to use SimpleEventSourceMonitor sample from Microsoft TraceEvent Library Samples package.
If you suspect that this is an issue, then it is better ask its owners Vance Morrison and Cosmin Radu.
This can be done by enabling the kernel provider, and then maintaining a lookup of process id to process name. Here's a rough example - no error checking, but you get the idea.
// create a lookup collection for future use
var pidToProcessName = new Dictionary<int, string>();
var session = new TraceEventSession(...);
// enable the kernel provider - note! this most come first
session.EnableKernelProvider(KernelTraceEventParser.Keywords.Process);
...
session.Source.Kernel.ProcessStart += ProcessStart;
session.Source.Dynamic.All += TraceEvent;
...
session.Source.Procces();
void ProcessStart(ProcessTraceData obj)
{
if(obj.OpCode == TraceEventOpcode.Start)
{
pidToProcessName[obj.ProcessID] = obj.ProcessName;
}
}
void TraceEvent(TraceEvent obj)
{
// pull the process name from our lookup
var processNameOfEvent = pidToProcessName[obj.ProcessID];
}