I'll begin this post by noting that I'm entirely new to the .NET world. ASP, EntityFramework, Linq, etc. are all mostly unknown magic at this point.
Having said that, I've built myself a neat Web API chat-like application with SignalR support for real-time events. It works quite well, but I'm having some performance problems with the Add function.
In my chat application, there are "Pads" (chat rooms) which contain a number of "Mates" and "Messages". Here's my Pad model for reference:
public class Pad
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public Guid PadId { get; set; }
public string StreetAddress { get; set; }
public int ZipCode { get; set; }
public virtual ICollection<Mate> Mates { get; set; }
public virtual ICollection<Message> Messages { get; set; }
}
My problem lies in my SignalR hub that processes a new Message sent to a particular pad. These two lines take about half a second to process.
pad.Messages.Add(msg); // pad is the Pad entity already fetched from the db context
db.Messages.Add(msg);
But they only take that long when Pad.Messages contains a large number of messages. Thousands. If I am sending to a pad with few to no messages, it executes almost instantly.
My initial 'trick' to improve the perceived performance here is to move the adding functions to after I send the notification back to the clients, but I realize something like this could present a potential problem later when there are tens or hundreds of thousands of messages in one pad.
Any advice here would be greatly appreciated!
Here is the entire message send method for reference:
public void SendMessage(string pad_id, string body)
{
var user_id = IdentityExtensions.GetUserId(Context.User.Identity);
body = body.Trim();
if (body.Length <= 0)
{
return;
}
// Check that the user belongs in this pad...
var user = (from u in db.Users
where u.Id == user_id
select u).First();
var pad = (from p in user.Pads where p.PadId == new Guid(pad_id) select p).FirstOrDefault();
if (pad != null) {
// Save the message to the database
var msg = new Message()
{
MessageId = new Guid(),
Author = user,
Body = body,
SendTime = DateTimeOffset.UtcNow,
Pad = pad
};
pad.Messages.Add(msg); // These two lines
db.Messages.Add(msg); // Are the culprit.
db.SaveChangesAsync();
Clients.Group(pad_id).messageReceived(user.Id, pad_id, body, DateTimeOffset.UtcNow); // Send message to clients
}
}
EDIT: I'm on EF 6.0.0
From your code, all you want to do is add 1 row (Message) to the table.
While constructing the Message entity, just use the padid instead of the Pad object.
That way you don't need to deal with all the pad objects. From a DB perspective, you just need to add a Message row with a PadId key.
var msg = new Message()
{
MessageId = new Guid(),
Author = user,
Body = body,
SendTime = DateTimeOffset.UtcNow,
PadId = new Guid(pad_id)
};
// pad.Messages.Add(msg); // don't need this.
db.Messages.Add(msg);
The above should always insert one row and not depend on the number of messages in the pad.
If it still gives a performance problem, then adding a single row is causing a lot of index updates to your table.
Related
I have the following database in Firebase:
Whenever I try to update some items, it deletes the unaltered ones, instead of letting them keep their values. I have the following code:
FirebaseClient firebaseClient = new FirebaseClient("FirebaseLink");
MyDatabaseRecord databaserecord = new MyDatabaseRecord
{
Plate1 = EntryPlate1.Text.ToString(),
Plate2 = EntryPlate2.Text.ToString()
};
string restName = "Rest1";
await firebaseClient.Child("Menus/" + restName).PutAsync(databaserecord); //Adicionar reload à página 2 após o click no botão de adicionar ou noutro click
EntryPlate1.Text = "";
EntryPlate2.Text = "";
MyDatabaseRecord.cs:
public class MyDatabaseRecord
{
public string Plate1 { get; set; }
public string Plate2 { get; set; }
}
What I mean by "it deletes the unaltered ones" is that, in this example, although my only changes are to "Plate1" and "Plate2" values, when its executed, it deletes "Plate3" from the database, instead of just replacing the ones I'm targeting.
What can I change to my code in order for this to work like intended?
The PutAsync method is the equivalent of a HTTP PUT call of the Firebase REST API, which rewrites the data at the given path with the data that you pass into the call. If you want to instead patch the data, use PatchAsync which performs a selective update only overwriting the properties that you pass in.
I made a MEME command of Discord Bot that posts a list of memes from phpMyAdmin DB. But it is not about that. I want to add specific emojis on that bot message that people could press on those emojis and kinda vote.
It is how it posts:
It is what I want to do:
[Command("meme")]
public async Task meme(CommandContext commandInfo, int id = -1, SocketUserMessage userMsg, string emoteName)
{
var description = "";
MySqlClient.Connect();
if (id == -1)
{
var query = "SELECT * FROM memes;";
var memeReader = MySqlClient.GetDataReader(query);
if (memeReader == null) return;
while (memeReader.Read())
{
var memeid = memeReader.GetUInt64("id");
var title = memeReader.GetString("title");
description += $"**{memeid}** {title}\n";
}
}
var memeEmbed = new DiscordEmbedBuilder()
{
Color = DiscordColor.Gold,
Title = "**The Meme of The Year**",
Description = description
};
var msg = await commandInfo.Channel.SendMessageAsync(embed: memeEmbed);
}
Adding Reactions
Assuming you are using Discord.Net, you need to react to your own message using the AddReactionAsync method.
Adding Emojis
Emojis are unicode characters. To use them, pass the utf-code (e.g. "\uD83D\uDC4C") as an argument to the constructor of the Emoji` object.
await userMsg.AddReactionAsync(new Emoji("\U0001f643"));
Adding Emotes
An Emote is a 'custom Emoji' which can be set by server admins/mods.
As an argument you need to pass a custom Emote string, like this "<:thonkang:282745590985523200>".
A safe wrapper to catch non-existing emotes could look like that (taken from the docs).
public async Task ReactWithEmoteAsync(SocketUserMessage userMsg, string escapedEmote)
{
if (Emote.TryParse(escapedEmote, out var emote))
{
await userMsg.AddReactionAsync(emote);
}
}
https://docs.stillu.cc/guides/emoji/emoji.html
Counting Reactions
As a second step you need to listen to user-events, when they react to your post. This should be done in a separate event handler, as your poll will likely be up for several hours/days.
Here is an example on how to implement an event handler
You then need to keep track of the reactions and need to associate it with the matching database entry.
On a single-instance MongoDB server, even with the write concern on the client set to journaled, one in every couple of thousand documents isn't replacable immediately after inserting.
I was under the impression that once journaled, documents are immediately available for querying.
The code below inserts a document, then updates the DateModified property of the document and tries to update the document based on the document's Id and the old value of that property.
public class MyDocument
{
public BsonObjectId Id { get; set; }
public DateTime DateModified { get; set; }
}
static void Main(string[] args)
{
var r = Task.Run(MainAsync);
Console.WriteLine("Inserting documents... Press any key to exit.");
Console.ReadKey(intercept: true);
}
private static async Task MainAsync()
{
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("updateInsertedDocuments");
var concern = new WriteConcern(journal: true);
var collection = database.GetCollection<MyDocument>("docs").WithWriteConcern(concern);
int errorCount = 0;
int totalCount = 0;
do
{
totalCount++;
// Create and insert the document
var document = new MyDocument
{
DateModified = DateTime.Now,
};
await collection.InsertOneAsync(document);
// Save and update the modified date
var oldDateModified = document.DateModified;
document.DateModified = DateTime.Now;
// Try to update the document by Id and the earlier DateModified
var result = await collection.ReplaceOneAsync(d => d.Id == document.Id && d.DateModified == oldDateModified, document);
if (result.ModifiedCount == 0)
{
Console.WriteLine($"Error {++errorCount}/{totalCount}: doc {document.Id} did not have DateModified {oldDateModified.ToString("yyyy-MM-dd HH:mm:ss.ffffff")}");
await DoesItExist(collection, document, oldDateModified);
}
}
while (true);
}
The code inserts at a rate of around 250 documents per second. One in around 1,000-15,000 calls to ReplaceOneAsync(d => d.Id == document.Id && d.DateModified == oldDateModified, ...) fails, as it returns a ModifiedCount of 0. The failure rate depends on whether we run a Debug or Release build and with debugger attached or not: more speed means more errors.
The code shown represents something that I can't really easily change. Of course I'd rather perform a series of Update.Set() calls, but that's not really an option right now. The InsertOneAsync() followed by a ReplaceOneAsync() is abstracted by some kind of repository pattern that updates entities by reference. The non-async counterparts of the methods display the same behavior.
A simple Thread.Sleep(100) between inserting and replacing mitigates the problem.
When the query fails, and we wait a while and then attempt to query the document again in the code below, it'll be found every time.
private static async Task DoesItExist(IMongoCollection<MyDocument> collection, MyDocument document, DateTime oldDateModified)
{
Thread.Sleep(500);
var fromDatabaseCursor = await collection.FindAsync(d => d.Id == document.Id && d.DateModified == oldDateModified);
var fromDatabaseDoc = await fromDatabaseCursor.FirstOrDefaultAsync();
if (fromDatabaseDoc != null)
{
Console.WriteLine("But it was found!");
}
else
{
Console.WriteLine("And wasn't found!");
}
}
Versions on which this occurs:
MongoDB Community Server 3.4.0, 3.4.1, 3.4.3, 3.4.4 and 3.4.10, all on WiredTiger storage engine
Server runs on Windows, other OSes as well
C# Mongo Driver 2.3.0 and 2.4.4
Is this an issue in MongoDB, or are we doing (or assuming) something wrong?
Or, the actual end goal, how can I ensure an insert is immediately retrievable by an update?
ReplaceOneAsync returns 0 if the new document is identical to the old one (because nothing changed).
It looks to me like if your test executes fast enough the various calls to DateTime.Now could return the same value, so it is possible that you are passing the exact same document to InsertOneAsync and ReplaceOneAsync.
I'm developing a "Task Control System" that will allow its users to enter task description information including when to execute the task and what environment (OS, browser, etc.) the task requires.
The 'controller' saves the description information and schedules the task. When the scheduled time arrives, the scheduler retrieves the task information and 'queues' the task for a remote machine that matches the required environment.
My first cut at this used a relational database to persist the task descriptions and enough history information to track problems (about 2 weeks worth). But this is not a 'big data' problem and the relationships are simple and I need better performance.
So I'm looking for something that offers more performance.
I'm trying to use redis for this, but I'm having some problems. I'm using ServiceStack.Redis version 3.9.71.0 for the client and Redis 2.8.4 is the server.
This sample code is taken from Dan Swain's tutorial. It's updated to work with ServiceStack.Redis client v 3.9.71.0. Much of it works, but 'currentShippers.Remove(lameShipper);' does NOT work.
Can anyone see why that might be?
Thanks
public void ShippersUseCase()
{
using (var redisClient = new RedisClient("localhost"))
{
//Create a 'strongly-typed' API that makes all Redis Value operations to apply against Shippers
var redis = redisClient.As<Shipper>();
//Redis lists implement IList<T> while Redis sets implement ICollection<T>
var currentShippers = redis.Lists["urn:shippers:current"];
var prospectiveShippers = redis.Lists["urn:shippers:prospective"];
currentShippers.Add(
new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "Trains R Us",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.Trains,
UniqueRef = Guid.NewGuid()
});
currentShippers.Add(
new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "Planes R Us",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.Planes,
UniqueRef = Guid.NewGuid()
});
var lameShipper = new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "We do everything!",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.All,
UniqueRef = Guid.NewGuid()
};
currentShippers.Add(lameShipper);
Dump("ADDED 3 SHIPPERS:", currentShippers);
currentShippers.Remove(lameShipper);
.
.
.
}
}
Fixed the problem by adding these overrides to the 'Shipper' class:
public override bool Equals(object obj)
{
if (obj == null)
{
return false;
}
var input = obj as Shipper;
return input != null && Equals(input);
}
public bool Equals(Shipper other)
{
return other != null && (Id.Equals(other.Id));
}
public override int GetHashCode()
{
return (int)Id;
}
This working example shows how to implement List<>.Contains, List<>.Find, and List<>.Remove. Once applied to the 'Shipper' class the problem was solved!
We have an email queue table in the database. It holds the subject, HTML body, to address, from address etc.
In Global.asax every interval, the Process() function is called which despatches a set number of emails. Here's the code:
namespace v2.Email.Queue
{
public class Settings
{
// How often process() should be called in seconds
public const int PROCESS_BATCH_EVERY_SECONDS = 1;
// How many emails should be sent in each batch. Consult SES send rates.
public const int EMAILS_PER_BATCH = 20;
}
public class Functions
{
private static Object QueueLock = new Object();
/// <summary>
/// Process the queue
/// </summary>
public static void Process()
{
lock (QueueLock)
{
using (var db = new MainContext())
{
var emails = db.v2EmailQueues.OrderBy(c => c.ID).Take(Settings.EMAILS_PER_BATCH);
foreach (var email in emails)
{
var sent = Amazon.Emailer.SendEmail(email.FromAddress, email.ToAddress, email.Subject,
email.HTML);
if (sent)
db.ExecuteCommand("DELETE FROM v2EmailQueue WHERE ID = " + email.ID);
else
db.ExecuteCommand("UPDATE v2EmailQueue Set FailCount = FailCount + 1 WHERE ID = " + email.ID);
}
}
}
}
The problem is that every now and then it's sending one email twice.
Is there any reason from the code above that could explain this double sending?
Small test as per Matthews suggestion
const int testRecordID = 8296;
using (var db = new MainContext())
{
context.Response.Write(db.tblLogs.SingleOrDefault(c => c.ID == testRecordID) == null ? "Not Found\n\n" : "Found\n\n");
db.ExecuteCommand("DELETE FROM tblLogs WHERE ID = " + testRecordID);
context.Response.Write(db.tblLogs.SingleOrDefault(c => c.ID == testRecordID) == null ? "Not Found\n\n" : "Found\n\n");
}
using (var db = new MainContext())
{
context.Response.Write(db.tblLogs.SingleOrDefault(c => c.ID == testRecordID) == null ? "Not Found\n\n" : "Found\n\n");
}
Returns when there is a record:
Found
Found
Not Found
If I use this method to clear the context cache after the delete sql query it returns:
Found
Not Found
Not Found
However still not sure if it's the root cause of the problem though. I would of thought the locking would definitely stop double sends.
The issue that your having is due to the way Entity Framework does its internal cache.
In order to increase performance, Entity Framework will cache entities to avoid doing a database hit.
Entity Framework will update its cache when you are doing certain operations on DbSet.
Entity Framework does not understand that your "DELETE FROM ... WHERE ..." statement should invalidate the cache because EF is not an SQL engine (and does not know the meaning of the statement you wrote). Thus, to allow EF to do its job, you should use the DbSet methods that EF understands.
for (var email in db.v2EmailQueues.OrderBy(c => c.ID).Take(Settings.EMAILS_PER_BATCH))
{
// whatever your amazon code was...
if (sent)
{
db.v2EmailQueues.Remove(email);
}
else
{
email.FailCount++;
}
}
// this will update the database, and its internal cache.
db.SaveChanges();
On a side note, you should leverage the ORM as much as possible, not only will it save time debugging, it makes your code easier to understand.