Develop Reference

ReplaceOneAsync() immediately after InsertOneAsync() not always working, even when journaled

On a single-instance MongoDB server, even with the write concern on the client set to journaled, one in every couple of thousand documents isn't replacable immediately after inserting.
I was under the impression that once journaled, documents are immediately available for querying.
The code below inserts a document, then updates the DateModified property of the document and tries to update the document based on the document's Id and the old value of that property.
public class MyDocument
{
public BsonObjectId Id { get; set; }
public DateTime DateModified { get; set; }
}
static void Main(string[] args)
{
var r = Task.Run(MainAsync);
Console.WriteLine("Inserting documents... Press any key to exit.");
Console.ReadKey(intercept: true);
}
private static async Task MainAsync()
{
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("updateInsertedDocuments");
var concern = new WriteConcern(journal: true);
var collection = database.GetCollection<MyDocument>("docs").WithWriteConcern(concern);
int errorCount = 0;
int totalCount = 0;
do
{
totalCount++;
// Create and insert the document
var document = new MyDocument
{
DateModified = DateTime.Now,
};
await collection.InsertOneAsync(document);
// Save and update the modified date
var oldDateModified = document.DateModified;
document.DateModified = DateTime.Now;
// Try to update the document by Id and the earlier DateModified
var result = await collection.ReplaceOneAsync(d => d.Id == document.Id && d.DateModified == oldDateModified, document);
if (result.ModifiedCount == 0)
{
Console.WriteLine($"Error {++errorCount}/{totalCount}: doc {document.Id} did not have DateModified {oldDateModified.ToString("yyyy-MM-dd HH:mm:ss.ffffff")}");
await DoesItExist(collection, document, oldDateModified);
}
}
while (true);
}
The code inserts at a rate of around 250 documents per second. One in around 1,000-15,000 calls to ReplaceOneAsync(d => d.Id == document.Id && d.DateModified == oldDateModified, ...) fails, as it returns a ModifiedCount of 0. The failure rate depends on whether we run a Debug or Release build and with debugger attached or not: more speed means more errors.
The code shown represents something that I can't really easily change. Of course I'd rather perform a series of Update.Set() calls, but that's not really an option right now. The InsertOneAsync() followed by a ReplaceOneAsync() is abstracted by some kind of repository pattern that updates entities by reference. The non-async counterparts of the methods display the same behavior.
A simple Thread.Sleep(100) between inserting and replacing mitigates the problem.
When the query fails, and we wait a while and then attempt to query the document again in the code below, it'll be found every time.
private static async Task DoesItExist(IMongoCollection<MyDocument> collection, MyDocument document, DateTime oldDateModified)
{
Thread.Sleep(500);
var fromDatabaseCursor = await collection.FindAsync(d => d.Id == document.Id && d.DateModified == oldDateModified);
var fromDatabaseDoc = await fromDatabaseCursor.FirstOrDefaultAsync();
if (fromDatabaseDoc != null)
{
Console.WriteLine("But it was found!");
}
else
{
Console.WriteLine("And wasn't found!");
}
}
Versions on which this occurs:
MongoDB Community Server 3.4.0, 3.4.1, 3.4.3, 3.4.4 and 3.4.10, all on WiredTiger storage engine
Server runs on Windows, other OSes as well
C# Mongo Driver 2.3.0 and 2.4.4
Is this an issue in MongoDB, or are we doing (or assuming) something wrong?
Or, the actual end goal, how can I ensure an insert is immediately retrievable by an update?

ReplaceOneAsync returns 0 if the new document is identical to the old one (because nothing changed).
It looks to me like if your test executes fast enough the various calls to DateTime.Now could return the same value, so it is possible that you are passing the exact same document to InsertOneAsync and ReplaceOneAsync.

redis servicestack client List.Remove(item) does not work

I'm developing a "Task Control System" that will allow its users to enter task description information including when to execute the task and what environment (OS, browser, etc.) the task requires.
The 'controller' saves the description information and schedules the task. When the scheduled time arrives, the scheduler retrieves the task information and 'queues' the task for a remote machine that matches the required environment.
My first cut at this used a relational database to persist the task descriptions and enough history information to track problems (about 2 weeks worth). But this is not a 'big data' problem and the relationships are simple and I need better performance.
So I'm looking for something that offers more performance.
I'm trying to use redis for this, but I'm having some problems. I'm using ServiceStack.Redis version 3.9.71.0 for the client and Redis 2.8.4 is the server.
This sample code is taken from Dan Swain's tutorial. It's updated to work with ServiceStack.Redis client v 3.9.71.0. Much of it works, but 'currentShippers.Remove(lameShipper);' does NOT work.
Can anyone see why that might be?
Thanks
public void ShippersUseCase()
{
using (var redisClient = new RedisClient("localhost"))
{
//Create a 'strongly-typed' API that makes all Redis Value operations to apply against Shippers
var redis = redisClient.As<Shipper>();
//Redis lists implement IList<T> while Redis sets implement ICollection<T>
var currentShippers = redis.Lists["urn:shippers:current"];
var prospectiveShippers = redis.Lists["urn:shippers:prospective"];
currentShippers.Add(
new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "Trains R Us",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.Trains,
UniqueRef = Guid.NewGuid()
});
currentShippers.Add(
new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "Planes R Us",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.Planes,
UniqueRef = Guid.NewGuid()
});
var lameShipper = new Shipper
{
Id = redis.GetNextSequence(),
CompanyName = "We do everything!",
DateCreated = DateTime.UtcNow,
ShipperType = ShipperType.All,
UniqueRef = Guid.NewGuid()
};
currentShippers.Add(lameShipper);
Dump("ADDED 3 SHIPPERS:", currentShippers);
currentShippers.Remove(lameShipper);
.
.
.
}
}
Fixed the problem by adding these overrides to the 'Shipper' class:
public override bool Equals(object obj)
{
if (obj == null)
{
return false;
}
var input = obj as Shipper;
return input != null && Equals(input);
}
public bool Equals(Shipper other)
{
return other != null && (Id.Equals(other.Id));
}
public override int GetHashCode()
{
return (int)Id;
}

This working example shows how to implement List<>.Contains, List<>.Find, and List<>.Remove. Once applied to the 'Shipper' class the problem was solved!

Entity Framework Add Performance Degrades With More Rows

I'll begin this post by noting that I'm entirely new to the .NET world. ASP, EntityFramework, Linq, etc. are all mostly unknown magic at this point.
Having said that, I've built myself a neat Web API chat-like application with SignalR support for real-time events. It works quite well, but I'm having some performance problems with the Add function.
In my chat application, there are "Pads" (chat rooms) which contain a number of "Mates" and "Messages". Here's my Pad model for reference:
public class Pad
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public Guid PadId { get; set; }
public string StreetAddress { get; set; }
public int ZipCode { get; set; }
public virtual ICollection<Mate> Mates { get; set; }
public virtual ICollection<Message> Messages { get; set; }
}
My problem lies in my SignalR hub that processes a new Message sent to a particular pad. These two lines take about half a second to process.
pad.Messages.Add(msg); // pad is the Pad entity already fetched from the db context
db.Messages.Add(msg);
But they only take that long when Pad.Messages contains a large number of messages. Thousands. If I am sending to a pad with few to no messages, it executes almost instantly.
My initial 'trick' to improve the perceived performance here is to move the adding functions to after I send the notification back to the clients, but I realize something like this could present a potential problem later when there are tens or hundreds of thousands of messages in one pad.
Any advice here would be greatly appreciated!
Here is the entire message send method for reference:
public void SendMessage(string pad_id, string body)
{
var user_id = IdentityExtensions.GetUserId(Context.User.Identity);
body = body.Trim();
if (body.Length <= 0)
{
return;
}
// Check that the user belongs in this pad...
var user = (from u in db.Users
where u.Id == user_id
select u).First();
var pad = (from p in user.Pads where p.PadId == new Guid(pad_id) select p).FirstOrDefault();
if (pad != null) {
// Save the message to the database
var msg = new Message()
{
MessageId = new Guid(),
Author = user,
Body = body,
SendTime = DateTimeOffset.UtcNow,
Pad = pad
};
pad.Messages.Add(msg); // These two lines
db.Messages.Add(msg); // Are the culprit.
db.SaveChangesAsync();
Clients.Group(pad_id).messageReceived(user.Id, pad_id, body, DateTimeOffset.UtcNow); // Send message to clients
}
}
EDIT: I'm on EF 6.0.0

From your code, all you want to do is add 1 row (Message) to the table.
While constructing the Message entity, just use the padid instead of the Pad object.
That way you don't need to deal with all the pad objects. From a DB perspective, you just need to add a Message row with a PadId key.
var msg = new Message()
{
MessageId = new Guid(),
Author = user,
Body = body,
SendTime = DateTimeOffset.UtcNow,
PadId = new Guid(pad_id)
};
// pad.Messages.Add(msg); // don't need this.
db.Messages.Add(msg);
The above should always insert one row and not depend on the number of messages in the pad.
If it still gives a performance problem, then adding a single row is causing a lot of index updates to your table.

Avoiding the new upsert in Azure Table Storage

Steve Marx writes about new extension methods to perform upserts in Azure Table Storage as part of the new storage protocol version here:
http://blog.smarx.com/posts/extension-methods-for-the-august-storage-features
However, what if I want to do the original operation of unconditional-merge-or-throw, rather than an upsert. I want to merge an object, updating a single field, but throw if the entity doesn't exist rather than create a new entity that contains only the properties I'm merging.
Is this possible? Note that I want to use upsert elsewhere, so I've taken to having IoC provide me with contexts created from GetDataServiceContext2011 instead of GetDataServiceContext. I suppose I could alternate between the two, but that won't help when the Azure team updates the official libraries.
According to MSDN:
The Insert Or Merge Entity operation uses the MERGE verb and must be
called using the 2011-08-18 version or newer. In addition, it does not
use the If-Match header. These attributes distinguish this operation
from the Update Entity operation, though the request body is the same
for both operations.
So, how do I get the storage library to emit a wildcard If-Match on save rather than emit no If-Match at all?

Just use AttachTo with an asterisk for an etag. That will result in an If-Match: *. Here's a complete working example:
class Entity : TableServiceEntity
{
public string Text { get; set; }
public Entity() { }
public Entity(string rowkey) : base(string.Empty, rowkey) { }
}
class Program
{
static void Update(CloudStorageAccount account)
{
var ctx = account.CreateCloudTableClient().GetDataServiceContext();
var entity = new Entity("foo") { Text = "bar" };
ctx.AttachTo("testtable", entity, "*");
ctx.UpdateObject(entity);
ctx.SaveChangesWithRetries();
}
static void Main(string[] args)
{
var account = CloudStorageAccount.Parse(args[0]);
var tables = account.CreateCloudTableClient();
tables.CreateTableIfNotExist("testtable");
var ctx = tables.GetDataServiceContext();
try { Update(account); } catch (Exception e) { Console.WriteLine("Exception (as expected): " + e.Message); }
ctx.AddObject("testtable", new Entity("foo") { Text = "foo" });
ctx.SaveChangesWithRetries();
try { Update(account); } catch (Exception e) { Console.WriteLine("Unexpected exception: " + e.Message); }
Console.WriteLine("Now text is: " + tables.GetDataServiceContext().CreateQuery<Entity>("testtable").Where(e => e.PartitionKey == string.Empty && e.RowKey == "foo").Single().Text);
tables.DeleteTableIfExist("testtable");
}
}

How to access entity's properties outside context using Entity Framework?

I'm new to Entity Framework (working mostly with NHibernate with ActiveRecord before) and I'm stuck with something, that I think should be easy...
I have a User Entity, and created partial User class so I can add some methods (like with NHibernate). I added GetByID to make getting user easier:
public static User GetByID(int userID)
{
using (var context = new MyEntities())
{
return context.Users.Where(qq => qq.UserID == userID).Single();
}
}
Now in the same class I want to log moment of logging in, and I try to do:
public static void LogLoginInfo(int userID)
{
using (var context = new MyEntities())
{
var user = User.GetByID(userID);
var log = new LoginLog { Date = DateTime.Now };
user.LoginLogs.Add(log);
context.SaveChanges();
}
}
The problem is I can't access user.LoginLogs because user's context is already disposed... Most likely I'm missing something obvious here, but creating always full queries like:
context.Users.Where(qq => qq.UserID == userID).Single().LoginLogs.Add(log);
doesn't seem like a good option...
I've read about Repository pattern but I think it's too big gun for such task. Please explain me what am I doing wrong. Thanks in advance!
EDIT
To picture what I'd like to do:
//somewhere in business logic
var user = User.GetByID(userID);
var posts = user.GetAllPostsForThisMonth();
foreach(var post in posts)
{
Console.WriteLine(post.Answers.Count);
}
Normally I'm not allowed to do this because I can't get post.Answers without context...

You are closing the object context and then trying to add a log to the user that is detached. You need to attach the user so the objectContext know what has been changed or added.
public static void LogLoginInfo(int userID)
{
using (var context = new MyEntities())
{
var user = context.User.Where(p=>p.UserID == userID); //<= The Context now knows about the User, and can track changes.
var log = new LoginLog { Date = DateTime.Now };
user.LoginLogs.Add(log);
context.SaveChanges();
}
}
Update
You can also attach the object.
public static void LogLoginInfo(int userID)
{
using (var context = new MyEntities())
{
var user = User.GetByID(userID);
var log = new LoginLog { Date = DateTime.Now };
user.LoginLogs.Add(log);
context.User.Attach(user);
context.SaveChanges();
}
}
Update
var getFirstLogin = from p in User.GetUserById(userId)
select p.LoginLogs.FirstOrDefault();
NB if LoginLogs is a different table you will need to use Include.
public static User GetByID(int userID)
{
using (var context = new MyEntities())
{
return context.Users.Include("LoginLogs").Where(qq => qq.UserID == userID).FirstOrDefault();
}
}

If you are open to using stored procedures (and they work nicely with EF), you can return the user object and simultaneously add to the log table with a single call to the database.
I used to do everything with SP's in my pre-EF/ORM days, when I went to EF I tried very hard to avoid using stored procedures to avoid falling back into my old habits, but now I have found that the selective use of stored procedures you can have the benefits of both -the EF way of doing things, and the super functionality/performance that a well written SP can provide.