Steve Marx writes about new extension methods to perform upserts in Azure Table Storage as part of the new storage protocol version here:
http://blog.smarx.com/posts/extension-methods-for-the-august-storage-features
However, what if I want to do the original operation of unconditional-merge-or-throw, rather than an upsert. I want to merge an object, updating a single field, but throw if the entity doesn't exist rather than create a new entity that contains only the properties I'm merging.
Is this possible? Note that I want to use upsert elsewhere, so I've taken to having IoC provide me with contexts created from GetDataServiceContext2011 instead of GetDataServiceContext. I suppose I could alternate between the two, but that won't help when the Azure team updates the official libraries.
According to MSDN:
The Insert Or Merge Entity operation uses the MERGE verb and must be
called using the 2011-08-18 version or newer. In addition, it does not
use the If-Match header. These attributes distinguish this operation
from the Update Entity operation, though the request body is the same
for both operations.
So, how do I get the storage library to emit a wildcard If-Match on save rather than emit no If-Match at all?
Just use AttachTo with an asterisk for an etag. That will result in an If-Match: *. Here's a complete working example:
class Entity : TableServiceEntity
{
public string Text { get; set; }
public Entity() { }
public Entity(string rowkey) : base(string.Empty, rowkey) { }
}
class Program
{
static void Update(CloudStorageAccount account)
{
var ctx = account.CreateCloudTableClient().GetDataServiceContext();
var entity = new Entity("foo") { Text = "bar" };
ctx.AttachTo("testtable", entity, "*");
ctx.UpdateObject(entity);
ctx.SaveChangesWithRetries();
}
static void Main(string[] args)
{
var account = CloudStorageAccount.Parse(args[0]);
var tables = account.CreateCloudTableClient();
tables.CreateTableIfNotExist("testtable");
var ctx = tables.GetDataServiceContext();
try { Update(account); } catch (Exception e) { Console.WriteLine("Exception (as expected): " + e.Message); }
ctx.AddObject("testtable", new Entity("foo") { Text = "foo" });
ctx.SaveChangesWithRetries();
try { Update(account); } catch (Exception e) { Console.WriteLine("Unexpected exception: " + e.Message); }
Console.WriteLine("Now text is: " + tables.GetDataServiceContext().CreateQuery<Entity>("testtable").Where(e => e.PartitionKey == string.Empty && e.RowKey == "foo").Single().Text);
tables.DeleteTableIfExist("testtable");
}
}
Related
We have an email queue table in the database. It holds the subject, HTML body, to address, from address etc.
In Global.asax every interval, the Process() function is called which despatches a set number of emails. Here's the code:
namespace v2.Email.Queue
{
public class Settings
{
// How often process() should be called in seconds
public const int PROCESS_BATCH_EVERY_SECONDS = 1;
// How many emails should be sent in each batch. Consult SES send rates.
public const int EMAILS_PER_BATCH = 20;
}
public class Functions
{
private static Object QueueLock = new Object();
/// <summary>
/// Process the queue
/// </summary>
public static void Process()
{
lock (QueueLock)
{
using (var db = new MainContext())
{
var emails = db.v2EmailQueues.OrderBy(c => c.ID).Take(Settings.EMAILS_PER_BATCH);
foreach (var email in emails)
{
var sent = Amazon.Emailer.SendEmail(email.FromAddress, email.ToAddress, email.Subject,
email.HTML);
if (sent)
db.ExecuteCommand("DELETE FROM v2EmailQueue WHERE ID = " + email.ID);
else
db.ExecuteCommand("UPDATE v2EmailQueue Set FailCount = FailCount + 1 WHERE ID = " + email.ID);
}
}
}
}
The problem is that every now and then it's sending one email twice.
Is there any reason from the code above that could explain this double sending?
Small test as per Matthews suggestion
const int testRecordID = 8296;
using (var db = new MainContext())
{
context.Response.Write(db.tblLogs.SingleOrDefault(c => c.ID == testRecordID) == null ? "Not Found\n\n" : "Found\n\n");
db.ExecuteCommand("DELETE FROM tblLogs WHERE ID = " + testRecordID);
context.Response.Write(db.tblLogs.SingleOrDefault(c => c.ID == testRecordID) == null ? "Not Found\n\n" : "Found\n\n");
}
using (var db = new MainContext())
{
context.Response.Write(db.tblLogs.SingleOrDefault(c => c.ID == testRecordID) == null ? "Not Found\n\n" : "Found\n\n");
}
Returns when there is a record:
Found
Found
Not Found
If I use this method to clear the context cache after the delete sql query it returns:
Found
Not Found
Not Found
However still not sure if it's the root cause of the problem though. I would of thought the locking would definitely stop double sends.
The issue that your having is due to the way Entity Framework does its internal cache.
In order to increase performance, Entity Framework will cache entities to avoid doing a database hit.
Entity Framework will update its cache when you are doing certain operations on DbSet.
Entity Framework does not understand that your "DELETE FROM ... WHERE ..." statement should invalidate the cache because EF is not an SQL engine (and does not know the meaning of the statement you wrote). Thus, to allow EF to do its job, you should use the DbSet methods that EF understands.
for (var email in db.v2EmailQueues.OrderBy(c => c.ID).Take(Settings.EMAILS_PER_BATCH))
{
// whatever your amazon code was...
if (sent)
{
db.v2EmailQueues.Remove(email);
}
else
{
email.FailCount++;
}
}
// this will update the database, and its internal cache.
db.SaveChanges();
On a side note, you should leverage the ORM as much as possible, not only will it save time debugging, it makes your code easier to understand.
Trying to unit test functions which access a entity framework. So i tried to put all the entity code into the test function below? However it stops at the Linq statement; obviously trying to access the database is too much drama for it. Maybe a work around would be too to create a replica database within the unit test function based on sql lite or compact;(Its not a big database anyways) then execution would not have to leave the test function? Is this possible and how would i implement it?
public void RetreiveKeyFnTest()
{
StegApp target = new StegApp(); // TODO: Initialize to an appropriate value
string username = "david"; // TODO: Initialize to an appropriate value
string password = "david1"; // TODO: Initialize to an appropriate value
string ConnectionString = ConfigurationManager.ConnectionStrings["DatabaseEntities"].ToString();
var dataContext = new DatabaseEntities(ConnectionString);
var user = dataContext.Users.FirstOrDefault(u => u.Username.Equals(username) && u.Password.Equals(password));
Assert.IsNotNull(user);
//target.RetreiveKeyFn(username, password);
//Assert.IsInstanceOfType(target.RetreiveLogs,typeof(DataAccess));
//Assert.IsInstanceOfType(target.p);
//Assert.IsNotNull(target.RetreiveLogs.AuthenitcateCredentials(username,password));
//Assert.Inconclusive("A method that does not return a value cannot be verified.");
}
Below is the code i am trying to test:
public void RetreiveKeyFn(string username, string password)
{
BusinessObjects.User p = RetreiveLogs.AuthenitcateCredentials(username,password);
if (p != null)
{
if (RetreiveLogs.RetreiveMessages(p.UserId) == null)
{
DisplayLogs.Text = "Sorry No messages for you recorded in Database, your correspondant might have chose not to record the entry";
}
else
{
MessageBox.Show("LogId = " + RetreiveLogs.RetreiveMessages(p.UserId).LogId + "\n" +
"UserId = " + RetreiveLogs.RetreiveMessages(p.UserId).UserId + "\n" +
"Message Key = " + RetreiveLogs.RetreiveMessages(p.UserId).MessageKey + "\n" + "PictureId = " + RetreiveLogs.RetreiveMessages(p.UserId).PictureId +
" Date & time = " + RetreiveLogs.RetreiveMessages(p.UserId).SentDateTime);
DisplayLogs.Visible = true;
}
}
else
{
MessageBox.Show("Please enter your correct username and password in order to retreive either key, image or both from Databse");
}
}
First, you should be able to access the same database in your test application as the one you're using in your main/actual application. You just need to make sure that your Test project contains your connection string in its own App.config.
The initialization of the context should be done either inside your StegApp(), or you should be able to pass a context to your StegApp() from a different scope. From what I read of your code, your StegApp() will not be able to access the dataContext variable you created.
Your test for null user already happens inside the RetrieveKeyFn() under the AuthenticateCredentials() method so there's no need for the first "Assert.IsNotNull(user)". I would recommend separating your business logic for RetrieveKeyFn from your UI behaviors so that you can easily do unit tests. You can bind the "Messagebox" operations to say a button click event handler which calls just RetrieveKeyFn(). I would suggest maybe something like this:
public class StegApp
{
public DatabaseEntities context;
//other properties
public StegApp()
{
//assuming your DatabaseEntities class inherits from DbContext.
//You should create other constructors that allow you to set options
//like lazy loading and mappings
this.context = new DatabaseEntities();
}
//ASSUMING YOUR RetrieveLogs.RetrieveMessages() function returns
//a Message object. replace this type with whatever type the
//RetrieveLogs.RetrieveMessages() method returns.
public Message RetrieveKeyFn (string username, string password)
{
BusinessObjects.User p = RetreiveLogs.AuthenitcateCredentials(username,password);
if (p != null)
{
var message = RetrieveLogs.RetrieveMessages(p.UserId);
if (message == null)
// handle behavior for no messages. In this case
// I will just create a new Message object with a -1 LogId
return new Message {LogId =-1};
else
return message;
}
else
//handle behavior when the user is not authenticated.
//In this case I throw an exception
throw new Exception();
}
//on your button click handler, do something like:
// try
// {
// var message = RetrieveKeyFn(txtUsername.Text.Trim(), txtPassword.Text.Trim());
// if (message.LogId == -1)
// DisplayLogs.Text = "Sorry No messages for you recorded in Database, your correspondant might have chose not to record the entry";
// else
// {
// MessageBox.Show("Log Id = " + message.LogId)
// etc. etc. etc.
// }
// }
// catch
// {
// MessageBox.Show ("user is not authenticated");
// }
}
When you do your unit test, remember to have the appropriate configuration strings in your test project's App.Config If the app.config does not yet exist, go ahead and create one. You should create tests for all possibilities (i.e. 1) user is valid, you get the message, 2) user is valid, there are no messages, 3) user is invalid).
Here's an example for case 2
[TestMethod]
public void RetrieveKeyFnTest1()
{
StegApp target = new StegApp(); // this creates your context. I'm assuming it also creates your RetrieveLogs object, etc
var username = "UserWithNotMessages"; //this user should exist in your database but should not have any messages. You could insert this user as part of your TestInitialize method
var password = "UserWithNotMessagesPassword"; //this should be the proper password
var message = target.RetrieveKeyFn(username, password);
Assert.AreEqual (-1, message.LogId);
}
I got my unit tests to work fine. The mistake i had was not to copy the app.config file into the test project! Although to be honest i expected Visual studio would have done that anyways.
When i try to send an instance of MULTIMEDIA type, with
hasStream="true"
property set to true, the WCF Data Server seems not to receive entity data.
On the client side i iterate over a collection of objects and i try to send them to another wcf data service. The reference to the "other wcf data service" is:
this.centralCtx
Also i set the saved stream for the each new entity and initialize all properties copying them from the source entity:
foreach (LOCAL_TYPE localObject in localObjects)
{
if (entityName == "MULTIMEDIA")
{
CentralService.ARTICOLI article = null;
CentralService.MULTIMEDIA multimedia = new CentralService.MULTIMEDIA();
LocalService.MULTIMEDIA lMultimedia = localObject as LocalService.MULTIMEDIA;
multimedia.ID_MULTIMEDIA = lMultimedia.ID_MULTIMEDIA;
multimedia.DATA_CREAZIONE = lMultimedia.DATA_CREAZIONE;
multimedia.DATA_ULTIMA_MODIFICA = lMultimedia.DATA_ULTIMA_MODIFICA;
multimedia.ARTICOLO_ID = lMultimedia.ARTICOLO_ID;
this.centralCtx.TryGetEntity(
new Uri(this.centralCtx.BaseUri + "ARTICOLI('" + multimedia.ARTICOLO_ID
+ "')", UriKind.Absolute), out article);
article.MULTIMEDIA.Add(multimedia);
this.centralCtx.AddRelatedObject(article, "MULTIMEDIA", multimedia);
DataServiceStreamResponse streamResponse = this.localCtx.GetReadStream(localObject);
this.centralCtx.SetSaveStream(multimedia, streamResponse.Stream,
true, "image/jpeg", "");
//this.centralCtx.UpdateObject(article);
}
else {
CENTRAL_TYPE cloned = DbHelper.FlatCloneFromType<LOCAL_TYPE, CENTRAL_TYPE>
(localObject, centralCtx);
this.centralCtx.AddObject(entityName, cloned);
}
}
try
{
this.centralCtx.SaveChanges();
Notify(progressAction, "Exported table " + entityName, null);
successAction(this.Log);
}
catch (Exception ex)
{
Notify(progressAction, "Error exporting table " + entityName, ex);
this.synchResult = SynchResultType.Error;
exceptionAction(ex);
}
This is change interceptor code:
[ChangeInterceptor("MULTIMEDIA")]
public void OnChangeMultimedia(MULTIMEDIA changedObject, UpdateOperations op)
{
switch (op)
{
case UpdateOperations.Add:
if(changedObject.ID_MULTIMEDIA == null)
changedObject.ID_MULTIMEDIA = Guid.NewGuid().ToString();
changedObject.STATO_INTERNO = "TRASFERITO";
changedObject.DATA_ULTIMA_MODIFICA = changedObject.DATA_ULTIMA_MODIFICA == null
? DateTime.Now.ToLocalTime() : changedObject.DATA_ULTIMA_MODIFICA;
this.CurrentDataSource.SaveChanges();
break;
default: break;
}
}
All the properties of changedObject on the server inside MULTIMEDIA change interceptor are always null. Why?
I finally got the answer.
The sending of an entity marked with the attribute hasStream implies two requests.
The first is a POST during which the server creates a record in the database and save the file on the file system.
The second is a MERGE during which the client performs an update to the ID passed from the server
That's why during the first request to the server all the properties of the object are null.
The changes to the database were committed successfully, but an error occurred while updating the object context. The ObjectContext might be in an inconsistent state. Inner exception message: AcceptChanges cannot continue because the object's key values conflict with another object in the ObjectStateManager. Make sure that the key values are unique before calling AcceptChanges.
Is the error message i get. here are the two functions i use...
public IList<string> GenerateVersions(decimal id, decimal fId, string folderName, string filename, string objFile)
{
List<string> generatedFiles = new List<string>();
foreach (var tCmdSets in db.IMG_SETTINGS_CMDSETS.Where("it.SETTINGS_FOLDER_ID = #folderid", new ObjectParameter("folderid", id)))
{
var strDestinationPath = ImageResizer.Util.PathUtils.RemoveExtension(Path.Combine(tmpDefaultFolder, tCmdSets.SETTINGS_CMDSET_DESTINATION, filename));
ResizeSettings objResizeCommand = new ResizeSettings(tCmdSets.SETTINGS_CMDSET_COMMAND);
var strCreatedFile = ImageBuilder.Current.Build(objFile, strDestinationPath, objResizeCommand, false, true);
generatedFiles.Add("### File created: (" + folderName + " » " + tCmdSets.SETTINGS_CMDSET_NAME + " ») " + Path.GetFileName(strCreatedFile));
IMG_UPLOAD_GENERATED_FILES tObjGenerated = new IMG_UPLOAD_GENERATED_FILES();
tObjGenerated.UPLOAD_GENERATED_FILE_NAME = Path.GetFileName(strCreatedFile);
tObjGenerated.UPLOAD_GENERATED_PATH = Path.GetDirectoryName(strCreatedFile);
tObjGenerated.SETTINGS_CMDSET_ID = tCmdSets.SETTINGS_CMDSET_ID;
tObjGenerated.UPLOAD_FILE_ID = fId;
dbHandler.IMG_UPLOAD_GENERATED_FILES.AddObject(tObjGenerated);
dbHandler.SaveChanges();
}
return generatedFiles;
}
public ActionResult UploadBulkFiles(decimal id)
{
IMG_SETTINGS_FOLDERS img_settings_folders = db.IMG_SETTINGS_FOLDERS.Single(i => i.SETTINGS_FOLDER_ID == id);
string strBulkDirectory = Path.Combine(tmpDefaultFolder, img_settings_folders.SETTINGS_FOLDER_BULK);
string[] objFiles = Directory.GetFiles(strBulkDirectory);
List<string> lstOuput = new List<string>();
foreach (var tFile in objFiles)
{
System.IO.File.Move(tFile, Path.Combine(tmpDefaultFolder, "masters", img_settings_folders.SETTINGS_FOLDER_NAME, Path.GetFileName(tFile)));
lstOuput.Add("### File moved to masters (" + img_settings_folders.SETTINGS_FOLDER_NAME + " ») " + Path.GetFileName(tFile));
IMG_UPLOAD_FILES tObjUploadedFile = new IMG_UPLOAD_FILES();
tObjUploadedFile.UPLOAD_FILE_NAME = Path.GetFileName(tFile);
tObjUploadedFile.SETTINGS_FOLDER_ID = img_settings_folders.SETTINGS_FOLDER_ID;
dbHandler.IMG_UPLOAD_FILES.AddObject(tObjUploadedFile);
dbHandler.SaveChanges();
var objGeneratedFiles = GenerateVersions(img_settings_folders.SETTINGS_FOLDER_ID,tObjUploadedFile.UPLOAD_FILE_ID, img_settings_folders.SETTINGS_FOLDER_NAME, Path.GetFileName(tFile), Path.Combine(tmpDefaultFolder, "masters", img_settings_folders.SETTINGS_FOLDER_NAME, Path.GetFileName(tFile)));
lstOuput.AddRange(objGeneratedFiles);
}
if (lstOuput.Count > 0)
{
return PartialView("UploadSingleFile", lstOuput);
}
else
{
return PartialView("NoUploads");
}
}
DATA MODEL
IMG_UPLOAD_FILE
UPLOAD_FILE_ID (PK)
UPLOAD_FILE_NAME
SETTINGS_FOLDER_ID
IMG_UPLOAD_GENERATED_FILES
UPLOAD_GENERATED_FILE_ID (PK)
UPLOAD_GENERATED_FILE_NAME
UPLOAD_GENERATED_FILE_PATH
SETTINGS_CMDSET_ID
UPLOAD_FILE_ID
I had the exact same scenario with Entity Model based on Oracle database. The implementation of Identity is done by trigger so when adding the tables to the model it does not set the StoreGenertedPattern property of the identity column to Identity since it doens't aware that this column is identity.
There is a need to open model editor, locate the entity in the model, click on the key column and set the StoreGenertedPattern property to 'Identity' manually.
The closest I can come to finding an answer is:
Because Oracle uses a Sequence + Trigger to make "Auto Ident" values, it seems like when the entity framework adds an object at saves it, the value return is still 0, because the trigger/sequence haven't updated it yet.
Because of the 0 number, the ObjectMannager will think that multiple objects with the entity key of 0 are in conflict.
I don't have a "bullet proof" solutions, but have rewritten my solutions to handle it another way.
\T
This might not be related to your problem but I was getting this problem on a web page with an ajax manager running until I did this:
...
private static _objectContext;
protected void Page_Init(object sender, EventArgs e)
{
_objectContext = new ObjectContext();
}
...
protected void _ContextCreating(object sender, EntityDataSourceContextCreatingEventArgs e)
{
e.Context = _objectContext;
}
protected void _ContextDisposing(object sender, EntityDataSourceContextDisposingEventArgs e)
{
e.Cancel = true;
}
Creating the ObjectContext in Page_Load when not a postback caused that very exception for me.
I'm tryring to do a simple insert with foreign key, but it seems that I need to use db.SaveChanges() for every record insert. How can I manage to use only one db.SaveChanges() at the end of this program?
public static void Test()
{
using (var entities = new DBEntities())
{
var sale =
new SalesFeed
{
SaleName = "Stuff...",
};
entities.AddToSalesFeedSet(sale);
var phone =
new CustomerPhone
{
CreationDate = DateTime.UtcNow,
sales_feeds = sale
};
entities.AddToCustomerPhoneSet(phone);
entities.SaveChanges();
}
}
After running the above code I get this exception:
System.Data.UpdateException: An error occurred while updating the entries. See the InnerException for details. The specified value is not an instance of a valid constant type
Parameter name: value.
EDIT: Changed example code and added returned exception.
Apperantly using UNSIGNED BIGINT causes this problem. When I switched to SIGNED BIGINT everything worked as it supposed to.
I tried to do this "the right way":
And then I wrote this little test app to scan a directory, store the directory and all its files in two tables:
static void Main(string[] args)
{
string directoryName = args[0];
if(!Directory.Exists(directoryName))
{
Console.WriteLine("ERROR: Directory '{0}' does not exist!", directoryName);
return;
}
using (testEntities entities = new testEntities())
{
StoredDir dir = new StoredDir{ DirName = directoryName };
entities.AddToStoredDirSet(dir);
foreach (string filename in Directory.GetFiles(directoryName))
{
StoredFile stFile = new StoredFile { FileName = Path.GetFileName(filename), Directory = dir };
entities.AddToStoredFileSet(stFile);
}
try
{
entities.SaveChanges();
}
catch(Exception exc)
{
string message = exc.GetType().FullName + ": " + exc.Message;
}
}
}
As you can see, I only have a single call to .SaveChanges() at the very end - this works like a charm, everything's as expected.
Something about your approach must be screwing up the EF system.....
it might be related with the implementation of AddToSalesFeedSet etc..
there is chance that you are doing commit inside ?
any way, my point is that i encountered very close problem, was tring to add relation to new entity with existed entity that been queried earlier - that has unsigned key
and got the same exception;
the solution was to call Db.collection.Attach(previouslyQueriedEntityInstance);