I'm working on an application using an Ignite.Net cache infront of an Oracle database.
I read that I can write to multiple caches at once safely using Ignite Transactions (https://apacheignite-net.readme.io/v1.5/docs/transactions#Cross-cache transactions).
I also read that each cache can have it's own CacheStore that writes to the underlying database but I've yet to find any documentation that explains how I should implement the CacheStore classes so the database writes are safe across the whole Ignite transaction.
I've seen information on SessionEnd and CacheStoreSession (https://apacheignite-net.readme.io/v2.6/docs/persistent-store#section-sessionend-) but these don't mention multiple CacheStores.
The following article explains how transactions are handled for 3rd party persistence but this again only talks of a single Cache/CacheStore (https://www.gridgain.com/resources/blog/apache-ignite-transactions-architecture-transaction-handling-level-3rd-party)
Can anyone advise how this works (assuming it does) or point me to further examples/documentation?
For a definitive answer (appreciate your time #alamar), I've spoken with one of the nice people at Gridgain and can confirm it is possible to safely perform transactions across multiple CacheStores, where all stores write to the same database without data inconsistency. It's not done via a mechanism specifically coded into Ignite as I had wondered about but can be implemented safely via a simple shared database connection.
For this to work you need to:
Make your caches transactional (AtomicityMode = CacheAtomicityMode.Transactional, WriteThrough = true)
Share a single database connection between the data stores (either inject via the CacheStoreFactory or use a singleton)
In all write operations on the CacheStores, write to the shared session database but do not commit. Mark the session as requiring a commit (your own boolean flag).
Implement SessionEnd (https://apacheignite-net.readme.io/docs/persistent-store#section-sessionend-) in each of your CacheStores. The implementation should call commit on your shared database connection if it has not already been called (check the boolean flag from the step before and reset after commit). You could always encapsulate that logic in your database connection class.
A simplified code example:
public class SharedDatabaseSession
{
private bool commitRequired;
private DatabaseConnection databaseConnection;
// ....
public void Write( /*xyz*/)
{
databaseConnection.Write( /*xyz*/);
commitRequired = true;
}
public void Commit()
{
if (commitRequired)
{
databaseConnection.Commit();
commitRequired = false;
}
}
public static SharedDatabaseSession GetInstance()
{
return instance;
}
}
public class FirstCacheStore : CacheStoreAdapter<int, int>
{
private SharedDatabaseSession database = SharedDatabaseSession.GetInstance();
/// ......
public override void Write(int key, int val)
{
database.Write( /*xyz*/);
}
public override void SessionEnd(bool commit)
{
if (commit)
{
database.Commit();
}
}
}
public class SecondCacheStore : CacheStoreAdapter<int, int>
{
private SharedDatabaseSession database = SharedDatabaseSession.GetInstance();
/// ......
public override void Write(int key, int val)
{
database.Write( /*xyz*/);
}
public override void SessionEnd(bool commit)
{
if (commit)
{
database.Commit();
}
}
}
Have you tried it?
My expectation, it should technically be supported, but since Cache Store uses two phase commit, and multiple Cache Stores will need to use "three phase commit", and there's no such thing - that you can expect data inconsistency on edge cases.
Happy path should, however, work OK.
Related
I am working on a test suit implementation which uses the SpecFlow + SpecRunner and XUnit. and we are trying to do parallel test execution and i would like to know is there are a way that i can run a hook in the begining of the test run and store the token value in a static variable so that that can be shared among threads.
to summarize is there a way that specflow offers a mechanism to share data between threads during parallel execution.
We can share the data using any one of the below approach
Scenario Context
Context Injection
Here, Approach 1 and 2 will not have any issue in multiple thread. Since, Context Injection life is specific to the scenario Level.
Approach 1 : we can define the Token Generation Step within the BeforeScenario hooks and the generated Token values can be updated in the ScenarioContext.
we can directly access the token from the scenario context in any place like below
Here, Token will be generated before each scenario run and it will not affect the Parallel execution.For more Details, Parallel-Execution
Scenarios and their related hooks (Before/After scenario, scenario block, step) are isolated in the different threads during execution and do not block each other. Each thread has a separate (and isolated) ScenarioContext.
Hooks Class:
public class CommonHooks
{
[BeforeScenario]
public static void Setup()
{
// Add Token Generation Step
var adminToken = "<Generated Token>";
ScenarioContext.Current["Token"] = adminToken;
}
}
Step Class:
[Given(#"I Get the customer details""(.*)""")]
public void WhenIGetTheCustomerDetails(string endpoint)
{
if(ScenarioContext.Current.ContainsKey("Token"))
{
var token = ScenarioContext.Current["Token"].ToString();
//Now the Token variable holds the token value from the scenario context and It can be used in the subsequent steps
}
else
{
Assert.Fail("Unable to get the Token from the Scenario Context");
}
}
If you wish to share the same token across multiple Step, then you can assign this token value within constructor and it can be used
For Example,
[Binding]
public class CustomerManagementSteps
{
public readonly string token;
public CustomerManagementSteps()
{
token= ScenarioContext.Current["Token"].ToString();
}
[Given(#"I Get the customer details""(.*)""")]
public void WhenIGetTheCustomerDetails(string endpoint)
{
//Now the Token variable holds the token value from the scenario context and It can be used in the subsequent steps
}
}
Approach 2: Context Injection details can be referred in the below link with an example
Context Injection
Updated
Given the downvote and comments, I've updated my code example to better show exactly one way you can use dependency injection here with code of your own design. This shared data will last the lifetime of the scenario and be used by all bindings. I think that's what you're looking for unless I'm mistaken.
//Stores whatever data you want to share
//Write this however you want, it's your code
//You can use more than one of these custom data classes of course
public class SomeCustomDataStructure
{
//If this is run in paralell, this should be thread-safe. Using List<T> for simplicity purposes
//Use EF, ConcurrentCollections, synchronization (like lock), etc...
//Again, do NOT copy this code for parallel uses as List<int> is NOT thread-safe
//You can force things to not run in parallel so this can be useful by itself
public List<int> SomeData { get; } = new List<int>();
}
//Will be injected and the shared instance between any number of bindings.
//Lifespan is that of a scenario.
public class CatalogContext : IDisposable
{
public SomeCustomDataStructure CustomData { get; private set; }
public CatalogContext()
{
//Init shared data however you want here
CustomData = new SomeCustomDataStructure();
}
//Added to show Dispose WILL be called at the end of a scenario
//Feel free to do cleanup here if necessary.
//You do NOT have to implement IDiposable, but it's supported and called.
public void Dispose()
{
//Below obviously not thread-safe as mentioned earlier.
//Simple example is all.
CustomData.SomeData.Clear();
}
}
[Binding]
public class SomeSteps
{
//Data shared here via instane variable, accessable to multiple steps
private readonly CatalogContext catalogContext;
//Dependency injection handled automatically here.
//Will get the same instance between other bindings.
public SomeSteps(CatalogContext catalogContext)
{
this.catalogContext = catalogContext;
}
[Given(#"the following ints")]
public void GivenTheFollowingInts(int[] numbers)
{
//This will be visible to all other steps in this binding,
//and all other bindings sharing the context
catalogContext.CustomData.SomeData.AddRange(numbers);
}
}
The Situation
I'm building a C# web application and I want to model my app configuration as an explicit dependency to be handed in through the constructor of a service instead of relying on System.Configuration.ConfigurationManager directly in each of the classes. This did bite my quite often in the past so I want the dependency to be explicit so that the next maintainer of the project (probably future me) doesn't have to guess where my services get their configuration settings - on top of that it is more TDD friendly. Furthermore I'm currently reading Eric Evan's Domain Driven Design and I really want to embrace his DDD approach.
I started modeling the configuration class and corresponding value objects to avoid Primitive Obsession but I hit some bumps on the way and I'm not sure how to handle them appropriately. Here is my current approach:
// Role interface that can be requested via constructor injection
interface IAppConnectionStringsConfig
{
OleDbConnectionString AuthenticationConnectionString { get; }
}
// A base class for handling common functionality like
// parsing comma separated lists or default values
class abstract AppConfigBase
{
protected string GetStringAppSetting(string key)
{
// Get the appropriate string or a default value from
// System.Configuration.ConfigurationManager
return theSettingFromSomeConfigSource;
}
}
// A value object for OLEDB connection strings that also has a
// convenient implicit conversion to string
class OleDbConnectionString
{
public readonly string Value;
public OleDbConnectionString(string connectionString)
{
Contract.Requires(connectionString != null);
this.VerifyStructure(connectionString);
this.Value = connectionString;
}
private void VerifyStructure(string text)
{
Contract.Requires(text != null);
// Verify that the given string fulfills the special
// needs of an OleDbConnectionString (including Provider=...)
if (!/* isValidOleDbConnectionString */)
{
throw new FormatException();
}
}
public implicit operator string(ConnectionString conn)
{
return conn.Value;
}
}
// The actual app config that implements our role interface
class AppConfig : AppConfigBase, IAppConnectionStringsConfig
{
public OleDbConnectionString AuthenticationConnectionString
{
get
{
return new OleDbConnectionString(this.GetStringAppSetting("authconn"));
}
}
}
The Problem
I know that constructor logic should be minimal and that is not a good idea to call virtual methods from the constructor. My questions are as follows:
1) Where should I put the validation logic for the OleDbConnectionString? I really want to prevent the creation of value objects in an invalid state - that's excrutiatingly usefull at a day to day basis :-)
I have the feeling that this is domain logic that should be owned by the class itself but on the other hand the constructor should do as little as possible - wouldn't the string parsing be too much or is this ok?
I could create a validator but I most certainly had to hand that in through the constructor for being able to test that thing properly and then I have to wire that manually or use a factory (I'm definitely not using a Service Locator). On top of that the validation now would be hidden in a separate service; I wouldn't have the temporal coupling since the constructor requires the validator but still that doesn't look right.
2) I wonder if it would be appropriate to make DDD value objects structs? They - like the name suggests - represent a single value and this value is immutable. But they would contain business logic in the form of validation
3) Is it OK to use a property for retrieving the connection string? It could throw an exception if the format for the string isn't valid. Furthermore it's perfectly possible that the implementation will be changed from reading from an xml config file to querying a database.
4) Any other comments on the design are welcome!
As a side note, I'm already using Code Contracts and there is a way to specify object invariants but I don't know whether this is really a good idea since these contracts are opt-in and in the case that they are inactive the invariants are no longer actively protected. I'm not sure about this, for development purposes to catch errors early it might be fine but for production it seems off.
Thx!
I never really thought about general settings as a DDD problem - are you modelling a domain that is about settings and how they are saved, or just allowing settings to be saved and used in an application that has some inner parts modeled as DDD?
You can split this out by separating concerns of getting settings away from the things that use the settings.
Is it OK to use a property for retrieving the connection string? It could throw an exception if the format for the string isn't valid.
I don't think its a good idea to throw an exception if a setting cannot be retrieved so you can return defaults which would allow the program to continue.
But also remember that the default returned value (i.e. a password, or network address) will probably cause the thing that depends on that setting to throw an exception.
I would look at allowing the construction to happen OK but when coming to use the service i.e. Sender.Send() or Sender.Connect() is when you would throw an exception.
Where should I put the validation logic for the OleDbConnectionString? I really want to prevent the creation of value objects in an invalid state
I create objects that can never return an invalid result, but they do return a default settings value:
public class ApplicationSettings : IIdentityAppSettings, IEventStoreSettings
{
/* snip */
static readonly object KeyLock = new object();
public byte[] StsSigningKey
{
get
{
byte[] key = null;
lock (KeyLock)
{
var configManager = WebConfigurationManager.OpenWebConfiguration("/");
var configElement = configManager.AppSettings.Settings["StsSigningKey"];
if (configElement == null)
{
key = CryptoRandom.CreateRandomKey(32);
configManager.AppSettings.Settings.Add("StsSigningKey", Convert.ToBase64String(key));
configManager.Save(ConfigurationSaveMode.Modified); // save to config file
}
else
{
key = Convert.FromBase64String(configElement.Value);
}
}
return key;
}
/* snip */
}
}
What I generally do
I have the settings interfaces for each bounded context defined in the domain model as part of the infrastructure - this allows a number of known interfaces which I can reference and trust to provide some form of settings.
ApplicationSettings is defined in the code that hosts my bounded context(s) be it a Console app or WebAPI or MVC etc, I may have multiple bounded contexts hosted under the same process, or may split them out as separate processes, either way it is the job of the hosting application to provide the relevant application settings and wiring can be done via the IoC container.
public class ApplicationSettings : IIdentityAppSettings, IEventStoreSettings
{
// implement interfaces here
}
public interface IEventStoreSettings
{
string EventStoreUsername { get; }
string EventStorePassword { get; }
string EventStoreAddress { get; }
int EventStorePort { get; }
}
public interface IIdentityAppSettings
{
byte[] StsSigningKey { get; }
}
I use SimpleInjector .NET IoC container to wire up my applications. I then register all the application interfaces with SimpleInjector (so i can query based on any of the application interfaces and have the settings class object returned):
resolver.RegisterAsImplementedInterfaces<ApplicationSettings>();
I can then have the specific interface injected in, an example is a command handler that uses an IRepository, which in turn the EventStoreRepository (which is wired up as an implementation of IRepository) uses IEventStoreSettings (which is wired up as the ApplicationSettings instance):
public class HandleUserStats : ICommandHandler<UserStats>
{
protected IRepository repository;
public HandleUserStats(IRepository repository)
{
this.repository = repository;
}
public void Handle(UserStats stats)
{
// do something
}
}
And my repository would in turn be wired up:
public class EventStoreRepository : IRepository
{
IEventStoreSettings eventStoreSettings;
public EventStoreRepository(IEventStoreSettings eventStoreSettings)
{
this.eventStoreSettings = eventStoreSettings;
}
public void Write(object obj)
{
// just some mockup code to show how to access setting
var eventStoreClient = new EventStoreClient(
this.eventStoreSettings.EventStoreUsername,
this.eventStoreSettings.EventStorePassword,
this.eventStoreSettings.EventStoreAddress,
this.eventStoreSettings.Port
);
// if ever there was an exception either during setup of the connection, or
// exception (if you don't return a default value) accessing settings, it
// could be caught and bubbled up as an InfrastructureException
// now do something with the event store! ....
}
}
I allow settings to be passed in from some external source (like a WCF receive, or MVC controller action) and wired up by getting resolver.GetInstance<CommandHandler<UserStats>>(); which wires up all the settings for me all the way down to the implementation level.
If I use Microsoft implementation unit of work from this tutorial:
http://www.asp.net/mvc/tutorials/getting-started-with-ef-5-using-mvc-4/implementing-the-repository-and-unit-of-work-patterns-in-an-asp-net-mvc-application
public class UnitOfWork : IDisposable
{
private SchoolContext context = new SchoolContext();
private GenericRepository<Department> departmentRepository;
private GenericRepository<Course> courseRepository;
public GenericRepository<Department> DepartmentRepository
{
get
{
if (this.departmentRepository == null)
{
this.departmentRepository = new GenericRepository<Department>(context);
}
return departmentRepository;
}
}
public GenericRepository<Course> CourseRepository
{
get
{
if (this.courseRepository == null)
{
this.courseRepository = new GenericRepository<Course>(context);
}
return courseRepository;
}
}
public void Save()
{
context.SaveChanges();
}
//......
}
I don't need to use transactions when I must add related items? For example when I must add order and order positions to database I don't need to start transaction because if something will go wrong then method Save() won't execute yes? Am I right?
_unitOfWork.OrdersRepository.Insert(order);
_unitOfWork.OrderPositionsRepository.Insert(orderPosition);
_unitOfWork.Save();
??
SaveChanges itself is transactional. Nothing happens at the database level when you call Insert, which based on the tutorial merely calls Add on the DbSet. Only once SaveChanges is called on the context does the database get hit and everything that happened up to that point is sent in one transaction.
You need transactions if you have multiple save changes in one method ... or chain of method calls using the same context.
Then you can roll back over the multiple save changes when your final update fails.
An example would be multiple repositories wrapping crud for an entity under the unit of work (IE a generic class). You may have many functions inserting and saving in each repository. However at the end you may find an issue which causes you to roll back previous saves.
EG in a service layer that needs to hit many repositories and execute a complex operation.
I have a web application and two sql databases. My client wants different languages and I have solved it with resource files one the public parts of the site, no probs. The thing is they have two databases with the same structure and type of data but in different languages, and the users differs aswell. I have overrided "Initialize" from SqlMembershipProvider as:
public class MyqlMembershipProvider : SqlMembershipProvider
{
public override void Initialize(string name, System.Collections.Specialized.NameValueCollection config)
{
var connectionString = GetConnectionStringFromSelectedLanguage();
config["connectionStringName"] = connectionString;
base.Initialize(name, config);
}
}
But the Initialize gets invoked only once, I need to set the connectionString everytime I use Membership.yadayada depending on the language selected. I dont want to use Membership.Providers["one_provider"].DeleteUser(a_username) because It is used EVERYWHERE. How should I go about this, is there anything else I can override?
/Mike
This isn't possible. The membership provider is a static object initialized once and shared between users. Changing any of the values for it changes them for all users of the application. See this answer for a more thorough explanation...
How do I use multiple databases with sqlmembershipprovider at runtime?
You could use a DependencyInjection framework like Ninject for the application and build your repositories based on a named dependency.
For example, you could have an Interface which just provides the connection string name:
Interface IConnectionProvider
{
ConnectionName { get;}
}
class LanguageAProvider : IConnectionProvider
{
public string ConnectionName { get { return "dbnameLangA"; } }
}
class LanguageBProvider : IConnectionProvider
{
public string ConnectionName { get { return "dbnameLangB"; } }
}
public class MyqlMembershipProvider : SqlMembershipProvider
{
private readonly string _connectionStringName;
public MyqlMembershipProvider (IConnectionProvider connection)
{
_connectionStringName = connection.ConnectionName;
}
public override void Initialize(string name, System.Collections.Specialized.NameValueCollection config)
{
var connectionString = GetConnectionStringFromSelectedLanguage();
config[_connectionStringName] = _connectionStringName;//I'm not sure how you need to use the connection name here.
base.Initialize(name, config);
}
}
I haven't tested any of this, but it could be an idea. You could then have Named Bindings to select the required dependency at run time.
I need to set the connectionString everytime I use Membership.yadayada depending on the language selected
This isn't feasible: consider what happens when two requests from users with different languages are processed concurrently. If you changed the connection string for user A, it would affect user B.
What you could do is write a custom Membership provider which aggregates two (or more) SqlMembershipProviders, and redirects based on the current user's language.
public class MyMembershipProvider : MembershipProvider
{
private Dictionary<string, MembershipProvider> _languageMembeshipProviders;
public override void Initialize(string name, System.Collections.Specialized.NameValueCollection config)
{
... create an SqlMembershipProvider for each language and add to
... the languageMembeshipProviders dictionary
}
public override bool ValidateUser(...)
{
... Get language from HttpContext.Current somehow
... select a provider from the dictionary
... call the provider
}
}
However I find it difficult to understand how this could work in practice. Presumably there's something in the request (querystring, cookie, accept-languages header) to indicate the user's language. If so, a user could potentially be authenticated using one language, then switch to access the other database by changing the language in his request.
I'd have thought it would be simpler and cleaner to have separate applications for each language. If a user arrives on site A indicating he wants language B, then he should be redirected to the application B.
I have a occasionally connected application where there is a server that stores information about products. I am using a database cache to store this information locally so when a connection is unavailable the application will still work when trying to read the database.
Since the database is configured and I do not have access to modify the tables, I did not implement 2 way updating and it only downloads a snapshot. A side question is if it is possible to create a database cache and have 2-way sync with only tracking columns on the client machine? I cannot add any columns or tables to the server. I know this might be a question for a separate post, but if this is true then it would change my direction for this problem completely, to a separate module detecting and syncing the database and handling any sync errors that are thrown and always connecting to the cache.
I am using a generic repository and I am wondering what the best practice to go about handling if a connection is available or not and using either a local or remote database depending on this status.
Should I add an interface to the generic repository that handles returning the correct string, and lets the repository know if it is live or not? I need to enable/disable certain features depending on the connection state so I also will need a property somewhere so that when this repository is used there can be a way to bind various controls enabled state to this status.
Instead should I have a wrapper that contains for example an IRepository and IConnectionStringManager and then handles feeding and initializing the repository connection string based on availability? This wrapper would expose the repository and any status properties required.
I guess I am not sure if I should be setting up my program to use IRepository with all the automatic connection sensing behind the scenes, or if I should have IRepositoryManager that has a IRepository and IConnectionStringManager in it.
Maybe both of those options are wrong?
I like the way Entity Framework allows you to provide a connection string as a constructor argument to its contexts. That way you can leverage a dependency injection framework to apply special logic when creating the context, and you only have to change the code in one place (assuming you're using DI principles). You could do something similar with your repository implementation.
Update
Since you're using Entity Framework, here's an idea of what it might look like in code:
// DI Bindings, Ninject style
Bind<Func<MyContext>>().ToMethod(
c => new MyContext(
c.Kernel.Get<IConnectionManager>().IsOnline()
? OnlineConnectionString
: OfflineConnectionString));
// Usage
public class ThingRepository : IThingRepository
{
private Func<MyContext> _getContext;
public ThingRepository(Func<MyContext> getContext)
{
_getContext = getContext;
}
public IEnumerable<Thing> GetAllThings()
{
using(var context = _getContext())
{
return context.Things.ToList();
}
}
}
Or, if you prefer to use a more explicit factory implementation:
public interface IMyContextFactory
{
MyContextFactory Get();
}
public class MyContextFactory : IMyContextFactory
{
private const string OnlineConnectionString = "...";
private const string OfflineConnectionString = "...";
private IConnectionManager _connectionManager;
public MyContextFactory(IConnectionManager connectionManager)
{
_connectionManager = connectionManager;
}
public MyContextFactory Get()
{
var connectionString = _connectionManager.IsOnline()
? OnlineConnectionString
: OfflineConnectionString
return new MyContext(connectionString);
}
}
// DI Bindings, Ninject style
Bind<IMyContextFactory>().To<MyContextFactory>();
// Usage
public class ThingRepository : IThingRepository
{
private IMyContextFactory _myContextFactory;
public ThingRepository(IMyContextFactory myContextFactory)
{
_myContextFactory = myContextFactory;
}
public IEnumerable<Thing> GetAllThings()
{
using(var context = _myContextFactory.Get())
{
return context.Things.ToList();
}
}
}