Develop Reference

AWS ElastiCache (Memcached) continually climbing connection count

I have a Lambda function in AWS (.NET Core) created to sweep Parameter Store parameters from Simple Systems Management into Memcached for easy / fast access by my serverless application. Despite my configuration function using the 'using' syntax for the Memcached client (which I expected to mean the connection would be released from the cache once the work was done) the connection count is continually climbing. My function is triggered once every 60 seconds to sync the parameter values.
Here is what the connection count looks like.
Here is my prototype function code:
public async Task<bool> LcsConfigurationSweeper(ILambdaContext context)
{
try
{
context.Logger.LogLine($"Configuration Sweeper activated");
ElastiCacheClusterConfig config = new ElastiCacheClusterConfig("*******", *******);
using (var client = new MemcachedClient(config))
{
GetParametersByPathRequest getParametersByPathRequest = new GetParametersByPathRequest
{
Recursive = true,
WithDecryption = true,
Path = "***"
};
var getParametersByPathResponse = await _systemsManagementClient.GetParametersByPathAsync(getParametersByPathRequest);
foreach (var parameter in getParametersByPathResponse.Parameters)
{
Object value; // Memcached value
if (client.TryGet(parameter.Name, out value))
{
if (value.ToString() != parameter.Value)
{
context.Logger.LogLine($"Configuration sweeper has detected {parameter.Name} has changed. Updating.");
if (client.Store(Enyim.Caching.Memcached.StoreMode.Replace, parameter.Name, parameter.Value))
{
context.Logger.LogLine($"Parameter {parameter.Name} updated in cache OK.");
return true;
}
else
{
context.Logger.LogLine($"Exception: Parameter {parameter.Name} unable to be updated in cache.");
}
}
}
else
{
if (client.Store(Enyim.Caching.Memcached.StoreMode.Set, parameter.Name, parameter.Value))
{
context.Logger.LogLine($"Parameter {parameter.Name} stored in cache OK.");
return true;
}
else
{
context.Logger.LogLine($"Exception: Parameter {parameter.Name} unable to be stored in cache.");
}
}
}
}
}
catch (Exception ex)
{
context.Logger.LogLine($"Exception: {ex.ToString()}");
}
return false;
}

Is it safe to call timer callback method like this?

Please correct me if I have some errors in this logic (not some elegancy things like getting rid of constructor initialization and using Init method instead for Poll). I have not had experience with timer callbacks so far. The code is pretty self-explanatory, I hope. What confuses me a bit is some mix of async things (like connection client creation) and further code - though, I just reused IClient class, it's not mine):
public async Task<WaitForLoanActivationDto> WaitForLoanActivation(string userName, string accountGuid, int timeout)
{
const int dueTime = 0;
const int pollPeriod = 500;
Poll<WaitForLoanActivationDto> state = new Poll<WaitForLoanActivationDto>
{
Client = await _rpcConnectionPool.GetClientAsync(userName),
AutoResetEvent = new AutoResetEvent(false),
StartTime = DateTime.Now,
Timeout = timeout,
Parameters = new Variant[] { accountGuid },
Result = new WaitForLoanActivationDto { Active = false }
};
Timer timer = new Timer(new TimerCallback(WaitForLoanActivationCallback), state, dueTime, pollPeriod);
state.AutoResetEvent.WaitOne();
timer.Dispose(state.AutoResetEvent);
if (state.ThreadException != null)
{
throw state.ThreadException;
}
return state.Result;
}
private void WaitForLoanActivationCallback(object state)
{
Poll<WaitForLoanActivationDto> pollState = (Poll<WaitForLoanActivationDto>)state;
if (pollState.StartTime.AddMilliseconds(pollState.Timeout) >= DateTime.Now)
{
try
{
using (RPCReply reply = ResultHelper.Check(pollState.Client.ExecuteRemoteCommand(WaitForLoanActivationRpcName, pollState.Parameters)))
{
pollState.Result.Active = reply[2].IDList["Active"].AsBoolean().Value;
VariantList statusList = reply[2].IDList["statuses"].List;
if (statusList.Count > 0)
{
var statuses = CustomerInformationConverter.GetStatusesList(statusList);
pollState.Result.Statuses = statuses.ToArray();
}
if (pollState.Result.Active)
{
pollState.AutoResetEvent.Set();
}
}
}
catch (Exception ex)
{
pollState.Result = null;
pollState.ThreadException = ex;
pollState.AutoResetEvent.Set();
}
}
else
{
pollState.AutoResetEvent.Set();
}
}

Thank you guys.
#ckuri, based on your idea I came up with the code below. I have not used Task.Delay though, because as I understood it creates delay even if the task is successfully complete - after it. The objective of my case is to run RPC method each pollPeriod milliseconds during timeout milliseconds. It the method returns Active == false - keep polling, otherwise - return the result. RPC execution time might take more than pollPeriod milliseconds, so if it's already running - no sense to spawn another task.
public async Task<WaitForLoanActivationDto> WaitForLoanActivation(string userName, string accountGuid, int timeout)
{
var cancellationTokenSource = new CancellationTokenSource();
try
{
const int pollPeriod = 500;
IClient client = await _rpcConnectionPool.GetClientAsync(userName);
DateTime startTime = DateTime.Now;
WaitForLoanActivationDto waitForLoanActivationDto = null;
while (startTime.AddMilliseconds(timeout) >= DateTime.Now)
{
waitForLoanActivationDto = await Task.Run(() => WaitForLoanActivationCallback(client, accountGuid), cancellationTokenSource.Token);
if (waitForLoanActivationDto.Active)
{
break;
}
else
{
await Task.Delay(pollPeriod, cancellationTokenSource.Token);
}
}
return waitForLoanActivationDto;
}
catch (AggregateException ex)
{
cancellationTokenSource.Cancel();
throw ex.InnerException;
}
}
private WaitForLoanActivationDto WaitForLoanActivationCallback(IClient client, string accountGuid)
{
using (RPCReply reply = ResultHelper.Check(client.ExecuteRemoteCommand(WaitForLoanActivationRpcName, accountGuid)))
{
var waitForLoanActivationDto = new WaitForLoanActivationDto
{
Active = reply[2].IDList["Active"].AsBoolean().Value
};
VariantList statusList = reply[2].IDList["statuses"].List;
if (statusList.Count > 0)
{
var statuses = CustomerInformationConverter.GetStatusesList(statusList);
waitForLoanActivationDto.Statuses = statuses.ToArray();
}
return waitForLoanActivationDto;
}
}

Task does not always return upon completion

So I am trying to build a program to control a machine. Communications with said machine is via a serial port for which I have written a driver. Continuous polling to the machine is necessary for status feedback etc. In my program I have a dedicated ExecutionEngine() class to handle serial send and receive. I also need to have two separate control sequences running, which I have put into methods RunSequenceA() and RunSequenceB() respectively. During normal operation, all three methods need to run until both control sequences finish, at which point the StopSequence() method is called. My issue is that sometimes, for whatever reason, the StopSequence() method is never called, leaving my ExecutionEngine() method in an infinite loop!
Code for ExecutionEngine():
private static void ExecutionEngine()
{
// Clear both lists in case they have old data
_commandList.Clear();
_pollingList.Clear();
// Poll while user has not yet clicked "STOP"
while (!_cTokenSource.Token.IsCancellationRequested)
{
// If there are commands to be sent, send them first
if (_commandList.Count > 0)
{
Command[] tempCommandArray;
lock (_commandList)
tempCommandArray = _commandList.ToArray();
foreach (var c in tempCommandArray)
{
if (_cTokenSource.Token.IsCancellationRequested)
break;
var response = SerialDriver.ComCycle(c.CommandBytes, _serialPort);
var success = CheckErrorReturn(response, false);
if (success)
{
AddPolling(c);
RemoveCommand(c);
}
}
}
// Do polling operation on applicable controllers
if (_pollingList.Count > 0)
{
Command[] tempPollingArray;
lock (_pollingList)
tempPollingArray = _pollingList.ToArray();
foreach (var c in tempPollingArray)
{
if (_cTokenSource.Token.IsCancellationRequested)
break;
var response = SerialDriver.ComCycle(c.PollBytes, _serialPort);
var success = ProcessPollReturn(response);
if (success)
{
c.FlagDone();
RemovePolling(c);
}
}
}
if (_commandList.Count + _pollingList.Count == 0)
{
// Will get stuck here if neither list gets new items added
Console.WriteLine("Bad place");
Thread.Sleep(500);
}
}
// Cancellation has been requested
lock (_commandList)
_commandList.Clear();
lock (_pollingList)
_pollingList.Clear();
ResetTriggers();
var endCommand = new Command("GL_SYSCMD", 0);
SerialDriver.ComCycle(endCommand.CommandBytes, _serialPort);
_serialPort.Close();
_vm.SequenceRunning = false;
return;
}
Code for running sequences:
private static async Task RunSequencesAsync()
{
var taskArray = new Task[2];
var a = new Action(RunSequenceA);
var b = new Action(RunSequenceB);
taskArray[0] = Task.Run(a);
taskArray[1] = Task.Run(b);
await Task.WhenAll(taskArray).ConfigureAwait(continueOnCapturedContext: false);
// Sometimes this never fires, WHY?
UpdateStatus("All done!");
StopSequence();
}
// Run A sequence
internal static void RunSequenceA()
{
if (_sequenceA1 != null && _sequenceA1.Count > 0)
{
foreach (var s in _sequenceA1)
{
if (_cTokenSource.Token.IsCancellationRequested)
return;
s.Execute();
if (s.Reference != null && TriggerStepCompleted != null)
TriggerStepCompleted(s, EventArgs.Empty);
}
// This part always fires
Console.WriteLine("Sequence A finished");
return;
}
else
return;
}
And finally, the methods to start and stop everything:
private static async Task StartSequenceAsync()
{
_serialPort.PortName = _vm.SelectedComPort;
_serialPort.Open();
_serialPort.DiscardInBuffer();
_serialPort.DiscardOutBuffer();
// Start
_cTokenSource = new CancellationTokenSource();
_vm.SequenceRunning = true;
var taskArray = new Task[2];
taskArray[0] = Task.Run(() => ExecutionEngine());
Thread.Sleep(50);
taskArray[1] = Task.Run(() => RunSequencesAsync());
await Task.WhenAll(taskArray).ConfigureAwait(continueOnCapturedContext: false);
}
private static void StopSequence()
{
_cTokenSource.Cancel();
}
To reiterate, the problem doesn't happen every time. In fact, most times the program runs fine. It seems that problems only arise if I manually call the StopSequence() method half way through execution. Then it's 50/50 as to whether the problem shows up. I'm pretty sure my issue is threading related, but not sure exactly what is going wrong. Any help pointing me in the right direction will be greatly appreciated!

Parallel.ForEach Error showing dbcontext

this method - doDayBegin(item.BranchId) is taking long time to execute. so i am using Parallel.ForEach to execute it parallel. when i am using normal foreach loop its working fine but when i am using Parallel.ForEach it showing this error
The context cannot be used while the model is being created Parallel.ForEach
public ActionResult Edit([DataSourceRequest] DataSourceRequest request)
{
try
{
JavaScriptSerializer js = new JavaScriptSerializer();
List<DB0010020Vm> _listDB0010020Vm = new List<DB0010020Vm>();
string dataDB0010020vm = Request.Form["griddetailsvm"];
if (!string.IsNullOrEmpty(dataDB0010020vm))
{
_listDB0010020Vm = js.Deserialize<List<DB0010020Vm>>(dataDB0010020vm).
Where(d => d.IsValid == "YES").ToList();
}
DateTime start = DateTime.UtcNow;
Parallel.ForEach(_listDB0010020Vm, item =>
{
doDayBegin(item.BranchId);
});
DateTime end = DateTime.UtcNow;
TimeSpan duration = end - start;
return Json(new
{
success = true,
message = "Day Begin Process Completed Successfully!" + duration
});
}
catch (Exception e)
{
return Json(new
{
success = false,
message = e.Message
});
}
}
public void doDayBegin(int BranchId)
{
using (var dbThread = new NeoSampleDBEntities()) // new db connection
{
EBS.DAL.Model.DB0010020 branchDetails = _idDB0010020Repository.FindOne(d => d.BranchId == BranchId);
if (branchDetails == null)
{
ModelState.AddModelError("", "Branch not found!");
}
else
{
branchDetails.LastOpenDate = Convert.ToDateTime(Request.Form["LastOpenDate"]);
OperationStatus status = _idDB0010020Repository.UpdateAndSave(branchDetails);
if (status != null && !status.Status)
ModelState.AddModelError("Updation failed", status.ExceptionMessage);
}
EBS.DAL.Model.DB0010044 dayBegin = new DB0010044();
dayBegin.BankId = 1;
dayBegin.BranchId = BranchId;
dayBegin.DayBeginFlag = 1;
dayBegin.DayDate = Convert.ToDateTime(Request.Form["LastOpenDate"]);
dayBegin.DayEndFlag = 0;
dayBegin.DayEndStage = 1;
dayBegin.DayReopenFlag = 0;
OperationStatus status2 = _idDB0010044Repository.AddAndSave(dayBegin);
if (status2 != null && !status2.Status)
ModelState.AddModelError("Updation failed", status2.ExceptionMessage);
else
{
CreateInwardSessionsForBranch(BranchId);
CreateOutwardSessionsForBranch(BranchId);
}
}
}
this is error
what will be the issue?

You create a new instance of the NeoSampleDBEntities DbContext in the doDayBegin method, but you use an existing (and as far as I can tell, single) instance of the repository _idDB0010020Repository, presumably encapsulating its own instance of DbContext or a pre-existing DbContext instance that was passed in on construction.
The error you're seeing is the result of a second thread's call to <repo>.FindOne making a DbContext method call while the model is being generated as a result of the first thread's call to <repo>.FindOne. Even if the model creation finished, you'll more than likely run into conflicts as DbContext is not thread safe.
From what you've posed, my suggestion would be to create a new repo in the using statement of the doDayBegin method instead of a new instance of the NeoSampleDBEntities DbContext.

My guess it's that _listDB0010020Vm has not been calculated yet when the parallel.foreach is called. If you step through the code in a standard non parallel foreach, it would show this by skipping the .where selection line, going into the foreach, then when it needs the list value, step next will show it jump back to the where selector. I believe List.ToArray will force it to calculate. Try paying that result to the parallel loop.

Duplicate entries on server response .net

Scenario
One windows service polls a url every two minutes to retrieve certain data.
If any data has been added since the previous call, the data is retrieved and stored otherwise the loop carries on.
Issue
Sometimes a request takes more than two minutes to return a response.
When this happens, the next request is still made and finds new data, since the previous request hasn't return a response yet
This results in duplicate entries when the data is stored.
What I've tried
I tried to handle that by using a boolean like so:
Boolean InProgress = true;
foreach (var item in Lists)
{
\\Make a request and return new data (if any)
InProgress = false;
if (InProgress = false)
{
\\Store new data
}
}
This doesn't solve the issue. I believe I'm using the boolean in wrong place, but I'm not sure where it should.
This is the loop that makes the request and store the data
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
Data getCredentials = new Data();
DataTable credentials = getCredentials.loadCredentials();
Boolean InProgress = true;
for (int i = 0; i < credentials.Rows.Count; i++)
{
if (credentials != null)
{
var PBranchID = (int)credentials.Rows[i]["PortalBranchID"];
var negRef = (int)credentials.Rows[i]["NegotiatorRef"];
var Username = credentials.Rows[i]["Username"].ToString();
var Password = credentials.Rows[i]["Password"].ToString();
var Domain = credentials.Rows[i]["Domain"].ToString();
var FooCompanyBaseUrl = "https://" + Domain + ".FooCompany.com/";
Data getCalls = new Data();
DataTable calls = getCalls.loadCalls(PBranchID);
//If it's not the first call
if (calls != null && calls.Rows.Count > 0)
{
//Makes a call
DateTime CreatedSince = DateTime.SpecifyKind((DateTime)calls.Rows[0]["LastSuccessOn"], DateTimeKind.Local);
string IssueListUrl = FooCompany.WebApi.V2.URLs.Issues(BaseUrl, null, CreatedSince.ToUniversalTime(), null);
FooCompany.WebApi.V2.DTO.PrevNextPagedList resultIssueList;
resultIssueList = FooCompany.WebApi.Client.Helper.Utils.Getter<Foocompany.WebApi.V2.DTO.PrevNextPagedList>(IssueListUrl, Username, Password);
InProgress = false;
if (InProgress == false)
{
if (resultIssueList.Items.Count > 0)
{
//If call returns new issues, save call
Data saveCalls = new Data();
saveCalls.saveCalls(PBranchID);
foreach (var item in resultIssueList.Items)
{
var Issue = FooCompany.WebApi.Client.Helper.Utils.Getter<FooCompany.WebApi.V2.DTO.Issue>(item, Username, Password);
string TenantSurname = Issue.Surname;
string TenantEmail = Issue.EmailAddress;
Data tenants = new Data();
int tenantPropRef = Convert.ToInt32(tenants.loadTenantPropRef(PBranchID, TenantSurname, TenantEmail));
Data Properties = new Data();
DataTable propAddress = Properties.loadPropAddress(PBranchID, tenantPropRef);
var Address1 = propAddress.Rows[0]["Address1"];
var Address2 = propAddress.Rows[0]["Address2"];
var AddressFolder = Address1 + "," + Address2;
if (!Directory.Exists("path"))
{
Directory.CreateDirectory("path");
}
string ReportPDFDestination = "path";
if (File.Exists(ReportPDFDestination))
{
File.Delete(ReportPDFDestination);
}
FooCompany.WebApi.Client.Helper.Utils.DownloadFileAuthenticated(FooCompany.WebApi.V2.URLs.IssueReport(BaseUrl, Issue.Id), Username, Password, ReportPDFDestination);
//Store data
}
IssueListUrl = resultIssueList.NextURL;
}
}
}
else
{
continue;
}
}
}
catch (Exception ex)
{
//write to log
}
}
Question
I'm sure there is a better way than a boolean.
Could anyone advice a different method to handle the issue properly?
Thanks.
Solution
I ended up using a combination of both Thomas and Mason suggestions. I wrapped a lock statement around the main function of my windows service and used a boolean inside the function section that makes the call to the remote server.
Tested many times and it's error free.

You seems to have a problem of synchronisation, just surround the code that iterate though the List with a lock, and you will be fine.
public class MyClass{
private readonly object internalLock= new object();
private bool AlreadyRunning { get; set; }
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(AlreadyRunning){
return;
}
try{
lock(internalLock){
Thread.MemoryBarrier();
if(AlreadyRunning){
return;
}
AlreadyRunning = true;
...Do all the things...
}
}
catch(Exception e){
..Exception handling
}
finally
{
AlreadyRunning = false;
}
}

bool InProgress=false;
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(!InProgress)
{
InProgress=true;
//retrieve data
InProgress=false;
}
}
Your InProgress variable needs to be declared outside the event handler. When you enter the method, check to see if it's already running. If it is, then we do nothing. If it's not running, then we say it's running, retrieve our data, then reset our flag to say we've finished running.
You'll probably need to add appropriate locks for thread safety, similar to Thomas's answer.