Develop Reference

Query parameters are persisting values

I have to simulate a real-time data flow from the server, for that I have implemented a Timer class from the System.Threading namespace.
public class DataManager
{
private Timer _timer;
private AutoResetEvent _autoResetEvent;
private Action _action;
public DateTime TimerStarted { get; }
public DataManager(Action action)
{
_action = action;
_autoResetEvent = new AutoResetEvent(false);
_timer = new Timer(Execute, _autoResetEvent, 0, 10000);
TimerStarted = DateTime.Now;
}
public void Execute(object stateInfo)
{
_action();
if ((DateTime.Now - TimerStarted).Seconds > 60)
{
_timer.Dispose();
}
}
}
And using this class to execute repository method to get latest data from controller.
[HttpGet]
public async Task<IActionResult> Get([FromQuery] PagedTransactionDataRequest queryParams)
{
var pageSize = queryParams.PageSize ?? 1;
var pageNumber = queryParams.PageNumber ?? 10;
<b>var timerManager = new DataManager(async () =>
await _paymentDraftHub.Clients.All.SendAsync(SignalRConstants.TransferPaymentDraftServiceData, await _paymentTransactionRepository.GetAllDeclinedAsync(pageSize, pageNumber))
);</b>
var response = new ResponseMessage { Message = "Accepted", Code = "201" };
return Ok(response);
}
PROBLEM: First request to the API is working perfectly fine but during the second request pageSize and pageNumber have the old values too. When the timer runs every 10 seconds, I can see the old query parameter values and new query parameter values both are getting executed.
I am not sure why the old values are even getting executed.

When you call your first GET, you create a DataManager. That DataManager only knows about the first query params. It then executes on a timer with those parameters.
When you call your second GET, you are creating a second DataManager. That second manager will also start executing on a timer with the 2nd set of parameters, but it does not affect the first one at all.
One solution is to make the DataManager a member of your controller, and expose the desired action as a property:
private DataManager _manager;
[HttpGet]
public async Task<IActionResult> Get([FromQuery] PagedTransactionDataRequest queryParams)
{
var pageSize = queryParams.PageSize ?? 1;
var pageNumber = queryParams.PageNumber ?? 10;
if(_manager == null) {_manager = new DataManager(async () =>
await _paymentDraftHub.Clients.All.SendAsync(SignalRConstants.TransferPaymentDraftServiceData, await _paymentTransactionRepository.GetAllDeclinedAsync(pageSize, pageNumber))
);
}
else
{_manager.action = async () => await _paymentDraftHub.Clients.All.SendAsync(SignalRConstants.TransferPaymentDraftServiceData, await _paymentTransactionRepository.GetAllDeclinedAsync(pageSize, pageNumber));
}
var response = new ResponseMessage { Message = "Accepted", Code = "201" };
return Ok(response);
}
I didn't have time to test this solution, so it may not even compile, but it does give you an idea.

Possible HttpClient deadlock

I have a decorator class that adds the ability to rate limit http requests for hitting an API:
public class RateLimitedHttpClient : IHttpClient
{
public RateLimitedHttpClient(System.Net.Http.HttpClient client)
{
_client = client;
_client.Timeout = TimeSpan.FromMinutes(30);
//ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
}
public async Task<string> ReadAsync(string url)
{
if (!_sw.IsRunning)
_sw.Start();
await Delay();
using var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
private async Task Delay()
{
var totalElapsed = GetTimeElapsedSinceLastRequest();
while (totalElapsed < MinTimeBetweenRequests)
{
await Task.Delay(MinTimeBetweenRequests - totalElapsed);
totalElapsed = GetTimeElapsedSinceLastRequest();
};
_timeElapsedOfLastHttpRequest = (int)_sw.Elapsed.TotalMilliseconds;
}
private int GetTimeElapsedSinceLastRequest()
{
return (int)_sw.Elapsed.TotalMilliseconds - _timeElapsedOfLastHttpRequest;
}
private readonly System.Net.Http.HttpClient _client;
private readonly Stopwatch _sw = new Stopwatch();
private int _timeElapsedOfLastHttpRequest;
private const int MinTimeBetweenRequests = 100;
}
However, I'm noticing that at the line indicated below I get a message in the debugger that says that the next statement will execute when the current thread returns.
var epsDataPoints = await _downloader.GetEPSData(cik);
foreach (var eps in epsDataPoints)
{
// getting VS2019 debugger message here!
// assuming that the line above is deadlocking....
Console.WriteLine($"{cik} :: {eps.DateInterval} :: {eps.EPS}");
}
When I open up the Task Manager, the network bandwidth goes to 0 and everything stops with the application other than it sitting at the Console.WriteLine above.
The EPSDownloader class that uses the IHttpClient is below:
public class EPSDownloader
{
public EPSDownloader(IHttpClient client)
{
_client = client;
}
public async Task<IEnumerable<EPSDataPoint>> GetEPSData(int cik)
{
var epsDataPoints = new Dictionary<LocalDate, EPSDataPoint>();
var reportLinks = await GetReportLinks(cik);
foreach (var reportLink in reportLinks)
{
var xbrlLink = await GetXBRLLink(reportLink);
var epsData = await GetEPSData(xbrlLink);
foreach (var eps in epsData)
{
if (!epsDataPoints.ContainsKey(eps.DateInterval.End))
epsDataPoints.Add(eps.DateInterval.End, eps);
}
}
var list = epsDataPoints.OrderBy(d => d.Key).Select(e => e.Value).ToList();
return list;
}
private async Task<IList<string>> GetReportLinks(int cik)
{
// move this url elsewhere
var url = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=" + cik +
"&type=10&dateb=&owner=include&count=100";
var srBody = await _client.ReadAsync(url); // consider moving this to srPage
var srPage = new SearchResultsPage(srBody);
return srPage.GetAllReportLinks();
}
private async Task<string> GetXBRLLink(string link)
{
var url = SEC_HOSTNAME + link;
var fdBody = await _client.ReadAsync(url);
var fdPage = new FilingDetailsPage(fdBody);
return fdPage.GetInstanceDocumentLink();
}
private async Task<IList<EPSDataPoint>> GetEPSData(string xbrlLink)
{
var xbrlBody = await _client.ReadAsync(SEC_HOSTNAME + xbrlLink);
var xbrlDoc = new XBRLDocument(xbrlBody);
return xbrlDoc.GetAllQuarterlyEPSData();
}
private readonly IHttpClient _client;
private const string SEC_HOSTNAME = "https://www.sec.gov";
}
It seems to be that there is an issue with HttpClient, but I don't know why. No exceptions are being thrown, but I do occasionally see that threads have exited with code 0.
Update: I actually restarted my computer while the application was running and it began running fine again for about 20 minutes before the Task Manager showed 0 for the network speed and the application just sat there.

How to store in sql server response of bot

I'm creating a bot using botframework , c# . I'm using also qnamaker. My questions is how to store in database the questions and answer of users. For the moment i can store in log table only message send by users as below :
MessagesController.cs :
// if (activity.Type == ActivityTypes.Message)
{
*************************
// Log to Database
// *************************
// Instantiate the BotData dbContext
Model.qnamakerbotdataEntities DB = new Model.qnamakerbotdataEntities();
// Create a new UserLog object
Model.UserLog NewUserLog = new Model.UserLog();
// Set the properties on the UserLog object
NewUserLog.Channel = activity.ChannelId;
NewUserLog.UserID = activity.From.Id;
NewUserLog.UserName = activity.From.Name;
NewUserLog.created = DateTime.UtcNow;
NewUserLog.Message = activity.Text.Truncate(500);
// Add the UserLog object to UserLogs
DB.UserLogs.Add(NewUserLog);
// Save the changes to the database
DB.SaveChanges();
Do you have an idea how to store message send by users and response of the bot ?
using System;
using System.Threading.Tasks;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Connector;
using QnABot.API;
using Microsoft.Bot.Builder.Dialogs.Internals;
namespace QnABot.Dialogs
{
[Serializable]
public class RootDialog : IDialog<object>
{
public Task StartAsync(IDialogContext context)
{
context.Wait(MessageReceivedAsync);
return Task.CompletedTask;
}
private async Task MessageReceivedAsync(IDialogContext context,
IAwaitable<object> result)
{
//var activity = await result as Activity;
//// Prompt text
//await context.PostAsync("Feel free to ask me");
var privateData = context.PrivateConversationData;
var privateConversationInfo = IncrementInfoCount(privateData,
BotStoreType.BotPrivateConversationData.ToString());
var conversationData = context.ConversationData;
var conversationInfo = IncrementInfoCount(conversationData,
BotStoreType.BotConversationData.ToString());
var userData = context.UserData;
var userInfo = IncrementInfoCount(userData,
BotStoreType.BotUserData.ToString());
context.Wait(QnADialog);
PrivateData.SetValue(BotStoreType.BotPrivateConversationData.ToString(),
privateConversationInfo);
conversationData.SetValue(BotStoreType.BotConversationData.ToString(), conversationInfo);
userData.SetValue(BotStoreType.BotUserData.ToString(), userInfo);
}
private async Task QnADialog(IDialogContext context, IAwaitable<object> result)
{
var activityResult = await result as Activity;
var query = activityResult.Text;
var qnaResult = QnaApi.GetFirstQnaAnswer(query);
if (qnaResult == null)
{
string message = $"Sorry, I did not understand . Please
reformulate your question";
}
else
{
await context.PostAsync(qnaResult.answers[0].answer);
}
context.Wait(MessageReceivedAsync);
}
public class BotDataInfo
{
public int Count { get; set; }
}
private BotDataInfo IncrementInfoCount(IBotDataBag botdata, string key)
{
BotDataInfo info = null;
if (botdata.ContainsKey(key))
{
info = botdata.GetValue<BotDataInfo>(key);
info.Count++;
}
else
info = new BotDataInfo() { Count = 1 };
return info;
}
}
}

You could so something like below. Please note I did not have all your code to make this compile, but you should be able to adjust it if needed. the main part is creating a reply like this var reply = activityResult.CreateReply(); and then setting the text in the message variable in your if-else then setting the text of the reply then sending it
NewUserLog.Message = reply.Text;
await context.PostAsync(reply);
private async Task QnADialog(IDialogContext context, IAwaitable<object> result)
{
var activityResult = await result as Activity;
var query = activityResult.Text;
var reply = activityResult.CreateReply();
var qnaResult = QnaApi.GetFirstQnaAnswer(query);
string message = "";
if (qnaResult == null)
{
message = $"Sorry, I did not understand. Please reformulate your question";
}
else
{
message = qnaResult.answers[0].answer;
}
reply.Text = message;
Model.qnamakerbotdataEntities DB = new Model.qnamakerbotdataEntities();
// Create a new UserLog object
Model.UserLog NewUserLog = new Model.UserLog();
// Set the properties on the UserLog object
NewUserLog.Channel = reply.ChannelId;
NewUserLog.UserID = reply.From.Id;
NewUserLog.UserName = reply.From.Name;
NewUserLog.created = DateTime.UtcNow;
NewUserLog.Message = reply.Text;
await context.PostAsync(reply);
context.Wait(MessageReceivedAsync);
}

Create delay between two message reads of a Queue?

I am using Azure Queues to perform a bulk import.
I am using WebJobs to perform the process in the background.
The queue dequeues very frequently. How do I create a delay between 2 message
reads?
This is how I am adding a message to the Queue
public async Task<bool> Handle(CreateFileUploadCommand message)
{
var queueClient = _queueService.GetQueueClient(Constants.Queues.ImportQueue);
var brokeredMessage = new BrokeredMessage(JsonConvert.SerializeObject(new ProcessFileUploadMessage
{
TenantId = message.TenantId,
FileExtension = message.FileExtension,
FileName = message.Name,
DeviceId = message.DeviceId,
SessionId = message.SessionId,
UserId = message.UserId,
OutletId = message.OutletId,
CorrelationId = message.CorrelationId,
}))
{
ContentType = "application/json",
};
await queueClient.SendAsync(brokeredMessage);
return true;
}
And Below is the WebJobs Function.
public class Functions
{
private readonly IValueProvider _valueProvider;
public Functions(IValueProvider valueProvider)
{
_valueProvider = valueProvider;
}
public async Task ProcessQueueMessage([ServiceBusTrigger(Constants.Constants.Queues.ImportQueue)] BrokeredMessage message,
TextWriter logger)
{
var queueMessage = message.GetBody<string>();
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(_valueProvider.Get("ServiceBaseUri"));
var stringContent = new StringContent(queueMessage, Encoding.UTF8, "application/json");
var result = await client.PostAsync(RestfulUrls.ImportMenu.ProcessUrl, stringContent);
if (result.IsSuccessStatusCode)
{
await message.CompleteAsync();
}
else
{
await message.AbandonAsync();
}
}
}
}

As far as I know, azure webjobs sdk enable concurrent processing on a single instance(the default is 16).
If you run your webjobs, it will read 16 queue messages(peeklock and calls Complete on the message if the function finishes successfully, or calls Abandon) and create 16 processes to execute the trigger function at same time. So you feel the queue dequeues very frequently.
If you want to disable concurrent processing on a single instance.
I suggest you could set ServiceBusConfiguration's MessageOptions.MaxConcurrentCalls to 1.
More details, you could refer to below codes:
In the program.cs:
JobHostConfiguration config = new JobHostConfiguration();
ServiceBusConfiguration serviceBusConfig = new ServiceBusConfiguration();
serviceBusConfig.MessageOptions.MaxConcurrentCalls = 1;
config.UseServiceBus(serviceBusConfig);
JobHost host = new JobHost(config);
host.RunAndBlock();
If you want to create a delay between 2 message reads, I suggest you could create a custom ServiceBusConfiguration.MessagingProvider.
It contains CompleteProcessingMessageAsync method, this method completes processing of the specified message, after the job function has been invoked.
I suggest you could add thread.sleep method in CompleteProcessingMessageAsync to achieve delay read.
More detail, you could refer to below code sample:
CustomMessagingProvider.cs:
Notice: I override the CompleteProcessingMessageAsync method codes.
public class CustomMessagingProvider : MessagingProvider
{
private readonly ServiceBusConfiguration _config;
public CustomMessagingProvider(ServiceBusConfiguration config)
: base(config)
{
_config = config;
}
public override NamespaceManager CreateNamespaceManager(string connectionStringName = null)
{
// you could return your own NamespaceManager here, which would be used
// globally
return base.CreateNamespaceManager(connectionStringName);
}
public override MessagingFactory CreateMessagingFactory(string entityPath, string connectionStringName = null)
{
// you could return a customized (or new) MessagingFactory here per entity
return base.CreateMessagingFactory(entityPath, connectionStringName);
}
public override MessageProcessor CreateMessageProcessor(string entityPath)
{
// demonstrates how to plug in a custom MessageProcessor
// you could use the global MessageOptions, or use different
// options per entity
return new CustomMessageProcessor(_config.MessageOptions);
}
private class CustomMessageProcessor : MessageProcessor
{
public CustomMessageProcessor(OnMessageOptions messageOptions)
: base(messageOptions)
{
}
public override Task<bool> BeginProcessingMessageAsync(BrokeredMessage message, CancellationToken cancellationToken)
{
// intercept messages before the job function is invoked
return base.BeginProcessingMessageAsync(message, cancellationToken);
}
public override async Task CompleteProcessingMessageAsync(BrokeredMessage message, FunctionResult result, CancellationToken cancellationToken)
{
if (result.Succeeded)
{
if (!MessageOptions.AutoComplete)
{
// AutoComplete is true by default, but if set to false
// we need to complete the message
cancellationToken.ThrowIfCancellationRequested();
await message.CompleteAsync();
Console.WriteLine("Begin sleep");
//Sleep 5 seconds
Thread.Sleep(5000);
Console.WriteLine("Sleep 5 seconds");
}
}
else
{
cancellationToken.ThrowIfCancellationRequested();
await message.AbandonAsync();
}
}
}
}
Program.cs main method:
static void Main()
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
var sbConfig = new ServiceBusConfiguration
{
MessageOptions = new OnMessageOptions
{
AutoComplete = false,
MaxConcurrentCalls = 1
}
};
sbConfig.MessagingProvider = new CustomMessagingProvider(sbConfig);
config.UseServiceBus(sbConfig);
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
Result:

Unit testing MassTransit consumers that make utilize asynchronous calls

We are using MassTransit asynchronous messaging (on top of RabbitMQ) for our microservice architecture.
We ran into issues testing consumers that in turn make asynchronous calls.
The example below shows a simple MassTransit consumer that uses RestSharp to make an outbound call and utilized the ExecuteAsync asynchronous method.
public class VerifyPhoneNumberConsumer : Consumes<VerifyPhoneNumber>.Context
{
IRestClient _restClient;
RestRequest _request;
PhoneNumber _phoneNumber;
PhoneNumberVerificationResponse _responseData;
public VerifyPhoneNumberConsumer(IRestClient client)
{
_restClient = client;
}
public void Consume(IConsumeContext<VerifyPhoneNumber> context)
{
try
{
//we can do some standard message verification/validation here
_restClient.ExecuteAsync<PhoneNumberVerificationResponse>(_request, (response) =>
{
//here we might do some standard response verification
_responseData = response.Data;
_phoneNumber = new PhoneNumber()
{
Number = _responseData.PhoneNumber
};
context.Respond(new VerifyPhoneNumberSucceeded(context.Message)
{
PhoneNumber = _phoneNumber
});
});
}
catch (Exception exception)
{
context.Respond(new VerifyPhoneNumberFailed(context.Message)
{
PhoneNumber = context.Message.PhoneNumber,
Message = exception.Message
});
}
}
}
A sample unit test for this might look like the following:
[TestFixture]
public class VerifyPhoneNumberConsumerTests
{
private VerifyPhoneNumberConsumer _consumer;
private PhoneNumber _phoneNumber;
private RestResponse _response;
private VerifyPhoneNumber _command;
private AutoResetEvent _continuationEvent;
private const int CONTINUE_WAIT_TIME = 1000;
[SetUp]
public void Initialize()
{
_continuationEvent = new AutoResetEvent(false);
_mockRestClient = new Mock<IRestClient>();
_consumer = new VerifyPhoneNumberConsumer(_mockRestClient.Object);
_response = new RestResponse();
_response.Content = "Response Test Content";
_phoneNumber = new PhoneNumber()
{
Number = "123456789"
};
_command = new VerifyPhoneNumber(_phoneNumber);
}
[Test]
public void VerifyPhoneNumber_Succeeded()
{
var test = TestFactory.ForConsumer<VerifyPhoneNumberConsumer>().New(x =>
{
x.ConstructUsing(() => _consumer);
x.Send(_command, (scenario, context) => context.SendResponseTo(scenario.Bus));
});
_mockRestClient.Setup(
c =>
c.ExecuteAsync(Moq.It.IsAny<IRestRequest>(),
Moq.It
.IsAny<Action<IRestResponse<PhoneNumberVerificationResponse>, RestRequestAsyncHandle>>()))
.Callback<IRestRequest, Action<IRestResponse<PhoneNumberVerificationResponse>, RestRequestAsyncHandle>>((
request, callback) =>
{
var responseMock = new Mock<IRestResponse<PhoneNumberVerificationResponse>>();
responseMock.Setup(r => r.Data).Returns(GetSuccessfulVericationResponse());
callback(responseMock.Object, null);
_continuationEvent.Set();
});
test.Execute();
_continuationEvent.WaitOne(CONTINUE_WAIT_TIME);
Assert.IsTrue(test.Sent.Any<VerifyPhoneNumberSucceeded>());
}
private PhoneNumberVerificationResponse GetSuccessfulVericationResponse()
{
return new PhoneNumberVerificationResponse
{
PhoneNumber = _phoneNumber
};
}
}
Because of the invocation of the ExecuteAsync method in the consumer, this test method would fall through if we did not put something to block it until it was signaled (or timed out). In the sample above, we are using AutoResetEvent to signal from the callback to continue and run assertions.
THIS IS A TERRIBLE METHOD and we are exhausting all resources to try to find out alternatives. If its not obvious, this can potentially cause false failures and race conditions during testing. Not too mention potentially crippling automated testing times.
What alternatives do we have that are BETTER than what we currently have.
EDIT Here is a source that I originally used for how to mock RestSharp asynchronous calls.
How to test/mock RestSharp ExecuteAsync(...)

Honestly, the complexity of doing asynchronous methods is one of the key drivers of MassTransit 3. While it isn't ready yet, it makes asynchronous method invocation from consumers so much better.
What you're testing above, because you are calling ExecuteAsync() on your REST client, and not waiting for the response (using .Result, or .Wait) in the consumer, the HTTP call is continuing after the message consumer has returned. So that might be part of your problem.
In MT3, this consumer would be written as:
public async Task Consume(ConsumeContext<VerifyPhoneNumber> context)
{
try
{
var response = await _restClient
.ExecuteAsync<PhoneNumberVerificationResponse>(_request);
var phoneNumber = new PhoneNumber()
{
Number = response.PhoneNumber
};
await context.RespondAsync(new VerifyPhoneNumberSucceeded(context.Message)
{
PhoneNumber = _phoneNumber
});
}
catch (Exception exception)
{
context.Respond(new VerifyPhoneNumberFailed(context.Message)
{
PhoneNumber = context.Message.PhoneNumber,
Message = exception.Message
});
}
}

I was able to come up with the following solution which seems far more elegant and proper. Feel free to correct me if I am wrong in assuming this.
I modified the RestSharp execution in my consumer so my consumer looks like the following:
public class VerifyPhoneNumberConsumer : Consumes.Context
{
IRestClient _restClient;
RestRequest _request;
PhoneNumber _phoneNumber;
PhoneNumberVerificationResponse _responseData;
public VerifyPhoneNumberConsumer(IRestClient client)
{
_restClient = client;
}
public void Consume(IConsumeContext<VerifyPhoneNumber> context)
{
try
{
//we can do some standard message verification/validation here
var response = await _restClient.ExecuteGetTaskAsync<PhoneNumberVerificationResponse>(_request);
_responseData = response.Data;
_phoneNumber = new PhoneNumber()
{
Number = _responseData.PhoneNumber
};
}
catch (Exception exception)
{
context.Respond(new VerifyPhoneNumberFailed(context.Message)
{
PhoneNumber = context.Message.PhoneNumber,
Message = exception.Message
});
}
}
}
This utilizes the TPL async capabilities of RestSharp so that I don't have to do it myself.
Because of this, I am able to change my test code to the following:
[Test]
public void VerifyPhoneNumber_Succeeded()
{
var test = TestFactory.ForConsumer<VerifyPhoneNumberConsumer>().New(x =>
{
x.ConstructUsing(() => _consumer);
x.Send(_command, (scenario, context) => context.SendResponseTo(scenario.Bus));
});
var response = (IRestResponse<PhoneNumberVerificationResponse>)new RestResponse<PhoneNumberVerificationResponse>();
response.Data = GetSuccessfulVericationResponse();
var taskResponse = Task.FromResult(response);
Expect.MethodCall(
() => _client.ExecuteGetTaskAsync<PhoneNumberVerificationResponse>(Any<IRestRequest>.Value.AsInterface))
.Returns(taskResponse);
test.Execute();
Assert.IsTrue(test.Sent.Any<VerifyPhoneNumberSucceeded>());
}