Develop Reference

Azure queue handling via ReceiveAsync returns null right away

The normal expected behaviour for the code below, would be that ReceiveAsync, looks at the Azure queue for up to 1 minute before returning null or a message if one is received. The intended use for this is to have an IoT hub resource, where multiple messages may be added to a queue intended for one of several DeviceClient objects. Each DeviceClient will continuously poll this queue to receive message intended for it. Messages for other DeviceClients are thus left in the queue for those others.
The actual behaviour is that ReceiveAsync is immediately returning null each time it's called, with no delay. This is regardless of the value that is given with TimeSpan - or if no parameters are given (and the default time is used).
So, rather than seeing 1 log item per minute, stating there was a null message received, I'm getting 2 log items per second (!). This behaviour is different from a few months ago,. so I started some research - with little result so far.
using Microsoft.Azure.Devices;
using Microsoft.Azure.Devices.Client;
public static TimeSpan receiveMessageWaitTime = new TimeSpan(0, 1 , 0);
Microsoft.Azure.Devices.Client.Message receivedMessage = null;
deviceClient = DeviceClient.CreateFromConnectionString(Settings.lastKnownConnectionString, Microsoft.Azure.Devices.Client.TransportType.Amqp);
// This code is within an infinite loop/task/with try/except code
if(deviceClient != null)
{
receivedMessage = await deviceClient.ReceiveAsync(receiveMessageWaitTime);
if(receivedMessage != null)
{
string Json = Encoding.ASCII.GetString(receivedMessage.GetBytes());
// Handle the message
}
else
{
// Log the fact that we got a null message, and try again later
}
await Task.Delay(500); // Give the CPU some time, this is an infinite loop after all.
}
I looked at the Azure hub, and noticed 8 messages in the queue. I then added 2 more, and neither of the new messages were received, and the queue is now on 10 items.
I did notice this question: Azure ServiceBus: Client.Receive() returns null for messages > 64 KB
But I have no way to see whether there is indeed a message that big currently in the queue (since receivemessage returns null...)
As such the questions:
Could you preview the messages in the queue?
Could you get a queue size, e.g. ask the number of messages in the queue before getting them?
Could you delete messages from the queue without getting them?
Could you create a callback based receive instead of an infinite loop? (I guess internally the code would just do a peek and the same as we are already doing)
Any help would be greatly appreciated.

If you use the Azure ServiceBus, I recommend that you could use the Service Bus Explorer to preview the message, get the number of message in the queue. And Also you could delete the message without getting them.

Is there a way to "flush" the messages into Facebook Messenger?

TL;DR: the bot takes 2-9 seconds to send any message to the user, even if I send the message from the very first line in MessagesController.cs.
I have used the Bot Framework to create a couple of bots for Facebook Messenger and I noticed that one of them is significantly slower than the other.
Both bots have the code below as the very first lines in the MessagesController.cs. This is supposed to send the Typing indicator to the user.
One bot consistently takes 2 seconds to show this typing indicator to the user on Facebook, while the other takes 9 seconds.
Activity typing = activity.CreateReply(null);
typing.ServiceUrl = activity.ServiceUrl;
typing.Type = ActivityTypes.Typing;
ConnectorClient connector = new ConnectorClient(new Uri(typing.ServiceUrl));
await connector.Conversations.SendToConversationAsync(typing);
The second bot indeed does much more work (calling various web APIs), but since these lines are the very first in the controller, I expect the typing indicator to be sent to the user immediately, after which the bot can continue doing its work.
However, it seems that the messages (including the typing indicator) are not sent until the bot completes its work.
Is there a way to "flush" the messages to the user to have a typing indicator sent immediately, or am I encountering some other issue?
Update: I've tried the ConnectorClient.Dispose() method but it doesn't seem to help make it any faster to send messages.

I am not sure what you mean by "flushing" the messages to the user but your bot should show the typing indicator immediately until you type a message and there is some process in the background. To avoid processing information until the user types a message you can use the ActivityTypes like this:
if (activity.Type == ActivityTypes.Message)
{
Activity typing = activity.CreateReply(null);
typing.ServiceUrl = activity.ServiceUrl;
typing.Type = ActivityTypes.Typing;
ConnectorClient connector = new ConnectorClient(new Uri(typing.ServiceUrl));
await connector.Conversations.SendToConversationAsync(typing);
}
else
{
HandleSystemMessage(activity);
}
...
However, it makes sense that the user cannot type anything until the application has already given a answer back to the previous question.

How do I trigger a subscription error to test recovery on exchange web service

My application uses the EWS API with a Streaming Subscription and everything is working fine and thats a problem for me as i havn't been able to exercise my recovery code for the OnSubscriptionError event.
Here is the code i use to subscribe for streaming notifications
private void SetStreamingNotifications(List<FolderId> folder_ids)
{
streaming_subscriptions_connection = new StreamingSubscriptionConnection(exchange_service, 30);
streaming_subscriptions_connection.OnDisconnect += OnDisconnect;
streaming_subscriptions_connection.OnSubscriptionError += OnSubscriptionError;
streaming_subscriptions_connection.OnNotificationEvent += OnNotificationEvent;
foreach (var folder_id in folder_ids)
{
StreamingSubscription sub = exchange_service.SubscribeToStreamingNotifications(
new[] { folder_id },
EventType.Created,
EventType.Modified,
EventType.Deleted,
EventType.Moved,
EventType.Copied
);
streaming_subscriptions_connection.AddSubscription(sub);
}
streaming_subscriptions_connection.Open();
}
private void OnSubscriptionError(object sender, SubscriptionErrorEventArgs args)
{
/* What exceptions can i expect to find in "args.Exception" */
/* Can the streaming subscription be recovered or do i need to create a new one? */
}
So my question is how can i trigger a subscription error so i can ensure my code can recover where possible and log / alert when not possible?
EDIT
Following a comment from #kat0r i feel i should add:
I'm currently testing against Exchange 2013 and also intend to test against Exchange 2010 SP1.

I logged a call with Microsoft to find out if it was possible. The short answer is no you can't trigger the OnSubscriptionError event.
Here are the email responses from MS:
In answer to your question, I don’t believe that there is a way you can trigger the OnSubscriptionError event. The correct action to take if you do encounter this event is to attempt to recreate the subscription that encountered the error. I will see if I can find out any further information about this, but the event is generated rarely and only when an unexpected error is encountered on the Exchange server (which is why it probably isn’t possible to trigger it).
It occurred to me that the EWS Managed API has been open-sourced, and is now available on Github: https://github.com/officedev/ews-managed-api
Based on this, we can see exactly what causes the OnSubscriptionError event to be raised – and as far as I can see, this only occurs in the IssueSubscriptionFailures and IssueGeneralFailure methods, both of which can be found in StreamingSubscriptionConnection.cs. Any error that is not ServiceError.ErrorMissedNotificationEvents and is tied to a subscription will result in this event being raised (and the subscription being removed). The error is read from the Response Xml. Of course, this doesn’t really answer your question of how to trigger the event, as that involves causing Exchange to generate such an error (and I’m afraid there is no information on this). It may be possible to inject some Xml (indicating an error) into a response in a test environment – in theory, you may be able to use Fiddler to do this (as it allows you to manipulate requests/responses).

A few things you could do is
Throttling will restrict the maximum number of subscriptions you can create so if you just keep creating new subscription you should get a throttling response from the server once you exceed 20.
The other thing is if you take the SubscriptionId and use a different process to unsubscribe you other code should get a Subscription not found.
You also want to test underlying network issue eg break the connection , dns if you have dev environment see what happens when you bounce the Exchange Server etc.
Cheers
Glen

Cant get MSMQ to work properly

I keep receiving an error when i try to Send to the queues
The specified format name does not support the requested operation. For example, a direct queue format name cannot be deleted.
My queue name is stored in a config file and looks like this
"FormatName:Direct=OS:MyComputerName\PRIVATE$\MyQueue"
There are 2 queues that exist, each queue has permissions set so that EVERYONE has Full Control.
SyncUser you can assume = "EVERYONE" or "My_Domain\operator"
//init the queues
qSync = new MessageQueue(queueName + "-sync")
{Formatter = new XmlMessageFormatter(new Type[] {typeof (String)})};
qClient = new MessageQueue(queueName + "-client")
{Formatter = new XmlMessageFormatter(new Type[] { typeof(String) })};
Creating the queues doesn't cause any errors, but I receive errors when i call this Send function
**Send(qSync, "This is a message");** //Example
public void Send(MessageQueue queue, string msg)
{
try
{
queue.Send(msg);
}
catch (MessageQueueException mqx)
{
var e = string.Format("Failed to Send - {0} | {1}", queue.QueueName, mqx.Message);
OnSynchronizerMonitor(new SyncEvent(e));
}
}
Any help would be greatly appreciated.
EDIT
It seems that my messages ARE making it to their remote queue destination, but the application is still throwing that error on every message sent. If I look at the MessageQueues (qSync|qClient) while debugging, that error shows up in several of the inner fields, before it is even used.

I hope someone can confirm this for me as I cant seem to find any forum questions or documentation that support this, but it seems that the MessageQueue has an issue with my Event setup.
I removed the event call and simply hardcoded the queue creation and send in its place within my main process and everything worked fine.
My setup is like so:
A Service - Creates a Processor, a Synchronizer, and sets the events.
Processor starts a thread in which it processes string transactions, on a successful process the transaction needs to be sent via the Synchronizer, so the Processor triggers event.
Synchronizer has already created the queues and is simply waiting for events to be triggered to send messages.
If anyone could shed some light on if this could be a cross thread issue or just a quirk in my system it would be appreciated.
**After a little more research and some trial and error, it seems I get the error due to a cross thread complication with the event. My guess is that it's a similar issue to UI objects being updated from a different thread than the one it was created on. The error is essentially an ownership issue. The messages still go though but the queue complains that its original owner isn't the one that initiated the call.

Bloomberg API request timing out

Having set up a ReferenceDataRequest I send it along to an EventQueue
Service refdata = _session.GetService("//blp/refdata");
Request request = refdata.CreateRequest("ReferenceDataRequest");
// append the appropriate symbol and field data to the request
EventQueue eventQueue = new EventQueue();
Guid guid = Guid.NewGuid();
CorrelationID id = new CorrelationID(guid);
_session.SendRequest(request, eventQueue, id);
long _eventWaitTimeout = 60000;
myEvent = eventQueue.NextEvent(_eventWaitTimeout);
Normally I can grab the message from the queue, but I'm hitting the situation now that if I'm making a number of requests in the same run of the app (normally around the tenth), I see a TIMEOUT EventType
if (myEvent.Type == Event.EventType.TIMEOUT)
throw new Exception("Timed Out - need to rethink this strategy");
else
msg = myEvent.GetMessages().First();
These are being made on the same thread, but I'm assuming that there's something somewhere along the line that I'm consuming and not releasing.
Anyone have any clues or advice?
There aren't many references on SO to BLP's API, but hopefully we can start to rectify that situation.

I just wanted to share something, thanks to the code you included in your initial post.
If you make a request for historical intraday data for a long duration (which results in many events generated by Bloomberg API), do not use the pattern specified in the API documentation, as it may end up making your application very slow to retrieve all events.
Basically, do not call NextEvent() on a Session object! Use a dedicated EventQueue instead.
Instead of doing this:
var cID = new CorrelationID(1);
session.SendRequest(request, cID);
do {
Event eventObj = session.NextEvent();
...
}
Do this:
var cID = new CorrelationID(1);
var eventQueue = new EventQueue();
session.SendRequest(request, eventQueue, cID);
do {
Event eventObj = eventQueue.NextEvent();
...
}
This can result in some performance improvement, though the API is known to not be particularly deterministic...

I didn't really ever get around to solving this question, but we did find a workaround.
Based on a small, apparently throwaway, comment in the Server API documentation, we opted to create a second session. One session is responsible for static requests, the other for real-time. e.g.
_marketDataSession.OpenService("//blp/mktdata");
_staticSession.OpenService("//blp/refdata");
The means one session operates in subscription mode, the other more synchronously - I think it was this duality which was at the root of our problems.
Since making that change, we've not had any problems.

My reading of the docs agrees that you need separate sessions for the "//blp/mktdata" and "//blp/refdata" services.

A client appeared to have a similar problem. I solved it by making hundreds of sessions rather than passing in hundreds of requests in one session. Bloomberg may not be to happy with this BFI (brute force and ignorance) approach as we are sending the field requests for each session but it works.

Nice to see another person on stackoverflow enjoying the pain of bloomberg API :-)
I'm ashamed to say I use the following pattern (I suspect copied from the example code). It seems to work reasonably robustly, but probably ignores some important messages. But I don't get your time-out problem. It's Java, but all the languages work basically the same.
cid = session.sendRequest(request, null);
while (true) {
Event event = session.nextEvent();
MessageIterator msgIter = event.messageIterator();
while (msgIter.hasNext()) {
Message msg = msgIter.next();
if (msg.correlationID() == cid) {
processMessage(msg, fieldStrings, result);
}
}
if (event.eventType() == Event.EventType.RESPONSE) {
break;
}
}
This may work because it consumes all messages off each event.

It sounds like you are making too many requests at once. BB will only process a certain number of requests per connection at any given time. Note that opening more and more connections will not help because there are limits per subscription as well. If you make a large number of time consuming requests simultaneously, some may timeout. Also, you should process the request completely(until you receive RESPONSE message), or cancel them. A partial request that is outstanding is wasting a slot. Since splitting into two sessions, seems to have helped you, it sounds like you are also making a lot of subscription requests at the same time. Are you using subscriptions as a way to take snapshots? That is subscribe to an instrument, get initial values, and de-subscribe. If so, you should try to find a different design. This is not the way the subscriptions are intended to be used. An outstanding subscription request also uses a request slot. That is why it is best to batch as many subscriptions as possible in a single subscription list instead of making many individual requests. Hope this helps with your use of the api.

By the way, I can't tell from your sample code, but while you are blocked on messages from the event queue, are you also reading from the main event queue while(in a seperate event queue)? You must process all the messages out of the queue, especially if you have outstanding subscriptions. Responses can queue up really fast. If you are not processing messages, the session may hit some queue limits which may be why you are getting timeouts. Also, if you don't read messages, you may be marked a slow consumer and not receive more data until you start consuming the pending messages. The api is async. Event queues are just a way to block on specific requests without having to process all messages from the main queue in a context where blocking is ok, and it would otherwise be be difficult to interrupt the logic flow to process parts asynchronously.