Cosmos DB - BulkExecutor InitializeAsync awaiting forever for the response - c#

I am working on a Console Application to load multiple docs into Cosmos from CSV File through C# code. When I am debugging the code, the debugger stops at await bulkExecutor.InitializeAsync(); and waiting for the response forever.
Even when I am running the executables locally, I am facing the same issue.
But when I am creating the web job with the same code and deploy to Azure Portal, there it works seamlessly. For the same web job, i am again facing the same issue in local machine during debugging mode.
ConnectinMode - I have tried both Direct and Gateway but still no luck.
NOTE :
Microsoft.Azure.CosmosDB.BulkExecutor version 1.8.8
Microsoft.Azure.DocumentDB version 2.10.3
All NuGET Packages are uptodate
var client = new DocumentClient(new Uri(config.CosmosDbendpointUrl), config.CosmosDbAuthorizationKey,
new ConnectionPolicy { ConnectionMode = ConnectionMode.Gateway, ConnectionProtocol = Protocol.Tcp });
DocumentCollection dataCollection = null;
dataCollection = Utils.GetCollectionIfExists(client, config.CosmosDatabase, config.CosmosDBCollection);
if (dataCollection == null)
{
throw new Exception("The data collection does not exist");
}
// Set retry options high for initialization (default values).
client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30;
client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9;
IBulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection);
await bulkExecutor.InitializeAsync();

BulkExecutor has been deprecated and is no longer supported. If you are working on a new app and want bulk features use the .NET SDK v3 and set the AllowBulk = true in the client options.
You can find a sample here.
Thanks.

Related

Azure CosmosDB SDK v3 with ASP.NET Web Forms

I have an issue with the following code, working well in a Console App project and not working in a ASP.NET Web Forms project, both targeting .NET Framework 4.7.2.
The goal is to use the last Azure Cosmos DB SDK (v3) to get all documents in a container, without specifying a type (use of dynamic).
I've tried to target both the emulator (the last version 2.4.5) and the Azure Cosmos service.
In the ASP.NET project, the execution of queryResultSetIterator.ReadNextAsync().Result never ends (no timeout).
string endpointUri = "https://localhost:8081";
string primaryKey = "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==";
string database = "SendGridEvents";
string collection = "SendGridEvents";
using (CosmosClient client = new CosmosClient(endpointUri, primaryKey))
{
Container container = client.GetDatabase(database).GetContainer(collection);
QueryDefinition queryDefinition = new QueryDefinition("SELECT * FROM c");
FeedIterator<dynamic> queryResultSetIterator = container.GetItemQueryIterator<dynamic>(queryDefinition);
List<dynamic> documents = new List<dynamic>();
while (queryResultSetIterator.HasMoreResults)
{
FeedResponse<dynamic> currentResultSet = queryResultSetIterator.ReadNextAsync().Result;
foreach (var document in currentResultSet)
{
documents.Add(document);
}
}
}
This is caused due to a deadlock because of the use of the problematic .Result.
There are countless of sources that explain this deadlock but I will link this answer from StackOverflow.
The TL;DR is that you are blocking an otherwise asynchronous call in the UI thread which is causing the deadlock. You need to properly await your call like this:
FeedResponse<dynamic> currentResultSet = await queryResultSetIterator.ReadNextAsync();
You could technically block the call if you used the ConfigureAwait(false) approach but the v3 SDK does not cascade this call all the way to the network calls so it wouldn't make any difference. WebForms allows you to have async handlers so you would be fine to make your method async and try again.

Operation returned an invalid status code 'Forbidden`. Botframework v4

So i created a bot in azure and downloaded it. The free 1000 calls from LUIS reached its limit. I created a subscription in azure portal (I did do the docker container something). Followed this guide until step 6. When i click the endpoint url and query directly in the browser it is working fine.
I added it to the bot via Bot Emulator by clicking + sign in services and adding the bot model there. But when i run bot i get the title error. I noticed in the .bot file the authoring key and subscription key added by the bot emulator is the same.
So i changed the subscription key to the one of the keys generated by azure and still the same error. I have tried reseting the authoring key still same and deleting my luis.ai account and created a new one. (still same email because that is the one logged in azure portal.) and still the same.
Here are some pictures for reference and the error.
I also tried testing it in luis.ai and got this result.
but when i check it is set to the new resource.
Here is a pic of the bot file after adding luis via Bot emulator. It has same authoring key and subscription key (still forbidden)
so i changed it now with subscription key (still forbidden).
Here it is working properly when tested directly in the URL.
For reference:
azure portal
luis.ai
and the error
How i add luis in the bot.
Here is the code for the bot service.
using System;
using System.Collections.Generic;
using Microsoft.Bot.Builder.AI.Luis;
using Microsoft.Bot.Configuration;
namespace Microsoft.BotBuilderSamples
{
public class BotServices
{
public BotServices(BotConfiguration botConfiguration)
{
foreach (var service in botConfiguration.Services)
{
switch (service.Type)
{
case ServiceTypes.Luis:
{
var luis = (LuisService)service;
if (luis == null)
{
throw new InvalidOperationException("The LUIS service is not configured correctly in your '.bot' file.");
}
var endpoint = (luis.Region?.StartsWith("https://") ?? false) ? luis.Region : luis.GetEndpoint();
var app = new LuisApplication(luis.AppId, luis.AuthoringKey, endpoint);
var recognizer = new LuisRecognizer(app);
this.LuisServices.Add(luis.Name, recognizer);
break;
}
}
}
}
public Dictionary<string, LuisRecognizer> LuisServices { get; } = new Dictionary<string, LuisRecognizer>();
}
}
I am trying to solve this for 4 days already. Thanks!
Thank you for all of the images. That is a HUGE help! Here's the problem:
By default, your code looks for the AuthoringKey in this section (second line):
var endpoint = (luis.Region?.StartsWith("https://") ?? false) ? luis.Region : luis.GetEndpoint();
var app = new LuisApplication(luis.AppId, luis.AuthoringKey, endpoint);
var recognizer = new LuisRecognizer(app);
this.LuisServices.Add(luis.Name, recognizer);
Since your .bot file still has the authoringKey set to the one that starts with ad9c..., which has hit its limit, your bot keeps running into the 403 error.
So, in your .bot file, replace that authoringKey with one of your endpointKeys (they start with 12ccc... or b575...).
I understand your confusion with this, especially since this requires you putting an endpointKey in your authoringKey property. I know there's some changes on the horizon to how LUIS bots will use keys, but those are probably a month or more out.
Alternatively, you can change:
var app = new LuisApplication(luis.AppId, luis.AuthoringKey, endpoint);
to:
var app = new LuisApplication(luis.AppId, luis.SubscriptionKey, endpoint);
Note: If you make either of these changes, LUIS can only query (which is usually fine), since Authoring Keys do everything else (see reference, below)
References
These are not so much for you as much as others that might come across this.
Authoring vs. Endpoint Keys
Key Limits
Troubleshooting LUIS 403 Errors

Repository created by Octokit doesn't appear on website

I cannot open repository created via Octokit from Github web browser (using standard url github.com/user/repo-name)
But what works is url github.com/user/repo-name.git
Additionally, Github shows this message on the page:
Cannot retrieve the latest commit at this time.
I assume that it might be some problems in .git configuration files by they looks pretty much the same as in repos I create in web browser.
What might be causing the problem?
I can clone this repo using context.CloneUrl which works just fine.
I'm creating a repository via Octokit, which works fine
var basicAuth = new Octokit.Credentials(Login, Password);
var client = new GitHubClient(new ProductHeaderValue(repoName)) {Credentials = basicAuth};
var repository = new NewRepository(repoName)
{
AutoInit = false,
Description = null,
Private = false
};
var context = await client.Repository.Create(repository);
Additionally I'm using LibGit2Sharp to create and initialize local repository
Github was experiencing difficulties since last night check out https://status.github.com/messages . It might be back up now (10hrs later)

How to track MongoDB requests from a console application

I have a Console Application project written in C# which I've added Application Insights to with the following NuGet packages.
Microsoft.ApplicationInsights
Microsoft.ApplicationInsights.Agent.Intercept
Microsoft.ApplicationInsights.DependencyCollector
Microsoft.ApplicationInsights.NLogTarget
Microsoft.ApplicationInsights.PerfCounterCollector
Microsoft.ApplicationInsights.Web
Microsoft.ApplicationInsights.WindowsServer
Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel
I've configured my InstrumentationKey in the config file and I'm firing up a TelemetryClient on startup using the with the following code:
var telemetryClient = new TelemetryClient();
telemetryClient.Context.User.Id = Environment.UserName;
telemetryClient.Context.Session.Id = Guid.NewGuid().ToString();
telemetryClient.Context.Device.OperatingSystem = Environment.OSVersion.ToString();
Everything is working well except AI is not capturing any requests that get sent to Mongo, I can see requests going off to SQL server in the 'Application map' but no sign of any other external requests. Is there any way that I can see telemetry of requests made to Mongo?
EDIT - Thanks to Peter Bons I ended up with pretty much the following which works like a charm and allows me to distinguish between success and failure:
var telemetryClient = new TelemetryClient();
var connectionString = connectionStringSettings.ConnectionString;
var mongoUrl = new MongoUrl(connectionString);
var mongoClientSettings = MongoClientSettings.FromUrl(mongoUrl);
mongoClientSettings.ClusterConfigurator = clusterConfigurator =>
{
clusterConfigurator.Subscribe<CommandSucceededEvent>(e =>
{
telemetryClient.TrackDependency("MongoDB", e.CommandName, DateTime.Now.Subtract(e.Duration), e.Duration, true);
});
clusterConfigurator.Subscribe<CommandFailedEvent>(e =>
{
telemetryClient.TrackDependency("MongoDB", $"{e.CommandName} - {e.ToString()}", DateTime.Now.Subtract(e.Duration), e.Duration, false);
});
};
var mongoClient = new MongoClient(mongoClientSettings);
I am not familiar with MongoDB but as far as I can tell there is no default support for it when it comes to Application Insights. But that does not mean you cannot do this, it will just involve some more code.
Again, I am not familiar with MongoDB but according to http://www.mattburkedev.com/logging-queries-from-mongodb-c-number-driver/ there is built-in support for logging the generated queries. Now, we only need to hook this up to Application Insights.
Since you already know how to use the TelemetryClient we can use the custom tracking methods provided by that class. See https://learn.microsoft.com/nl-nl/azure/application-insights/app-insights-api-custom-events-metrics for the available custom tracking methods.
All you need to do is to insert some code like this:
telemetryClient.TrackDependency(
"MongoDB", // The name of the dependency
query, // Text of the query
DateTime.Now, // Time that query is executed
TimeSpan.FromSeconds(0), // Time taken to execute query
true); // Indicates success
The class telemetryClient is thread-safe so you can reuse it.
Now, according to the referenced blogpost you should be able to do something like this:
var client = new MongoClient(new MongoClientSettings()
{
Server = new MongoServerAddress("localhost"),
ClusterConfigurator = cb =>
{
cb.Subscribe<CommandStartedEvent>(e =>
{
telemetryClient.TrackDependency(
"MongoDB", // The name of the dependency
e.Command.ToJson() // Text of the query
DateTime.Now, // Time that query is executed
TimeSpan.FromSeconds(0), // Time taken to execute query
true); // Indicates success
});
}
});
Again, I am not familiar with MongoDB but I hope this is a starting point for your imagination on how to adapt it to your needs using your knowledge of MongoDB.
EDIT:
If there is also a CommandCompletedEvent or similar event as opposed to the CommandStartedEvent event you should probably track the dependency there because you should then be able to calculate (or simpel read) the time spent and maybe get the actual value for the success indicator.

Application Insights with WPF, nullreferenceexception

I'm trying to get Application Insights to work with my WPF application, but whenever I try to call any of the Track functions, I get a nullreferenceexception with this stacktrace
at Microsoft.ApplicationInsights.TelemetryClient.Track(ITelemetry telemetry)
at Microsoft.ApplicationInsights.TelemetryClient.TrackTrace(TraceTelemetry telemetry)
at Microsoft.ApplicationInsights.TelemetryClient.TrackTrace(String message)
I added an application in Azure so that I got an instrumentationKey (hidden in the code below), added the nuget package and entered this code:
var config = new TelemetryConfiguration();
client = new TelemetryClient(config);
config.InstrumentationKey = "myKey";
client.InstrumentationKey = "myKey";
client.TrackTrace("testing testing");
client.Flush();
The crash occurs at the 5:th line above, and It occurs no matter which version of the nuget package I use.
I've heard some people mention an ApplicationInsights.config, but no such file has been generated for me.
Use TelemetryConfiguration.Active instead of create a new instance.
No need to set InstrumentationKey on client instance. When you set the Active configuration, it will used that key for every new client instance. Only explicit set the key on the telemetry client if you're sending that specific telemetry to a custom / different key than the configuration.
TelemetryConfiguration.Active.InstrumentationKey = "myKey";
client = new TelemetryClient();
client.TrackTrace("testing testing");
client.Flush();
Just ran into the same issue. We got it working with code like:
// setup the client
TelemetryClient tc = new TelemetryClient();
tc.InstrumentationKey = "key copied from portal";
// Set session data:
tc.Context.User.Id = Environment.UserName;
tc.Context.Session.Id = Guid.NewGuid().ToString();
tc.Context.Device.OperatingSystem = Environment.OSVersion.ToString();
tc.TrackTrace("some data....");
tc.Flush(); // only for desktop apps
You can see more details at Application Insights on Windows Desktop apps, services and worker roles

Categories

Resources