I am very new to ElasticSearch, and have set up an AWS Lambda function in c# to take the content of S3 object(s) (which contain JSON data) with the hopes of posting them to ES to be searchable.
I'm using the Elasticsearch.Net nuget library.
In the documentation here - https://github.com/elastic/elasticsearch-net there is samples of configuring the node URI etc, but my understanding is that any requests to ES need to be signed with AWS Signature V4 (based on Access/Secret key). I have created an IAM user for this purpose, but nowhere in the documentation can I find how to sign the POST requests. The samples show post methods, but no place to include the signature?
E.g.
var person = new Person
{
FirstName = "Martijn",
LastName = "Laarman"
};
var indexResponse = lowlevelClient.Index<BytesResponse>("people", "person", "1", PostData.Serializable(person));
byte[] responseBytes = indexResponse.Body;
var asyncIndexResponse = await lowlevelClient.IndexAsync<StringResponse>("people", "person", "1", PostData.Serializable(person));
string responseString = asyncIndexResponse.Body;
Even when instantiating the connection, nowhere is there a place to add your credentials?
var settings = new ConnectionConfiguration(new Uri("http://example.com:9200"))
.RequestTimeout(TimeSpan.FromMinutes(2));
var lowlevelClient = new ElasticLowLevelClient(settings);
I have checked ConnectionConfiguration object but there's no methods or properties that seem related. What have I missed?
Using the elasticsearch-net-aws package you should be able to set up the low level client like so:
var httpConnection = new AwsHttpConnection("us-east-1"); // or whatever region you're using
var pool = new SingleNodeConnectionPool(new Uri(TestConfig.Endpoint));
var config = new ConnectionConfiguration(pool, httpConnection);
config.DisableDirectStreaming();
var client = new ElasticLowLevelClient(config);
// ...
// if using an access key
var httpConnection = new AwsHttpConnection("us-east-1", new StaticCredentialsProvider(new AwsCredentials
{
AccessKey = "My AWS access key",
SecretKey = "My AWS secret key",
}));
// if using app.config, environment variables, or roles
var httpConnection = new AwsHttpConnection("us-east-1");
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var config = new ConnectionSettings(pool, httpConnection);
var client = new ElasticClient(config);
Related
I am trying to create a database and container using C# I am trying to use the newest version of Azure.ResourceManager.CosmosDB 1.2.0.
In a previos version I used:
var client = new CosmosDBManagementClient("someendpoint", new DefaultAzureCredential());
var database = await client.SqlResources.StartCreateUpdateSqlDatabaseAsync(
AzureResourceGroup, AccountName, DatabaseName,
new SqlDatabaseCreateUpdateParameters(
new SqlDatabaseResource(DatabaseName),
new CreateUpdateOptions()
)
);
var armcontainer = await client.SqlResources.StartCreateUpdateSqlContainerAsync(
AzureResourceGroup, AccountName, DatabaseName,
ContainerName,
GetContainerParameters()
);
However, the CosmosDBManagementClient is no longer in library.
I know there is:
var client = new CosmosClient(endpoint, new DefaultAzureCredential());
await client.CreateDatabaseIfNotExistsAsync("testing",
throughput: null, new RequestOptions());
I also can't get this to work due to 403 error, even with the contributor permissions, but I know this was an error because you are supposed to use the resource manager.
Any suggestions would be appreciated.
I haven't worked with passwordless access but have you tried providing the id explictely?
string userAssignedClientId = "<your managed identity client Id>";
var credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions { ManagedIdentityClientId = userAssignedClientId });
var client = new CosmosClient(endpoint, new DefaultAzureCredential());
If this doesn't work you could try the connection string approach:
using CosmosClient client = new(
accountEndpoint: "endpoint here",
authKeyOrResourceToken: "key here");
I keep getting Grpc.Core.RpcException: Status(StatusCode="Unimplemented" ... exception when I try to use the ImageClassificationInstance objects to classify using a model I have created in the Vertex AI. Is there any way (either by using Google AiPlatform library or using HttpRequest ) to perform ImageClassification with some given image?
The following is the code that I am using
var client = builder.Build();
var instance = new ImageClassificationPredictionInstance() { Content = rawdata };
var parameters = new ImageClassificationPredictionParams() { ConfidenceThreshold = 0.5f, MaxPredictions = 5 };
var pv = ValueConverter.ToValue(parameters);
var iv = new[] { ValueConverter.ToValue(instance) };
var ep = new EndpointName("My project ID", "us-central1", "My endpoint id"); // Of course I have set the string to my project ID and my endpoint ID
var returnobj = client.Predict(ep, iv, pv); // This is where the exception pops up
I can confirm that my vertex AI AutoML model is working on the web browser to classify any uploaded picture.
You must set Endpoint on PredictionServiceClientBuilder :
var client = new PredictionServiceClientBuilder
{
Endpoint = "us-central1-aiplatform.googleapis.com"
}.Build();
How do i fix this. I want to set my authentication in my code and not on the machine.
I have checked almost every answer on stackoverflow and github, but none has a good explanation.
How do i pass the credentials to the create intent, it throws this error.
InvalidOperationException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
GoogleCredential credential =
GoogleCredential.FromFile(file);
//var credential = GoogleCredential.FromStream(
// Assembly.GetExecutingAssembly().GetManifestResourceStream("chatbot-a90a9-8f2fb910202d.json"))
// .CreateScoped(IntentsClient.DefaultScopes);
var storage = StorageClient.Create(credential);
var client = IntentsClient.Create();
var text = new Intent.Types.Message.Types.Text();
text.Text_.Add("Message Text");
var message = new Intent.Types.Message()
{
Text = text
};
var trainingPhrasesParts = new List<string>
{
"Book a fligt",
"check cheap flights"
};
var phraseParts = new List<Intent.Types.TrainingPhrase.Types.Part>();
foreach (var part in trainingPhrasesParts)
{
phraseParts.Add(new Intent.Types.TrainingPhrase.Types.Part()
{
Text = part
});
}
var trainingPhrase = new Intent.Types.TrainingPhrase();
trainingPhrase.Parts.AddRange(phraseParts);
var intent = new Intent();
intent.DisplayName = "test";
intent.Messages.Add(message);
intent.TrainingPhrases.Add(trainingPhrase);
var newIntent = client.CreateIntent(
parent: new AgentName("chatbot-a90a9"),
intent: intent
);
SOLVED.
I change
var client = IntentsClient.Create();
To
IntentsClientBuilder builder = new IntentsClientBuilder
{
CredentialsPath = file, // Relative to where the code is executing or absolute path.
// Scopes = IntentsClient.DefaultScopes // Commented out because there's no need to specify this since you are using the defaults and all default values will be automatically used for values not specified in the builder.
};
IntentsClient client = builder.Build();
I have a .Net Standard client application running on UWP.
My client application contacts the server that generates a sas key like so:
var myPrivateStorageAccount = CloudStorageAccount.Parse(mystorageAccountKey);
var myPrivateTableClient = myPrivateStorageAccount.CreateCloudTableClient();
SharedAccessTablePolicy pol = new SharedAccessTablePolicy()
{
SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(59),
Permissions = SharedAccessTablePermissions.Query | SharedAccessTablePermissions.Add
};
CloudTable myPrivateTable = myPrivateTableClient.GetTableReference(tableName);
String sas = myPrivateTable.GetSharedAccessSignature(pol);
return sas;
My client application then runs the following:
StorageCredentials creds = new StorageCredentials(sas);
this.tableClient = new CloudTableClient(tableServiceURI, creds);
this.table = tableClient.GetTableReference(tableName);
TableQuery<DynamicTableEntity> projectionQuery = new TableQuery<DynamicTableEntity>().Select(new string[] { "DocumentName" }).Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, cc));
var res = await table.ExecuteQuerySegmentedAsync<DynamicTableEntity>(projectionQuery, null);
and gets the following error:
Server failed to authenticate the request. Make sure the value of
Authorization header is formed correctly including the signature. sr
is mandatory. Cannot be empty
but as this is tablestorage I dont think sr is required
and my SAS key looks fine to me:
?sv=2018-03-28&tn=MyTable&sig=RandomSig151235341543&st=2019-01-17T12%3A00%3A28Z&se=2019-01-17T12%3A59%3A28Z&sp=ra
so whats the problem here?
Ok so this is kind of stupid but I will post anyway.
I was sending the request to:
https://myaccount.blob.core.windows.net/
and should have been sending the request to:
https://myaccount.table.core.windows.net/
I have tried the following code to create a new storage account in Azure:
Getting the token (success - I received a token):
var cc = new ClientCredential("clientId", "clientSecret");
var context = new AuthenticationContext("https://login.windows.net/subscription");
var result = context.AcquireTokenAsync("https://management.azure.com/", cc);
Create cloud storage credentials:
var credential = new TokenCloudCredentials("subscription", token);
Create the cloud storage account (fails):
using (var storageClient = new StorageManagementClient(credentials))
{
await storageClient.StorageAccounts.CreateAsync(new StorageAccountCreateParameters
{
Label = "samplestorageaccount",
Location = LocationNames.NorthEurope,
Name = "myteststorage",
AccountType = "RA-GRS"
});
}
Error:
ForbiddenError: The server failed to authenticate the request. Verify
that the certificate is valid and is associated with this
subscription.
I am not sure if this is one of those misleading messages or if I misconfigured something in Azure?
As far as I know, Azure provides two types of storage management library now.
Microsoft.Azure.Management.Storage
Microsoft.WindowsAzure.Management.Storage
Microsoft.Azure.Management.Storage is used to create new ARM storage.
Microsoft.WindowsAzure.Management.Storage is used to create classic ARM storage.
I guess you want to create the new arm storage but you used the "Microsoft.WindowsAzure.Management.Storage" library. Since the "Microsoft.WindowsAzure.Management.Storage" uses the certificate to auth requests, you will get the error. If you want to know how to use "Microsoft.WindowsAzure.Management.Storage" to create classic storage, I suggest you refer to this article.
I assume you want to create new ARM storage, I suggest you install the "Microsoft.Azure.Management.Storage" Nuget package.
More details, you could refer to the following code.
static void Main(string[] args)
{
var subscriptionId = "your subscriptionId";
var clientId = "your client id";
var tenantId = "your tenantid";
var secretKey = "secretKey";
StorageManagementClient StorageManagement = new StorageManagementClient(new Microsoft.Azure.TokenCloudCredentials(subscriptionId, GetAccessToken(tenantId, clientId, secretKey)));
var re= StorageManagement.StorageAccounts.CreateAsync("groupname", "sotrage name",new Microsoft.Azure.Management.Storage.Models.StorageAccountCreateParameters() {
Location = LocationNames.NorthEurope,
AccountType = Microsoft.Azure.Management.Storage.Models.AccountType.PremiumLRS
},new CancellationToken() { }).Result;
Console.ReadKey();
}
static string GetAccessToken(string tenantId, string clientId, string secretKey)
{
var authenticationContext = new AuthenticationContext($"https://login.windows.net/{tenantId}");
var credential = new ClientCredential(clientId, secretKey);
var result = authenticationContext.AcquireTokenAsync("https://management.core.windows.net/",
credential);
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the JWT token");
}
var token = result.Result.AccessToken;
return token;
}