Develop Reference

Error while saving bot state to azure blob storage - c#

i am trying to save bot framework using C#. I am trying to save bot state to azure blob storage
using nuget package Microsoft.Bot.Builder.Azure.Blobs version 4.19.2.
What i am doing is in Startup.cs file is
var storage = new BlobsStorage(
"connectionstring",
"containername"
);
var userState = new UserState(storage);
services.AddSingleton(userState);
var conversationState = new ConversationState(storage);
services.AddSingleton(conversationState);
But i am getting following error
System.InvalidOperationException: Unable to find the following types in the 'AllowedTypes' collection.
Please provide the 'AllowedTypesSerializationBinder' in the custom 'JsonSerializerSettings' instance, with the list of types to allow.
Example:
new JsonSerializerSettings
{
SerializationBinder = new AllowedTypesSerializationBinder(
new List<Type>
{
typeof(PromptOptions),
typeof(DialogInstance),
typeof(DialogState),
}),
}
When i check container blob files are generated anyways

Related

I want my application to be able to initialize the data protection key storage at startup.
But it will only work if I use the Azure Storage Access Key and not the MSI or Azure user in my Visual Studio (az login) that I expected. I have given the user and VM Scale Set the roles Owner, Contributor, Storage blob Data Contributor without success.
Is it a must to use SAS or the Access Key in order for the blob to be created automatically?
Error when using the token provider (MSI, az login)
Microsoft.AspNetCore.DataProtection.Internal.DataProtectionHostedService:
Information: Key ring failed to load during application startup.
Request Information
StatusMessage:The specified resource does not exist.
ErrorCode:ResourceNotFound
ErrorMessage:The specified resource does not exist.
public void ConfigureServices( IServiceCollection services )
{
var accessKey = "ddBU/...==";
var blobUri = new Uri( $"https://mystorageaccount.blob.core.windows.net/mycontainer/keys.xml" );
var tp = new Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider();
var token = tp.GetAccessTokenAsync( $"https://mystorageaccount.blob.core.windows.net/" ).Result;
// OK - creates and updates blob when neccessary
var sc = new Microsoft.Azure.Storage.Auth.StorageCredentials( "mystorageaccount", accessKey );
// NOK - can only read the blob
//sc = new Microsoft.Azure.Storage.Auth.StorageCredentials( token );
var cbb = new Microsoft.Azure.Storage.Blob.CloudBlockBlob( blobUri, sc );
services.AddDataProtection()
.PersistKeysToAzureBlobStorage( cbb )
.SetApplicationName( "MyFrontends" );
}

The role "Storage blob Data Contributor" does allow PersistKeysToAzureBlobStorage() to create and maintain the key in the Azure Storage Account.
I must have made a mistake while testing, maybe I did not wait long enough after applying the role or maybe I had not created the target container.

We have a parquet formatfile (500 mb) which is located in Azure blob.How to read the file directly from blob and save in memory of c# ,say eg:Datatable.
I am able to read parquet file which is physically located in folder using the below code.
public void ReadParqueFile()
{
using (Stream fileStream = System.IO.File.OpenRead("D:/../userdata1.parquet"))
{
using (var parquetReader = new ParquetReader(fileStream))
{
DataField[] dataFields = parquetReader.Schema.GetDataFields();
for (int i = 0; i < parquetReader.RowGroupCount; i++)
{
using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
{
DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();
DataColumn firstColumn = columns[0];
Array data = firstColumn.Data;
//int[] ids = (int[])data;
}
}
}
}
}
}
(I am able to read csv file directly from blob using sourcestream).Please kindly suggest a fastest method to read the parquet file directly from blob

Per my experience, the solution to directly read the parquet file from blob is first to generate the blob url with sas token and then to get the stream of HttpClient from the url with sas and finally to read the http response stream via ParquetReader.
First, please refer to the sample code below of the section Create a service SAS for a blob of the offical document Create a service SAS for a container or blob with .NET using Azure Blob Storage SDK for .NET Core.
private static string GetBlobSasUri(CloudBlobContainer container, string blobName, string policyName = null)
{
string sasBlobToken;
// Get a reference to a blob within the container.
// Note that the blob may not exist yet, but a SAS can still be created for it.
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
if (policyName == null)
{
// Create a new access policy and define its constraints.
// Note that the SharedAccessBlobPolicy class is used both to define the parameters of an ad hoc SAS, and
// to construct a shared access policy that is saved to the container's shared access policies.
SharedAccessBlobPolicy adHocSAS = new SharedAccessBlobPolicy()
{
// When the start time for the SAS is omitted, the start time is assumed to be the time when the storage service receives the request.
// Omitting the start time for a SAS that is effective immediately helps to avoid clock skew.
SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24),
Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
};
// Generate the shared access signature on the blob, setting the constraints directly on the signature.
sasBlobToken = blob.GetSharedAccessSignature(adHocSAS);
Console.WriteLine("SAS for blob (ad hoc): {0}", sasBlobToken);
Console.WriteLine();
}
else
{
// Generate the shared access signature on the blob. In this case, all of the constraints for the
// shared access signature are specified on the container's stored access policy.
sasBlobToken = blob.GetSharedAccessSignature(null, policyName);
Console.WriteLine("SAS for blob (stored access policy): {0}", sasBlobToken);
Console.WriteLine();
}
// Return the URI string for the container, including the SAS token.
return blob.Uri + sasBlobToken;
}
Then to get the http response stream of HttpClient from the url with sas token .
var blobUrlWithSAS = GetBlobSasUri(container, blobName);
var client = new HttpClient();
var stream = await client.GetStreamAsync(blobUrlWithSAS);
Finally to read it via ParquetReader, the code comes from Reading Data of GitHub repo aloneguid/parquet-dotnet.
var options = new ParquetOptions { TreatByteArrayAsString = true };
var reader = new ParquetReader(stream, options);

From Azure DevOps portal, I can manually add file/ folder into repository irrespective of the fact that source code is cloned or not - Image for illustration.
However, I want to programmatically create a folder and a file inside that folder within a Repository from c# code in my ASP .NET core application.
Is there a Azure DevOps service REST API or any other way to do that? I'll use BASIC authentication through PAT token only.
Note : I'm restricted to clone the source code at local repository.
Early reply is really appreciated.
I tried HttpClient, GitHttpClient and LibGit2Sharp but failed.

Follow below steps in your C# code
call GetRef REST https://dev.azure.com/{0}/{1}/_apis/git/repositories/{2}/refs{3}
this should return the object of your repository branch which you can use to push your changes
Next, call Push REST API to create folder or file into your repository
https://dev.azure.com/{0}/{1}/_apis/git/repositories/{2}/pushes{3}
var changes = new List<ChangeToAdd>();
//Add Files
//pnp_structure.yml
var jsonContent = File.ReadAllText(#"./static-files/somejsonfile.json");
ChangeToAdd changeJson = new ChangeToAdd()
{
changeType = "add",
item = new ItemBase() { path = string.Concat(path, "/[your-folder-name]/somejsonfile.json") },
newContent = new Newcontent()
{
contentType = "rawtext",
content = jsonContent
}
};
changes.Add(changeJson);
CommitToAdd commit = new CommitToAdd();
commit.comment = "commit from code";
commit.changes = changes.ToArray();
var content = new List<CommitToAdd>() { commit };
var request = new
{
refUpdates = refs,
commits = content
};
var personalaccesstoken = _configuration["azure-devOps-configuration-token"];
var authorization = Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(string.Format("{0}:{1}", "", personalaccesstoken)));
_logger.LogInformation($"[HTTP REQUEST] make a http call with uri: {uri} ");
//here I making http client call
// https://dev.azure.com/{orgnizationName}/{projectName}/_apis/git/repositories/{repositoryId}/pushes{?api-version}
var result = _httpClient.SendHttpWebRequest(uri, method, data, authorization);

I'm using Microsoft.WindowsAzure.Storage C# library in .Net Core 2.0.
Using this library I'm trying to configure CORS dynamically in Azure Storage Emulator but getting error:
"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature".
public async void ConfigureCors() {
ServiceProperties serviceProperties = await this.blobClient.GetServicePropertiesAsync();
serviceProperties.Cors = new CorsProperties();
serviceProperties.Cors.CorsRules.Clear();
serviceProperties.Cors.CorsRules.Add(new CorsRule() {
AllowedHeaders = allowedCorsHeaders,
ExposedHeaders = allowedCorsExposedHeaders,
AllowedOrigins = allowedCorsOrigin,
AllowedMethods = allowedCorsMethods,
MaxAgeInSeconds = allowedCorsAgeInSeconds
});
await blobClient.SetServicePropertiesAsync(serviceProperties);
}
I'm able to generate SAS key for upload files on local server directly, but is not able to configure CORS dynamically so that I can access storage via C# code.
Strange thing to note is that the above code is working perfectly fine when using Azure Storage Cloud but local emulator is throwing this error.
Version info:
WindowsAzure.Storage version is 8.4.0
Windows Azure Storage Emulator version 5.2.0.0
Azure storage explorer version is 0.9.01
Credentials used for connection:
AccountName=devstoreaccount1;
AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuF‌​q2UVErCz4I6tq/K1SZFP‌​TOtr/KBHBeksoGMGw==;

According to your description, I have created a test demo on my side. It works well.
I guess the reason why you get the Server failed to authenticate the request error is the wrong azure storage package version and storage emulator version.
I suggest you could update the storage version to 8.4.0 and storage emulator version to 5.2 firstly and try again.
More details about my test demo, you could refer to below codes:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("UseDevelopmentStorage=true");
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
ServiceProperties serviceProperties = blobClient.GetServicePropertiesAsync().Result;
serviceProperties.Cors = new CorsProperties();
serviceProperties.Cors.CorsRules.Clear();
serviceProperties.Cors.CorsRules.Add(new CorsRule()
{
AllowedHeaders = new List<string>() { "*" },
ExposedHeaders = new List<string>() { "*" },
AllowedOrigins = new List<string>() { "*" },
AllowedMethods = CorsHttpMethods.Put | CorsHttpMethods.Get | CorsHttpMethods.Head | CorsHttpMethods.Post,
MaxAgeInSeconds = 1800
});
var re = blobClient.SetServicePropertiesAsync(serviceProperties);
Result:

I write function for search in youtube.
I create project on "console.developers.google.com" . the name project is youtubesearch and I get apiKey.
I have the error
"An unhandled exception of type
'Google.GData.Client.InvalidCredentialsException' occurred in
Google.GData.Client.dll"
at foreach loop
my code is:
private void getsearch(string serchFor)
{
YouTubeRequestSettings setting = new YouTubeRequestSettings("youtubesearch", "APIkey","ise34857#gmail.com","password for my email");
YouTubeRequest Request = new YouTubeRequest(setting);
YouTubeQuery query = new YouTubeQuery(YouTubeQuery.DefaultVideoUri);
query.OrderBy = "viewCount";
query.Query = serchFor;
query.SafeSearch = YouTubeQuery.SafeSearchValues.Moderate;
Feed<Video> videofeed = Request.Get<Video>(query);
videoLookUp = new Dictionary<string, string>();
foreach (Video v in videofeed.Entries)
{
if (v.Media.Content!=null)
{
Console.WriteLine(v.Title);
Console.WriteLine(v.Media.Content.Url);
comboBox1.Items.Add(v.Title);
videoLookUp.Add(v.Title,v.Media.Content.Url);
}
}
}
Please check object setting if my parameter is right and tell me if I had any proplem in my code

Although the YouTube Data API (v2) has been officially deprecated, you can check in this documentation on how to properly authenticate your application using .NET client library. It also shows here how to properly use the YouTubeRequestSettings object.
To perform any operation using the YouTube API, you create a YouTubeRequestSettings object, which specifies the authentication information and authentication scheme to be used. With that object, you then create a YouTubeRequest object that you will use to actually perform the operations. (If you do not specify authentication information when creating the YouTubeRequestSettings object, then you will only be able to use the YouTubeRequest object to perform operations that do not require authentication.
YouTubeRequestSettings settings =
new YouTubeRequestSettings("example app", clientID, developerKey);
YouTubeRequest request = new YouTubeRequest(settings);
For more information and sample code, check this tutorial.