Apologies, I'm new to Azure. I created a service bus and queue via the Azure portal using this tutorial.
I can write and read from the queue ok. The problem is, to deploy to the next environment, I have to either update the ARM template to add the new queue or create the queue in code. I can't create the queue through the portal in the next environment.
I've chosen the latter: check to see if the queue exists and create as required via code. I already have an implementation for this for a CloudQueueClient (in the Microsoft.WindowsAzure.Storage.Queue namespace). This uses a CloudStorageAccount entity to create the CloudQueueClient, if it doesnt exists.
I was hoping it would be this simple but it appears not. I'm struggling to find a way to create a QueueClint (in the Microsoft.Azure.ServiceBus namespace). All I have is the Service Bus connection string and the queue name but having scoured Microsoft docs, there's talk of a NamespaceManager and MessagingFactory (in a different namespace) involved in the process.
Can anyone point me in the direction of how to achieve this and more importantly, is this the right approach? I'll be using DI to instantiate the queue so the check/creation will only be done once.
The solution is required for a service bus queue and not a storage account queue. Differences outlined here
Thanks
Sean Feldman's answer pointed me in the right direction. The main nuget packages/namespaces required (.net core ) are
Microsoft.Azure.ServiceBus
Microsoft.Azure.ServiceBus.Management
Here's my solution:
private readonly Lazy<Task<QueueClient>> asyncClient;
private readonly QueueClient client;`
public MessageBusService(string connectionString, string queueName)
{
asyncClient = new Lazy<Task<QueueClient>>(async () =>
{
var managementClient = new ManagementClient(connectionString);
var allQueues = await managementClient.GetQueuesAsync();
var foundQueue = allQueues.Where(q => q.Path == queueName.ToLower()).SingleOrDefault();
if (foundQueue == null)
{
await managementClient.CreateQueueAsync(queueName);//add queue desciption properties
}
return new QueueClient(connectionString, queueName);
});
client = asyncClient.Value.Result;
}
Not the easiest thing to find but hope it helps someone out.
To create entities with the new client Microsoft.Azure.ServiceBus you will need to use ManagemnetClient by creating an instance and invoking CreateQueueAsync().
The Microsoft.Azure.ServiceBus nuget package in the accepted answer is now deprecated. To use the Azure.Messaging.ServiceBus package instead, the code you want is as follows:
using Azure.Messaging.ServiceBus.Administration;
var client = new ServiceBusAdministrationClient(connectionString);
if (!await client.QueueExistsAsync(queueName))
{
await client.CreateQueueAsync(queueName);
}
You can create a Service Bus Queue using NamespaceManager likewise,
QueueDescription _serviceBusQueue = new QueueDescription("QUEUENAME"); //assign the required properties to _serviceBusQueue
NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString("CONNECTIONSTRING");
var queue = await namespaceManager.CreateQueueAsync(_azureQueue);
In case you need a more up to date implementation using the newer Azure.Messaging.ServiceBus library, you'll need to install the package System.Linq.Async
It's an extension method for creating "missing" queues from your service bus namespace
using Azure.Messaging.ServiceBus.Administration;
using System.Threading.Tasks;
using System.Linq;
using System;
public static class ServiceBusAdministrationClientExtensions
{
public static async Task CreateMissingQueuesAsync(this ServiceBusAdministrationClient serviceBusAdministrationClient, params string[] queueNames)
{
var allQueues = serviceBusAdministrationClient.GetQueuesAsync();
var queueList = await allQueues.ToListAsync();
foreach (var queueName in queueNames) {
var foundQueue = queueList.Where(q => q.Name == queueName.ToLower()).Any();
if (!foundQueue)
{
var queueOptions = new CreateQueueOptions(queueName)
{
DefaultMessageTimeToLive = TimeSpan.FromHours(1),
LockDuration = TimeSpan.FromSeconds(30)
};
await serviceBusAdministrationClient.CreateQueueAsync(queueOptions);
}
}
}
}
Then call the extension method with a call like this.
var _serviceBusAdminClient = new ServiceBusAdministrationClient(ServiceBusConnectionString);
await _serviceBusAdminClient.CreateMissingQueuesAsync("queueName");
I adapted this code from the accepted answer in this thread.
Related
I have an Azure Function that has a blob trigger, in my function method args I expose the Blob itself via BlobClient and the name of the file uploaded.
[FunctionName("MyFunc")]
public async Task RunAsync([BlobTrigger("upload/{name}", Connection = "DataLake")]
BlobClient blob, string name)
{
var propertiesResponse = await blob.GetPropertiesAsync();
var properties = propertiesResponse.Value;
var metadata = properties.Metadata;
//do stuff with metadata
if (metadata.TryGetValue("activityId", out var activityId))
{
}
using (var stream = await blob.OpenReadAsync())
using (var sr = new StreamReader(stream))
{
//do some stuff with blob
}
}
I would like to unit test this function and was trying to mock BlobClient but having issues using the Moq library. I have found BlobsModelFactory that aims to help mocking but I can't see anything for BlobClient. Has anyone managed to mock BlobClient?
In line with the new Azure SDK guidelines public methods are marked virtual so they can be mocked:
A service client is the main entry point for developers in an Azure SDK library. Because a client type implements most of the “live” logic that communicates with an Azure service, it’s important to be able to create an instance of a client that behaves as expected without making any network calls.
Each of the Azure SDK clients follows mocking guidelines that allow their behavior to be overridden:
Each client offers at least one protected constructor to allow inheritance for testing.
All public client members are virtual to allow overriding.
In case of the BlobClient mocking can be done like this*:
var mock = new Mock<BlobClient>();
var responseMock = new Mock<Response>();
mock
.Setup(m => m.GetPropertiesAsync(null, CancellationToken.None).Result)
.Returns(Response.FromValue<BlobProperties>(new BlobProperties(), responseMock.Object))
A lot of stuff, like BlobProperties, can be mocked using the static class BlobsModelFactory, some examples:
var blobProps = BlobsModelFactory.BlobProperties(blobType: BlobType.Block);
var result = BlobsModelFactory.BlobDownloadResult(content: null);
Additional references:
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/README.md#mocking
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/samples/Mocking.md
*Code is for demonstration only, the references give clues on how to use BlobsModelFactory
I have an Azure Function that has a blob trigger, in my function method args I expose the Blob itself via BlobClient and the name of the file uploaded.
[FunctionName("MyFunc")]
public async Task RunAsync([BlobTrigger("upload/{name}", Connection = "DataLake")]
BlobClient blob, string name)
{
var propertiesResponse = await blob.GetPropertiesAsync();
var properties = propertiesResponse.Value;
var metadata = properties.Metadata;
//do stuff with metadata
if (metadata.TryGetValue("activityId", out var activityId))
{
}
using (var stream = await blob.OpenReadAsync())
using (var sr = new StreamReader(stream))
{
//do some stuff with blob
}
}
I would like to unit test this function and was trying to mock BlobClient but having issues using the Moq library. I have found BlobsModelFactory that aims to help mocking but I can't see anything for BlobClient. Has anyone managed to mock BlobClient?
In line with the new Azure SDK guidelines public methods are marked virtual so they can be mocked:
A service client is the main entry point for developers in an Azure SDK library. Because a client type implements most of the “live” logic that communicates with an Azure service, it’s important to be able to create an instance of a client that behaves as expected without making any network calls.
Each of the Azure SDK clients follows mocking guidelines that allow their behavior to be overridden:
Each client offers at least one protected constructor to allow inheritance for testing.
All public client members are virtual to allow overriding.
In case of the BlobClient mocking can be done like this*:
var mock = new Mock<BlobClient>();
var responseMock = new Mock<Response>();
mock
.Setup(m => m.GetPropertiesAsync(null, CancellationToken.None).Result)
.Returns(Response.FromValue<BlobProperties>(new BlobProperties(), responseMock.Object))
A lot of stuff, like BlobProperties, can be mocked using the static class BlobsModelFactory, some examples:
var blobProps = BlobsModelFactory.BlobProperties(blobType: BlobType.Block);
var result = BlobsModelFactory.BlobDownloadResult(content: null);
Additional references:
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/README.md#mocking
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/samples/Mocking.md
*Code is for demonstration only, the references give clues on how to use BlobsModelFactory
I am using AmazonS3Client to Read/Write data to S3 Object Storage. In my code i am creating a new connection everytime while doing operations like Read,List Buckets, Upload, Rename, Delete etc. After deploying my application to production i encountered some performance issues. After going throughh few blogs it was recommended to use single amazonS3 client connection. My code below ->
For every below CRUD operations if you see i am creating a new connection and then disposing it by using block. I am planning to have single connection and use it without using block on every call. Does maintaining a single connection good choice ? I have ~400 users accessing application at the same time.
public ObjectFileInfo(string path)
{
StorageClient = ObjectFileManager.GetClient();
objectFileInfo = ObjectFileManager.getFileInfo(StorageClient, path);
}
public class ObjectFileManager
{
public static Amazon.S3.AmazonS3Client GetClient()
{
AmazonS3Config Config = new AmazonS3Config();
AmazonS3Client StorageClient;
Config.RegionEndpoint = null;
Config.ServiceURL = ConfigurationManager.NGDMSobjECSEndPoint;
Config.AllowAutoRedirect = true;
Config.ForcePathStyle = true;
Config.Timeout = TimeSpan.FromMinutes(30);
StorageClient = new AmazonS3Client(ConfigurationManager.NGDMSobjECSUser, ConfigurationManager.NGDMSobjECSKey, Config);
return StorageClient;
}
public static string[] ListBuckets()
{
ListBucketsResponse Response;
//Creating AmazonS3Client and disposing it in using
using (AmazonS3Client StorageClient = GetClient())
{
Response = StorageClient.ListBuckets();
}
var BucketNames = from Bucket in Response.Buckets select Bucket.BucketName;
return BucketNames.ToArray();
}
public static bool DeleteFile(string keyName)
{
var delRequest = new DeleteObjectRequest
{
BucketName = bucketName,
Key = keyName
};
//Creating AmazonS3Client and disposing it in using
using (AmazonS3Client StorageClient = GetClient())
{
StorageClient.DeleteObject(delRequest);
}
return true;
}
}
Planning to use Singleton as below and removing using block ->
class S3ObjectStorageClient
{
/// <summary>
/// Singleton implementation of Object Storage Client
/// </summary>
private S3ObjectStorageClient()
{
}
public static AmazonS3Client Client
{
get
{
return S3Client.clientInstance;
}
}
/// <summary>
/// Nested private class to ensure Singleton
/// </summary>
private class S3Client
{
static S3Client()
{
}
internal static readonly AmazonS3Client clientInstance = ObjectFileManager.GetClient();
}
}
public ObjectFileInfo(string path)
{
StorageClient = S3ObjectStorageClient.Client; //Singleton
objectFileInfo = ObjectFileManager.getFileInfo(StorageClient, path);
}
public static string[] ListBuckets()
{
ListBucketsResponse Response;
//Singleton and removed using block
AmazonS3Client StorageClient = S3ObjectStorageClient.Client;
Response = StorageClient.ListBuckets();
var BucketNames = from Bucket in Response.Buckets select Bucket.BucketName;
return BucketNames.ToArray();
}
public static bool DeleteFile(string keyName)
{
var delRequest = new DeleteObjectRequest
{
BucketName = bucketName,
Key = keyName
};
//Singleton and removed using block
AmazonS3Client StorageClient = S3ObjectStorageClient.Client;
StorageClient.DeleteObject(delRequest);
return true;
}
}
As one of the authors of the AWS .NET SDK I can give a little more context. Under the cover the AmazonS3Client along with all of the other service clients in the SDK it manages a pool of HttpClients which are the expensive object to create. So when you are creating a new AmazonS3Client the SDK is reusing an HttpClient from a pool the SDK is managing.
If you are using a proxy with proxy credentials then the SDK does have to create a new HttpClient each time a service client is created.
An area where there could be potential performance issues with creating service clients all the time is determining the AWS credentials to use when an AWSCredentials object is not passed into the constructor. That means each service client will have to resolve the credentials which if you are using an assume role profile that could cause a lot of extra calls to perform the assume role. Getting credentials from instance metadata is optimized so that only one thread is refreshing those credentials per process.
Actually you can safely reuse it, according to the docs it is not a bad idea to create and reuse a client. But creating a new client is not very expensive:
The best-known aspect of the AWS SDK for .NET are the various service clients that you can use to interact with AWS. Client objects are thread safe, disposable, and can be reused. (Client objects are inexpensive, so you are not incurring a large overhead by constructing multiple instances, but it’s not a bad idea to create and reuse a client.)
Thus, according to this the performance benefits are probably not that huge. But since there is a small cost to creating a new client I would always reuse the client. That said, according to the docs your code
using (AmazonS3Client StorageClient = GetClient())
{
Response = StorageClient.ListBuckets();
}
is not really bad, but just a bit less efficient than using a singleton. If you think it hurts your performance in a noticable way, best bet is to measure it and if it is really the cause refactor to using a singleton.
Both are valid approach but you'll certainly gain code efficiency using a singleton.
Moreover, dependency injection is promoted by AWS as the right pattern when it comes to using clients. For example, new AWS service CodeGuru profiler highlights multiple client instances as a source of optimization.
See also : https://aws.amazon.com/fr/blogs/developer/working-with-dependency-injection-in-net-standard-inject-your-aws-clients-part-1/
I'm using next code to push Tag to Git repository
#addin "Cake.Git"
using LibGit2Sharp;
var solutionFolder = "./";
var versionTag = "someTag";
Task("Default")
.Does(() =>
{
var remoteRepository = EnvironmentVariable("bamboo_planRepository_repositoryUrl");
var repositoryRevision = EnvironmentVariable("bamboo_planRepository_revision");
var absolutePath = MakeAbsolute(Directory(solutionFolder));
var repoName = "central";
//LibGit2Sharp add remote
using (var repo = new Repository(absolutePath.FullPath))
{
repo.Network.Remotes.Add(repoName, remoteRepository);
}
GitTag(solutionFolder, versionTag, repositoryRevision);
GitPushRef(solutionFolder, gitUser, gitPassword, repoName, versionTag);
}
});
Stuck into the next issue: because our bamboo configured to use SSH protocol, and Cake.Git(LibGit2Sharp) currently doesn't support it receiving next error
Error: unsupported URL protocol
Thanks
I would suspect the issue is to due using shallow clones, which are enabled by default.
Shallow clones allows Bamboo to perform clones with i.e. history truncated to a specified number of revisions.
This should increase the speed of the initial code checkouts, however if your build depends on the full repository history, we recommend that you do not use this option.
GIT operations in general need the full repo to work reliably.
A little bit hacky but it works, will update answer when will find better approach.
Done based on the How to tag a git repo in a bamboo build.
Cake.Git currently doesn't support adding repository but under the hood using LibGit2Sharp so just added LibGit2Sharp namespace to the code.
Core issue is that Cake.Git(LibGit2Sharp) doesn't support SSH yet (Issue on GitHub Is it possible to use Cake.Git with SSH), as workaraound calling git push through cmd How to execute cmd
command
#addin "Cake.Git"
using LibGit2Sharp;
var solutionFolder = "./";
var versionTag = "someTag";
var remoteRepository = EnvironmentVariable("bamboo_planRepository_repositoryUrl");
var repositoryRevision = EnvironmentVariable("bamboo_planRepository_revision");
Task("Default")
.Does(() =>
{
var absolutePath = MakeAbsolute(Directory(solutionFolder));
var repoName = "central";
//LibGit2Sharp add remote
using (var repo = new Repository(absolutePath.FullPath))
{
repo.Network.Remotes.Add(repoName, remoteRepository);
}
GitTag(solutionFolder, versionTag, repositoryRevision);
Cmd($"git push {repoName} {versionTag}");
}
});
private void Cmd(params object[] parameters)
{
if (parameters.Any())
{
var args = new ProcessArgumentBuilder()
.Append(#"/c");
foreach (var param in parameters)
args.Append($"{param}");
StartProcess("cmd", new ProcessSettings { Arguments = args });
}
}
Using NotificationHubClient I can get all registered devices using GetAllRegistrationsAsync(). But if I do not use the registration model but the installation model instead, how can I get all installations? There are methods to retrieve a specific installation but none to get everything.
You're correct, as of July 2016 there's no way to get all installations for a hub. In the future, the product team is planning to add this feature to the installations model, but it will work in a different way. Instead of making it a runtime operation, you'll provide your storage connection string and you'll get a blob with everything associated with the hub.
Sorry for visiting an old thread... but in theory you could use the GetAllRegistrationsAsyc to get all the installations. I guess this will return everything without an installation id as well, but you could just ignore those if you choose.
Could look something like this
var allRegistrations = await _hub.GetAllRegistrationsAsync(0);
var continuationToken = allRegistrations.ContinuationToken;
var registrationDescriptionsList = new List<RegistrationDescription>(allRegistrations);
while (!string.IsNullOrWhiteSpace(continuationToken))
{
var otherRegistrations = await _hub.GetAllRegistrationsAsync(continuationToken, 0);
registrationDescriptionsList.AddRange(otherRegistrations);
continuationToken = otherRegistrations.ContinuationToken;
}
// Put into DeviceInstallation object
var deviceInstallationList = new List<DeviceInstallation>();
foreach (var registration in registrationDescriptionsList)
{
var deviceInstallation = new DeviceInstallation();
var tags = registration.Tags;
foreach(var tag in tags)
{
if (tag.Contains("InstallationId:"))
{
deviceInstallation.InstallationId = new Guid(tag.Substring(tag.IndexOf(":")+1));
}
}
deviceInstallation.PushHandle = registration.PnsHandle;
deviceInstallation.Tags = new List<string>(registration.Tags);
deviceInstallationList.Add(deviceInstallation);
}
I am not suggesting this to be the cleanest chunk of code written, but it does the trick for us. We only use this for debugging type purposes anyways