I need to fetch the metadata for an Azure blob if it exists and would like to achieve this with minimal REST calls (by the storage SDK to the storage service)
I know I can do something like
CloudBlockBlob.ExistsAsync() and then
CloudBlockBlob.FetchAttributesAsync() if the blob exists
I tried to combine these 2 calls into one
CloudBlockBlob.FetchAttributesAsync(AccessCondition.GenerateIfExistsCondition(),new BlobRequestOptions(), new OperationContext());
Docs on 'AccessCondition.GenerateIfExistsCondition()' say -
Constructs an access condition such that an operation will be
performed only if the resource exists.
but it still fails with a 404 not found.
Any idea if what I want to achieve is even possible and what I might be doing wrong?
Looking at the documentation for the action: https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob-properties.
It's basically a HEAD request to the blob, and there is no mention of If-Match etc. for headers.
So I think the most optimal way of doing it is just calling FetchAttributesAsync.
If that causes a 404, then the blob did not exist.
It only does 1 HTTP request.
Related
We're just getting started with Azure Storage.
In our scenario we upload to private blobs that we later need to access directly from our client app, e.g. images.
Is there a way to address private blobs in Azure Storage with a URL containing the access key?
Sifting through the MS docs all I could find so far is simple URL access via the blob URI, e.g. as given by the URI property of the CloudBlockBlob instance when listing blobs via the .net API.
Naturally accessing this from a web browser fails due to the blob not being public.
However, can we qualify the URL to also include the access key in order to allow authorized clients to access the blob..?
You can generate an SAS URL and token for the private blob. Here's the process for generating this manually in the Azure portal, to test the concept. It will work even if your storage container is private, as it allows temporary, time limited access to the file using a URL that contains a token in it's query string.
Click on your file within the storage container, select the 'Generate SAS' tab, and in the right pane select
This will generate a token, and a URL that includes the token, like below:
You can test downloading the URL as a file by using curl. Use the 2nd URL shown in the image above (the one that includes the full token and other parameters in the querystring), then do this (IMPORTANT - the URL must be in double quotes):
curl "<YOUR_URL>" --output myFileName.txt
Tip - this is also a good method for making files available to an Azure VM, if you need to install a file directly on the VM for any reason (I needed to do this to install an SSL certificate), you can generate the URL then curl to download the file, on the VM itself. E.g. connect to the VM first with Bastion or SSH, then use curl to download the file somewhere.
This is the API for how you read blobs from storage:
https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob
There is no URL-Parameter to pass the access key, only the header value Authorization. So you could do the request manually and e.g. add the resulting data as a base64 encoded image. I would advise against it if at all possible.
You must also be aware that by passing your access key to the client, you are effectively making your blob public anyways. You would be putting your data at more risk than anonymous access, since the access key allows more operations than anonymous access. This would also hold true for your objective-c app, even though its much more obfuscated there. SAS is the way to go there - create a backend service that creates a defined set of SAS tokens for given resources. It is however much more effort than simply obfuscating the full access key somewhere.
See "Features available to anonymous users":
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources
Good morning,
I'm trying to implement Azure Blog Storage for the first time using their example code provided. However my app is through a very broad 400 Bad Request error when trying to UploadFromStream().
I have done a bunch of searching on this issue. Almost everything i have come across identifies naming conventions of the container or blob to be the issue. this is NOT my issue, i'm using all lowercase, etc.
My code is no different from their example code:
The connection string:
<add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=xxxxxx;EndpointSuffix=core.windows.net" />
And the code:
// Retrieve storage account from connection string
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
// Retrieve reference to a blob named "myblob"
CloudBlockBlob blob = container.GetBlockBlobReference("myblob");
// Create the container if it doesn't already exist
container.CreateIfNotExists();
// Create or overwrite the "myblob" blob with contents from a local file.
using (var fileStream = System.IO.File.OpenRead(#"D:\Files\logo.png"))
{
blob.UploadFromStream(fileStream);
}
Here is the exception details:
This is all i have to go on. The only other thing i can think of is that i'm running this on my development environment with HTTP not HTTPS. Not sure if this might be a issue?
EDIT:
Additionally, when attempting to upload a file directily in the Azure portal to the container i recieve a
Validation error for TestAzureFileUpload.txt. Details: "The page blob
size must be aligned to a 512-byte boundary. The current file size is
56."
Could this be related to my issue? Am i missing some setting here?
I know i do not have enough to go on here for anyone to help me identify the exact issue, but i am hoping that someone can at least point me in the right direction to resolve this?
Any help would be appreciated
I use a Premium storage account to test the code and get the same "400 bad request" as yours.
From the exception details, you can see the "Block blobs are not supported" message.
Here is an image of my exception details
To solve your problem, I think you should know the difference between block blob and page blob.
Block blobs are comprised of blocks, each of which is identified by a block ID. You create or modify a block blob by writing a set of blocks and committing them by their block IDs. they are for you discrete storage objects like jpg, txt, log, etc. That you'd typically view as a file in your local OS. Supported by standard storage account only.
Page blobs are a collection of 512-byte pages optimized for random read and write operations, such as VHD's. To create a page blob, you initialize the page blob and specify the maximum size the page blob will grow. The truth is, page blobs are designed for Azure Virtual Machine disks. Supported by both standard and Premium storage account.
Since you are using the Premium Storage, which is currently available only for storing data on disks used by Azure Virtual Machines.
So my suggestion is :
If you want your application to support streaming and random access scenarios, and be able to access application data from anywhere. You should use block blobs with the standard account.
If you want to lift and shift applications that use native file system APIs to read and write data to persistent disks. Or you want to store data that is not required to be accessed from outside the virtual machine to which the disk is attached. You should use Page blobs.
Reference link:
Understanding Block Blobs, Append Blobs, and Page Blobs
Follow up question on previous question
I created a container with access type as 'container' and not blob or private still it needs https to get connected!!.
anyway i created cluster multiple times but still I am getting 403: Forbidden or 401 Credentials required. If not connect to company vpn then 404.There should be no SAS issue as I am having access type of container as 'container'
Also I am able to get all container blob list using fiddler rest call: http://nileshhdstore.blob.core.windows.net/nileshhdstore?restype=container&comp=list
When I run my dot net code (Please see my .net C# code at the top) i get the 401 or 403 error. When i check my blob i see that reducer and mapper exe are uploaded in the blob.
I went through https://msdn.microsoft.com/en-us/library/dd179428.aspx.
I have no clue how to add header in the above mapreduce code. I know I have added azure storage key but its not helping out. its been more than a week and I am already late on the proof of concept.
I'm confused about how to get size of the blob in Windows Azure.
In my case, I first get the blob reference with CloudBlockBlob blob = container.GetBlockBlobReference(foo);(here foo is name of blob and I'm sure the blob exists). Then I try to get blob size blob.Property.Length; However, it always return 0. I breakpoint at this statement and track content inside blob: uri of the blob is correct, can I infer that the blob is correctly retrieved from that? While all the fields in Properties is either null or 0. I cannot figure out a solution. Is it because I currently emulate the app locally in Storage Emulator and will be OK after the deployment?
Thanks and Best Regards.
Call blob.FetchAttributes(). GetBlockBlobReference doesn't actually make any calls to the blob service. It just constructs a local object that represents the blob.
I have written a blog on exact same issue about 4 days back:
http://blogs.msdn.com/b/avkashchauhan/archive/2012/04/27/windows-azure-blob-size-return-0-even-when-blob-is-accessible-and-downloaded-without-any-problem.aspx
We're hosting SSIS reports on our servers and we are storing their paths in a sql server table. From .Net, I want to be able to make sure the path entered is correct.
This should return "true" because there's a report there, for example:
http://server/Reports/Pages/Viewer.aspx?%2fShopFloor%2fProduction+by+Turn&rs:Command=Render
This should return false.
http://I am an invalid location/I am laughing at you/now I'm UEEing
I was looking at WebRequests, but I don't know if that's the route I should be taking.
Any help would be appreciated. Thanks!
You can try making a HEAD request to validate that the resource exists. With a HEAD request, you would only need the HTTP Code (200 = Success, 404 = Not Found) without consuming resources or excess memory to download the entire resource. Take a look at HttpWebRequest class for performing the actual request.
Do a http HEAD request to the URL, which should just fetch headers. This way you dont need to download the entire page if it exists. from the returned headers(if there are any) you should be able to determine if its a correct URL or not.
I'm unsure of the exact layout of your network, but if the .net application has visibility to the location the reports are stored, you could use File.Exists(). Otherwise, mellamokb and red-X have the right idea.