Azure storage - mapreduce invalid credentials on running C# mapreduce job - c#

Follow up question on previous question
I created a container with access type as 'container' and not blob or private still it needs https to get connected!!.
anyway i created cluster multiple times but still I am getting 403: Forbidden or 401 Credentials required. If not connect to company vpn then 404.There should be no SAS issue as I am having access type of container as 'container'
Also I am able to get all container blob list using fiddler rest call: http://nileshhdstore.blob.core.windows.net/nileshhdstore?restype=container&comp=list
When I run my dot net code (Please see my .net C# code at the top) i get the 401 or 403 error. When i check my blob i see that reducer and mapper exe are uploaded in the blob.
I went through https://msdn.microsoft.com/en-us/library/dd179428.aspx.
I have no clue how to add header in the above mapreduce code. I know I have added azure storage key but its not helping out. its been more than a week and I am already late on the proof of concept.

Related

Fetch Azure blob metadata only if the blob exists

I need to fetch the metadata for an Azure blob if it exists and would like to achieve this with minimal REST calls (by the storage SDK to the storage service)
I know I can do something like
CloudBlockBlob.ExistsAsync() and then
CloudBlockBlob.FetchAttributesAsync() if the blob exists
I tried to combine these 2 calls into one
CloudBlockBlob.FetchAttributesAsync(AccessCondition.GenerateIfExistsCondition(),new BlobRequestOptions(), new OperationContext());
Docs on 'AccessCondition.GenerateIfExistsCondition()' say -
Constructs an access condition such that an operation will be
performed only if the resource exists.
but it still fails with a 404 not found.
Any idea if what I want to achieve is even possible and what I might be doing wrong?
Looking at the documentation for the action: https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob-properties.
It's basically a HEAD request to the blob, and there is no mention of If-Match etc. for headers.
So I think the most optimal way of doing it is just calling FetchAttributesAsync.
If that causes a 404, then the blob did not exist.
It only does 1 HTTP request.

URL to access private blob in Azure Storage

We're just getting started with Azure Storage.
In our scenario we upload to private blobs that we later need to access directly from our client app, e.g. images.
Is there a way to address private blobs in Azure Storage with a URL containing the access key?
Sifting through the MS docs all I could find so far is simple URL access via the blob URI, e.g. as given by the URI property of the CloudBlockBlob instance when listing blobs via the .net API.
Naturally accessing this from a web browser fails due to the blob not being public.
However, can we qualify the URL to also include the access key in order to allow authorized clients to access the blob..?
You can generate an SAS URL and token for the private blob. Here's the process for generating this manually in the Azure portal, to test the concept. It will work even if your storage container is private, as it allows temporary, time limited access to the file using a URL that contains a token in it's query string.
Click on your file within the storage container, select the 'Generate SAS' tab, and in the right pane select
This will generate a token, and a URL that includes the token, like below:
You can test downloading the URL as a file by using curl. Use the 2nd URL shown in the image above (the one that includes the full token and other parameters in the querystring), then do this (IMPORTANT - the URL must be in double quotes):
curl "<YOUR_URL>" --output myFileName.txt
Tip - this is also a good method for making files available to an Azure VM, if you need to install a file directly on the VM for any reason (I needed to do this to install an SSL certificate), you can generate the URL then curl to download the file, on the VM itself. E.g. connect to the VM first with Bastion or SSH, then use curl to download the file somewhere.
This is the API for how you read blobs from storage:
https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob
There is no URL-Parameter to pass the access key, only the header value Authorization. So you could do the request manually and e.g. add the resulting data as a base64 encoded image. I would advise against it if at all possible.
You must also be aware that by passing your access key to the client, you are effectively making your blob public anyways. You would be putting your data at more risk than anonymous access, since the access key allows more operations than anonymous access. This would also hold true for your objective-c app, even though its much more obfuscated there. SAS is the way to go there - create a backend service that creates a defined set of SAS tokens for given resources. It is however much more effort than simply obfuscating the full access key somewhere.
See "Features available to anonymous users":
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources

After Changing Bot Framework Folder Structure, the Messaging Endpoint returns a Bad Request error

I created my Bot Service using the Azure portal and choose the basic template to use, which had the run.csx file in the "messages" folder. I have updated the folder structure to match my needs better and removed the "messages" folder, replacing it with a folder called "Default" instead.
I tried to update my messaging endpoint to point towards "/api/Default?code=", where was the value previously there when the messaging endpoint pointed towards "/api/messages", but now I am getting a Bad Request error:
BadRequest
{
"Message": "The 'code' query parameter provided in the HTTP request did not match the expected value."
}
I'm not sure how to get this "code" value and what it should be. Logs don't say what it should be and I can't find documentation on this.
Someone mentioned in another forum:
Here are two ways to retrieve the bot service key code:
1) go through the kudo console and find the function's secrets .json file: https://[YourBotId].scm.azurewebsites.net/api/vfs/data/Functions/secrets/messages.json
2) open the function app's blade instead of the bot service blade and then find the function's (messages in bot's case) key/code/secrets

Xbox Music RESTful API on Azure access problems

The simple goal here is to access the Xbox Music RESTful API on Azure and I stumble on the first part of it which is getting the Access Token. By following this procedure: http://msdn.microsoft.com/en-us/library/dn546686.aspx it's simply not allowing me to get a token.
Thus far I have:
I have a datamarket registration
I registered an application, got a client_id and client_secret
Subscribed to the Xbox Music RESTful API
Then I simply try to get a token either through a POST using XMLHttpRequest or Advanced Rest Client for Chrome and thus far all I receive is this Json response:
{
"error":"invalid_client",
"error_description":"ACS50012: Authentication failed.\r\nTrace ID: d2469189-d620-4725-98c6-544e3899d711\r\nCorrelation ID: 3726a6c7-de19-4873-a90c-b51c7ca447a7\r\nTimestamp: 2014-05-08 14:18:31Z"
}
I did some research for this error here: http://msdn.microsoft.com/en-us/library/gg429787.aspx but can't seem to pull an explanation for this.
The data explorer is also broken as I receive this:
The request resulted in a backend time out or backend error. The team is investigating the issue. We are sorry for the inconvenience. (502)
In the end I assume something is bogus with my account but can't figure it out. Any help with this would be greatly appreciated.
See comment, recreate a new application and copy client secret properly.

Getting the authenticated user, authed by Apache Basic Auth under Mono and ServiceStack

I'm running a Rest-Service using ServiceStack under Apache2 in C#/Mono. The Apache-Server is using SSL and BasicAuthentication with a htpasswd-file.
If I access my Rest-Service I get the auth-Request of the apache which is good. After authenticating my RestService is accessed. Now my question is: How am I able to get the user (the apache-session user) which has requested the service and authenticated to it?
I allready tried to get the Environment-Variables but didn't get the requested Information there.
The built in Authentication of ServiceStack requires me, as I understand, to implement authentication on my own. But I want to use the authentication of the apache.
Does anyone know a sollution?
Greetings
daily
P.S.
Below are the Mono and SSL relevant Sections of my Site-config for apache:
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/apache.pem
SSLCertificateKeyFile /etc/apache2/ssl/apache.key
SSLOptions StrictRequire +StdEnvVars
SSLProtocol all -SSLv2
MonoServerPath wcs.service.de "/usr/bin/mod-mono-server4"
MonoDebug wcs.service.de true
MonoSetEnv wcs.service.de MONO_IOMAP=all
MonoAutoApplication disabled
MonoApplications wcs.service.de "/:/var/www/wcs"
A little bit later than pronounced, but here is the answer to the question.
In your ServiceStack Service you get your RequestDTO as parameter. See this link:
https://github.com/ServiceStack/ServiceStack/wiki/Access-HTTP-specific-features-in-services
Under the point Services you can see what I mean. You don't need the RequestFilter as I suggested in my answer above. However.. you have this IHttpRequest of Servicestack. On this object you can get the OriginalRequest. This is the original ASP.net-Request. You have to cast the Result to HttpRequest (ASP.net).
var orgRequest = (HttpRequest)request.OriginalRequest;
On this newly casted object you have an accessable field Param. You can now get the REMOTE_USER param which contains the SSL-User I needed.
var sslUser = orgRequest.Param["REMOTE_USER"];
I hope this may help someone running into the same problem.
Greetings, daily

Categories

Resources