Is there a simple API using which I can get the size of an ADLS directory? preferably something in C#, but it's not a must.
We could use the Get Content Summary of a Directory REST API to do that.
curl -i "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETCONTENTSUMMARY"
C# code demo
var url = "https://tomdatalake.azuredatalakestore.net/webhdfs/v1/tomtest?api-version=2017-08-01&op=GETCONTENTSUMMARY";
var token = "eyJ0eX.....";
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
var result = client.GetAsync(url).Result;
var data = result.Content.ReadAsStringAsync().Result;
}
I also test it with PostMan.
Yes, you can use DataLakeStoreFileSystemManagementClient.FileSystem.GetContentSummary:
var client = new DataLakeStoreFileSystemManagementClient(credentials);
ContentSummaryResult result = client.FileSystem.GetContentSummary(dataLakeAccount, path);
var dirSize = result.ContentSummary.Length;
Documentation.
Related
I am developing (so new at it) an application with Windows Forms C# in Visual Studio and I need to get a specific commit from a GitLab project.
I have seen many examples about for GitHub. I have tried this way (do not know if is the correct one):
Download repository at a particular commit using the 7-digit SHA1:
var Token = "xxxx";
var url = "http://{my_domain}/{user}/{project_name}/repository/archive/{shor_id}.zip";
var path = #"C:\GitLab\My_Projects";
try
{
using (var client = new System.Net.Http.HttpClient())
{
var credentials = string.Format(System.Globalization.CultureInfo.InvariantCulture, "{0}:", Token);
credentials = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(credentials));
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", credentials);
var contents = client.GetByteArrayAsync(url).Result;
System.IO.File.WriteAllBytes(path, contents);
}
}
catch (System.UnauthorizedAccessException)
{
backgroundWorker1.CancelAsync();
Console.WriteLine("Problem downloading");
}
But I get this error:
'System.UnauthorizedAccessException' in mscorlib.dll. Access denied to 'C:\GitLab\My_Projects'.
As I said before, I am new at it and probably may have said something stupid, sorry in advance.
If someone knows about the subject, I would love to explain or help me with this concept.
Finally, I achived the solution. I took another direction using GitLab API.
If someone would like to look at the result I arrived, here you have the code:
var url_sha = $"http://{my_gitlab_domain}/api/v4/projects/{id_project}/repository/archive.zip?
private_token={my_token}&sha={short_id_commit}";
var client = new HttpClient();
var response = await client.GetAsync(url_sha);
var local_path = #"C:\GitLab\My_Projects";
using (var stream = await response.Content.ReadAsStreamAsync())
{
var fileInfo = new FileInfo(local_path + ".zip");
using (var fileStream = fileInfo.OpenWrite())
{
await stream.CopyToAsync(fileStream);
}
}
I'm trying to use Azure App Insights API using https://dev.applicationinsights.io/
Got the metrics data using this "https://api.applicationinsights.io/v1/apps/889f31e2-8281-4d8b-bb80-53a9f83d4a09/metrics/requests/duration?interval=PT1H", Passed the api key through headers.
But not able to get the Query:
Tried this:
https://api.applicationinsights.io/v1/apps/889f31e2-8281-4d8b-bb80-53a9f83d4a09/Query/traces | where operation_Id contains 62c5a318-c2cf-434e-a102-0e6efc0a84e7?interval=P7D
Getting Status Code as NotFound.
Any suggestions, Am I missing something?
According to my research, if you want to do query with Azure Application insight rest api, the URL should be like as below
https://api.applicationinsights.io/v1/apps/{appId}/query?query={1}×pan={2}
For example
var client = new HttpClient();
var appId = "";
var query = "traces | where operation_Id contains '33f491236bb412419002b006e1c3058b'";
var timespan = "P7D";
var apiKey=""
string url = string.Format("https://api.applicationinsights.io/v1/apps/{0}/query?query={1}×pan={2}",
appId,
query,
timespan);
var request = new HttpRequestMessage();
request.Method = HttpMethod.Get;
request.Headers.Add("x-api-key", apiKey);
request.RequestUri = new Uri(url);
using (var response = await client.SendAsync(request)) {
var str = await response.Content.ReadAsStringAsync();
Console.WriteLine(JsonConvert.DeserializeObject<JObject>(str));
Console.Read();
}
For more details, please refer to the document. Besides, you also can try the API vai the explorer.
Update
If you want to Azure AD auth to access Azure application insights API, please refer to the following steps
Register Azure AD application in your tenant
Configure API permissions
Create a client secret for the application
Configure assign contributor to the AD application in your Application Insights resource
call the api
// install sdl MSAL.NET to get access token
var client = new HttpClient();
var appId = "";
var query = "traces | where operation_Id contains '33f491236bb412419002b006e1c3058b'";
var timespan = "P7D";
var apiKey=""
string url = string.Format("https://api.applicationinsights.io/v1/apps/{0}/query?query={1}×pan={2}",
appId,
query,
timespan);
string ClientId = "<your ad application clientID>";
string ClientSecret = "<your ad application ClientSecret>";
string tenant = "";
var app = ConfidentialClientApplicationBuilder.Create(ClientId)
.WithClientSecret(ClientSecret)
.WithAuthority(new Uri($"https://login.microsoftonline.com/{tenant}"))
.Build();
string[] scopes = new string[] { "https://api5.applicationinsights.io/.default" };
var result =await app.AcquireTokenForClient(scopes).ExecuteAsync();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", result.AccessToken);
var request = new HttpRequestMessage();
request.Method = HttpMethod.Get;
request.RequestUri = new Uri(url);
using (var response = await client.SendAsync(request)) {
var str = await response.Content.ReadAsStringAsync();
Console.WriteLine(JsonConvert.DeserializeObject<JObject>(str));
Console.Read();
}
For more details, please refer to the document
I am trying to upload an image to the Microsoft custom vision API prediction endpoint using Restsharp, I am trying to use the AddFile method but I am getting a BadRequest as the result, here is the code I am using
public IRestResponse<PredictionResponse> Predict(string imageFileName)
{
var file = new FileInfo(imageFileName);
var serviceUrl = ConfigurationManager.AppSettings["api.custom-vision.prediction.url.file"];
var serviceKey = ConfigurationManager.AppSettings["api.custom-vision.key"];
var client = new RestClient(serviceUrl);
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/octet-stream");
request.AddHeader("Prediction-Key", serviceKey);
request.AddFile(file.Name, file.FullName);
var response = client.Execute<PredictionResponse>(request);
return response;
}
When I execute the method I am getting the following response back from the service
{
"code": "BadRequestImageFormat",
"message": "Bad Request Image Format, Uri: 1062fe0480714281abe2daf17beb3ac5"
}
After looking for ways in the restsharp documentation to properly upload a file, I came to the solution that it needs to be passed as parameter with an array of bytes with the parameter type of ParameterType.RequestBody
Here is the example of the method that actually works
public IRestResponse<PredictionResponse> Predict(string imageFileName)
{
var file = new FileInfo(imageFileName);
var serviceUrl = ConfigurationManager.AppSettings["api.custom-vision.prediction.url.file"];
var serviceKey = ConfigurationManager.AppSettings["api.custom-vision.key"];
var client = new RestClient(serviceUrl);
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/octet-stream");
request.AddHeader("Prediction-Key", serviceKey);
request.AddParameter("content", File.ReadAllBytes(file.FullName), ParameterType.RequestBody);
var response = client.Execute<PredictionResponse>(request);
return response;
}
I simply want to send a rest request to Tableau's REST API but for some reason .NET isn't sending the raw XML (although tested and it works using Postman in chrome)
var admin = "\hardcoded_admin_user"\"";
var pass = "\hardcoded_pass"\"";
var tableau_signin = String.Format("<tsRequest> <credentials name={0} password={1}> </credentials> <site contentUrl=\"\"/> </tsRequest>", admin, pass);
//if user is validated make a REST call to Tableau Server
string endPoint = #"http://server/api/2.0/auth/signin";
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/xml"));
var post = client.PostAsync(endPoint,
new StringContent(tableau_signin)).Result;
}
Any help would be appreciated.
Provide Encoding and Content Type in the StringContent.
var post = client.PostAsync(endPoint,
new StringContent(tableau_signin, Encoding.UTF8, "application/xml")).Result;
var user = FormatTextBodyForUserSignIn(userName, password);
var httpContent = new StringContent(user, Encoding.UTF8, "application/xml");
var response = client.PostAsync($"api/{TableauAPIVersion}/auth/signin", httpContent).Result;
I have OneDrive & Google Drive successfully processing chunked download however Dropbox is giving me grief because I cannot get the correct http request path to the file.
I am not an expert in rest url's & endpoints, maybe someone can point me in the right direction for the acceptable dropbox request format for the latest UWP SDK.
using (var httpRequest = new HttpRequestMessage())
{
string url = "https://content.dropboxapi.com/1/files/auto" + uri;
string accessKey = ApplicationData.Current.LocalSettings.Values[CommonData.dropboxAccessToken_Key].ToString();
httpRequest.Method = HttpMethod.Get;
httpRequest.RequestUri = new Uri(url);
httpRequest.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("bearer", accessKey);
}
I have read docs on Dropbox and it is not clear on the formatting for me, also I could not find a clear example anywhere.
Thanks again!
According to your code, the problem here is in your authorization header. For Dropbox API, the correct authorization header should like following:
Authorization: Bearer <access token>
So we should change httpRequest.Headers.Authorization to
httpRequest.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
Then your code should be albe to work. Using "file.mp3" under "temp" folder for example.
The code may like:
var uri = "/temp/file.mp3";
using (var httpClient = new HttpClient())
{
using (var httpRequest = new HttpRequestMessage())
{
string url = "https://content.dropboxapi.com/1/files/auto" + Uri.EscapeDataString(uri);
httpRequest.Method = HttpMethod.Get;
httpRequest.RequestUri = new Uri(url);
httpRequest.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", accessToken);
var response = await httpClient.SendAsync(httpRequest);
if (response.IsSuccessStatusCode)
{
//TODO
}
}
}