Using HttpClient to post large MultipartFormDataContent to IIS server web page - c#

I am trying to upload up to 2GB of data using a HttpClient.
The data is sent through the request body into an aspx page where it is read (horrible, I know but I cannot change this.)
The data is placed in a MultipartFormDataContent and posted like this:
var apiRequest = new MultipartFormDataContent();
apiRequest.Add(new StreamContent(file), fileName, fileName);
apiRequest.Headers.ContentDisposition =
new ContentDispositionHeaderValue("form-data") { Name = fileName, FileName = fileName };
HttpClient client = new HttpClient();
client.Timeout = TimeSpan.FromMinutes(30);
HttpResponseMessage response = null;
try
{
response = client.PostAsync(apiEndPoint, form).Result;
response.EnsureSuccessStatusCode();
string responseBody = response.Content.ReadAsStringAsync().Result;
}
catch (HttpRequestException e)
{
log.LogError($"logging here");
}
Things I have tried :
-HttpClient http version 1.0 instead of default
-HttpClient MaxRequestContentBufferSize
-web.config maxAllowedContentLength
-web.config AspMaxRequestEntityAllowed
-web.config maxRequestLength
Currently, the files get added to the httpClient correctly but I cannot get them to post to the web app. I got up to 900MB through but anything over that simply redirects to the main page and I get HTML from the web app in the response body.
Thanks in advance!

After a lot of hair pulling we found the solution by looking at the IIS logs.
The logs revealed that the Microsoft URLScan tool was blocking the requests.
When the request body was not approved by the scan, IIS would redirect you straight to the main page with no error.
You have to configure a max request length in the urlscan.ini file.
More info here: https://ajaxuploader.com/large-file-upload-iis-debug.htm
The file is located at C:\WINDOWS\system32\inetsrv\urlscan

Related

How do you make a Post request with binary file in this situation?

i have System A (Web Interface for people to upload files) and System B (file archive system).
step 1) User submits a files via System A
step 2) System A receives the files ,and encrypt it
step 3) System A sends a Post request with the decrypted file by calling System B 's end point (http://127.0.0.1:8080/rest/bookstore/).
step 4) System B receives the POST request. then archive the file .
at this moment , i am stuck in step 3. i am abled to receive the files, but throw a exception when System A make a Post Request with the file
i am trying to post a csv file to http://127.0.0.1:8080/rest/bookstore/. However it return
RestSharp.RestClient.Execute returned "StatusCode: NotAcceptable, Content-Type: , Content-Length: 0)" RestSharp.RestResponse
Do you have any idea what wrong in below code?
string path = "THE PATH OF THE FILE";
byte[] file = System.IO.File.ReadAllBytes(path);
string filevalue = System.Convert.ToBase64String(file);
var client = new RestClient("http://127.0.0.1:8080/rest/bookstore/");
client.Timeout = -1;
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "text/csv");
request.AddParameter("text/csv", file , ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);
You should not need to redirect, should simply be:
public class BookstoreController : ApiController
{
public void Get() => Download();
}
public static void Download()
{
var client = new RestClient("...");
var request = new RestRequest("...", Method.POST);
var response = client.Execute(request);
client.DownloadData(new RestRequest("/file", Method.GET)).SaveAs("filename.csv");
Console.WriteLine("Output: ");
Console.WriteLine(response.Content);
}
In essence, you could do the method content to download the file. But create the client and request, then execute the response but download the data from the request into the response. Then it should force the browser to download the file for the user.
Today i have checked the system Log . i found this message (which means my file is archived).
INFO 11:15:01.938 (FedoraLdp) GET resource 'bookstore/cc51bc52-e59a-4e29-801a-e7d979326197'
So that i try to submit Post Request and check if file is archived .
and finally found that the file is archived as showed in system log. even though ,In the Http Post response ,it said that my Request is not acceptable ...
So that i think that The API End point does not return my a correct message and not giving me a clear information about what is missing in Http Post Request i made.

Asana API 403 Response in C#

I am trying to implement a Xamarin app that works with the Asana API.
I have successfully implemented the OAuth as documented in the Asana documentation here... at least I assume it is successful. I get an access token from the token endpoint in an HTTPResponse with HTTP Status "OK".
But then when I turn around and try to make an API call with that same access token, I get a 403 Forbidden error. I tried the same API call in my browser (after logging in to Asana), and it works fine, which leads me to believe that I do have access to the resource, I must have an issue with authorizing the request on my end.
The API call in question is (documented here): https://app.asana.com/api/1.0/workspaces.
My C# code is as follows (abbreviated to relevant parts, and assume that ACCESS_TOKEN contains the access token I got from the token exchange endpoint):
HttpClient client = new HttpClient();
client.BaseAddress = "https://app.asana.com/api/1.0";
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", ACCESS_TOKEN);
client.DefaultRequestHeaders.Add("Accept", "application/json");
And then I use this HttpClient (named client) in the following function:
// Returns a list of the Asana workspace names for the logged in user.
private async Task<List<string>> GetWorkspacesAsync()
{
List<string> namesList = new List<string>();
// Send the HTTP Request and get a response.
this.UpdateToken(); // Refreshes the token if needed using the refresh token.
using (HttpResponseMessage response = await client.GetAsync("/workspaces"))
{
// Handle a bad (not ok) response.
if (response.StatusCode != HttpStatusCode.OK)
{
// !!!THIS KEEPS TRIGGERING WITH response.StatusCode AS 403 Forbidden!!!
// Set up a stream reader to read the response.
// This is for TESTING ONLY
using (StreamReader reader = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
// Extract the json object from the response.
string content = reader.ReadToEnd();
Debug.WriteLine(content);
}
throw new HttpRequestException("Bad HTTP Response was returned.");
}
// If execution reaches this point, the Http Response returned with code OK.
// Set up a stream reader to read the response.
using (StreamReader reader = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
// Extract the json object from the response.
string content = reader.ReadToEnd();
JsonValue responseJson = JsonValue.Parse(content);
foreach (JsonValue workspaceJson in responseJson["data"])
{
string workspaceName = workspaceJson["name"];
Debug.WriteLine("Workspace Name: " + workspaceName);
namesList.Add(workspaceName);
}
}
}
// I have other awaited interactions with app storage in here, hence the need for the function to be async.
return namesList;
}
Finally found the answer. It looks like I was using HttpClient incorrectly; a subtle thing that should be equivalent, but is not due to the way it is implemented.
The answer
I needed to place the final slash at the end of the BaseAddress property of HttpClient, and NOT at the start of the relative address for the specific request. This answered question explains this.
To fix my code
I needed to change the setting up of the BaseAddress:
HttpClient client = new HttpClient();
client.BaseAddress = "https://app.asana.com/api/1.0/"; // FINAL SLASH NEEDED HERE
And remove the slash from the request's relative address:
// DO NOT put slash before relative address "workspaces" here
using (HttpResponseMessage response = await client.GetAsync("workspaces"))
Why I got the original error
When HttpClient combined the BaseAddress with the relative URI I specified in GetAsync(), it dropped off some of the base address, since the final slash was not included. The resulting address from combining the BaseAddress with the relative URI was a valid URL, but not a valid page/API call in Asana. Asana thus did an automatic redirect to a login page, which, of course, the rest of the API call would be forbidden from there.
How I discovered this
In debugging, I grabbed the access token returned during my app's authorization with Asana. I then recreated the request to the "/workspaces" API myself in Postman, and the request worked as expected. This confirmed that my authorization worked fine, and the issue must be with the specific request rather than the authorization. In debugging I then looked into the HttpResponseMessage, which has a property called RequestMessage, that includes the actual URL the GetAsync() made the request against. I observed the Login URL from Asana, rather than the BaseAddress I specified... which led me to the question/
answer linked above.
Hope this explanation helps anyone who comes across a similar error!

Sharepoint online API returns: HTTP Error 400. A request header field is too long

I have a loop that will loop through records in my DB, pulling information i need and then creating 3 folders & upload a file.
This works OK for like 40 records but then it starts erroring out with the below response back from sharepoint: <!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Bad Request</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Bad Request - Header Field Too Long</h2>\r\n<hr><p>HTTP Error 400. A request header field is too long.</p>\r\n</BODY></HTML>
I am not sure whats going on, i read online its todo with cookies but i am using HTTPClient to send the request so i dont know how that would effect it? I also seen onlne about changing the kestrel?
Can anybody shed some light on this for me? Provide me with an easy but working solution? I dont use CSOM for integrating to sharepoint online, i use HTTP Requests, below is a sample of how i interact with sharepoint.
It seems as if i get blocked or banned temporarily cause if i wait a good bit, i can then make the same request that failed previously, and it will work! So strange.
Sample code (Used to create a resource at Sharepoint):
//Set Endpoint
var sharePointEndpoint = $"https://{hostname}/sites/{site}/_api/web/folders";
//Set default headers
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", sharePointToken); //Set token
client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
//Pre-Body data setup
var metaData = new MetaDataModel();
metaData.type = "SP.Folder";
//Body data setup
var bodyModel = new ExpandoObject() as IDictionary<string, object>;
bodyModel.Add("__metadata", metaData);
bodyModel.Add("ServerRelativeUrl", location + "/" + directoryName + "/");
//Set content headers
HttpContent strContent = new StringContent(JsonConvert.SerializeObject(bodyModel));
strContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");
strContent.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("odata", "verbose"));
// Send request, grab response
var response = await client.PostAsync(sharePointEndpoint, strContent);
//Return response message
return response;
It turns out I needed to use Content-Length header when sending the request, once done I was able to successfully communicate with sharepoint without encountering this error.
More information here: https://social.technet.microsoft.com/Forums/en-US/26459f1c-945d-4112-9200-69c5a33a37ff/sharepoint-online-rest-api-returns-http-error-400-a-request-header-field-is-too-long?forum=sharepointdevelopment
Thanks.

Using Azure Hybrid Connection to connect to internal Web API

Very new to Azure, and I have an internal web API on an internal address http://internal-server:182/api/policies. I have set up a Hybrid Connection internal-service.servicebus.windows.net. This is connected and working.
My struggle is getting the C# code working to connect and retrieve the data. After a number of days, I have reviewed various articles, videos etc and all seem more advanced than what I am trying to do, which is just call the Web API and read the JSON. I have tried to simplify the code but receive the error:
401 MalformedToken: Invalid authorization header: The request is missing WRAP authorization credentials.
At present I have the followed code:
using (var client = new HttpClient())
{
var url = "http://internal-service.servicebus.windows.net";
var tp = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", "<key goes here>");
var token = tp.GetWebTokenAsync(url, string.Empty, true, TimeSpan.FromHours(1))
.GetAwaiter()
.GetResult();
client.BaseAddress = new Uri(url);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("ServiceBusAuthorization", token);
var response = client.GetAsync("/api/policies").Result;
string res = "";
using (HttpContent content = response.Content)
{
// ... Read the string.
Task<string> result = content.ReadAsStringAsync();
res = result.Result;
Label1.Text = res;
}
}
Any help or direction would be much appreciated? Once this code is working the Web App will be published as an Azure Web App.
Seems that your are not sending the right header.
First suggestion: intercept the call with a proxy like fiddler, to do that add a proxy config to your call to localhost port 8888, after this you can try some request and see the raw http you are sending to the server, you can also modify it until it works, once you have this modify your code until it send the same raw http.
You can find more info about this here:
Microsoft Azure CreateQueue using Simple REST Client
https://github.com/ytechie/event-hubs-sas-generator

WebAPI using PostAsync throws exception only when filesize is large

I am trying to send a file to a WebAPI controller that does some processing with the file on a server. Everything seems to work well until I tried files that are large than 2mb... files large than this seem to be throwing an odd exception.
Here is the snippet:
var progress = new ProgressMessageHandler();
progress.HttpSendProgress += ProgressEventHandler;
HttpClient client = HttpClientFactory.Create(progress);
client.Timeout = TimeSpan.FromMinutes(20);
try
{
using (
var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read, 1024,
useAsync: true))
{
var content = new StreamContent(fileStream, 1024);
var address = new Uri(string.Format("{0}api/File/Upload?submittalId={1}&fileName={2}&documentTypeId={3}", FileServiceUri, tabTag.submittalId, Path.GetFileName(file), documentTypeId));
client.MaxResponseContentBufferSize = 2147483647;
var response = await client.PostAsync(address, content);
var result = response.Content.ReadAsAsync<object>();
if (!response.IsSuccessStatusCode)
continue;
}
The exception is thrown on the line:
var response = await client.PostAsync(address, content);
and is:
No MediaTypeFormatter is available to read an object of type 'Object' from content with media type 'text/html'
It's not even hitting the breakpoint at the beginning of my service controller so I didnt include that code(although I can if thats potentially an issue). As I said above, this ONLY happens with files > 2mb -- small files work just fine(thank god so I have something to show for a demo ^^).
Anyhelp with this would be greatly appreciated.
Cory's observation is right that Web API doesn't have a in-built formatter to either serialize or deserialize text/html content. My guess is that you are most probably getting an error response in html. If its indeed that, you can do the following:
When uploading files to a IIS hosted Web API application, you need to take care of the following stuff.
You need to look for the following 2 settings in Web.config to increase the upload size:
NOTE(maxRequestLength="size in Kilo bytes"):
<system.web> <httpRuntime targetFramework="4.5" maxQueryStringLength="" maxRequestLength="" maxUrlLength="" />
NOTE(maxAllowedContentLength is in bytes):
<system.webServer> <security> <requestFiltering> <requestLimits maxAllowedContentLength="" maxQueryString="" maxUrl=""/>
Also note that the default buffer policy of Web API in IIS hosted scenarios is buffered, so if you are uploading huge files, your request would be consuming lot of memory. To prevent that you can change the policy like the following:
config.Services.Replace(typeof(IHostBufferPolicySelector), new CustomBufferPolicySelector());
//---------------
public class CustomBufferPolicySelector : WebHostBufferPolicySelector
{
public override bool UseBufferedInputStream(object hostContext)
{
return false;
}
}
The response is coming back with a text/html Content-Type, and ReadAsAsync<object>() doesn't know how to deserialize text/html into an object.
Likely, your web app is configured to only accept files up to a certain size and is returning an error with a friendly HTML message. You should be checking the response code before trying to deserialize the content:
var response = await client.PostAsync(address, content);
response.EnsureSuccessStatusCode();

Categories

Resources