Develop Reference

Using HttpClient to post large MultipartFormDataContent to IIS server web page

I am trying to upload up to 2GB of data using a HttpClient.
The data is sent through the request body into an aspx page where it is read (horrible, I know but I cannot change this.)
The data is placed in a MultipartFormDataContent and posted like this:
var apiRequest = new MultipartFormDataContent();
apiRequest.Add(new StreamContent(file), fileName, fileName);
apiRequest.Headers.ContentDisposition =
new ContentDispositionHeaderValue("form-data") { Name = fileName, FileName = fileName };
HttpClient client = new HttpClient();
client.Timeout = TimeSpan.FromMinutes(30);
HttpResponseMessage response = null;
try
{
response = client.PostAsync(apiEndPoint, form).Result;
response.EnsureSuccessStatusCode();
string responseBody = response.Content.ReadAsStringAsync().Result;
}
catch (HttpRequestException e)
{
log.LogError($"logging here");
}
Things I have tried :
-HttpClient http version 1.0 instead of default
-HttpClient MaxRequestContentBufferSize
-web.config maxAllowedContentLength
-web.config AspMaxRequestEntityAllowed
-web.config maxRequestLength
Currently, the files get added to the httpClient correctly but I cannot get them to post to the web app. I got up to 900MB through but anything over that simply redirects to the main page and I get HTML from the web app in the response body.
Thanks in advance!

After a lot of hair pulling we found the solution by looking at the IIS logs.
The logs revealed that the Microsoft URLScan tool was blocking the requests.
When the request body was not approved by the scan, IIS would redirect you straight to the main page with no error.
You have to configure a max request length in the urlscan.ini file.
More info here: https://ajaxuploader.com/large-file-upload-iis-debug.htm
The file is located at C:\WINDOWS\system32\inetsrv\urlscan

Any added headers to DefaultRequestHeaders will not make it through to the API

In my API project I have the following controller that works fine when called with Postman:
[HttpPost]
public async Task<ActionResult> Upload([FromHeader] string authorization, IFormFile payLoad) { ... }
When I use Postman, I add a string (Token) in the Auth section and specify the Type as Bearer Token. I then go to the Body section and set the Key payLoad as a File and choose a file to upload.
Postman generates C# - RestSharp code as follows:
var client = new RestClient("http://localhost:11764/api/logdata");
client.Timeout = -1;
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer exampleTokenString");
request.AddFile("payLoad", "/C:/path/oqwi.zip");
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);
I'm not using RestSharp so I haven't verified that the code above works but the Postman post itself within the tool works fine and my API gets all the data as I would expect.
In a separate client application, whenever I attempt to make a POST call, the API controller (at the top of this page) always receives null for the authorization parameter. The file loads in fine. Here is the client code trying to POST to the API with every example I attempted to add the header (I did not try them all at once):
Uri EndPoint = new Uri("http://localhost:11764/api/logdata");
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization
= new AuthenticationHeaderValue("Bearer", AccessToken);
client.DefaultRequestHeaders.Add("Authorization", "Bearer " + AccessToken);
var request = new HttpRequestMessage(HttpMethod.Post, EndPoint)
{
Content = fileAsFormData
};
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", AccessToken);
request.Headers.Add("Authorization", "Bearer " + AccessToken);
var response = await client.SendAsync(request);
...
Note: The above is an example of every different attempt I made at adding the authorization and token; I didn't do all of the above at once.
I have inspected the object in the client during runtime and it appears as if the header(s) are added on where I would expect them to be. Using Fiddler, I can confirm this:
UPDATE: I've tried adding other headers, like CacheControl, and none of them make it through to the API. I see it on the client side during runtime, I see it in Fiddler, but then they're all scrubbed by the time they get to the API. I'm wondering if this Github discussion has anything to do with it:
https://github.com/dotnet/runtime/issues/26475

According to this Github ticket within the dotnet runtime team,
karelz commented on Oct 24, 2018 •
FYI: 2 weeks ago we released a security fix to remove Authorization request headers from redirects.
Thinking that redirects may be at the heart of the issue, by using Fiddler or the Visual Studio inspection tool, I was able to observe that the callback to my client was from https://localhost:5001. I was not expecting this...
When I was constructing the original client code to POST to my API, I was simply copying all of the values I had used when I was exercising these calls from Postman. As an example from Postman's C# RestSharp:
var client = new RestClient("http://localhost:11764/api/logdata");
This ultimately was a red herring caused by the intelligent way Postman handles redirects. Postman was indeed posting to http://localhost:11764 -- but then getting a secure redirect to https://localhost:5001. Postman would then subtly resend the original request with reattached headers to this new secure endpoint.
So after updating the endpoint that the client will POST to, from http://localhost:11764/api/logdata to https://localhost:5001/api/logdata, everything works as expected.
But why https://localhost:5001? It's setup this way in (most) launchSettings.json:
"MyProj.UploadApi.WebApi": {
"commandName": "Project",
"launchBrowser": false,
"environmentVariables": {
"ASPNETCORE_ENVIRONMENT": "Development"
},
"applicationUrl": "https://localhost:5001;http://localhost:11764"
}

Sharepoint online API returns: HTTP Error 400. A request header field is too long

I have a loop that will loop through records in my DB, pulling information i need and then creating 3 folders & upload a file.
This works OK for like 40 records but then it starts erroring out with the below response back from sharepoint: <!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"http://www.w3.org/TR/html4/strict.dtd\">\r\n<HTML><HEAD><TITLE>Bad Request</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Bad Request - Header Field Too Long</h2>\r\n<hr><p>HTTP Error 400. A request header field is too long.</p>\r\n</BODY></HTML>
I am not sure whats going on, i read online its todo with cookies but i am using HTTPClient to send the request so i dont know how that would effect it? I also seen onlne about changing the kestrel?
Can anybody shed some light on this for me? Provide me with an easy but working solution? I dont use CSOM for integrating to sharepoint online, i use HTTP Requests, below is a sample of how i interact with sharepoint.
It seems as if i get blocked or banned temporarily cause if i wait a good bit, i can then make the same request that failed previously, and it will work! So strange.
Sample code (Used to create a resource at Sharepoint):
//Set Endpoint
var sharePointEndpoint = $"https://{hostname}/sites/{site}/_api/web/folders";
//Set default headers
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", sharePointToken); //Set token
client.DefaultRequestHeaders.Add("Accept", "application/json;odata=verbose");
//Pre-Body data setup
var metaData = new MetaDataModel();
metaData.type = "SP.Folder";
//Body data setup
var bodyModel = new ExpandoObject() as IDictionary<string, object>;
bodyModel.Add("__metadata", metaData);
bodyModel.Add("ServerRelativeUrl", location + "/" + directoryName + "/");
//Set content headers
HttpContent strContent = new StringContent(JsonConvert.SerializeObject(bodyModel));
strContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");
strContent.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("odata", "verbose"));
// Send request, grab response
var response = await client.PostAsync(sharePointEndpoint, strContent);
//Return response message
return response;

It turns out I needed to use Content-Length header when sending the request, once done I was able to successfully communicate with sharepoint without encountering this error.
More information here: https://social.technet.microsoft.com/Forums/en-US/26459f1c-945d-4112-9200-69c5a33a37ff/sharepoint-online-rest-api-returns-http-error-400-a-request-header-field-is-too-long?forum=sharepointdevelopment
Thanks.

Using Azure Hybrid Connection to connect to internal Web API

Very new to Azure, and I have an internal web API on an internal address http://internal-server:182/api/policies. I have set up a Hybrid Connection internal-service.servicebus.windows.net. This is connected and working.
My struggle is getting the C# code working to connect and retrieve the data. After a number of days, I have reviewed various articles, videos etc and all seem more advanced than what I am trying to do, which is just call the Web API and read the JSON. I have tried to simplify the code but receive the error:
401 MalformedToken: Invalid authorization header: The request is missing WRAP authorization credentials.
At present I have the followed code:
using (var client = new HttpClient())
{
var url = "http://internal-service.servicebus.windows.net";
var tp = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", "<key goes here>");
var token = tp.GetWebTokenAsync(url, string.Empty, true, TimeSpan.FromHours(1))
.GetAwaiter()
.GetResult();
client.BaseAddress = new Uri(url);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("ServiceBusAuthorization", token);
var response = client.GetAsync("/api/policies").Result;
string res = "";
using (HttpContent content = response.Content)
{
// ... Read the string.
Task<string> result = content.ReadAsStringAsync();
res = result.Result;
Label1.Text = res;
}
}
Any help or direction would be much appreciated? Once this code is working the Web App will be published as an Azure Web App.

Seems that your are not sending the right header.
First suggestion: intercept the call with a proxy like fiddler, to do that add a proxy config to your call to localhost port 8888, after this you can try some request and see the raw http you are sending to the server, you can also modify it until it works, once you have this modify your code until it send the same raw http.
You can find more info about this here:
Microsoft Azure CreateQueue using Simple REST Client
https://github.com/ytechie/event-hubs-sas-generator

WebAPI using PostAsync throws exception only when filesize is large

I am trying to send a file to a WebAPI controller that does some processing with the file on a server. Everything seems to work well until I tried files that are large than 2mb... files large than this seem to be throwing an odd exception.
Here is the snippet:
var progress = new ProgressMessageHandler();
progress.HttpSendProgress += ProgressEventHandler;
HttpClient client = HttpClientFactory.Create(progress);
client.Timeout = TimeSpan.FromMinutes(20);
try
{
using (
var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read, 1024,
useAsync: true))
{
var content = new StreamContent(fileStream, 1024);
var address = new Uri(string.Format("{0}api/File/Upload?submittalId={1}&fileName={2}&documentTypeId={3}", FileServiceUri, tabTag.submittalId, Path.GetFileName(file), documentTypeId));
client.MaxResponseContentBufferSize = 2147483647;
var response = await client.PostAsync(address, content);
var result = response.Content.ReadAsAsync<object>();
if (!response.IsSuccessStatusCode)
continue;
}
The exception is thrown on the line:
var response = await client.PostAsync(address, content);
and is:
No MediaTypeFormatter is available to read an object of type 'Object' from content with media type 'text/html'
It's not even hitting the breakpoint at the beginning of my service controller so I didnt include that code(although I can if thats potentially an issue). As I said above, this ONLY happens with files > 2mb -- small files work just fine(thank god so I have something to show for a demo ^^).
Anyhelp with this would be greatly appreciated.

Cory's observation is right that Web API doesn't have a in-built formatter to either serialize or deserialize text/html content. My guess is that you are most probably getting an error response in html. If its indeed that, you can do the following:
When uploading files to a IIS hosted Web API application, you need to take care of the following stuff.
You need to look for the following 2 settings in Web.config to increase the upload size:
NOTE(maxRequestLength="size in Kilo bytes"):
<system.web> <httpRuntime targetFramework="4.5" maxQueryStringLength="" maxRequestLength="" maxUrlLength="" />
NOTE(maxAllowedContentLength is in bytes):
<system.webServer> <security> <requestFiltering> <requestLimits maxAllowedContentLength="" maxQueryString="" maxUrl=""/>
Also note that the default buffer policy of Web API in IIS hosted scenarios is buffered, so if you are uploading huge files, your request would be consuming lot of memory. To prevent that you can change the policy like the following:
config.Services.Replace(typeof(IHostBufferPolicySelector), new CustomBufferPolicySelector());
//---------------
public class CustomBufferPolicySelector : WebHostBufferPolicySelector
{
public override bool UseBufferedInputStream(object hostContext)
{
return false;
}
}

The response is coming back with a text/html Content-Type, and ReadAsAsync<object>() doesn't know how to deserialize text/html into an object.
Likely, your web app is configured to only accept files up to a certain size and is returning an error with a friendly HTML message. You should be checking the response code before trying to deserialize the content:
var response = await client.PostAsync(address, content);
response.EnsureSuccessStatusCode();