Develop Reference

.NET Core 3 MVC download a file from HTTP and re-deliver to client using minimum memory

I have a .NET Core 3 MVC app that needs to read a file from one location over HTTP and then re-deliver it back out to the response. Some of these files will be ~200MB in size.
What I have works, but it reads the whole file into memory before sending the File result out to the client. Is there a way to make it essentially a passthrough where the read stream flows into the response stream so that very little memory is required on the server?
This is what I have now but I do not think will perform well with large files:
if (requestedFile != null)
{
using (var client = new System.Net.Http.HttpClient())
{
using (var result = await client.GetAsync(requestedFile.DownloadUrl))
{
if (result.IsSuccessStatusCode)
{
var bytes = await result.Content.ReadAsByteArrayAsync();
return File(bytes, "application/zip", "largefile.zip");
}
}
}
}
I have also tried this which results in a runtime error of "Cannot access a closed Stream":
using (var client = new System.Net.Http.HttpClient())
{
using (var httpResponseMessage = await client.GetAsync(requestedFile.DownloadUrl))
{
return File(await httpResponseMessage.Content.ReadAsStreamAsync(), "application/zip", "largefile.zip");
}
}
Edit:
Solution after some trial and error was remocing all using statements and letting the FileStreamResult close the stream on its own. So I ended up with:
var client = new HttpClient();
var result = await client.GetAsync(requestedFile.DownloadUrl);
var stream = await result.Content.ReadAsStreamAsync();
return new FileStreamResult(stream, "application/zip")
{
FileDownloadName = "largefile.zip"
};

One of the overloads for File is a Stream. Just get that URL as a Stream or read the response body as a stream and immediately return that in the overload:
var client = new System.Net.Http.HttpClient();
var result = await client.GetAsync(requestedFile.DownloadUrl);
var stream = await result.Content.ReadAsStreamAsync();
return File(stream,"application/pdf", "Invoice.pdf");
Note: this will fail if you wrap the Stream in a using block as the FileResult already closes the Stream.

For Export Excel
var client = new System.Net.Http.HttpClient();
var comp = client.GetAsync($"RewardEmployee/ExportExcelCalculate?rewardConfigId={id}").Result;
`var stream = await result.Content.ReadAsStreamAsync();
return File(stream, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"ExportExcelCalculateRewardEmployee.xlsx");

Nancy: How to log body of response in Nancy

In case we search for a failure in our application, we want to add the possibility to log all request and responses given by our Nancy service. In order to accomplish this, we added some pipeline hooks.
But unfortunately I'm not able to get the body of our responses. When I execute the delegate that renders the contents to the response stream (Contents), my response stream (ms) is disposed.
public void LogResponse(NancyContext context) {
string CreateResponseMessage() {
return $"Response (Client: {context.Request.UserHostAddress}) [{context.Request.Method}] {context.Request.Url}, " +
$"Status code: {(int) context.Response.StatusCode} {context.Response.StatusCode}";
}
string CreateContentMessage() {
using (var ms = new MemoryStream())
{
// For some reason, when executing this line, my ms is disposed
context.Response.Contents(ms);
ms.Seek(0, SeekOrigin.Begin);
using (var sr = new StreamReader(ms)) {
var headers = FormatHeaders(context.Response.Headers);
var content = sr.ReadToEnd();
return
$"Headers:{NewLine}{headers}{NewLine}" +
$"Body:{NewLine} {content}";
}
}
}
Log.Info(CreateResponseMessage());
Log.Verbose(CreateContentMessage);
}
Is this the wrong way to get the content of the body? Is there a way at all?

Not able to download file using C# from a web api which works using postman

I have recently started working with web api's.
I need to download a file in C# project from a web api, which works fine when I hit the web api using postman's send and download option. Refer to the image, also please check the response in header's tab. This way, I am able to directly download the file to my computer.
I want to do the same from my C# project, I found following two links which shows how to download a file from web api.
https://code.msdn.microsoft.com/HttpClient-Downloading-to-4cc138fd
http://blogs.msdn.com/b/henrikn/archive/2012/02/16/downloading-a-google-map-to-local-file.aspx
I am using the following code in C# project to get the response:
private static async Task FileDownloadAsync()
{
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Add("Accept", "text/html");
try
{
// _address is exactly same which I use from postman
HttpResponseMessage response = await client.GetAsync(_address);
if (response.IsSuccessStatusCode)
{
}
else
{
}
}
catch (Exception)
{
throw;
}
}
}
However I am not getting the response at all (before I can start to convert the response to a file), please check the error message coming:
What am I missing here, any help would be appreciated.

As the (500s) error says - it's the Server that rejects the request. The only thing I see that could cause an issues is the charset encoding. Yours is the default UTF-8. You could try with other encodings.

Below method uses:
SSL certificate (comment out code for cert, if you don't use it)
Custom api header for additional layer of security (comment out Custom_Header_If_You_Need code, if you don't need that)
EnsureSuccessStatusCode will throw an error, when response is not 200. This error will be caught in and converted to a human readable string format to show on your screen (if you need to). Again, comment it out if you don't need that.
private byte[] DownloadMediaMethod(string mediaId)
{
var cert = new X509Certificate2("Keystore/p12_keystore.p12", "p12_keystore_password");
var handler = new WebRequestHandler { ClientCertificates = { cert } };
using (var client = new HttpClient(handler))
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Add("Custom_Header_If_You_Need", "Value_For_Custom_Header");
var httpResponse = client.GetAsync(new Uri($"https://my_api.my_company.com/api/v1/my_media_controller/{mediaId}")).Result;
//Below one line is most relevant to this question, and will address your problem. Other code in this example if just to show the solution as a whole.
var result = httpResponse.Content.ReadAsByteArrayAsync().Result;
try
{
httpResponse.EnsureSuccessStatusCode();
}
catch (Exception ex)
{
if (result == null || result.Length == 0) throw;
using (var ms = new MemoryStream(result))
{
var sr = new StreamReader(ms);
throw new Exception(sr.ReadToEnd(), ex);
}
}
return result;
}
}
Once you have your http response 200, you can use the received bytes[] as under to save them in a file:
using (var fs = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
fs.Write(content, 0, content.Length);
}

Your request header says that you accept HTML responses only. That could be a problem.

How can I send a File over a REST API?

I am trying to send a file to a server over a REST API. The file could potentially be of any type, though it can be limited in size and type to things that can be sent as email attachments.
I think my approach will be to send the file as a binary stream, and then save that back into a file when it arrives at the server. Is there a built in way to do this in .Net or will I need to manually turn the file contents into a data stream and send that?
For clarity, I have control over both the client and server code, so I am not restricted to any particular approach.

I'd recommend you look into RestSharp
http://restsharp.org/
The RestSharp library has methods for posting files to a REST service. (RestRequst.AddFile()). I believe on the server-side this will translate into an encoded string into the body, with the content-type in the header specifying the file type.
I've also seen it done by converting a stream to a base-64 string, and transferring that as one of the properties of the serialized json/xml object. Especially if you can set size limits and want to include file meta-data in the request as part of the same object, this works really well.
It really depends how large your files are though. If they are very large, you need to consider streaming, of which the nuances of that is covered in this SO post pretty thoroughly: How do streaming resources fit within the RESTful paradigm?

You could send it as a POST request to the server, passing file as a FormParam.
#POST
#Path("/upload")
//#Consumes(MediaType.MULTIPART_FORM_DATA)
#Consumes("application/x-www-form-urlencoded")
public Response uploadFile( #FormParam("uploadFile") String script, #HeaderParam("X-Auth-Token") String STtoken, #Context HttpHeaders hh) {
// local variables
String uploadFilePath = null;
InputStream fileInputStream = new ByteArrayInputStream(script.getBytes(StandardCharsets.UTF_8));
//System.out.println(script); //debugging
try {
uploadFilePath = writeToFileServer(fileInputStream, SCRIPT_FILENAME);
}
catch(IOException ioe){
ioe.printStackTrace();
}
return Response.ok("File successfully uploaded at " + uploadFilePath + "\n").build();
}
private String writeToFileServer(InputStream inputStream, String fileName) throws IOException {
OutputStream outputStream = null;
String qualifiedUploadFilePath = SIMULATION_RESULTS_PATH + fileName;
try {
outputStream = new FileOutputStream(new File(qualifiedUploadFilePath));
int read = 0;
byte[] bytes = new byte[1024];
while ((read = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
outputStream.flush();
}
catch (IOException ioe) {
ioe.printStackTrace();
}
finally{
//release resource, if any
outputStream.close();
}
return qualifiedUploadFilePath;
}

Building on to #MutantNinjaCodeMonkey's suggestion of RestSharp. My use case was posting webform data from jquery's $.ajax method into a web api controller. The restful API service required the uploaded file to be added to the request Body. The default restsharp method of AddFile mentioned above caused an error of The request was aborted: The request was canceled. The following initialization worked:
// Stream comes from web api's HttpPostedFile.InputStream
(HttpContext.Current.Request.Files["fileUploadNameFromAjaxData"].InputStream)
using (var ms = new MemoryStream())
{
fileUploadStream.CopyTo(ms);
photoBytes = ms.ToArray();
}
var request = new RestRequest(Method.PUT)
{
AlwaysMultipartFormData = true,
Files = { FileParameter.Create("file", photoBytes, "file") }
};

Detect the file/s being transported with the request.
Decide on a path where the file will be uploaded (and make sure CHMOD 777 exists for this directory)
Accept the client connect
Use ready library for the actual upload
Review the following discussion:
REST file upload with HttpRequestMessage or Stream?

First, you should login to the server and get an access token.
Next, you should convert your file to stream and post the stream:
private void UploadFile(FileStream stream, string fileName)
{
string apiUrl = "http://example.com/api";
var formContent = new MultipartFormDataContent
{
{new StringContent(fileName),"FileName"},
{new StreamContent(stream),"formFile",fileName},
};
using HttpClient httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("Authorization", accessToken);
var response = httpClient.PostAsync(#$"{apiUrl}/FileUpload/save", formContent);
var result = response.Result.Content.ReadAsStringAsync().Result;
}
In this example, we upload the file to http://example.com/api/FileUpload/save and the controller has the following method in its FileUpload controller:
[HttpPost("Save")]
public ActionResult Save([FromForm] FileContent fileContent)
{
// ...
}
public class FileContent
{
public string FileName { get; set; }
public IFormFile formFile { get; set; }
}

Is HttpClient flawed when sending large files/content?

After reading/googling about HttpClient, I have the impression that this component is not suitable for uploading large files or contents to REST services.
It seems that if the upload takes more than the established timeout, the transmission will fail. Does it make sense? What does this timeout means?
Getting progress information seems hard or requires add-ons.
So my questions are: Is it possible to sove these two issues without too much hassle? Otherwise, what's the best approach when working with large contents and REST services?

Yes, if the upload takes longer that the TimeOut, the upload will fail. This is a limitation of HttpClient. The most robust solution to this problem is the one that Thomas Levesque has written an article about, and linked in his comments to your question. You have to use HttpWebRequest instead of HttpClient.
If you want to get progress messages, open the file as a FileStream and manually iterate through it, copying bytes in increments onto the (upload) request stream. As you go, you can calculate your progress relative to the file size.
TL's code example. Be sure to read the article though!:
long UploadFile(string path, string url, string contentType)
{
// Build request
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = WebRequestMethods.Http.Post;
request.AllowWriteStreamBuffering = false;
request.ContentType = contentType;
string fileName = Path.GetFileName(path);
request.Headers["Content-Disposition"] = string.Format("attachment; filename=\"{0}\"", fileName);
try
{
// Open source file
using (var fileStream = File.OpenRead(path))
{
// Set content length based on source file length
request.ContentLength = fileStream.Length;
// Get the request stream with the default timeout
using (var requestStream = request.GetRequestStreamWithTimeout())
{
// Upload the file with no timeout
fileStream.CopyTo(requestStream);
}
}
// Get response with the default timeout, and parse the response body
using (var response = request.GetResponseWithTimeout())
using (var responseStream = response.GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
string json = reader.ReadToEnd();
var j = JObject.Parse(json);
return j.Value<long>("Id");
}
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.Timeout)
{
LogError(ex, "Timeout while uploading '{0}'", fileName);
}
else
{
LogError(ex, "Error while uploading '{0}'", fileName);
}
throw;
}
}