I have a REST controller that streams the response in csv format using a helper method like below:
public static void CsvStreamHelper(IEnumerable<T> data, Stream stream)
{
using (var writer = new StreamWriter(stream))
{
foreach (var line in data)
{
// format csv lines here
writer.WriteLine(lineString);
}
writer.Flush();
}
}
Then, I'm using this in my Controller like:
public Task<IActionResult> MyController()
{
var data = // Get data here.
CsvStreamHelper(data, this.HttpContext.Response.Body);
return new EmptyResult();
}
This is working fine. However, I would like to use content negotiation middleware with custom formatter like here while continue to stream response.
I can override WriteResponseBodyAsync method using my helper method. What I'm unsure about is if I use it in my Rest controller like this.Ok(data), instead of streaming the response, it will just build the response and send it in one chunk. How can I achieve streaming response with content negotiation middleware?
Related
I am using Insomnia for testing an API, but the same happens with Postman.
I want to test a file upload, with the following controller:
public async Task<IActionResult> Post([FromForm]IFormFile File)
If I set the request as a multipart request:
it works.
However, if I set it as a binary file:
I don't know how to get the data. How can it be done?
Also, in the controller method's signature, if I change [FromForm] to [FromBody], I'm not getting data.
Can someone clarify this for me?
As you've noticed already, using binary file option in Postman/Insomnia doesn't work the standard way. There are three different ways to upload file via RESTful API, and you have to choose one.
I've included code snippets that read the uploaded file contents to a string and output it -- try sending a text file, and you should get the contents of the file in the 200 response.
Form-data upload
This is the most popular/well-known upload method formatting the data you send as a set of key/value pairs. You normally need to specify Content-Type to multipart/form-data in the request, and then use [FromForm] attribute in MVC to bind values to variables. Also, you can use the built-in IFormFile class to access the file uploaded.
[HttpPost]
public async Task<IActionResult> PostFormData([FromForm] IFormFile file)
{
using (var sr = new StreamReader(file.OpenReadStream()))
{
var content = await sr.ReadToEndAsync();
return Ok(content);
}
}
Body upload
You can send body in the format that MVC understands, e.g. JSON, and embed the file inside it. Normally, the file contents would be encoded using Base64 or other encoding to prevent character encoding/decoding issues, especially if you are sending images or binary data. E.g.
{
"file": "MTIz"
}
And then specify [FromBody] inside your controller, and use class for model deserialization.
[HttpPost]
public IActionResult PostBody([FromBody] UploadModel uploadModel)
{
var bytes = Convert.FromBase64String(uploadModel.File);
var decodedString = Encoding.UTF8.GetString(bytes);
return Ok(decodedString);
}
// ...
public class UploadModel
{
public string File { get; set; }
}
When using large and non-text files, the JSON request becomes clunky and hard to read though.
Binary file
The key point here is that your file is the whole request. The request doesn't contain any additional info to help MVC to bind values to variables in your code. Therefore, to access the file, you need to read Body in the Request.
[HttpPost]
public async Task<IActionResult> PostBinary()
{
using (var sr = new StreamReader(Request.Body))
{
var body = await sr.ReadToEndAsync();
return Ok(body);
}
}
Note: the example reads Body as string. You may want to use Stream or byte[] in your application to avoid file data encoding issues.
In addition of the above, in case of multipart file conversion to base64String you can refer to the below:
if (File.Length> 0)
{
using (var ms = new MemoryStream())
{
File.CopyTo(ms);
var fileBytes = ms.ToArray();
string s = Convert.ToBase64String(fileBytes);
}
}
Note: I am using this code for .NET CORE 2.1
I'm trying to POST a MultipartFormDataContent in a real case scenario, a data content object could contain anything from a simple string to a video file I'm using a serialized object down there, just a proof of concept.
Also I would like to note that using JSON objects wont serve my real life scenarios
public class GzipMultipartContent : MultipartFormDataContent
{
public GzipMultipartContent()
{
Headers.ContentEncoding.Add("gzip");
}
protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
return Task.Factory.StartNew(() =>
{
using (var gzip = new GZipStream(stream, CompressionMode.Compress, true))
base.SerializeToStreamAsync(gzip, context);
});
}
}
and here's how I call it
var gzipped = new GzipMultipartContent();
var test = new TestClass();
gzipped.Add(new StringContent(JsonConvert.SerializeObject(test)), "value");
var client = new HttpClient();
var result = client.PostAsync("http://localhost:60001/api/Home/", gzipped).Result;
and here's the post action in the controller
// POST: api/Home
[HttpPost]
public void Post([FromForm] object value)
{
}
I have added a break point at the server side and made sure it doesn't even reach the Post method, also I have tried with a normal POST request to make sure that it's not a server configuration problem or a URL mistyping
Client side
If the code in question is your real code, then there are at least two issues:
Did not wait on base.SerializeToStreamAsync
You created a new task, but you did not wait until the base class completed writing to the compressed stream in the task. So you could send unpredictable content to server.
Did not override Content-Length
MultipartFormDataContent calculates length of content based on data not compressed, since you have compressed data, you must re-compute length for the compressed data.
Frankly, I don't think you need to inherit from MultipartFormDataContent to make it compressed. Instead, you could compress the entire MultipartFormDataContent in a wrapper HttpContent:
public class GzipCompressedContent : HttpContent
{
private readonly HttpContent _content;
public GzipCompressedContent(HttpContent content)
{
// Copy original headers
foreach (KeyValuePair<string, IEnumerable<string>> header in content.Headers)
{
Headers.TryAddWithoutValidation(header.Key, header.Value);
}
Headers.ContentEncoding.Add("gzip");
_content = content;
}
protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
using (var gzip = new GZipStream(stream, CompressionMode.Compress, true))
{
// Compress the entire original content
await _content.CopyToAsync(gzip);
}
}
protected override bool TryComputeLength(out long length)
{
// Content-Lenght is optional, so set to -1
length = -1;
return false;
}
}
And use it:
var test = new TestClass();
using (var client = new HttpClient())
{
var form = new MultipartFormDataContent();
form.Add(new StringContent(JsonConvert.SerializeObject(test)), "value");
var compressed = new GzipCompressedContent(form);
var result = await client.PostAsync(..., compressed);
}
Server side
Your server needs to support compressed stream.
For example, by default, ASP.NET Core does not support compressed request, if you send GZip compressed request to an ASP.NET Core application, you will see exception:
System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.
at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
The above exception happens in action invocation pipeline before any controller action takes place. So controller actions in this case could not be reached.
To fix such, you will need to enable server side request decompression support.
If you are using ASP.NET Core, check out this nuget package.
I am not sure if I understood the issue, but if it is that your request isn't getting to the server, while your "normal" POST requests are, then I think that I found your problem.
I think that the issue is that your server doesn't know what Content-Type is coming to it. I literally copy-pasted your code, but added
Headers.ContentType = new MediaTypeHeaderValue("application/x-gzip");
to GzipMultipartContent.cs ctor.
After I added the type, I was hitting my breakpoint in the localhost server.
Source: Content-Type
In requests, (such as POST or PUT), the client tells the server what type of data is actually sent.
I have some the following function that formats some data into csv:
public static void WriteToStream(IEnumerable<MyClass> data, Stream stream)
{
var writer = new StreamWriter(stream);
foreach(var record in data)
{
var line = // code that generates the csv line.
writer.WriteLine(line);
}
}
In my REST controller method, I'm trying to send this stream as a response. I'm trying the following:
[HttpGet]
public async Task<IActionResult> GetCsvStream()
{
var data = // code to get raw data.
var stream = new MemoryStream();
WriteToStream(data, stream);
return new FileStreamResult(stream, "text/csv");
}
However, I'm getting Internal Server Error with this code. What I would like to achieve is to continuously stream data as csv.
EDIT:
By the way, I'm currently doing it as below as John suggested. But, does this actually stream data or does it write all the content to response body first and then sends the final response? My understanding is it's the latter.
[HttpGet]
public async Task<IActionResult> GetCsvStream()
{
var data = // code to get raw data.
WriteToStream(data, this.HttpContext.Response.Body);
return this.Ok();
}
I am using Insomnia for testing an API, but the same happens with Postman.
I want to test a file upload, with the following controller:
public async Task<IActionResult> Post([FromForm]IFormFile File)
If I set the request as a multipart request:
it works.
However, if I set it as a binary file:
I don't know how to get the data. How can it be done?
Also, in the controller method's signature, if I change [FromForm] to [FromBody], I'm not getting data.
Can someone clarify this for me?
As you've noticed already, using binary file option in Postman/Insomnia doesn't work the standard way. There are three different ways to upload file via RESTful API, and you have to choose one.
I've included code snippets that read the uploaded file contents to a string and output it -- try sending a text file, and you should get the contents of the file in the 200 response.
Form-data upload
This is the most popular/well-known upload method formatting the data you send as a set of key/value pairs. You normally need to specify Content-Type to multipart/form-data in the request, and then use [FromForm] attribute in MVC to bind values to variables. Also, you can use the built-in IFormFile class to access the file uploaded.
[HttpPost]
public async Task<IActionResult> PostFormData([FromForm] IFormFile file)
{
using (var sr = new StreamReader(file.OpenReadStream()))
{
var content = await sr.ReadToEndAsync();
return Ok(content);
}
}
Body upload
You can send body in the format that MVC understands, e.g. JSON, and embed the file inside it. Normally, the file contents would be encoded using Base64 or other encoding to prevent character encoding/decoding issues, especially if you are sending images or binary data. E.g.
{
"file": "MTIz"
}
And then specify [FromBody] inside your controller, and use class for model deserialization.
[HttpPost]
public IActionResult PostBody([FromBody] UploadModel uploadModel)
{
var bytes = Convert.FromBase64String(uploadModel.File);
var decodedString = Encoding.UTF8.GetString(bytes);
return Ok(decodedString);
}
// ...
public class UploadModel
{
public string File { get; set; }
}
When using large and non-text files, the JSON request becomes clunky and hard to read though.
Binary file
The key point here is that your file is the whole request. The request doesn't contain any additional info to help MVC to bind values to variables in your code. Therefore, to access the file, you need to read Body in the Request.
[HttpPost]
public async Task<IActionResult> PostBinary()
{
using (var sr = new StreamReader(Request.Body))
{
var body = await sr.ReadToEndAsync();
return Ok(body);
}
}
Note: the example reads Body as string. You may want to use Stream or byte[] in your application to avoid file data encoding issues.
In addition of the above, in case of multipart file conversion to base64String you can refer to the below:
if (File.Length> 0)
{
using (var ms = new MemoryStream())
{
File.CopyTo(ms);
var fileBytes = ms.ToArray();
string s = Convert.ToBase64String(fileBytes);
}
}
Note: I am using this code for .NET CORE 2.1
i tried to send a basic file from my integration test project in C# to a web api .
But i don't know why, each call i get an exception .
Json.JsonSerializationException : Error getting value from 'ReadTimeout' on 'System.Io.FileStream'
I found this property can't be read , so maybe that why my httpclient can't serialize it.
So how can i send a file to a web api ?
This is my code from the client:
using (StreamReader reader = File.OpenText("SaveMe.xml"))
{
response = await client.PostAsJsonAsync($"api/registration/test/", reader.BaseStream);
response.EnsureSuccessStatusCode();
}
And my controller:
[Route("api/registration")]
public class RegistrationController : Controller
{
[HttpPost, Route("test/")]
public ActionResult AddDoc(Stream uploadedFile)
{
if (uploadedFile != null)
{
return this.Ok();
}
else
{
return this.NotFound();
}
}
Here the screenShot we can see , the property [ReadTimeout] can't be access.
I'm not sure if they still support PostAsJsonAsync in .NET Core, which I am on. So I decided to rewrite your snippet as follows using PostAsync:
using (StreamReader reader = File.OpenText("SaveMe.xml"))
{
var response = await client.PostAsync($"api/registration/test/", new StreamContent(reader.BaseStream));
}
Update your API method to look like:
[Route("api/registration")]
public class RegistrationController : Controller
{
[HttpPost, Route("test/")]
public ActionResult AddDoc()
{
//Get the stream from body
var stream = Request.Body;
//Do something with stream
}
First you have to read all the data from file and only then send it. To open the .xml files use XmlReader. look here Reading Xml with XmlReader in C#