I have a ASP.Net MVC application that is generating xml files dynamically when the user requests (via http request). The only problem is that it will also write them into a file stored locally. So I really have two questions:
Is it possible to not write the file locally but still send the response with the file to the client only
Or can I block a user from typing in the file URL and viewing it on their browser such as example.com/test.xml
I have an authentication system in place to control access of certain views, but I don't know how to do this for something without a view
Current way of writing file and sending response:
[AcceptVerbs(HttpVerbs.Get)]
public void SendXML()
{
if(!User.Identity.IsAuthenticated)
{
return;
}
string path = Server.MapPath("~/Temp/test.xml");
using (Stream s = File.Open(path, FileMode.OpenOrCreate))
{
using (XmlWriter writer = XmlWriter.Create(s, settings))
{
// Write Xml Document
}
}
Response.ContentType = "text/xml";
Response.TransmitFile(path);
}
Current Way of sending request:
var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
if(this.readyState == 4 && this.status == 200){
// Good
}
};
xmlhttp.open("GET", "http://example.com/MyController/SendXML", true);
xmlhttp.send();
Is it possible to not write the file locally but still send the
response with the file to the client only
Yes. It is possible. Once you have the XML in a string. Instead of writing that XML to a file and sending, You can convert the XML string to a byte array and then to an octet stream and send it down from API. This will download the file for the end user and ofcouse you are not storing anything to filesystem
var xml = #"<root></root>";
byte[] xmlBytes = System.Text.Encoding.UTF8.GetBytes(xml);
var stream = new MemoryStream(xmlBytes);
return File(stream, "application/octet-stream", "DownloadName.xml");
Or can I block a user from typing in the file URL and viewing it on
their browser such as example.com/test.xml
It depends, You can have multiple options. You can integrate an authentication mechanism like cookie authentication or token authentication like IdentityServer to facilitate protected endpoints. If you have protected endpoints you can restrict access to only specific users.
When I use Postman to try uploading a large file to my server (written in .NET Core 2.2), Postman immediately shows the HTTP Error 404.13 - Not Found error: The request filtering module is configured to deny a request that exceeds the request content length
But when I use my code to upload that large file, it gets stuck at the line to send the file.
My client code:
public async void TestUpload() {
StreamContent streamContent = new StreamContent(File.OpenRead("D:/Desktop/large.zip"));
streamContent.Headers.Add("Content-Disposition", "form-data; name=\"file\"; filename=\"large.zip\"");
MultipartFormDataContent multipartFormDataContent = new MultipartFormDataContent();
multipartFormDataContent.Add(streamContent);
HttpClient httpClient = new HttpClient();
Uri uri = new Uri("https://localhost:44334/api/user/testupload");
try {
HttpResponseMessage httpResponseMessage = await httpClient.PostAsync(uri, multipartFormDataContent);
bool success = httpResponseMessage.IsSuccessStatusCode;
}
catch (Exception ex) {
}
}
My server code:
[HttpPost, Route("testupload")]
public async Task UploadFile(IFormFileCollection formFileCollection) {
IFormFileCollection formFiles = Request.Form.Files;
foreach (var item in formFiles) {
using (var stream = new FileStream(Path.Combine("D:/Desktop/a", item.FileName), FileMode.Create)) {
await item.CopyToAsync(stream);
}
}
}
My client code gets stuck at the line HttpResponseMessage httpResponseMessage = await httpClient.PostAsync(uri, multipartFormDataContent), while the server doesn't receive any request (I use a breakpoint to ensure that).
It gets stuck longer if the file is bigger. Looking at Task Manager, I can see my client program uses up high CPU and Disk as it is actually uploading the file to the server. After a while, the code moves to the next line which is
bool success = httpResponseMessage.IsSuccessStatusCode
Then by reading the response content, I get exactly the result as of Postman.
Now I want to know how to immediately get the error to be able to notify the user in time, I don't want to wait really long.
Note that when I use Postman to upload large files, my server doesn't receive any request as well. I think I am missing something, maybe there is problem with my client code.
EDIT: Actually I think it is the client-side error. But if it is server-side error, then it still doesn't mean too much for me. Because, let me clear my thought. I want to create this little helper class that I can use across projects, maybe I can share it with my friends too. So I think it should be able, like Postman, to determine the error as soon as possible. If Postman can do, I can too.
EDIT2: It's weird that today I found out Postman does NOT know before hand whether the server accepts big requests, I uploaded a big file and I saw it actually sent the whole file to the server until it got the response. Now I don't believe in myself anymore, why I thought Postman knows ahead of time the error, I must be stupid. But it does mean that I have found a way to do my job even better than Postman, so I think this question might be useful for someone.
Your issue has nothing to do with your server-side C# code. Your request gets stuck because of what is happening between the client and the server (by "server" I mean IIS, Apache, Nginx..., not your server-side code).
In HTTP, most clients don't read response until they send all the request data. So, even if your server discovers that the request is too large and returns an error response, the client will not read that response until the server accepts the whole requests.
When it comes to server-side, you can check this question, but I think it would be more convenient to handle it on the client side, by checking the file size before sending it to the server (this is basically what Postman is doing in your case).
Now I am able to do what I wanted. But first I want to thank you #Marko Papic, your informations do help me in thinking about a way to do what I want.
What I am doing is:
First, create an empty ByteArrayContent request, with the ContentLength of the file I want to upload to the server.
Second, surround HttpResponseMessage = await HttpClient.SendAsync(HttpRequestMessage) in a try-catch block. The catch block catches HttpRequestException because I am sending a request with the length of the file but my actual content length is 0, so it will throw an HttpRequestException with the message Cannot close stream until all bytes are written.
If the code reaches the catch block, it means the server ALLOWS requests with the file size or bigger. If there is no exception and the code moves on to the next line, then if HttpResponseMessage.StatusCode is 404, it means the server DENIES requests bigger than the file size. The case when HttpResponseMessage.StatusCode is NOT 404 will never happen (I'm not sure about this one though).
My final code up to this point:
private async Task<bool> IsBigRequestAllowed() {
FileStream fileStream = File.Open("D:/Desktop/big.zip", FileMode.Open, FileAccess.Read, FileShare.Read);
if(fileStream.Length == 0) {
fileStream.Close();
return true;
}
HttpRequestMessage = new HttpRequestMessage();
HttpMethod = HttpMethod.Post;
HttpRequestMessage.Method = HttpMethod;
HttpRequestMessage.RequestUri = new Uri("https://localhost:55555/api/user/testupload");
HttpRequestMessage.Content = new ByteArrayContent(new byte[] { });
HttpRequestMessage.Content.Headers.ContentLength = fileStream.Length;
fileStream.Close();
try {
HttpResponseMessage = await HttpClient.SendAsync(HttpRequestMessage);
if (HttpResponseMessage.StatusCode == HttpStatusCode.NotFound) {
return false;
}
return true; // The code will never reach this line though
}
catch(HttpRequestException) {
return true;
}
}
NOTE: Note that my approach still has a problem. The problem with my code is the ContentLength property, it shouldn't be exact the length of the file, it should be bigger. For example, if my file is exactly 1000 bytes in length, then if the file is successfully uploaded to the server, the Request that the server gets has greater ContentLength value. Because HttpClient doesn't just only send the content of the file, but it has to send many informations in addition. It has to send the boundaries, content types, hyphens, line breaks, etc... Generally speaking, you should somehow find out before hand the exact bytes that HttpClient will send along with your files to make this approach work perfectly (I still don't know how so far, I'm running out of time. I will find out and update my answer later).
Now I am able to immediately determine ahead of time whether the server can accept requests that are as big as the file my user wants to upload.
I'm having some trouble with finding the right incantation that will allow me to write to a response stream and then later read the contents in a test. Currently I have this
var res = new HttpResponseMessage(System.Net.HttpStatusCode.OK);
var ms = new MemoryStream();
res.Content = new StreamContent(ms);
using (var sw = new StreamWriter(ms, System.Text.Encoding.UTF8))
using (var csv = new CsvHelper.CsvWriter(sw))
csv.WriteRecords(allData.ToList());
return res;
In my test I'm trying to read this response
var controller = appContainer().Resolve<MyController>();
var res = (await controller.Get()) as HttpResponseMessage;
res.ShouldNotEqual(null);
var csv = await res.Content.ReadAsStringAsync();
the last line generates an error
Error while copying content to a stream.
----> System.ObjectDisposedException : Cannot access a closed Stream.
So there's a couple things here
Why is this error happening and how can I prevent it properly in the test?
The use of MemoryStream doesn't sit right with me, shouldn't I be able to write directly to the content's stream? Isn't MemoryStream potentially hugely increasing my memory usage?
Just put this out there, though it's not perfect... Using PushStreamContent does a lot of the job but it comes with its own headaches - namely that any exceptions that your anonymous method may produce will get swallowed and be difficult to track down without a full repro of the problem. When the bomb goes off is well passed the point of the pipeline where web api unhandled exception handlers would come into effect, and the xmlhttprequest doesn't seem to recognize the closure.
E.g. something like
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new PushStreamContent((stream, content, context) =>
{
// write your output here
});
return response;
will get you what you want, provided that internal method never slips up or goes wrong in any way.
PushStreamContent flushes your http headers immediately before the anonymous method is called, so you're chunked and no way to reel it back in later.
You can add a try/catch in your anonymous method to leave yourself a note if something goes wrong, but in my experience XmlHttpRequest doesn't recognize when the remote server forcibly closes the request so it keeps on waiting. Only started to figure out what was going on when I put Fiddler in there, and Fiddler squawked.
I'm working on an MVC webapplication that streams data from many resources.
My problem is when want to get data (music file) from a stream resource and then stream it to my web page, I don't know how not to download completely and then stream it to my web page.
Here is my webapi code:
[HttpGet]
public HttpResponseMessage Downlaod(int Data)
{
WebClient myWebClient = new WebClient();
Uri u =new Uri("https://api.soundcloud.com/tracks/" + Data + "/stream?client_id=*******************");
byte[] myDataBuffer = myWebClient.DownloadData(u);
MemoryStream st = new MemoryStream(myDataBuffer);
/*heres when i download data and convert it to memory stream*/
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Headers.AcceptRanges.Add("bytes");
result.StatusCode = HttpStatusCode.OK;
result.Content = new StreamContent(st);
result.Content.Headers.ContentLength = st.Length;
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");
return result;
}
I want to stream immediately when I receive bytes from my resource.
note: I'm not asking about how to stream data to the client it's about streaming from server to server.
I want to get file from another server and stream it to my clients without downloading the full content before start streaming.
note2: I also don't want to download the full content in once because the full content is very big, I want to get a byte from my content and then send that byte to the client not downloading the full content.
I think I'm doing it in wrong way and it is not possible with an MVC application if anyone can introduce an application that can proxy bytes from destination to client it would be the answer. the main reason that I want this,is to proxy a music file from my content server to a javascript music player and not to expose the main file.
Actually I'm trying to return multiple files(within same response as result to jQuery mobile client request) which include(html, .js, .css, .png images files) altogether at one time so that mobile client can download them as new updates. And I have been asked to make it a POST request rather than a GET request anyway.
Here is my Web API code
[Route("availableupdates")]
[HttpPost]
public HttpResponseMessage FilePackage()
{
HttpResponseMessage response = null;
var localFilePath = Directory.GetFiles(#"C:\Temp\Flower_Shop\Flower_Shop\images\i.png");
if (!File.Exists(localFilePath[0]))
{
response = Request.CreateResponse(HttpStatusCode.Gone);
}
else
{
response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StreamContent(new FileStream(localFilePath[0], FileMode.Open, FileAccess.Read));
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = "i";
}
return response;
}
Now the thing is that all those (html, .js, .css, .png) files are part of another little mobile app (and apparently lying in different folders within same application) which is sitting on the same server as the Web API RESTFUL service itself and in the code above I just hardcoded physical path of one of the image files. I don't know how to get all the files from different folders and sub-folders and compress them together and send them to the client.
Another big thing that I will really appreciate if someone explains me what MIMETYPE will work for the compressed files and do I need to send the compressed files as JSON response as I don't really know that It would be a good idea to send back a JSON response. What will happen to the contents of the files? Will the contents be converted into JSON as well?