I have a .Net Core 2.0 application that is sending files to a Web API endpoint, using multipart content. Everywhere I've looked, such as C# HttpClient 4.5 multipart/form-data upload, makes it seem that it should be as easy as passing a FileStream to a StreamContent. However, when I make the post, it looks like the file is attaching as text, not bits.
Actual code:
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri( "http://localhost:10442/filetest" )
};
var multiContent = new MultipartFormDataContent();
var filestream = File.OpenRead( path );
var filename = Path.GetFileName( path );
var streamContent = new StreamContent( filestream );
streamContent.Headers.Add( "Content-Type", "application/octet-stream" );
streamContent.Headers.Add( "Content-Disposition", $"form-data; name=\"file1\"; filename=\"{filename}\"" );
multiContent.Add( streamContent, "file", filename );
request.Content = multiContent;
var response = await new HttpClient().SendAsync( request );
The request looks like this which, as you may notice, is not all on one line (which I think is a/THE problem):
POST http://localhost:10442/filetest HTTP/1.1
Content-Type: multipart/form-data; boundary="c5295887-425d-4ec7-8638-20c6254f9e4b"
Content-Length: 88699
Host: localhost:10442
--c5295887-425d-4ec7-8638-20c6254f9e4b
Content-Type: application/octet-stream
Content-Disposition: form-data; name="file1"; filename="somepdf.pdf"
%PDF-1.7
%
1 0 obj
<</Type/Catalog/Version/1.7/Pages 3 0 R/Outlines 2 0 R/Names 8 0 R/Metadata 31 0 R>>
endobj
Fiddler shows the entire post all the way down to the end boundary, but await Request.Content.ReadAsStringAsync() in the endpoint only shows the first couple dozen bytes (it looks as if the stream wasn't finished, but if Fiddler got it all, shouldn't my endpoint have too?).
I was having similar trouble trying to hit a remote endpoint; I built this endpoint to test locally.
The exception I'm getting is:"Unexpected end of MIME multipart stream. MIME multipart message is not complete." To me, this makes sense both if I'm really only getting part of my stream, or if the line breaks are throwing something off.
I have also tried throwing some of the Idisposables into Usings but, as expected, that closes the streams and I get exceptions that way.
And for completeness's sake, here's the endpoint I'm calling:
public async void ReceiveFiles()
{
// exception happens here:
var mpData = await Request.Content.ReadAsMultipartAsync();
await Task.FromResult( 0 );
}
Try something like this:
static int Main(string[] args)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("http://localhost:10442/filetest")
};
var path = "c:\\temp\\foo.bak";
using (var filestream = File.OpenRead(path))
{
var length = filestream.Length.ToString();
var streamContent = new StreamContent(filestream);
streamContent.Headers.Add("Content-Type", "application/octet-stream");
streamContent.Headers.Add("Content-Length", length);
request.Content = streamContent;
Console.WriteLine($"Sending {length} bytes");
var response = new HttpClient().SendAsync(request).Result;
Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}
Console.WriteLine("Hit any key to exit");
Console.ReadKey();
return 0;
}
and
[HttpPost]
public async Task<IActionResult> Upload()
{
var buf = new byte[1024 * 64];
long totalBytes = 0;
using (var rs = Request.Body)
{
while (1 == 1)
{
int bytesRead = await rs.ReadAsync(buf, 0, buf.Length);
if (bytesRead == 0) break;
totalBytes += bytesRead;
}
}
var uploadedData = new
{
BytesRead = totalBytes
};
return new JsonResult(uploadedData) ;
}
I'm trying to solve a similar issue, and I'm not 100% to a solution yet, but maybe some of my research can help you.
It was helpful to me to read through the microsoft docs for .NET core file uploads, specifically for large files that use streams and multipart form data:
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.1#uploading-large-files-with-streaming
You already referenced it, but there's some relevant useful information in this answer:
C# HttpClient 4.5 multipart/form-data upload
This explains the details of the content-disposition header and how it is used with multipart form data requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition#As_a_header_for_a_multipart_body
As to your specific problem of the file being sent as text instead of bits, since http is text-based, it can only be sent as text, but that text can be encoded as you see fit. Perhaps your StreamContent needs a specific encoding to be used, like base64 encoding or similar? I do believe the newlines are significant in the multipart request, so hopefully setting the encoding for the file content as needed would be enough.
Another possibility: could it be that you need to set additional information on the file section's headers or in the definition of the StreamContent to indicate that it should expect to continue, or that the boundary information isn't put in correctly? See Multipart forms from C# client
I use this lib : https://github.com/jgiacomini/Tiny.RestClient
It's make easier to send multiplart file to send multipart files.
Here a sample :
await client.PostRequest("MultiPart/Test").
AsMultiPartFromDataRequest().
AddStream(stream1, "request", "request2.bin").
AddStream(stream2, "request", "request2.bin")
ExecuteAsync();
Related
When calling a specific endpoint in C# which works without issues in Postman (or via Firefox), I'm getting an empty response.
The url I'm calling is returning a collection of data. In the url parameters I can specify how much of said data I want.
I've inspected the response size in Postman, and when I limit the amount of data requested in my C# call such that the response is around 700kb, then I get a JSON response back.
However, if I exceed this size in the C# call, then the response is empty '{ }' and the ContentLength returned = -1 (the statusCode returned is 200, so this seems fine at least). This same request which fails in C# works fine within Postman and Firefox however...
I somehow suspect this occurs because either the deserializer's buffer is not big enough OR because the response is still in transit and the code somehow continues executing before it has read the whole response body...
See below for the 3 implementations which I've tested:
1:
var httpClient = new HttpClient();
var responseMessage = await httpClient.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead);
if (responseMessage.StatusCode == System.Net.HttpStatusCode.OK)
{
using (var httpStream = await responseMessage.Content.ReadAsStreamAsync())
{
using (var sr = new StreamReader(httpStream))
{
Info(await sr.ReadToEndAsync()); //Info logs the string to a file
}
}
}
2 (RESTSharp):
var client = new RestClient();
var request = new RestRequest(requestUrl, Method.GET, DataFormat.Json);
Info(request.Content); //Info logs the string to a file
3:
var httpWebRequest = (HttpWebRequest)WebRequest.Create(requestUrl);
httpWebRequest.ContentType = "application/json; charset-utf8";
var httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
var binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
{
ms.Write(buffer, 0, count);
}
responseBytes = ms.ToArray();
}
Info(Encoding.UTF8.GetString(responseBytes, 0, responseBytes.Length)); //Info logs the string to a file
I'm not modifying the HttpClient.MaxResponseContentBufferSize property, but for good measure I've also tried changing this value, to no avail.
How can I resolve this?
I've found the issue, it was being caused by the backend service to which I was connecting. Thank you again #Panagiotis Kanavos
I'm trying my hand at .NET Core but I'm stuck trying to convert multipart/form-data to an application/octet-stream to send via a PUT request. Anybody have any expertise I could borrow?
[HttpPost("fooBar"), ActionName("FooBar")]
public async Task<IActionResult> PostFooBar() {
HttpResponseMessage putResponse = await _httpClient.PutAsync(url, HttpContext.Request.Body);
}
Update: I think I might have two issues here:
My input format is multipart/form-data so I need to split out the file from the form data.
My output format must be application-octet stream but PutAsync expects HttpContent.
I had been trying to do something similar and having issues. I needed to PUT large files (>1.5GB) to a bucket on Amazon S3 using a pre-signed URL. The implementation on Amazon for .NET would fail for large files.
Here was my solution:
static HttpClient client = new HttpClient();
client.Timeout = TimeSpan.FromMinutes(60);
static async Task<bool> UploadLargeObjectAsync(string presignedUrl, string file)
{
Console.WriteLine("Uploading " + file + " to bucket...");
try
{
StreamContent strm = new StreamContent(new FileStream(file, FileMode.Open, FileAccess.Read));
strm.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
HttpResponseMessage putRespMsg = await client.PutAsync(presignedUrl, strm);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
return false;
}
return true;
}
Turns out Request has a Form property that contains a Files property that has an OpenReadStream() function on it to convert it into a stream. How exactly I was supposed to know that, I'm not sure.
Either way, here's the solution:
StreamContent stream = new StreamContent(HttpContext.Request.Form.Files[0].OpenReadStream());
HttpResponseMessage putResponse = await _httpClient.PutAsync(url, stream);
I'm trying to implement HMAC security for an API. Everything works fine until I try to post a file.
The HMAC solution can be found here - https://github.com/gavinharriss/WebAPI.HMAC - it's a fork from the original to allow GET requests as well as POST requests.
The code to attach a file:
var requestContent = new MultipartFormDataContent();
var fileContent = new ByteArrayContent(file);
requestContent.Add(fileContent, "file", filename);
if I immediately call HttpContent.ReadAsByteArrayAsync() there is no issue, the byte array is available.
However, the HMAC HttpClient (HMACHttpClient) implements a DelegatingHandler (HMACDelegatingHandler) in order to attach the HMAC header to requests.
In the HMACDelegatingHandler the request is passed along as a HttpRequestMessage from which the HttpRequestMessage.Content property is used in a helper to build the HMAC signature.
When building the signature, the following code is called from a helper class:
private static async Task<byte[]> ComputeHash(HttpContent httpContent)
{
using (var md5 = MD5.Create())
{
byte[] hash = null;
if (httpContent != null)
{
var content = await httpContent.ReadAsByteArrayAsync(); // <-- Fails here
if (content.Length != 0)
{
hash = md5.ComputeHash(content);
}
}
return hash;
}
}
When stepping through the code the var content = await httpContent.ReadAsByteArrayAsync() line is hit, then nothing, no error. The requests just seems to go poof but everything is still running and the HttpClient request never gets sent.
Any ideas what's going on?
Having tested this with various sizes of file, I found the issue arose when files got around the 50,000 byte mark.
This post provided a solution: HttpContent.ReadAsStringAsync causes request to hang (or other strange behaviours).
If you replace erroring line in HMACHelper (line 66):
var content = await httpContent.ReadAsByteArrayAsync();
with this:
var ms = new MemoryStream();
await httpContent.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
var content = ms.ToArray();
It should stop hanging.
I am using RestSharp (version 105.2.3.0 in Visual Studio 2013, .net 4.5) to call a NodeJS hosted webservice. One of the calls I need to make is to upload a file. Using a RESTSharp request, if I retrieve the stream from my end into a byte array and pass that to AddFile, it works fine. However, I would much rather stream the contents and not load up entire files in server memory (the files can be 100's of MB).
If I set up an Action to copy my stream (see below), I get an exception at the "MyStream.CopyTo" line of System.Net.ProtocolViolationException (Bytes to be written to the stream exceed the Content-Length bytes size specified). This exception is thrown within the Action block after client.Execute is called.
From what I read, I should not be manually adding a Content-Length header, and it doesn't help if I do. I have tried setting CopyTo buffer too small and large values, as well as omitting it entirely, to no avail. Can somebody give me a hint on what I've missed?
// Snippet...
protected T PostFile<T>(string Resource, string FieldName, string FileName,
string ContentType, Stream MyStream,
IEnumerable<Parameter> Parameters = null) where T : new()
{
RestRequest request = new RestRequest(Resource);
request.Method = Method.POST;
if (Parameters != null)
{
// Note: parameters are all UrlSegment values
request.Parameters.AddRange(Parameters);
}
// _url, _username and _password are defined configuration variables
RestClient client = new RestClient(_url);
if (!string.IsNullOrEmpty(_username))
{
client.Authenticator = new HttpBasicAuthenticator(_username, _password);
}
/*
// Does not work, throws System.Net.ProtocolViolationException,
// Bytes to be written to the stream exceed the
// Content-Length bytes size specified.
request.AddFile(FieldName, (s) =>
{
MyStream.CopyTo(s);
MyStream.Flush();
}, FileName, ContentType);
*/
// This works, but has to load the whole file in memory
byte[] data = new byte[MyStream.Length];
MyStream.Read(data, 0, (int) MyStream.Length);
request.AddFile(FieldName, data, FileName, ContentType);
var response = client.Execute<T>(request);
// check response and continue...
}
I had the same issue. I ended up using the .Add() on the Files collection. It has a FileParameter param which has the same params as AddFile(), you just have to add the ContentLength:
var req = GetRestRequest("Upload", Method.POST, null);
//req.AddFile("file",
// (s) => {
// var stream = input(imageObject);
// stream.CopyTo(s);
// stream.Dispose();
// },
// fileName, contentType);
req.Files.Add(new FileParameter {
Name = "file",
Writer = (s) => {
var stream = input(imageObject);
stream.CopyTo(s);
stream.Dispose();
},
FileName = fileName,
ContentType = contentType,
ContentLength = contentLength
});
The following code works for me for uploading a csv file using rest sharp. Web services API has been called.
var client = new RestClient(<YOUR API END URL >);
var request = new RestRequest(Method.POST) ;
request.AlwaysMultipartFormData = true;
request. AddHeader("Content-Type", "multipart/form-data");
request.AddHeader("X-API-TOKEN", <Your Unique Token - again not needed for certain calls>);
request.AddParameter(<Your parameters.....>);
request.AddFile("file", currentFileLocation, contentType);
request.AddParameter("multipart/form-data", fileName, ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
var response = client.Execute(request);
I am adding a file (bytearray) to post data via RestSharp and am sending it to a node.js express server. I want to write the file with the filename from content-disposition.
My first problem is that when writing the data to the file using fs, it also writes a wrapper with some of the header info:
-------------------------------28947758029299
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: application/octet-stream
Hello from c#
-------------------------------28947758029299--
The other problem is, that although it writes it to the file, Content-Disposition does not seem to be part of the headers object:
[ 'content-type',
'accept',
'x-forwarded-port',
'user-agent',
'accept-encoding',
'content-length',
'host',
'x-forwarded-for' ]
The only solution I can think of is, to temporarily write the file and extract what I need with regex, but I believe that would corrupt image files besides being rather a result of me not properly understanding http requests than a legit solution. I am very new to both C# and node, so it is very patched together from examples I found online:
This is my C# code:
public ActionResult Documents(HttpPostedFileBase file)
{
if (ModelState.IsValid)
{
if (file == null)
{
ModelState.AddModelError("File", "Please Upload Your file");
}
else if (file.ContentLength > 0)
{
var fileName = Path.GetFileName(file.FileName);
var inputStream = file.InputStream;
System.IO.Stream MyStream;
int FileLen = file.ContentLength;
byte[] input = new byte[FileLen];
// Initialize the stream.
MyStream = file.InputStream;
// Read the file into the byte array.
MyStream.Read(input, 0, FileLen);
var client = new RestClient("http://128.199.53.59");
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/octet-stream");
request.AddFile("file", input, fileName, "application/octet-stream");
RestResponse response = (RestResponse)client.Execute(request);
ModelState.Clear();
ViewBag.Message = "File uploaded successfully";
}
}
}
And this my relevant node.js:
app.post('/', function(request, response){
var fileData = [];
var size = 0;
request.on('data', function (chunk) {
size += chunk.length;
fileData.push(chunk);
})
request.on('end', function(){
var buffer = Buffer.concat(fileData);
fs.writeFile('logo.txt', buffer, 'binary', function(err){
if (err) throw err
})
})
});
You need to use a multipart/form-data parser. For Express, there is multer, multiparty, and formidable.
If you want to work with the incoming files as streams instead of always saving them to disk, you could use busboy/connect-busboy (busboy is what powers multer) instead.