Develop Reference

c# JSON REST response via 3 different approaches (WebRequest, RESTSharp, HttpClient) is empty, but Postman and browser works

When calling a specific endpoint in C# which works without issues in Postman (or via Firefox), I'm getting an empty response.
The url I'm calling is returning a collection of data. In the url parameters I can specify how much of said data I want.
I've inspected the response size in Postman, and when I limit the amount of data requested in my C# call such that the response is around 700kb, then I get a JSON response back.
However, if I exceed this size in the C# call, then the response is empty '{ }' and the ContentLength returned = -1 (the statusCode returned is 200, so this seems fine at least). This same request which fails in C# works fine within Postman and Firefox however...
I somehow suspect this occurs because either the deserializer's buffer is not big enough OR because the response is still in transit and the code somehow continues executing before it has read the whole response body...
See below for the 3 implementations which I've tested:
1:
var httpClient = new HttpClient();
var responseMessage = await httpClient.GetAsync(requestUrl, HttpCompletionOption.ResponseHeadersRead);
if (responseMessage.StatusCode == System.Net.HttpStatusCode.OK)
{
using (var httpStream = await responseMessage.Content.ReadAsStreamAsync())
{
using (var sr = new StreamReader(httpStream))
{
Info(await sr.ReadToEndAsync()); //Info logs the string to a file
}
}
}
2 (RESTSharp):
var client = new RestClient();
var request = new RestRequest(requestUrl, Method.GET, DataFormat.Json);
Info(request.Content); //Info logs the string to a file
3:
var httpWebRequest = (HttpWebRequest)WebRequest.Create(requestUrl);
httpWebRequest.ContentType = "application/json; charset-utf8";
var httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
var binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
{
ms.Write(buffer, 0, count);
}
responseBytes = ms.ToArray();
}
Info(Encoding.UTF8.GetString(responseBytes, 0, responseBytes.Length)); //Info logs the string to a file
I'm not modifying the HttpClient.MaxResponseContentBufferSize property, but for good measure I've also tried changing this value, to no avail.
How can I resolve this?

I've found the issue, it was being caused by the backend service to which I was connecting. Thank you again #Panagiotis Kanavos

C# StreamContent and File.OpenRead() Not Producing HTTP'able multipart content

I have a .Net Core 2.0 application that is sending files to a Web API endpoint, using multipart content. Everywhere I've looked, such as C# HttpClient 4.5 multipart/form-data upload, makes it seem that it should be as easy as passing a FileStream to a StreamContent. However, when I make the post, it looks like the file is attaching as text, not bits.
Actual code:
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri( "http://localhost:10442/filetest" )
};
var multiContent = new MultipartFormDataContent();
var filestream = File.OpenRead( path );
var filename = Path.GetFileName( path );
var streamContent = new StreamContent( filestream );
streamContent.Headers.Add( "Content-Type", "application/octet-stream" );
streamContent.Headers.Add( "Content-Disposition", $"form-data; name=\"file1\"; filename=\"{filename}\"" );
multiContent.Add( streamContent, "file", filename );
request.Content = multiContent;
var response = await new HttpClient().SendAsync( request );
The request looks like this which, as you may notice, is not all on one line (which I think is a/THE problem):
POST http://localhost:10442/filetest HTTP/1.1
Content-Type: multipart/form-data; boundary="c5295887-425d-4ec7-8638-20c6254f9e4b"
Content-Length: 88699
Host: localhost:10442
--c5295887-425d-4ec7-8638-20c6254f9e4b
Content-Type: application/octet-stream
Content-Disposition: form-data; name="file1"; filename="somepdf.pdf"
%PDF-1.7
%
1 0 obj
<</Type/Catalog/Version/1.7/Pages 3 0 R/Outlines 2 0 R/Names 8 0 R/Metadata 31 0 R>>
endobj
Fiddler shows the entire post all the way down to the end boundary, but await Request.Content.ReadAsStringAsync() in the endpoint only shows the first couple dozen bytes (it looks as if the stream wasn't finished, but if Fiddler got it all, shouldn't my endpoint have too?).
I was having similar trouble trying to hit a remote endpoint; I built this endpoint to test locally.
The exception I'm getting is:"Unexpected end of MIME multipart stream. MIME multipart message is not complete." To me, this makes sense both if I'm really only getting part of my stream, or if the line breaks are throwing something off.
I have also tried throwing some of the Idisposables into Usings but, as expected, that closes the streams and I get exceptions that way.
And for completeness's sake, here's the endpoint I'm calling:
public async void ReceiveFiles()
{
// exception happens here:
var mpData = await Request.Content.ReadAsMultipartAsync();
await Task.FromResult( 0 );
}

Try something like this:
static int Main(string[] args)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("http://localhost:10442/filetest")
};
var path = "c:\\temp\\foo.bak";
using (var filestream = File.OpenRead(path))
{
var length = filestream.Length.ToString();
var streamContent = new StreamContent(filestream);
streamContent.Headers.Add("Content-Type", "application/octet-stream");
streamContent.Headers.Add("Content-Length", length);
request.Content = streamContent;
Console.WriteLine($"Sending {length} bytes");
var response = new HttpClient().SendAsync(request).Result;
Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}
Console.WriteLine("Hit any key to exit");
Console.ReadKey();
return 0;
}
and
[HttpPost]
public async Task<IActionResult> Upload()
{
var buf = new byte[1024 * 64];
long totalBytes = 0;
using (var rs = Request.Body)
{
while (1 == 1)
{
int bytesRead = await rs.ReadAsync(buf, 0, buf.Length);
if (bytesRead == 0) break;
totalBytes += bytesRead;
}
}
var uploadedData = new
{
BytesRead = totalBytes
};
return new JsonResult(uploadedData) ;
}

I'm trying to solve a similar issue, and I'm not 100% to a solution yet, but maybe some of my research can help you.
It was helpful to me to read through the microsoft docs for .NET core file uploads, specifically for large files that use streams and multipart form data:
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.1#uploading-large-files-with-streaming
You already referenced it, but there's some relevant useful information in this answer:
C# HttpClient 4.5 multipart/form-data upload
This explains the details of the content-disposition header and how it is used with multipart form data requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition#As_a_header_for_a_multipart_body
As to your specific problem of the file being sent as text instead of bits, since http is text-based, it can only be sent as text, but that text can be encoded as you see fit. Perhaps your StreamContent needs a specific encoding to be used, like base64 encoding or similar? I do believe the newlines are significant in the multipart request, so hopefully setting the encoding for the file content as needed would be enough.
Another possibility: could it be that you need to set additional information on the file section's headers or in the definition of the StreamContent to indicate that it should expect to continue, or that the boundary information isn't put in correctly? See Multipart forms from C# client

I use this lib : https://github.com/jgiacomini/Tiny.RestClient
It's make easier to send multiplart file to send multipart files.
Here a sample :
await client.PostRequest("MultiPart/Test").
AsMultiPartFromDataRequest().
AddStream(stream1, "request", "request2.bin").
AddStream(stream2, "request", "request2.bin")
ExecuteAsync();

How to Post a web request and receive a file from web API response?

I have an Infopath Form Template on Sharepoint, I want to add a button there so when the user clicks on it, it will POST an string to the following Web API. The following web API is tested and returns an excel file as shown:
I want to Post the FileName of the excel file using post request and it is important for me the request method to be POST type. and then the user will download a file with the specified 'FileName'.
Actually i want to use post method because at the next stage i will send the content of the excel file too.
Important Note: I only can use .Net FrameWork 3.5 because this is the only framework supported in InfoPath Form Templates.
[HttpPost]
public HttpResponseMessage Post([FromBody]string FileName)
{
string reqBook = "c:\somefile.xlsx";
//converting Excel(xlsx) file into bytes array
var dataBytes = File.ReadAllBytes(reqBook);
//adding bytes to memory stream
var dataStream = new MemoryStream(dataBytes);
HttpResponseMessage httpResponseMessage = Request.CreateResponse(HttpStatusCode.OK);
httpResponseMessage.Content = new StreamContent(dataStream);
httpResponseMessage.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
httpResponseMessage.Content.Headers.ContentDisposition.FileName = FileName;
httpResponseMessage.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
return httpResponseMessage;
}

When you perform the HttpPost on the client side, you will want to read the HttpResponseStream to get the byte data of the response stream.
Once you have the response stream data, you can then deserialize it to the type of object in C# you want, or you could alternatively just write it to the disk as
File.WriteAllBytes("someexcel.xlsx",data);
An easy way to do it would be with the HttpClient class.
HttpClient client = new HttpClient();
var response = client.PostAsync("", null).Result;
var content = response.Content.ReadAsByteArrayAsync().Result;
File.WriteAllBytes("excel.xlsx", content);
Just fill in the PostAsync bit with the Url and the content you wish to post.
I am using .Result to keep everything synchronous - but you can use 'await' if you prefer.
If you are working with HttpWebRequests - then the process becomes more complicated, as you need to manage the streams yourself.
The HttpClient will manage and handle it all for you - so I recommend it, unless there is something special it needs to do that it currently does not.
Due to your .Net 3.5 requirement:
private static HttpWebResponse MakeRequest(string url, string postArgument)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "multipart/form-data;";
Stream stream = request.GetRequestStream();
string result = string.Format("arg1={0}", postArgument);
byte[] value = Encoding.UTF8.GetBytes(result);
stream.Write(value, 0, value.Length);
stream.Close();
return (HttpWebResponse)request.GetResponse();
}
You can then do:
var response = MakeRequest("http://mywebsite.com/ProcessExcel", "accounts.xlsx");
And then do
Stream objStream = response .GetResponseStream();
BinaryReader breader = new BinaryReader(objStream);
byte[] data= breader.ReadBytes((int)webresponse.ContentLength);
File.WriteAllBytes("excel.xlsx",data);

Webrequest GET Body ViolationException

I'm using SolrExpress to search and index documents within c# (dotnet core). Inserting (indexing) documents works fine since this is a nice post request.
However when i'm trying to do a select query (to retrieve documents) i'm getting aggregation exceptions. By digging down the source in SolrExpress i came upon the following source:
private WebRequest Prepare(SecurityOptions options, string requestMethod, string handler, string data)
{
var baseUrl = $"{this.HostAddress}/{handler}";
var encoding = new UTF8Encoding();
var bytes = encoding.GetBytes(data);
var request = WebRequest.Create(baseUrl);
if (options.AuthenticationType == AuthenticationType.Basic)
{
var encoded = Convert.ToBase64String(Encoding.GetEncoding("ISO-8859-1").GetBytes(options.UserName + ":" + options.Password));
request.Headers[HttpRequestHeader.Authorization] = "Basic " + encoded;
}
request.Method = requestMethod;
request.ContentType = "application/json";
#if NET451
request.ContentLength = bytes.Length;
#endif
#if NETCORE
var taskStream = request.GetRequestStreamAsync();
taskStream.Wait();
var stream = taskStream.Result;
stream.Write(bytes, 0, bytes.Length);
#else
var stream = request.GetRequestStream();
stream.Write(bytes, 0, bytes.Length);
stream.Close();
#endif
return request;
}
The method calling this method is the following for GET:
public string Get(SecurityOptions options, string handler, string data)
{
var request = this.Prepare(options, "GET-X", handler, data);
#if NETCORE
var task = this.ExecuteAsync(request, data);
task.Wait();
return task.Result;
#else
return this.Execute(request, data);
#endif
}
This Get method caused an error by using a request method GET-X which Solr itself (6.4.1) did not understand. I've changed this to a normal request method: GET therefore solving the error on solr's side.
However currently i'm getting a System.Net.ProtocolViolationException with the message: Cannot send a content-body with this verb-type. This is happening when waiting for the taskStream to finish and write its result to the request-body.
My question:
How would one send a GET-request with a body (in json format (as string)) within dotnet core?
Since RFC2616 says it's not forbidden i'd like to use this 'feature' as answered in the following question
See RFC2616 - Hypertext Transfer Protocol -- HTTP/1.1, section 4.3 "Message Body":
A message-body MUST NOT be included in a request if the specification of the > > request method (section 5.1.1) does not allow sending an entity-body in requests.
In section 9.3 "GET" including an entity-body is not forbidden.
So, yes, you are allowed to send an entity-body with a HTTP GET request.

Http request: Writing binary text file (or image) without headers with filename from content-disposition

I am adding a file (bytearray) to post data via RestSharp and am sending it to a node.js express server. I want to write the file with the filename from content-disposition.
My first problem is that when writing the data to the file using fs, it also writes a wrapper with some of the header info:
-------------------------------28947758029299
Content-Disposition: form-data; name="file"; filename="test.txt"
Content-Type: application/octet-stream
Hello from c#
-------------------------------28947758029299--
The other problem is, that although it writes it to the file, Content-Disposition does not seem to be part of the headers object:
[ 'content-type',
'accept',
'x-forwarded-port',
'user-agent',
'accept-encoding',
'content-length',
'host',
'x-forwarded-for' ]
The only solution I can think of is, to temporarily write the file and extract what I need with regex, but I believe that would corrupt image files besides being rather a result of me not properly understanding http requests than a legit solution. I am very new to both C# and node, so it is very patched together from examples I found online:
This is my C# code:
public ActionResult Documents(HttpPostedFileBase file)
{
if (ModelState.IsValid)
{
if (file == null)
{
ModelState.AddModelError("File", "Please Upload Your file");
}
else if (file.ContentLength > 0)
{
var fileName = Path.GetFileName(file.FileName);
var inputStream = file.InputStream;
System.IO.Stream MyStream;
int FileLen = file.ContentLength;
byte[] input = new byte[FileLen];
// Initialize the stream.
MyStream = file.InputStream;
// Read the file into the byte array.
MyStream.Read(input, 0, FileLen);
var client = new RestClient("http://128.199.53.59");
var request = new RestRequest(Method.POST);
request.AddHeader("Content-Type", "application/octet-stream");
request.AddFile("file", input, fileName, "application/octet-stream");
RestResponse response = (RestResponse)client.Execute(request);
ModelState.Clear();
ViewBag.Message = "File uploaded successfully";
}
}
}
And this my relevant node.js:
app.post('/', function(request, response){
var fileData = [];
var size = 0;
request.on('data', function (chunk) {
size += chunk.length;
fileData.push(chunk);
})
request.on('end', function(){
var buffer = Buffer.concat(fileData);
fs.writeFile('logo.txt', buffer, 'binary', function(err){
if (err) throw err
})
})
});

You need to use a multipart/form-data parser. For Express, there is multer, multiparty, and formidable.
If you want to work with the incoming files as streams instead of always saving them to disk, you could use busboy/connect-busboy (busboy is what powers multer) instead.