Develop Reference

Read big files in chunks(C#) [duplicate]

I need to upload large files (~200MB) over HTTP protocol. I want to avoid loading the files to memory and want to send them directly.
Thanks to this article I was able to make it with HttpWebRequest.
HttpWebRequest requestToServer = (HttpWebRequest)WebRequest.Create("....");
requestToServer.AllowWriteStreamBuffering = false;
requestToServer.Method = WebRequestMethods.Http.Post;
requestToServer.ContentType = "multipart/form-data; boundary=" + boundaryString;
requestToServer.KeepAlive = false;
requestToServer.ContentLength = ......;
using (Stream stream = requestToServer.GetRequestStream())
{
// write boundary string, Content-Disposition etc.
// copy file to stream
using (var fileStream = new FileStream("...", FileMode.Open, FileAccess.Read))
{
fileStream.CopyTo(stream);
}
// add some other file(s)
}
However, I would like to do it via HttpClient. I found article which describes using of HttpCompletionOption.ResponseHeadersRead and I tried something like this but it does not work unfortunately.
WebRequestHandler handler = new WebRequestHandler();
using (var httpClient = new HttpClient(handler))
{
httpClient.DefaultRequestHeaders.Add("ContentType", "multipart/form-data; boundary=" + boundaryString);
httpClient.DefaultRequestHeaders.Add("Connection", "close");
var httpRequest = new HttpRequestMessage(HttpMethod.Post, "....");
using (HttpResponseMessage responseMessage = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead))
{
using (Stream stream = await responseMessage.Content.ReadAsStreamAsync())
{
// here I wanted to write content to the stream, but the stream is available only for reading
}
}
}
Maybe I overlooked or missed something...
UPDATE
On top of it, it is important to use StreamContent with proper headers:
Content-Disposition
Content-Type

See the StreamContent class:
HttpResponseMessage response =
await httpClient.PostAsync("http://....", new StreamContent(streamToSend));
In your example, you are getting a response stream and trying to write to it. Instead, you must pass in your content for the request, as above.
The HttpCompletionOption.ResponseHeadersRead is to disable buffering of the response stream, but does not affect the request. You would typically use it if your response is large.
For posting multiple files of form data, use the MultipartFormDataContent:
var content = new MultipartFormDataContent();
content.Add(new StreamContent(stream1), "file1.jpg");
content.Add(new StreamContent(stream2), "file2.jpg");
HttpResponseMessage response =
await httpClient.PostAsync("http://...", content);

Use C# to Save an Excel File Contained in a Tableau API Response

I'm trying to download an Excel file from a Tableau view, using a snippet based on this article: Downloading Files with the WebRequest and WebResponse Classes.
WebResponse response = null;
Stream webStream = null;
Stream localStream = null;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.Headers.Add("X-Tableau-Auth", token);
response = request.GetResponse();
webStream = response.GetResponseStream();
localStream = File.Create("testing.xlsx");
byte[] buffer = new Byte[1024];
int bytesRead;
int bytesProcessed = 0;
do
{
bytesRead = webStream.Read(buffer, 0, buffer.Length);
localStream.Write(buffer, 0, bytesRead);
bytesProcessed += bytesRead;
} while (bytesRead > 0);
response.Close();
webStream.Close();
localStream.Close();
But when I try to open the Excel file it says "Nope, maybe it is corrupt?". The response is a complete Excel file "ready to be saved", which means that it's encoded using UTF-8. Indeed, if I use Postman to do the call, and then save the response, it's saved and opens without any problem.
Before finding the page I said above, I thought the problem was because the response is UTF-8 and the class String is UTF-16. So, I made some testing getting the data from the file generated by Postman and then writing to a new file. The result of the testings: indeed, if the data passes through a String, it's not well saved. Then I tried with that code, and got the same result: fail. I'm pretty sure this code is not using any UTF-16 encoding variable, but maybe I'm wrong.
Anyway, do anyone knows what is my problem with that code, or point me to the right way to accomplish my task? That is, to download a Tableau report to Excel, using the Tableau API.
Thanks in advance.

Unless you're stuck on a particularly old version of .NET, that referenced link is from 2004. The following code will work in .NET 5 / C# 9, and in earlier versions with just some minor tweaking of the using statements. It's showing for Tableau Online, but should work fine for recent versions of Server, if I had to guess. This is sample-grade code, so I would recommend following best practices for the HttpClient if you intend to make this call frequently.
//using System.IO;
//using System.Net.Http;
var token = "your-session-token";
var uri = "https://your-pod.online.tableau.com/api/.../sites/.../views/.../crosstab/excel";
var yourFile = "D:/file/test.xlsx";
using var client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, uri);
request.Headers.Add("X-Tableau-Auth", token);
using var response = await client.SendAsync(request);
using FileStream outputFileStream = new FileStream(yourFile, FileMode.Create);
await response.Content.CopyToAsync(outputFileStream);
UPDATE: If you're constrained to WebRequest and non-async methods, you can try the following:
var token = "your-session-token";
var uri = "https://your-pod.online.tableau.com/api/.../sites/.../views/.../crosstab/excel";
var yourFile = "D:/file/test.xlsx";
WebRequest request = WebRequest.Create(uri);
request.Method = "GET";
request.Headers.Add("X-Tableau-Auth", token);
var response = request.GetResponse();
if(((HttpWebResponse)response).StatusCode == HttpStatusCode.OK)
{
using (Stream dataStream = response.GetResponseStream())
using (FileStream fileStream = new FileStream(yourFile, FileMode.CreateNew))
{
dataStream.CopyTo(fileStream);
}
}
response.Close();

Trouble making POST request from c# [.NET]

We have a created an API for the application which takes the image via POST request process it and sends the result in JSON format.
We tried calling API from different sources like python, postman app, c#. We can successfully call end point using python and postman app but with c# getting error
c# code [Not working]
byte[] img_data = System.IO.File.ReadAllBytes(#"file_path");
string url_ep = "http://ip:port/get";
Dictionary<string, byte[]> fl_image = new Dictionary<string, byte[]>();
fl_image.Add("image", img_data);
string data = JsonConvert.SerializeObject(fl_image);
var dataToSend = Encoding.UTF8.GetBytes(data);
var request = HttpWebRequest.Create(url_ep);
request.ContentType = "application/json";
request.ContentLength = dataToSend.Length;
request.Method = "POST";
request.GetRequestStream().Write(dataToSend, 0, dataToSend.Length);
var response = request.GetResponse();
System.IO.Stream dataStream = response.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
Console.WriteLine(responseFromServer);
python code [working]
import requests
url = 'http://ip:port/get'
fl_image = {'image': open('file_path', 'rb')}
res = requests.post(url, files=fl_image)
print(res.json())
API Endpoint
from flask import Flask, request
import numpy as np
import cv2 as cv
#app.route('/get', methods = ['POST'])
def get_image():
if request.method == 'POST':
file = request.files['image']
# Read file
f = file.read()
# convert string of image data to uint8
f1 = np.fromstring(f, np.uint8)
# decode image
f2 = cv.imdecode(f1,cv.IMREAD_COLOR)

There are several issues with the way you are posting data from C#. The most relevant one is that you are trying to post a file as a JSON object, with file contents as string.
This cannot work: your python server is clearly expecting multipart/form-data as content-type.
I also strongly recommend you to use HttpClient and not the old HttpWebRequest class to send HTTP Requests.
var filePath = #"file_path";
var url = "http://ip:port/get";
using (var client = new HttpClient())
using (var content = new MultipartFormDataContent())
using (var fileStream = File.OpenRead(filePath))
{
var imageContent = new StreamContent(fileStream);
// NOTE: the line below is not required, but useful when you know the media type
imageContent.Headers.ContentType = MediaTypeHeaderValue.Parse("image/jpeg");
content.Add(imageContent, "image", Path.GetFileName(filePath));
var response = await client.PostAsync(url, content);
var stringResponse = await response.Content.ReadAsStringAsync();
// do what you need with the response
}
Other minor issues:
Do not read the entire file in memory (using File.ReadAllBytes), but open a stream for reading instead.
Use async/await when possible, do not block on async code (do not use .Result, .Wait() or .GetAwaiter().GetResult() on Task or Task<T>)
Always call Dispose() on IDisposable objects when you have finished using them (wrapping them inside a using block)

You need to dispose the connections
reader.Close();
dataStream.Close();
response.Close();
Hope this helps
Or try using HttpClient for .net within the using block

C# StreamContent and File.OpenRead() Not Producing HTTP'able multipart content

I have a .Net Core 2.0 application that is sending files to a Web API endpoint, using multipart content. Everywhere I've looked, such as C# HttpClient 4.5 multipart/form-data upload, makes it seem that it should be as easy as passing a FileStream to a StreamContent. However, when I make the post, it looks like the file is attaching as text, not bits.
Actual code:
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri( "http://localhost:10442/filetest" )
};
var multiContent = new MultipartFormDataContent();
var filestream = File.OpenRead( path );
var filename = Path.GetFileName( path );
var streamContent = new StreamContent( filestream );
streamContent.Headers.Add( "Content-Type", "application/octet-stream" );
streamContent.Headers.Add( "Content-Disposition", $"form-data; name=\"file1\"; filename=\"{filename}\"" );
multiContent.Add( streamContent, "file", filename );
request.Content = multiContent;
var response = await new HttpClient().SendAsync( request );
The request looks like this which, as you may notice, is not all on one line (which I think is a/THE problem):
POST http://localhost:10442/filetest HTTP/1.1
Content-Type: multipart/form-data; boundary="c5295887-425d-4ec7-8638-20c6254f9e4b"
Content-Length: 88699
Host: localhost:10442
--c5295887-425d-4ec7-8638-20c6254f9e4b
Content-Type: application/octet-stream
Content-Disposition: form-data; name="file1"; filename="somepdf.pdf"
%PDF-1.7
%
1 0 obj
<</Type/Catalog/Version/1.7/Pages 3 0 R/Outlines 2 0 R/Names 8 0 R/Metadata 31 0 R>>
endobj
Fiddler shows the entire post all the way down to the end boundary, but await Request.Content.ReadAsStringAsync() in the endpoint only shows the first couple dozen bytes (it looks as if the stream wasn't finished, but if Fiddler got it all, shouldn't my endpoint have too?).
I was having similar trouble trying to hit a remote endpoint; I built this endpoint to test locally.
The exception I'm getting is:"Unexpected end of MIME multipart stream. MIME multipart message is not complete." To me, this makes sense both if I'm really only getting part of my stream, or if the line breaks are throwing something off.
I have also tried throwing some of the Idisposables into Usings but, as expected, that closes the streams and I get exceptions that way.
And for completeness's sake, here's the endpoint I'm calling:
public async void ReceiveFiles()
{
// exception happens here:
var mpData = await Request.Content.ReadAsMultipartAsync();
await Task.FromResult( 0 );
}

Try something like this:
static int Main(string[] args)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("http://localhost:10442/filetest")
};
var path = "c:\\temp\\foo.bak";
using (var filestream = File.OpenRead(path))
{
var length = filestream.Length.ToString();
var streamContent = new StreamContent(filestream);
streamContent.Headers.Add("Content-Type", "application/octet-stream");
streamContent.Headers.Add("Content-Length", length);
request.Content = streamContent;
Console.WriteLine($"Sending {length} bytes");
var response = new HttpClient().SendAsync(request).Result;
Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}
Console.WriteLine("Hit any key to exit");
Console.ReadKey();
return 0;
}
and
[HttpPost]
public async Task<IActionResult> Upload()
{
var buf = new byte[1024 * 64];
long totalBytes = 0;
using (var rs = Request.Body)
{
while (1 == 1)
{
int bytesRead = await rs.ReadAsync(buf, 0, buf.Length);
if (bytesRead == 0) break;
totalBytes += bytesRead;
}
}
var uploadedData = new
{
BytesRead = totalBytes
};
return new JsonResult(uploadedData) ;
}

I'm trying to solve a similar issue, and I'm not 100% to a solution yet, but maybe some of my research can help you.
It was helpful to me to read through the microsoft docs for .NET core file uploads, specifically for large files that use streams and multipart form data:
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.1#uploading-large-files-with-streaming
You already referenced it, but there's some relevant useful information in this answer:
C# HttpClient 4.5 multipart/form-data upload
This explains the details of the content-disposition header and how it is used with multipart form data requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition#As_a_header_for_a_multipart_body
As to your specific problem of the file being sent as text instead of bits, since http is text-based, it can only be sent as text, but that text can be encoded as you see fit. Perhaps your StreamContent needs a specific encoding to be used, like base64 encoding or similar? I do believe the newlines are significant in the multipart request, so hopefully setting the encoding for the file content as needed would be enough.
Another possibility: could it be that you need to set additional information on the file section's headers or in the definition of the StreamContent to indicate that it should expect to continue, or that the boundary information isn't put in correctly? See Multipart forms from C# client

I use this lib : https://github.com/jgiacomini/Tiny.RestClient
It's make easier to send multiplart file to send multipart files.
Here a sample :
await client.PostRequest("MultiPart/Test").
AsMultiPartFromDataRequest().
AddStream(stream1, "request", "request2.bin").
AddStream(stream2, "request", "request2.bin")
ExecuteAsync();

Send a 'Stream' over a PutAsync request

I'm trying my hand at .NET Core but I'm stuck trying to convert multipart/form-data to an application/octet-stream to send via a PUT request. Anybody have any expertise I could borrow?
[HttpPost("fooBar"), ActionName("FooBar")]
public async Task<IActionResult> PostFooBar() {
HttpResponseMessage putResponse = await _httpClient.PutAsync(url, HttpContext.Request.Body);
}
Update: I think I might have two issues here:
My input format is multipart/form-data so I need to split out the file from the form data.
My output format must be application-octet stream but PutAsync expects HttpContent.

I had been trying to do something similar and having issues. I needed to PUT large files (>1.5GB) to a bucket on Amazon S3 using a pre-signed URL. The implementation on Amazon for .NET would fail for large files.
Here was my solution:
static HttpClient client = new HttpClient();
client.Timeout = TimeSpan.FromMinutes(60);
static async Task<bool> UploadLargeObjectAsync(string presignedUrl, string file)
{
Console.WriteLine("Uploading " + file + " to bucket...");
try
{
StreamContent strm = new StreamContent(new FileStream(file, FileMode.Open, FileAccess.Read));
strm.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
HttpResponseMessage putRespMsg = await client.PutAsync(presignedUrl, strm);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
return false;
}
return true;
}

Turns out Request has a Form property that contains a Files property that has an OpenReadStream() function on it to convert it into a stream. How exactly I was supposed to know that, I'm not sure.
Either way, here's the solution:
StreamContent stream = new StreamContent(HttpContext.Request.Form.Files[0].OpenReadStream());
HttpResponseMessage putResponse = await _httpClient.PutAsync(url, stream);