I need to upload large files (~200MB) over HTTP protocol. I want to avoid loading the files to memory and want to send them directly.
Thanks to this article I was able to make it with HttpWebRequest.
HttpWebRequest requestToServer = (HttpWebRequest)WebRequest.Create("....");
requestToServer.AllowWriteStreamBuffering = false;
requestToServer.Method = WebRequestMethods.Http.Post;
requestToServer.ContentType = "multipart/form-data; boundary=" + boundaryString;
requestToServer.KeepAlive = false;
requestToServer.ContentLength = ......;
using (Stream stream = requestToServer.GetRequestStream())
{
// write boundary string, Content-Disposition etc.
// copy file to stream
using (var fileStream = new FileStream("...", FileMode.Open, FileAccess.Read))
{
fileStream.CopyTo(stream);
}
// add some other file(s)
}
However, I would like to do it via HttpClient. I found article which describes using of HttpCompletionOption.ResponseHeadersRead and I tried something like this but it does not work unfortunately.
WebRequestHandler handler = new WebRequestHandler();
using (var httpClient = new HttpClient(handler))
{
httpClient.DefaultRequestHeaders.Add("ContentType", "multipart/form-data; boundary=" + boundaryString);
httpClient.DefaultRequestHeaders.Add("Connection", "close");
var httpRequest = new HttpRequestMessage(HttpMethod.Post, "....");
using (HttpResponseMessage responseMessage = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead))
{
using (Stream stream = await responseMessage.Content.ReadAsStreamAsync())
{
// here I wanted to write content to the stream, but the stream is available only for reading
}
}
}
Maybe I overlooked or missed something...
UPDATE
On top of it, it is important to use StreamContent with proper headers:
Content-Disposition
Content-Type
See the StreamContent class:
HttpResponseMessage response =
await httpClient.PostAsync("http://....", new StreamContent(streamToSend));
In your example, you are getting a response stream and trying to write to it. Instead, you must pass in your content for the request, as above.
The HttpCompletionOption.ResponseHeadersRead is to disable buffering of the response stream, but does not affect the request. You would typically use it if your response is large.
For posting multiple files of form data, use the MultipartFormDataContent:
var content = new MultipartFormDataContent();
content.Add(new StreamContent(stream1), "file1.jpg");
content.Add(new StreamContent(stream2), "file2.jpg");
HttpResponseMessage response =
await httpClient.PostAsync("http://...", content);
Related
I was told to migrate a post request written in HttpWebRequest to HttpRequestMessage.
I've provided the old code and the new code that I've developed below.
In the old code they are sending the byte stream over to the
endpoint. I'm not sure whether using HttpWebRequest demands data to be sent as bytes or not.
In the new code, I've created a StringContent to be sent to the endpoint. Are both the codes equivalent and works the intended way?
If not, some help to modify the new code is appreciated.
Using HttpWebRequest (old code)
//postData -> data to be sent(type string)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("someurl.net");
req.ContentType = "application/xml";
req.Accept = "application/xml";
byte[] postBytes = Encoding.UTF8.GetBytes(postData);
req.ContentLength = postBytes.Length;
Stream postStream = req.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Flush();
postStream.Close();
WebResponse resp = req.GetResponse();
Using HttpRequestMessage (new code)
//postData -> data to be sent(type string)
var request = new HttpRequestMessage(HttpMethod.Post, $"{address}");
request.Headers.Add("Accept", "application/xml");
request.Headers.Add("Content-Type", "application/xml");
request.Content = new StringContent(RSAEncryptDecrypt.EncryptResponse(postData));
await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken)
I suppose it depends on what the intent was originally :-)
HttpWebRequest is very flexible to what it will send out. Byte[], FormUrlEncodedContent, Multipart Content, Stream, String, ...you get the idea.
There are some performance considerations, again depending on your content. End of the day if you test it and it passes within the required parameters, then you should be OK. Given someone is telling you to do it, it might be in your best interest to ask said someone a few additional questions.
One thing that does stick out to me, in the original code your string is encoded in UTF8 and your new code appears to perhaps be something else. May want to explicitly call out the encoding parameter of StringContent if you choose to keep going that way.
Here's a quick example that should get you up and going. A little inflated, but hopefully it helps you out :-)
public static void PostSomeContent(string data_parameter, string uri_parameter){
HttpRequestMessage _httpRequestMessage;
MemoryStream _memoryStream;
StreamWriter _streamWriter;
StreamContent _streamContent;
HttpClient _client;
HttpResponseMessage _httpResponse;
//Initilize a HttpRequest Message
_httpRequestMessage =
new HttpRequestMessage();
//Set the end point
//You could also do this in the intilization
_httpRequestMessage.RequestUri =
new Uri(uri_parameter);
//Set your headers
_httpRequestMessage.Headers.Add("Accept", "application/xml");
_httpRequestMessage.Headers.Add("Content-Type", "application/xml");
//Set the methood
_httpRequestMessage.Method =
HttpMethod.Post;
//I'm asuming you want it to be async
Task.Run(async()=>{
//Set up the stream
using(_memoryStream = new MemoryStream()){
using(_streamWriter = new StreamWriter(_memoryStream)){
_streamWriter.Write(data_parameter);
_streamWriter.Flush();
_memoryStream.Seek(
offset:0,
loc: System.IO.SeekOrigin.Begin
);
using(_streamContent = new StreamContent(_memoryStream)){
_httpRequestMessage.Content = _streamContent;
_client =
new HttpClient();
try{
//Use the client to send your message
//Configure what you want back via completion option
using(_httpResponse = await _client.SendAsync(
request: _httpRequestMessage,
completionOption: HttpCompletionOption.ResponseHeadersRead)){
//Or however you would like to make sure there was no error
_httpResponse.EnsureSuccessStatusCode();
//If you want the response as a string:
string _content = await _httpResponse.Content.ReadAsStringAsync();
//Or response in a stream:
using(Stream _responseStream = await _httpResponse.Content.ReadAsStreamAsync()){
}
}
}
catch(Exception){
//Handle your exception
}
}
}
}
});
}
I'm trying to download an Excel file from a Tableau view, using a snippet based on this article: Downloading Files with the WebRequest and WebResponse Classes.
WebResponse response = null;
Stream webStream = null;
Stream localStream = null;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";
request.Headers.Add("X-Tableau-Auth", token);
response = request.GetResponse();
webStream = response.GetResponseStream();
localStream = File.Create("testing.xlsx");
byte[] buffer = new Byte[1024];
int bytesRead;
int bytesProcessed = 0;
do
{
bytesRead = webStream.Read(buffer, 0, buffer.Length);
localStream.Write(buffer, 0, bytesRead);
bytesProcessed += bytesRead;
} while (bytesRead > 0);
response.Close();
webStream.Close();
localStream.Close();
But when I try to open the Excel file it says "Nope, maybe it is corrupt?". The response is a complete Excel file "ready to be saved", which means that it's encoded using UTF-8. Indeed, if I use Postman to do the call, and then save the response, it's saved and opens without any problem.
Before finding the page I said above, I thought the problem was because the response is UTF-8 and the class String is UTF-16. So, I made some testing getting the data from the file generated by Postman and then writing to a new file. The result of the testings: indeed, if the data passes through a String, it's not well saved. Then I tried with that code, and got the same result: fail. I'm pretty sure this code is not using any UTF-16 encoding variable, but maybe I'm wrong.
Anyway, do anyone knows what is my problem with that code, or point me to the right way to accomplish my task? That is, to download a Tableau report to Excel, using the Tableau API.
Thanks in advance.
Unless you're stuck on a particularly old version of .NET, that referenced link is from 2004. The following code will work in .NET 5 / C# 9, and in earlier versions with just some minor tweaking of the using statements. It's showing for Tableau Online, but should work fine for recent versions of Server, if I had to guess. This is sample-grade code, so I would recommend following best practices for the HttpClient if you intend to make this call frequently.
//using System.IO;
//using System.Net.Http;
var token = "your-session-token";
var uri = "https://your-pod.online.tableau.com/api/.../sites/.../views/.../crosstab/excel";
var yourFile = "D:/file/test.xlsx";
using var client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Get, uri);
request.Headers.Add("X-Tableau-Auth", token);
using var response = await client.SendAsync(request);
using FileStream outputFileStream = new FileStream(yourFile, FileMode.Create);
await response.Content.CopyToAsync(outputFileStream);
UPDATE: If you're constrained to WebRequest and non-async methods, you can try the following:
var token = "your-session-token";
var uri = "https://your-pod.online.tableau.com/api/.../sites/.../views/.../crosstab/excel";
var yourFile = "D:/file/test.xlsx";
WebRequest request = WebRequest.Create(uri);
request.Method = "GET";
request.Headers.Add("X-Tableau-Auth", token);
var response = request.GetResponse();
if(((HttpWebResponse)response).StatusCode == HttpStatusCode.OK)
{
using (Stream dataStream = response.GetResponseStream())
using (FileStream fileStream = new FileStream(yourFile, FileMode.CreateNew))
{
dataStream.CopyTo(fileStream);
}
}
response.Close();
We have a created an API for the application which takes the image via POST request process it and sends the result in JSON format.
We tried calling API from different sources like python, postman app, c#. We can successfully call end point using python and postman app but with c# getting error
c# code [Not working]
byte[] img_data = System.IO.File.ReadAllBytes(#"file_path");
string url_ep = "http://ip:port/get";
Dictionary<string, byte[]> fl_image = new Dictionary<string, byte[]>();
fl_image.Add("image", img_data);
string data = JsonConvert.SerializeObject(fl_image);
var dataToSend = Encoding.UTF8.GetBytes(data);
var request = HttpWebRequest.Create(url_ep);
request.ContentType = "application/json";
request.ContentLength = dataToSend.Length;
request.Method = "POST";
request.GetRequestStream().Write(dataToSend, 0, dataToSend.Length);
var response = request.GetResponse();
System.IO.Stream dataStream = response.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
Console.WriteLine(responseFromServer);
python code [working]
import requests
url = 'http://ip:port/get'
fl_image = {'image': open('file_path', 'rb')}
res = requests.post(url, files=fl_image)
print(res.json())
API Endpoint
from flask import Flask, request
import numpy as np
import cv2 as cv
#app.route('/get', methods = ['POST'])
def get_image():
if request.method == 'POST':
file = request.files['image']
# Read file
f = file.read()
# convert string of image data to uint8
f1 = np.fromstring(f, np.uint8)
# decode image
f2 = cv.imdecode(f1,cv.IMREAD_COLOR)
There are several issues with the way you are posting data from C#. The most relevant one is that you are trying to post a file as a JSON object, with file contents as string.
This cannot work: your python server is clearly expecting multipart/form-data as content-type.
I also strongly recommend you to use HttpClient and not the old HttpWebRequest class to send HTTP Requests.
var filePath = #"file_path";
var url = "http://ip:port/get";
using (var client = new HttpClient())
using (var content = new MultipartFormDataContent())
using (var fileStream = File.OpenRead(filePath))
{
var imageContent = new StreamContent(fileStream);
// NOTE: the line below is not required, but useful when you know the media type
imageContent.Headers.ContentType = MediaTypeHeaderValue.Parse("image/jpeg");
content.Add(imageContent, "image", Path.GetFileName(filePath));
var response = await client.PostAsync(url, content);
var stringResponse = await response.Content.ReadAsStringAsync();
// do what you need with the response
}
Other minor issues:
Do not read the entire file in memory (using File.ReadAllBytes), but open a stream for reading instead.
Use async/await when possible, do not block on async code (do not use .Result, .Wait() or .GetAwaiter().GetResult() on Task or Task<T>)
Always call Dispose() on IDisposable objects when you have finished using them (wrapping them inside a using block)
You need to dispose the connections
reader.Close();
dataStream.Close();
response.Close();
Hope this helps
Or try using HttpClient for .net within the using block
I have an Infopath Form Template on Sharepoint, I want to add a button there so when the user clicks on it, it will POST an string to the following Web API. The following web API is tested and returns an excel file as shown:
I want to Post the FileName of the excel file using post request and it is important for me the request method to be POST type. and then the user will download a file with the specified 'FileName'.
Actually i want to use post method because at the next stage i will send the content of the excel file too.
Important Note: I only can use .Net FrameWork 3.5 because this is the only framework supported in InfoPath Form Templates.
[HttpPost]
public HttpResponseMessage Post([FromBody]string FileName)
{
string reqBook = "c:\somefile.xlsx";
//converting Excel(xlsx) file into bytes array
var dataBytes = File.ReadAllBytes(reqBook);
//adding bytes to memory stream
var dataStream = new MemoryStream(dataBytes);
HttpResponseMessage httpResponseMessage = Request.CreateResponse(HttpStatusCode.OK);
httpResponseMessage.Content = new StreamContent(dataStream);
httpResponseMessage.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
httpResponseMessage.Content.Headers.ContentDisposition.FileName = FileName;
httpResponseMessage.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
return httpResponseMessage;
}
When you perform the HttpPost on the client side, you will want to read the HttpResponseStream to get the byte data of the response stream.
Once you have the response stream data, you can then deserialize it to the type of object in C# you want, or you could alternatively just write it to the disk as
File.WriteAllBytes("someexcel.xlsx",data);
An easy way to do it would be with the HttpClient class.
HttpClient client = new HttpClient();
var response = client.PostAsync("", null).Result;
var content = response.Content.ReadAsByteArrayAsync().Result;
File.WriteAllBytes("excel.xlsx", content);
Just fill in the PostAsync bit with the Url and the content you wish to post.
I am using .Result to keep everything synchronous - but you can use 'await' if you prefer.
If you are working with HttpWebRequests - then the process becomes more complicated, as you need to manage the streams yourself.
The HttpClient will manage and handle it all for you - so I recommend it, unless there is something special it needs to do that it currently does not.
Due to your .Net 3.5 requirement:
private static HttpWebResponse MakeRequest(string url, string postArgument)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "multipart/form-data;";
Stream stream = request.GetRequestStream();
string result = string.Format("arg1={0}", postArgument);
byte[] value = Encoding.UTF8.GetBytes(result);
stream.Write(value, 0, value.Length);
stream.Close();
return (HttpWebResponse)request.GetResponse();
}
You can then do:
var response = MakeRequest("http://mywebsite.com/ProcessExcel", "accounts.xlsx");
And then do
Stream objStream = response .GetResponseStream();
BinaryReader breader = new BinaryReader(objStream);
byte[] data= breader.ReadBytes((int)webresponse.ContentLength);
File.WriteAllBytes("excel.xlsx",data);
I'm trying to migrate a Windows C# library to Windows Phone 8 and I'm forced to make some changes in how the library gets data from an online URI.
I'm using the BCL's HttpClient to perform my data retrieval and everything's fine for now.
The library also requires an upload feature, and I can't find a way to do this using th HttpClient.
Here's the code:
// Client setup
var httpclient = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Post, string.Format(SubmitURI, value));
// Add the headers
request.Headers.Add("header", header);
var postData = GetPostData();
var data = Encoding.UTF8.GetBytes(postData);
// Side question -> Content is null here!
request.Content.Headers.Add("ContentType", "multipart/form-data; boundary=" + Boundary);
// BEGIN ORIGINAL CODE
var stream = request.GetRequestStream();
stream.Write(data, 0, data.Length);
stream.Close();
// END ORIGINAL CODE
// Get response
var response = await httpclient.SendAsync(request);
var responseContent = new StreamReader(await response.Content.ReadAsStreamAsync()).ReadToEnd();
Between the BEGIN ORIGINAL CODE and END ORIGINAL CODE comments, there's the code that I'm not able to migrate, so that you can understand what it does and I may need to make it work on WP.
The other of the code is already working on WP, except for the
request.Content.Headers.Add("ContentType", "multipart/form-data; boundary=" + Boundary);
because, for some reasons, request.Content is null.
So my question is: how can I migrate those 3 lines to WP using HttpClient (or any better way)?
And, as a little side question, why is request.Content null?
EDIT: based on #Johan van der Slikke's answer I've edited my code. It compiles, but the server reports that no file was uploaded, so I guess that there are still some issues.
var stream = new MemoryStream(data);
var streamContent = new StreamContent(stream);
request.Content = streamContent;
request.Content.Headers.Add("ContentType", "multipart/form-data; boundary=" + Boundary);
// Get response
var response = await httpclient.SendAsync(request);
You should wrap your stream in the StreamContent class (a subclass of HttpContent) and send it with the HttpClient using the PostAsync or PutAsync methods.
Edit
You don't need to use HttpRequestMessage.
var stream = new MemoryStream(data);
var streamContent = new StreamContent(stream);
// Get response
var response = await httpclient.PostAsync(streamContent)
Also you don't need to create a MemoryStream with your byte array. You can wrap it in a ByteArrayContent directly.
var response = await httpclient.PostAsync(new ByteArrayContent(data))
Maybe (because I see you using multipart/form-data header) you should use MultipartFormDataContent classes or FormUrlEncodedContentClasses.
You can send multi-part content like this,
var client = new HttpClient();
var content = new MultipartContent();
content.Add(new ByteArrayContent(data));
var response = await client.PostAsync(SubmitUrl, content);
However, I am curious what media type your "PostData" is. You may want to set the content-type header on the ByteArrayContent.