I have a requirement to upload large files (upto 2GB) from my .NET web application to a java web service which in turn consumes a .jar file.
The java webservice API accepts MultipartFormDataContent as its parameter.
Problem I am facing is, I am unable to load the entire 2GB into a byte array as it throws "SystemOutOfMemoryException" when I attempt to upload a file anything larger that 300MB.
I also tried BufferedReader, StreamWriter but in vain.
Provided my code below for your reference:
public bool SendMessage(Dictionary<string, byte[]> files, string fromAddress, string toAddresses, string ccAddresses, string subject, string body)
{
JavaScriptSerializer jss = new JavaScriptSerializer();
Dictionary<string, long> fileSizes = new Dictionary<string, long>();
Dictionary<string, ByteArrayContent> fileContent = new Dictionary<string, ByteArrayContent>();
HttpContent fileSizesContent = null;
try
{
HttpContent messageContent = new StringContent(jss.Serialize(new
{
to = toAddress,
cc = ccAddresses,
subject = subject,
body = "Test"
}));
if (files != null)
{
foreach (var entry in files)
{
fileSizes.Add(entry.Key, entry.Value.Length);
fileContent.Add(entry.Key, new ByteArrayContent(entry.Value));
}
fileSizesContent = new StringContent(jss.Serialize(fileSizes));
}
using (var client = new HttpClient())
{
using (var formData = new MultipartFormDataContent())
{
if (fileContent.Count > 0)
{
foreach (var entry in fileContent)
{
formData.Add(entry.Value, "attachments", entry.Key);
}
formData.Add(fileSizesContent, "fileSizes");
}
formData.Add(messageContent, "message");
var response = client.PostAsync(<java web service url>, formData).Result;
if (!response.IsSuccessStatusCode)
{
return false;
}
return true;
}
}
}
catch (Exception ex)
{
return false;
}
}
Issue is: I am unable to opulate the parameter ByteArrayContent as it throws SystemOutOfMemoeyException for files >300MB.
Please help me out.
Thanks.
Related
I am uploading multiple files through a form and then iterating through each file and am creating a zero byte temp file and getting the file name using the GetTempFileName. Is there a way to copy the file itself not just the file name and the zero byte?. The code runs but i always get the zero byte file on the other end of my API which should happen. My code is like below.
public async Task<IActionResult> UploadDocumentsAsync(Documents files)
{
try
{
List<string> tempFilePath = new List<string>();
List<string> fileName = new List<string>();
foreach (var doc in files.documents)
{
tempFilePath.Add(Path.GetTempFileName());
fileName.Add(doc.FileName);
}
MultipartFormDataContent formData = new MultipartFormDataContent();
var filePath = tempFilePath;
string token = Token();
List<FileStream> streams = new List<FileStream>();
using (var client = new HttpClient())
{
using (var content = new MultipartFormDataContent())
{
for (int i = 0; i < tempFilePath.Count; i++)
{
var fileStream = new FileStream(tempFilePath[i], FileMode.Open);
formData.Add(new StreamContent(fileStream), fileName[i], fileName[i]);
streams.Add(fileStream);
}
var request = new HttpRequestMessage(HttpMethod.Post, new Uri(_configuration["Application:AppDomain"]) + "api/Document")
{
Content = formData
};
request.Headers.Add("accept", "application/json");
var response = await client.SendAsync(request);
streams.ForEach(stream =>
{
stream.Dispose();
});
if (response.IsSuccessStatusCode)
{
//Handle success
}
//Handle failure
}
}
}
catch (Exception e)
{
//Handle the exception
}
}
You need to save the actual file to temp fileName if you want to access it later
public async Task<IActionResult> UploadDocumentsAsync(Documents files)
{
try
{
List<string> tempFilePath = new List<string>();
List<string> fileName = new List<string>();
foreach (var doc in files.documents)
{
string fileTempFileName = Path.GetTempFileName();
tempFilePath.Add(fileTempFileName );
fileName.Add(doc.FileName);
using(Stream outStream = File.OpenWrite(Path.Combine(Path.GetTempPath(), fileTempFilePath )))
{
doc.CopyTo(outStream);
}
}
//Continue your code [...]
}
catch (Exception e)
{
//Handle the exception
}
}
I am trying to connect to an api, that returns GZip encoded JSON, from a WCF service (WCF service to WCF service). I am using the HTTPClient to connect to the API and have been able to return the JSON object as a string. However I need to be able to store this returned data in a database and as such I figured the best way would be to return and store the JSON object in an array or byte or something along those lines.
What I am having trouble with specifically is the decompressing of the GZip encoding and have been trying lots of different example but still cant get it.
The below code is how I am establishing my connection and getting a response, this is the code that returns a string from the API.
public string getData(string foo)
{
string url = "";
HttpClient client = new HttpClient();
HttpResponseMessage response;
string responseJsonContent;
try
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
response = client.GetAsync(url + foo).Result;
responseJsonContent = response.Content.ReadAsStringAsync().Result;
return responseJsonContent;
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return "";
}
}
I have been following a few different examples like these StackExchange API, MSDN, and a couple on stackoverflow, but I haven't been able to get any of these to work for me.
What is the best way to accomplish this, am I even on the right track?
Thanks guys.
Just instantiate HttpClient like this:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (var client = new HttpClient(handler)) //see update below
{
// your code
}
Update June 19, 2020:
It's not recommended to use httpclient in a 'using' block as it might cause port exhaustion.
private static HttpClient client = null;
ContructorMethod()
{
if(client == null)
{
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
client = new HttpClient(handler);
}
// your code
}
If using .Net Core 2.1+, consider using IHttpClientFactory and injecting like this in your startup code.
var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
TimeSpan.FromSeconds(60));
services.AddHttpClient<XApiClient>().ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
}).AddPolicyHandler(request => timeout);
I used code from below link to decompress GZip stream.Then used the decompressed byte array to get the required JSON object. Hope it may help some one.
var readTask = result.Content.ReadAsByteArrayAsync().Result;
var decompressedData = Decompress(readTask);
string jsonString = System.Text.Encoding.UTF8.GetString(decompressedData, 0, decompressedData.Length);
ResponseObjectClass responseObject = Newtonsoft.Json.JsonConvert.DeserializeObject<ResponseObjectClass>(jsonString);
https://www.dotnetperls.com/decompress
static byte[] Decompress(byte[] gzip)
{
using (GZipStream stream = new GZipStream(new MemoryStream(gzip), CompressionMode.Decompress))
{
const int size = 4096;
byte[] buffer = new byte[size];
using (MemoryStream memory = new MemoryStream())
{
int count = 0;
do
{
count = stream.Read(buffer, 0, size);
if (count > 0)
{
memory.Write(buffer, 0, count);
}
}
while (count > 0);
return memory.ToArray();
}
}
}
Ok so I eventually solved my problem. If there are better ways please let me know :-)
public DataSet getData(string strFoo)
{
string url = "foo";
HttpClient client = new HttpClient();
HttpResponseMessage response;
DataSet dsTable = new DataSet();
try
{
//Gets the headers that should be sent with each request
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//Returned JSON
response = client.GetAsync(url).Result;
//converts JSON to string
string responseJSONContent = response.Content.ReadAsStringAsync().Result;
//deserializes string to list
var jsonList = DeSerializeJsonString(responseJSONContent);
//converts list to dataset. Bad name I know.
dsTable = Foo_ConnectAPI.ExtentsionHelpers.ToDataSet<RootObject>(jsonList);
//Returns the dataset
return dsTable;
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return null;
}
}
//deserializes the string to a list. Utilizes JSON.net. RootObject is a class that contains the get and set for the JSON elements
public List<RootObject> DeSerializeJsonString(string jsonString)
{
//Initialized the List
List<RootObject> list = new List<RootObject>();
//json.net deserializes string
list = (List<RootObject>)JsonConvert.DeserializeObject<List<RootObject>>(jsonString);
return list;
}
The RootObject contains the get set that will get the values of the JSON.
public class RootObject
{
//These string will be set to the elements within the JSON. Each one is directly mapped to the JSON elements.
//This only takes into account a JSON that doesn't contain nested arrays
public string EntityID { get; set; }
public string Address1 { get; set; }
public string Address2 { get; set; }
public string Address3 { get; set; }
}
The easiest way to create the above class(es) is to use json2charp which will format it accordingly and also provide the correct datatypes.
The following is from another answer on Stackoverflow
again it does not take into account nested JSON.
internal static class ExtentsionHelpers
{
public static DataSet ToDataSet<T>(this List<RootObject> list)
{
try
{
Type elementType = typeof(RootObject);
DataSet ds = new DataSet();
DataTable t = new DataTable();
ds.Tables.Add(t);
try
{
//add a column to table for each public property on T
foreach (var propInfo in elementType.GetProperties())
{
try
{
Type ColType = Nullable.GetUnderlyingType(propInfo.PropertyType) ?? propInfo.PropertyType;
t.Columns.Add(propInfo.Name, ColType);
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
}
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
try
{
//go through each property on T and add each value to the table
foreach (RootObject item in list)
{
DataRow row = t.NewRow();
foreach (var propInfo in elementType.GetProperties())
{
row[propInfo.Name] = propInfo.GetValue(item, null) ?? DBNull.Value;
}
t.Rows.Add(row);
}
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
insert.insertCategories(t);
return ds.
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return null;
}
}
};
Then finally to insert the above dataset into a table with columns that were mapped to the JSON I utilized SQL bulk copy and the following class
public class insert
{
public static string insertCategories(DataTable table)
{
SqlConnection objConnection = new SqlConnection();
//As specified in the App.config/web.config file
objConnection.ConnectionString = System.Configuration.ConfigurationManager.ConnectionStrings["foo"].ToString();
try
{
objConnection.Open();
var bulkCopy = new SqlBulkCopy(objConnection.ConnectionString);
bulkCopy.DestinationTableName = "dbo.foo";
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.WriteToServer(table);
return "";
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return "";
}
finally
{
objConnection.Close();
}
}
};
So the above works to insert JSON from a webAPI into a database. This is something that I get to work. But by no means do I expect it to be perfect. If you have any improvements then please update it accordingly.
Sooner or later your code might brake if your server use another compression scheme like 'Brotli' with content-type: br
I have made a video to handle decompression in httpClient with the clientHandler:
compression algos handled by httpClientHandler
We are building a web application that consist of an Angular2 frontend, a ASP.NET Core web api public backend, and a ASP.NET Core web api private backend.
Uploading files from Angular2 to the public backend works. But we would prefer to post them forward to the private backend.
Current working code
[HttpPost]
public StatusCodeResult Post(IFormFile file)
{
...
}
From there I can save the file to disk using file.CopyTo(fileStream);
However, I want to re-send that file, or those files, or, ideally, the whole request to my second web api core.
I am not sure how to achieve this with the HttpClient class of asp.net core.
I've tried all kinds of things such as
StreamContent ss = new StreamContent(HttpContext.Request.Body);
var result = client.PostAsync("api/Values", ss).Result;
But my second backend gets an empty IFormFile.
I have a feeling it is possible to send the file(s) as a stream and reconstruct them on the other side, but can't get it to work.
The solution must use two web api core.
Solution
Public backend in DMZ
[HttpPost]
public StatusCodeResult Post(IFormFile file)
{
try
{
if (file != null && file.Length > 0)
{
using (var client = new HttpClient())
{
try
{
client.BaseAddress = new Uri(currentPrivateBackendAddress);
byte[] data;
using (var br = new BinaryReader(file.OpenReadStream()))
data = br.ReadBytes((int)file.OpenReadStream().Length);
ByteArrayContent bytes = new ByteArrayContent(data);
MultipartFormDataContent multiContent = new MultipartFormDataContent();
multiContent.Add(bytes, "file", file.FileName);
var result = client.PostAsync("api/Values", multiContent).Result;
return StatusCode((int)result.StatusCode); //201 Created the request has been fulfilled, resulting in the creation of a new resource.
}
catch (Exception)
{
return StatusCode(500); // 500 is generic server error
}
}
}
return StatusCode(400); // 400 is bad request
}
catch (Exception)
{
return StatusCode(500); // 500 is generic server error
}
}
Private backend
[HttpPost]
public void Post()
{
//Stream bodyStream = HttpContext.Request.Body;
if (Request.HasFormContentType)
{
var form = Request.Form;
foreach (var formFile in form.Files)
{
var targetDirectory = Path.Combine(_appEnvironment.WebRootPath, "uploads");
var fileName = GetFileName(formFile);
var savePath = Path.Combine(targetDirectory, fileName);
using (var fileStream = new FileStream(savePath, FileMode.Create))
{
formFile.CopyTo(fileStream);
}
}
}
}
Hi i had the same issue and this is what worked for me :
My setup is netCore MVC netCoreApi.
My MVC Controller looks like :
[HttpPost("UploadFiles")]
public async Task<IActionResult> Post(List<IFormFile> files)
{
Sp4RestClient dataPovider = new Sp4RestClient("http://localhost:60077/");
long size = files.Sum(f => f.Length);
foreach (var file in files)
{
await dataPovider.ImportFile(file);
}
return Ok();
}
DataProvider Method :
public async Task ImportFile(IFormFile file)
{
RestClient restClient = new RestClient(_queryBulder.BuildImportFileRequest());
using (var content = new MultipartFormDataContent())
{
content.Add(new StreamContent(file.OpenReadStream())
{
Headers =
{
ContentLength = file.Length,
ContentType = new MediaTypeHeaderValue(file.ContentType)
}
}, "File", "FileImport");
var response = await restClient.Post<IFormFile>(content);
}
}
And least my WebApi Controller :
[HttpPost]
[Route("ImportData")]
public IActionResult Import(IFormFile file)
{
return Ok();
}
To see the complete code here is my RestClient Post method :
public async Task<RestResult<T>> Post<T>(HttpContent content)
{
using (HttpClient httpClient = new HttpClient())
{
HttpResponseMessage response = await httpClient.PostAsync(Endpoint, content);
if (response.StatusCode == HttpStatusCode.Created)
{
T result = JsonConvert.DeserializeObject<T>(await response.Content.ReadAsStringAsync());
return new RestResult<T> { Result = result, ResultCode = HttpStatusCode.OK };
}
RestResult<T> nonOkResult =
new RestResult<T> { Result = default(T), ResultCode = response.StatusCode, Message = await response.Content.ReadAsStringAsync() };
return nonOkResult;
}
}
// Yeah i know im not getting HttpStatusCode.Created back ;)
happy coding ;)
API Code
[Route("api/upload/{id}")]
[HttpPost]
public async Task<IActionResult> Post(string id)
{
var filePath = #"D:\" + id; //+ Guid.NewGuid() + ".png";
if (Request.HasFormContentType)
{
var form = Request.Form;
foreach (var formFile in form.Files)
{
if (formFile.Length > 0)
{
using (var stream = new FileStream(filePath, FileMode.Create))
{
await formFile.CopyToAsync(stream);
}
}
}
}
return Ok(new { Path = filePath });
}
Back End
[Route("home/UploadFile")]
[HttpPost]
public IActionResult UploadFile(IFormFile file)
{
if (file == null || file.Length == 0)
return Content("file not selected");
var client = new HttpClient();
byte[] data;
using (var br = new BinaryReader(file.OpenReadStream()))
data = br.ReadBytes((int)file.OpenReadStream().Length);
ByteArrayContent bytes = new ByteArrayContent(data);
MultipartFormDataContent multiContent = new MultipartFormDataContent
{
{ bytes, "file", file.FileName }
};
var result = client.PostAsync("http://localhost:2821/api/upload/" + file.FileName, multiContent).Result;
return RedirectToAction("file");
}
Download Source
I was in a similar situation - I needed a proxy method for forwarding not only files but also JSON data and whatnot. I did not want to do any analysis of the data in my proxy to let the final receiver deal with it.
So with some help from #Anton Tykhyy I came to the following working solution:
byte[] arr = null;
using (var mems = new MemoryStream())
{
// read entire body into memory first because it might be chunked with unknown length
await request.Body.CopyToAsync(mems);
await mems.FlushAsync(); // not sure if needed after CopyToAsync - better safe then sorry
arr = mems.ToArray();
}
msg.Content = new ByteArrayContent(arr);
msg.Content.Headers.ContentLength = arr.Length;
// keep content-type header "as is" to preserve multipart boundaries etc.
msg.Content.Headers.TryAddWithoutValidation("Content-Type", request.ContentType);
var response = await _httpClient.SendAsync(msg);
I tested it with complex request that contained multipart form data with JSON field and multiple attached files, and all the data reached my backend server without any issues.
Ignoring the HttpClient when you call the private backend API, can you reference the private Core API project from the public Core API project and call the controller directly from the Core API project? See the request is still null/empty. If the request comes out with a value then the issue is with the use of the HttpClient.
Ideally, you want to create a package library(kind of SDK) for your private Core API that you want to distribute to consuming clients. This acts like a wrapper/proxy. This way you can isolate the private backend system and you can troubleshoot it in isolation. So you public Core API project(which is the private backend client) can reference it as nuget package.
I am working with RestSharp and create object of RestRequest for sending FileData to API. But after getting response I want to delete the file from my local machine but when I try to do the same it gives me the error "File is being used by other process". The reason I think is that I am unable to dispose the object of RestRequest. Please help me to solve it. Below is the code. Thanks in Advance..!!!
public string PostMultiformDataAPI(Method method, string apiUrl, string data = "", Dictionary<string, string> headers = null)
{
string[] files = null;
try
{
RestClient client = new RestClient(apiUrl);
var request = new RestRequest();
request.Method = method;
//Add header values to request
if (headers != null)
{
foreach (var header in headers)
{
request.AddHeader(header.Key, header.Value);
}
}
string filename = string.Empty;
if (Directory.Exists(HttpContext.Current.Server.MapPath("/Upload")))
{
files = Directory.GetFiles(HttpContext.Current.Server.MapPath("/Upload"));
foreach (string file in files)
{
request.AddFile(file.Split('/').Last(), file);
}
}
// execute the request
IRestResponse response = client.Execute(request);
var content = response.Content; // raw content as string
foreach (string file in files)
{
GC.Collect();
GC.WaitForPendingFinalizers();
var fileInfo = new FileInfo(file);
fileInfo.Refresh();
fileInfo.Delete();
//File.Delete(file);
}
return content;
}
finally
{
}
}
You just need to assign null to request object instance to remove the reference it has to file. Worked for me. Please let me know if it works for you.
// execute the request
IRestResponse response = client.Execute(request);
var content = response.Content; // raw content as string
request = null;
response = null;
foreach (string file in files)
{
GC.Collect();
GC.WaitForPendingFinalizers();
var fileInfo = new FileInfo(file);
fileInfo.Refresh();
fileInfo.Delete();
File.Delete(file);
}
return content;
I need some help for posting a MultipartFormDataContentto my web service.
I have an error 500 but I didn't know why.
There is a mean to see what kind of data i send with visual studio? Or there is a mistake in my code?
My c# code Client
using (var client = new HttpClient())
{
using (var content =
new MultipartFormDataContent())
{
MemoryStream s = new MemoryStream();
StreamWriter writer = new StreamWriter(s);
writer.Write(json);
writer.Flush();
s.Position = 0;
content.Add(new StreamContent(s), "JSON");
foreach (KeyValuePair<String, byte[]> pair in data)
{
Stream stream = new MemoryStream(pair.Value);
content.Add(new StreamContent(stream),"uploaded");
}
using (
var message =
await client.PostAsync(urlFinal, content))
{
var input = await message.Content.ReadAsStringAsync();
}
}
}
And the Web Service part in JAVA
public static Result Method() {
MultipartFormData data = request().body().asMultipartFormData();
return ok(toJson(MyObject.myMethode(data)));
}
public static int myMethode(MultipartFormData data) {
FilePart JSON = data.getFile("JSON");
FilePart picture = data.getFile("uploaded");
return 1;
}