Develop Reference

C# - How to upload bigger files using FTP? [duplicate]

This question already has an answer here:
Upload and download a file to/from FTP server in C#/.NET
(1 answer)
Closed 4 years ago.
The code below works when i upload files around 12kb and below but for some reason anything above that size results in the program freezing and getting an error:
"The underlying connection was closed: An unexpected error occurred on a receive"
Is there something I am doing wrong?
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.IO;
using System.Net;
namespace FTP3
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void uploadFile(string filePath)
{
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create("ftp://175.137.158.136" + "/" + Path.GetFileName(filePath));
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential(username, password);
request.UsePassive = true;
request.UseBinary = true;
request.KeepAlive = false;
FileStream stream = File.OpenRead(filePath);
byte[] buffer = new byte[stream.Length];
stream.Read(buffer, 0, buffer.Length);
stream.Close();
Stream reqStream = request.GetRequestStream();
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
MessageBox.Show("Upload complete!");
}
private void button1_Click(object sender, EventArgs e)
{
uploadFile(#"C:\Users\User\Desktop\test\data.xlsx");
}
}
}

you should upload the file in 'chunks', not completely. So please try the following:
using (FileStream fileStream = File.OpenRead(filePath))
{
using (Stream reqStream = request.GetRequestStream())
{
long curFileStreamPos = 0;
long chunkSize = 512;
while (curFileStreamPos < fileStream.Length)
{
if (fileStream.Length - curFileStreamPos < chunkSize)
chunkSize = fileStream.Length - curFileStreamPos;
byte[] buff = new byte[chunkSize];
fileStream.Read(buff, 0, buff.Length);
reqStream.Write(buff, 0, buff.Length);
curFileStreamPos += chunkSize;
}
reqStream.Close();
}
fileStream.Close();
}
I have not tested this snippet, so please verify indexes especially.
If you upload a file at once it may happen that the tcp/ip stack does get timeouts due to preparing the data to upload.
Further, this way you do not create big memory objects when you upload files with hundreds of mb (which would end up completely in memory at your solution)

Embedded Resources(resx) are giving me trouble- memory consumption

Consider the following class file that I use to access a .resx that contains multiple sound files with different file sizes.
using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using System.Reflection;
using System.Text;
namespace Resources
{
public class ResManager
{
public static void SoundCopy(String ResourceName, String Dest)
{
try
{
var rm = Resources.Sounds.ResourceManager;
var resStream = rm.GetStream(ResourceName);
if (!File.Exists(Dest))
{
File.Create(Dest).Close();
}
FileStream fs = new FileStream(Dest, FileMode.OpenOrCreate);
CopyStream(resStream, fs);
resStream.Close();
resStream = null;
fs.Close();
fs = null;
rm.ReleaseAllResources();
rm = null;
}
catch (Exception e)
{
// nothlng
}
} // S copy();
private static void CopyStream(Stream from, Stream to)
{
int bufSize = 1024, count;
byte[] buffer = new byte[bufSize];
count = from.Read(buffer, 0, bufSize);
while (count > 0)
{
// StackOverflow -- working set raises here
to.Write(buffer, 0, count);
count = from.Read(buffer, 0, bufSize);
}
// clean up
buffer = null;
} // c str()
} //class
} //namespace
Whatever I copy a different file to the filesystem, I noticed that the program keeps a certain Working Set permanently allocated after i do that, but copying the same file for more than 2 times doesn't.
I am trying to keep my app efficient , i tried to make a duplicate class that needs to be instanciated to do this, then i gave it null value and i've waited 40 minutes to see if the GC did something, but nope.
Normally my app runs at 29 MB working set jumping every 10 s to 37~ MB. After I copied 3 files to temp folder, the working set raised at 133 MB, and it stayed at that during my 40 minute pause.
Additional note: I am targetting .NET 2.0

How to check start and end of a message coming in stream in c#?

I am receiving data from streaming of bytes.
Code of TCPListerner :
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading.Tasks;
class Program
{
static void Main(string[] args)
{
TcpListener tcpListener = new TcpListener(IPAddress.Any, 6054);
tcpListener.Start();
Console.WriteLine("Waiting for a client to connect...");
//blocks until a client connects
Socket socketForClient = tcpListener.AcceptSocket();
Console.WriteLine("Client connected");
//Read data sent from client
NetworkStream networkStream = new NetworkStream(socketForClient);
int bytesReceived, totalReceived = 0;
string fileName = "testing.txt";
byte[] receivedData = new byte[10000];
do
{
bytesReceived = networkStream.Read
(receivedData, 0, receivedData.Length);
totalReceived += bytesReceived;
Console.WriteLine("Total bytes read: " + totalReceived.ToString());
if (!File.Exists(fileName))
{
using (File.Create(fileName)) { };
}
using (var stream = new FileStream(fileName, FileMode.Append))
{
stream.Write(receivedData, 0, bytesReceived);
}
}
while (bytesReceived != 0);
Console.WriteLine("Total bytes read: " + totalReceived.ToString());
socketForClient.Close();
Console.WriteLine("Client disconnected...");
Console.ReadLine();
}
}
I want to add functionality to process only that message which is coming from streaming starting with <Product> and want to check when the message ends. For end we can use </Products>.
How to perform this validation check in existing code, Any Idea ?

MemoryStream VS FileStream which one is better to use in Webservice for GZipStream decompress?

I have a asp.net webservice. some functionality of this webservice is to decompress clients request first. for this i have wrote 2 methods one uses MemoryStream and other uses FileStream.
When using MemoryStream sometiomes it get OutofMemoryException. so i have planned to use FileStream instead of MemoryStream for this reason.
before using this i just need a clarification that i am doing the right
thing for the right job.
N:B: Sometiome my clients will send 10MB+ data to the webservice which i need to decompress in webservice side. i have more then 200 clients running. and where the webservice is hosted there are more then 30 web application and webservice also hosted though my webservice is under different app pool.
I did see : GZIP decompression C# OutOfMemory
i get some knowledge from there but for webservice which one should better i have quite confusion based on my situation. and i need a clear understanding about this.
Decompress method (uses MemoryStream) is below:
public static string Decompress(string compressedText)
{
try
{
byte[] gzBuffer = Convert.FromBase64String(compressedText);
using (MemoryStream ms = new MemoryStream())
{
int msgLength = BitConverter.ToInt32(gzBuffer, 0);
ms.Write(gzBuffer, 4, gzBuffer.Length - 4);
byte[] buffer = new byte[msgLength];
ms.Position = 0;
using (GZipStream zip = new GZipStream(ms, CompressionMode.Decompress))
{
zip.Read(buffer, 0, buffer.Length);
}
return Encoding.UTF8.GetString(buffer);
}
}
catch (Exception ex)
{
DataSyncLog.Debug(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType + "::" + System.Reflection.MethodBase.GetCurrentMethod().ToString() + ":" + ex.ToString()+" : "+ex.StackTrace);
}
return string.Empty;
}
Decompress method (uses FileStream) is below:
public static string Decompress(string compressedText)
{
string SourceDirectory = System.Guid.NewGuid().ToString();
string DestinationDirectory = System.Guid.NewGuid().ToString();
try
{
File.WriteAllBytes(SourceDirectory, Convert.FromBase64String(compressedText));
using (FileStream fd = File.Create(DestinationDirectory))
{
using (FileStream fs = File.OpenRead(SourceDirectory))
{
fs.Seek(4, 0);
using (Stream csStream = new GZipStream(fs, CompressionMode.Decompress))
{
byte[] buffer = new byte[1024];
int nRead;
while ((nRead = csStream.Read(buffer, 0, buffer.Length)) > 0)
{
fd.Write(buffer, 0, nRead);
}
}
}
}
return Encoding.UTF8.GetString(File.ReadAllBytes(DestinationDirectory));
}
catch (Exception ex)
{
DataSyncLog.Debug(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType + "::" + System.Reflection.MethodBase.GetCurrentMethod().ToString() + ":" + ex.ToString() + " : " + ex.StackTrace);
return string.Empty;
}
finally
{
ClearFiles(SourceDirectory);
ClearFiles(DestinationDirectory);
}
}
Someone could you please show me the right direction that which one
should i use or any modification needed to the method that uses
MemoryStream that can overcome this error. i will be grateful to you
if you give me a clear understanding about this or any code change
suggestion.

Working with stream looks more efficient memory-wise in the second case: with memory stream you hold entire stream in memory, with file stream you have only buffer of limited size.
Both of your methods can have problems with memory just because of their signature: when client sends 10MB this amount of memory will be allocated for compressedText argument and for return value.
You can look into changing interface of your service, so data is transfered in chunks (here you can find example of similar approach -http://www.codeproject.com/Articles/43272/Uploading-Large-Files-Through-Web-Service)
Or, if you can consider switching to WCF, it supports streaming transfer mode - http://msdn.microsoft.com/en-us/library/ms751463.aspx

Partially download and serialize big file in C#?

As part of an upcoming project at my university, I need to write a client that downloads a media file from a server and writes it to the local disk. Since these files can be very large, I need to implement partial download and serialization in order to avoid excessive memory use.
What I came up with:
namespace PartialDownloadTester
{
using System;
using System.Diagnostics.Contracts;
using System.IO;
using System.Net;
using System.Text;
public class DownloadClient
{
public static void Main(string[] args)
{
var dlc = new DownloadClient(args[0], args[1], args[2]);
dlc.DownloadAndSaveToDisk();
Console.ReadLine();
}
private WebRequest request;
// directory of file
private string dir;
// full file identifier
private string filePath;
public DownloadClient(string uri, string fileName, string fileType)
{
this.request = WebRequest.Create(uri);
this.request.Method = "GET";
var sb = new StringBuilder();
sb.Append("C:\\testdata\\DownloadedData\\");
this.dir = sb.ToString();
sb.Append(fileName + "." + fileType);
this.filePath = sb.ToString();
}
public void DownloadAndSaveToDisk()
{
// make sure directory exists
this.CreateDir();
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
var rStream = response.GetResponseStream();
int bytesRead = -1;
do
{
var buf = new byte[2048];
bytesRead = rStream.Read(buf, 0, buf.Length);
rStream.Flush();
this.SerializeFileChunk(buf);
}
while (bytesRead != 0);
}
private void CreateDir()
{
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
}
private void SerializeFileChunk(byte[] bytes)
{
Contract.Requires(!Object.ReferenceEquals(bytes, null));
FileStream fs = File.Open(filePath, FileMode.Append);
fs.Write(bytes, 0, bytes.Length);
fs.Flush();
fs.Close();
}
}
}
For testing purposes, I've used the following parameters:
"http://itu.dk/people/janv/mufc_abc.jpg" "mufc_abc" "jpg"
However, the picture is incomplete (only the first ~10% look right) even though the content length prints 63780 which is the actual size of the image.
So my questions are:
Is this the right way to go for partial download and serialization or is there a better/easier approach?
Is the full content of the response stream stored in client memory? If this is the case, do I need to use HttpWebRequest.AddRange to partially download data from the server in order to conserve my client's memory?
How come the serialization fails and I get a broken image?
Do I introduce a lot of overhead when I use the FileMode.Append? (msdn states that this option "seeks to the end of the file")
Thanks in advance

You could definitely simplify your code using a WebClient:
class Program
{
static void Main()
{
DownloadClient("http://itu.dk/people/janv/mufc_abc.jpg", "mufc_abc.jpg");
}
public static void DownloadClient(string uri, string fileName)
{
using (var client = new WebClient())
{
using (var stream = client.OpenRead(uri))
{
// work with chunks of 2KB => adjust if necessary
const int chunkSize = 2048;
var buffer = new byte[chunkSize];
using (var output = File.OpenWrite(fileName))
{
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
}
}
}
Notice how I am writing only the number of bytes I have actually read from the socket to the output file and not the entire 2KB buffer.

I don't know if this is the source of the problem, however I would change the loop like this
const int ChunkSize = 2048;
var buf = new byte[ChunkSize];
var rStream = response.GetResponseStream();
do {
int bytesRead = rStream.Read(buf, 0, ChunkSize);
if (bytesRead > 0) {
this.SerializeFileChunk(buf, bytesRead);
}
} while (bytesRead == ChunkSize);
The serialize method would get an additional argument
private void SerializeFileChunk(byte[] bytes, int numBytes)
and then write the right number of bytes
fs.Write(bytes, 0, numBytes);
UPDATE:
I do not see the need for closing and reopening the file each time. I also would use the using statement, which closes the resources, even if an exception should occur. The using statement calls the Dispose() method of the resource at the end, which in turn calls Close() in the case of file streams. using can be applied to all types implementing IDisposable.
var buf = new byte[2048];
using (var rStream = response.GetResponseStream()) {
using (FileStream fs = File.Open(filePath, FileMode.Append)) {
do {
bytesRead = rStream.Read(buf, 0, buf.Length);
fs.Write(bytes, 0, bytesRead);
} while (...);
}
}
The using statement does something like this
{
var rStream = response.GetResponseStream();
try
{
// do some work with rStream here.
} finally {
if (rStream != null) {
rStream.Dispose();
}
}
}

Here is the solution from Microsoft: http://support.microsoft.com/kb/812406
Updated 2021-03-16: seems the original article is not available now. Here is the archived one: https://mskb.pkisolutions.com/kb/812406