This question already has answers here:
FtpWebRequest FTP download with ProgressBar
(3 answers)
Closed 2 years ago.
I'm with something new, I'm blocked and I don't know how to do the truth.
I am making a program that downloads by FTP to do a series of steps, the first thing is to download.
Here I have the code that makes the download which works perfect:
public static void DescargarFichero(string ficFTP, string user, string pass, string dirLocal, Boolean UsePassive, Boolean UseBinary)
{
FtpWebRequest dirFtp = ((FtpWebRequest)FtpWebRequest.Create(ficFTP));
dirFtp.KeepAlive = true;
dirFtp.UsePassive = UsePassive;
dirFtp.UseBinary = UseBinary;
// Los datos del usuario (credenciales)
NetworkCredential cr = new NetworkCredential(user, pass);
dirFtp.Credentials = cr;
FtpWebResponse response = (FtpWebResponse)dirFtp.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
using (FileStream writer = new FileStream(dirLocal, FileMode.Create))
{
long length = response.ContentLength;
int bufferSize = 2048;
int readCount;
byte[] buffer = new byte[2048];
readCount = responseStream.Read(buffer, 0, bufferSize);
while (readCount > 0)
{
writer.Write(buffer, 0, readCount);
readCount = responseStream.Read(buffer, 0, bufferSize);
Console.WriteLine("Descargando...");
}
}
reader.Close();
response.Close();
}
I am doing tests in a console application, but the future is to use windows form and that it looks good, my blocking is this: How can I show this to happen:
while (readCount > 0)
{
writer.Write(buffer, 0, readCount);
readCount = responseStream.Read(buffer, 0, bufferSize);
Console.WriteLine("Descargando...");
}
I can find out that this function is executing and that it is "downloading" I am looking for that every time that cycle is iterating I can return a value without having to break the cycle. My idea is to be able to say If Value is = X then "downloading"
Newbie question? Yes, I am blocked and I don't know how to get out of the "rat race". I hope you can help me.
Showing progress percentage
readCount = responseStream.Read(buffer, 0, bufferSize);
int current = 0; //Create a new int variable to count iterations
while (readCount > 0)
{
writer.Write(buffer, 0, readCount);
readCount = responseStream.Read(buffer, 0, bufferSize);
current++; //Increment the counter
double percentProgress = (current/readCount) * 100; //Calculate the percent
string percentString = percentProgress.ToString() + "%"; //Append a %
Console.WriteLine(percentString); //Write to console
}
See this post if you wish to keep it on one line: Can Console.Clear be used to only clear a line instead of whole console?
Related
I have tried to download an FTP file using C# and have had various problems. What I want to achieve is to be able to show download progress in a progressBar. It is important that I use Windows Form and .Net.
I have tried two codes;
My first code works perfectly, that is, I can download the FTP file without problems.
CODE 1
FtpWebRequest dirFtp = ((FtpWebRequest)FtpWebRequest.Create(ficFTP));
dirFtp.KeepAlive = true;
dirFtp.UsePassive = UsePassive;
dirFtp.UseBinary = UseBinary;
// Los datos del usuario (credenciales)
NetworkCredential cr = new NetworkCredential(user, pass);
dirFtp.Credentials = cr;
FtpWebResponse response = (FtpWebResponse)dirFtp.GetResponse();
long size = (long)response.ContentLength;
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
using (FileStream writer = new FileStream(dirLocal, FileMode.Create))
{
int bufferSize = 2048;
int readCount;
byte[] buffer = new byte[2048];
readCount = responseStream.Read(buffer, 0, bufferSize);
while (readCount > 0)
{
writer.Write(buffer, 0, readCount);
readCount = responseStream.Read(buffer, 0, bufferSize);
}
}
lblDescarga.Text = "¡Downloaded!";
reader.Close();
response.Close();
Problem with this code
My problem with this code is that I can't get the size of the FTP file to be able to use the progressBar, In theory this section of code would tell me the size of my file but it always returns -1:
long size = (long)response.ContentLength;
As this did not work as I wanted, I made a post and people recommended this solution FtpWebRequest FTP download with ProgressBar:
CODE 2
try
{
const string url = "ftp://185.222.111.11:21/patch/archive.zip";
NetworkCredential credentials = new NetworkCredential("user", "pass");
// Query size of the file to be downloaded
WebRequest sizeRequest = WebRequest.Create(url);
sizeRequest.Credentials = credentials;
sizeRequest.Method = WebRequestMethods.Ftp.GetFileSize;
int size = (int)sizeRequest.GetResponse().ContentLength;
progressBar1.Invoke(
(MethodInvoker)(() => progressBar1.Maximum = size));
// Download the file
WebRequest request = WebRequest.Create(url);
request.Credentials = credentials;
request.Method = WebRequestMethods.Ftp.DownloadFile;
using (Stream ftpStream = request.GetResponse().GetResponseStream())
using (Stream fileStream = File.Create(#"C:\tmp\archive.zip"))
{
byte[] buffer = new byte[10240];
int read;
while ((read = ftpStream.Read(buffer, 0, buffer.Length)) > 0)
{
fileStream.Write(buffer, 0, read);
int position = (int)fileStream.Position;
progressBar1.Invoke(
(MethodInvoker)(() => progressBar1.Value = position));
}
}
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
Problem with this code
The problem with this code is when it gets to this point:
int size = (int) sizeRequest.GetResponse (). ContentLength;
Remote server error: (550) File not available (eg file not found or not accessed).
The truth is that it is impossible to tell that you do not have permission if code 1 works well. However I have the normal permissions in FTP, could someone give me an idea please?
I have created some avro files. I can use the following commands to convert them to json, just to check whether the files are ok
java -jar avro-tools-1.8.2.jar tojson FileName.avro>outputfilename.json
Now, I have some big avro files and the REST API I m trying to upload to, has size limitations and thus I am trying to upload it in chunks using streams.
The following sample, which just reads from the original file in chunks and copies to another avro file, creates the file perfectly
using System;
using System.IO;
class Test
{
public static void Main()
{
// Specify a file to read from and to create.
string pathSource = #"D:\BDS\AVRO\filename.avro";
string pathNew = #"D:\BDS\AVRO\test\filenamenew.avro";
try
{
using (FileStream fsSource = new FileStream(pathSource,
FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[(20 * 1024 * 1024) + 100];
long numBytesToRead = (int)fsSource.Length;
int numBytesRead = 0;
using (FileStream fsNew = new FileStream(pathNew,
FileMode.Append, FileAccess.Write))
{
// Read the source file into a byte array.
//byte[] bytes = new byte[fsSource.Length];
//int numBytesToRead = (int)fsSource.Length;
//int numBytesRead = 0;
while (numBytesToRead > 0)
{
int bytesRead = fsSource.Read(buffer, 0, buffer.Length);
byte[] actualbytes = new byte[bytesRead];
Array.Copy(buffer, actualbytes, bytesRead);
// Read may return anything from 0 to numBytesToRead.
// Break when the end of the file is reached.
if (bytesRead == 0)
break;
numBytesRead += bytesRead;
numBytesToRead -= bytesRead;
fsNew.Write(actualbytes, 0, actualbytes.Length);
}
}
}
// Write the byte array to the other FileStream.
}
catch (FileNotFoundException ioEx)
{
Console.WriteLine(ioEx.Message);
}
}
}
How do I know this creates a ok avro. Because the earlier command to convert to json, again works i.e.
java -jar avro-tools-1.8.2.jar tojson filenamenew.avro>outputfilename.json
However, when I use the same code, but instead of copying to another file, just call a rest api, the file gets uploaded but upon downloading the same file from the server and running the command above to convert to json says - "Not a Data file".
So, obviously something is getting corrupted and I am struggling to figure out what.
This is the snippet
string filenamefullyqualified = path + filename;
Stream stream = System.IO.File.Open(filenamefullyqualified, FileMode.Open, FileAccess.Read, FileShare.None);
long? position = 0;
byte[] buffer = new byte[(20 * 1024 * 1024) + 100];
long numBytesToRead = stream.Length;
int numBytesRead = 0;
do
{
var content = new MultipartFormDataContent();
int bytesRead = stream.Read(buffer, 0, buffer.Length);
byte[] actualbytes = new byte[bytesRead];
Array.Copy(buffer, actualbytes, bytesRead);
if (bytesRead == 0)
break;
//Append Data
url = String.Format("https://{0}.dfs.core.windows.net/raw/datawarehouse/{1}/{2}/{3}/{4}/{5}?action=append&position={6}", datalakeName, filename.Substring(0, filename.IndexOf("_")), year, month, day, filename, position.ToString());
numBytesRead += bytesRead;
numBytesToRead -= bytesRead;
ByteArrayContent byteContent = new ByteArrayContent(actualbytes);
content.Add(byteContent);
method = new HttpMethod("PATCH");
request = new HttpRequestMessage(method, url)
{
Content = content
};
request.Headers.Add("Authorization", "Bearer " + accesstoken);
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
position = position + request.Content.Headers.ContentLength;
Array.Clear(buffer, 0, buffer.Length);
} while (numBytesToRead > 0);
stream.Close();
I have looked through the forum threads but haven't come across anything which deals with splitting of avro files.
I have a hunch that my "content" for the http request isn't right. what is it that I am missing?
If you need more details, I will be happy to provide.
I have found the problem now. The problem was because of MultipartFormDataContent. When an avro file is uploaded with that, it adds extra text like content Type etc, along with removal of many lines (I do not know why).
So, the solution was to upload the contents as "ByteArrayContent" itself and not add it to MultipartFormDataContent like I was doing earlier.
Here is the snippet, almost similar to the one in the question, except that I no longer use MultipartFormDataContent
string filenamefullyqualified = path + filename;
Stream stream = System.IO.File.Open(filenamefullyqualified, FileMode.Open, FileAccess.Read, FileShare.None);
//content.Add(CreateFileContent(fs, path, filename, "text/plain"));
long? position = 0;
byte[] buffer = new byte[(20 * 1024 * 1024) + 100];
long numBytesToRead = stream.Length;
int numBytesRead = 0;
//while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
//{
do
{
//var content = new MultipartFormDataContent();
int bytesRead = stream.Read(buffer, 0, buffer.Length);
byte[] actualbytes = new byte[bytesRead];
Array.Copy(buffer, actualbytes, bytesRead);
if (bytesRead == 0)
break;
//Append Data
url = String.Format("https://{0}.dfs.core.windows.net/raw/datawarehouse/{1}/{2}/{3}/{4}/{5}?action=append&position={6}", datalakeName, filename.Substring(0, filename.IndexOf("_")), year, month, day, filename, position.ToString());
numBytesRead += bytesRead;
numBytesToRead -= bytesRead;
ByteArrayContent byteContent = new ByteArrayContent(actualbytes);
//byteContent.Headers.ContentType= new MediaTypeHeaderValue("text/plain");
//content.Add(byteContent);
method = new HttpMethod("PATCH");
//request = new HttpRequestMessage(method, url)
//{
// Content = content
//};
request = new HttpRequestMessage(method, url)
{
Content = byteContent
};
request.Headers.Add("Authorization", "Bearer " + accesstoken);
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
position = position + request.Content.Headers.ContentLength;
Array.Clear(buffer, 0, buffer.Length);
} while (numBytesToRead > 0);
stream.Close();
But the streaming by record will not be able to handle the AVRO file as a whole in a transaction. We may end up in partial success, if some records fail, for example.
If we have a small tool that can split AVRO files based on a threshold number of records, it will be great.
The spark-based split by partition technique does allow to split data set to a pre-defined number of files; but, it does not allow splitting based on the number of records. I.e., I do not want an AVRO file with more than 500 records.
So we have to devise a batching logic based on the comfortable heap size the application can handle along with a two-phase commit, to handle transactions
Im connecting 2 devices over TCPClient and TCPListener and im sending just a string for now and its all working:
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] buffer = encoder.GetBytes("Hello Server!");
clientStream.Write(buffer, 0 , buffer.Length);
clientStream.Flush();
and then
bytesRead = clientStream.Read(message, 0, 4096);
ASCIIEncoding encoder = new ASCIIEncoding();
Console.WriteLine("Mensageee"+ encoder.GetString(message, 0, bytesRead));
But now i need to send a large file over it like 10mb or maybe more so should i use this?
string doc = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
byte[] file = File.ReadAllBytes(doc + filedir)
byte[] fileBuffer = new byte[file.Length];
TcpClient clientSocket = new TcpClient(ip, port);
NetworkStream networkStream = clientSocket.GetStream();
networkStream.Write(file.ToArray(), 0, fileBuffer.GetLength(0));
networkStream.Close();
And how should i receive all this file and then save it somewhere?
Any help is welcome thanks o/
The short answer is, you send a byte[] multiple times...
Essentially, you will need to fill a buffer ('byte[]') with a subset of the file:
int count = fileIO.Read(buffer, 0, buffer.Length);
And then send the buffer over the socket:
clientSocket.Send(buffer, 0, count);
Just do these two processes until you have sent the entire file... (Which will be when count <= 0) However, the server has to know how many bytes to read... so we should start out by sending a Int64 with the file's length.
What we have so far...
using (var fileIO = File.OpenRead(#"C:\temp\fake.bin"))
using(var clientSocket = new System.Net.Sockets.TcpClient(ip, port).GetStream())
{
// Send Length (Int64)
clientSocket.Write(BitConverter.GetBytes(fileIO.Length, 0, 8));
var buffer = new byte[1024 * 8];
int count;
while ((count = fileIO.Read(buffer, 0, buffer.Length)) > 0)
clientSocket.Write(buffer, 0, count);
}
Server Side
Int64 bytesReceived = 0;
int count;
var buffer = new byte[1024*8];
// Read length - Int64
clientStream.Read(buffer, 0, 8);
Int64 numberOfBytes = BitConverter.ToInt64(buffer, 0);
using(var fileIO = File.Create("#c:\some\path"))
while(bytesReceived < numberOfBytes && (count = clientStream.Read(buffer, 0, buffer.Length)) > 0)
{
fileIO.Write(buffer, 0, count);
bytesReceived += count;
}
I want to calculate downloading time in my ftp download manager.
I am using these code to download file through ftp.
try
{
string DirectoryCreate = localPath;
if (!Directory.Exists(DirectoryCreate))
{
Directory.CreateDirectory(DirectoryCreate);
}
FtpWebRequest requestFileDownload = (FtpWebRequest)WebRequest.Create("ftp://xxxxxx.com);
requestFileDownload.Method = WebRequestMethods.Ftp.DownloadFile;
FtpWebResponse responseFileDownload = (FtpWebResponse)requestFileDownload.GetResponse();
Stream responseStream = responseFileDownload.GetResponseStream();
FileStream writeStream = new FileStream(localPath + "\\" + fileName, FileMode.Create);
int Length = 2048;
Byte[] buffer = new Byte[Length];
int bytesRead = responseStream.Read(buffer, 0, Length);
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = responseStream.Read(buffer, 0, Length);
}
responseStream.Close();
writeStream.Close();
requestFileDownload = null;
responseFileDownload = null;
}
catch(WebException ex)
{
}
Can anyone please say me what i should change in my code for calculating download time.
There would be great appreciation if someone could help me.
DateTime t1 = DateTime.Now;
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = responseStream.Read(buffer, 0, Length);
}
DateTime t2 = DateTime.Now;
TimeSpan diff = t2 - t1;//you can return diff or display it using its properties ..
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
int bytesRead = responseStream.Read(buffer, 0, Length);
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = responseStream.Read(buffer, 0, Length);
}
responseStream.Close();
writeStream.Close();
requestFileDownload = null;
responseFileDownload = null;
}
catch(WebException ex)
{
}
stopwatch.Stop();
stopwatch.Elapsed // this will give you the elapsed time
With help of this question C# 4.0: Convert pdf to byte[] and vice versa i was able to convert byte[] to PDF. Byte array length is 25990 approx. When i try to open the PDF it says file is corrupted. What could be the reason?
I tried the BinaryWriter but it creates PDF of 0 KB.
It's a response from a Web Service
Sample Code
WebResponse resp = request.GetResponse();
var buffer = new byte[4096];
Stream responseStream = resp.GetResponseStream();
{
int count;
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, responseStream.Read(buffer, 0, buffer.Length));
} while (count != 0);
}
resp.Close();
byte[] memoryBuffer = memoryStream.ToArray();
System.IO.File.WriteAllBytes(#"E:\sample1.pdf", memoryBuffer);
int s = memoryBuffer.Length;
BinaryWriter binaryWriter = new BinaryWriter(File.Open(#"E:\sample2.pdf", FileMode.Create));
binaryWriter.Write(memoryBuffer);
You are reading twice from the stream but only writing one buffer. Change this:
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, responseStream.Read(buffer, 0, buffer.Length));
To this:
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, count);
It seems your missing some bytes there because you have one unnecessary read. Try this:
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, count);
} while (count != 0);