Develop Reference

Read large files - 2GB+ for Google Drive API Upload

I'm currently working on a small backup tool written in C# that is supposed to upload files contained within a specified folder to Google Drive via its API. The program largely functions as it's supposed to, the only problem that it is unable to handle files larger than 2GB.
The problem is caused by the upload function itself which is attached down below, it uses a byte array to read the file to subsequently create a Memory Stream. As far as I'm aware (I'm still a beginner when it comes to c#), a byte array can only contain 2GB of information before returning an overflow exception. To combat this I've tried to utilize FileStream.Read (second bit of code attached below) instead of System.IO.File.ReadAllBytes, though this again lead to an overflow exception of the byte Array. I know that at this point I'd have to split the file up, however, due to the rather limited documentation of the GDrive API for C# - at least from what I've seen - and my limited knowledge of C# I've got little to no clue on how to tackle this problem.
I'm sorry for the long read, all help on this matter is highly appreciated.
Upload Function V1 (System.IO.File.ReadAllBytes):
private static Google.Apis.Drive.v3.Data.File UploadFile(Boolean useFolder, String mime, DriveService _service, string _uploadFile, string _parent, string _descrp = "")
{
if (System.IO.File.Exists(_uploadFile))
{
Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File
{
Name = System.IO.Path.GetFileName(_uploadFile),
Description = _descrp,
MimeType = mime
};
if (useFolder)
{
body.Parents = new List<string> { _parent };
}
byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
MemoryStream stream = new System.IO.MemoryStream(byteArray);
try
{
FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, mime);
request.SupportsTeamDrives = true;
request.Upload();
return request.ResponseBody;
}
catch (Exception e)
{
Console.WriteLine("Error Occured: " + e);
return null;
}
}
else
{
Console.WriteLine("The file does not exist. 404");
return null;
}
}
Upload Method V2 (FileStream):
private static Google.Apis.Drive.v3.Data.File UploadFile(Boolean useFolder, String mime, DriveService _service, string _uploadFile, string _parent, string _descrp = "")
{
if (System.IO.File.Exists(_uploadFile))
{
Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File
{
Name = System.IO.Path.GetFileName(_uploadFile),
Description = _descrp,
MimeType = mime
};
if (useFolder)
{
body.Parents = new List<string> { _parent };
}
//byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
using (FileStream fileStream = new FileStream(_uploadFile, FileMode.Open, FileAccess.Read))
{
Console.WriteLine("ByteArrayStart");
byte[] byteArray = new byte[fileStream.Length];
int bytesToRead = (int)fileStream.Length;
int bytesRead = 0;
while (bytesRead > 0)
{
int n = fileStream.Read(byteArray, bytesRead, bytesToRead);
if (n == 0)
{
break;
}
bytesRead += n;
Console.WriteLine("Bytes Read: " + bytesRead);
bytesToRead -= n;
Console.WriteLine("Bytes to Read: " + bytesToRead);
}
bytesToRead = byteArray.Length;
MemoryStream stream = new System.IO.MemoryStream(byteArray);
try
{
FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, mime);
request.SupportsTeamDrives = true;
request.Upload();
return request.ResponseBody;
}
catch (Exception e)
{
Console.WriteLine("Error Occured: " + e);
return null;
}
}
}
else
{
Console.WriteLine("The file does not exist. 404");
return null;
}
}

MemoryStream's constructors only work with byte arrays that are limited to Int32.MaxValue bytes. Why not just use your FileStream object directly?
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Name = "flag.jpg"
};
FilesResource.CreateMediaUpload request;
using (var stream = new System.IO.FileStream(#"C:\temp\flag.jpg", System.IO.FileMode.Open))
{
request = service.Files.Create(fileMetadata, stream, "image/jpeg");
request.Fields = "id";
request.Upload();
}
var file = request.ResponseBody;
Really a file that big you should be using resumable upload but im going to have to dig around for some sample code for that.

Request big file stream failed with wcf rest

I have a REST GET API that is written using WCF library to return Stream of a specific requested file that is located on API server that hosts that web service application. The service works well if the size of the requested file is small; that is less than 100 MB. But if file size is greater than > 100 MB, then the service returns 0 bytes without any logged information I can get the library method (saying, the "catch" block).
The library method (the class library project) returns Stream of needed file is
public Stream GetFile(string fileId, string seekStartPosition=null)
{
_lastActionResult = string.Empty;
Stream fileStream = null;
try
{
Guid fileGuid;
if (Guid.TryParse(fileId, out fileGuid) == false)
{
_lastActionResult = string.Format(ErrorMessage.FileIdInvalidT, fileId);
}
else
{
ContentPackageItemService contentItemService = new ContentPackageItemService();
string filePath = DALCacheHelper.GetFilePath(fileId);
if (File.Exists(filePath))
{
fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read);
long seekStart = 0;
// if seek position is specified, move the stream pointer to that location
if (string.IsNullOrEmpty(seekStartPosition) == false && long.TryParse(seekStartPosition, out seekStart))
{
// make sure seek position is smaller than file size
FileInfo fi = new FileInfo(filePath);
if (seekStart >= 0 && seekStart < fi.Length)
{
fileStream.Seek(seekStart, SeekOrigin.Begin);
}
else
{
_lastActionResult = string.Format(ErrorMessage.FileSeekInvalidT, seekStart, fi.Length);
}
}
}
else
{
_lastActionResult = string.Format(ErrorMessage.FileNotFoundT, fileId);
Logger.Write(_lastActionResult,
"General", 1, Constants.LogId.RESTSync, System.Diagnostics.TraceEventType.Error, System.Reflection.MethodBase.GetCurrentMethod().Name);
}
}
}
catch(Exception ex)
{
Logger.Write(ex,"General", 1, Constants.LogId.RESTSync, System.Diagnostics.TraceEventType.Error, System.Reflection.MethodBase.GetCurrentMethod().Name);
}
return fileStream;
}
API method on the client side project (where .svc file is):
[WebGet(UriTemplate = "files/{fileid}")]
public Stream GetFile(string fileid)
{
ContentHandler handler = new ContentHandler();
Stream fileStream = null;
try
{
fileStream = handler.GetFile(fileid);
}
catch (Exception ex)
{
Logger.Write(string.Format("{0} {1}", ex.Message, ex.StackTrace), "General", 1, Constants.LogId.RESTSync, System.Diagnostics.TraceEventType.Error, System.Reflection.MethodBase.GetCurrentMethod().Name);
throw new WebFaultException<ErrorResponse>(new ErrorResponse(HttpStatusCode.InternalServerError, ex.Message), HttpStatusCode.InternalServerError);
}
if (fileStream == null)
{
throw new WebFaultException<ErrorResponse>(new ErrorResponse(handler.LastActionResult), HttpStatusCode.InternalServerError);
}
return fileStream;
}

As you are using REST, I presume you are using the WebHttpBinding. You need to set the MaxReceivedMessageSize on the client binding to be sufficient for the maximum expected response size. The default is 64K. Here's the msdn documentation for the property if you are creating your binding in code. If you are creating your binding in your app.config, then this is the documentation you need.

c# ftpwebrequest performance

I have an application that download files from a Unix FTP server. It works fine, just have this performance problem: Files which size is <= 1K takes in average between 2084 and 2400 milliseconds to download, while applications like Filezilla download the same files in less than 1 second (per each file).
Maybe this time its OK for some average users, but is not acceptable for my application, since I need to download THOUSANDS of files.
I optimize the code as much as I could:
- The cache and buffer to read the content are created 1 time in the constructor of the class.
- I create 1 time the network credentials, and re-use on every file download. I know this is working, since for the first file it takes like 7s to download, and all subsequent downloads are on the range of 2s.
- I change the size of the buffer from 2K until 32K. I dont know if this will help or not, since the files Im downloading are less than 1K, so in theory the buffer will be fill with all the information in 1 round from network.
Maybe is not related to the network, but to the way Im writing and/or windows handles the write of the file??
Can someone please give me some tips on how to reduce the time to something similar to filezilla??
I need to reduce the time, otherwise my ftp will be running for 3 days 24 hours a day to finish its task :(
Many thanks in advance.
The code here: Its not complete, it just show the download part.
//Create this on the constructor of my class
downloadCache = new MemoryStream(2097152);
downloadBuffer = new byte[32768];
public bool downloadFile(string pRemote, string pLocal, out long donwloadTime)
{
FtpWebResponse response = null;
Stream responseStream = null;
try
{
Stopwatch fileDownloadTime = new Stopwatch();
donwloadTime = 0;
fileDownloadTime.Start();
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(pRemote);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.UseBinary = false;
request.AuthenticationLevel = AuthenticationLevel.None;
request.EnableSsl = false;
request.Proxy = null;
//I created the credentials 1 time and re-use for every file I need to download
request.Credentials = this.manager.ftpCredentials;
response = (FtpWebResponse)request.GetResponse();
responseStream = response.GetResponseStream();
downloadCache.Seek(0, SeekOrigin.Begin);
int bytesSize = 0;
int cachedSize = 0;
//create always empty file. Need this because WriteCacheToFile just append the file
using (FileStream fileStream = new FileStream(pLocal, FileMode.Create)) { };
// Download the file until the download is completed.
while (true)
{
bytesSize = responseStream.Read(downloadBuffer, 0, downloadBuffer.Length);
if (bytesSize == 0 || 2097152 < cachedSize + bytesSize)
{
WriteCacheToFile(pLocal, cachedSize);
if (bytesSize == 0)
{
break;
}
downloadCache.Seek(0, SeekOrigin.Begin);
cachedSize = 0;
}
downloadCache.Write(downloadBuffer, 0, bytesSize);
cachedSize += bytesSize;
}
fileDownloadTime.Stop();
donwloadTime = fileDownloadTime.ElapsedMilliseconds;
//file downloaded OK
return true;
}
catch (Exception ex)
{
return false;
}
finally
{
if (response != null)
{
response.Close();
}
if (responseStream != null)
{
responseStream.Close();
}
}
}
private void WriteCacheToFile(string downloadPath, int cachedSize)
{
using (FileStream fileStream = new FileStream(downloadPath, FileMode.Append))
{
byte[] cacheContent = new byte[cachedSize];
downloadCache.Seek(0, SeekOrigin.Begin);
downloadCache.Read(cacheContent, 0, cachedSize);
fileStream.Write(cacheContent, 0, cachedSize);
}
}

Sounds to me your problem is related to Nagels algorithm used in the TCP client.
You can try turning the Nagel's algorithm off and also set SendChunked to false.

Can't download complete image file from skydrive using REST API

I'm working on a quick wrapper for the skydrive API in C#, but running into issues with downloading a file. For the first part of the file, everything comes through fine, but then there start to be differences in the file and shortly thereafter everything becomes null. I'm fairly sure that it's just me not reading the stream correctly.
This is the code I'm using to download the file:
public const string ApiVersion = "v5.0";
public const string BaseUrl = "https://apis.live.net/" + ApiVersion + "/";
public SkyDriveFile DownloadFile(SkyDriveFile file)
{
string uri = BaseUrl + file.ID + "/content";
byte[] contents = GetResponse(uri);
file.Contents = contents;
return file;
}
public byte[] GetResponse(string url)
{
checkToken();
Uri requestUri = new Uri(url + "?access_token=" + HttpUtility.UrlEncode(token.AccessToken));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
request.Method = WebRequestMethods.Http.Get;
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] contents = new byte[response.ContentLength];
responseStream.Read(contents, 0, (int)response.ContentLength);
return contents;
}
This is the image file I'm trying to download
And this is the image I am getting
These two images lead me to believe that I'm not waiting for the response to finish coming through, because the content-length is the same as the size of the image I'm expecting, but I'm not sure how to make my code wait for the entire response to come through or even really if that's the approach I need to take.
Here's my test code in case it's helpful
[TestMethod]
public void CanUploadAndDownloadFile()
{
var api = GetApi();
SkyDriveFolder folder = api.CreateFolder(null, "TestFolder", "Test Folder");
SkyDriveFile file = api.UploadFile(folder, TestImageFile, "TestImage.png");
file = api.DownloadFile(file);
api.DeleteFolder(folder);
byte[] contents = new byte[new FileInfo(TestImageFile).Length];
using (FileStream fstream = new FileStream(TestImageFile, FileMode.Open))
{
fstream.Read(contents, 0, contents.Length);
}
using (FileStream fstream = new FileStream(TestImageFile + "2", FileMode.CreateNew))
{
fstream.Write(file.Contents, 0, file.Contents.Length);
}
Assert.AreEqual(contents.Length, file.Contents.Length);
bool sameData = true;
for (int i = 0; i < contents.Length && sameData; i++)
{
sameData = contents[i] == file.Contents[i];
}
Assert.IsTrue(sameData);
}
It fails at Assert.IsTrue(sameData);

This is because you don't check the return value of responseStream.Read(contents, 0, (int)response.ContentLength);. Read doesn't ensure that it will read response.ContentLength bytes. Instead it returns the number of bytes read. You can use a loop or stream.CopyTo there.
Something like this:
WebResponse response = request.GetResponse();
MemoryStream m = new MemoryStream();
response.GetResponseStream().CopyTo(m);
byte[] contents = m.ToArray();

As LB already said, you need to continue to call Read() until you have read the entire stream.
Although Stream.CopyTo will copy the entire stream it does not ensure that read the number of bytes expected. The following method will solve this and raise an IOException if it does not read the length specified...
public static void Copy(Stream input, Stream output, long length)
{
byte[] bytes = new byte[65536];
long bytesRead = 0;
int len = 0;
while (0 != (len = input.Read(bytes, 0, Math.Min(bytes.Length, (int)Math.Min(int.MaxValue, length - bytesRead)))))
{
output.Write(bytes, 0, len);
bytesRead = bytesRead + len;
}
output.Flush();
if (bytesRead != length)
throw new IOException();
}

Partially download and serialize big file in C#?

As part of an upcoming project at my university, I need to write a client that downloads a media file from a server and writes it to the local disk. Since these files can be very large, I need to implement partial download and serialization in order to avoid excessive memory use.
What I came up with:
namespace PartialDownloadTester
{
using System;
using System.Diagnostics.Contracts;
using System.IO;
using System.Net;
using System.Text;
public class DownloadClient
{
public static void Main(string[] args)
{
var dlc = new DownloadClient(args[0], args[1], args[2]);
dlc.DownloadAndSaveToDisk();
Console.ReadLine();
}
private WebRequest request;
// directory of file
private string dir;
// full file identifier
private string filePath;
public DownloadClient(string uri, string fileName, string fileType)
{
this.request = WebRequest.Create(uri);
this.request.Method = "GET";
var sb = new StringBuilder();
sb.Append("C:\\testdata\\DownloadedData\\");
this.dir = sb.ToString();
sb.Append(fileName + "." + fileType);
this.filePath = sb.ToString();
}
public void DownloadAndSaveToDisk()
{
// make sure directory exists
this.CreateDir();
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
var rStream = response.GetResponseStream();
int bytesRead = -1;
do
{
var buf = new byte[2048];
bytesRead = rStream.Read(buf, 0, buf.Length);
rStream.Flush();
this.SerializeFileChunk(buf);
}
while (bytesRead != 0);
}
private void CreateDir()
{
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
}
private void SerializeFileChunk(byte[] bytes)
{
Contract.Requires(!Object.ReferenceEquals(bytes, null));
FileStream fs = File.Open(filePath, FileMode.Append);
fs.Write(bytes, 0, bytes.Length);
fs.Flush();
fs.Close();
}
}
}
For testing purposes, I've used the following parameters:
"http://itu.dk/people/janv/mufc_abc.jpg" "mufc_abc" "jpg"
However, the picture is incomplete (only the first ~10% look right) even though the content length prints 63780 which is the actual size of the image.
So my questions are:
Is this the right way to go for partial download and serialization or is there a better/easier approach?
Is the full content of the response stream stored in client memory? If this is the case, do I need to use HttpWebRequest.AddRange to partially download data from the server in order to conserve my client's memory?
How come the serialization fails and I get a broken image?
Do I introduce a lot of overhead when I use the FileMode.Append? (msdn states that this option "seeks to the end of the file")
Thanks in advance

You could definitely simplify your code using a WebClient:
class Program
{
static void Main()
{
DownloadClient("http://itu.dk/people/janv/mufc_abc.jpg", "mufc_abc.jpg");
}
public static void DownloadClient(string uri, string fileName)
{
using (var client = new WebClient())
{
using (var stream = client.OpenRead(uri))
{
// work with chunks of 2KB => adjust if necessary
const int chunkSize = 2048;
var buffer = new byte[chunkSize];
using (var output = File.OpenWrite(fileName))
{
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
}
}
}
Notice how I am writing only the number of bytes I have actually read from the socket to the output file and not the entire 2KB buffer.

I don't know if this is the source of the problem, however I would change the loop like this
const int ChunkSize = 2048;
var buf = new byte[ChunkSize];
var rStream = response.GetResponseStream();
do {
int bytesRead = rStream.Read(buf, 0, ChunkSize);
if (bytesRead > 0) {
this.SerializeFileChunk(buf, bytesRead);
}
} while (bytesRead == ChunkSize);
The serialize method would get an additional argument
private void SerializeFileChunk(byte[] bytes, int numBytes)
and then write the right number of bytes
fs.Write(bytes, 0, numBytes);
UPDATE:
I do not see the need for closing and reopening the file each time. I also would use the using statement, which closes the resources, even if an exception should occur. The using statement calls the Dispose() method of the resource at the end, which in turn calls Close() in the case of file streams. using can be applied to all types implementing IDisposable.
var buf = new byte[2048];
using (var rStream = response.GetResponseStream()) {
using (FileStream fs = File.Open(filePath, FileMode.Append)) {
do {
bytesRead = rStream.Read(buf, 0, buf.Length);
fs.Write(bytes, 0, bytesRead);
} while (...);
}
}
The using statement does something like this
{
var rStream = response.GetResponseStream();
try
{
// do some work with rStream here.
} finally {
if (rStream != null) {
rStream.Dispose();
}
}
}

Here is the solution from Microsoft: http://support.microsoft.com/kb/812406
Updated 2021-03-16: seems the original article is not available now. Here is the archived one: https://mskb.pkisolutions.com/kb/812406