Develop Reference

Read large files - 2GB+ for Google Drive API Upload

I'm currently working on a small backup tool written in C# that is supposed to upload files contained within a specified folder to Google Drive via its API. The program largely functions as it's supposed to, the only problem that it is unable to handle files larger than 2GB.
The problem is caused by the upload function itself which is attached down below, it uses a byte array to read the file to subsequently create a Memory Stream. As far as I'm aware (I'm still a beginner when it comes to c#), a byte array can only contain 2GB of information before returning an overflow exception. To combat this I've tried to utilize FileStream.Read (second bit of code attached below) instead of System.IO.File.ReadAllBytes, though this again lead to an overflow exception of the byte Array. I know that at this point I'd have to split the file up, however, due to the rather limited documentation of the GDrive API for C# - at least from what I've seen - and my limited knowledge of C# I've got little to no clue on how to tackle this problem.
I'm sorry for the long read, all help on this matter is highly appreciated.
Upload Function V1 (System.IO.File.ReadAllBytes):
private static Google.Apis.Drive.v3.Data.File UploadFile(Boolean useFolder, String mime, DriveService _service, string _uploadFile, string _parent, string _descrp = "")
{
if (System.IO.File.Exists(_uploadFile))
{
Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File
{
Name = System.IO.Path.GetFileName(_uploadFile),
Description = _descrp,
MimeType = mime
};
if (useFolder)
{
body.Parents = new List<string> { _parent };
}
byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
MemoryStream stream = new System.IO.MemoryStream(byteArray);
try
{
FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, mime);
request.SupportsTeamDrives = true;
request.Upload();
return request.ResponseBody;
}
catch (Exception e)
{
Console.WriteLine("Error Occured: " + e);
return null;
}
}
else
{
Console.WriteLine("The file does not exist. 404");
return null;
}
}
Upload Method V2 (FileStream):
private static Google.Apis.Drive.v3.Data.File UploadFile(Boolean useFolder, String mime, DriveService _service, string _uploadFile, string _parent, string _descrp = "")
{
if (System.IO.File.Exists(_uploadFile))
{
Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File
{
Name = System.IO.Path.GetFileName(_uploadFile),
Description = _descrp,
MimeType = mime
};
if (useFolder)
{
body.Parents = new List<string> { _parent };
}
//byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
using (FileStream fileStream = new FileStream(_uploadFile, FileMode.Open, FileAccess.Read))
{
Console.WriteLine("ByteArrayStart");
byte[] byteArray = new byte[fileStream.Length];
int bytesToRead = (int)fileStream.Length;
int bytesRead = 0;
while (bytesRead > 0)
{
int n = fileStream.Read(byteArray, bytesRead, bytesToRead);
if (n == 0)
{
break;
}
bytesRead += n;
Console.WriteLine("Bytes Read: " + bytesRead);
bytesToRead -= n;
Console.WriteLine("Bytes to Read: " + bytesToRead);
}
bytesToRead = byteArray.Length;
MemoryStream stream = new System.IO.MemoryStream(byteArray);
try
{
FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, mime);
request.SupportsTeamDrives = true;
request.Upload();
return request.ResponseBody;
}
catch (Exception e)
{
Console.WriteLine("Error Occured: " + e);
return null;
}
}
}
else
{
Console.WriteLine("The file does not exist. 404");
return null;
}
}

MemoryStream's constructors only work with byte arrays that are limited to Int32.MaxValue bytes. Why not just use your FileStream object directly?
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Name = "flag.jpg"
};
FilesResource.CreateMediaUpload request;
using (var stream = new System.IO.FileStream(#"C:\temp\flag.jpg", System.IO.FileMode.Open))
{
request = service.Files.Create(fileMetadata, stream, "image/jpeg");
request.Fields = "id";
request.Upload();
}
var file = request.ResponseBody;
Really a file that big you should be using resumable upload but im going to have to dig around for some sample code for that.

The server does not allow messages larger than 5242880 bytes

Elmah has recently reported this bug;
Microsoft.SharePoint.Client.ServerException: The request message is too big. The server does not allow messages larger than 5242880 bytes.
The code where it fell over was;
public SharepointFileInfo Save(byte[] file, string fileName)
{
using (var context = new ClientContext(this.SharepointServer))
{
context.Credentials = new NetworkCredential(this.UserName, this.Password, this.Domain);
var list = context.Web.Lists.GetByTitle(this.DocumentLibrary);
var fileCreationInformation = new FileCreationInformation
{
Content = file,
Overwrite = true,
Url = fileName
};
var uploadFile = list.RootFolder.Files.Add(fileCreationInformation);
var listItem = uploadFile.ListItemAllFields;
listItem.Update();
context.ExecuteQuery();
if (this.Metadata.Count > 0)
{
this.SaveMetadata(uploadFile, context);
}
return GetSharepointFileInfo(context, list, uploadFile);
}
}
I am using Sharepoint 2013.
How do I fix this?

It's a normal problme. You use the classic API (new FileCreationInformation [...] context.ExecuteQuery()) which sent a HTTP requet to the server. You file is up to 5 Mb. So, IIS receive a huge request, and reject it.
To Upload a file to SharePoint you need to use :
File.SaveBinaryDirect
(with this you don't need to change settings ;) )
using (FileStream fs = new FileStream(filePath, FileMode.Open))
{
Microsoft.SharePoint.Client.File.SaveBinaryDirect(ctx, string.Format("/{0}/{1}", libraryName, System.IO.Path.GetFileName(filePath)), fs, true);
}
Check this links to see how to upload a file to SharePoint using CSOM :
Upload large files sample app for SharePoint
good luck

There is several approche to do that (upload file with metaData). I propose 2 methods to you (one simple, second more complex)
In 2 Times (simple)
Upload the the file with the File.SaveBinaryDirect
Get the SPFile with CSOM by the file URL with SP.Web.getFileByServerRelativeUrl and File.listItemAllFields methodes.
Here an exemple : get listitem by file URL
With FileCreationInformation but more complex.
You need to use : File.StartUpload, File.ContinueUpload and File.FinishUpload
The code is from Microsoft the last part of the tuto, not mine
public Microsoft.SharePoint.Client.File UploadFileSlicePerSlice(ClientContext ctx, string libraryName, string fileName,int fileChunkSizeInMB = 3){
// Each sliced upload requires a unique ID.
Guid uploadId = Guid.NewGuid();
// Get the name of the file.
string uniqueFileName = Path.GetFileName(fileName);
// Ensure that target library exists, and create it if it is missing.
if (!LibraryExists(ctx, ctx.Web, libraryName))
{
CreateLibrary(ctx, ctx.Web, libraryName);
}
// Get the folder to upload into.
List docs = ctx.Web.Lists.GetByTitle(libraryName);
ctx.Load(docs, l => l.RootFolder);
// Get the information about the folder that will hold the file.
ctx.Load(docs.RootFolder, f => f.ServerRelativeUrl);
ctx.ExecuteQuery();
// File object.
Microsoft.SharePoint.Client.File uploadFile;
// Calculate block size in bytes.
int blockSize = fileChunkSizeInMB * 1024 * 1024;
// Get the information about the folder that will hold the file.
ctx.Load(docs.RootFolder, f => f.ServerRelativeUrl);
ctx.ExecuteQuery();
// Get the size of the file.
long fileSize = new FileInfo(fileName).Length;
if (fileSize <= blockSize)
{
// Use regular approach.
using (FileStream fs = new FileStream(fileName, FileMode.Open))
{
FileCreationInformation fileInfo = new FileCreationInformation();
fileInfo.ContentStream = fs;
fileInfo.Url = uniqueFileName;
fileInfo.Overwrite = true;
uploadFile = docs.RootFolder.Files.Add(fileInfo);
ctx.Load(uploadFile);
ctx.ExecuteQuery();
// Return the file object for the uploaded file.
return uploadFile;
}
}
else
{
// Use large file upload approach.
ClientResult<long> bytesUploaded = null;
FileStream fs = null;
try
{
fs = System.IO.File.Open(fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (BinaryReader br = new BinaryReader(fs))
{
byte[] buffer = new byte[blockSize];
Byte[] lastBuffer = null;
long fileoffset = 0;
long totalBytesRead = 0;
int bytesRead;
bool first = true;
bool last = false;
// Read data from file system in blocks.
while ((bytesRead = br.Read(buffer, 0, buffer.Length)) > 0)
{
totalBytesRead = totalBytesRead + bytesRead;
// You've reached the end of the file.
if (totalBytesRead == fileSize)
{
last = true;
// Copy to a new buffer that has the correct size.
lastBuffer = new byte[bytesRead];
Array.Copy(buffer, 0, lastBuffer, 0, bytesRead);
}
if (first)
{
using (MemoryStream contentStream = new MemoryStream())
{
// Add an empty file.
FileCreationInformation fileInfo = new FileCreationInformation();
fileInfo.ContentStream = contentStream;
fileInfo.Url = uniqueFileName;
fileInfo.Overwrite = true;
uploadFile = docs.RootFolder.Files.Add(fileInfo);
// Start upload by uploading the first slice.
using (MemoryStream s = new MemoryStream(buffer))
{
// Call the start upload method on the first slice.
bytesUploaded = uploadFile.StartUpload(uploadId, s);
ctx.ExecuteQuery();
// fileoffset is the pointer where the next slice will be added.
fileoffset = bytesUploaded.Value;
}
// You can only start the upload once.
first = false;
}
}
else
{
// Get a reference to your file.
uploadFile = ctx.Web.GetFileByServerRelativeUrl(docs.RootFolder.ServerRelativeUrl + System.IO.Path.AltDirectorySeparatorChar + uniqueFileName);
if (last)
{
// Is this the last slice of data?
using (MemoryStream s = new MemoryStream(lastBuffer))
{
// End sliced upload by calling FinishUpload.
uploadFile = uploadFile.FinishUpload(uploadId, fileoffset, s);
ctx.ExecuteQuery();
// Return the file object for the uploaded file.
return uploadFile;
}
}
else
{
using (MemoryStream s = new MemoryStream(buffer))
{
// Continue sliced upload.
bytesUploaded = uploadFile.ContinueUpload(uploadId, fileoffset, s);
ctx.ExecuteQuery();
// Update fileoffset for the next slice.
fileoffset = bytesUploaded.Value;
}
}
}
} // while ((bytesRead = br.Read(buffer, 0, buffer.Length)) > 0)
}
}
finally
{
if (fs != null)
{
fs.Dispose();
}
}
}
return null;}
hope this help you

How to merge MP3 files in WP8 (edit)

I have list of MP3 files that I would like to merge them into one file. I downloaded files locally into isolated storage, but I have no idea how to merge them. Google doesn't help either. I don't know if its possible in WP8. (2) If not possible what specific solution you could advice (I also have my files in web).
I write below code for merging, but this always writes the last file.
string[] files = new string[] { "001000.mp3", "001001.mp3", "001002.mp3", "001003.mp3", "001004.mp3", "001005.mp3", "001006.mp3", "001007.mp3" };
using (IsolatedStorageFile storage = IsolatedStorageFile.GetUserStoreForApplication())
{
string newFileName = "001.mp3";
foreach (string _fileName in files)
{
if (storage.FileExists(_fileName))
{
byte[] bytes = new byte[1];
using (var f = new IsolatedStorageFileStream(_fileName, FileMode.Open, storage))
{
bytes = new byte[f.Length];
int byteCount;
using (var isfs = new IsolatedStorageFileStream(newFileName, FileMode.OpenOrCreate, storage))
{
while ((byteCount = f.Read(bytes, 0, bytes.Length)) > 0)
{
isfs.Write(bytes, 0, byteCount);
isfs.Flush();
}
}
}
}
}
}
How can i add (merge) files to end of the newly created file?

I added isfs.Seek(0, SeekOrigin.End); before writing to file. This is how we can simply merge same bitrate mp3 files.
the code looks like below:
using (var isfs = new IsolatedStorageFileStream(newFileName, FileMode.OpenOrCreate, storage))
{
isfs.Seek(0, SeekOrigin.End); //THIS LINE DID SOLVE MY PROBLEM
while ((byteCount = f.Read(bytes, 0, bytes.Length)) > 0)
{
isfs.Write(bytes, 0, byteCount);
isfs.Flush();
}
}
Please improve my code if it seems to be longer.

download zip files by use of reader in c#

I have been working on this application that enables user to log in into another website, and then download specified file from that server. So far I have succeeded in logging on the website and download the file. But everything ruins when it comes to zip files.
Is there any chunk of code that could be helpful in reading the .zip files byte by byte or by using stream reader?
I m using downloadfile() but its not returning the correct zip file.
I need a method by which I can read zip files. Can I do it by using ByteReader()
The code used to download zip file is
string filename = "13572_BranchInformationReport_2012-05-22.zip";
string filepath = "C:\\Documents and Settings\\user\\Desktop\\" + filename.ToString();
WebClient client = new WebClient();
string user = "abcd", pass = "password";
client.Credentials = new NetworkCredential(user, pass);
client.Encoding = System.Text.Encoding.UTF8;
try
{
client.DownloadFile("https://web.site/archive/13572_BranchInformationReport_2012-05-22.zip", filepath);
Response.Write("Success");
}
catch (Exception ue)
{
Response.Write(ue.Message);
}
Thanks in advance.

is there any chunk of code that could be helpful in reading the zip files bytes by bytes aur by using stream reader.
Absolutely not. StreamReader - and indeed any TextReader is for reading text content, not binary content. A zip file is not text - it's composed of bytes, not characters.
If you're reading binary content such as zip files, you should be using a Stream rather than a TextReader of any kind.
Note that WebClient.DownloadFile and WebClient.DownloadData can generally make things easier for downloading binary content.

Another simple way to downlaod zip file
<asp:HyperLink ID="HyperLink1" runat="server" NavigateUrl="~/DOWNLOAD/Filename.zip">Click To Download</asp:HyperLink>
Another solution
private void DownloadFile()
{
string getPath = "DOWNLOAD/FileName.zip";
System.IO.Stream iStream = null;
byte[] buffer = new Byte[1024];
// Length of the file:
int length;
// Total bytes to read:
long dataToRead;
// Identify the file to download including its path.
string filepath = Server.MapPath(getPath);
// Identify the file name.
string filename = System.IO.Path.GetFileName(filepath);
try
{
// Open the file.
iStream = new System.IO.FileStream(filepath, System.IO.FileMode.Open,
System.IO.FileAccess.Read, System.IO.FileShare.Read);
// Total bytes to read:
dataToRead = iStream.Length;
// Page.Response.ContentType = "application/vnd.android.package-archive";
// Page.Response.ContentType = "application/octet-stream";
Page.Response.AddHeader("Content-Disposition", "attachment; filename=" + filename);
// Read the bytes.
while (dataToRead > 0)
{
// Verify that the client is connected.
if (Response.IsClientConnected)
{
// Read the data in buffer.
length = iStream.Read(buffer, 0, 1024);
// Write the data to the current output stream.
Page.Response.OutputStream.Write(buffer, 0, length);
// Flush the data to the HTML output.
Page.Response.Flush();
// buffer = new Byte[1024];
dataToRead = dataToRead - length;
}
else
{
//prevent infinite loop if user disconnects
dataToRead = -1;
}
}
}
catch (Exception ex)
{
// Trap the error, if any.
Page.Response.Write(ex.Message);
}
finally
{
if (iStream != null)
{
//Close the file.
iStream.Close();
Page.Response.Close();
}
}
}

Your answer
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);
objResponse = objRequest.GetResponse();
byte[] buffer = new byte[32768];
using (Stream input = objResponse.GetResponseStream())
{
using (FileStream output = new FileStream ("test.doc",
FileMode.CreateNew))
{
int bytesRead;
while ( (bytesRead=input.Read (buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
This is how i achieved it. Thanks everyone for ur help

Unzipping a Stream in C#

I'm working in C#, and I'm downloading for the internet a zip file with one XML file in it. and I wish to load this XML file. This is what I have so far:
byte[] data;
WebClient webClient = new WebClient();
try {
data = webClient.DownloadData(downloadUrl);
}
catch (Exception ex) {
Console.WriteLine("Error in DownloadData (Ex:{0})", ex.Message);
throw;
}
if (data == null) {
Console.WriteLine("Bulk data is null");
throw new Exception("Bulk data is null");
}
//Create the stream
MemoryStream stream = new MemoryStream(data);
XmlDocument document = new XmlDocument();
//Gzip
GZipStream gzipStream = new GZipStream(stream, CompressionMode.Decompress);
//Load report straight from the gzip stream
try {
document.Load(gzipStream);
}
catch (Exception ex) {
Console.WriteLine("Error in Load (Ex:{0})", ex.Message);
throw;
}
in document.Load I'm always getting the following exception:
The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
What I'm doing wrong?

Apparently SharpZipLib is now unmaintained and you probably want to avoid it:
https://stackoverflow.com/a/593030
In .NET 4.5 there is now built in support for zip files, so for your example it would be:
var data = new WebClient().DownloadData(downloadUrl);
//Create the stream
var stream = new MemoryStream(data);
var document = new XmlDocument();
//zip
var zipArchive = new ZipArchive(stream);
//Load report straight from the zip stream
document.Load(zipArchive.Entries[0].Open());

If you have a byte array that contains a zip archive with a single file, you can use the ZipArchive class to get an unzipped byte array with the file's data.
ZipArchive is contained in .NET 4.5, in the assembly System.IO.Compression.FileSystem (you need to reference it explicitly).
The following function, adapted from this answer, works for me:
public static byte[] UnzipSingleEntry(byte[] zipped)
{
using (var memoryStream = new MemoryStream(zipped))
{
using (var archive = new ZipArchive(memoryStream))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
using (var entryStream = entry.Open())
{
using (var reader = new BinaryReader(entryStream))
{
return reader.ReadBytes((int)entry.Length);
}
}
}
}
}
return null; // To quiet my compiler
}

I am using SharpZipLib and it's working great !
Below is a function that encapsulate the library
public static void Compress(FileInfo sourceFile, string destinationFileName,string destinationTempFileName)
{
Crc32 crc = new Crc32();
string zipFile = Path.Combine(sourceFile.Directory.FullName, destinationTempFileName);
zipFile = Path.ChangeExtension(zipFile, ZIP_EXTENSION);
using (FileStream fs = File.Create(zipFile))
{
using (ZipOutputStream zOut = new ZipOutputStream(fs))
{
zOut.SetLevel(9);
ZipEntry entry = new ZipEntry(ZipEntry.CleanName(destinationFileName));
entry.DateTime = DateTime.Now;
entry.ZipFileIndex = 1;
entry.Size = sourceFile.Length;
using (FileStream sourceStream = sourceFile.OpenRead())
{
crc.Reset();
long len = sourceFile.Length;
byte[] buffer = new byte[bufferSize];
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
crc.Update(buffer, 0, readSoFar);
len -= readSoFar;
}
entry.Crc = crc.Value;
zOut.PutNextEntry(entry);
len = sourceStream.Length;
sourceStream.Seek(0, SeekOrigin.Begin);
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
zOut.Write(buffer, 0, readSoFar);
len -= readSoFar;
}
}
zOut.Finish();
zOut.Close();
}
fs.Close();
}
}

As the others have mentioned GZip and Zip are not the same so you might need to use a zip library. I use a library called: DotNetZip - available from the below site:
http://dotnetzip.codeplex.com/

From GZipStream Class description:
Compressed GZipStream objects written to a file with an extension of .gz can be decompressed using many common compression tools; however, this class does not inherently provide functionality for adding files to or extracting files from .zip archives
So unless you control server-side files, I'd suggest looking for specific zip-targeted library (SharpZipLib for example).