Develop Reference

Storing text file in Oracle DB as BLOB cuts off the end of the file

I'm generating a text file in a process which at the end loops through a list of strings that were fed to it, and through a MemoryStream and StreamWriter it converts that list to byte[]. The byte[] is then saved to an Oracle Database using a BLOB datatype. While it works for the majority of the data (typically thousands of lines. I've had anywhere between 5,000 and 40,000, and it's the same result regardless), I have a specific message that goes at the end, but it's always missing. Generally the last line that does end up in the file is cut off halfway.
The function that generates the byte[]:
public byte[] GenerateFileData()
{
var fileData = new byte[0];
using (var ms = new MemoryStream())
{
using (var sw = new StreamWriter(ms))
{
Messages.ForEach(x => sw.WriteLine(x)); // Messages is a list of strings in this class
fileData = ms.ToArray();
}
}
return fileData;
}
The function that saves the byte[] to the database:
public void SaveLogFile(int entityId, byte[] fileData)
{
using (var context = new SomeDBContext())
{
var entity= context.SomeEntity.FirstOrDefault(x => x.Id == runId);
if(entity != null)
{
entity.LOG_FILE = fileData;
context.SaveChanges();
}
}
}
And lastly, the function that turns the data into a file:
[HttpGet]
public FileResult GetLogFile(int id = 0)
{
var fileData = new byte[0];
using (var context = new SomeDbContext())
{
var entity = context.SomeEntity.FirstOrDefault(x => x.Id == id);
fileData = entity.LOG_FILE;
}
var fileName = "SomethingSomething" + id.ToString();
return File(fileData, "text/plain", fileName);
}

Try to get the MemoryStream content after the writer close asthis code:
public byte[] GenerateFileData()
{
var fileData = new byte[0];
using (var ms = new MemoryStream())
{
using (var sw = new StreamWriter(ms))
{
Messages.ForEach(x => sw.WriteLine(x)); // Messages is a list of strings in this class
}
ms.Flush();
fileData = ms.ToArray();
}
return fileData;
}

Reading from a ZipArchiveEntry cause exception and MemoryLeak if using a MemoryStream

I have the following code that generate two kinds of errors. First with the current code I get an exception 'NotSupportedException: This stream from ZipArchiveEntry does not support reading.'. How am I supposed to read the data ?
Furthermore if i use a MemoryStream (as the commented code ) then I can read the data and deserialize correctly but the memorystream i created still remains in memory even if the dispose method has been called on it , causing some memory leaks . Any idea what is wrong with this code ?
void Main()
{
List<Product> products;
using (var s = GetDb().Result)
{
products = Utf8Json.JsonSerializer.Deserialize<List<Product>>(s).ToList();
}
}
// Define other methods and classes here
public static Task<Stream> GetDb()
{
var filepath = Path.Combine("c:/users/tom/Downloads", "productdb.zip");
using (var archive = ZipFile.OpenRead(filepath))
{
var data = archive.Entries.Single(e => e.FullName == "productdb.json");
return Task.FromResult(data.Open());
//using (var reader = new StreamReader(data.Open()))
//{
// var ms = new MemoryStream();
// data.Open().CopyTo(ms);
// ms.Seek(0, SeekOrigin.Begin);
// return Task.FromResult((Stream)ms);
//}
}
}

With the commented code you open the stream into a reader, don't use the reader, then open the stream again and copy over to the memory stream without closing the second opened stream.
It is the second opened stream that remains in memory, not the MemoryStream.
Refactor
public static async Task<Stream> GetDb() {
var filepath = Path.Combine("c:/users/tom/Downloads", "productdb.zip");
using (var archive = ZipFile.OpenRead(filepath)) {
var entry = archive.Entries.Single(e => e.FullName == "productdb.json");
using (var stream = entry.Open()) {
var ms = new MemoryStream();
await stream.CopyToAsync(ms);
return ms;
}
}
}

Using a generic stream to create a zipped file with SharpCompress

Since System.IO.Compression seems to be out of reach for now if I want to use both dotnet core + net461, I've tried with SharpCompress.
The "read zip" part was easy, but I am having trouble finding out how to write to a zip stream.
The wiki of the project is a bit outdated. This is the only example that I've found that applies to writing to streams. I've tried to follow it and adapt it to my needs, but I am stuck at the exception it throws:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using SharpCompress.Common;
using SharpCompress.Compressors.Deflate;
using SharpCompress.Writers;
using System;
using System.IO;
namespace DbManager.DjdbCore.Tests
{
[TestClass]
public class ZipTests
{
public ZipTests()
{
Directory.SetCurrentDirectory(AppContext.BaseDirectory);
}
[TestMethod]
public void Test()
{
var zip = File.OpenWrite(#"..\..\..\..\..\test-resources\zip_file_test.zip");
var writerOptions = new WriterOptions(CompressionType.Deflate);
var zipWriter = WriterFactory.Open(zip, ArchiveType.Zip, writerOptions);
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
binaryWriter.Write("Test string inside binary file - text to fill it up: qoiwjqefñlawijfñlaskdjfioqwjefñalskvndñaskvnqo`wiefowainvñaslkfjnwpowiqjfeopwiqjnfjñlaskdjfñlasdfjiowiqjefñaslkdjfñalskjfpqwoiefjqw");
var deflateStream = new DeflateStream(memoryStream, SharpCompress.Compressors.CompressionMode.Compress);
deflateStream.Write(memoryStream.ToArray(), 0, Convert.ToInt32(memoryStream.Length));
// EXCEPTION: SharpCompress.Compressors.Deflate.ZlibException: 'Cannot Read after Writing.'
// Source code: if (_streamMode != StreamMode.Reader) { throw new ZlibException("Cannot Read after Writing."); }
zipWriter.Write("test_file_inside_zip.bin", deflateStream, DateTime.Now);
zip.Flush();
zipWriter.Dispose();
zip.Dispose();
}
}
}
In case it helps, this is what I used (and it worked, but only in dotnet core) using the library System.IO.Compression:
private void WriteAsZipBinary()
{
//Open the zip file if it exists, else create a new one
var zip = ZipPackage.Open(this.FileFullPath, FileMode.OpenOrCreate, FileAccess.ReadWrite);
var zipStream = ZipManager.GetZipWriteStream(zip, nameOfFileInsideZip);
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
// Here is where strings etc are written to the binary file:
WriteStuffInBinaryStream(ref binaryWriter);
//Read all of the bytes from the file to add to the zip file
byte[] bites = new byte[Convert.ToInt32(memoryStream.Length - 1) + 1];
memoryStream.Position = 0;
memoryStream.Read(bites, 0, Convert.ToInt32(memoryStream.Length));
binaryWriter.Dispose();
binaryWriter = null;
memoryStream.Dispose();
memoryStream = null;
zipStream.Position = 0;
zipStream.Write(bites, 0, bites.Length);
zip.Close();
}
public static Stream GetZipWriteStream(Package zip, string renamedFileName)
{
//Replace spaces with an underscore (_)
string uriFileName = renamedFileName.Replace(" ", "_");
//A Uri always starts with a forward slash "/"
string zipUri = string.Concat("/", Path.GetFileName(uriFileName));
Uri partUri = new Uri(zipUri, UriKind.Relative);
string contentType = "Zip"; // System.Net.Mime.MediaTypeNames.Application.Zip;
//The PackagePart contains the information:
// Where to extract the file when it's extracted (partUri)
// The type of content stream (MIME type): (contentType)
// The type of compression: (CompressionOption.Normal)
PackagePart pkgPart = zip.CreatePart(partUri, contentType, CompressionOption.Normal);
//Compress and write the bytes to the zip file
return pkgPart.GetStream();
}

I'll post here the answer on github from #adamhathcock (the owner of the project):
[TestMethod]
public void Test()
{
var writerOptions = new WriterOptions(CompressionType.Deflate);
using(var zip = File.OpenWrite(#"..\..\..\..\..\test-resources\zip_file_test.zip"))
using(var zipWriter = WriterFactory.Open(zip, ArchiveType.Zip, writerOptions))
{
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
binaryWriter.Write("Test string inside binary file - text to fill it up: qoiwjqefñlawijfñlaskdjfioqwjefñalskvndñaskvnqo`wiefowainvñaslkfjnwpowiqjfeopwiqjnfjñlaskdjfñlasdfjiowiqjefñaslkdjfñalskjfpqwoiefjqw");
memoryStream.Position = 0;
zipWriter.Write("test_file_inside_zip.bin", memoryStream, DateTime.Now);
}
}
2 things:
You forgot to reset the MemoryStream after writing to it so it can be read.
You don't need to manually use the DeflateStream. You've told the ZipWriter what compression to use. If it worked, you would have double compressed the bytes which would be garbage really.

iTextSharp System.OutOfMemoryException

I have an issue with trying to create a large PDF file. Basically I have a list of byte arrays, each containing a PDF in a form of a byte array. I wanted to merge the byte arrays into a single PDF. This works great for smaller files (under 2000 pages), but when I tried creating a 12,00 page file it bombed). Originally I was using MemoryStream but after some research, a common solution was to use a FileStream instead. So I tried a file stream approach, however get similar results. The List contains 3,800 records, each containing 4 pages. MemoryStream bombs after around 570. FileStream after about 680 records. The current file size after the code crashed was 60MB. What am I doing wrong? Here is the code I have, and the code crashes on "copy.AddPage(curPg);" directive, inside the "for(" loop.
private byte[] MergePDFs(List<byte[]> PDFs)
{
iTextSharp.text.Document doc = new iTextSharp.text.Document();
byte[] completePDF;
Guid uniqueId = Guid.NewGuid();
string tempFileName = Server.MapPath("~/" + uniqueId.ToString() + ".pdf");
//using (MemoryStream ms = new MemoryStream())
using(FileStream ms = new FileStream(tempFileName, FileMode.Create, FileAccess.Write, FileShare.Read))
{
iTextSharp.text.pdf.PdfCopy copy = new iTextSharp.text.pdf.PdfCopy(doc, ms);
doc.Open();
int i = 0;
foreach (byte[] PDF in PDFs)
{
i++;
// Create a reader
iTextSharp.text.pdf.PdfReader reader = new iTextSharp.text.pdf.PdfReader(PDF);
// Cycle through all the pages
for (int currentPageNumber = 1; currentPageNumber <= reader.NumberOfPages; ++currentPageNumber)
{
// Read a page
iTextSharp.text.pdf.PdfImportedPage curPg = copy.GetImportedPage(reader, currentPageNumber);
// Add the page over to the rest of them
copy.AddPage(curPg);
}
// Close the reader
reader.Close();
}
// Close the document
doc.Close();
// Close the copier
copy.Close();
// Convert the memorystream to a byte array
//completePDF = ms.ToArray();
}
//return completePDF;
return GetPDFsByteArray(tempFileName);
}

A couple of notes:
PdfCopy implements iDisposable, so you should try and see if a using helps.
PdfCopy.FreeReader() will help.
Anyway, not sure if you're using MVC or WebForms, but here's a simple working HTTP handler tested with a 15 page 125KB test file that runs on my workstation:
<%# WebHandler Language="C#" Class="MergeFiles" %>
using System;
using System.Collections.Generic;
using System.Web;
using System.IO;
using iTextSharp.text;
using iTextSharp.text.pdf;
public class MergeFiles : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
List<byte[]> pdfs = new List<byte[]>();
var pdf = File.ReadAllBytes(context.Server.MapPath("~/app_data/test.pdf"));
for (int i = 0; i < 4000; ++i) pdfs.Add(pdf);
var Response = context.Response;
Response.ContentType = "application/pdf";
Response.AddHeader(
"content-disposition",
"attachment; filename=MergeLotsOfPdfs.pdf"
);
Response.BinaryWrite(MergeLotsOfPdfs(pdfs));
}
byte[] MergeLotsOfPdfs(List<byte[]> pdfs)
{
using (var ms = new MemoryStream())
{
using (Document document = new Document())
{
using (PdfCopy copy = new PdfCopy(document, ms))
{
document.Open();
for (int i = 0; i < pdfs.Count; ++i)
{
using (PdfReader reader = new PdfReader(
new RandomAccessFileOrArray(pdfs[i]), null))
{
copy.AddDocument(reader);
copy.FreeReader(reader);
}
}
}
}
return ms.ToArray();
}
}
public bool IsReusable { get { return false; } }
}
Tried to make the output file similar to what you described in the question, but YMMV, depending on how large the individual PDFs you're dealing with are in size. Here's the test output from my run:

So after a lot of messing around, I realized that there just was no way around it. However, I did manage to find a work-around. Instead of returning byte array, I return a temp file path, which I then transmit and delete there after.
private string MergeLotsOfPDFs(List<byte[]> PDFs)
{
Document doc = new Document();
Guid uniqueId = Guid.NewGuid();
string tempFileName = Server.MapPath("~/__" + uniqueId.ToString() + ".pdf");
using (FileStream ms = new FileStream(tempFileName, FileMode.Create, FileAccess.Write, FileShare.Read))
{
PdfCopy copy = new PdfCopy(doc, ms);
doc.Open();
int i = 0;
foreach (byte[] PDF in PDFs)
{
i++;
// Create a reader
PdfReader reader = new PdfReader(new RandomAccessFileOrArray(PDF), null);
// Cycle through all the pages
for (int currentPageNumber = 1; currentPageNumber <= reader.NumberOfPages; ++currentPageNumber)
{
// Read a page
PdfImportedPage curPg = copy.GetImportedPage(reader, currentPageNumber);
// Add the page over to the rest of them
copy.AddPage(curPg);
// This is a lie, it still costs money, hue hue hue :)~
copy.FreeReader(reader);
}
reader.Close();
}
// Close the document
doc.Close();
// Close the document
copy.Close();
}
// Return temp file path
return tempFileName;
}
And here is how I send that data to the client.
// Send the merged PDF file to the user.
System.Web.HttpResponse response = System.Web.HttpContext.Current.Response;
response.ClearContent();
Response.ClearHeaders();
response.ContentType = "application/pdf";
response.AddHeader("Content-Disposition", "attachment; filename=1094C.pdf;");
response.WriteFile(tempFileName);
HttpContext.Current.Response.Flush(); // Sends all currently buffered output to the client.
DeleteFile(tempFileName); // Call right after flush but before close
HttpContext.Current.Response.SuppressContent = true; // Gets or sets a value indicating whether to send HTTP content to the client.
HttpContext.Current.ApplicationInstance.CompleteRequest(); // Causes ASP.NET to bypass all events and filtering in the HTTP pipeline chain of execution and directly execute the EndRequest event.
Lastly, here is a fancy DeleteFile method
private void DeleteFile(string fileName)
{
if (File.Exists(fileName))
{
try
{
File.Delete(fileName);
}
catch (Exception ex)
{
//Could not delete the file, wait and try again
try
{
System.GC.Collect();
System.GC.WaitForPendingFinalizers();
File.Delete(fileName);
}
catch
{
//Could not delete the file still
}
}
}
}

Method that return a string with an specifc extension

Im a long time trying to solve one problem. I have one method that Serialize a string, follows the code:
XmlRetorno()
var algumasDef = new XmlWriterSettings {
Indent = true,
OmitXmlDeclaration = true
};
var nameSpace = new XmlSerializerNamespaces();
nameSpace.Add(string.Empty, "urn:sngpc-schema");
var meuXml = new XmlSerializer(GetType(), "urn:sngpc-schema");
using (var minhaString = new StringWriterWithEncoding(Encoding.GetEncoding("iso-8859-1"))) {
using (var escreve = XmlWriter.Create(minhaString, algumasDef)) {
meuXml.Serialize(escreve, this, nameSpace);
}
return minhaString.ToString();
}
Then, my next step is to compact that string to a zip file, my method to zip.
CompactXml()
string ziparEssaString = msg.XmlRetorno();
byte[] byteArray = new byte[ziparEssaString.Length];
int indexBA = 0;
foreach (char item in ziparEssaString.ToArray()) {
byteArray[indexBA++] = (byte)item;
}
//prepare to compress
using (MemoryStream ms = new MemoryStream()) {
using (GZipStream sw = new GZipStream(ms, CompressionMode.Compress)) {
sw.Write(byteArray, 0, byteArray.Length);
}
//transform bytes[] zip to string
byteArray = ms.ToArray();
StringBuilder sB = new StringBuilder(byteArray.Length);
foreach (byte item in byteArray) {
sB.Append((char)item);
}
return sB.ToString();
}
I need to compress a string that is formatted .xml and when I unpack I need the extension to be .xml too, my webservice return an error. Please, i need one light.