I have some code which processes one or more DataTables, writing the row and column data as CSV files to a stream. Each DataTable's contents are saved to a separate CSV file and finally saved to zip file (using DotNetZip. This code works fine when there is only one DataTable that needs to be proceesed, but when there are multiple DataTables, row and column data is being saved to only one CSV (the other CSVs are empty) and the data is chopped off at random places.
MemoryStream stream = new MemoryStream();
MemoryStream outputStream = new MemoryStream();
StreamWriter streamWriter = new StreamWriter(stream);
StreamWriter outStreamWriter = new StreamWriter(stream);
CsvConfiguration config = new CsvConfiguration();
config.QuoteAllFields = true;
streamWriter.WriteLine("sep=" + config.Delimiter);
var zip = new ZipFile();
var csv = new CsvWriter(streamWriter, config);
foreach (DataTable dt in dataTables)
{
foreach (DataColumn dc in dt.Columns)
{
csv.WriteField(dc.ColumnName.ToString());
}
csv.NextRecord();
foreach (DataRow dr in dt.Rows)
{
foreach (DataColumn dc in dt.Columns)
{
csv.WriteField(dr[dc].ToString());
}
csv.NextRecord();
}
zip.AddEntry(report.Title.ToString() + dt.GetHashCode() + ".csv", stream);
stream.Position = 0;
}
zip.Save(outputStream);
streamWriter.Flush();
outStreamWriter.Flush();
outputStream.Position = 0;
return outputStream;
I suspect that my usage of zip.AddEntry() may not be the correct way to save files to a stream. Any help appreciated as always. Also note, I know I don't have any using statements in my code: I was too lazy to add this in for this example.
There are two possible problem places I see:
1) outputStream.Position = 0;
and
2) var csv = new CsvWriter(streamWriter, config);
First is not the correct way to reset the stream. Second may have some problems with rewinded streams;
1)To rectify the first one either:
ms.Seek(0, IO.SeekOrigin.Begin)
ms.SetLength(0)
or just create new MemoryStream for each table.
2) To rectify the second one just create new CsvWriter for each table.
foreach (DataTable dt in dataTables)
{
var csv = new CsvWriter(streamWriter, config);
...
}
I'd recommended you to handle both problems, because there aren't any enormous advantages in reusing old objects(or have you profiled your code?), but it may lead to all sorts of inconsistent behaviours in case of misdisposing and misresetting.
Related
I want to export my data as csv file. For that I'm using CsvHelper library. But I don't want to have all data in one csv file. The limitation should be 1000 per file.
What I tried for that limitation:
var fileStream = new FileStream("/static/export.csv", FileMode.Create, FileAccess.Write)
var memoryStream = new MemoryStream();
var stream = new StreamWriter();
var writer = new CsvWriter(stream, config);
for(i = 0; i < indexCount; i += 1000)
{
var items = result.Skip(i).Take(1000);
.
.// logic for writing records with CsvHelper
.
writer.Flush();
memoryStream.Position = 0;
byte[] data = memoryStream.ToArray();
fileStream.Write(data, 0, data.Length);
}
For example if I have 100000 rows in database, I want to have 100 csv files. How can I download chunk files from stream?
var i = 1;
foreach (var chunk in result.Chunk(1000))
{
using var fileStream = new FileStream($"/static/export{i++}.csv", FileMode.Create, FileAccess.Write);
using var streamWriter = new StreamWriter(fileStream);
var writer = new CsvWriter(streamWriter , config);
// logic for writing records with CsvHelper
}
This question already has an answer here:
Writing Large File To Disk Out Of Memory Exception
(1 answer)
Closed 2 years ago.
We have an endpoint that loads records from the database and creates a CSV from the records and then returns the file stream. But when the records are greater than 200K, we get OutOfMemoryException.
public async Task<IActionResult> Export()
{
var records = // get all records from the database
var memoryStream = new MemoryStream();
var streamWriter = new StreamWriter(memoryStream);
var csvWriter = new CsvWriter(streamWriter, CultureInfo.InvariantCulture);
await csvWriter.WriteRecordsAsync(records);
csvWriter.Flush();
streamWriter.Flush();
memoryStream.Flush();
string filename = $"Records_{DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ss")}.csv";
memoryStream.Seek(0, SeekOrigin.Begin);
return File(memoryStream, "text/csv", filename);
}
Is there a better way of doing this to prevent OutOfMemoryException.
I dont know why you are getting this error. but from my below code, I am able to read large size of data, I tested with 3Gb approx data. What is your data size?
Here is my code using CSV helper.
private IEnumerable<Dictionary<string, EntityProperty>> ReadCSV(Stream source, IEnumerable<TableField> cols)
{
using (TextReader reader = new StreamReader(source, Encoding.UTF8))
{
var cache = new TypeConverterCache();
cache.AddConverter<float>(new CSVSingleConverter());
cache.AddConverter<double>(new CSVDoubleConverter());
var csv = new CsvReader(reader,
new CsvHelper.Configuration.CsvConfiguration(global::System.Globalization.CultureInfo.InvariantCulture)
{
Delimiter = ";",
HasHeaderRecord = true,
CultureInfo = global::System.Globalization.CultureInfo.InvariantCulture,
TypeConverterCache = cache
});
csv.Read();
csv.ReadHeader();
var map = (
from col in cols
from src in col.Sources()
let index = csv.GetFieldIndex(src, isTryGet: true)
where index != -1
select new { col.Name, Index = index, Type = col.DataType }).ToList();
while (csv.Read())
{
yield return map.ToDictionary(
col => col.Name,
col => EntityProperty.CreateEntityPropertyFromObject(csv.GetField(col.Type, col.Index)));
}
}
}
When writing to stream (Maybe other destinations too) CsvHelper does not return anything if my DataTable contains less than 12 rows. I tested adding rows one by one until I get a result in the string myCsvAsString variable.
Anyone ran into this problem? Here is the code I am using to reproduce it:
var stream = new MemoryStream();
using (var writer = new StreamWriter(stream))
using (var csvWriter = new CsvWriter(writer, CultureInfo.InvariantCulture))
{
if (includeHeaders)
{
foreach (DataColumn column in dataTable.Columns)
{
csvWriter.WriteField(column.ColumnName);
}
csvWriter.NextRecord();
}
foreach (DataRow row in dataTable.Rows)
{
for (var i = 0; i < dataTable.Columns.Count; i++)
{
csvWriter.WriteField(row[i]);
}
csvWriter.NextRecord();
}
csvWriter.Flush();
stream.Position = 0;
StreamReader reader = new StreamReader(stream);
string myCsvAsString = reader.ReadToEnd();
}
Ok I found the issue, I was flushing the csvWriter but I did not flush the StreamWriter.
I added writer.Flush() just after csvWriter.Flush() and the stream is complete.
I am having an .xls file containg 5 columns (id, name, address, phone, mobile) and respective values. I have to create a xml file programatically and write the 3 out of 5 column values only (id, name , mobile) in the xml file.
I am using below code to get the data from web and first writing it to an xls file which is working fine. However getting data from xls and creating and writing to xml is missing..
FtpWebResponse response = (FtpWebResponse)req.GetResponse();
Stream stream = response.GetResponseStream();
byte[] buffer = new byte[2048];
FileStream fs = new FileStream(DownloadedxlsFilePath, FileMode.Create);
int ReadCount = stream.Read(buffer, 0, buffer.Length);
while (ReadCount > 0)
{
fs.Write(buffer, 0, ReadCount);
ReadCount = stream.Read(buffer, 0, buffer.Length);
}
ResponseDescription = response.StatusDescription;
fs.Close();
stream.Close();
so there are two ways:
1. Get the whole data and select the required one and then create and write to an xml
2. Write the data to an .xls file easily using above code and then get the required data using C# logic and then write to xml.
Can any one help on either of above mentioned approach in c#.
This may help, but will require some modification on your part. I use the below code to retrieve data from an Excel file, write it to a datatable, and then iterate over each row in the datatable and write it out to a database table.
namespace CopyExcel
{
class Program
{
static void Main(string[] args)
{
var fileName = string.Format("{0}\\MaintList.xlsx", "C:\\Import");
var connectionString = string.Format("Provider=Microsoft.Jet.OLEDB.4.0; data source={0}; Extended Properties=Excel 8.0;", fileName);
var db = new ModulesDataContext();
var adapter = new OleDbDataAdapter("SELECT * FROM [WCR$]", connectionString);
var ds = new DataSet();
adapter.Fill(ds, "ExcelData");
DataTable ExcelData = ds.Tables["ExcelData"]; // datatable to store retrieved Excel data
var counter = 0;
foreach (DataRow row in ExcelData.Rows)
{
Module mod = new Module(); // this is the class built over my Module table
mod.SystemID = Convert.ToInt32(ExcelData.Rows[counter]["SysID"]);
mod.Module1 = ExcelData.Rows[counter]["Num"].ToString();
mod.Title = ExcelData.Rows[counter]["Title"].ToString();
mod.Type = "CAI";
// Save data to Module database table
db.Modules.InsertOnSubmit(mod);
db.SubmitChanges();
counter++;
}
}
}
}
You could modify so that instead of writing to datatable, you could output to XML:
var xElement = new XElement("systemid", Convert.ToInt32(ExcelData.Rows[counter]["SysID"]);
xElement.Add(new XElement("module", ExcelData.Rows[counter]["Num"].ToString());
xElement.Add(new XElement("title", ExcelData.Rows[counter]["Title"].ToString());
etc...
Of course, you'd be dealing with your own columns like id, name, and mobile.
In crystal report I am trying to display both tables of datatset ds ,but only table[0] is getting displayed in crystal report ,table[1] is not getting displayed why this is so ,I checked table[1] data is filled properly in that ,where is the problem?
private void ViewR_Load(object sender, EventArgs e)
{
String str = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
string path = str + "\\images\\";
CrystalReportP objRpt;
// Creating object of our report.
objRpt = new CrystalReportP();
DataSetPatient ds = new DataSetPatient(); // .xsd file name
DataTable dt = DBHandling.GetPatient();
ds.Tables[0].Merge(dt);
DataTable dt1 = new DataTable();
//dt1.Columns.Add("Images", typeof(Bitmap));
dt1.Columns.Add("Images", typeof(Byte[]));
if (System.IO.Directory.Exists(path))
{
string[] allImg = System.IO.Directory.GetFiles(path);
foreach (string imgName in allImg)
{
drow = dt1.NewRow();
// define the filestream object to read the image
FileStream fs;
// define te binary reader to read the bytes of image
BinaryReader br;
// check the existance of image
// open image in file stream
fs = new FileStream(imgName, FileMode.Open);
// initialise the binary reader from file streamobject
br = new BinaryReader(fs);
// define the byte array of filelength
byte[] imgbyte = new byte[fs.Length + 1];
// read the bytes from the binary reader
imgbyte = br.ReadBytes(Convert.ToInt32((fs.Length)));
drow[0] = imgbyte;
// add the image in bytearray
dt1.Rows.Add(drow[0]);
// add row into the datatable
br.Close();
// close the binary reader
fs.Close();
// close the file stream
}
}
try
{
ds.Tables[1].Merge(dt1);
}
catch (Exception r)
{
}
objRpt.SetDataSource(ds);
crystalReportViewer1.ReportSource = objRpt;
}
In Field Explorer of Crystal report Just Cheking whether both tables added in its Database Field,If not than After adding It works Properly .