Sorry for my English. I used the library Epplus and I really like it. But I've got a problem: Out of Memory. Need to write large amounts of data, no matter what. I want to know is it possible to append to the end of the Excel file is not stored in the memory of all. Or create multiple files and then concatenate into one file. Thanks in advance.
1)if you retrieve your data from database
use a datareader instead of datatable
2)write the excel to a temp file, delete it after done(if it's web environment, use response.writefile then delete it)
3)write the header first then append data to it
something like this (using my phone to type this)
var pck = new ExcelPackage();
var ws = pck.AddSheet("sheet1");
//write header here
pck.saveas(fileinfo);
pck.dispose(); // not sure if function existed
pck= new excelpage(fileino.fullname);
ws = pck.worksheets[1];
var rowIndex =0;
while (reader.read())
{
if (++rowindex % 100000 == 0)
{
// save and re-open
}
//write row here
}
pck.save();
//dispose / send file / delete file etc
Related
I want to generate big xlsx file, but don't want to keep it in memory of the server and get OutOfMemoryException .
So I read the data from database page by page, generate rows with OpenXmlWriter and send it part by part to the client:
// I use `MemoryStream` as the buffer
OpenXmlWriter = OpenXmlWriter.Create(OutputStream);
OpenXmlWriter.WriteStartElement(new Worksheet());
OpenXmlWriter.WriteStartElement(new SheetData());
foreach(var row in rows)
{
//...write cells with OpenXmlWriter and then
OutputStream.Position = 0;
var buffer = new byte[OutputStream.Length];
OutputStream.Read(buffer, 0, (int)OutputStream.Length);
FlushCalback(buffer);
OutputStream.SetLength(0);
System.Web.HttpContext.Current.Response.BinaryWrite(dataBuffer);
}
But after downloading I found, that it generates only xml markup, while xlsx is actualy "package". I can't find any example how to do that. Is there any solution? Or another libraries ?
UPDATE:
SpreadsheetDocument could help, but it writes ALL the data to the stream after calling Save() method. And it will rewrite everithing after each call.
SpreadsheetDocument = SpreadsheetDocument.Create(OutputStream, SpreadsheetDocumentType.Workbook);
//... generate rows
SpreadsheetDocument.Save();
let me outline my requirement. I have an excel spreadsheet with multiple pivot tables ( linked to charts / slicers etc ) and 2 worksheets with the data that those pivot tables refer to. Currently I have to manually execute a SQL query, copy the data, paste it over the current data in the spreadsheet and then refresh the pivot tables every day.
This is sub-optimal at best. So what I am trying to achieve is some C# code that I can execute on a schedule.
Using EPPlus, I have managed to load the excel file as a template, create a new one, get the data from SQL, update the 2 datasheets with the new data and then save the file.
using (var templateStream = new MemoryStream(File.ReadAllBytes(#"PATH_TO_TEMPLATE_FILE")))
{
using (var newStream = new MemoryStream())
{
//Create e NEW excel doc from the given template
using (ExcelPackage excelPackage = new ExcelPackage(newStream, templateStream))
{
//load the data from SQL
DataSet data = LoadDatasetFromQuery(configs, QueueItem);
//loop over the DataTables inside the DataSet
for (int i = 1; i <= data.Tables.Count; i++)
{
//Resolve the worksheet to put the data on
var worksheetName = configs.FirstOrDefault(c => c.Name.StartsWith($"Worksheet.{i}."));
ExcelWorksheet worksheet = excelPackage.Workbook.Worksheets[worksheetName.Value];
//Put the data on the worksheet top/left = B3
worksheet.Cells["B3"].LoadFromDataTable(data.Tables[i - 1], false);
}
//Save the file to the memory stream
excelPackage.Save();
}
//Write the file to the file system
File.WriteAllBytes(#"PATH_TO_OUTPUT_FILE", newStream.ToArray());
}
}
The problem is, when I try and open the excel file, it says it is corrupt and tries to repair it, which is does, by removing the pivot tables completely. My template file makes use of named ranges as referred to in this SO post but that has not resolved the issue.
Herewith the excel log of how it completed the "repair"
I have also dabbled a little bit in using the interop library ( Microsoft.Office.Interop.Excel ) but that is really like a black hole when it comes to debugging / documentation etc. I'm not averse to using it, I just don't know how. ( well nothing I have tried works properly anyways )
Any help with the above will be greatly appreciated. If you need more information, feel free to ask.
Ok, so it seems my above code was correct, but the excel template I was loading was dodgy. In order to correct the issue I had to make sure that all the pivot tables used named ranges to refer to the data ( click anywhere on the pivot table, then click on the Formulas tab in the top ribbon and then click on Name Manager ) source and then use the offset calculation ( to enable a dynamic range ) as suggested in the link in my post above.
=OFFSET(DataSource!$A$1,0,0,COUNTA(DataSource!$A:$A),COUNTA(DataSource!$1:$1))
where DataSource = the name of the worksheet with the data
Finally, I set up the pivots to refresh their data on opening ( right click on the pivot table, go to data tab and tick the "refresh on open" option )
There is a bit of a pain in that when I open the generated doc it is in "Protected mode" so the data + calcs dont refresh, but if I just click "Enable Editing" it all updates and normal service is resumed, happy days!
I use Spreadsheetgear to export the results of custom SQL queries as excel files.
Now I want to improve this system: The user will be able to upload an excel template file into the database (currently as varbinary). For example, it could have one worksheet with calculations, then when exporting data into that template it'll fill a different worksheet with the datatable from the query.
Can spreadsheetgear do this? If so, how does it work - mainly how can I load an existing excel file as a Spreadsheetgear workbook/workbookset? I could not find anything in their documentation (though I am still looking).
Edit: Solved.
I create the workbook manually, load the template from the database as a byte[], then open said template with the OpenFromMemory function:
// Create workbookSet
SpreadsheetGear.IWorkbookSet workbookSet = SpreadsheetGear.Factory.GetWorkbookSet();
// Create a new empty workbook in the workbookSet.
SpreadsheetGear.IWorkbook workbook = workbookSet.Workbooks.Add();
if(TemplateID != -1) // If this case requires a template
{
// Get template from SQL database (.xlsx stored as varbinary(max))
byte[] template = GetTemplateByID(VorlagenID);
workbook = workbookSet.Workbooks.OpenFromMemory(template);
}
// Create export worksheet
SpreadsheetGear.IWorksheet worksheet = workbook.Worksheets[0];
worksheet.Name = "Export";
[...]
Templates always use the Worksheet[1] in my case, but it should be easy to create a Worksheet[1] for the export.
Yes it is possible to set up pre-constructed template files using SpreadsheetGear. We use this extensively using .NET / C# / MSSSQL. The method allows you to create quite sophisticated templates and then simply add the required data. This of course includes any calculations you build into the template.
Method 1 - Store the template on a webserver, extract and write the created user spreadsheet to a folder on the web server. Return the filename to allow extraction by code or by the user from the server.
public static String SaveTemplateSpreadsheetToServer()
{
// Open the workbook.
var templatename = HostingEnvironment.MapPath("~/Files/MyTemplate.xlsx");
var workbook = Factory.GetWorkbook(templatename );
// Read and write to the spreadsheet
// Save a copy to disk and return filename
var filename = "The_exported_file.xlsx";
var filePath = HostingEnvironment.MapPath("~/FilesTemp/" + filename);
workbook.SaveAs(filePath, FileFormat.OpenXMLWorkbook);
// close workbook
workbook.Close();
// Return the filename
return fileName;
}
Method 2: Store the template on a webserver, extract and save modified spreadsheet as a byte array. Download directly an attachment
public static byte[] SaveTemplateSpreadsheetToServer()
{
// Open the workbook.
var templatename = HostingEnvironment.MapPath("~/Files/MyTemplate.xlsx");
var workbook = Factory.GetWorkbook(templatename );
// Read and write to the spreadsheet
// Save as byte array and send to user
var byteArray = workbook.SaveToMemory(FileFormat.OpenXMLWorkbook);
// close workbook
workbook.Close();
// Return the byte array
return byteArray;
}
We have done some work with binary template files saved in a database but find it more convenient to work with physical template files on a web server. It is easier to manage changes to the template.
My only caution is to avoid working with very big templates that have lots of "junk" in them (e.g. images). The process becomes affected by the time it takes to load the file into memory prior to the read / write / export activity. Less than 1MB is ideal and less than 2MB is manageable.
I'm currently working with an Excel file that has leading rows that have information I don't need. These extra rows also mess with importing that data in the header row below. So I'm trying to remove them to work with the data.
using (var pack = new ExcelPackage(myFileInfo))
{
// Should return the sheet name
var ws = pack.Workbook.Worksheets.FirstOrDefault();
// Should Delete rows 1-5 and shift up the rows after deletion
ws.DeleteRow(1,5,true);
}
I was thinking something like the above would work, but I've not had much success with it.
The goal would be to delete rows 1-5, shift up the rest of the data (maybe a merge would work?) then convert it into a datatable.
Anyone have tips tips or resources on removing rows from my excel sheet (prior to moving it into a datatable since that is where the issue occurs)
The code as you have it will remove the first 5 rows but you also need to do something with the amended file. You could save it in place with:
pack.Save();
or save to a new location with:
pack.SaveAs(new FileInfo(outputFilePath));
I have uploaded a complete example here:
static void Main(string[] args)
{
var myFileInfo = new FileInfo("Demo.xlsx");
using (var pack = new ExcelPackage(myFileInfo))
{
var ws = pack.Workbook.Worksheets.FirstOrDefault();
ws.DeleteRow(1, 5, true);
pack.SaveAs(new FileInfo("output.xlsx"));
}
}
If you build and run the solution you can see that it transforms the demo file from this in the input file (Demo.xlsx):
to this in the output file:
with the first 5 rows removed and everything shifted up.
I need a way to read a Excel file from a stream. It doesn't seem to work with the ADO.NET way of doing things.
The scenario is that a user uploads a file through a FileUpload and i need to read some values from the file and import to a database.
For several reasons I can't save the file to disk, and there is no reason to do so either.
So, anyone know of a way to read a Excel file from a FileUpload stream?
It seems i found a soultion to the problem myself.
http://www.codeplex.com/ExcelDataReader
This library seems to work nicely and it takes a stream to read the excel file.
ExcelDataReader reader = new ExcelDataReader(ExcelFileUpload.PostedFile.InputStream);
This can be done easily with EPPlus.
//the excel sheet as byte array (as example from a FileUpload Control)
byte[] bin = FileUpload1.FileBytes;
//gen the byte array into the memorystream
using (MemoryStream ms = new MemoryStream(bin))
using (ExcelPackage package = new ExcelPackage(ms))
{
//get the first sheet from the excel file
ExcelWorksheet sheet = package.Workbook.Worksheets[1];
//loop all rows in the sheet
for (int i = sheet.Dimension.Start.Row; i <= sheet.Dimension.End.Row; i++)
{
//loop all columns in a row
for (int j = sheet.Dimension.Start.Column; j <= sheet.Dimension.End.Column; j++)
{
//do something with the current cell value
string currentCellValue = sheet.Cells[i, j].Value.ToString();
}
}
}
SpreadsheetGear can do it:
SpreadsheetGear.IWorkbook workbook = SpreadsheetGear.Factory.GetWorkbookSet().Workbooks.OpenFromStream(stream);
You can try it for yourself with the free evaluation.
Disclaimer: I own SpreadsheetGear LLC
Infragistics has an excel component that can read an excel file from a stream.
I'm using it in a project here and it works well.
Also the open source myXls component could easily be modified to support this. The XlsDocument contstructor only supports loading from a file given by a file name, but it works by creating a FileStream and then reading the Stream, so changing it to support loading from streams should be trivial.
Edit:
I see that you found a solution but I just wanted to note that I updated the source code for the component so that it now can read an excel file directly from a stream. :-)
I use ClosedXML nuget package to read excel content from stream. It has a constructor overload in XLWorkbook class which takes stream pointing to an excel file (aka workbook).
imported namespace at the top of your code file:
using ClosedXML.Excel;
Source code:
var stream = /*obtain the stream from your source*/;
if (stream.Length != 0)
{
//handle the stream here
using (XLWorkbook excelWorkbook = new XLWorkbook(stream))
{
var name = excelWorkbook.Worksheet(1).Name;
//do more things whatever you like as you now have a handle to the entire workbook.
var firstRow = excelWorkbook.Worksheet(1).Row(1);
}
}