Develop Reference

Replacing Invalid XML characters from an excel file and writing it back to disk causes file is corrupted error on opening in MS Excel

A little background on problem:
We have an ASP.NET MVC5 Application where we use FlexMonster to show the data in grid. The data source is a stored procedure that brings all the data into the UI grid, and once user clicks on export button, it exports the report to Excel. However, in some cases export to excel is failing.
Some of the data has some invalid characters, and it is not possible/feasible to fix the source as suggested here
My approach so far:
EPPlus library fails on initializing the workbook as the input excel file contains some invalid XML characters. I could find that the file is dumped with some invalid character in it. I looked into the possible approaches .
Firstly, I identified the problematic character in the excel file. I first tried to replace the invalid character with blank space manually using Notepad++ and the EPPlus could successfully read the file.
Now using the approaches given in other SO thread here and here, I replaced all possible occurrences of invalid chars. I am using at the moment
XmlConvert.IsXmlChar
method to find out the problematic XML character and replacing with blank space.
I created a sample program where I am trying to work on the problematic excel sheet.
//in main method
String readFile = File.ReadAllText(filePath);
string content = RemoveInvalidXmlChars(readFile);
File.WriteAllText(filePath, content);
//removal of invalid characters
static string RemoveInvalidXmlChars(string inputText)
{
StringBuilder withoutInvalidXmlCharsBuilder = new StringBuilder();
int firstOccurenceOfRealData = inputText.IndexOf("<t>");
int lastOccurenceOfRealData = inputText.LastIndexOf("</t>");
if (firstOccurenceOfRealData < 0 ||
lastOccurenceOfRealData < 0 ||
firstOccurenceOfRealData > lastOccurenceOfRealData)
return inputText;
withoutInvalidXmlCharsBuilder.Append(inputText.Substring(0, firstOccurenceOfRealData));
int remaining = lastOccurenceOfRealData - firstOccurenceOfRealData;
string textToCheckFor = inputText.Substring(firstOccurenceOfRealData, remaining);
foreach (char c in textToCheckFor)
{
withoutInvalidXmlCharsBuilder.Append((XmlConvert.IsXmlChar(c)) ? c : ' ');
}
withoutInvalidXmlCharsBuilder.Append(inputText.Substring(lastOccurenceOfRealData));
return withoutInvalidXmlCharsBuilder.ToString();
}
If I replaces the problematic character manually using notepad++, then the file opens fine in MSExcel. The above mentioned code successfully replaces the same invalid character and writes the content back to the file. However, when I try to open the excel file using MS Excel, it throws an error saying that file may have been corrupted and no content is displayed (snapshots below). Moreover, Following code
var excelPackage = new ExcelPackage(new FileInfo(filePath));
on the file that I updated via Notepad++, throws following exception
"CRC error: the file being extracted appears to be corrupted. Expected 0x7478AABE, Actual 0xE9191E00"}
My Questions:
Is my approach to modify content this way correct?
If yes, How can I write updated string to an Excel file?
If my approach is wrong then, How can I proceed to get rid of invalid XML chars?
Errors shown on opening file (without invalid XML char):
First Pop up
When I click on yes
Thanks in advance !

It does sounds like a binary (presumable XLSX) file based on your last comment. To confirm, open the file created by the FlexMonster with 7zip. If it opens properly and you see a bunch of XML files in folders, its a XLSX.
In that case, a search/replace on a binary file sounds like a very bad idea. It might work on the XML parts but might also replace legit chars in other parts. I think the better approach would be to do as #PanagiotisKanavos suggests and use ZipArchive. But you have to do rebuild it in the right order otherwise Excel complains. Similar to how it was done here https://stackoverflow.com/a/33312038/1324284, you could do something like this:
public static void ReplaceXmlString(this ZipArchive xlsxZip, FileInfo outFile, string oldString, string newstring)
{
using (var outStream = outFile.Open(FileMode.Create, FileAccess.ReadWrite))
using (var copiedzip = new ZipArchive(outStream, ZipArchiveMode.Update))
{
//Go though each file in the zip one by one and copy over to the new file - entries need to be in order
foreach (var entry in xlsxZip.Entries)
{
var newentry = copiedzip.CreateEntry(entry.FullName);
var newstream = newentry.Open();
var orgstream = entry.Open();
//Copy non-xml files over
if (!entry.Name.EndsWith(".xml"))
{
orgstream.CopyTo(newstream);
}
else
{
//Load the xml document to manipulate
var xdoc = new XmlDocument();
xdoc.Load(orgstream);
var xml = xdoc.OuterXml.Replace(oldString, newstring);
xdoc = new XmlDocument();
xdoc.LoadXml(xml);
xdoc.Save(newstream);
}
orgstream.Close();
newstream.Flush();
newstream.Close();
}
}
}
When it is used like this:
[TestMethod]
public void ReplaceXmlTest()
{
var datatable = new DataTable("tblData");
datatable.Columns.AddRange(new[]
{
new DataColumn("Col1", typeof (int)),
new DataColumn("Col2", typeof (int)),
new DataColumn("Col3", typeof (string))
});
for (var i = 0; i < 10; i++)
{
var row = datatable.NewRow();
row[0] = i;
row[1] = i * 10;
row[2] = i % 2 == 0 ? "ABCD" : "AXCD";
datatable.Rows.Add(row);
}
using (var pck = new ExcelPackage())
{
var workbook = pck.Workbook;
var worksheet = workbook.Worksheets.Add("source");
worksheet.Cells.LoadFromDataTable(datatable, true);
worksheet.Tables.Add(worksheet.Cells["A1:C11"], "Table1");
//Now similulate the copy/open of the excel file into a zip archive
using (var orginalzip = new ZipArchive(new MemoryStream(pck.GetAsByteArray()), ZipArchiveMode.Read))
{
var fi = new FileInfo(#"c:\temp\ReplaceXmlTest.xlsx");
if (fi.Exists)
fi.Delete();
orginalzip.ReplaceXmlString(fi, "AXCD", "REPLACED!!");
}
}
}
Gives this:
Just keep in mind that this is completely brute force. Anything you can do to make the file filter smarter rather then simply doing ALL xml files would be a very good thing. Maybe limit it to the SharedString.xml file if that is where the problem lies or in the xml files in the worksheet folders. Hard to say without knowing more about the data.

Record count for Excel file with extension CSV

I am trying to count the record of excel file with extension csv with the following code but I am not getting exact number of rows.
int lineCount = 0;
using (var reader = File.OpenText(#fileFullPath))
{
while (reader.ReadLine()!= null)
{
lineCount++;
}
}
Can anyone please advise me on this.

You can do:
var lineCount = File.ReadAllLines(#fileFullPath).Length

First try to open the mentioned .csv file using notepad. If the file opens and readable, It means that your file is human readable and it can be read by File reader line by line without further processing.
If it is so you can use var lineCount = File.ReadAllLines(#fileFullPath).Length. Otherwise you need to have library for reading Excel file

C# splitting large excel file to smaller files

I want to split one large Excel file to few smaller and accessible files.
I already tried to use this code but the files are not accessible:
using (System.IO.StreamReader sr = new System.IO.StreamReader("path"))
{
int fileNumber = 0;
while (!sr.EndOfStream)
{
int count = 0;
using (System.IO.StreamWriter sw = new System.IO.StreamWriter("other path" + ++fileNumber + ".xlsx"))
{
sw.AutoFlush = true;
while (!sr.EndOfStream && ++count < 20000)
{
sw.WriteLine(sr.ReadLine());
}
}
}
}
Any ideas?
thanks.

Files, other than text files, don't work this way. You can't simply cut at a certain point and obtain a working copy.
As for Excel files, you may look into the following tutorial, which illustrates how to automate Excel from C#:
https://support.microsoft.com/en-us/help/302084/how-to-automate-microsoft-excel-from-microsoft-visual-c--net
Basically, what you want to do is open your large Excel file, decide where you want to split it (every n rows, every n sheets and so on), read each portion and write into a newly created xlsx.

Using FileHelpers to import Excel data using MVC 5

I'm trying to write an application in MVC 5 that will accept a file specified by a user and upload that file information into the database. The file itself has multiple worksheets, which I think FileHelpers handles gracefully, but I can't find any good documentation about working with a byte array. I can get the file just fine, and get to my controller, but don't know where to go from there. I am currently doing this in the controller:
public ActionResult UploadFile(string filepath)
{
//we want to check here that the first file in the request is not null
if (Request.Files[0] != null)
{
var file = Request.Files[0];
byte[] data = new byte[file.ContentLength];
ParseInputFile(data);
//file.InputStream.Read(data, 0, data.Length);
}
ViewBag.Message = "Success!";
return View("Index");
}
private void ParseInputFile(byte[] data)
{
ExcelStorage provider = new ExcelStorage(typeof(OccupationalGroup));
provider.StartRow = 3;
provider.StartColumn = 2;
provider.FileName = "test.xlsx";
}
Am I able to use the Request like that in conjunction with FileHelpers? I just need to read the Excel file into the database. If not, should I be looking into a different way to handle the upload?

So, I decided instead to use ExcelDataReader to do my reading from Excel. It puts the stream (in the below code, test) into a DataSet that I can just manipulate manually. I'm sure it might not be the cleanest way to do it, but it made sense for me, and allows me to work with multiple worksheets fairly easily as well. Here is the snippet of regular code that I ended up using:
//test is a stream here that I get using reflection
IExcelDataReader excelReader = ExcelReaderFactory.CreateOpenXmlReader(test);
DataSet result = excelReader.AsDataSet();
while(excelReader.Read())
{
//process the file
}
excelReader.Close();

getting info from dataset and putting in worksheet?

I am having some issues with the following:
I give my method a dataset and it is supposed to throw the information into an excel file, which is a template of the formatting I desire. The excel file i created has a header, some filters and things of the sort, and I set my method to populate the file AFTER the header, etc but the problem is, when I do that, I lose all the formatting i had on the template. I am using this class http://www.codeproject.com/KB/office/biffcsharp.aspx . Im not sure, it may be that the format for the implementation of the class is a real simple one or that it overwrites all the information I had.
my method looks like this, using the class on the link above :
public void PopularSheet()
{
string filename = "C:\\test" + System.Web.HttpContext.Current.Session["SYSTEMCLIENTID"].ToString()+ System.Web.HttpContext.Current.Session["SYSTEMUSERTYPEID"].ToString()+ System.Web.HttpContext.Current.Session["CLIENTID"].ToString()+".xls";
File.Copy("C:\\test.xls", filename);
FileStream stream = new FileStream(filename, FileMode.OpenOrCreate);
ExcelWriter writer = new ExcelWriter(stream);
DataSet ds = GetDataSet();
writer.BeginWrite();
int jValue = ds.Tables[0].Columns.Count;
int iValue = ds.Tables[0].Rows.Count;
// Passa os dados do dataset para a planilha
for (int i = 0; i < iValue; i++)
{
// Lê todas as colunas da linha i
for (int j = 0; j < jValue; j++)
{
writer.WriteCell(i+2, j, ds.Tables[0].Rows[i][j].ToString());
}
}
writer.EndWrite();
stream.Close();
}
I also tried using an excel library,http://www.carlosag.net/tools/excelxmlwriter/,but i think in order to LOAD a file (so that i can insert the information i need into it) i need to load a xml file, which is impossible!
Another library I used presented a problem when saving, I was able to edit the worksheet and then when i saved and opened the excel file that was generated thru the code, it would come out empty .
I cannot use anything that will force me to install excel, which is why i am trying these alternatives. Are there any suggestions to what I could do?
What i need to do :
Load an existing excel file as "template"
Throw a dataset into the file
Save the file with the information that i threw with the template format

There is a library called ClosedXML which is useful for creating openxml file.

Quick thought: can you use a .csv instead of a standard .xls file?
If so, you can easily fill your table.