Develop Reference

I need helping adding two different rows(with same columns) to the same datatable

Working on a windows form application that reads in data from csv files and adds the data to a Datagridview. I ran into an issue with all of the rows being added to the datable and being displayed on the datagridview. The datagridview displays the datarows from the first two if conditions and OneRow if condition only. It will not add the rows from the twoRow if condition if the datable and datagridview rows are populated with the OneRow if condition rows. But i want the rows from both OneRow and TwoRow to be displyed. Also the rows from TwoRow do populate the datatable and datagridview when I comment(/**/) out the OneRow if condition. But I need both to populate the table. Thanks in advance!
Construct.MainDataTable.Columns.Add("Date", typeof(DateTime));
Construct.MainDataTable.Columns.Add("Time");
Construct.MainDataTable.Columns.Add("Serial");
Construct.MainDataTable.Columns.Add("Type");
Construct.MainDataTable.Columns.Add("level");
Construct.MainDataTable.Columns.Add("price");
Construct.MainDataTable.Columns.Add(" Limit");
Construct.MainDataTable.Columns.Add("last Limit");
Construct.MainDataTable.Columns.Add("Data");
..........................
...............................................
DataRow oneRow = Construct.MainDataTable.NewRow();
DataRow twoRow = Construct.MainDataTable.NewRow();
dataGridView2.AllowUserToAddRows = false;
if (line.Split(',')[2].Equals("Time"))
{
time = line.Split(',')[3];
date = line.Split(',')[1];
}
if (line.Split(',')[2].Equals("Level"))
{
level = line.Split(',')[3];
}
//OneROw(IF condition)
if ((Convert.ToDecimal(line.Split(',')[8])) < (Convert.ToDecimal (line.Split(',')[12])))
{
type = line.Split(',')[1];
serial = line.Split(',')[7];
price = line.Split(',')[3];
Limit = line.Split(',')[8];
lastLimit = line.Split(',')[10];
Data = line.Split(',')[12];
oneRow["Date"] = date;
oneRow["Time"] = time;
oneRow["Serial"] = serial;
oneRow["Type"] = type;
oneRow["level"] = level;
oneRow["price"] = price;
oneRow[" Limit"] = Limit;
oneRow["last Limit"] = lastlimit;
oneRow["Data"] = Data;
Construct.MainDataTable.Rows.Add(oneRow);
}
//TwoROw(IF condition)
if ((line.Contains('"')) && ((line.Contains("NG"))))
{
price = line.Split(',')[3];
type = line.Split(',')[1];
serial = line.Split(',')[7];
Limit = line.Split('"')[7];
var valLimit = Limit.Split(',').Select(a => Convert.ToInt32(a, 16));
var limitJoin = String.Join(",", valLimit);
lastlimit = line.Split('"')[1];
var vallastLimit = lastlimit.Split(',').Select(d => Convert.ToInt32(d, 16));
var lastJoin = String.Join(",", vallastLimit);
Data = line.Split('"')[5];
var valDatas = Data.Split(',').Select(s => Convert.ToInt32(s, 16));
var dataJoin = String.Join(",", valDatas);
twoRow["Date"] = date;
twoRow["Time"] = time;
twoRow["Serial"] = serial;
twoRow["Type"] = type;
twoRow["level"] = level;
twoRow["price"] = price;
twoRow["Limit"] = limitJoin;
twoRow["last Limit"] = lastJoin;
twoRow["Data"] = dataJoin;
Construct.MainDataTable.Rows.Add(twoRow);
}
dataGridView2.DataSource = Construct.MainDataTable;

Can't add a comment because I don't have enough karma so I ask my questions here: So, if I understood your problem you can't add data from one .csv file if it have more then one row? Why are you using 2 different if conditions for row in .csv file?
If you have empty data in row never mind you can still place them to your DataTable column, so you can use loop to add data from .csv to your DataTable. Try some thing like this:
public static DataTable CsvToDataTable(string csv)
{
DataTable dt = new DataTable();
string[] lines = csv.Split(new[] { "\r\n", "\r", "\n" }, StringSplitOptions.RemoveEmptyEntries);
Regex onlyDeimiterComma = new Regex(",(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))");
for (int i = 0; i < lines.Length; i++)
{
DataRow row = dt.NewRow();
string[] cells = onlyDeimiterComma.Split(lines[i]);
for (int j = 0; j < cells.Length; j++)
{
if (i == 0)
{
if (j == 0)
{
dt.Columns.Add(cells[j], typeof(DateTime));
}
else
{
dt.Columns.Add(cells[j]);
}
}
else
{
row[j] = cells[j];
}
}
dt.Rows.Add(row);
}
return dt;
}
Just call this method anywhere in your code and give it string read from your .csv file.
You can try to compile this code here and see how it works on .csv data with different data (empty columns, quoted text, quoted commas)
UPD: If you need to fill DataTable from two different .csv files you can still use code above. Just call it twice for both files and then Merge two DataTable, like this:
DataTable dt = CsvToDataTable(csvFileOne);
DataTable dtTwo = CsvToDataTable(csvFileTwo);
dt.Merge(dtTwo);

Google BigQuery REST API C# copy/export table with nested schema as CSV

Is there a way to export a whole table with a nested schema from Google BigQuery using the REST API as a CSV?
There is an example for doing this (https://cloud.google.com/bigquery/docs/exporting-data) with a not nested schema. This works fine on the not nested columns in my table. Here is the code of this part:
PagedEnumerable<TableDataList, BigQueryRow> result2 = client.ListRows(datasetId, result.Reference.TableId);
StringBuilder sb = new StringBuilder();
foreach (var row in result2)
{
sb.Append($"{row["visitorId"]}, {row["visitNumber"]}, {row["totals.hits"]}{Environment.NewLine}");
}
using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(sb.ToString())))
{
var obj = gcsClient.UploadObject(bucketName, fileName, contentType, stream);
}
In BQ there are columns like totals.hits, totals.visits...If I try to address them I got the errormessage that there is not such a column. If I am addressing "totals" I get the objectname "System.Collections.Generic.Dictionary`2[System.String,System.Object]" in the rows in my csv.
Is there any possibility to do something like that? In the end I want my table from GA in BQ as a CSV somewhere else.

It is possible. Select every column you need like in the following shema und flatten everything was need to be flattened.
string query = $#"
#legacySQL
SELECT
visitorId,
visitNumber,
visitId,
visitStartTime,
date,
hits.hitNumber as hitNumber,
hits.product.productSKU as product.productSKU
FROM
FLATTEN(FLATTEN({tableName},hits),hits.product)";
//Creating a job for the query and activating legacy sql
BigQueryJob job = client.CreateQueryJob(query,
new CreateQueryJobOptions { UseLegacySql = true });
BigQueryResults queryResult = client.GetQueryResults(job.Reference.JobId,
new GetQueryResultsOptions());
StringBuilder sb = new StringBuilder();
//Getting the headers from the GA table and write them into the first row of the new table
int count = 0;
for (int i = 0; i <= queryResult.Schema.Fields.Count() - 1; i++)
{
string columenname = "";
var header = queryResult.Schema.Fields[0].Name;
if (i + 1 >= queryResult.Schema.Fields.Count)
columenname = queryResult.Schema.Fields[i].Name;
else
columenname = queryResult.Schema.Fields[i].Name + ",";
sb.Append(columenname);
}
//Getting the data from the GA table and write them row by row into the new table
sb.Append(Environment.NewLine);
foreach (var row in queryResult.GetRows())
{
count++;
if (count % 1000 == 0)
Console.WriteLine($"item {count} finished");
int blub = queryResult.Schema.Fields.Count;
for (Int64 j = 0; j < Convert.ToInt64(blub); j++)
{
try
{
if (row.RawRow.F[Convert.ToInt32(j)] != null)
sb.Append(row.RawRow.F[Convert.ToInt32(j)].V + ",");
}
catch (Exception)
{
}
}
sb.Append(Environment.NewLine);
}

How to load column data from DataReader?

I'm porting application to ASP.Net 5.0 with EF7 and found several problems. One of the issues is MS have dropped DataTable. I'm trying to transfer a bit of code to not use DataTable but read from SQLDataReader and record this into entities I have. I have to read data by columns, but looks like datareader can read only once.
The old code:
Series[] arrSeries = new Series[dt.Columns.Count - 1];
IList<Categories> arrCats = new List<Categories>();
Categories arrCat = new Categories();
foreach (DataColumn dc in dt.Columns)
{
var strarr = dt.Rows.Cast<DataRow>().Select(row => row[dc.Ordinal]).ToList();
if (dc.Ordinal == 0)
{
arrCat.category = strarr.Select(o => new Category { label = o.ToString() }).ToList();
}
else
{
Series s = new Series()
{
seriesname = dc.ColumnName,
renderas = null,
showvalues = false,
data = strarr.Select(o => new SeriesValue { value = o.ToString() }).ToList()
};
arrSeries[dc.Ordinal - 1] = s;
}
}
arrCats.Add(arrCat);
MultiFusionChart fusChart = new MultiFusionChart
{
chart = dictAtts,
categories = arrCats,
dataset = arrSeries
};
return fusChart;
The new code:
Series[] arrSeries = new Series[colColl.Count - 1];
IList<Categories> arrCats = new List<Categories>();
Categories arrCat = new Categories();
arrCat.category = new List<Category>();
for (int i = 0; i < reader.FieldCount; i++)
{
Series s = new Series()
{
seriesname = reader.GetName(i),
renderas = null,
showvalues = false
};
while (reader.Read())
{
if (i == 0)
{
Category cat = new Category();
cat.label = reader.GetValue(i).ToString();
arrCat.category.Add(cat);
}
else
{
SeriesValue sv = new SeriesValue();
sv.value = reader.GetValue(i).ToString();
s.data.Add(sv);
arrSeries[i - 1] = s;
}
}
}
arrCats.Add(arrCat);
MultiFusionChart fusChart = new MultiFusionChart
{
chart = dictAtts,
categories = arrCats,
dataset = arrSeries
};
return fusChart;
Where the code works, it returns null for Series. And I believe this is because reader went to the end while recording Categories. As far as I know it is not possible to reset DataReader?
Is there a way to load column data from DataReader to List? Or maybe there is other way how I can replace DataTable in this example?

You can read value from multiple columns by specifying its index like this
int totalColumns = reader.FieldCount;
for(int i=0;i<totalColumns;i++)
{
var label = reader.GetValue(i);
}
You can select value of all columns by looping through columns.
GetValue takes int as argument which is index of column not the index of row.

The problem is in your for loop. At first when your i is 0, You have initialized a 'Series` object S. After that,
while (reader.Read())
will read your entire Reader while your i will still be zero. SO only the
if(i == 0)
condition will return true and your entire reader will be spent. Afterwards, for i >= 1,
while (reader.Read())
will always return false keeping your array, arrSeries blank. Remove the while loop and directly read value from the dataReader using the index,
reader.GetValue(i);

Upload XLSX to XDocument changes the number format

I have the following method to take an XLSX file and convert it to an XDocument:
public static XDocument ConvertXlsx2Xml(string fileName, string sheetName)
{
// Return the value of the specified cell.
const string documentRelationshipType = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument";
const string worksheetSchema = "http://schemas.openxmlformats.org/spreadsheetml/2006/main";
const string sharedStringsRelationshipType = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/sharedStrings";
const string sharedStringSchema = "http://schemas.openxmlformats.org/spreadsheetml/2006/main";
string cellValue = null;
var xsurvey = new XDocument(new XDeclaration("1.0", "UTF-8", "yes"));
var xroot = new XElement("Root"); //Create the root
using (Package xlPackage = Package.Open(fileName, FileMode.Open, FileAccess.Read))
{
PackagePart documentPart = null;
Uri documentUri = null;
// Get the main document part (workbook.xml).
foreach (System.IO.Packaging.PackageRelationship relationship in xlPackage.GetRelationshipsByType(documentRelationshipType))
{
// There should only be one document part in the package.
documentUri = PackUriHelper.ResolvePartUri(new Uri("/", UriKind.Relative), relationship.TargetUri);
documentPart = xlPackage.GetPart(documentUri);
// There should only be one instance, but get out no matter what.
break;
}
if (documentPart != null)
{
// Load the contents of the workbook.
var doc = new XmlDocument();
doc.Load(documentPart.GetStream());
/*
doc now contains the following important nodes:
<bookViews>
<workbookView xWindow="-15615" yWindow="2535" windowWidth="26835" windowHeight="13095" activeTab="2" />
<sheets>
<sheet name="Sheet1" sheetId="2" r:id="rId1" />
*/
// Create a namespace manager, so you can search.
// Add a prefix (d) for the default namespace.
var nt = new NameTable();
var nsManager = new XmlNamespaceManager(nt);
nsManager.AddNamespace("d", worksheetSchema);
nsManager.AddNamespace("s", sharedStringSchema);
//If value for sheetName isn't found, take the first sheet
string searchString = string.Format("//d:sheet[#name='{0}']", sheetName);
XmlNode sheetNode = doc.SelectSingleNode(searchString, nsManager) ??
doc.SelectSingleNode("//d:sheet", nsManager);
/*
* 11/15/12 DS Added to avoid pulling the data each time
* Create a dictionary of the shared strings from the associated string file
*/
#region Shared String Dictionary
var sharedStrings = new Dictionary<int, string>();
foreach (System.IO.Packaging.PackageRelationship stringRelationship in documentPart.GetRelationshipsByType(sharedStringsRelationshipType))
{
// There should only be one shared string reference, so you exit this loop immediately.
Uri sharedStringsUri = PackUriHelper.ResolvePartUri(documentUri, stringRelationship.TargetUri);
PackagePart stringPart = xlPackage.GetPart(sharedStringsUri);
{
// Load the contents of the shared strings.
var stringDoc = new XmlDocument(nt);
stringDoc.Load(stringPart.GetStream());
nsManager.AddNamespace("s", sharedStringSchema);
const string strSearch = "//s:sst";
XmlNode stringNode = stringDoc.SelectSingleNode(strSearch, nsManager);
int keyInt = 0;
if (stringNode != null)
foreach (XmlElement nd in stringNode)
{
//string test = nd.InnerText;
sharedStrings.Add(keyInt, nd.InnerText);
keyInt = keyInt + 1;
}
}
}
#endregion
var hrowList = new List<string>();
var hrowArray = new string[] {};
if (sheetNode != null && sheetNode.Attributes != null)
{
// Get the relId attribute:
XmlAttribute relationAttribute = sheetNode.Attributes["r:id"];
if (relationAttribute != null)
{
string relId = relationAttribute.Value;
// First, get the relation between the document and the sheet.
PackageRelationship sheetRelation = documentPart.GetRelationship(relId);
Uri sheetUri = PackUriHelper.ResolvePartUri(documentUri, sheetRelation.TargetUri);
PackagePart sheetPart = xlPackage.GetPart(sheetUri);
// Load the contents of the workbook.
var sheetDoc = new XmlDocument(nt);
sheetDoc.Load(sheetPart.GetStream());
/*
* sheetDoc now contains the following important nodes:
* <dimension ref="A1:V81" /> range of sheet data
* <sheetData>
* <row r="1" spans="1:22"> <row> r = row number, spans = columns containing the data
* <c r="A1" t="s"><v>0</v></c> <c> r = Cell address (A1,B4,etc), t = data type ("s"=string,"b"=bool, null=decimal)
* <v> contents are the index num if t="s", or value of t=null
*/
XmlNode sheetDataNode = sheetDoc.SelectSingleNode("//d:sheetData", nsManager);
int roNum = 0;
if (sheetDataNode != null)
{
var isSkip = false;
foreach (XmlElement row in sheetDataNode)
{
var xrow = new XElement("Row");
foreach (XmlElement cell in row)
{
XmlAttribute typeAttr = cell.Attributes["t"];
string cellType = typeAttr != null ? typeAttr.Value : string.Empty;
XmlNode valueNode = cell.SelectSingleNode("d:v", nsManager);
cellValue = valueNode != null ? valueNode.InnerText : cellValue;
// Check the cell type. At this point, this code only checks for booleans and strings individually.
switch (cellType)
{
case "b":
cellValue = cellValue == "1" ? "TRUE" : "FALSE";
break;
case "s":
cellValue = sharedStrings[Convert.ToInt32(cellValue)];
break;
}
if (cellValue == null) continue;
cellValue = cellValue.Replace("\r", "");
cellValue = cellValue.Replace("\n", " ");
cellValue = cellValue.Trim();
if (roNum == 0)
{
hrowList.Add(cellValue);
}
else
{
//XmlAttribute rowAttr = cell.Attributes["r"];
//int intStart = rowAttr.Value.IndexOfAny("0123456789".ToCharArray());
//colLet = rowAttr.Value.Substring(0, intStart);
//int colNum = NumberFromExcelColumn(colLet);
int colNum = GetColNum(cell);
/* 05/29/13 DS force column names to UPPER to remove case sensitivity */
var xvar = new XElement(hrowArray[colNum - 1].ToUpper());
xvar.SetValue(cellValue);
xrow.Add(xvar);
}
/* 6/18/2013 DS You must clear the cellValue so it is carried into the next cell value if it is empty. */
cellValue = "";
}
if (roNum == 0) hrowArray = hrowList.ToArray();
else xroot.Add(xrow);
roNum = roNum + 1;
}
}
}
}
}
}
xsurvey.Add(xroot);
return xsurvey;
}
For the most part, it works very well. However, I have just noticed that if one of the cell values contains a number like 0.004 it becomes 4.0000000000000001E-3.
The resulting XML gets imported and that value is loaded as a string, but before the final transfer to the production tables, this particular field is converted to numeric. That format doesn't work with numeric.
How do I prevent that change on load? If I can't, is there a better method to prevent a system error, other than specifically scrubbing that field and changing it back as part of the transfer process?
UPDATE
It is only numbers less than .01 that have a problem. 1, 1.004, and .04 are fine, but .004 is not.
UPDATE 2
If I format the cells as text BEFORE populating the data, I do not have this issue. There is something about how ManEx stores the data that prevents a clean upload.

Read value as double with invariant culture, and then convert it to decimal
var cellValueDouble = Convert.ToDouble(cellValue,System.Globalization.CultureInfo.InvariantCulture);
var cellValueDecimal = Convert.ToDecimal(cellValueDouble);

Duplicate fields when generating PDF from template

I am reading an XML file and using the data to fill out a PDF template.
The XML file has a field called <my:SpecialInstructions>Test for special instructions</my:SpecialInstructions>
whereas the PDF template fields are:
and the fields are called:
The problem I am getting is Test for special instructions is getting duplicated once for the first line and once for the second line
question: how does the <my:SpecialInstructions> tag get translated into my:SpecialInstructions1 & my:SpecialInstructions2?
I am using XMLParse, c# for this.
Per Robert's suggestion I am including the code that is populating the PDF:
public void InsertDataToTemplate(DataTable dt)
{
// cycle through datatable and find field to field matches
DataRow row = dt.Rows[0];
int fieldType = 0;
string checkBoxInsert = "";
List<string> notFoundList = new List<string>();
foreach (DataColumn col in dt.Columns)
{
if (pdfStamper.AcroFields.Fields.Where(afd => afd.Key == col.ColumnName).Count() != 0)
{
fieldType = pdfStamper.AcroFields.GetFieldType(col.ColumnName);
if (fieldType == AcroFields.FIELD_TYPE_CHECKBOX)
{
checkBoxInsert = (row[col.ColumnName].ToString().ToUpper() == "FALSE") ? "NO" : "Yes";
pdfStamper.AcroFields.SetField(col.ColumnName, checkBoxInsert);
}
else
{
pdfStamper.AcroFields.SetField(col.ColumnName, row[col.ColumnName].ToString());
}
}
else
notFoundList.Add(col.ColumnName);
}
}