Develop Reference

Serialize parquet data with C#

Is there a way to serialize data in Apache Parquet format using C#, I can't find any implementation of that. In the oficial Parquet docs it is said that "Thrift can be also code-genned into any other thrift-supported language." but I'm not sure what this actually means.
Thanks

I have started an opensource project for .NET implementation of Apache Parquet, so anyone is welcome to join. https://github.com/aloneguid/parquet-dotnet

We've just open sourced our .NET wrapper around Apache Parquet C++. It's a different approach compared to Parquet.NET, the latter being a pure .NET implementation.
You're welcome to give it a try and share your feedback:
https://github.com/G-Research/ParquetSharp

Here is another one to the list. Cinchoo ETL - an open source library, can do parquet files read and write.
Method 1: POCO Method
Define POCO class
public partial class Employee
{
public int Id { get; set; }
public string Name { get; set; }
}
Serialization code
List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();
Employee rec1 = new Employee();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
Employee rec2 = new Employee();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
using (var parser = new ChoParquetWriter<Employee>("emp.parquet"))
{
parser.Write(objs);
}
Method 2: Dynamic Method
List<ExpandoObject> objs = new List<ExpandoObject>();
dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
using (var parser = new ChoParquetWriter("emp.parquet"))
{
parser.Write(objs);
}
Disclaimer: I'm author of this library

No there isn't. I've spent a week trying to write my own parquet writer for .NET and it's just too complicated i.e. needs much more time. I ended up using Python and fastparquet library to do any processing outside of Hadoop clusters. I must say fastparquet is an amazing piece of work and very easy to work with but there is a lot of functionality missing i.e. nested columns and ability to effectively append to the file. Not mentioning dependency on Python3 which can be a headache to deploy.
You can generate Thrift protocols into C# but that doesn't get you far, it just means your output will be compatible with Parquet specification.
I'm still keen to create an opensource Parquet library for .NET Core/.NET 4.5 so if anyone is keen to cooperate please let me know.

How to cache reading .csv files in C#

This may be a noob question, but I need some help. I have written two simple methods in C#: ReadCsv_IT and GetTranslation. The ReadCsv_IT method reads from a csv file. The GetTransaltion method calls the ReadCsv_IT method and returns the translated input (string key).
My problem is that in the future I will have to request a lot of times GetTranslation, but I obviously don't want to read the .csv files every time. So I was thinking about ways to use cache Memory to optimize my program, so that I don't have to read the .csv file on every request. But I am not sure how to do it and what I could do to optimize my program. Can anyone please help ?
public string ReadCsv_IT(string key)
{
string newKey = "";
using (var streamReader = new StreamReader(#"MyResource.csv"))
{
CsvReader csv = new CsvReader(streamReader);
csv.Configuration.Delimiter = ";";
List<DataRecord> rec = csv.GetRecords<DataRecord>().ToList();
DataRecord record = rec.FirstOrDefault(a => a.ORIGTITLE == key);
if (record != null)
{
//DOES THE LOCALIZATION with the help of the .csv file.
}
}
return newKey;
}
Here is the GetTranslation Method:
public string GetTranslation(string key, string culture = null)
{
string result = "";
if (culture == null)
{
culture = Thread.CurrentThread.CurrentCulture.Name;
}
if (culture == "it-IT")
{
result = ReadCsv_IT(key);
}
return result;
}
Here is also the class DataRecord.
class DataRecord
{
public string ORIGTITLE { get; set; }
public string REPLACETITLE { get; set; }
public string ORIGTOOLTIP { get; set; }
}
}

Two options IMO:
Turn your stream into an object?
In other words:
Make a class stream so you can refer to that object of the class stream.
Second:
Initialize your stream in the scope that calls for GetTranslation, and pass it on as an attribute to GetTranslation and ReadCSV_IT.

Brecht C and Thom Hubers have already given you good advice. I would like to add one more point, though: using csv files for localization in .NET is not really a good idea. Microsoft recommends using a resource-based approach (this article is a good starting point). It seems to me that you are trying to write code for something that is already built into .NET.
From a translation point of view csv files are not the best possible format either. First of all, they are not really standardized: many tools have slightly different ways to handle commas, quotes, and line breaks that are part of the translated text. Besides, translators will be tempted to open them in Excel, and -unless handled with caution- Excel will write out translations in whatever encoding it deems best.
If the project you are working on is for learning please feel free to go ahead with it, but if you are developing software that will be used by customers, updated, translated into several target languages, and redeployed, I would recommend to reconsider your internationalization approach.

#Brecht C is right, use that answer to start. When a variable has to be cached to be used by multiple threads or instances: take a look at InMemoryCache or Redis when perfomance and distribution over several clients gets an issue.

Api for working with a classes as OOP?

I'm writing a 3rd party app that needs to read in .cs files and be able to manipulate classes, then ultimately save back to file.
The type of code I am looking at would be something like:
var classManager = new classManager();
var classes = classManager.LoadFromFile(filePath);
var class = classes[0]; // Just illustrating more than 1 class can exist in a file
var prop = new ClassProperty {Type=MyType.GetType() };
prop.AddGet("return x+y < 50");
//stuff like prop.ReadOnly = true;
class.AddProperty(prop);
var method = new ClassMethod {signature="int id, string name"};
method.MethodBody = GetMethodBodyAsString(); //not writing out an entire method body here
class.AddMethod(method);
class.SaveToFile(true); //Format code
Does such a library exist?

The .NET Compiler Platform Roslyn is what you're looking for. It supports parsing and editting cs files. Check out this post for an example

Keeping track of user customization's c#

Good evening; I have an application that has a drop down list; This drop down list is meant to be a list of commonly visited websites which can be altered by the user.
My question is how can I store these values in such a manor that would allow the users to change it.
Example; I as the user, decide i want google to be my first website, and youtube to be my second.
I have considered making a "settings" file however is it practical to put 20+ websites into a settings file and then load them at startup? Or a local database, but this may be overkill for the simple need.
Please point me in the right direction.

Given you have already excluded database (probably for right reasons.. as it may be over kill for a small app), I'd recommend writing the data to a local file.. but not plain text..
But preferably serialized either as XML or JSON.
This approach has at least two benefits -
More complex data can be stored in future.. example - while order can be implicit, it can be made explicit.. or additional data like last time the url was used etc..
Structured data is easier to validate against random corruption.. If it was a plain text file.. It will be much harder to ensure its integrity.

The best would be to use the power of Serializer and Deserializer in c#, which will let you work with the file in an Object Oriented. At the same time you don't need to worry about storing into files etc... etc...
Here is the sample code I quickly wrote for you.
using System;
using System.IO;
using System.Collections;
using System.Xml.Serialization;
namespace ConsoleApplication3
{
public class UrlSerializer
{
private static void Write(string filename)
{
URLCollection urls = new URLCollection();
urls.Add(new Url { Address = "http://www.google.com", Order = 1 });
urls.Add(new Url { Address = "http://www.yahoo.com", Order = 2 });
XmlSerializer x = new XmlSerializer(typeof(URLCollection));
TextWriter writer = new StreamWriter(filename);
x.Serialize(writer, urls);
}
private static URLCollection Read(string filename)
{
var x = new XmlSerializer(typeof(URLCollection));
TextReader reader = new StreamReader(filename);
var urls = (URLCollection)x.Deserialize(reader);
return urls;
}
}
public class URLCollection : ICollection
{
public string CollectionName;
private ArrayList _urls = new ArrayList();
public Url this[int index]
{
get { return (Url)_urls[index]; }
}
public void CopyTo(Array a, int index)
{
_urls.CopyTo(a, index);
}
public int Count
{
get { return _urls.Count; }
}
public object SyncRoot
{
get { return this; }
}
public bool IsSynchronized
{
get { return false; }
}
public IEnumerator GetEnumerator()
{
return _urls.GetEnumerator();
}
public void Add(Url url)
{
if (url == null) throw new ArgumentNullException("url");
_urls.Add(url);
}
}
}

You clearly need some sort of persistence, for which there are a few options:
Local database
- As you have noted, total overkill. You are just storing a list, not relational data
Simple text file
- Pretty easy, but maybe not the most "professional" way. Using XML serialization to this file would allow for complex data types.
Settings file
- Are these preferences really settings? If they are, then this makes sense.
The Registry - This is great for settings you don't want your users to ever manually mess with. Probably not the best option for a significant amount of data though
I would go with number 2. It doesn't sound like you need any fancy encoding or security, so just store everything in a text file. *.ini files tend to meet this description, but you can use any extension you want. A settings file doesn't seem like the right place for this scenario.

Not able to understand .Net Code

Is there any idea what he is doing in the below code at line
"chartSeries.AddItem(new ChartSeriesItem(listData[i].x, listData[i].y));"
I took this code from one forum.
http://www.telerik.com/community/forums/aspnet-ajax/chart/performance-problem-on-line-chart.aspx
I was not able to understand what kind of data holder it is?
Am pulling the data from in dataset format, If I could bind my dataset with listdata, I would acheive this development. But am not undestanding how I need to make this binding.
I have tried as below but didn't worked and throghing an error, please find the attached.
Dim listdata As new DataList
listdata.DataSource = ds.Tables.Item(0)
listdata.DataBind()
Original code took from forum
ChartSeries chartSeries = new ChartSeries("Altitude", ChartSeriesType.Line);
chartSeries.Appearance.ShowLabels = false;
incrementCount = 1;
for (int i = 0; i < listData.Count; i = i + incrementCount)
{
chartSeries.AddItem(new ChartSeriesItem(listData[i].x, listData[i].y));
}
RadChart2.AddChartSeries(chartSeries);
RadChart2.PlotArea.XAxis.AxisLabel.TextBlock.Text = "Distance (Miles)";
RadChart2.PlotArea.YAxis.AxisLabel.TextBlock.Text = "Yaxislabel";

I think its as simple as this, the error is that you can't use the word New, the code from the forum says to use new, capitalisation can make all the difference in .NET.