I am trying to fetch data from database and export it to .csv file format, which is similar to ETL process. But I want to do this in C#.
SQL Query to fetch data from database.
Format the data to a specified file specification.
Convert it to .csv file.
I know that 1st step is easy to do, I am struggling to find a way for 2nd and 3rd step.
I can't comment on the file specification, as you haven't described it, but writing CSV files is pretty easy. Unlike XML etc the format is so simplistic you can write directly to a StreamWriter using WriteLine. All you need is to output a first line that contains the column names separated with commas, then for each row returned from your SQL Query write the column values in the same order separated by commas. The only real gotcha is escaping, e.g. dealing with commas, quotes, etc by surrounding each value with quotes and escaping any quotes in the value.
The example below does just that for a DataTable:
http://dotnetguts.blogspot.co.nz/2007/01/exporting-datatable-to-csv-file-format.html
I figured out a simple and efficient way to do this:
1.SQL Query to fetch data from database.
2. Format the data to a specified file specification.
For the first 2 steps, I have used the Dapper.NET, which
took care of the database part as well as formatting. Dapper helped to convert the SQL results to a LIST by which I fulfilled the file specifications.
3.Convert it to .csv file.
For converting the SQL results to CSV file, I have used FileHelpers
Library which is more simple than I expected.
Hope it will help someone.
Related
I'm struggling to figure out how to convert a CSV file into a database. I've tried a few methods here but I can't wrap my head around it. I have a CSV file with thousands of rows and I need to convert that into a
SQLite database using C#. Any help is appreciated!
You can leverage MS Excel, open the CSV file and specify your character separator as needed (I believe it will default to tab limited). You can now save your thousands of rows as an Excel spreadsheet versus the character separated file (CSV) format.
Then you can leverage the open source OpenXML libraries to open the Excel document and work with it using object model. In an object oriented fashion, you can programatically create your new database using SQL statements.
Query for the spreadsheet headers to be used as your column names. Of course you'll need to ensure that your source CSV had provided appropriate headers. These can easily be added to the top of the large file if not.
E.g.
https://learn.microsoft.com/en-us/office/open-xml/how-to-get-a-column-heading-in-a-spreadsheet
Next, you simply iterate the rows and construct your SQL statement to insert the rows. You can review the Open XML docs, Microsoft docs, or existing StackOverflow docs for sample code on how this is easily done.
How to read xslx with open XML SDK based on columns in each row in C#?
I have a CSV file with the following fields.
DocPath, Data
C:\CR-12-12-02-03-12-9AM-JG.docx, 0x50004b000300040014000000060000000800000000000 (not the whole thing, it is a massively long string)
The data comes from an MSaccess database but all i have is the CSV.
I need to convert the files back to their original document files or put it into an acceptable format in sql where i can extract them later. The data was stored in a field type 'long binary' in MSaccess and extracted to this CSV
I have tried researching but have come up empty. Maybe I am asking the wrong questions so I am explaining it here in hopes that someone can help me?
I have to create a fix length record file using C#. I read the records from database and then after some business logic I would like to write some of the fields into a text file.
I researched this and some recommend using an XML to define the file, while others format the string and then write that into a file.
Is there a proper way to do this? I would like to keep this object oriented.
Thanks for any hints.
take a look at http://www.filehelpers.com/ to export the data to a fixed file format. you may also want to look at http://hibernatingrhinos.com/open-source/rhino-etl to create a process that runs the export for you. rhino.etl includes operations for FileHelpers.
You can use Jet to parse fixed length records in a text file. This is a decent overview that should be able to get you started:
http://msmvps.com/blogs/deborahk/archive/2009/08/25/reading-fixed-length-files.aspx
In the past when i've done this, I pull the data out of SQL as a Char(Length) so that the padding is handled by SQL. This makes the C# pretty easy to put together. You can do this by casting or converting the data when you select it, or by creating a table that is the exact format of your file and inserting the data in there before pulling it out into C#.
How do I convert an exported IXF file (using db2 export) to a human-readable format, like CSV or XML? I am comfortable with doing it in Python or .NET C#.
The PC/IXF format is fairly complex, and is practically unknown to programs outside of DB2. Writing your own PC/IXF parser just to convert an IXF file directly to some other format might take a while. A faster alternative is to issue an IMPORT command on the DB2 server and specify CREATE INTO instead of INSERT INTO , which will generate a brand new table that can accommodate the contents of the file being imported. This will allow you to run an EXPORT command on the new table to dump the rows to a delimited format.
In case you still want write your own PC/IXF parser, you can take a look at this project, that converts IXF file to JSON
IXF is an old and also well documented file format. It is possible to read and process it, I've done this couple of years ago. So don't let you be discouraged. It's hard, but not too hard for developers.
Today you can find also solutions on GitHub, e.g. ixfcvt
I'm writing times into a CSV file time using the format HH:mm:ss. When excel opens the file it automatically recognizes this column format as time. Is it possible to prevent Excel from doing this so the column is formatted as regular text?
Thank you
There was recently a similar question on SuperUser: link
The same principle as the accepted answer can be employed here if all you want is a CSV that can be opened with Excel without the ill-effects of autoformatting. You'll need to write your values to the CSV in this format:
="yourdatetimehere"
Of course the downside is that the equal signs and quotation marks will be stored in your CSV as text. This means that this will probably cause problems for you if you plan to use the CSV in any context outside Excel. But as a hack to get around Excel's autoformatting, this should work.
You have no control over formatting in a CSV file, unless you want to go through the full custom-import setup in Excel each time.
If you want to force Excel to treat something as text, then use a proper Excel file, generated using PHPExcel