Create table with > 65.000 rows efficiently from Excel file in WPF - c#

I am currently working on a feature in my project that should fulfill the following function:
- the user can browse for an excel file (with as much as 65.000 rows)
- my tool should then somehow (not so important how as long as it's more or less efficient) populate a datagrid with that data.
- the tool should then modify the table in several ways and at last the user can select several rows that will be stored to a database.
I have already tried:
- using Interop, but going through all the rows in the Excel file and displaying them in the datagrid alone took more than 10 minutes
- creating a SSIS package in SQL Server Management Studio, which was much faster (just a few seconds). However, I haven't found a way to use that in my WPF application. Plus I'd need to add at least one column via code-behind (again, I don't know how to do that) which contains a reference ID.
I know my 'question' is very vague, but I really don't know what I should look for. I hope some of you have already done something similar and can tell me how they did it or how they would do it.
Cheers

I have found a way.
I converted the Excel file into cvs, used a StreamReader and read the lines into a DataTable as plain text.
Then I went over that DataTable, created Objects that I then added to an Observable Collection.
It takes less than 3 seconds :)
Thanks for the comments

Related

How do I save and load data from a DataGridView?

I have been looking for a while now for a way to do this but nothing seems to be what I am looking for, but all I want is to enter in some data into a DataGridView, save it and then if I want, later load it back up. All the tutorial seem to focus on log in data from some database or something like that when all I want is a grid where I can put things in and add a search filter tool so I can look things up. It is for a project I am doing and I was wanting a way to save and organise data efficiently like how you can as if it was a text document but in a grid with a search filter as well as the way you can send file over to someone else who has the application and read the data. Sorry if I have put it in a confusing way.
I am doing this in Visual Studio 2019 in C#
You could use this one as a staring point. Although such approaches seem to be outdated. (Using datatables etc) Your best bet is using classes for serializing the data to json and sending the json file

How many rows a normal C# window form application can load easily in datagridview

I am using datagridview for data rendering in window form application.
application is actually fetching a lot of rows and the system gets stuck and goes to not responding state. it is normally getting 100K + rows.
How many rows a normal C# window form application can load easily in datagridview
I'm not aware of any hard limitations. This would be based on factors like the amount of resources available to the application. If you're rendering high amounts of data, you may want to consider using virutal mode.
https://learn.microsoft.com/en-us/dotnet/framework/winforms/controls/implementing-virtual-mode-wf-datagridview-control
I don't think its good idea to first of fetch 100k+ rows and show all on grid, use can use paging that will be quicker and more efficient. You can control how many records you want to fetch at once based on traffic and performance of your database.
#pageNumber AS INT,
#rowsPerPage AS INT,
OFFSET((#PageNumber -1) * #RowsPerPage) ROWS
FETCH NEXT #RowsPerPage ROWS ONLY;
GO
Read this article :
https://10tec.com/articles/why-datagridview-slow.aspx
I have analysed the application and found out that there is nothing wrong with the datagridview. it can contain 100K rows if required.
there are some other functions that executes once the data loads which takes lot of time and puts the application in not responding state.
Virtual mode and Paging are also good options.
And I agree with your points that 100K rows are too much for a human to process. thats just the wastage of resources and effort. I understand that. I am actually trying to improve someone else's application.
Thanks guys !

Using a local database

I'm busy writing a small app and I only want a database to load small amounts of data, in other words I don't need the functionality of SQL.
So I've installed filehelpers, but it seems very limited in the sense that I can read/write and even append data, but it seem impossible to delete one row of data in a table?
Does anyone know how to do this with filehelpers or point me to a different solution where I can just add a local db to my app without any other external software required?
PS. My visual studio does not have the "create local db" from the item selection.
For something like this, I'd use an embedded SQLite database. It gives you the best of both worlds, one file database for local data and most of the features of SQL.
See here: https://sqlite.org/
On their download page, they have lots of stuff and a plugin for VS:
https://sqlite.org/download.html

dynamic column to sql table via .net OR dynamic report field

Scenario: Client maintains financial/compliance record in a spreadsheet for each quarter of the year. Spreadsheet contains columns which are not static, they can change from quarter to quarter and will not be same the next year. They want a portal by which all the regional managers (of stores/franchise) can enter their data and at the end aggregated at the national level.
Issue: As you could have guessed, I want to develop the spreadsheet column into sql table with all the columns, but the issue is that they want to add new columns dynamically (for future quarters) via admin side. Thinking of providing a textbox (columnname), dropdown (sqltypes possible) and add button which will basically add the column. But the columns can grow and that's not the right option, I guess. Other alternative is, instead of adding columns, I can add it as rows and then use PIVOT to do the sql part.
If anyone of you have developed this kind of application, could you please aware me of any complications before I proceed further with my idea of adding rows instead of columns for adding fields for the reports dynamically. If anyone has got sample example or reference online, please divert me there.
Take a look at Umbraco. It's a cms that puts the concept of PIVOT into practice. (Actually I believe most CMS does).
Actually, I'd go so far as to suggest building that application of yours in Umbraco. You will need to customise it a bit, but you will have quite a bit of heavy lifting already done for you, such as authorisation, membership, and the schema/content mechanism.

How do I build a c# treeview from SQL Hierarchyid?

I'm developing an C# .NET Windows Form application that 'll manage a hierarchical tree-view structure. (I have a single SQL table with a hierarchyid column. The data set is small ~300 rows/nodes at the moment, but is bound to grow large eventually)
After I exit the application, I want to be able to restart it with the last modified state. I understand I can do this by serializing it to XML.
However, I'm trying to find a method by which the treeview could be generated directly from the table at run time, using the hierarchy id column.
After Googling around, I've found some others who wish to the same, but unfortunately I've found no solutions. Is there a good way to do this?
Thanks.
Look for IHierarchicalEnumerable, HierarchicalDataSourceControl and HierarchicalDataSourceView. Thats exactly what are you looking for.

Categories

Resources