On a winform application,I've the following scenario:
A socket receives data with high frequency (150-200packets x second )
I've a grid view that is populated on the decoded packet received
the received item can be a new item or an update of a previous one..there're 5possible value that can be updated
the gridview is in virtualization mode : on
Since I can't bind to a BindingList (it will freeze the Ui thread) I've decided to use a Queue and to update the grid view based on a Timer. When the timer's elapsed event is processed I update the gridview calling the
gridview.TemplateMaster.Refresh() causing the cellValueNeeded to be called
This is the fastest solution I've found but I have some problems:
I can't know which column has been updated
Using the virtualization on cause me to call the cellValueNeeded each time and my List is almost useless, I can use it as a bi dimensional array
I've to custom implement sort/filter/column reorder
Anyone of you has got a better approach ?
Thanks
I would also recommend having a look at this article: High Performance with RadGridView and Virtual Mode including Filtering, Sorting and Grouping
Given the following additional information:
I've got 8 columns... the row number depends of which item is dragged
from a treeview... in this test case I've got 256 rows but can be more
...for the second point If I rebing the whole itemsource I loose what
has been updated
your issue is much more complex than your OP suggests. You actually have two sources of changes that modify the grid and the underlying data. One is from your incoming data packets and the other is the user. You also have 8*256=2048 items to show the user. That is a lot. And from my understanding of th OP the UI can potentially be updated with as much as 1,000 new values every second. This is not a useful feature since no human being can process information that quickly.
So, given the fact that the user can arbitrarily pull in data into the grid from a tree view and your packet receiver could add or update a thousand cells of grid data per second I would suggest a very different approach:
Keep the queue collection that holds incoming data
Add three new pieces of UI just above the grid:
A read-only control (simple label control would do) that displays the count of updates currently in the queue
A time-stamp of when the grid was last updated from the data packet queue
A button which the user would hit manually to pull in the queued data. Clicking the button would reset the time-stamp to the current time and the count of queued items to 0
You could also keep your timer to automatically update the data but make it smarter:
Extend the time period between automatic updates
Reset the timer whenever the user interacts with the UI at all (this would basically create a situation where the grid is automatically updated only when the user is "idle", or not interacting with the UI)
The columns being moved around is not a problem since you can find/and refer to columns by their column name rather than their index
Since the grid can be sorted arbitrarily by the user you need to use a find function to identify the row to update. You should have some sort of (primary/unique) key column, visible or hidden, by which to identify a row and the underlying object
All other functionality stays the same
Original answer for posterity:
I suggest creating a new collection which would be a shared resource between the UI and your socket thread (assuming you have a thread for the socket communication). You would update this collection as well as a flag time-stamp of the latest change. This flag is important because you would not change it unless data did actually change. You could also have an additional field which could count the number of updates since the latest change. (I'm assuming/hoping that much of this data coming in would not actually be different.)
So now you have an internal cache that can be shared by both threads. You don't have to worry about synchronization issues because one thread only writes and the other only reads.
Related
I have rapid incoming data (which are basically rows of a tabular data set). As i receive these rows, I merge/upsert them into the data cache. The data has to be shown on a WPF control (an item control).
The problem :
The data is not directly bound to the user control. A series of filter and grouping/aggregation (done using LINQ) is applied on the data before it is shown to the data. Thus what the user sees on the control can drastically change (say he changes the grouping, then all the rows will change).
This is what I am doing as of now :
As the data is coming in VERY rapidly, a thread is picking up the data every 2 seconds, applying the filter and then grouping the data and binding the data set to the wpf item control.
This is definitely not good as a new data table is being set as the datasource every 2 seconds. The application becomes laggy after some time.
What will be the best approach for me to solve this problem ? Thanks.
I'd say the best way to approach this would be to create a view source and bind the display element directly to that - that way you've got the grouping and filtering systems using the built in stuff.
I'd also then look into virtualisation and optimisation which will help with showing a lot of data.
As a more general note, I'd say explore whether you really need all that data to be displayed and at that frequency; it might be a better user experience (and easier on the machine) to aggregate the data and just surface pertinent details, that the user can then drill-down into further as required.
I have a form, "StartForm" where the user makes a combo box choice, and then clicks a button. From there I would like to:
Generate an instance of another form, "MainForm"
Populate the DataGridView on that form with around 50 000 rows
Loop through each row in the datagrid view and perform some processing on it. In my case, it's formatting the row colour based on the value of a cell, and updating some columns.
Display the form.
This is straight forward, however because of the large number of rows and the processing time on each one, the UI freezes when constructing the MainForm. So I need the MainForm to be constructed on a background thread, and a progress bar to fill up on the StartForm while this is going on.
I keep getting cross thread exceptions or my formatting is lost when using backgroundWorker, and control.Invoke() also seems to throw exceptions. Maybe I'm just not using these correctly...
Could someone please explain how to accomplish the above? I have tried to keep the question basic to understand so it can be helpful to others but if you would like my code then please ask.
Thanks!
I think you need to implement virtual mode for your data grid. Take a look on msdn article about implementing Virtual Mode in the DataGridView Control. Small sample:
grid.VirtualMode = true; // enable virtual mode
grid.RowCount = source.Count; // 50000
grid.CellValueNeeded += grid_CellValueNeeded;
grid.CellPainting += grid_CellPainting;
Use CellValueNeeded event handler to provide value for cell (i.e. select value from source). Use CellPainting event handler to set cell color based on cell value (use e.CellStyle property). If it takes long time to fill your source with data, you can do that in BackgroundWorker. But do formatting and displaying data in virtual mode - this is the best option, when you have huge amount of rows (btw consider apply some filtering - rarely users need 50000 rows of data at once).
By default, all the work is done on the UI thread which gives the illusion of 'crashing' since the UI becomes inaccessible during the heavy work. Creating a form itself on a thread that's not the UI thread is possible but would mean that the form is only ever accessible by the thread that created it, so it's unlikely that's what you want. You're getting the cross-thread exception because you're trying to modify the UI thread from another thread.
What you could try is to load and process all the data on a background thread, but load it into a new, temporary DataTable. Then, use DataSet.Merge on the UI thread (possibly in the RunWorkerCompleted event if you're using a BackgroundWorker) and merge the temporary DataTable with your main DataSet which is bound to the DataGridView.
That should mean all the processing and heavy work is done on a background thread, but the DataGridView control is updated on the UI thread, thus bypassing any invalid cross-threading. One side note is that you should test the performance of the DataSet.Merge.
I am building a monitoring application. Data is presented to a user in a girdview. This data changes frequently(an application adds 5-6 entries every second). Total number of entires loaded may be a few thousand.
What is the best way to load these rows into gridview?
Should I load data incrementally? And add each row manually?
Or
Would calling DataBind again be too expensive?
I have noticed a flicker on screen with this approach and so don't want to use this.
Are there any better methods to populate a gridview with incremental data?
I am building a monitoring application. Data is presented to a user in a grid view. This data changes frequently(an application adds 5-6 entries every second)
Can the human eye catch 5-6 updates a second? I think your good with one
update every 500 ms.
Performance tips:
Use BindingList or List. Stay away from datatable/dataset.
Maintain a cache of all the updates. Use a timer with 500 ms
interval, update the List datasource from the cache when interval elapses.
Refresh the CurrencyManager and do not rebind the grid.
CurrencyManager currencyManager = (CurrencyManager)this.BindingContext[Grid.DataSource];
currencyManager.Refresh();
I think loading data incrementally is much better option, and especial suits monitoring application.
I'm having an application, that periodically polls a web service (about every 10 seconds). In my application logic I'm having a List<Message> holding the messages. All messages have an id, and might be received out of order. Therefore the class implements the Comparable Interface.
What WinForm control would fit to be regurarly updated (with the items in order). I plan to hold the last 500 messages. Should I sort the list and then update the whole form? Or is data binding approriate (concerning performance)?
If you are working with WinForms, I shall recommend a ListView-control with View-property set to Details along with a BackgroundWorker-control.
If the list is going to be changing each time, and your order is not coming in consistently, I'd just sort your list and reupdate the entire UI.
You should be able to do the list fetching + sorting in a background thread, so the main "UI" blocking will just be redrawing the UI. With 500 elements, this won't be fast, but it should be tolerable.
i'd use databinding with the BindingList in System.ComponentModel wrapping your List.
BindingList in this case, would allow you to update your List object with new data and will automatically re-bind your data to the control that you are using for display.
which control you use would depend on how you want it to look. any control that supports databinding to a collection would work, though.
I have a SQL table containing currently 1 million rows that will grow over time.
There is a specific user requirement to present a sortable grid that displays all rows without paging. The user expects to be able to very quickly jump from row to row and top to bottom by using the scrollbar.
I am familiar with "virtual mode" grids that only present a visible subset of the overall data. They can provide excellent UI performance and minimal memory requirements, (I've even implemented an application using this technique many years ago).
The Windows Forms DataGridView provides a virtual mode that looks like it should be the answer. However unlike other virtual modes I've encountered, it still allocates memory for every row (confirmed in ProcessExplorer). Obviously this causes overall memory usage to needlessly greatly increase and, while allocating these rows, there is a noticeable delay. Scrolling performance also suffers on 1 million + rows.
A real virtual mode would have no need to allocate any memory for rows not being displayed. You just give it the total row count (eg 1,000,000) and all the grid does is scale the scrollbar accordingly. When it is first displayed the grid simply asks for data the first n (say 30) visible rows only, instantaneous display.
When the user scrolls the grid, a simple row offset and the number of visible rows are provided and can be used to retrieve data from the data store.
Here's an example of the DataGridView code I'm currently using:
public void AddVirtualRows(int rowCount)
{
dataGridList.ColumnCount = 4;
dataGridList.AutoSizeColumnsMode = DataGridViewAutoSizeColumnsMode.None;
dataGridList.AutoSizeRowsMode = DataGridViewAutoSizeRowsMode.None;
dataGridList.VirtualMode = true;
dataGridList.RowCount = rowCount;
dataGridList.CellValueNeeded += new DataGridViewCellValueEventHandler(dataGridList_CellValueNeeded);
}
void dataGridList_CellValueNeeded(object sender, DataGridViewCellValueEventArgs e)
{
e.Value = e.RowIndex;
}
Am I missing anything here, or is the "virtual" mode of the DataGridView not really virtual at all?
[Update]
It looks like the good old ListView implements exactly the sort of virtual mode I'm looking for. But unfortunately the ListView does not have the cell formatting capabilities of the DataGridView, so I can't use it.
For others that might be able to, I tested it with a four column ListView (in Detail mode), VirtualMode= True and VirtualListSize =100,000,000 rows.
The list is displayed immediately with the first 30 rows visible. I can then scroll rapidly to the bottom of the list with no delay. The memory usage is constant 10 MB at all times.
We just had a similar requirement to be able to display arbitrary, unindexed 1M+ row tables in our application with "very good" performance, using the stock DataGridView. At first I thought it wasn't possible, but with enough head scratching, we came up with something that works very well after spending days pouring over Reflector and .NET Profiler. This was difficult to do, but the results were well worth it.
The way we tackled this problem was by creating a class that implements ITypedList and IBindingList (you can call it LargeTableView, for example) to manage the asynchronous retrieval and caching of information from the database. We also created a single PropertyDescriptor-inheriting class (e.g. LargeTableColumnDescriptor) to retrieve data from each column.
When the DataGridView.DataSource property is set to a class implementing IBindingList, it goes into a pseudo-virtual mode that differs from regular VirtualMode, where as when each row is painted (such as when the user scrolls), the DataGridView accesses the indexer [] on the IBindingList and the respective GetValue methods on each column's PropertyDescriptor to retrieve the values as needed. The CellValueNeeded event is not raised. In our case, we access the database when the indexer is accessed, and then cache the value, so that subsequent re-paints don't hit the database.
I performed similar tests re: memory usage. The DataGridView does allocate an array that is the size of the list (i.e. 1M rows), however each item in the array initially references a single DataGridViewRow, so memory usage is acceptible. I am not sure if the behavior is the same when VirtualMode is true. We were able to eliminate scroll lag by immediately returning String.Empty in the GetValue method if the row is not cached, and then performing the database query asynchronously. When the async request is finished, you can raise the IBindingList.ListChanged event to signal to the DataGridView that it should repaint the cells, except this time reading from the cache which is readily available. That way, the UI is never blocked waiting for database calls.
One thing we noticed is that performance is significantly better if you set the DataSource or number of virtual rows before adding the DataGridView to the form - it cut initialization time in half. Also, make sure that you have both Row and Column autosizing set to None or else you will have additional performance problems.
Side note: the way we accomplished "loading" such a large table in our .NET application was by creating a temporary table on the SQL server that listed the primary keys in the desired sort order along with an IDENTITY (row number), and then persisting the connection for subsequent row requests. This naturally takes time to initialize (approx. 3-5s on a reasonably fast SQL server), but without knowledge of available indexes, we have no better alternative. Then, in our ITypedList implementation, we request rows in pages of 100 rows, where the 50th row is the row that is being painted, so that we limit the number of queries performed each time the indexer is accessed, and that we give the appearance of having all of the data available in our application.
Further reading:
http://msdn.microsoft.com/en-us/library/ms404298.aspx
http://msdn.microsoft.com/en-us/library/system.componentmodel.ibindinglist.aspx
The answer is NO
see first comment here
if anyone knows a better way please tell us
I would say yes... as long as you stick to the events triggered by the Virtual Mode behavior (e.g. CellValueNeeded) and that you take good care of clearing your hand built-buffer. I already displayed large amount of data, more than 1M without a fuss.
I'm a little bit curious about the implementation of Kevin McCormick using a DataSource based on ITypedList or any IList related interface implementations. I guess it's just yet another abstraction layer who makes the use of an internal and transparent buffer in addition to let the user or the developer powers the DataGridView with this, but still internally dealing with the native VirtualMode to display the information you have loaded in the buffer.
Besides of the fashion way to workaround the virtual mode, to me, the the only tough problem remaining with the DataGridView is the RowCount limitation: it's still a Int32.Max. It is probably due to the legacy of the Winforms Drawing System... just like the Images or even every Size Members, Width, Height of Winform Controls... why not stick to a UInt32 Type?
I assume that nobody has seen a Control or a picture with negative dimensions, but still the type is not properly appropriate to the context of use.
See my answer there below, could help you if you are still stuck with that trouble, event if, I guess it's already resolved for ages now.
https://stackoverflow.com/a/16373108/1906567