I have 5 tables in database Table1, Table2 and so on [All tables have same column name or Table Definition]. I am using Entity Framework in MVC application and C#.
First creating an object of db of Database.
Getting table data as db.Table1.ToList();.
I want to do some thing like this.
list<string> TableNames = new list<string>();
db.TableNames[1].ToList();
Now I know this won't work but is there any way I can get data without hard coding the table names as my Project will deal with 100s of tables with same column names but different data.
This is a Project for a Hospital which will receive data from different locations. Lets say for location A I am expecting 100 cases a day and right now I have 10 locations. So if I combine all this data into one which means 1000 records each day in a single day therefore overtime searching through this table will become performance sensitive.
I am writing this for those who might occur into this same dilemma.....
I had reference a table through EF so the classes got generated into the Model.
Lets say I have 3 tables of same schema tbl_Loc1, tbl_Loc2 and tblLoc3.
public void getDataFromTable(string TableName)
{
using(var ctx = new DBEntities())
{
string query ="Select * from " +TableName;
var data=ctx.tbl_Loc1.SqlQuery(query);
}
}
DBEntities is Database Connection String
In ctx.tbl_Loc1.SqlQuery(query);.............. tbl_loc1 has a class in model which will help in getting data in the same format[As all tables have the same table definition]
There is a model of tbl_Loc1 in EF whereas tbl_Loc2 and tbl_Loc3 are only in Database.
Return this data as IEnumerable list
http://www.entityframeworktutorial.net/Querying-with-EDM.aspx
I echo other commenter's thoughts that you probably can handle this all in one table with a distinguishing column (and some proper indexes on the table). What you've mentioned so far only amounts to hundreds of thousands of records, something that should still perform very well.
However, in order to do what you want the way you state it, you can use reflection to examine the properties of your db object. Any property in there that is a hashset is a property that represents a table, so you can get a list of all the hashset properties, and their names (perhaps with a few tweaks regarding pluralization), which will give you your table names.
For a more sophisticated use of metadata within EF, take a look at How I can read EF DbContext metadata programmatically?.
Also, you may find that SMO is a helpful approach to this kind of thing (nothing preventing you from using it and EF).
Related
I'm a little new to Linq (C#) so I could use some help. I have a list of objects that is a bunch of records that I have pulled from a SQL Server table.
I need to query a second table from a different database that has additional related info for the records contained in the list of objects taken from the first table and I'm not exactly sure how to do that.
I have an object called ReportData that I then need to take certain properties from that first collection and certain related properties from the List from the second table and store it into the ReportData objects which will then be used to populate a .csv file.
I've got the population of the .csv file done, it's just the rest of it that I'm not entirely sure how to tackle.
At the risk of over-explaining my question, I'm going to err on the side of too much information.
I am creating a bulk upload process that inserts data into two tables. The two tables look roughly as follows. TableA is a self-referencing table that allows N levels of reference.
Parts (self-referencing table)
--------
PartId (PK Int Non-Auto-Incrementing)
DescriptionId (Fk)
ParentPartId
HierarchyNode (HierarchyId)
SourcePartId (VARCHAR(500) a unique Part Id from the source)
(other columns)
Description
--------
DescriptionId (PK Int Non-Auto-Incrementing)
Language (PK either 'EN' or 'JA')
DescriptionText (varchar(max))
(I should note too that there are other tables that will reference our PartID that I'm leaving out of this for now.)
In Description, the combo of Description and Language will be unique, but the actual `DescriptionID will always have at least two instances.
Now, for the bulk upload process, I created two staging tables that look a lot like Parts and Description but don't have any PK's, Indexes, etc. They are Parts_Staging and Description_Staging.
In Parts_Staging there is an extra column that contains a Hierarchy Node String, which is the HierarchyNode in this kind of format: /1/2/3/ etc. Then when data is copied from the _Staging table to the actual table, I use a CAST(Source.Column AS hierarchyid).
Because of the complexity of the ID's shared across the two tables, the self-referencing id's and the hierarchyid in Parts, and the number of rows to be inserted (possible in the 100,000's) I decided to 100% compile ALL of the data in a C# model first, including the PK ID's. So the process looks like this in C#:
Query the two tables for MAX ID
Using those Max ID's, compile a complete model of all the data for both tables (inlcuding the hierarchyid /1/2/3/)
Do a bulk insert into both _Staging Tables
Trigger a SP that copies non-duplicate data from the two _Staging tables into the actual tables. (This is where the CAST(Source.Column AS hierarchyid) happens).
We are importing lots of parts books, and a single part may be replicated across multiple books. We need to remove the duplicates. In step 4, duplicates are weeded out by checking the SourcePartId in the Parts table and the Description in the DescriptionText in the Description table.
That entire process works beautifully! And best of all, it's really fast. But, if you are reading this carefully (and I thank if you are) then you have already noticed one glaring, obvious problem.
If multiple processes are happening at the same time (and that absolutely WILL happen!) then there is a very real risk of getting the ID's mixed up and the data becoming really corrupted. Process1 could do the GET MAX ID query and before it manages to finish, Process2 could also do a GET MAX ID query, and because Process1 hasn't actually written to the tables yet, it would get the same ID's.
My original thought was to use a SEQUENCE object. And at first, that plan seemed to be brilliant. But it fell apart in testing because it's entirely possible that the same data will be processed more than once and eventually ignored when the copy happens from the _Staging tables to the final tables. And in that case, the SEQUENCE numbers will already be claimed and used, resulting in giant gaps in the ID's. Not that this is a fatal flaw, but it's an issue we would rather avoid.
So... that was a LOT of background info to ask this actual question. What I'm thinking of doing is this:
Lock both of the tables in question
Steps 1-4 as outlined above
Unlock both of the tables.
The lock would need to be a READ lock (which I think is an Exclusive lock?) so that if another process attempts to do the GET MAX ID query, it will have to wait.
My question is: 1) Is this the best approach? And 2) How does one place an Exclusive lock on a table?
Thanks!
I'm not sure in regards to what's the best approach but in terms of placing an 'exclusive' lock on a table, simply using with (TABLOCKX) in your query will put one on the table.
If you wish to learn about it;
https://msdn.microsoft.com/en-GB/library/ms187373.aspx
This past week I was tasked with moving a PHP based database to a new SQL database. There are a handful of requirements, but one of those was using ASP.Net MVC to connect to the SQL database...and I have never used ASP.Net or MVC.
I have successfully moved the database to SQL and have the foundation of the ASP site set up (after spending many hours pouring through tutorials). The issue I am having now is that one of the pages is meant to display a handful of fields (User_Name, Work_Date, Work_Description, Work_Location, etc) but the only way of grabbing all of those fields is by combining two of the tables. Furthermore, I am required to allow the user to search the combined table for any matching rows between a user inputted date range.
I have tried having a basic table set up that displays the correct fields and have implemented a search bar...but that only allows me to search by a single date, not a range. I have also tried to use GridView with its Query Builder feature to grab the data fields I needed (which worked really well), but I can't figure out how to attach textboxes/buttons to the newly made GridView. Using a single table with GridView works perfectly and using textboxes/buttons is very intuitive. I just can't seem to make the same connection with a joined view.
So I suppose my question is this: what is the best way for me to combine these two tables while also still having the ability to perform searches on the displayed data? If I could build this database from scratch I would have just made a table with the relevant data attached to it, but because this is derived from a previously made database it has 12+ years of information that I need to dump into it.
Any help would be greatly appreciated. I am kind of dead in the water here. My inexperience with these systems is getting the better of me. I could post the code that I have, but I am mainly interested in my options and then I can do the research on my own.
Thanks!
It's difficult to offer definitive answers to your questions due to the need for guesswork.
But here are some hints.
You can say WHERE datestamp >= '2017-01-01' AND datestamp < '2018-01-01' to filter all the rows in calendar year 2017. Many variations on this sort of date range filter are available.
Your first table probably has some kind of ID number on each row. Let's call it first.first_id. Your second table probably has its own id, let's call it second.second_id. And, it probably has another id that identifies a row in your first table, let's call it second.first_id. That second.first_id is called a foreign key in the second table to the first table. There can be any number of rows in your second table corresponding to your first table via this foreign key.
If this is the case you can do something like this:
SELECT first.datestamp, first.val1, first.val2, second.val1, second.val2
FROM first
JOIN second ON first.first_id = second.first_id
WHERE first.datestamp >= '2018-06-01' AND first.datestamp < '2018-07-01'
AND (first.val1 = 'some search term' OR second.val1 = 'some search term')
ORDER BY first.datestamp
This makes a virtual table by joining together your two physical tables (FROM...JOIN...).
Then it filters the rows you want from that virtual table (FROM ...).
Then it puts them in the order you want (ORDER BY...).
Finally, it chooses the columns from the virtual table you want in your result set (SELECT ...).
SQL database servers (MySQL, SQL Server, postgreSQL, Oracle and the rest) are very smart about doing this sort of thing efficiently.
I have been using linq to sql for a little while and often come up against this type of problem....
e.g
I have 2 db tables
-Table: Invoice ("Id" int auto-increment, "InvoiceDate" datetime)
-Table: InvoiceItems ("Id" int auto-increment, "InvoiceId" int (FK), "SomeReference" varchar(50))
The "SomeReference" field holds a value that is a combination of the Id from the parent Invoice record and some random characters. eg. "145AHTL"
Before i can set the value of SomeReference I need to know the value of the Invoice Id, but this only gets populated when it is saved to the DB. I have both parent and child records in the same Linq to SQl DB Context but I only want to perform "SubmitChanges" to the parent Invoice record only, so that i can then populate the SomeReference in the child record. I dont want to have the child InvoiceItem record saved to the DB before SomeReference is set.
How can I achieve this using Linq to Sql?
I understand that linq to sql uses the "Unit of Work" idea for saving to db, but I dont understand how I can avoid unnecessarily saving records to the db when they are not ready to be saved just yet. If there is no way around this, then why do developers bother with linq to sql, as this seems like such a huge drawback?
edit: should note that this example is just something i came up with to help describe my problem.
You can not. Not this way. And this is the only way (linq dues not support sequences). Brutally speaking - you have to fix your logic. The Id of an invoice is not a refernce field. It should not ever never be the number. This is a logical field and should be handled by your logic, outside the Id.
You example can be done, but you need to forget about the SQL and the database, but think in an ORM way.
Two issues need to be addressed in your example
First inserting the master and detail at the same time
Pseudo code for how it works:
using (var dc = new datacontext())
var master = new masterentity;
master.somedata = "data";
dc.tb_master.InsertOnSumbut(master)
var detail = new detailentity
detail.tb_master = master
dc.tb_detail.InsertOnSubmit(detail)
Submitchanges()
So you assign the entities to eachother, not the keys.
Second: the SomeReference
This first part however, does not give you the somereference field, only sets the the foreign key properly.
Your somereference field contains redundant data (not necessary) so that needs to be solved.
The somereference is a string + the ID.
So you store the string part in a column in the database (and only that) and you implement a custom property somereference by using a partial class.
public partial class tb_detail
{
public string somereference
{
get
{
return _id.ToString() + _somestring;
}}}
I'm creating a data-entry application where users are allowed to create the entry schema.
My first version of this just created a single table per entry schema with each entry spanning a single or multiple columns (for complex types) with the appropriate data type. This allowed for "fast" querying (on small datasets as I didn't index all columns) and simple synchronization where the data-entry was distributed on several databases.
I'm not quite happy with this solution though; the only positive thing is the simplicity...
I can only store a fixed number of columns. I need to create indexes on all columns. I need to recreate the table on schema changes.
Some of my key design criterias are:
Very fast querying (Using a simple domain specific query language)
Writes doesn't have to be fast
Many concurrent users
Schemas will change often
Schemas might contain many thousand columns
The data-entries might be distributed and needs syncronization.
Preferable MySQL and SQLite - Databases like DB2 and Oracle is out of the question.
Using .Net/Mono
I've been thinking of a couple of possible designs, but none of them seems like a good choice.
Solution 1: Union like table containing a Type column and one nullable column per type.
This avoids joins, but will definitly use a lot of space.
Solution 2: Key/value store. All values are stored as string and converted when needed.
Also use a lot of space, and of course, I hate having to convert everything to string.
Solution 3: Use an xml database or store values as xml.
Without any experience I would think this is quite slow (at least for the relational model unless there is some very good xpath support).
I also would like to avoid an xml database as other parts of the application fits better as a relational model, and being able to join the data is helpful.
I cannot help to think that someone has solved (some of) this already, but I'm unable to find anything. Not quite sure what to search for either...
I know market research is doing something like this for their questionnaires, but there are few open source implementations, and the ones I've found doesn't quite fit the bill.
PSPP has much of the logic I'm thinking of; primitive column types, many columns, many rows, fast querying and merging. Too bad it doesn't work against a database.. And of course... I don't need 99% of the provided functionality, but a lot of stuff not included.
I'm not sure this is the right place to ask such a design related question, but I hope someone here has some tips, know of any existing work, or can point me to a better place to ask such a question.
Thanks in advance!
Have you already considered the most trivial solution: having one table for each of your datatypes and storing the schema of your dataset in the database as well. Most simple solution:
DATASET Table (Virtual "table")
ID - primary key
Name - Name for the dataset/table
COLUMNSCHEMA Table (specifies the columns for one "dataset")
DATASETID - int (reference to Dataset-table)
COLID - smallint (unique # of the column)
Name - varchar
DataType - ("varchar", "int", whatever)
Row Table
DATASETID
ID - Unique id for the "row"
ColumnData Table (one for each datatype)
ROWID - int (reference to Row-table)
COLID - smallint
DATA - (varchar/int/whatever)
To query a dataset (a virtual table), you must then dynamically construct a SQL statement using the schema information in COLUMNSCHEMA table.