I am writing a C# form based application that stores its data in SQL Server 2005.
My client wants to use this data in SPSS.
My original plan had been to create a database view which matches the SPSS structure. This could then be exported as a CSV file and copied into SPSS.
However, I just heard there may be a way to import data from a relational database directly within SPSS? I haven't been able to work out how to do this (I can see an export to db option, but not an import)
Does anybody know if data import from SQL Server is possible in SPSS? Or if there is an easier way to achieve this than by using the approach I outlined above?
Many thanks!
--- L
Just to add to this, you can import from a number of ODBC-compliant data sources. IMO, SPSS does this much easier than SAS, if you were comparing those two tools only. It is true that you have to save an ODBC connection up front, but after that it is a breeze!
In addition, with newer versions of SPSS (I know it was in Versions 18+), you can WRITE data back to a database. This is an unbelievebe feature and they make it very easy to do. We recently started leveraging this feature at work and it gets around needing to store multiple versions of the save SPSS file on a shared network drive, helps with reporting (can feed predictions) to a data warehouse that helps with reporting (i.e. Cognos), etc etc.
For any future searchers... I solved this by using the "Open Database" option under File menu, then selecting "New Query..."
I had to add appropriate ODBC drivers, then was able to create new joins between tables, and rename variables as required.
Easy once I stopped looking for something called 'import'! DOH!
Related
I have to design a system with an sql database whose work is to get data from different databases which may be in another database such as mysql or oracle etc. Then the system will map the attributes of that database with my database schema and store them..
Example reference link: https://msdn.microsoft.com/en-us/library/aa728893(v=vs.71).aspx
Since I am new, I can't attach images which is why I am providing links.
All my searches end up with getting the mapping tools but what actually I want is how to create that tool myself.
I am not a professional but a little push will be enough for me and highly appreciated. Thanks in advance.
This, as I said in the comment, sounds to me a job for an Integration Services package.
If you would like to use Microsoft SQL Server Integration Services you should first have Microsoft SQL Server Data Tools installed on your development machine.
Afterwards you start by creating a new Integration Service project inside Visual Studio. Then you can add an ODBC Connection Manager to manage your different databases input of data. After that you can add a different transformation container objects in your package to transform the data as you need it. At the end you need to specify the output of all those elements into your database where you want to store the information that you collect from other sources.
You can also create a different package for every source database you have so that the tasks can be separated. Unfortunately a complete tutorial is very long for me to post here but you can check out the tutorial on Microsoft web site. An other example here.
As as warning you should be really careful with data types because if you don't match/convert them correctly the package will fail with not so obvious errors.
If you choose the .tt(T4 Template) solution in which you create the application then you should start by connecting to the sources database and loop through tables definitions to get the columns and then store them as a xml file. The matching you will have to do it inside the text template file so that you have the matching already done when the table is read from the data source.
Here is an example that should get you started. Note that in the example the output file will be a .cs file not .xml but you can configure that very easily with this T4 directive <## output extension=".xml" #>.
Please, ignore for this task, any form of programming, Do not do it. Especially ignore Entity framework.
I would recommend you to take one of the many ETL tools out there, and design the work with it. I can recommend you Talend . It is very easy to learn and in time, even a developer starts to work with it.
The best part about it that it can connect to webservices, enterprise solutions and probably any imaginable Database out there.
What you do is to design Jobs that run on its own in parallel, and then you export the jobs as standalone JARs. A chroneJob, or a scheduled service completes the process to have it executed periodically
I'm currently developing app that requires storage of lots of information from different controls. I've never done this before, and after reading couple things on this topic got a little confused. Do I need to download SQL server 2008 and work with it in order to store data from WinForm? If yes, then what is the service-side item for? Can I use for storage?I don't need to import data from database(at least not for now), I just need stuff to save somewhere, and I would like to know where exactly. Thank you!
You can choose to store your data in a serverless database (SQL CE, SQLite for example) if you don't need fancy database stuff like stored procedures, weird indexes etc. Both of the above mentioned technologies are compatible with Entity Framework, hence you can use code first approach (you can write the classes and the database would be generated on the fly).
here you can read more about it.
http://www.codeproject.com/Articles/318010/Entity-Framework-Code-First-Let-s-Try-It
And here's SQLite assembly for .Net
http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki
Good luck
You don't have to store anything in a database, you could for example serialize whatever you need to store into isolated storage
I have read one line from your question " SQL server 2008 and work with it in order to store data from WinForm"; Mean you want to save some data and that will be coming from data entry from winform.. RIght?
If that is the case you can adopt any of the RDBMS software that suites your finance + capacity needs and other perameters.
You can surely go for SQL server 2008 or Oracle or MySql or (List will go one).
What is the nature of your data?
Just because it's big does not necessarily mean a full-blown DBMS (or even an embedded database) is the best fit. Have you considered simply serializing the data to plain files?
But if your data is "relational" and you need to do the things such as querying, integrity enforcement, transactions, concurrent access by multiple clients (etc..), then you should definitely consider a database approach, including MS SQL Server, Oracle, IBM DB2, Sybase etc...
There are good open-source DBMSes as well: PostgreSQL, Firebird, MySQL...
I have created an application that will be importing CSV files into a database table, and I've got multiple CSV files I need to import into a table in a SQL Server database.
I've got a couple approaches in mind but I'm not sure which is most practical. The application works by asking the user to select the files they want to import (from their local file system) and then they simply click a [Load Files] button. These files may contain 100,000+ rows at times.
What would be better for the above scenario?
Import CSV file into datatable using C# and open-source GenericParser then using a traditional method of BulkCopy to push the datatable to the database
Note: my concern is the strain on users PC when doing this for files with 100,000+ rows. How will this affect the processing or would it crash the program?
Use Bulk Insert which requires the file name and path. My concern for this option is I'm not sure if the server would be able to process the Bulk Insert command without the physical file being located on the server? The file path would relate to the users local machine. The only time I've used Bulk Insert is when I was logged onto the server itself, which is not possible for this app.
Is there a way to do it with Linq? While I'm not really familiar with Linq if it can be accomplished I'm open to trying it.
Any insight is appreciated. I know what I need to do just not sure of how to accomplish it practically.
Thanks
My recommendation would be to use the SqlBulkCopy class in .NET. It will allow you to import rows almost as fast at the BULK INSERT statement, but only requires that you populate a DataTable with the rows, and then send them to SQL Server.
Another consideration you might want to look at would be (and this is my personal favorite for simple file import programs) to use PowerShell instead of C#, which has a built-in cmdlet for imporing CSV files. Pretty cool stuff.
1) loader app in .Net is a good choice, generally. 100,000 rows is really not a strenuous workload, especially for simple loads. Only if there is a ton of multiple-table-joins involved in order to look up values on the fly would that really be a big concern.
2) although strictly speaking physical file location is just a performance question, I wouldn't do it. It will introduce administrative headaches.
3) I don't have experience with Linq, I cannot remark.
Just for bonus alternate idea: if you have IIS running somewhere, maybe even on the DB server, you can whip up a lightweight, one-page "webapp" which is just a CGI script with ODBC connection to the DB and the user just feeds the CSV in as a "web/CGI" upload. No utility application to install on user workstations this way.
To solve your problem, you have to see on in two basic views:
Do you need make some operations with data before insert in into database (some sumarization, correction,...)?
If yes, than here is the best way to upload rows from file to object (each row into one object instance). And than you can elegantly move with list of items with Linq.
Do you need only insert rows from file to database as they are?
I this case, use process described in point 2 of your question.
I'd prefer to upload file to server before any action. It's more safe.
In my application programming experience, I have always worked with a SQL Server (or Access) database on the back end that stores application data. I'm now looking at some business requirements that work with data that would fit well in a relational database, but they require it to be stored in a single, portable, custom file that the application will create, and load from. I know it's a very common concept for an application to save off a single file or document that it can later load and continue to work on, but I'm not sure how to achieve this with complex data. Encrypting xml comes to mind, but that would be very slow to work with or potentially eat up a lot of memory if I had to load it all back into objects first. What are some options?
I recommend you use a SQLLite or Firebird embedded database. There are other options as well. They support single-file usage and will give you a clear upgrade path for future versions of you schema (upgrade SQL scripts).
I did not understand how encryption plays into this.
When running in a .NET environment I think that SQL Server Compact is worth looking into. It is basically a mini SQL Server that doesn't have to be installed and configured as a service, but instead is an dll that you reference. You can use normal data access tools like linq-to-sql and entity framework.
SQLite comes to my mind. Its a single file based DB. Here is a link to convert SQL server databases to SQLite. Also check out Using SQLite in your C# Application
How do i create a db file in C#? a friend told me it was in the toolbox and not to use sqlite. I dont see anything that could be it, nor what it is called. google didnt help:(
Could it be...
SQL Server Compact Edition – A lightweight, in-process database engine designed to run on devices and desktops and is geared toward local data storage. Compact Edition includes a subset of SQL Server 2005 data types and shares common elements of the Transact-SQL (T-SQL) language with the data service engines.
There is no file-based database provider built in to c# or the .NET Framework. There are of course pre-existing connectors for using SQL Server (which includes SQL Express), but if you need a fully functional RDBMS that is file-based, you need to use something like SQLite or Firebird (also a fan of VistaDB, not free or open source but VERY solid and pretty affordable).
On the Add New Item menu: "Service-based Database" or "Sql Server Database" if it is an asp.net application. I am sure your friend meant it as "create a sql express db file in Visual Studio".
That said, if you wanted to fill an empty database, with tables that correspond to a c# model, you could create a linq2sql model, and use its CreateDatabase to do that for you :)
You might want to check this http://quickstarts.asp.net/QuickStartv20/aspnet/doc/data/vwd.aspx (visual web developer link, but it applies).
For a full/in-depth explanation of how SQL express can be used with a semi file based approach check and its limitations:
http://www.databasejournal.com/features/mssql/article.php/3704171/SQL-Server-2005-Express-Edition---Part-8---XCopy-Deployment.htm
Perhaps you might try Microsoft LocalDB.
It is file based but uses a low level SQLExpress Installation to host it.
If you are using Visual Studio or Web Developer Express, there are indeed ways to easily create a MS SQLExpress database. Just go to Add New Item... and it should be one of the available file types.
Keep in mind you have to have installed either Microsoft SQL Express Edition (free, as in beer!) or Microsoft SQL (very un-free!, in all senses). If you haven't done this, you don't get the option of creating a database file so easily. If you have got it yet, you can get it here.
As other answerers have mentioned, strictly speaking this is NOT a C# feature. MS SQL and it's derivatives, are database applications, much like Oracle, MySQL, or PostgreSQL. It's just that Microsoft Visual Studio makes using the Microsoft database product very easy by default. Differentiating between C#, Visual Studio, and any database programs will probably get you better answers, faster, no matter where you ask. :)
Every database has a file system in some binary format more than likely custom and uses a cache to control the flow of the database(s) lifetime.
If you create a database system, you will need some type of cache because you only want to read from the file if the cache has already released it.
If you have 1000 clients tapping into the same db, you certainly don't want to read/write to the file for each client request, so you want to manage a queue of clients and run it against the cache so that the cache knows not to release the db after its time span for lifetime is reached put to rather update the time span, therefore, not having to reload the file, if disposed and queued again, until the queue referencing the db object is empty.
Creating a well designed cache it used by all rdbms's so that duplicate objects are not created and files are not reloaded if not need be.
You can use
FileDB - A C# database to store files
http://filedb.codeplex.com
There is a MVC Example in the source that allows you to upload files and also has drag and drop support.
It then saves it into just one file in a location that you specified.
Like this:
private string pathDB = #"C:\CMS-MVC\Parts\FileManager\filedb-19055\trunk\MvcTest\Data\MvcData.dat";
That one file will store all of your files in that one "container".
You are mistaken. Databases are not developed in C#. Databases are built using a database system such as Oracle, MS SQL Server, MySQL, and numerous others.
Once you build a database using one of the above providers, you can then perform actions on the database using your programming language of choice (in your case C#) to get data out of and put data into it.