What is the best way to store your Data outside of SharePoint. I want to use the default Edit/View options in SharePoint (though I do plan on extending their functionality). I need to store the data outside of the SharePoint Lists as I am expecting a large record set(150,000 to start with).
I totally agree with GalacticJello, storing data outside of the regular sharepoint content database is at the moment (MOSS 2007) a complete and utter nightmare. MS provided a ExternalStorage provider baseclas for us to override, but there are major cons against using this:
writing and using your own custom ExternalStorage provider implementation is doable, but very difficult
ExternalStorage provider implementations only do just that, making sharepoint store stuff in a different location than the regular content db.
This means that you would need to write your own code to keep the external storage and the list items in sync, and i'm not even mentioning workflows and versioning.
last but not least (and IMHO the worst), creating and using a custom ExternalStorage provider is not web app or site collection targetable, it's usage is farm wide (and there are NO workarounds). So any site collection you create in that farm will have it's doc libs use the external storage provider.
You can store the items in the list, the trick is to create efficient views that return paged data quickly back to the user.
Another option is to use folders to split up the data.
If you really need to store it outside of SharePoint, I would consider waiting for SharePoint 2010 and thier "External Lists" features, as there are a ton of pitfalls and things to consider if you want to mimic that functionality in SharePoint 2007 (been there, done that).
You could create a SQL Data source and use the Data Form Web part to connect it to an edit form etc.:
Data Forms and SQL Server, Part 1
Data Forms and SQL Server, Part 2
Personally I prefer to have the level of control of a totally custom web part.. with this you will not run into limitations eventually.
In this case it sounds like all the elements will be in the same list, otherwise SLAM (free tool on CodePlex) is a very neat data replication tool for related lists.
Based on the comments above it sure looks like a plain ol' ASP.NET page might be the best option, unless there is some good reason why SharePoint must be used.
Related
As per requirement, we need to convert the existing MS Access database to a web application. Is there any easy way to convert the MS Access database to web application? As of now they are inserting the data to access db using access Forms. User also wish to continue access form feature even if we create new web application for the same. That means user should have the option to access the MS access database through Access forms as well as web application.
Please guide me away to solve this issue.
Best Regards,
Ranish
You can use Office 365 and have somewhat of a web-based application.
https://blogs.office.com/en-us/2012/07/30/get-started-with-access-2013-web-apps/
Or, store Access in SharePoint, but your functionality will be quite limited. Keep in mind, no VBA will run on a web-based application.
The alternative is to use SQL Server Express, and ASP.NET, both of which are free from Microsoft. I'll tell you now, though, the learning curve will be quite steep if you have never used these technologies before. This combo, however, will give you the most control!
You can get the .NET framework from here.
https://www.microsoft.com/en-us/download/details.aspx?id=30653
You can get SQL Server Express from here.
https://www.microsoft.com/en-US/download/details.aspx?id=42299
Four years after and according to this:
https://www.comparitech.com/net-admin/microsoft-access/
still a question for many. Access can be converted to an Web App in almost no time. Particularly Access Forms are super easy to crate with the library like Jam.py.
The process was discussed on Reddit in April 2021:
https://www.reddit.com/r/MSAccess/comments/mj4aya/moving_ms_access_to_web/
I see quite a few Access databases with more than 100 tables, all converted successfully to SQLite3. After inspecting the imported tables via provided link, Forms are automatically created. Which leaves the Access Reports and Business Logic untouched. Reports can be designed in LibreOffice for Jam.py as Templates. Business Logic can be moved from VB to Python, if there is a need to do so.
The SQLite was selected as the default DB for the conversion, since it is very portable. Looks like the converted App can be moved to any DB that Jam.py supports, by Export/Import.
Cheers
First of all, Database and Web Application are not mutually exclusive.
Back to original question, I have done multiple projects like that. A client started with small Microsoft Access database with a couple of user; then they migrate to to web application when they get more traffic.
At first, you want to convert data from MS Access Database to SQL Server. MS Access Database is not meant to access multiple users simultaneously. Then you develop the Web Application which uses SQL server as back end database.
Right before you go live, you convert the data again from MS Access Database to SQL Server very last time. Then do not let them use old MS Access Database anymore.
Easy way to convert the MS Access database to Web application
Most of the time whoever created MS Access database are not software engineer, so table are not normalized and do not have relationship at all. I normally create new normalized database in SQL Server. Then write a small program to convert those data from MS Access to SQL database.
There are generally two approaches with more details covered in this article looking at ways to convert microsoft access to web application
Direct Port means simply a basic migration whereby you port more or less verbatim basic Access forms into a web portal i.e. Microsft Access to a browser-based version as is using a third-party tool. Some of these are quite mundane in that it just allows you to run the Access application inside an internet browser (whoopee!) or can be quite drawn out and then limits you on how much you can change afterward. With even more complex cases requiring a consultant to help you migrate the system. Though it does help to know your user count as the higher you tend to be, the less appealing a third-party porting service becomes due to subscription-based models.
Upsize -the more involved or complex your data structure is an upsize using custom development and splitting the system across web and data tiers might be worth it if
You've got a special process or some secret sauce you're looking to keep.
Likely going to have a significant user count and want to avoid subscription
Inherently cynical or cautious, and want to handle your own architecture and security
Looking for a specific user experience
If you mean how to convert automatically and you want to keep both Access and the Web application (I don't recommend that, I would move everything to the Web app) I would do the following:
Export your Access data in CSV/Excel
Use a platform like DaDaBIK to import the CSV/Excel file and automatically create a Web app based on that file, with data stored on SQL server, MySQL, PostgreSQL or SQLite.
connect your Access to the SQL Server (or Mysql, ...) database created by DaDaBIK, from now on Access will only be used as a frontend.
Now you have a web app created with DaDaBIK and your Access frontend both working on the same DB. As I said I would skip 3) and keep only the Web app, this helps with handling data integrity when two users are accessing the same record.
Depending on how complex is your Access Application (e.g. complex validation rules or custom VB code you added), you could reach your goal without any coding or with some coding.
My WinForms C#/.NET application requires a table/grid control to display records to the end user. The records will be simple, containing only two fields, a string and a date/time field. I need to persist the data and I am wondering what the most efficient control and storage back-end is to use. The data is non-critical (i.e. - not health or financial records, or anything sensitive requiring extensive safety or any encryption).
One solution I have found so far is the DataGrid control in conjunction with SQL Server Compact Edition. I learned about this solution from this tutorial:
http://www.dotnetperls.com/datagridview-tutorial
It seems though that this may be overkill for my application. In addition, I am worried about the complexities of installing SQL Server CE, especially when it comes to admin vs. user account privilege issues during installation:
http://msdn.microsoft.com/en-us/library/aa983326(v=vs.80).aspx
Is there a table or grid control with built-in file load/save capabilities that uses a simple disk file as the storage method, perhaps a comma delimited ASCII file? I'd like something that I can still use SQL (via LINQ) to interface with. also, I am hoping that this can be done transparently. That is, if I want to upgrade to an SQL database engine solution later, the code from my end that interfaces with the data would not change (except perhaps for the database open/create code of course).
Or am I better off simply biting the bullet and going with SQL Server CE or perhaps SQLite:
Good embedded database solution (like SQLite) for .Net
If you have any caveats or anecdotes regarding installation issues and ease of use, they would be appreciated.
In my projects, we use Object datasources. Grid's can be bound to collections of objects just as easily as they can dataTables. You can store/restore the data using a simple serialization engine (XmlSerializer is rather easy to implement). Make a basic object, use List or BindingList as the dataset, and serialize/de-serialize it in the backEnd when you need it.
List and BindingList both support Linq queries.
Adding database save later is as simple as writing the code that saves the object to the database, in place of the serialization code, no change to the front end at all.
As far as a "Correct" solution is concerned...there are so many different ways to do it that it boils down to personal preference, and possibly actual requirements and expected future development. I find it easier to code using objects because the data manipulation is easier, but if you are going for straight record entry, no data manipulation required, going direct to a database is easier. It just depends on the data and what you plan on doing with it.
I strongly recommend you to use an embedded database, because it will be easier to go to a full database in a near future. SQL Server CE is a good option, and if you want to go big you can simply go to a full SQL Server Database with minimal changes in your code, the only downside of SQL Server CE is that you need to install it and it requires the .NET Framework 4, aside from that I don't see a big problem with it.
I have a C# application that allows one user to enter information about customers and job sites. The information is very basic.
Customer: Name, number, address, email, associated job site.
Job Site: Name, location.
Here are my specs I need for this program.
No limit on amount of data entered.
Single user per application. No concurrent activity or multiple users.
Allow user entries/data to be exported to an external file that can be easily shared between applications/users.
Allows for user queries to display customers based on different combinations of customer information/job site information.
The data will never be viewed or manipulated outside of the application.
The program will be running almost always, minimized to the task bar.
Startup time is not very important, however I would like the queries to be considerably fast.
This all seems to point me towards a database, but a very lightweight one. However I also need it to have no limitations as far as data storage. If you agree I should use a database, please let me know what would be best suited for my needs. If you don't think I should use a database, please make some other suggestions on what you think would be best.
My suggestion would be to use SQLite. You can find it here: http://sqlite.org/. And you can find the C# wrapper version here: http://sqlite.phxsoftware.com/
SQLite is very lightweight and has some pretty powerful stuff for such a lightweight engine. Another option you can look into is Microsoft Access.
You're asking the wrong question again :)
The better question is "how do I build an application that lets me change the data storage implementation?"
If you apply the repository pattern and properly interface it you can build interchangable persistence layers. So you could start with one implementation and change it as-needed wihtout needing to re-engineer the business or application layers.
Once you have a repository interface you could try implementations in a lot of differnt approaches:
Flat File - You could persist the data as XML, and provided that it's not a lot of data you could store the full contents in-memory (just read the file at startup, write the file at shutdown). With in-memory XML you can get very high throughput without concern for database indexes, etc.
Distributable DB - SQLite or SQL Compact work great; they offer many DB benefits, and require no installation
Local DB - SQL Express is a good middle-ground between a lightweight and full-featured DB. Access, when used carefully, can suffice. The main benefit is that it's included with MS Office (although not installed by default), and some IT groups are more comfortable having Access installed on machines than SQL Express.
Full DB - MySql, SQL Server, PostGreSQL, et al.
Given your specific requirements I would advise you towards an XML-based flat file--with the only condition being that you are OK with the memory-usage of the application directly correlating to the size of the file (since your data is text, even with the weight of XML, this would take a lot of entries to become very large).
Here's the pros/cons--listed by your requirements:
Cons
No limit on amount of data entered.
using in-memory XML would mean your application would not scale. It could easily handle a 10MB data-file, 100MB shouldn't be an issue (unless your system is low on RAM), above that you have to seriously question "can I afford this much memory?".
Pros
Single user per application. No concurrent activity or multiple users.
XML can be read into memory and held by the process (AppDomain, really). It's perfectly suited for single-user scenarios where concurrency is a very narrow concern.
Allow user entries/data to be exported to an external file that can be easily shared between applications/users.
XML is perfect for exporting, and also easy to import to Excel, databases, etc...
Allows for user queries to display customers based on different combinations of customer information/job site information.
Linq-to-XML is your friend :D
The data will never be viewed or manipulated outside of the application.
....then holding it entirely in-memory doesn't cause any issues
The program will be running almost always, minimized to the task bar.
so loading the XML at startup, and writing at shutdown will be acceptible (if the file is very large it could take a while)
Startup time is not very important, however I would like the queries to be considerably fast
Reading the XML would be relatively slow at startup; but when it's loaded in-memory it will be hard to beat. Any given DB will require that the DB engine be started, that interop/cross-process/cross-network calls be made, that the results be loaded from disk (if not cached by the engine), etc...
It sounds to me like a database is 100% what you need. It offers both the data storage, data retrieval (including queries) and the ability to export data to a standard format (either direct from the database, or through your application.)
For a light database, I suggest SQLite (pronounced 'SQL Lite' ;) ). You can google for tutorials on how to set it up, and then how to interface with it via your C# code. I also found a reference to this C# wrapper for SQLite, which may be able to do much of the work for you!
How about SQLite? It sounds like it is a good fit for your application.
You can use System.Data.SQLite as the .NET wrapper.
You can get SQL Server Express for free. I would say the question is not so much why should you use a database, more why shouldn't you? This type of problem is exactly what databases are for, and SQL Server is a very powerful and widely used database, so if you are going to go for some other solution you need to provide a good reason why you wouldn't go with a database.
A database would be a good fit. SQLite is good as others have mentioned.
You could also use a local instance of SQL Server Express to take advantage of improved integration with other pieces of the Microsoft development stack (since you mention C#).
A third option is a document database like Raven which may fit from the sounds of your data.
edit
A fourth option would be to try Lightswitch when the beta comes out in a few days. (8-23-2010)
/edit
There is always going to be a limitation on data storage (the empty space of the hard disk). According to wikipedia, SQL Express is limited to 10 GB for SQL Server Express 2008 R2
I never thought I'd ever say this but I'd like to have something like the report generator in Microsoft Access. Very simple, just list data from a SQL query.
I don't really care what language is used as long as I can get it done fast.
C#,C++,Python,Javascript...
I want to know the quickest (development sense) way to display data from a database.
edit :
I'm using MySQL with web interface for data input. I would be much better if the user had some kind of GUI.
Depends on the database -- with [sqlite][1], for example, ...:
$ sqlite3 databasefile 'select foo, bar from baz'
is all it takes (see the URL I pointed to for more options you can use, e.g. to change the output format, etc). Mysql has a similar command-line client (see e.g. here), so does PostgreSQL (see here), etc, etc.
So, what specific DB engine are you concerned with? Or, if more than one, which set?
Some suggestions:
1) ASP.NET Gridview
---use the free Visual Studio to create an asp.net page
...can do VB, C#, etc.
---drag/drop a gridview control on your page, then connect it to your data and display fields, all via wizard (you did say quick and dirty, correct?). No coding required if you can live within the wizard's limitations (which aren't too bad actually).
The type of database (mySQL or otherwise) isn't relevant.
Other quick and dirty approach might be Access itself -- it can create 'pages', I think, that are web publishable.
If you want to put a little more work into it, ASP.NET has some other great controls / layout capability (non-wizard derived).
Also, you could look at SSRS if you have access to it. More initial setup work, but has the option to let your users create their own reports in a semi-Access-like fashion. Web accessible.
Good luck.
I have an ASP.Net MVC web app that includes a set of Forums. In order to maintain flexible security, I have chosen an access-control-list style of security.
However, this is getting to be a pretty heavy chunk of data to retrieve every time somebody views the forum index.
I am using the EnterpriseLibrary.Caching functionality to cache various non-LINQ items on the site. (The BBCode interpreter, the Skins, and etc.)
My question is this:
What is the safest and most elegant way to cache a LINQ result?
Essentially, I would like to keep a copy of the ACL for each forum in memory to prevent the database hit. That way, for each person that hits the site, at most I would have to fetch group membership information.
All-in-all I'm really looking for a way to cache large abouts of LINQ data effectively, not just these specific rows.
If you've already got a caching system for general objects, all you should need is this:
var whatever = linkQuery.ToList();