I have two situations in this case:
I want to query a WCF service and hold the data somewhere, because one of the web pages renders based on the data that's retrieved from the service. I don't want the page itself querying the service, but I'd rather have some sort of scheduled worker that runs once every a couple of minutes, and retrieves the data and holds it somewhere.
Where should I cache the service response, and what is the correct way to create the task to query the service every couple minutes?
I think I could achieve this by saving the response to a static variable, alongside the last query date, and then check on the page load if enough time has passed, I call the service and refresh the data, else I use the static cache.
This would also account for the case where no users access the page for a long time, and the site not futilely querying the service.
But it seems kind of rough, are there other, better ways to accomplish this kind of task?
You could indeed take another approach like having a scheduled program query the information and put it in an in-memory cache available to all the web servers in your farm. However, whether that would be better for your scenario depends on the size of your app and how much time/effort you want to spend on it.
An in-memory cache is harder to implement/support than a static variable but it's sometimes better since static variables can be cleared up every time the server resets (e.g. after X number of minutes of inactivity)
Depending on the size of your system I would start with the static variable, test drive the approach for a while and then decide if you need something more sophisticated.
Have you taken a look at Velocity
Nico: Why don't you write a simple console daemon that gets the data and stores it on your end in a database and then have your web app get the data from your local copy? You can make that console app run every certain amount of time. Inserting the data should not be a problem if you are using sql server 2008. You can pass datatable parameters to a stored proc and insert a whole table in one call. If you don't use Sql Server 2008, then serialize the whole collection returned by the web service and store in a table in one big blob column and record the timestamp when you got the data. You then can read the content of that column, deserealize your collection and reconstruct it to native objects for displaying on your page.
I've never seen (and I don't think its possible) to have your web app query the web service every certain amount of time. Imagine the web site is idle for hours therefore no interaction from anybody. That means that no events will fire and nothing will be queried.
Alternatively, you could create a dummy page executing a javascript function at certain intervals and have that javascript function make an ajax request to the server to get the data from the web service and cache the data. The problem is that the minute you walk out of that page, nothing will happen and you'll stop querying the web service. I think this is silly.
Related
Is there a strategy for caching data in a console application that is maintained after the console application is terminated and starts up again?
For example, when my console application starts up, 4 calls are made to my database which returns a fair amount of data. The rest of the application runs and uses these lists. When the console application starts up again at the next scheduled interval it will have to retrieve these four lists again. Is there a way to have those lists cached for a certain amount of time to reduce the amount of times I have to call the database?
My current set up is a Powershell script that just pings a URL on my website which obviously can cache these 4 lists and maintain them. However I think I need to move this function into console applications to remove the load from the IIS process as I've had some high CPU spikes on my server and Im assuming its to do with this code.
One idea I had was to give an API endpoint for these four lists in my website (so they can be cached) and call that from my console application. Is that the best way to handle this or is there a proper way of caching data and maintaining it after a console application has ended and started up again?
You could use a local file and store the values. Maybe in conjunction with a database or endpoint, adding some expire date to a tag in the file.
A local file access will be much faster then accessing a database or a any remote call. A remote call, lets say from a database or IIS endpoint could be used for your first load time.
I need some suggestions from the community to a requirement I have. Below is the requirement and I need some approach suggestions.
Users from the client need to retrieve data from my source database (Let say SQL database in my production server). The users access the data by a intermediary service layer (WCF Rest service). On another server (Info Server) I have a SQL Database (Info DB) which will hold all queries that can be requested. Since in some cases my data is huge, I give the option to user to schedule the data retrieval and look at the data later. The schedule information per user would also be stored in the Info DB. I also allow user to retrieve data real time in case he wants.
In both cases I want to Query data from Source (Production DB), store them in file format (May be CSV or excel) and then when user wants the data I would send the data over to the client.
Since the queries are stored in InfoDB. I let the admin define schedule run time for the every Query. This is to enable Admin to adjust long running queries run at night time when calls to server is low. In case the user demands a query to be run at real time, I would allow that.
As a solution architecture I have thought of this :
I will have a WCF rest service which would be installed on Info Server. This service will act as calling point for the Users. When user calls a query real time, the service will get the results, save to a file format and transfer it over. If the user schedules the query, the service will add an entry for the user/ for the query in the info database.
I will have a Windows Service on the Info Server. This Windows Service will periodically check the Info DB for Scheduled Queries entries and if the queries fall within the scheduled time , it will start running the query, get the data and save that to a file location and add the file location entry to the Schedule entry. This will enable me to track which schedules are finished and where the data is available (File path).
Now here are my issues with this:
My data can be huge, will a WCF rest service be good enough to transfer large files over the wire ? Can I transfer files over wire or I need to transfer data as JSON ? What is the best approach.
If I use a windows service, is this a good approach or is there a better alternative ? The reason I am asking is because as per my understanding Windows Service will have to run always , because I need to figure out the entries which are scheduled. This means at specific interval the Windows Service would check the info database and see if the schedule entry should be run or not. In ideal scenario the windows service will run through out the day and check the database periodically without much action because preferably all schedules would be at night time.
I have used an intermediary service approach because if I need to move to cloud tomorrow, I can easily move this solution. Am I right in my assumption ?
If I move to cloud tomorrow, would I be bale to encrypt the data transfer (may be data encryption or file encryption). I have no idea on data encryption/decryption.
Need your suggestion(s) to this.
My data can be huge, will a WCF rest service be good enough to
transfer large files over the wire ? Can I transfer files over wire or
I need to transfer data as JSON ? What is the best approach.
When you say huge, how huge? Are we talking gigabytes, megabytes, or kilobytes. I regularly have 100mb rest responses (you will probably have to tweak some things, to increase your MaxMessageLength, but this should be enough to get you going. I would take their advice and use a streaming API though, especially if you are talking several megs of content.
If I use a windows service, is this a good approach or is there a better alternative ? The reason I am asking is because as per my understanding Windows Service will have to run always , because I need to figure out the entries which are scheduled. This means at specific interval the Windows Service would check the info database and see if the schedule entry should be run or not. In ideal scenario the windows service will run through out the day and check the database periodically without much action because preferably all schedules would be at night time.
Beware writing your own scheduler. You might be better off dropping things onto a queue for processing, then just firing up workers at the appropriate time. That way you can just invoke the worker directly for your realtime call. Plus you can run it whenever the database is idle, not on a scheduled basis. It's tricky "knowing" when a service will be idle. Especially in a world of round-the-clock users.
I have used an intermediary service approach because if I need to move to cloud tomorrow, I can easily move this solution. Am I right in my assumption ?
Yes, wrapping an endpoint in a rest service (WCF) will make moving to the cloud much easier.
If I move to cloud tomorrow, would I be bale to encrypt the data transfer (may be data encryption or file encryption). I have no idea on data encryption/decryption.
HTTPS is your friend here. Read this. Don't invent your own here, or use proprietary encryption. HTTPS is old, straightforward and good.
I have coded a C# Web API 2 web service, and am wanting to retrieve this data in an android application that I have coded.
I wish to have a local SQL database on my android device that stores the data from the web service.
Whenever data is to be used in my android application, I wish to check to see if the local database data is the most up to date, and if it isn't, then I wish to retrieve the latest data from the web service.
I am thinking of the following:
Each time data is saved into the local database, save an integer value as an update number.
Whenever data is required, retrieve the latest update number from the local database, send a request to the web service with this update number, and only return data if the web service data has a newer update number.
Would this work well? Is there a better approach to this problem than my above idea? Are there any disadvantages or errors that could occur?
Thanks in advance.
I did exactly same thing with my synchronization algorithm. It's like this.
My web service methods gets datetime parameters, web service sends objects only created or modified after that time.
On the mobile side, I keep synchronization start time, and for the each synchronization, i send previous sync start time. (By the way, I ask date to server just before sync starts)
I dont keep sync end time, because some of the data may be modified after sync started.
I have done similar thing in my sync adapter where user click on sync and it get data from web feeds and store only updated data into sqlite.
refer to SyncAdapter.java file of sync adapter sample provided on http://developer.android.com/training/sync-adapters/creating-sync-adapter.html
I had To move a pre built asp website from single worker process environment to multiple worker process environment on cloud servers.
I was having a class with static arraylist variable which use to contain last 2 minutes of all session information for tracking purpose. The admin use to access this arraylist to view live reports. But on moving it to cloud infrastructure this has breaked down results are no longer correct. It depends on which server behind load balancer is serving the pages thus we have multiple instances of the static variable per app pool. I tried to move to mysql but we needed to flush out data regularly and it was also having performance issue. Here the arraylist is processed heavily to churn out useful data thus I need something which is inmemory.
Please note that before also the use of static variable without lock was the downside but that only led to difference between 1 or 2 records but was blazing fast.
You can consider backing your session by SQL server based session storage.
Alternatively you can use a application caching server to back it. That will let you share it across multiple web servers.
We have an existing big application which contains a lot of data. We'd like to use it as a datasource for various internally written C# web applications, so we don't have more redundant data.
The data we are looking at doesn't change too much, so caching would work fine most of the time. So we are writing a C# Web Service against the data to be reused in various internally written applications.
However roughly once per month, the Oracle database source is unavailable.
What is the best way to handle this in the web service so that those other applications that rely on that data aren't disrupted also?
Set up replication or failover partners? Honestly, this doesn't seem like a job for more code; it sounds like a job for more infrastructure. I know Oracle licenses are expensive, but so is paying developers to work around unavailability.
If you simply had to solve it with code, then the web services should simply retain and return their cached data if any regularly-scheduled DB query fails with a timeout or connection failed-type message. The cached data should be kept as long as necessary in this circumstance, until a call to refresh that data succeeds. If there is no cached data, you can either swallow the error and return nothing, or return an error stating the data is unavailable from both places.
The solution was to use a secondary Cache which doesn't expires.
The secondary cache is updated with the latest values if the first (shorter) cache is successfully updated from the database. If the database querying fails and the first cache has expired, then the first cache is updated by the second cache. So there is always a secondary cache.