We are storing a serialized object in to Redis cache. We want to check the age of cache before retrieving new data and update the cache. If it is less than 10 mins, as data might not have changed in that duration, so we pull from cache and send to API output. if not, we will still return cached data to output, request new pull of data, update cache if new data is available and intimate the web listener to get latest update.
Is there a way with existing cache API, that we can check the age of Redis cache key? If no, any workarounds?
I think you're looking at it from the wrong perspective.
What would happen if you were to use Redis' EXPIRE with a TTL of 10min for each of your cache's keys? Redis will keep the data for 10min and then expire it, simplifying the application's logic. You would no longer need to actively check the "age" of the key - if it isn't in Redis then you need to fetch (for reference look up the so-called "Cache Aside" pattern).
Related
I am currently working on a task where I need to synchronize the data, example:
Client makes a request to get data to the API (API has access to the database), client received the data and saves it to the list. That list is used for some time etc..... Then in the meanwhile another client accesses the same API and database and makes some changes... to the same table... After a while the first client want's to update his current data, since the table is quite big let's say 10 thousand records, grabbing the entire table again is inefficient. I would like to grab only the records that have been modified,deleted, or newly created. And then update the current list the client 1 has. if the client has no records, he classifies all of them as newly created (at start up) and just grabs them all. I would like to do as much checking on the client's side.
How would I go about this ? I do have fields such as Modified, LastSync, IsDeleted. So I can find the records I need but main issue is how to do it efficiently with minimal repetition.
At the moment I tried to get all the rows at first, then after I want to update (Synchronize) I get the minimal required info LastSync Modified IsDeleted Key, from the API, which I compare with what I have on the client and then send only keys of the rows that don't match to the server to get the entire values that match the keys. But I am not sure about efficiency of this also... not sure how to update the current list with those values efficiently the only way I can think of is using loop in loop to compare keys and update the list, but I know it's not a good approach.
This will never work as long as you do not do the checks on the server side. There is always a chance that someone post between your api get call to server and you post call to server. Whatever test you can do at the client side, you can do it on the server side.
Depending on your DB and other setup, you could accomplish this by adding triggers on the tables/fields that you want to track in the database and setting up a cache (could use Redis, Memcached, Aerospike, etc.) and a cache refresh service . If something in the DB is added/updated/deleted, you can set up your trigger to write to a separate archive table in the DB. You can then set up a job from, e.g., Jenkins (or have a Kafka connector -- there are many ways to accomplish this) to poll the archive tables and the original table for changes based on an ID or date or whatever criteria you need. Anything that has changed will be refreshed and then written back to the cache. Then your API wouldn't be accessing the DB at all. It would just call the cache for the most recent data whenever a client requests it. Your separate service wold be responsible for synchronization of data, database access, and keeping the cache up to date.
Is it possible to cache once produced response on server-side and then redeliver it in response to the same request?
Let me explain:
I have an endloint that takes about 5 seconds to generate a response - this includes going to the database and fetching data, processing it, performing some computations on it, serealizing and gzipping the response - the entire thing takes 5 seconds.
Once this is done for the first time I want the result to be available for all the requests coming from all the users.
In my views client side caching, when you either cache the result on the client and do not hit the server at all for some time or when you hit the server but get 304 not-changed instead of the data is not good enough.
What i want is to hit the sever and if this enndpoint (with the same set of parameters) was already called by anyone then get the full response. Is it possible at all?
You have a number of options for this.
One option is API level caching, you create a key using the parameters required to generate the response, you go and fetch the data and save the pair in the cache. Then next time a request comes in, you recreate the key and go and check your cache first. If it's there, happy days, return it, if not, go fetch it and store it.
This of course depends on the amount of data you have, too muchm data or too big data and this will not work. You could also store it for a while, say 10 minutes, 1 hour etc.
If you have a lot of data and caching like this isn't possible then consider something else. Maybe create your own no-sql cache store ( using something like MongoDB maybe ),store it and retrieve it from there, without the need for any changes so it's a straight retrieve, thus very quick.
You could also use something like Redis Cache.
Lots of options, just choose whatever is appropriate.
I want to be stored some master data in web server Cache and that data should not be expired. Consider a master data like type of users -> Trader, Supplier, Producer and so on. So this type of master data is usually less intensive to add/delete/update. Now in a normal scenario the process of querying the database for each request is not cost-effective in terms of server resources, hence it is much better to cache or persist the data. So in this case what type of expiration cache need to be used? Is there any option in cache to persist the data alive in webserver until iisreset/recycle?
gayathri, you can explicitly save the value in cache like below:
Cache["UserType"]="Trader";//or any runtime value
Regards,
Aamir
I have a custom centralized MemoryCache and wrapper to be exposed over network. The objects to be stored are expected to be large and accessed frequently. The network traffic might become a bottleneck just because of the size of the data.
So, I want to return some kind of value that the clients may ask first to check if they need a new value from the cache server. I have 3 options:
Generate checksum in the cache server and return to client while adding object to cache.
Generate
Get current time tick in the cache server and return to client while adding object to cache.
In both the cases, the checksum or time tick will also be added in cache with different key so that clients may query it separately.
I can avoid this if either of these values is available from MemoryCache directly. Any idea?
Thanks in advance.
I need to maintaing the Session throughout the day? On or before session expires, how do I enforce session throughout the day without giving any session timeout either in IIS or web.config?
Note: Default Session Timeout in IIS – 20 minutes.
Thanks in Advance
The first thing you can do is decouple the session from the process by using a SQL (or other database) session state server that holds the serialized copy of the session data attached to a session id. Then you could use a persistent cookie to pass the session id back and forth. Any session that expires could then be regenerated or reinstated.
Since you're wanting to maintain a session all day (which is innately insecure), we'll assume that most security considerations have already been thrown out. If you're concerned about replay attacks, you could use an HMAC to validate cookie session requests by having the session id, datetime, username (if that's part of your data), maybe IP address in a string followed by a one-way salted hash of this data. This way you rehash the data and compare the hash sent with the request with the generated hash from that data to see if the request is valid.
Forcing the page to be refreshed every 10 minutes or so by javascript. The session timeouts are how long after the last requested page, the session becomes invalid. When you refresh the page every 10 minutes or so, the session is extended for 20 minutes on every refresh.
If you refuse to change the setting in the web.config to a longer value, you could try to force the browser to refresh within a short time of the session timeout (eg at 17 minutes). There might be other nifty ways.
If you can find a way to do without the session, it will save you pain later. Either use a caching strategy (memcached) or look at your resource usage.
eg Do you really need to serialise X into the session?
Use StateServer instead of InProc
This delegates all session handling to a separate database and removes it from the servers process. This will mitigate problems with eating up server resources which then allows you to increase the session timeout to 24 hours.
I believe this is the only viable solution.
Peiter is on the right track -- making fake requests to keep things alive. You definitely don't want to refresh the page, that leads to unwanted updates and such. You don't even need to refresh part of the page. You just need to send an ajax request every X minutes while a user has a browser open to an IHttpHandler that does have session state enabled. This is very, very effective -- we've got one app which is compltely dependent upon sessions for everything that has a 5 minute session timeout. And a session heartbeat that fires every 4.5 minutes. Its been in production upwards of 3 years and we have not lost a session yet. And yes, session state is stored in process in this case -- the use of sessions is so heavy we don't want to wait for them to go out to SQL and back.
PS: protip -- make sure to make the request and response unique; caching kicks in otherwise in certain browsers rendering this trick ineffective.
put a ajax call into a timer, to a method which does nothing "", that's how even the user is on the page but not doing anything means "totally inactive", would not log out, we have implemented the same logic and resolved our problem, our target was 3 to 4 hours, and we did not want to change the value in web.config or using any other way cause of some reasons.
that ajax call would be in a "timer", set the timer time to "any".