Develop Reference

Trying to sync data from third party api

This question has probably been asked correctly before, and I'll gladly accept an answer pointing me to the right spot. The problem is I don't know how to ask the question correctly to get anything returned in a search.
I'm trying to pull data from a 3rd party api (ADP) and store data in my database using asp.net core.
I am wanting to take the users returned from the API and store them in my database, where I have an ADP ancillary table seeded with the majority of the data from the api.
I would then like to update or add any missing or altered records in my database FROM the API.
I'm thinking that about using an ajax call to the api to retrieve the records, then either storing the data to another table and using sql to look for records that are changed between the two tables and making any necessary changes(this would be manually activated via a button), or some kind of scheduled background task to perform this through methods in my c# code instead of ajax.
The question I have is:
Is it a better fit to do this as a stored procedure in sql or rather have a method in my web app perform the data transformation.
I'm looking for any examples of iterating through the returned data and updating/creating records in my database.
I've only seen vague not quite what I'm looking for examples and nothing definitive on the best way to accomplish this. If I can find any reference material or examples, I'll gladly research but I don't even know where to start, or the correct terms to search for. I've looked into model binding, ajax calls, json serialization & deserialization. I'm probably overthinking this.
Any suggestions or tech I should look at would be appreciated. Thanks for you time in advance.
My app is written in asp.net core 2.2 using EF Core
* EDIT *
For anyone looking - https://learn.microsoft.com/en-us/dotnet/csharp/tutorials/console-webapiclient
This with John Wu's Answer helped me achieve what I was looking for.

If this were my project this is how I would break down the tasks, in this order.
First, start an empty console application.
Next, write a method that gets the list of users from the API. You didn't tell us anything at all about the API, so here is a dummy example that uses an HTTP client.
public async Task<List<User>> GetUsers()
{
var client = new HttpClient();
var response = await client.GetAsync("https://SomeApi.com/Users");
var users = await ParseResponse(response);
return users.ToList();
}
Test the above (e.g. write a little shoestring code to run it and dump the results, or something) to ensure that it works independently. You want to make sure it is solid before moving on.
Next, create a temporary table (or tables) that matches the schema of the data objects that are returned from the API. For now you will just want to store it exactly the way you retrieve it.
Next, write some code to insert records into the table(s). Again, test this independently, and review the data in the table to make sure it all worked correctly. It might look a little like this:
public async Task InsertUser(User user)
{
using (var conn = new SqlConnection(Configuration.ConnectionString))
{
var cmd = new SqlCommand();
//etc.
await cmd.ExecuteNonQueryAsync();
}
}
Once you know how to pull the data and store it, you can finish the code to extract the data from the API and insert it. It might look a little like this:
public async Task DoTheMigration()
{
var users = await GetUsers();
var tasks = users.Select
(
u => InsertUser(u)
);
await Task.WhenAll(tasks.ToArray());
}
As a final step, write a series of stored procedures or a DTS package to move the data from the temp tables to their final resting place. If you are using MS Access, you can write a series of queries and execute them in order with some VBA. At a high level it would:
Check for any records that exist in the temp table but not in the final table and insert them into the final table.
Check for any records that exist in the final table but not the temp table and remove them or mark them as deleted.
Check for any records in common that have different column values and update the final table.
Each of these development activities raises it own set of questions, of course, which you can post back to StackOverflow with details. As it is your question doesn't have enough specificity for a more in-depth answer.

Prevent insertion of duplicate documents into Lotus notes database

I have a c# web api hosted in iis which has a post method that takes a list of document ids to insert into a lotus notes database.
The post method can be called multiple times and I want to prevent insertion of duplicate documents.
This is the code(in a static class) that is called from the post:
lock (thisLock)
{
var id = "some unique id";
doc = vw.GetDocumentByKey(id, false);
if (doc == null)
{
NotesDocument docNew = db.CreateDocument();
//some more processing
docNew.Save(true, false, false);
}
}
Even with the lock in place, I am running into scenarios where duplicate documents are inserted. Is it because a request can be execute on a new process? What is the best way to prevent it from happening?

Your problem is: getdocumentbykey depends on the view index being up to date. On a busy server there is no guarantee that this is true. You can TRY to call a vw.Update, but unfortunately this does not trigger an update of the view index, so it might be without any effect (it just updates the vw object to represent what has changed in the backend, if the backend did not update, then it does nothing).
You could use db.Search('IdField ="' & id & '"', Nothing, 0) instead, as the search does not rely on an index to be rebuilt. This will be slightly slower, but should be way more accurate.

you might want to store the inserted ids in some singleton object or even simply static list. And lock on this list - whoever obtains the lock verifies that the ids it wants to insert are not present and then adds them to the list itself.
You need to keep them only for a short length of time, just so that 2 concurrent posts with the same content does not update plus normal view index gets updated. So rather store timestamp along id, so you can clean out older records if the list grows long.

How to get Indentity value back after insert

Given the following code (which is mostly irrelevant except for the last two lines), what would your method be to get the value of the identity field for the new record that was just created? Would you make a second call to the database to retrieve it based on the primary key of the object (which could be problematic if there's not one), or based on the last inserted record (which could be problematic with multithreaded apps) or is there maybe a more clever way to get the new value back at the same time you are making the insert?
Seems like there should be a way to get an Identity back based on the insert operation that was just made rather than having to query for it based on other means.
public void Insert(O obj)
{
var sqlCmd = new SqlCommand() { Connection = con.Conn };
var sqlParams = new SqlParameters(sqlCmd.Parameters, obj);
var props = obj.Properties.Where(o => !o.IsIdentity);
InsertQuery qry = new InsertQuery(this.TableAlias);
qry.FieldValuePairs = props.Select(o => new SqlValuePair(o.Alias, sqlParams.Add(o))).ToList();
sqlCmd.CommandText = qry.ToString();
sqlCmd.ExecuteNonQuery();
}
EDIT: While this question isn't a duplicate in the strictest manner, it's almost identical to this one which has some really good answers: Best way to get identity of inserted row?

It strongly depends on your database server. For example for Microsoft SQL Server you can get the value of the ##IDENTITY variable, that contains the last identity value assigned.
To prevent race conditions you must keep the insert query and the variable read inside a transaction.
Another solution could be to create a stored procedure for every type of insert you have to do and make it return the identity value and accept the insert arguments.
Otherwise, inside a transaction you can implement whatever ID assignment logic you want and be preserved from concurrency problems.

Afaik there is not finished way.
I solved by using client generated ids (guid) so that my method generated the id and returns it to the caller.
Perhaps you can analyse some SqlServer systables in order to see what has last changed. But you would get concurrency issues (What if someone else inserts a very similar record).
So I would recommend a strategy change and generate the id's on the clients

You can take a look at : this link.
I may add that to avoid the fact that multiple rows can exist, you can use "Transactions", make the Insert and the select methods in the same transaction.
Good luck.

The proper approach is to learn sql.
You can do a SQL command followed by a SELECT in one run, so you can go in and return the assigned identity.
See

Optimizing a RESTful Query in the client

READ FIRST before answering!
I have a RESTful service which wraps around the Entity Framework. Basically, all I did was create a database, add relations between the tables, create an Entity model around this database and finally expose the whole thing as a RESTful *.svc service. This is done, cannot be changed either.
Now I need to query data from it through a client application. All I can access is the service itself. I cannot add any server-side code, even if I wanted to. The server is now locked.
I need to retrieve data from a table called "ProductVoorwaarden" (product conditions) which is linked to three other tables. (Rubriek, Categorie and Datatype.) This data needs to be returned as XML, with a root node called "PRODUCTVOORWAARDEN" and every record in it's own XElement called "REC". Within this REC, there's an attribute for every field in the table plus references to related tables. Here's the code I have right now:
XElement PRODUCTVOORWAARDEN()
{
XElement Result = new XElement("PRODUCTVOORWAARDEN");
var Brondata = COBA.Productvoorwaarden.OrderBy(O => O.Code);
foreach (var item in Brondata)
{
COBA.LoadProperty(item, "Rubriek");
COBA.LoadProperty(item, "Categorie");
COBA.LoadProperty(item, "Datatype");
XElement REC = new XElement("REC",
Attribute("Rubriek", item.Rubriek.Code),
Attribute("Categorie", item.Categorie.Naam),
Attribute("Code", item.Code),
Attribute("Datatype", item.Datatype.Naam),
Attribute("Eenheid", item.Eenheid),
Attribute("Naam", item.Naam),
Attribute("Omschrijving", item.Omschrijving),
Attribute("UitgebreideTekstVeld", item.UitgebreideTekstVeld),
Attribute("Veld", item.Veld)
);
Result.Add(REC);
}
return Result;
}
This code works fine, but it's slow. It reads all ProductVoorwaarden records but then it has to make round-trips to the server again for every record to retrieve Rubriek.Code, Categorie.Naam and Datatype.Naam. (In the database, these relations are set by an auto-incremental Identity field but the XML code uses Code or Naam as reference.)
As you can imagine, every trip back to the RESTful service just eats up more time, which I'm trying to avoid. So is there any way to speed this all up a bit more just on the client-side?
The server is still under development and the next release will take a few more months. As a result, I have to deal with the options that the server provides right now. If there's no way to speed this up without modifying the server then fine. At least I've tried. There are 35 more tables that need to be processed with a deadline in a few days so if it works, then it works.

You could make each of your COBA.LoadProperty calls asynchronous and run them in parallel rather than sequentially. It will make your client code more complex since you'll have to handle the return of each async call and determine when they have all completed and you're ready to build your XML. Assuming each of your 4 REST calls is taking the same amount of time that would reduce the delay by half.
You've probably already double checked but I have come across cases where generating the enumerator from the lambda expression can be expensive. Still it was in the hundreds of milliseconds and I get the impression your delay is larger than that. May be worth checking.

Linq-To-SQL Invalid Cast on insert, not on update

Hey guys, I'm having a weird time with Linq-To-SQL
I'm building a postit system that allows people to make postits and attach them to multiple objects. The database for this is a table that has the postits themselves (id, message, etc) and a linkstable which holds the records to which a postit is attached.
the weird thing I'm experiancing is the following.
When I retrieve an object from the database (using Linq-To-SQL), do some data changing and submit it again, I experience no trouble whatsoever.
Yet, when I try to make a new one I get an exception in the Submitchanges method in the datacontext: Specified Cast is not valid.
I've been looking on the web and mostly it involves some change in the mapping, but this shouldn't be the case as I can update without any problems.
T_PostIt np = new T_PostIt();
np.CreatedOn = DateTime.Now;
np.CreatedBy = Request.ServerVariables["REMOTE_USER"].ToString();
np.MarkedForDeletion = false;
np.Message = txtNewPostitMessage.Text;
np.ModifiedBy = Request.ServerVariables["REMOTE_USER"].ToString();
foreach (int i in ServerIds)
{
T_PostIt_Link pil = new T_PostIt_Link();
pil.LinkType = 'S';
pil.LinkID = i;
pil.MarkedForDeletion = false;
np.T_PostIt_Links.Add(pil);
}
dc.T_PostIts.InsertOnSubmit(np);
dc.SubmitChanges();
I use the above code and can't seem to get what I'm doing wrong.
help anyone?

Have you tried updating the properties one by one, and then save the changes back to the database? It could be that updating the entity only fails when one specific value has changed. If I may guess, it could be that the value of CreatedOn cannot be cast to a valid DateTime in the database (due to culture settings). That would explain why updating goes OK - you're not changing the value of CreatedOn here. You do, however, when inserting a new entity.
Edit: maybe this is the problem you're facing. Also, be sure to read this thread, where the topic starter eventually points to the first thread mentioning that it is an apparant bug in Linq2Sql.