Develop Reference

Populate new field in SQLite database using existing field value and a C# function

I have a SQLite database for which I want to populate a new field based on an existing one. I want to derive the new field value using a C# function.
In pseudocode, it would be something like:
foreach ( record in the SQLite database)
{
my_new_field[record_num] = my_C#_function(existing_field_value[record_num]);
}
Having looked at other suggestions on StackOverflow, I'm using a SqliteDataReader to read each record, and then running a SQlite "UPDATE" command based on the specific RowId to update the new field value for the same record.
It works .... but it's REALLY slow and thrashes the hard drive like crazy. Is there really no better way to do this?
Some of the databases I need to update might be millions of records.
Thanks in advance for any help.
Edit:
In response to the comment, here's some real code in a legacy language called Concordance CPL. The important point to note is that you can read and write changes to the current record in one go:
int db;
cycle(db)
{
db->FIRSTFIELD = myfunction(db->SECONDFIELD);
}
myfunction(text input)
{
text output;
/// code in here to derive output from input
return output;
}
I have a feeling there's no equivalent way to do this in SQLite as SQL is inherently transactional, whereas Concordance allowed you to traverse and update the database sequentially.

The answer to this is to wrap all of the updates into a single transaction.
There is an example here that does it for bulk inserts:
https://www.jokecamp.com/blog/make-your-sqlite-bulk-inserts-very-fast-in-c/
In my case, it would be bulk updates based on RowID wrapped into a single transaction.
It's now working, and performance is many orders of magnitude better.
EDIT: per the helpful comment above, defining a custom C# function and then reference it in a single UPDATE command also works well, and in some ways is better than the above as you don't have to loop through within C# itself. See e.g. Create/Use User-defined functions in System.Data.SQLite?

Trying to sync data from third party api

This question has probably been asked correctly before, and I'll gladly accept an answer pointing me to the right spot. The problem is I don't know how to ask the question correctly to get anything returned in a search.
I'm trying to pull data from a 3rd party api (ADP) and store data in my database using asp.net core.
I am wanting to take the users returned from the API and store them in my database, where I have an ADP ancillary table seeded with the majority of the data from the api.
I would then like to update or add any missing or altered records in my database FROM the API.
I'm thinking that about using an ajax call to the api to retrieve the records, then either storing the data to another table and using sql to look for records that are changed between the two tables and making any necessary changes(this would be manually activated via a button), or some kind of scheduled background task to perform this through methods in my c# code instead of ajax.
The question I have is:
Is it a better fit to do this as a stored procedure in sql or rather have a method in my web app perform the data transformation.
I'm looking for any examples of iterating through the returned data and updating/creating records in my database.
I've only seen vague not quite what I'm looking for examples and nothing definitive on the best way to accomplish this. If I can find any reference material or examples, I'll gladly research but I don't even know where to start, or the correct terms to search for. I've looked into model binding, ajax calls, json serialization & deserialization. I'm probably overthinking this.
Any suggestions or tech I should look at would be appreciated. Thanks for you time in advance.
My app is written in asp.net core 2.2 using EF Core
* EDIT *
For anyone looking - https://learn.microsoft.com/en-us/dotnet/csharp/tutorials/console-webapiclient
This with John Wu's Answer helped me achieve what I was looking for.

If this were my project this is how I would break down the tasks, in this order.
First, start an empty console application.
Next, write a method that gets the list of users from the API. You didn't tell us anything at all about the API, so here is a dummy example that uses an HTTP client.
public async Task<List<User>> GetUsers()
{
var client = new HttpClient();
var response = await client.GetAsync("https://SomeApi.com/Users");
var users = await ParseResponse(response);
return users.ToList();
}
Test the above (e.g. write a little shoestring code to run it and dump the results, or something) to ensure that it works independently. You want to make sure it is solid before moving on.
Next, create a temporary table (or tables) that matches the schema of the data objects that are returned from the API. For now you will just want to store it exactly the way you retrieve it.
Next, write some code to insert records into the table(s). Again, test this independently, and review the data in the table to make sure it all worked correctly. It might look a little like this:
public async Task InsertUser(User user)
{
using (var conn = new SqlConnection(Configuration.ConnectionString))
{
var cmd = new SqlCommand();
//etc.
await cmd.ExecuteNonQueryAsync();
}
}
Once you know how to pull the data and store it, you can finish the code to extract the data from the API and insert it. It might look a little like this:
public async Task DoTheMigration()
{
var users = await GetUsers();
var tasks = users.Select
(
u => InsertUser(u)
);
await Task.WhenAll(tasks.ToArray());
}
As a final step, write a series of stored procedures or a DTS package to move the data from the temp tables to their final resting place. If you are using MS Access, you can write a series of queries and execute them in order with some VBA. At a high level it would:
Check for any records that exist in the temp table but not in the final table and insert them into the final table.
Check for any records that exist in the final table but not the temp table and remove them or mark them as deleted.
Check for any records in common that have different column values and update the final table.
Each of these development activities raises it own set of questions, of course, which you can post back to StackOverflow with details. As it is your question doesn't have enough specificity for a more in-depth answer.

dapper: #tempDB created but not available in next query

In my program, I want to select some bookIDs into a tempDB for later queries like this (using Dapper extension):
using (var conn = new SqlConnection(connStr)) {
conn.Execute("SELECT bookID INTO #tempdb WHERE ... FROM Books");
int count = conn.ExecuteScalar<int>("SELECT COUNT(*) FROM #tempdb");
var authors = conn.Query("SELECT * FROM #tempdb LEFT JOIN BookAuthors ON ...");
}
However when I execute the page, I get following exception:
Invalid object name '#tempdb'.
It seems that life-cycle of #tempdb is only valid in first query ?

It looks like you're using the implicit connection opening/closing. This will indeed cause problems with transient objects. If you need temp tables between queries, you will need to manually open the connection before you execute any such queries. This should then work fine, and many examples in the test suite make use of temp tables in the way.
However, from a practical standpoint, making use of temporary tables to transfer state between queries is ... awkward. In addition to being brittle, it isn't good for the plan cache, as #foo has a different meaning between all uses on different connection (including reset but reused connections).

I found a previous poster who met the same problem and his solution.
Using dapper, why is a temp table created in one use of a connection not available in a second use of the same connection
The post indicates that you have to "CREATE TABLE #tempdb" explicitly in your SQL first and everything goes fine. Even the poster himself don't know why such style of coding works.

MySQL insert based on datetime

This is my first post ever, so I apologize if I unknowingly break any rules regarding post etiquette.
I am migrating db entries using C# and the Redmine API. I need to pull the time entries in XML and insert into MySQL if and only if the time entries are newer than what exists currently in the database.
Right now I am prototyping this by simply rerouting the original MySQL db entries back to MySQL and into a new table. Later I will using ERP software similar to SAP software.
I am not sure if the program logic needs to be primarily in the C# code, or in a MySQL statement. Here is where I am stuck:
foreach (TimeEntry t in timeEntries.list)
{
comm.CommandText =
"INSERT INTO time_entries(time_entry_id,project_id,issue_id,user_id,activity_id,hours,comments,"
+ "spent_on,created_on,updated_on,customer_number)"
+ "VALUES(?time_entry_id,?project_id,?issue_id,?user_id,?activity_id,?hours,?comments,?spent_on,"
+ "?created_on,?updated_on,?customer_number)";
comm.Parameters.AddWithValue("?time_entry_id", t.id);
comm.Parameters.AddWithValue("?project_id", t.project.Id);
comm.Parameters.AddWithValue("?issue_id", t.issue.id);
comm.Parameters.AddWithValue("?user_id", t.user.id);
comm.Parameters.AddWithValue("?activity_id", t.activity.id);
comm.Parameters.AddWithValue("?hours", t.hours);
comm.Parameters.AddWithValue("?comments", t.comments);
comm.Parameters.AddWithValue("?spent_on", t.spent_on);
comm.Parameters.AddWithValue("?created_on", t.created_on);
comm.Parameters.AddWithValue("?updated_on", t.updated_on);
comm.Parameters.AddWithValue("?customer_number", t.custom_fields.list[0].value);
comm.ExecuteNonQuery();
comm.Parameters.Clear();
continue;
}
Of course, everything inserts just fine, and subsequent runs of the program will simply insert duplicate records. When this is finished, the idea is that it will run as a daily cron job that will update MySQL with new time entries only.
As I mentioned, at a later point I will be sending the XML data directly to an ERP program similar to SAP software, but right now I want to make sure the MySQL to MySQL prototype works first.
I am a relatively new programmer, and a first time poster, so if there are flaws with my code logic or if my question is not specific enough then please let me know. Thanks in advance.

Actually, I just figured out a working solution. Not sure if it is the best way to do it, but it works, so I am happy.
All I did was changed updated_on to be the primary key in MySQL, and then handled the exception that was thrown for duplicate entries:
try
{
comm.ExecuteNonQuery();
}
catch(MySql.Data.MySqlClient.MySqlException mySqlEx)
{
Console.WriteLine("Duplicate entry found");
}
I wanted to post the code in case it helps anyone else searching for an answer to this same issue. As far as I am concerned, this problem is now solved.

You can try using DUAL
INSERT INTO time_entries (time_entry_id, project_id, issue_id, user_id,
activity_id, hours,comments, spent_on,
created_on, updated_on, customer_number)
SELECT ?time_entry_id, ?project_id, ?issue_id, ?user_id, ?activity_id,
?hours, ?comments, ?spent_on, ?created_on, ?updated_on,
?customer_number
FROM DUAL
WHERE '2015-11-20' NOT IN (SELECT YourTimeField from time_entries)
I tryed to write a clear example (hope you understood the logic)

C# - Using database table to populate variables

EDIT: Thank you everyone, I figured out how to get it to work now! Details below...
I'm kind of a newbie to C#, and I'm trying to teach myself the language by programming a really simple RPG game.
Right now, I'm at the point where I want to start adding different enemies to fight (up until now I just used a single one hardcoded in for testing).
I've started setting up a database with enemy info (one column for name, one for HP, one for common stats and attacks, etc.). I have it so that when you start combat with an enemy, the player is able to select a creature from a dropdown, and whichever creature he has will set a variable called "EnemyID".
What I want to do is use that EnemyID variable to correspond to a row in my database, then pull the value of each column into variables that I can then reference during combat.
Is this something that's possible to do? If so, could someone explain the method to me in relatively simple terms? Even just a small example of how to import row data from any kind of database will do, I'm good at understanding code once I see it in use a couple of times.
(Oh yeah, if it matters, I'm using Visual Studio Express 2013, and my database is a SQL Server Express 2014 database.)
Thanks in advance!
EDIT:
After finding a simple tutorial for ADO.NET, and following a suggestion from one of the posters, I've come up with the following code.
public void DataPoll()
{
SqlConnection MonDat = new SqlConnection("Data Source=(local);
Initial Catalog=TestDatabase;Integrated Security=SSPI");
SqlDataReader rdr = null;
try
{
MonDat.Open();
SqlCommand cmd = new SqlCommand(
"select * from Monsters where Id = EnemyID", MonDat);
rdr = cmd.ExecuteReader();
while (rdr.Read())
{
EnemyIDBuffer = (int)rdr["Id"];
EnemyName = (string)rdr["Name"];
EnemyHPBase = (int)rdr["HP"];
EnemyAtkBase = (int)rdr["Atk"];
EnemyDefBase = (int)rdr["Def"];
EnemyMagBase = (int)rdr["Mag"];
PrimAtk = (string)rdr["PrimAtk"];
SecoAtk = (string)rdr["SecoAtk"];
TertAtk = (string)rdr["TertAtk"];
RareAtk = (string)rdr["RareAtk"];
}
}
finally
{
if (rdr != null)
{
rdr.Close();
}
if (MonDat != null)
{
MonDat.Close();
}
}
}
However, when I try to run it, my program stalls and crashes. I'm guessing I have something configured wrong (I just took script from the tutorial and tweaked it slightly). Can anyone give me a hand figuring out where I went wrong?
EnemyID is a variable I used to assign what enemy is fought, based on a menu selection. I'm trying to use that variable to generate the row ID to pull the rest of the row data from, but I think that might be causing an issue.
EDIT2: It took me longer than it really should have, but I figured it out. I had to change my code a little tiny bit.
SqlCommand cmd = new SqlCommand(
"select * from Monsters where Id = " + EnemyID, MonDat);
I have a habit of forgetting that you're able to join statements like this. I made a new project that only polled data and threw it into my variables, and from there put it into text boxes, and with this method I was able to poll two different sets of enemy stats by assigning different EnemyID values to two different buttons. Proof of concept, right there.
Thanks to both people who replied, both suggestions were equally useful to getting this working. :)

There's numerous tutorials out there on how to use a database, the first two use straight ADO.NET which is pure data access, making you responsible for its interaction in your code:
ADO.NET Overview
ADO.NET Tutorial for Beginners
The next two, one is for Entity, and the other for nHibernate, they connect to SQL databases and convert the objects there to usable code in your program through a process called object relational mapping.
Entity Framework Tutorials
nHibernate Tutorials
These are all relevant links to stuff in the most current years, with VS 2013; hopefully that provides you a good starting point.

You can do something like this:
Your SQL should pass in the procedure name and EnemyId.
The stored procedure would do a select * from Enemies where EnemyId = #EnemyId
DataSet dataSet = HSPDataAccessProxy.Instance.ExecuteDataSet(sql);
The dataSet has the table that is returned by the store procedure and you can retrieve the columns you need from that table.