Develop Reference

How can I write multiple updates in one BulkAll method in ElasticSearch NEST 7.13.2

Using ElasticSearch NEST .Net package 7.13.2 in Visual Studio 2019
For a list of products I am currently updating existing documents in my product index by using the following code:
var productIndex = "productindex";
foreach (var product in products)
{
productClassIdScript = $"ctx._source.productClassId = \"{product.ProductClassId}\"; ";
elasticClient.Update<productIndex, object>(product.Id,
q => q.Script(s => s.Source(productClassIdScript).Lang("painless")));
}
I do this for more than 10000 products and it takes about 2 hours.
I know I can insert new documents with the Bulk API.
Can I do the updates with the BulkAll method ?
Something like this:
var bulkAllObservable = elasticClient.BulkAll<Product>(myBulkAllRequest)
.Wait(TimeSpan.FromMinutes(15), next =>
{
// do something e.g. write number of pages to console
});
How should I construct myBulkAllRequest ?
Any help is much appreciated.

Bulk index will drastically reduce your indexing / updating time, so this is a good way to go.
You can still use BulkAll for updates, in case elasticsearch already has
document with provided id, the document will be updated.
var bulk = elasticClient.BulkAll<EsDocument>(new List<EsDocument> { new EsDocument { Id = "1", Name = "1" }}, d => d);
using var subscribe = bulk.Subscribe(new BulkAllObserver(onNext: response => Console.WriteLine("inserted")));
bulk.Wait(TimeSpan.FromMinutes(1), response => Console.WriteLine("Bulk insert done"));
var bulk2 = elasticClient.BulkAll<EsDocument>(new List<EsDocument> { new EsDocument { Id = "1", Name = "1_updated" }}, d => d);
using var subscribe2 = bulk2.Subscribe(new BulkAllObserver(onNext: response => Console.WriteLine("inserted")));
bulk2.Wait(TimeSpan.FromMinutes(1), response => Console.WriteLine("Bulk insert done"));
First BulkAll will insert document with Id "1" second, will update document with Id "1".
Index state after the first bulkd
and after second one

C#, Entity Framework, How to Update(delete, add, edit) multiple rows at same time?

Sometimes, we would like to change order details by adding, removing, and editing orders by customer's request or depends on stock quantity.
So now want get some list and update including remove, edit, add rows, then save on database
What's the best efficiently way as C#, EntityFramework?
public class OrderDetail
{
public int Id { get; set; }
public int OrderId {get; set; }
public int Qty{ get; set; }
public string ItemName { get; set; }
}
/// Dummy db, OrderDetail Table
{
{1, 1000, 24,"A"},
{2, 1000, 12,"B"}
}
public void Update()
{
using(var db = new xxEntities())
{
// Get All orders, OrderId==1000, total 2rows
List<OrderDetails> list = db.OrderDetails.Where(x=> x.OrderId==1000).ToList();
// remove some row or rows
var temp1 = list.First(x=> x.Id==1);
list.Remove(temp);
// edit some row or rows
var temp2 = list.First(x=> x.Id==2);
temp2.Qty=100;
// add some row or rows
list.Add(new OrderDetail{ Id=3, OrderId=1000, Qty=2, ItemName="C"});
list.Add(new OrderDetail{ Id=4, OrderId=1000, Qty=2, ItemName="D"});
// Apply all changes
db.SaveChanges();
}
}
Additional Question
public void UpdateOrder(int orderId, List<OrderDetail> newOrders)
{
var result = db.OrderDetails.Where(x=>x.OrderId==orderId).ToList();
result = newOrders;
// it does not work
//db.OrderDetails.Update(result);
db.OrderDetails.RemoveRange(result);
db.OrderDetails.AddRange(newOrders);
db.SaveChange();
}
is it right approach to update multiple rows?

As mentioned in another answer... EF will create individual statements for each of the changes that are detected (i.e., updates, inserts, deletes) and submit them inside a single transaction. Gets the job done but is potentially very "chatty". Benefit is that you don't need to worry about the details of how it's getting done. Pretty easy to just modify the data object and call SaveChanges.
If you can consider not using EF for updates such as this... one way we do this kind of update is by creating a System.Data.DataTable and using that as a table-valued parameter into a stored procedure (if your datastore supports it).
Meta-code:
var dt = new DataTable();
var newRow = dt.NewRow();
newRow["column1"] = newdata;
dt.Rows.Add(newRow);
Then just use dt as your input parameter and let the stored proc determine the insert/update/delete operations.

If you want to Add / Remove / Update rows from your tables in Entity Framework, you have to Add / Remove / Update the items in your DbSet, not in fetched data.
using (var dbContext = new OrderContext())
{
// Add one Order
Order orderToAdd = new Order
{
// fill required properties; don't fill primary key
}
var addedOrder = dbContext.Orders.Add(orderToAdd);
// note: addedOrder has no Id yet.
// Add several Orders
IEnumerable<Order> orders = ...
dbContext.Orders.AddRange(orders);
dbContext.SaveChanges();
// now they've got their id:
Debug.Assert(addedOrder.Id != 0);
Debug.Assert(orders.All(order => order.Id != 0);
}
To Remove, you'll first have to fetch the complete Order
int orderIdToDelete = ...
using (var dbContext = new OrderContext())
{
Order orderToDelete = dbContext.Orders.Find(orderIdToDelete);
dbContext.Orders.Remove(orderToDelete);
var ordersToDelete = dbContext.Orders
.Where(order => order.Date.Year < 2000)
.ToList();
dbContext.Orders.RemoveRange(ordersToDelete);
// the orders are not deleted yet.
dbContext.SaveChanges();
}
To Update, you first have to get the value:
int orderIdToUpdate = ...
Order orderToUpdate = dbContext.Orders.Find(orderIdToUpdate);
orderToUpdate.Date = DateTime.Today;
var today = Datetime.Today;
var dateLimit = today.AddDays(-28);
var nonPaidOrders = dbContext.Orders
.Where(order => !order.Paid && order.Date < dateLimit)
.ToList();
foreach (var order in nonPaidOrders)
{
this.SendReminder(order);
order.ReminderDate = today;
}
dbContext.SaveChanges();

There is no "most efficient" way outside of making all changes then calling SaveChanges. upon which Ef will issue a lot of SQL Statements (one per operation).
There is most efficient way because there is no way to change the way Ef works and there is exactly one way Ef does its updates. They do NOT happen at the same time. Period. They happen in one transaction, one after the other, when you call SaveChanges.

Update collection from DbSet object via Linq

i know it is not complicated but i struggle with it.
I have IList<Material> collection
public class Material
{
public string Number { get; set; }
public decimal? Value { get; set; }
}
materials = new List<Material>();
materials.Add(new Material { Number = 111 });
materials.Add(new Material { Number = 222 });
And i have DbSet<Material> collection
with columns Number and ValueColumn
I need to update IList<Material> Value property based on DbSet<Material> collection but with following conditions
Only one query request into database
The returned data from database has to be limited by Number identifier (do not load whole database table into memory)
I tried following (based on my previous question)
Working solution 1, but download whole table into memory (monitored in sql server profiler).
var result = (
from db_m in db.Material
join m in model.Materials
on db_m.Number.ToString() equals m.Number
select new
{
db_m.Number,
db_m.Value
}
).ToList();
model.Materials.ToList().ForEach(m => m.Value= result.SingleOrDefault(db_m => db_m.Number.ToString() == m.Number).Value);
Working solution 2, but it execute query for each item in the collection.
model.Materials.ToList().ForEach(m => m.Value= db.Material.FirstOrDefault(db_m => db_m.Number.ToString() == m.Number).Value);
Incompletely solution, where i tried to use contains method
// I am trying to get new filtered collection from database, which i will iterate after.
var result = db.Material
.Where(x=>
// here is the reasonable error: cannot convert int into Material class, but i do not know how to solve this.
model.Materials.Contains(x.Number)
)
.Select(material => new Material { Number = material.Number.ToString(), Value = material.Value});
Any idea ? For me it is much easier to execute stored procedure with comma separated id values as a parameter and get the data directly, but i want to master linq too.

I'd do something like this without trying to get too cute :
var numbersToFilterby = model.Materials.Select(m => m.Number).ToArray();
...
var result = from db_m in db.Material where numbersToFilterBy.Contains(db_m.Number) select new { ... }

How to delete several documents by ID in one operation using Elasticsearch Nest

I am building some abstraction functions for my application to call, which will hit elasticsearch through Nest. One of such functions is a Delete(string id) call, which is easy to accomplish. I have done this as follows:
public void Delete(string id)
{
esClient.Delete(id);
}
Now let's say I want to do the same thing, but operate on several documents simultaneously. My original hunch was to do something like this:
public void Delete(IEnumerable<string> ids)
{
esClient.DeleteMany(ids); // won't compile
}
As my comment states, doing this won't compile. What is the proper way of batch deleting documents by ID in Nest?

To use esClient.DeleteMany(..) you have to pass collection of objects to delete.
var objectsToDelete = new List<YourType> {.. };
var bulkResponse = client.DeleteMany<YourType>(objectsToDelete);
You can get around this by using following code:
var ids = new List<string> {"1", "2", "3"};
var bulkResponse = client.DeleteMany<YourType>(ids.Select(x => new YourType { Id = x }));
Third option, use bulk delete:
var bulkResponse = client.Bulk(new BulkRequest
{
Operations = ids.Select(x => new BulkDeleteOperation<YourType>(x)).Cast<IBulkOperation>().ToList()
});

I was working on a .NET client for ElasticSearch 5.x, and I was fortunate to have the following code running (as well as succeeding all unit tests) for bulk deletion using ID's:
//IList<string> ids = ...
var descriptor = new BulkDescriptor();
foreach (var id in ids.Where(x => !string.IsNullOrWhiteSpace(x)))
descriptor.Delete<T>(x => x
.Id(id))
.Refresh(Refresh.WaitFor);
var response = await _client.BulkAsync(descriptor);

Bulk delete using Ravendb

I am trying to delete documents ids from a collection in RavenDB using the below logic
var documentstore = new DocumentStore
{
Url = "http://localhost:8080",
DefaultDatabase = "Employee"
};
documentstore.Initialize();
using (var session = documentstore.OpenSession())
{
var test = new List<string>();
test.Add("emps/81993");
test.Add("emps/40319");
var w1 = session.Load<Employee>(test);
session.Delete(w1);
session.SaveChanges();
}
I get the below error
Models.Employee[] is not associated with the session, cannot delete unknown
How do I go about doing a bulk delete of document Ids from the collection?
Thanks

You are trying to delete the array of employees, not each employee itself. When you pass in an Enumerable in the Load, you will get an array back containing each entity.
Try this instead:
using (var session = documentstore.OpenSession())
{
var test = new List<string>();
test.Add("emps/81993");
test.Add("emps/40319");
Employee[] employees = session.Load<Employee>(test);
foreach (var employee in employees)
{
session.Delete(employee);
}
session.SaveChanges();
}
To further explain, the returned array is not tracked by RavenDb's UoW, but each individual item in the array is, that's why you get the message about Employee[] is not associated with the session.

You can use set based operation.
The following code will query specific index and delete everything matching the query
var operation = store.DatabaseCommands.DeleteByIndex("Index Name", new IndexQuery
{
Query = "Lucene query of stuff to delete"
}, options: null);
for example, the following code will delete all documents from a document collection called "Companies" (excerpt from one of RavenDB unit tests)
var operation = documentStore.DatabaseCommands.DeleteByIndex("Raven/DocumentsByEntityName", new IndexQuery
{
Query = "Tag:[[Companies]]"
});
You can read more in this link to RavenDB documentation