I have the following test:
public class ListingEventTest
{
public ListingEventTest()
{
Artists = new List<ArtistTest>();
}
public List<ArtistTest> Artists { get; set; }
public string Id { get; set; }
public string Name { get; set; }
public double Popularity { get; set; }
}
public class ArtistTest
{
public string Id { get; set; }
public Stat Stats { get; set; }
}
public class Stat
{
public double Popularity { get; set; }
}
public class ArtistsWithStats_ByName : AbstractIndexCreationTask<ListingEventTest>
{
public ArtistsWithStats_ByName()
{
Map = listingEvents => from listingEvent in listingEvents
let artists = LoadDocument<ArtistTest>(listingEvent.Artists.Select(x => x.Id))
select new
{
Popularity = artists.Average(x => x.Stats.Popularity),
listingEvent.Name
};
}
}
[TestFixture]
public class IndexCanDoSums
{
[Test]
public async void WhenListingEventArtistsHaveStatsIndexReturnsPopularity()
{
var store = new EmbeddableDocumentStore
{
UseEmbeddedHttpServer = true,
Configuration =
{
RunInUnreliableYetFastModeThatIsNotSuitableForProduction = true,
RunInMemory = true,
}
}.Initialize();
IndexCreation.CreateIndexes(typeof(ArtistsWithStats_ByName).Assembly, store);
using (var session = store.OpenAsyncSession())
{
for (int i = 0; i < 10; i++)
{
var le = new ListingEventTest
{
Name = "test-" + i
};
await session.StoreAsync(le);
for (int j = 0; j < 2; j++)
{
var artist = new ArtistTest
{
Stats = new Stat
{
Popularity = 0.89d
}
};
await session.StoreAsync(artist);
le.Artists.Add(artist);
}
await session.SaveChangesAsync();
}
}
Thread.Sleep(2000);
using (var session = store.OpenAsyncSession())
{
var query = session
.Advanced.AsyncLuceneQuery<ListingEventTest>("ArtistsWithStats/ByName");
var result = await query.ToListAsync();
result.First().Popularity.Should().NotBe(0);
}
}
}
When I query this index Popularity is always 0.
Any ideas?
Some funny things going on here.
First, you are storing ArtistTest under the ListingEventTest document, not as separate documents, so in your index there is no need to call LoadDocument, you could just do:
from listingEvent in listingEvents
from artist in listingEvent.Artists
select ...
Second, a Map-only index is a lot like a SQL index where you're just calling out the columns you want to be able to query on. Here, you're doing a calculation on a set of buried properties and you have a top-level property where you want to store that information, but how that ends up working is that your calculated property value goes into the Lucene index (so you could query by Popularity if you want) but the data that is returned is straight from the unaltered document. The map defines what goes into Lucene, which points to the document id, and then the document store returns the documents as the results.
This could be modified somewhat by calling Store(x => x.Popularity) in the index's constructor, which would store the value to be recalled later, but honestly offhand I'm not sure if your calculated value or the document's value (which is zero) would win.
Given that, it becomes pretty confusing to have a document property for the sole purpose of trying to fill it during indexing, which is why it's usually a better option to have a class that represents the mapped state, and then implementing AbstractIndexCreationTask<TDocument, TReduceResult> where the TReduceResult class would only contain the result of your mapping, namely the Name and Popularity columns.
Then when you query from, you can use .ProjectFromIndexFieldsInto<T>() to get your results from the stored index results, not from the document store.
Related
I want to use Random Forest method in my project.
I have this error trying to use EvaluateNonCalibrated:
System.ArgumentOutOfRangeException: „AUC is not defined when there is
no positive class in the data Arg_ParamName_Name”
var testSetTransform = _trainedModel.Transform(_dataSplit.TestSet);
return MlContext.BinaryClassification.EvaluateNonCalibrated(testSetTransform);
Here you have models:
public class MLCategoryPrediciton
{
public bool PredictedLabel { get; set; }
}
public class MLFinancialChange
{
[LoadColumn(1)]
public bool Label { get; set; }
[LoadColumn(2)]
public float Value { get; set; }
[LoadColumn(3)]
public float CategoryId { get; set; }
}
And way i preparing data:
public async Task FitAsync()
{
var list = await fcRepository.FindAllAsync();
var output = new List<MLFinancialChange>();
foreach(var item in list)
{
var x = new MLFinancialChange
{
Value = item.Value,
CategoryId = item.Category.Id,
};
output.Add(x);
}
IDataView data = MlContext.Data.LoadFromEnumerable<MLFinancialChange>(output);
DataOperationsCatalog.TrainTestData dataSplit = MlContext.Data.TrainTestSplit(data);
_dataSplit = dataSplit;
var dpp = BuildDataProcessingPipeline();
var tp = dpp.Append(_model);
_trainedModel = tp.Fit(_dataSplit.TrainSet);
}
private EstimatorChain<NormalizingTransformer> BuildDataProcessingPipeline()
{
var dataProcessPipeline = MlContext.Transforms.Concatenate("Features",
nameof(MLFinancialChange.Value),
nameof(MLFinancialChange.CategoryId)
)
.Append(MlContext.Transforms.NormalizeMinMax("Features", "Features"))
.AppendCacheCheckpoint(MlContext);
return dataProcessPipeline;
}
Thanks for help
I just want that to work and i tried to find solution in the internet. Unfortunatelly i spent a lot of time trying fix it.
I've had this problem before, and I think it was because the model wasn't predicting any positive cases. You can try inspecting the predictions and just make sure it's not predicting everything as negative.
I have a list which contains order property of string.
This is what the data looks like:
1
2
2.1
2.1.1
2.1.2
2.2
2.2.1
2.2.2
3
3.1
3.2
3.3
4
And the structure of class is like this
public class CMNavBarDefaultDto
{
public int ID { get; set; }
public string Name { get; set; }
public string? Description { get; set; }
public string? Comments { get; set; }
public string Order { get; set; }
public bool ToShowGray { get; set; }
public bool IsNew { get; set; }
}
if the user delete any order from the list, like user delete 3.1 then 3.2 and 3.3 rearrange
3.2 become 3.1, 3.3 become 3.1,
if the user delete any parent like 1 then all the hierarchy should be maintained in a new form
like 2 become 1 and its child and sub-child should start from 1.
Can anyone suggest to me what approach is helpful in this scenario?
Seems to me, best method is to use list of uints to specify level as needed. You could then parallel the html table with Order object to update table as needed by having delete operation actually modify Order, which then updates html table.
The question is what to do if user deletes middle level? For example, if user deletes 1.1 in the following order, what happens to 1.1.1 and 1.1.2?
1
1.1
1.1.1
1.1.2
1.2
Does it become like below?
1
1.1
1.2
1.3 (was 1.2)
Knowing these rules, you can create a conversion function to create parallel Order and then operate on it as needed. The below code assumes all prior levels are defined as you go deeper (i.e., if you have a level 2, it is assumed the prior level was 2 or 1). At any point you can jump up levels (i.e., you could be at level 4 and then have the next level 1). If you don't follow these rules, the ToString function would throw Exception.
public class Order
{
var list = new List<unit>();
// needed: static function or constructor that takes html table string and returns Order and methods to modify Order
public override string ToString()
{
var sb = new StringBuilder();
var stack = Stack<(uint, string))>();
uint current = 0;
string order = "";
foreach (var next in list)
{
if (next > stack.Count)
{
// next is in deeper level, make sure prior level(s) exist.
if ((current == 0) || (next != (stack.Count + 1))) throw new Exception("missing level");
// prior level(s) exist, push current level in stack in case lower level needed later, then restart count within next level
stack.Push((current, order));
order = $"{order}{current}.";
current = 0;
}
else if (next < stack.Count)
{
// at lower level! pop out levels from stack until we get back to next level
while (stack.Count > next)
{
(current, order) = stack.Pop();
}
}
// append next level to output
current++;
sb.AppendLine($"{order}{current}");
}
return sb.ToString();
}
}
The following will create the tree. You have to add code to do add and remove items in tree. When you add or remove you have to renumber the items
using System;
using System.Linq;
using System.Text;
using System.Collections;
using System.Collections.Generic;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
List<CMNavBarDefaultDto> list = new List<CMNavBarDefaultDto>()
{
new CMNavBarDefaultDto() { Order = "1"},
new CMNavBarDefaultDto() { Order = "2"},
new CMNavBarDefaultDto() { Order = "2.1"},
new CMNavBarDefaultDto() { Order = "2.1.1"},
new CMNavBarDefaultDto() { Order = "2.1.2"},
new CMNavBarDefaultDto() { Order = "2.2"},
new CMNavBarDefaultDto() { Order = "2.2.1"},
new CMNavBarDefaultDto() { Order = "2.2.2"},
new CMNavBarDefaultDto() { Order = "3"},
new CMNavBarDefaultDto() { Order = "3.1"},
new CMNavBarDefaultDto() { Order = "3.2"},
new CMNavBarDefaultDto() { Order = "3.3"},
new CMNavBarDefaultDto() { Order = "4"},
};
Tree root = new Tree();
root.MakeTree(list, root);
}
}
public class CMNavBarDefaultDto
{
public int ID { get; set; }
public string Name { get; set; }
public string? Description { get; set; }
public string? Comments { get; set; }
public string Order { get; set; }
public bool ToShowGray { get; set; }
public bool IsNew { get; set; }
}
public class Tree
{
public CMNavBarDefaultDto cmNav { get; set; }
public int[] orderInt { get; set; }
public List<Tree> children { get; set; }
public void MakeTree(List<CMNavBarDefaultDto> cmNavs, Tree parent)
{
List<Tree> list = new List<Tree>();
foreach(CMNavBarDefaultDto cmNav in cmNavs)
{
Tree node = new Tree() { cmNav = cmNav, orderInt = cmNav.Order.Split(new char[] { '.' }).Select(x => int.Parse(x)).ToArray() };
list.Add(node);
}
int level = 0;
list = list.OrderBy(x => x.orderInt.Length).ToList();
MakeTreeRecursive(list, parent, level);
}
public void MakeTreeRecursive(List<Tree> list, Tree parent, int level)
{
var groups = list.GroupBy(x => x.orderInt[level]).ToList();
foreach(var group in groups)
{
if (parent.children == null) parent.children = new List<Tree>();
parent.children.Add(group.First());
if (group.Count() > 1)
{
if (group.Last().orderInt.Length == level + 1)
{
group.First().children = new List<Tree>();
group.First().children = group.Skip(1).ToList();
}
else
{
MakeTreeRecursive(group.Skip(1).ToList(), group.First(), level += 1);
}
}
}
}
}
}
I am developing an inventory management system and I want to add a product with variants. I can add a product with three variants(color, size, material) and the options for each as below:
color - Black, Blue, Grey
size - S,M,L,XL
material - Cotton, Wool
If specify only 2 variants(e.g. color and size) my code is generating all the options correctly but if I add a 3rd variant then its not giving me the expected output.
Suppose I have a product called Jean my expected output would be as below:
Jean-Black/S/Cotton
Jean-Black/S/Wool
Jean-Black/M/Cotton
Jean-Black/M/Wool
Jean-Black/L/Cotton
Jean-Black/L/Wool
Jean-Black/XL/Cotton
Jean-Black/XL/Wool
===========================================
Jean-Blue/S/Cotton
Jean-Blue/S/Wool
Jean-Blue/M/Cotton
Jean-Blue/M/Wool
Jean-Blue/L/Cotton
Jean-Blue/L/Wool
Jean-Blue/XL/Cotton
Jean-Blue/XL/Wool
===========================================
Jean-Grey/S/Cotton
Jean-Grey/S/Wool
Jean-Grey/M/Cotton
Jean-Grey/M/Wool
Jean-Grey/L/Cotton
Jean-Grey/L/Wool
Jean-Grey/XL/Cotton
Jean-Grey/XL/Wool
My model is as below:
public class CreateModel : PageModel
{
[Required]
[BindProperty]
public string? Name { get; set; }
[BindProperty]
public List<ProductVariantModel> Variants { get; set; }
}
ProductVariantModel
public class ProductVariantModel
{
public string? Name { get; set; }
public string? Options { get; set; }
}
I'm creating the combinations as below:
List<ProductVariantOption> productOptions = new();
try
{
int variantsTotal = model.Variants.Count;
for (int a = 0; a < variantsTotal; a++)
{
string[] options = model.Variants[a].Options.Split(',');
for (int i = 0; i < options.Length; i++)
{
string? option = $"{model.Name}-{options[i]}";
if (variantsTotal > 1)
{
int index = a + 1;
if (index < variantsTotal)
{
var levelBelowOptions = model.Variants[index].Options.Split(',');
var ops = GetOptions(option, levelBelowOptions);
productOptions.AddRange(ops);
}
}
}
a += 1;
}
}
GetOptions method
private List<ProductVariantOption> GetOptions(string option, string[] options)
{
List<ProductVariantOption> variantOptions = new();
for (int i = 0; i < options.Length; i++)
{
string sku = $"{option}/{options[i]}";
string opt = $"{option}/{options[i]}";
variantOptions.Add(new ProductVariantOption(opt, sku));
}
return variantOptions;
}
ProductVariantOption
public class ProductVariantOption
{
public string Name { get; private set; }
public string SKU { get; private set; }
public Guid ProductVariantId { get; private set; }
public ProductVariant ProductVariant { get; private set; }
public ProductVariantOption(string name, string sku)
{
Guard.AgainstNullOrEmpty(name, nameof(name));
Name = name;
SKU = sku;
}
}
Where am I getting it wrong?
If you generalize your problem, you can describe it as follows:
For every potential variable of the model starting with a single variant (base model name), generate every possible combination of models so far with this variable
Having generated those combinations, map each generated combination into ProductVariantOption.
So you might want to generate cross products of all lists of variables. This could be achieved with an .Aggregate which does .SelectMany inside (note that I have simplified the definition of the final output, but you can construct it as you want inside the .BuildModel method:
private static ProductVariantOption BuildModel(string[] modelParts) {
if (modelParts.Length == 1) {
return new ProductVariantOption {
Name = modelParts.Single()
};
}
var baseName = modelParts.First();
var variantParts = string.Join('/', modelParts.Skip(1));
return new ProductVariantOption {
Name = $"{baseName}-{variantParts}"
};
}
public static IList<ProductVariantOption> GetVariants(CreateModel model) {
// Prepare all possible variables from the model in advance
var allVariables = model.Variants.Select(v => v.Options.Split(",")).ToArray();
var initialParts = new List<string[]> { new[] { model.Name } };
// Generate cross product for every subsequent variant with initial list as a seed
// Every iteration of aggregate produces all possible combination of models with the new variant
var allModels = allVariables.Aggregate(initialParts, (variantsSoFar, variableValues) =>
variantsSoFar
.SelectMany(variant => variableValues.Select(variableValue => variant.Append(variableValue).ToArray()))
.ToList()
);
// Map all lists of model parts into ProductVariantOption
return allModels.Select(BuildModel).ToList();
}
This approach has the benefit of being able to handle any amount of potential variables including cases where there are no variables (in this case only a single variant is produced - In your example it would be just "Jean")
Complete running example:
https://dotnetfiddle.net/XvkPZQ
I'm not sure why you have the name and SKU fields identical, but using your model, you need to initialize a list of the first level product variants, and then use that list as product variants so far and add the next variant to every product variant in the list.
I just used a List<string> to store the name so far, and added to it each variant's options:
var productVariantsName = model.Variants[0].Options.Split(',')
.Select(o => $"{model.Name}-{o}")
.ToList();
foreach (var variant in model.Variants.Skip(1)) {
var pvNameSoFar = productVariantsName;
productVariantsName = new();
foreach (var pvName in pvNameSoFar) {
foreach (var option in variant.Options.Split(','))
productVariantsName.Add($"{pvName}/{option}");
}
}
var productOptions = productVariantsName
.Select(pvName => new ProductVariantOption(pvName, pvName))
.ToList();
You can also do this with LINQ using the CartesianProduct LINQ Method from the answer (I use a slightly different lambda version).
With that defined, you can do:
var productOptions = model.Variants
.Select(v => v.Options.Split(','))
.CartesianProduct()
.Select(vs => $"{model.Name}-{vs.Join("/")}")
.Select(pvName => new ProductVariantOption(pvName, pvName))
.ToList();
PS: This uses the obvious definition for the Join string extension method.
For completeness, here are the extensions used:
public static class IEnumerableExt {
public static string Join(this IEnumerable<string> ss, string sep) => String.Join(sep, ss);
public static IEnumerable<T> AsSingleton<T>(this T item) => new[] { item };
// ref: https://stackoverflow.com/a/3098381/2557128
public static IEnumerable<IEnumerable<T>> CartesianProduct<T>(this IEnumerable<IEnumerable<T>> sequences) =>
sequences.Aggregate(Enumerable.Empty<T>().AsSingleton(),
(accumulator, sequence) => accumulator.SelectMany(_ => sequence,
(accseq, item) => accseq.Append(item)));
}
How to add rows into datatable without foreach loop by reading records from csv ?
var records = File.ReadLines(FD.FileName, Encoding.UTF8).Skip(1);
//not working because datatable is not thread safe
Parallel.ForEach(records, record => dataTable.Rows.Add(record.Split(',')));
//using for each is taking aroung 13 sec for 45000 records, need a better solution than this.
foreach (var record in records)
{
dataTable.Rows.Add(record.Split(','));
}
Have you tried using a library like LINQ to CSV?
LINQ to CSV library
I have used it recently and its pretty good.
Basically you set the structure in your class like below
using LINQtoCSV;
using System;
class Product
{
[CsvColumn(Name = "ProductName", FieldIndex = 1)]
public string Name { get; set; }
[CsvColumn(FieldIndex = 2, OutputFormat = "dd MMM HH:mm:ss")]
public DateTime LaunchDate { get; set; }
[CsvColumn(FieldIndex = 3, CanBeNull = false, OutputFormat = "C")]
public decimal Price { get; set; }
[CsvColumn(FieldIndex = 4)]
public string Country { get; set; }
[CsvColumn(FieldIndex = 5)]
public string Description { get; set; }
}
There after use the below code to read the file
CsvFileDescription inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
CsvContext cc = new CsvContext();
IEnumerable<Product> products =
cc.Read<Product>("products.csv", inputFileDescription);
// Data is now available via variable products.
var productsByName =
from p in products
orderby p.Name
select new { p.Name, p.LaunchDate, p.Price, p.Description };
// or ...
foreach (Product item in products) { .... }
Let me know how it goes
I'm sure, that you don't need to display 50000 of rows at once.
Just because this is not human-readable.
Consider pagination: load first N rows, then, when user clicks "Next page" button, load next N rows, and so on.
Having this class for data row:
public class MyDataRow
{
public int Id { get; set; }
public string Name { get; set; }
}
pagination demo will be like this:
public partial class Form1 : Form
{
private const int PageSize = 10;
private readonly BindingList<MyDataRow> dataRows;
private readonly IEnumerator<string> csvLinesEnumerator;
public Form1()
{
InitializeComponent();
// start enumeration
csvLinesEnumerator = ReadLines().Skip(1).GetEnumerator();
// assign data source to DGV
dataRows = new BindingList<MyDataRow>();
dataGridView1.DataSource = dataRows;
// fetch first page
FetchNextPage();
}
private void button1_Click(object sender, EventArgs e) => FetchNextPage();
private void FetchNextPage()
{
// disable notifications
dataRows.RaiseListChangedEvents = false;
try
{
dataRows.Clear();
while (dataRows.Count < PageSize)
{
if (csvLinesEnumerator.MoveNext())
{
// parse next line and make row object from line's data
var lineItems = csvLinesEnumerator.Current.Split(',');
var row = new MyDataRow
{
Id = int.Parse(lineItems[0]),
Name = lineItems[1]
};
// add next row to the page
dataRows.Add(row);
}
else
{
break;
}
}
}
finally
{
// enable notifications and reload data source into DGV
dataRows.RaiseListChangedEvents = true;
dataRows.ResetBindings();
}
}
/// <summary>
/// This is just File.ReadLines replacement for testing purposes
/// </summary>
private IEnumerable<string> ReadLines()
{
for (var i = 0; i < 50001; i++)
{
yield return $"{i},{Guid.NewGuid()}";
}
}
}
UI:
Another option is to use DataGridView.VirtualMode, but it mostly optimized for cases, when you know the total number of rows (e.g., you can count them via SQL query), which isn't true for CSV files.
I want to get all these values from table Trouble and paste into Combobox.
I have this code,but i get System.ObjectDisposedException error
var items = db.Trouble.Where(u => u.id_Проблемы > 0).Select(u => u.id_Проблемы);
id_Trouble_box.Items.Add(items);
You should force Immediately excution by .ToArray() or ToList()
var items = db.Trouble.Where(u => u.id_Проблемы > 0).Select(u => u.id_Проблемы).ToArray();
id_Trouble_box.Items.Add(items);
Read the following thread to have a better understanding.
LINQ performance - deferred v/s immediate execution
Add a .ToArray() at the end. Perhaps you have it adding items outside a using statement.
If you want to select them all then you don't need a Where criteria:
var items = db.Trouble();
and to add to a combo you could set the DataSource:
var items = db.Trouble();
id_Trouble_box.DataSource = items.ToList();
id_Trouble_box.DisplayMember = "columnToShow"; // display column
id_Trouble_box.ValueMember = "id_Проблемы"; // id to get on selection as a value
EDIT: For those who don't understand why this answer fixes the error:
string defaultConString = #"server=.\SQLExpress;Database=Northwind;Trusted_Connection=yes;";
void Main()
{
Form f = new Form();
ComboBox cb = new ComboBox { Top = 10, Left = 10 };
f.Controls.Add(cb);
using (var ctx = new MyContext(defaultConString))
{
var items = ctx.Customers.Where(c => c.ContactName.Contains("a")).Select(c => c.CompanyName);
cb.Items.Add(items);
}
f.Show();
}
public class MyContext : DbContext
{
public MyContext(string connectionString)
: base(connectionString)
{ }
public DbSet<Customer> Customers { get; set; }
}
public class Customer
{
[Key]
public string CustomerId { get; set; }
public string CompanyName { get; set; }
public string ContactName { get; set; }
// ...
// public virtual List<Order> Orders { get; set; }
}
I solved the problem this way
var items = db.Trouble.Where(u => u.id_Проблемы > 0).Select(u => u.id_Проблемы).ToArray();
for(int i = 0; i < items.Length; i++)
{
id_Trouble_box.Items.Add(items[i]);
}