Develop Reference

Kafka track offsets in all partitions

Will try to explain what I am trying to achieve.
All I know is topic name and by that I must scale down to partitions. First I try
consumer.Subscribe(topics)
And
consumer.Assignement
But if there is no delay between calls It returns empty list.
I could use consumer.Assign(..)
But I dont know exact partitions, offsets yet.
Next when Iam able to go down to partition I need to get offsets low/high by time range.
For example topi "test" has 5 partitions, and I need to extract all messages info (partition, offsets) for messages being inserted from 10:00 to 10:05.
If any additional info needed, just let me know.
Thanks

I'm not 100% clear on what you are aiming for but some information about assignment may help here.
The assignment() method returns an empty list prior to the first time the poll method is called on a consumer when joining the group, or after a rebalance - this is because when partitions are automatically assigned the consumer only finds out the assignment as one of the steps of the poll method, prior to fetching actual records.
You can find out the actual assigned partitions either by calling poll at least once before calling assignment() - I think that is what you have discovered - else by passing a ConsumerRebalanceListener when calling subscribe(). The onPartitionsAssigned method is called during the poll - essentially a callback - with an argument that is the collection of newly assigned partitions. This enables your code to discover the current assignment before any records are fetched.
Hope this helps a bit - I have written up a blog post about this aspect of assignment but haven't yet published it - I'll add a link when I do, if it sounds like this is the issue you are facing.

I went for a bit different approach.
From IAdminClient load metada to get all available partitions for it.
Create TopicPartitionTimestamp with start timestamp I need to consume from.
Assign to TopicPartitionTimestamp and consume from it.
Also I chose to start every partition consumption on different Thread.

C# Combobox - "deleted entities"?

It's a struggle with any application that provides select fields, that are populated by a certain datasource: Everything works fine in the first place, but once the application ages, some older entries might be deleted, leading to the problem that prior select fields can no longer access the entity in question.
Opening a view, where a select points to an already deleted datarow will (best case) show empty string.
We designed our system in a way, that deletions are not real delete-operations, but only the setting of a deleted flag. (So, all the information is still there)
However, when using Databindings along with C# (or even if not) the most blatant use case is still not covered by general mechanics (I assume):
Select-Field should show all NOT-Deleted-Entities while creating a new object containing references to the entity in question.
Select-Field (populated the very same way) should show the "deleted" entity, if it was selected "days/months/years" ago.
Is there a "handy" solution to this?
Currently we are using a "Proxy-Method" for every datasource, which will reload the data of the deleted entity, if it's not in the "available data" collection - but it's hard to believe there is no better way to deal with this, as this problem applies for almost every language out there?

In a normalized database you would have a constraint with ON DELETE NO ACTION/RESTRICT event that would prevent removal of a referenced element from the list. It would force you to decide what is to be done with the referencing rows.
With your manually-controlled deletions this could have been covered by a trigger. As none of these were implemented, you are left with only one thing to do: updating the dropdown with the selected option before rendering the UI. My approach (in Java, I'm not good at C#):
List<String> options = getNonDeletedWhatever();
if (!options.contains(currentEntity.getWhatever())) {
options.add(currentEntity.getWhatever()); // This optionally inserts an outdated value
}
or simply:
Set<String> options = getNonDeletedWhatever();
options.add(currentEntity.getWhatever()); // This optionally inserts an outdated value

I solve it by creating a list of available (non-deleted) items and if the selected item is a deleted one, then I add that item to the list.
This list becomes the data source for my dropdown.

Best approach to track Amount field on Invoice table when InvoiceItem items change?

I'm building an app where I need to store invoices from customers so we can track who has paid and who has not, and if not, see how much they owe in total. Right now my schema looks something like this:
Customer
- Id
- Name
Invoice
- Id
- CreatedOn
- PaidOn
- CustomerId
InvoiceItem
- Id
- Amount
- InvoiceId
Normally I'd fetch all the data using Entity Framework and calculate everything in my C# service, (or even do the calculation on SQL Server) something like so:
var amountOwed = Invoice.Where(i => i.CustomerId == customer.Id)
.SelectMany(i => i.InvoiceItems)
.Select(ii => ii.Amount)
.Sum()
But calculating everything every time I need to generate a report doesn't feel like the right approach this time, because down the line I'll have to generate reports that should calculate what all the customers owe (sometimes go even higher on the hierarchy).
For this scenario I was thinking of adding an Amount field on my Invoice table and possibly an AmountOwed on my Customer table which will be updated or populated via the InvoiceService whenever I insert/update/delete an InvoiceItem. This should be safe enough and make the report querying much faster.
But I've also been searching some on this subject and another recommended approach is using triggers on my database. I like this method best because even if I were to directly modify a value using SQL and not the app services, the other tables would automatically update.
My question is:
How do I add a trigger to update all the parent tables whenever an InvoiceItem is changed?
And from your experience, is this the best (safer, less error-prone) solution to this problem, or am I missing something?

There are many examples of triggers that you can find on the web. Many are poorly written unfortunately. And for future reference, post DDL for your tables, not some abbreviated list. No one should need to ask about the constraints and relationships you have (or should have) defined.
To start, how would you write a query to calculate the total amount at the invoice level? Presumably you know the tsql to do that. So write it, test it, verify it. Then add your amount column to the invoice table. Now how would you write an update statement to set that new amount column to the sum of the associated item rows? Again - write it, test it, verify it. At this point you have all the code you need to implement your trigger.
Since this process involves changes to the item table, you will need to write triggers to handle all three types of dml statements - insert, update, and delete. Write a trigger for each to simplify your learning and debugging. Triggers have access to special tables - go learn about them. And go learn about the false assumption that a trigger works with a single row - it doesn't. Triggers must be written to work correctly if 0 (yes, zero), 1, or many rows are affected.
In an insert statement, the inserted table will hold all the rows inserted by the statement that caused the trigger to execute. So you merely sum the values (using the appropriate grouping logic) and update the appropriate rows in the invoice table. Having written the update statement mentioned in the previous paragraphs, this should be a relatively simple change to that query. But since you can insert a new row for an old invoice, you must remember to add the summed amount to the value already stored in the invoice table. This should be enough direction for you to start.
And to answer your second question - the safest and easiest way is to calculate the value every time. I fear you are trying to solve a problem that you do not have and that you may never have. Generally speaking, no one cares about invoices that are of "significant" age. You might care about unpaid invoices for a period of time, but eventually you write these things off (especially if the amounts are not significant). Another relatively easy approach is to create an indexed view to calculate and materialize the total amount. But remember - nothing is free. An indexed view must be maintained and it will add extra processing for DML statements affecting the item table. Indexed views do have limitations - which are documented.
And one last comment. I would strongly hesitate to maintain a total amount at any level higher than invoice. Above that level one frequently wants to filter the results in any ways - date, location, type, customer, etc. At this level you are approaching data warehouse functionality which is not appropriate for a OLTP system.

First of all never use triggers for business logic. Triggers are tricky and easily forgettable. It will be hard to maintain such application.
For most cases you can easily populate your reporting data via entity framework or SQL query. But if it requires lots of joins then you need to consider using staging tables. Because reporting requires data denormalization. To populate staging tables you can use SQL jobs or other schedule mechanism (Azure Scheduler maybe). This way you won't need to work with lots of join and your reports will populate faster.

Assign unique numbers in database when saving the form

I have a form where you can create an order and when you save it, is checking in the database (using oracle) for the last order number and is assigning the next one to the currently saved order. What I found is that if two users are saving a new order both in the same time or at few seconds apart, because of the connection speed my app is unable to assign different numbers for the newly two created orders. The problem is that both are checking in the same time the last assigned number and both orders get the same number..
I have some ideas but all of them have advantages and disadvantages..
To have the system wait a few seconds and check the order number when the user saves the order. But if both saved in the same time, the check will be done in the same time later and I guess that I will end up with the same problem..
To have the system check the order number (a check is run every time the treeview is refreshed) and see if it’s been duplicated and then let the user know via the treeview with some highlight, that it’s been duplicated. But if any documents are assigned to the order before the check, then I will end up with documents having a different number in the name and inside from the order to which is assigned..
To have the system check all order numbers periodically and give one of the duplicates a new order number, but Here is the same problem with the documents as at #2.. And also might cause some performance issue..
Assigning the order number when a user requests a new order not when he saves the order. I could have the system do Solution #1 along with this solution and recheck to see if the number is being used within the database and then reassign it a new one. Once again, if documents get assigned, someone has to go fix those.
One way of possibly stopping the documents from being assigned to duplicates is that the user is only allowed put some of the information and then save it or apply it and it does the recheck of #1, and then if it doesn't find anything, allow the user to add documents. This part of the solution could be applied possibly to any of the above but I don't want to delay the users work while is checking the numbers..
Please if you see any improvements to the ideas above or if you have new ones, let me know.
I need to find the best solution and as much as possible not to affect the user's current workflow..

If your Order ID is only a number you can use Oracle Sequence.
CREATE SEQUENCE order_id;
And before you save the record get a new order number.
SELECT order_id.NEXTVAL FROM DUAL;
See also Oracle/PLSQL: Sequences (Autonumber)

Generating unique ID for clients on a system running over a LAN in C#

I've a simple client registration system that runs over a network. The system is supposed to generate a unique three digit ID (primary key) with the current year concatenated (e.g. 001-2013). However, I've encountered the problem that the same primary keys being generated when two users from different computers (over a LAN) try to register different clients.
What if the user cancels the registration after an ID is already generated? I've to reuse that ID for another client. I've read about static variable but it didn't solve my problem. I really appreciate your ideas.

Unique and sequential IDs are hard to implement. To completely achive it you would have to serialize commiting creation of client information so ID generated only when data is actually stored, otherwise you'll endup with holes when something wrong happened during submittion.
If you don't need strict sequential numbers - giving out ranges of ID (1-22, 23-44,...) to each system is common approach. Instead of ranges you can give out lists of IDs to use ({1,3,233,234}, {235,236,237}) if you need to use as many IDs as possible.

Issue:
New item -001 is created, but not saved yet
New item -002 is created, but not saved yet
Item -001 is cancelled
What to do with ID -001?
The easiest solution is to simply not assign an ID until an item is definitely stored.
An alternative is, when finally saving an item, you look up the first free ID. If the item from step 2 (#2) is saved before the one from step 1, #2 gets ID -001. When #1 then gets saved, the saving logic sees that its claimed ID (-001) is in use, so it'll assign -002. So ID's get reassigned.
Finally you can simply find the next free ID when creating a new item. In the three steps described above, this'll mean you initially have a gap where -001 is supposed to be. If you now create a new item, your code will see -001 is unused and will assign that to the new item.
But, and that totally depends on your requirements which you didn't specify, now -001 was created later in time than -002, I do not know if that is allowed. Furthermore at any given moment you can have a gap in your numbering where an item has been cancelled. If it happens at the end of a reporting period, this will cause errors (-033, -034, -036).
You also might want to include an auto-incrementing primary key instead of this invoice number or whatever it is.