Develop Reference

OutputBuffer not working for large c# list

I'm currently using SSIS to do an improvement on a project. need to insert single documents in a MongoDB collection of type Time Series. At some point I want to retrieve rows of data after going through a C# transformation script. I did this:
foreach (BsonDocument bson in listBson)
{
OutputBuffer.AddRow();
OutputBuffer.DatalineX = (string) bson.GetValue("data");
}
But this piece of code that works great with small file does not work with a 6 million line file. That is, there are no lines in the output. The other following tasks validate but react as if they had received nothing as input.
Where could the problem come from?

Your OuputBuffer has DatalineX defined as a string, either DT_STR or DT_WSTR and a specific length. When you exceed that value, things go bad. In normal strings, you'd have a maximum length of 8k or 4k respectively.
Neither of which are useful for your use case of at least 6M characters. To handle that, you'll need to change your data type to DT_TEXT/DT_NTEXT Those data types do not require a length as they are "max" types. There are lots of things to be aware of when using the LOB types.
Performance can suck depending on whether SSIS can keep the data in memory (good) or has to write intermediate values to disk (bad)
You can't readily manipulate them in a data flow
You'll use a different syntax in a Script Component to work with them
e.g.
// TODO: convert to bytes
Output0Buffer.DatalineX.AddBlobData(bytes);
Longer example of questionable accuracy with regard to encoding the bytes that you get to solve at https://stackoverflow.com/a/74902194/181965

ServiceStack TimeSpan Serialization can't be Cast by SQL to Time(7)

When TimeSpan properties are serialized they end up looking like this: -PT10M, PT30S, or PT6H. My real problem is that in MSSQL I can't use the Cast function to get them to a Time so if there's an easy way to do that I'm open to that solution however I believe the easiest thing will be to figure out how to get ServiceStack to change it's serialization behavior and instead serialize them like this -0:10:00, 0:00:30, or 6:00:00 respectively.
The way I'm serializing them now is to just call MyObject.ToJson().
Is changing the serialization the way to go and if so how can I do that with ServiceStack?
UPDATE
Please ignore the negative time values I'm aware SQL's Time type can only hold positive values. This actually doesn't affect what I'm trying to do at the moment. I have other fields where the time can sometimes be negative so I included them in my question as an example but it's not really relevent to my question.
After some more testing I can confirm that values stored like this like this -0:10:00, 0:00:30, or 6:00:00 deserialize fine. Still, is there a way to force them to serialize to this more easily readable value like this?
Here's almost the same question for Newtonsoft: https://stackoverflow.com/questions/39876232

C# .Net Linq and converting datetime to formatted sting

I have the query
var result = (from myView in db.loginSessions
where myView.machine.ToUpper().Equals(machine.ToUpper())
&& myView.start >= myStart
&& myView.end <= myEnd
orderby myView.start
select new loginSessionList {
Id = myView.id,
Machine = myView.machine.ToUpper(),
Start = myView.start.ToString("u"),
End = myView.end.ToString("u"),
User = myView.username
}).ToList();
I get ArgumentOutOfRange exceptions on the datetime conversions. I have tried different ToString conversion strings. I have tried the other To... date->string conversions offered by Intellisence. I have tried Convert.string(myView.start). Nothing has worked. I have googled and have found advice using all the things I have tried. Do I have to post-process the generated list?
What have I missed?

I have 3 rules for dealing with DateTimes that served me well:
Always store, retreive and transmit the UTC value. Translating into the proper local Timezone is the job of ToString(), wich asks Windows for the users timezone. You do not want to add timezones to your troubles.
Avoid store, retreive or transmission of DateTimes as strings. Keep it in proper types whenever possible
If you can not follow rule 2 (like when you deal with Serialsiation), at least pick a fixed Culture Format and String encoding on all endpoints. You do not want to get different Cultures or faulty implied Encodings to your existing troubles

So, the answer to my issue had nothing to do with Linq or .Net. #JoeEnos put me on the right track as mentioned in my comment to his comment. I had created a class to receive each row of query result. The date fields were initially DateTime types but I started having issues. I changed those fields to string and then ended up asking my question above.
Back when the receiving class still had the DateTime fields, all of the fields had lowercase names. I must have applied some sort of code formatting that CamelCased the field names. This meant that after serializing the results the CamelCased names could not be found and the javascript didn't care.
I fixed the field names in my js code, changed the field data types back to DateTime and all is good.
Thanks

Design better text file format for reading mixed type data of variable length

I am designing a text file format to be read in C#. I have a need to store types: int, double and string on a single line. I'm planning to use a .CSV format so the file can be manually opened and read. A particular record may have say 8 known types, then a variable number of "indicator" combinations of either (string, int, double) or (string, int, double, double), and some lines may include no "indicators". Thus, each record is may be of variable length.
In VB6 I would just input the data, split the data, into a variant array, then determine the number of elements on that line in the array, and use the ***VarType function to determine if the final "indicator" variables are string, int, or double and parse the field accordingly.
There may be a better way to design a text file and that may be the best solution. If so I'm interested in hearing ideas. I have searched but found no questions that specifically talk about reading variable length lines of text with mixed type into C#.
If a better format is not forthcoming, is there a way to duplicate the VB6 VarType function within C# as described two paragraphs above***? I can handle the text file reading and line splitting easily in C#.

you could use either json or xml as they are well supported in .NET and have automatic serialization capabilities

First I agree with Keith's suggestion to use Xml or JSON. You are reinventing a wheel here. This page has an introductory example of how to serialize objects to a file and some links to more info.
If you need to stick with your own file format and custom serialization/deserialization however take a look at the Convert class, as well as the various TryParse methods which hang off of the intrinsic value types like int and double.

c#: tryparse vs convert

Today I read an article where it's written that we should always use TryParse(string, out MMM) for conversion rather than Convert.ToMMM().
I agree with article but after that I got stuck in one scenario.
When there will always be some valid value for the string and hence we can also use Convert.ToMMM() because we don't get any exception from Convert.ToMMM().
What I would like to know here is: Is there any performance impact when we use TryParse because when I know that the out parameter is always going to be valid then we can use Convert.ToMMM() rather TryParse(string, out MMM).
What do you think?

If you know the value can be converted, just use Parse(). If you 'know' that it can be converted, and it can't, then an exception being thrown is a good thing.
EDIT: Note, this is in comparison to using TryParse or Convert without error checking. If you use either of the other methods with proper error checking then the point is moot. I'm just worried about your assumption that you know the value can be converted. If you want to skip the error checking, use Parse and die immediately on failure rather than possibly continuing and corrupting data.

When the input to TryParse/Convert.ToXXX comes from user input, I'd always use TryParse. In case of database values, I'd check why you get strings from the database (maybe bad design?). If string values can be entered in the database columns, I'd also use TryParse as you can never be sure that nobody modifies data manually.
EDIT
Reading Matthew's reply: If you are unsure and would wrap the conversion in a try-catch-block anyway, you might consider using TryParse as it is said to be way faster than doing a try-catch in that case.

There is significant difference regarding the developing approach you use.
Convert: Converting one "primitive" data in to another type and corresponding format using multiple options
Case and point - converting an integer number in to its bit by bit representation. Or hexadecimal number (as string) in to integer, etc...
Error Messages : Conversion Specific Error Message - for problems in multiple cases and at multiple stages of the conversion process.
TryParse: Error-less transfer from one data format to another. Enabling T/F control of possible or not.
Error Messages: NONE
NB: Even after passing the data in to a variable - the data passed is the default of the type we try to parse in to.
Parse: in essence taking some data in one format and transfer it in to another. No representations and nothing fancy.
Error Messages: Format-oriented
P.S. Correct me if I missed something or did not explain it well enough.