Develop Reference

OutputBuffer not working for large c# list

I'm currently using SSIS to do an improvement on a project. need to insert single documents in a MongoDB collection of type Time Series. At some point I want to retrieve rows of data after going through a C# transformation script. I did this:
foreach (BsonDocument bson in listBson)
{
OutputBuffer.AddRow();
OutputBuffer.DatalineX = (string) bson.GetValue("data");
}
But this piece of code that works great with small file does not work with a 6 million line file. That is, there are no lines in the output. The other following tasks validate but react as if they had received nothing as input.
Where could the problem come from?

Your OuputBuffer has DatalineX defined as a string, either DT_STR or DT_WSTR and a specific length. When you exceed that value, things go bad. In normal strings, you'd have a maximum length of 8k or 4k respectively.
Neither of which are useful for your use case of at least 6M characters. To handle that, you'll need to change your data type to DT_TEXT/DT_NTEXT Those data types do not require a length as they are "max" types. There are lots of things to be aware of when using the LOB types.
Performance can suck depending on whether SSIS can keep the data in memory (good) or has to write intermediate values to disk (bad)
You can't readily manipulate them in a data flow
You'll use a different syntax in a Script Component to work with them
e.g.
// TODO: convert to bytes
Output0Buffer.DatalineX.AddBlobData(bytes);
Longer example of questionable accuracy with regard to encoding the bytes that you get to solve at https://stackoverflow.com/a/74902194/181965

RFC_READ_TABLE throws Rfcabapexception after querying a lot of columns

Where I work, we have two systems that use SAP, one using Delphi and another using c#. I'm implementing the c# and both have the same problem, when I query for a great amount of columns using RFC_READ_TABLE, depending on the table ( usually 60+ ), it returns a Rfcabapexception with no description and no Inner Exception, just a title. What is causing this exception and what can I do to prevent it?

The function module RFC_READ_TABLE has to convert the data to a generic format because "really generic types" like DATA or STANDARD TABLE are not supported for RFC communication. Because of this, the outout is transmitted as a series of table lines, each a character field up to 512 characters in length.
This has several consequences:
If the total size of all fields you requested exceeds 512 characters, you will get a short dump (check with transaction ST22) and the exception you mentioned.
If you try to read fields that can not be converted to character fields and/or do hot have a fixed-length (!) character representation, bad things will happen. Most likely, RFC_READ_TABLE will either abort with a short dump or barf all over your output data.
You can bypass the first problem by slicing the table vertically and reading groups of columns sequentially. Be aware that RFC_READ_TABLE is not guaranteed to always return the data in the same order when stitching the results back together again. Also be aware that you might run into violations of transaction isolation, depending on how often the data you read changes.

Best Way to Load a File, Manipulate the Data, and Write a New File

I have an issue where I need to load a fixed-length file. Process some of the fields, generate a few others, and finally output a new file. The difficult part is that the file is of part numbers and some of the products are superceded by other products (which can also be superceded). What I need to do is follow the superceded trail to get information I need to replace some of the fields in the row I am looking at. So how can I best handle about 200000 lines from a file and the need to move up and down within the given products? I thought about using a collection to hold the data or a dataset, but I just don't think this is the right way. Here is an example of what I am trying to do:
Before
Part Number List Price Description Superceding Part Number
0913982 3852943
3852943 0006710 CARRIER,BEARING
After
Part Number List Price Description Superceding Part Number
0913982 0006710 CARRIER,BEARING 3852943
3852943 0006710 CARRIER,BEARING
As usual any help would be appreciated, thanks.
Wade

Create structure of given fields.
Read file and put structures in collection. You may use part number as key for hashtable to provide fastest searching.
Scan collection and fix the data.
200 000 objects from given lines will fit easily in memory.
For example.
If your structure size is 50 bytes then you will need only 10Mb of memory. It is nothing for modern PC.

How to read a text file into a List in C#

I have a text file that has the following format:
1234
ABC123 1000 2000
The first integer value is a weight and the next line has three values, a product code, weight and cost, and this line can be repeated any number of times. There is a space in between each value.
I have been able to read in the text file, store the first value on the first line into a variable, and then the subsequent lines into an array and then into a list, using first readline.split('').
To me this seems an inefficient way of doing it, and I have been trying to find a way where I can read from the second line where the product codes, weights and costs are listed down into a list without the need of using an array. My list control contains an object where I am only storing the weight and cost, not the product code.
Does anyone know how to read in a text file, take in some values from the file straight into a list control?
Thanks

What you do is correct. There is no generalized way of doing it, since what you did is that you descirbed the algorithm for it, that has to be coded or parametrized somehow.

Since your text file isn't as structured as a CSV file, this kind of manual parsing is probably your best bet.
C# doesn't have a Scanner class like Java, so what you wan't doesn't exist in the BCL, though you could write your own.

The other answers are correct - there's no generalized solution for this.
If you've got a relatively small file, you can use File.ReadAllLines(), which will at least get rid of a lot cruft code, since it'll immediately convert it to a string array for you.

If you don't want to parse strings from the file and to reserve an additional memory for holding split strings you can use a binary format to store your information in the file. Then you can use the class BinaryReader with methods like ReadInt32(), ReadDouble() and others. It is more efficient than read by characters.
But one thing: binary format is bad readable by humans. It will be difficult to edit the file in the editor. But programmatically - without any problems.

storing large data in string

i am trying to store large data more than 255 characters in a string datatype but it truncates after 255. how can i achive this basically i need to pass this data to database

C# strings do not have any particular character limit. However the database column you are writing to may have a limit. If you are storing large amounts of data, you should use a BLOB column instead of an ordinary varchar type.

StringBuilder class
Like they said the string class is not limited, but you can do this for large strings. I feel it handles them better.
StringBuilder sb = new StringBuilder();
sb.append("Some text...");
sb.append("more text...");
sb.append("even more text!");
sb.toString();

Okay, it sounds like you have several different technologies involved - Excel, XML, databases etc. Try to tackle just one at a time. First read the data out of Excel, and make sure you can do that without any truncation.
Write a small console app which will read the value, then write it to the console - and its length. If that works, you know the problem isn't in Excel.
Next you can write a small console app with hardcoded input data (so you don't need to keep using interop with Excel) and write the XML from that, or whatever your next stage is.
Basically, take the one big problem ("when I read data from Excel and write it to the database it truncates long values") and split it into smaller and smaller ones until you've found what's wrong.

The string type does not limit strings to 255 characters. Your database column must be 255 characters.

I know that c# strings can hold much longer data than that. If the truncation occurs on commiting to DB, check the length constraint on ur Db field

The problem lies in the Excel part; .Character has a 255 characters limitation.
To read the complete text from a shape the following VBA syntax would do:
Worksheets("YourSheet").Shapes("Shape1").OLEFormat.Object.Text