Send and receive large MTOMAttachments to IBM Content Manager WebService - c#

When trying to upload large files (100-500MB) I get the familiar "OutOfMemoryException" which is caused by trying to read the whole file into memory at once (asked and answered here on stackoverflow). I know that I should use a stream or divide the file into smaller parts. Changing the proxy code manually is an option, if that helps. I use the specific webservice (CMWebService).
Since I am unable to change IBMs code, is there any way to send the file in smaller parts? I have already found the classes UpdateItemRequestAdd and UpdateItemRequestAddPart but I can't get them to work. Unfortunately, there are also no samples available by IBM.
Receiving files pose the same problem, and I have not been able to find any classes that could help me there.
This is the code that I am currently using to upload files:
string resources0 = "tiffFileContent";
string resources1 = "image/tiff";
string resources2 = #"D:\myImageFile.tif";
CreateItemRequest createRequest = new CreateItemRequest()
{
AuthenticationData = data,
Item = new CreateItemRequestItem()
{
ItemXML = new ItemXML()
{
MYITEMTYPE = new MYITEMTYPE()
{
ArchiveId = "4719",
ICMBASE = new ICMBASE[] {
new ICMBASE(){
resourceObject = new LobObjectType()
{
label = new LobObjectTypeLabel()
{
name= resources0
},
MIMEType = resources1,
originalFileName = resources2
},
}
}
}
},
},
mtomRef = new MTOMAttachment[] { new MTOMAttachment() {
ID = resources0,
MimeType = resources1,
Value = System.IO.File.ReadAllBytes(resources2), // Error on large files
}},
};
var createReply = service.CreateItem(createRequest);

We "resolved" this by telling our customer to get a more potent system with more RAM. With 4-8GB of RAM we were able to upload files up to 200MB without problem.
Upon receiving large files, the Java-HeapSize in IBM Content Manager had to be increased.
http://www.mkyong.com/websphere/how-to-increase-websphere-jvm-memory/

Related

How do you add thumbnails to an embed message in DSharpPlus?

I have been bashing my head against the wall for 8 hours now, trying to figure out how to add thumbnails to embeded messages. I've been following this tutorial that uses ThumbnailURL, something that doesn't exist in the context he's using it for. Instead I just have Thumbnail which does not take a string, but rather an EmbedThumbnail which I can't access at all.
var avatarEmbed = new DiscordEmbedBuilder
{
Title = "Enter the game.",
Thumbnail = ctx.Client.CurrentUser.AvatarUrl,
Color = DiscordColor.Azure,
};
Please don't send links to docs and tutorials. I've tried. I've really tried. But this just seems like an issue that is very recent and therefore resources on it would be next to none. Also the docs don't seem to have example code, making it all more frustrating.
Not sure about your version but I use the latest nightly build.
and I do it like this
var builder = new DiscordEmbedBuilder
{
Title = "Title Here",
Color = DiscordColor.Azure
};
builder.WithThumbnail(ctx.Client.CurrentUser.AvatarUrl);
but you can also do it this way
var builder = new DiscordEmbedBuilder
{
Title = "Title Here",
Color = DiscordColor.Azure,
Thumbnail = new DiscordEmbedBuilder.EmbedThumbnail
{
Url = ctx.Client.CurrentUser.AvatarUrl
}
};

Usage of Document Extraction cognitive skill

I am trying to utilize Azure Cognitive services to perform basic document extraction.
My intent is to input PDFs and DOCXs (and possibly some other files) into the Cognitive Engine for parsing, but unfortunately, the implementation of this is not as simple as it seems.
According to the documentation (https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-document-extraction#sample-definition), I must define the skill and then I should be able to input files, but there is no examples on how this should be done.
So far I have been able to define the skill but I am still not sure where I should be dropping the files into.
Please see my code below, as it seeks to replicate the same data structure shown in the example code (albeit using the C# Library)
public static DocumentExtractionSkill CreateDocumentExtractionSkill()
{
List<InputFieldMappingEntry> inputMappings = new List<InputFieldMappingEntry>
{
new("file_data") {Source = "/document/file_data"}
};
List<OutputFieldMappingEntry> outputMappings = new List<OutputFieldMappingEntry>
{
new("content") {TargetName = "extracted_content"}
};
DocumentExtractionSkill des = new DocumentExtractionSkill(inputMappings, outputMappings)
{
Description = "Extract text (plain and structured) from image",
ParsingMode = BlobIndexerParsingMode.Text,
DataToExtract = BlobIndexerDataToExtract.ContentAndMetadata,
Context = "/document",
};
return des;
}
And then I build on this skill like so:
_indexerClient = new SearchIndexerClient(new Uri(Environment.GetEnvironmentVariable("SearchEndpoint")), new AzureKeyCredential(Environment.GetEnvironmentVariable("SearchKey"));
List<SearchIndexerSkill> skills = new List<SearchIndexerSkill> { Skills.DocExtractionSkill.CreateDocumentExtractionSkill() };
SearchIndexerSkillset skillset = new SearchIndexerSkillset("DocumentSkillset", skills)
{
Description = "Document Cracker Skillset",
CognitiveServicesAccount = new CognitiveServicesAccountKey(Environment.GetEnvironmentVariable("CognitiveServicesKey"))
};
await _indexerClient.CreateOrUpdateSkillsetAsync(skillset);
And... then what?
There is no clear method that would fit what I believe the next stage, actually parsing documents.
What is the next step from here to begin dumping files into the _indexerClient (of type SearchIndexerClient)?
As the next stage shown in the documentation is:
{
"values": [
{
"recordId": "1",
"data":
{
"file_data": {
"$type": "file",
"data": "aGVsbG8="
}
}
}
]
}
Which is not clear as to where I would be doing this.
According to the document that you have mentioned. They are actually trying to get the output through postman. They are using a GET Method to receive the extracted document content by sending JSON request to the mentioned URL(i.e. Cognitive skill url) and the files/documents are needed to be uploaded to your storage account in order to get extracted.
you can follow this tutorial to get more insights.

Running a TuesPechkin converter more than once

I'm using TuesPechkin for my web application, which I'm testing locally on IIS with VS2013. The user clicks a button and the page's current HTML is saved to a PDF file, which is then emailed out. This process is going to be run regularly as the site's data changes.
When converter.Convert(document) first runs, it converts without problem. Every subsequent attempt, however, results in the process hanging and me needing to restart VS.
Below is some default code I've been using to test.
public void MakePDF()
{
var document = new HtmlToPdfDocument
{
GlobalSettings =
{
ProduceOutline = true,
DocumentTitle = "Pretty Websites",
PaperSize = PaperKind.A4, // Implicit conversion to PechkinPaperSize
Margins =
{
All = 1.375,
Unit = TuesPechkin.Unit.Centimeters
}
},
Objects = {
new ObjectSettings { HtmlText = "<h1>Pretty Websites</h1><p>This might take a bit to convert!</p>" }
}
};
IConverter converter =
new ThreadSafeConverter(
new PdfToolset(
new Win32EmbeddedDeployment(
new TempFolderDeployment())));
byte[] result = converter.Convert(document);
}
Can anyone point me in the right direction on this? Most of my troubleshooting so far has led to some discussions on threading and pooling, but no concrete code solutions for running TuesPechkin more than once.
Have you tried the ThreadSafeConverter? The StandardConverter is only suitable for small console apps.
IConverter converter =
new ThreadSafeConverter(
new RemotingToolset<PdfToolset>(
new Win32EmbeddedDeployment(
new TempFolderDeployment())));
byte[] result = converter.Convert(document);
Note that you should keep the converter somewhere static, or as a singleton instance (as mentioned here).
Since this app on IIS, could get singleton converter, and use RemotingToolset
var toolSet = new RemotingToolset<PdfToolset>(winAnyCpuEmbeddedDeployment);
// Then
using TuesPechkin.Wkhtmltox.AnyCPU;
...
var converter = PDFHelper.Factory.GetConverter();
var result = converter.Convert(This.Document);
Reference : https://github.com/tloy1966/TuesPechkin

Copying annotations leads to a corrupted attachment

I'm implementing a plugin (POST Quote Create, Synchronous, Sandbox) to make it so that Notes are copied to the new record when a quote is revised.
My plugin boils down to this (snippet):
var serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
var Service = serviceFactory.CreateOrganizationService(context.UserId);
var notesQuery = new QueryExpression("annotation");
notesQuery.ColumnSet = new ColumnSet(true);
notesQuery.Criteria = new FilterExpression
{
FilterOperator = LogicalOperator.And,
Conditions =
{
new ConditionExpression("objecttypecode", ConditionOperator.Equal, "quote"),
new ConditionExpression("objectid", ConditionOperator.Equal, revisedQuoteId)
}
};
var notes = Service.RetrieveMultiple(notesQuery).Entities;
foreach (var n in notes)
{
var newNote = new Entity("annotation");
newNote.Attributes.Add("ownerid", n.GetAttributeValue<EntityReference>("ownerid"));
newNote.Attributes.Add("objectid", new EntityReference("quote", sourceEntity.Id));
newNote.Attributes.Add("objecttypecode", "quote");
newNote.Attributes.Add("subject", n.GetAttributeValue<string>("subject"));
newNote.Attributes.Add("notetext", n.GetAttributeValue<string>("notetext"));
newNote.Attributes.Add("isdocument", n.GetAttributeValue<bool>("isdocument"));
if (n.GetAttributeValue<bool>("isdocument"))
{
newNote.Attributes.Add("filesize", n.GetAttributeValue<int>("filesize"));
newNote.Attributes.Add("documentbody", n.GetAttributeValue<string>("documentbody"));
newNote.Attributes.Add("filename", n.GetAttributeValue<string>("filename"));
newNote.Attributes.Add("mimetype", n.GetAttributeValue<string>("mimetype"));
}
Service.Create(newNote);
}
Basically, I copy everything over, including an eventual attachment. Everything seems fine, the new revision shows fields, detail records and notes properly... everything but the attachment of the notes.
If I have a single note, with an attached test.txt which content is:
Test attachment
The OrganizationData service reads as follows:
<d:FileName>test.txt</d:FileName>
<d:FileSize m:type="Edm.Int32">39</d:FileSize>
<d:DocumentBody>H4sIAAAMaVMA/wtJLS5RSCwpSUzOyE3NK+HlAgCLmj1zEQAAAA==</d:DocumentBody>
Its "clone" has the correct subject and text, and also shows a test.txt attached which content is
‹ iS ÿI-.QH,)ILÎÈMÍ+áå ‹š=s
mimetype and filesize (while checking odata, I noticed that filesize is not actually correct!) appear to be correct (aka: the same as the original note I'm trying to copy), but OData seems to confirm something's off (it's different!):
<d:FileName>test.txt</d:FileName>
<d:FileSize m:type="Edm.Int32">60</d:FileSize
<d:DocumentBody>H4sIAED6aVMA/5Pv5mBg4MkMZvjP7amrF+iho+npc+6E71nth0+ZGLpn2RYLMjAwAABXqCwTJQAAAA==</d:DocumentBody>
The test.txt file was created from a command prompt (COPY CON test.txt, type, CTRL+Z).
I tried to change the file, and created a test.pdf through PDFCreator: AcroRead in turn whines and says the document is corrupted (so it seems like the issue is mimetype-agnostic).
I also tried re-implementing the same code through early binding (via the CRMSVCUTIL-generated classes) but it yields the exact same result (garbage instead of the attachment contents).
I attempted to hand-craft the documentbody like this:
// "VGVzdCBhdHRhY2htZW50" is Base64 for "Test attachment"
newNote.Attributes.Add("documentbody", "VGVzdCBhdHRhY2htZW50");
and the created file is correct.
I can't figure out what's going on: as far as I know, documentbody is supposed to be a Base64-encoded string which (again, as far as I know) shouldn't be any different when copied around. What am I missing ?
For reference, CRM is updated to UR13 but I repro'd it on a UR16 environment.
EDIT: Does NOT work (only for CRM 4)
Try this (not verified):
var notes = Service.RetrieveMultiple(notesQuery).Entities;
foreach (var newNote in notes)
{
newNote.annotationid = null;
newNote.Attributes.Add("objectid", new EntityReference("quote", sourceEntity.Id));
newNote.Attributes.Add("objecttypecode", "quote");
Service.Create(newNote);
}
Just saw this, in an article from MSDN:
Annotation setupAnnotation = new Annotation()
{
Subject = "Example Annotation",
FileName = "ExampleAnnotationAttachment.txt",
DocumentBody = Convert.ToBase64String(
new UnicodeEncoding().GetBytes("Sample Annotation Text")),
MimeType = "text/plain"
};
I see the document body is encoded according to the Unicode encoding. Maybe you should try to retrieve the encoding from the file and convert it to a string accordingly.

Creating FedEx Shipping Documnents using FedEx ship service WSDL

I am in the process of integrating with the FedEx international Ship Service. But I am really stuck on one part. I am trying to create a certificate of origin using their test environment. I have followed the xml schema and have come up with the code below
private static void SetCustomInvoice(ProcessShipmentRequest request)
{
request.RequestedShipment.ShippingDocumentSpecification = new ShippingDocumentSpecification();
request.RequestedShipment.ShippingDocumentSpecification.ShippingDocumentTypes = new RequestedShippingDocumentType[1] { new RequestedShippingDocumentType() };
request.RequestedShipment.ShippingDocumentSpecification.ShippingDocumentTypes[0] = RequestedShippingDocumentType.CERTIFICATE_OF_ORIGIN;
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin = new CertificateOfOriginDetail();
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin.DocumentFormat = new ShippingDocumentFormat { StockType = ShippingDocumentStockType.STOCK_4X6, ImageType = ShippingDocumentImageType.PDF, ImageTypeSpecified = true, StockTypeSpecified = true };
request.RequestedShipment.SpecialServicesRequested = new ShipmentSpecialServicesRequested();
request.RequestedShipment.SpecialServicesRequested.SpecialServiceTypes = new ShipmentSpecialServiceType[1] { new ShipmentSpecialServiceType() };
request.RequestedShipment.SpecialServicesRequested.SpecialServiceTypes[0] = ShipmentSpecialServiceType.ELECTRONIC_TRADE_DOCUMENTS;
request.RequestedShipment.SpecialServicesRequested.EtdDetail = new EtdDetail();
request.RequestedShipment.SpecialServicesRequested.EtdDetail.RequestedDocumentCopies = new RequestedShippingDocumentType[1] { RequestedShippingDocumentType.CERTIFICATE_OF_ORIGIN };
request.RequestedShipment.SpecialServicesRequested.EtdDetail.DocumentReferences = new UploadDocumentReferenceDetail[1] { new UploadDocumentReferenceDetail() };
request.RequestedShipment.SpecialServicesRequested.EtdDetail.RequestedDocumentCopies[0] = RequestedShippingDocumentType.CERTIFICATE_OF_ORIGIN;
}
But I keep getting an error message back from the web service stating “Invalid Stock Type”. Even though the shipmentDocumentStockType is an enum and I am using one of the values from it. I am still getting this error. Any ideas where I might be going wrong?
Any information will be a great help. I have tried getting in touch with FedEx technical support and they were not really a great help.
This might be a little late for you but just wanted to provide an answer in case someone else might be looking for it.
I was having a similar problem but instead of the Certificate of Origin it was with the Commercial Invoice. Turns out these forms need to be printed on a full 8.5 x 11 page in PDF format, so changing the StockType from STOCK_4x6 to PAPER_LETTER fixed it for me:
From:
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin.DocumentFormat = new ShippingDocumentFormat { StockType = ShippingDocumentStockType.STOCK_4X6, ImageType = ShippingDocumentImageType.PDF, ImageTypeSpecified = true, StockTypeSpecified = true };
To:
request.RequestedShipment.ShippingDocumentSpecification.CertificateOfOrigin.DocumentFormat = new ShippingDocumentFormat { StockType = ShippingDocumentStockType.PAPER_LETTER, ImageType = ShippingDocumentImageType.PDF, ImageTypeSpecified = true, StockTypeSpecified = true };
Hope this helps

Categories

Resources