I need to create a C# SQL CLR stored procedure to upload files (data exports) to AWS S3 buckets. These files will generally be very small.
The AWS SDK cannot be installed on the SQL Servers and I am finding it difficult to find any information about how to accomplish this.
I am looking for some examples or documentation on how to accomplish uploading files without using the SDK.
My experience is mainly SQL, limited amount of C#.
You can use Amazon S3 via a REST API: Amazon S3 REST API Introduction
However, it can get a little complex, especially when providing Authentication signatures.
Related
I have a CSV file stored in blob storage. The goal is to move this file into a Sharepoint site and set some metadata. What would be the best way to do this? The client does not want us to use Power Automate or Logic Apps.
I tried using Azure Data Factory but there seems to be an issue with writing data to SharePoint. I used the copy activity but the 'sink' to SharePoint failed. Does data factory support writing to Sharepoint?
The client does not want us to use Power Automate or Logic Apps.
Why not? This is the simplest way to achieve this, and is also better maintainable than for instance C# code.
Does data factory support writing to Sharepoint?
Yes, it does. However, using Data Factory only to copy a file to SharePoint is quite a bit of overkill.
If Logic Apps are not an option, have a look at an Azure Function to automatically trigger when the file is created in Azure Storage, and have a look at for instance Upload File To SharePoint Office 365 Programmatically Using C# CSOM – PNP for a C# way of uploading a file to SharePoint.
I am working on a C# MVC solution which needs to support the uploading of 1000s of scanned .PDF survey forms onto a system and then extract the data from each survey; in order to extract hand-written checkboxes I need to use the AWS Textract API. More information on my project can be found here: AWS textract with hand-written checkboxes
My problem is when I downloaded the AWS SDK for .NET I noticed that .Textract is not fully available at the minute for .NET
My question being, is there any other way apart from the AWS .NET SDK that would allow me to implement AWS Textracts checkbox feature into my .NET system? Neither Googles Vision OCR API or Azures Read API support checkbox extraction unfortunately.
I have some files(.txt, .doc, .xlsx etc) inside a bucket in my AmazonS3 drive and is it possible to perform a content level search through my C# application? That is, when we type a string and upon pressing key in my application, every files that contains the searched string in its content should list.
Is there any way to achieve this either using any method or even using WebAPI's.
Thanks in advance
Amazon S3 is purely a storage service. There is no search capability built into S3.
You could use services such as Amazon CloudSearch and Amazon Elasticsearch Service, which can index documents, but please note that this involves additional configuration and additional costs.
You won't be able to do all those file types you listed, but any of your files that are structured, or semi-structured, you could consider using the newly released AWS Athena which does allow searching of S3 file using an SQL-like language:
https://aws.amazon.com/athena/faqs/
Amazon Athena is an interactive query service that makes it easy to
analyze data in Amazon S3 using standard SQL. Athena is serverless, so
there is no infrastructure to setup or manage, and you can start
analyzing data immediately. You don’t even need to load your data into
Athena, it works directly with data stored in S3. To get started, just
log into the Athena Management Console, define your schema, and start
querying. Amazon Athena uses Presto with full standard SQL support and
works with a variety of standard data formats, including CSV, JSON,
ORC, Apache Parquet and Avro. While Amazon Athena is ideal for quick,
ad-hoc querying and integrates with Amazon QuickSight for easy
visualization, it can also handle complex analysis, including large
joins, window functions, and arrays.
I would like to use Amazon's .NET SDK for accessing a publicly readable bucket, but the AWSClientFactory.CreateAmazonS3Client method for creating an AmazonS3 forces you to input your AwsAccessKey and AwsSecretAccessKey.
I do not want to include these in my client-side application, but I still want to use the SDK provided by Amazon to access these publicly readable objects on S3.
Is this possible?
I also asked this question on the AWS forum and an official Amazon employee responded:
As of the 1.3.8.0 release of the SDK you can pass null for the access
and secret key and the SDK will skip the signing process and try the
operations like GetObject as a public operation.
Norm
It's worth noting that at the time of writting, the current Amazon .NET SDK is at version '1.3.14', so passing null for the credentials is currently possible.
I have a C# application that generates simple JPEG images. I need to be able to store these images and recall them at various times in the future. So, I'm looking for a turnkey, secure, CDN system. I have hacked my own together with a Windows server and IIS - I upload via FTP and request images over HTTP - but (1) there's, effectively, no need for it to be Windows and (2) its not very cost effective. I'll be generating approximately 1-2GB of images each month and I need to hold the images in perpetuity.
What are some of the turnkey options for storing this many images?
I suggest storing the images on Amazon S3. It's stable, widely supported, and can plug into a variety of workflows and security models. As of August 2011, pricing starts at $0.09/GB/month for storage, and $0.12/GB for transfer (with the first GB per month free).
While many people use S3 as a cheap and good-enough CDN, Amazon also offers Amazon Cloudfront, a "real" CDN that integrates neatly with S3.
Amazon maintains an official C# library that can talk to S3 and CloudFront, the AWS SDK for .NET.
I'm a fan of NetDNA http://www.netdna.com/ . I currently use them - good customer service, and inexpensive. Also, it is easy to plugin into Wordpress.
Check out Amazon Cloudfront http://aws.amazon.com/cloudfront/
It's their CDN product built on S3. You can use the available C# libraries, examples here http://aws.amazon.com/code/Amazon-S3/129