I need to copy the [EDIT: entire] contents of a bucket into another bucket.
The answers to this question implies that it can be done on the CLI, [EDIT: as a single operation (rather than looping over every record in the bucket)]
Can it be done via any C# SDKs?
We're currently using the AWSSDK.S3 Nuget package SDK, but I can't see any methods in that package's AmazonClient that would allow this sort of copy?
The AWS SDK for .NET supports copying objects from one bucket to another. However, you cannot copy ALL objects in a single call. See REF docs here:
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/index.html?page=TS3Client.html&tocid=Amazon_S3_AmazonS3Client
The business logic to copy an object from one bucket to another can be found in the AWS Code Library, which contains hundreds of tested examples in supported programming languages.
For this use case, see:
URL - https://docs.aws.amazon.com/code-library/latest/ug/s3_example_s3_CopyObject_section.html
The code lib should be the first place to reference when you want to learn how to use the AWS SDK.
S3 Batch Operations, AWS Datasync, or S3 Replication could be options as well. These tools require more initial setup, but the actual data transfer will happen in a single request. If there are a lot of objects to copy or this process is run frequently, it may be easier to automate using one of these options.
See https://aws.amazon.com/blogs/storage/considering-four-different-replication-options-for-data-in-amazon-s3/ for a comparison of a few options.
Related
I am going to develop a Web API in .NET Core to use Azure blob. It will contain methods to create container, Block Blob and Append Blob. Also, it needs to update and read Block Blob and Append Blob. Blob size will vary from 1 MiB to 300 MiB.
I have following questions in my mind:
Which library I should use Azure.Storage.Blobs or Microsoft.WindowsAzure.Storage.Blob?
In case of Azure.Storage.Blobs package, how to create a blob if its not exists (I couldn't find sample code for blob creation)?
In case of Azure.Storage.Blobs package, what will be the performance whether its been improved when compared to Microsoft.WindowsAzure.Storage.Blob or the same?
Thanks in advance.
The WindowsAzure library is old, and is replaced by the Azure library. I wouldn't use it for new development; it exists to keep legacy programs working but is no longer actively developed
You upload it. Seems to be some confusion in your mind about what a blob is? If a file on disk doesn't exist, you write one and then it exists. Did you mean to ask about creating a container?
There probably is some performance or feature improvement, but I don't know of one out of my head; take a look at the release notes for Azure since v9.3.3ish (it was around that time the package was switched) to see what features are added. The fact that WindowsAzure is the legacy library would be enough for me to not use it for new development, but we use both libs in various different projects and don't have any "wow, that's slow" or "wow, that's fast" thoughts on either of them- they both work
Right now I am saving my all visual studio projects in C drive.
Now I want to keep copy of that all projects in some other drive, so if my C drive get crashed then also I can access all projects.
So what is best way for that.
If I just make zip of current projects from C drive and paste it in another drive.
And when needed I extract it, then will it work OR any error will come.
Thanks for help in Advance.
If I just make zip of current projects from C drive and paste it in another drive.
Well you could do that but it's rather tedius; error-prone and rather brute-force. It's difficult to maintain history.
A better choice is to use some form of source control (SC) / software configuration management (SCM). SC is a tool for maintaining a code repository. It works by associating metadata about every source file and any changes you make.
e.g.
Git
Subversion
Microsoft TFS
Perforce
IBM/Rational ClearCase
Microsoft SourceSafe (ewww, retried thankfully)
Source control not only keeps a copy somewhere else (ideally a different computer) but it also allows you to
keep track of what changed
rollback a change
share with your friends or colleges
integrates nicely with your IDE of choice (VS) or command-line
But in this day and age there are plenty of free cloud-based solutions that offer you more than just acting as a code repository such as stats; wikis; bug tracking; and spiffy charts. Check out:
Microsoft Visual Studio Team Services (VSTS)
Github
Atlassian Bitbucket
Summary
Irrespective of whether you perform manual folder copies or use source control; both will lead to a copy of your code. However only the latter introduces workflows and due-diligence (via SCM) so that as you code you are unlikely to lose information due to the procedures and safeguards in place.
A word on file backup
If for some absolute reason you decide not to proceed with SC but rather stick with plain-old-file-backup then at least follow the fine wisdom of Scott Hanselman (MSFT) where he talks about file backup best practices:
I've got a number of backups because I practice the Backup Rule of Three.
3 copies of anything you care about - Two isn't enough if it's important.
2 different formats - Example: Dropbox+DVDs or Hard Drive+Memory Stick or CD+Crash Plan, or more
1 off-site backup - If the house burns down, how will you get your memories back?
...using apps like CrashPlan.
Scott will most likely agree that his plan wasn't intended for source code but at least you have 3 backups of files as he recommends.
See Also
Hanselman, S, "Is your stuff backed up? Recovering from a hardware failure"
Have a look at Visual Studio Team Services. You can add code to source control (I would use Git if I were you) and manage your projects there for free.
Having your code in a source control system has many benefits, like having history of each commit.
Next to that, VSTS has lots of opions like Continuous Integration / Continuous Deployment, Testing, project management support like making your project an Agile project.
What will you do if hard drive crash?
Code Management is a practice and there are many tools to help you to manage your code. Try GitHub or bitbucket
Moreover you can also zip the code and save on external disks but check how much risk is involved with your code.
You could use a .zip archive to back up your work, but this is slightly laborious and will likely include a number of files you do not need to get up and running again (for example the build output, nuget packages folders, etc.) which will bloat your archives.
A better option would be to use a Version Control System of some kind, which will allow you to back up those parts of the project/solution that actually need to be backed up while ignoring the parts that can be rebuilt from the code. A good walk through of what and why can be found in Version Control By Example, which also includes comparisons about different types of VCS as well as how to perform many of the usual tasks.
There are various free options out there, based on a number of different providers. As some examples, I've used the following services, and all of them will give you a free account, and some will also give you private repositories (so that random members of the public can't see your work if that's what you want):
GitHub - unlimited public repositories, uses Git.
BitBucket - unlimited private repositories, uses Git or Mercurial
Visual Studio Team Services - unlimited private repositories, uses Git or TFS
Using an online provider will give you the added benefit of the backups being on a third party - so if your disk fails you'll still have a backup, as well as the other benefits a VCS will provide (the ability to rollback to a specific point in time, annotations about changes, etc.).
I work in a small environment that consists of me (the developer) and my colleague (the designer).
We have a good intersection of skills so mostly we work pretty well together, currently we host all javascript files, css files and images in an s3 bucket so he can adjust our front end stuff without me having to deploy the changes to the server.
The only part of our workflow that lets us down is whenever he needs to adjust any of the HTML design. For that part he's dependent on me to deploy those changes to the production server.
We use ASP MVC and Razor templates.
Has anyone successfully worked out if its possible to host our cshtml razor templates in an s3 bucket rather than on our production server?
EDIT: We wouldn't want to host all of our view templates on s3, just a few that change regularly.
You can try implementing custom Razor engine. The base engine has overridable CreatePartialView, CreateView ETC methods which you can override and return an instance of IView with content fetched from S3.
Caution: you should not use this on a production server because it may have a performance hit. Also, make sure you have S3 bucket as private otherwise your server-side razor code will be accessible to anyone which is dangerous.
Generally view files are smaller in size than the assets - Images, JavaScript and CSS Files. You may keep the view files on the server and keep the assets on Amazon S3.
This would reduce the load from the server. Also, with this you might do well with public buckets only as most of the assets would probably be accessible publically (without authentication).
This is not an exact solution of what was asked in the question. It is just an alternative suggestion and should be treated accordingly.
You can sure host them in S3, as you already do. Moreover, you could definitely write code that would download and compile these templates on-demand, but it would be a non-trivial task. You will want to implement a VirtualPathProvider.
It sounds like you should consider using a versioning system. In addition to solving the problem you describe above, a versioning system would ensure that
you can track who changed what when
you have a full history and can undo changes
you are alerted if you and your mate make conflicting changes to a file
you always know what is "the newest version"
Many systems even allow you to deploy your code directly from the versioning system.
There are dozens of versioning system services online and most of them provide their services for free for teams of your size. Take a look at Microsoft's hosted TFS, BitBucket or GitHub.
If you have never worked with versioning systems you will need to learn a few new concepts and terms. However, it won't take long and it will be soo worth it.
i have a project where i do something i have same copy from another who have do something in it later i have a thing to do currently.
so that's three copy and i am unsure that i am not confused with it. sometime i call wrong copy for finding the thing i want from them.
without using any software outside our office window any trick to manage all project or how i can manage them and make them one copy.
the same problem with some other problem come with when the same script used in two software , website who are used same script with have a small difference in both.
so i want to know how exactly all other manage them. i want to have a answer with .net plateform.
There are several source control solutions that have plugins to integrate directly with Visual Studio. I would recommend looking into Subversion. Currently at my office we use Vault, but it has some short comings, especially when it comes to renaming or moving a file and then editing it prior to a check in.
http://ankhsvn.open.collab.net/
http://www.sourcegear.com/vault/
I recently started a small pet project. It's written in C# with ASP.NET MVC3. As it's grown, my needs for source control also has grown. Here's what I've done:
Day one, I placed my project in a Dropbox folder. That way, I could reach my project files from all my computers (and even my iPhone).
A few weeks later, when I first had to make some serious changes and neede a fallback, I created a Mercurial repository in my Dropbox folder and commited all my code. I've got a blogpost on that here: http://csharptutorial.blogspot.com/2011/06/using-dropbox-with-mercurial.html. Mercurial is a great versioning system, because it doesn't require a server installation and your repository is copied in full to all locations where it's checked out.
Now, I'm looking at moving my project over to BitBucket (http://bitbucket.org/) where you also share and discuss code with others.
A good alternative to Mercurial is Git, using github.com in stead of bitbucket.org. Both have a learning curve, but they both represent modern version control tools and is well worth having a look at.
Source control and discipline.
You need to get an approach and method for managing your source. A source control system is essential for this. I use SVN and Ankh, as does Clayton.
However, you need an approach for using this. I always have branches. You can have development branches and merge these changes into a release branch when your code has passed all the tests. This way takes a discipline and time, but you need to do this to build quality into your system.
We need to have documents shared between clients (CRM-like functionality). Users need to be able to:
Edit the documents and save them again
Attach new documents
Our application is coded in WPF with WCF for data-transport and NHibernate/SQL for data on the server.
what we're thinking is to use SVN and have the app create a local check-out of parts of the repository (when they click a document, it is checked out by SVN in the background and opened from the local path) - When saved it will silently (using monitoring of the path) be committed back to the repository.
Question: Is this feasible - or are there better solutions to this?
EDIT 1:
Summary so far:
I'll look into using Git/Mercurial instead of SVN
Document size (revisions) might be prohibitive pending tests
SharePoint is an option (although not viable in my case as the cost alone is prohibitive) - I will look into the alternatives for SharePoint, tho.
Not much experience out there about using repositories for many users although it works for small teams..
Wiki software might be an alternative to SVN.
Thanks for all the feedback - I'll keep it open a bit longer.
EDIT 2:
Summary after a few days of work - I have a client working - see my progress here.
Based on the heavy .NET references, are you all set up with MSDN? Perhaps you can make use of SharePoint...which may already be included within your MSDN account.
You might also want to consider using a Wiki for document management - I've seen this done and do it myself for my own organisation. We're using Atlassian's Confluence Wiki. Confluence provides for the versioning and general management of documents.
I wouldn't use SVN for this, SVN is not very efficient when dealing with binary files. By using SVN as a back channel for some content in your application you just complicate things by adding another technology and dependency, but you will not use much of its real potential.
I would store the documents as blobs in the database and get/store them through WCF.
Generally I don't think that SVN or any version control system is a good thing to use for sharing documents. Main disadvantage is the diff system on binary files... Your SVN repo will grow rapidly..
Maybe you should try using some of the commercial tools designed for document sharing (eg. Microsoft Sharepoint). Or some Open Source alternatives... Perhaps you should read this post...
It depends on the kind of documents you are using. If you have lots of changing, compressed binary files, then don't use it.
However, if the documents are in an open format like a Wiki language, (X)HTML, LaTeX or uncompressed ODF, then using a version control system makes absolutely sense. Also, a bunch of compressed ODF files or PDF files are handled very well, especially if the files are mostly smaller than 5 MB or so.
In addition, make sure to check some more recent version control systems like Mercurial and Git before sticking to the conceptually outdated SVN. In your scenario, you won't profit much from the "distributed" part of Mercurial and Git, but they are nevertheless easier to setup - at least to my experience. And they provide very advanced version control features which can save your day in the rare cases when you need them.
In case you stick to SVN, and if your client software runs under a modern Unix system, you can also try SVN-FS. This is a filesystem that uses a remote SVN server. Each read goes to the latest revision. Each write creates a new commit. This seems to be exactly what you wanted to build around SVN.
I think that using ready made and proven tech is great idea. Would like to see it's progress if you really go that way.
I would strongly go AGAINST SharePoint - you'll tie yourself to Microsoft in manners that are hard to describe here. From my point of view, SharePoint is a tech that needs taking care of just for itself.