I am fairly new to python, and come from a C# background. In C# l, third party libraries are commonly stored inside the project folder.
This means that libraries are totally internal to the project. The project then is not dependent on anything outside of the project folder (other than .net framework of course).
I really like this structure and have tried successfully to mirror this in python by copying the libraries into a lib directory, in the project root, and adding the lib folder to the python path on startup of the application.
I am worried that there may be something I am overlooking by doing this as I have looked around a bit amd have not really seen anyone else in thw python community doing this.
My question is simply - is this ok? Is there something that I may miss by simply dumping the necessary .py libraries in, rather than using easy install, and thus storing the libraries in site packages, at a system level?
Please feel free to let me know of any drawbacks you can see, no matter how simple.
Thanks!
I'll espouse the usage of virtualenv and pip for development purposes. This will give you exactly the sandbox that you are used to. As for distribution, use setup.py and reuse the requirements.txt file that you would use with pip install -r to install dependencies to generate the install_requires argument to setuptools.setup. I've been meaning to set up an example that shows this off a little - check out https://github.com/dave-shawley/setup-example for a nice example with some description too. I plan on adding a little more to this as time allows.
If you want to closely manage the dependencies of your code on the per project basis you might want to take a look at virtualenv.
Virtualenv will allow you to keep your dependencies close to your source but will remove the error prone manual copying of the .py files.
On top of that remamber that some packages are not pure python and they sometimes contain compiled C code - if you use virtualenv you do not have to worry about it.
Related
Ever since I've been using the (relatively) new .NET Standard Library project type in Visual Studio, I've been having some problems getting a complete set of DLL files that are required by my project.
The problem is usually limited to 3rd-party libraries which I reference as NuGet packages. I've noticed that these don't get copied to the output folder of my project when I build it. This didn't use to be the case in classic project types.
While I can appreciate the de-cluttering effect that this change has brought for .NET Standard projects, I'm now faced with a problem. I sometimes absolutely need to be able to get the entire list of all files that my project depends on!
I have several different cases, where I might require this list for one reason or another, but the one I believe is most crucial for me, is when I want to gather these files from the csproj itself, right after it's built. In there, I have a custom MSBuild <Target> which should take all the files from the output dir and zip them together for distribution. The problem is, I'm missing all the files that come from NuGet dependencies, because they're not there!
How can I solve this in a general (i.e. not project-specific) way?
UPDATE
There's this deps.json file that contains basically all I'm after and then some. It's just a matter of extracting the relevant information and find the files in the local NuGet cache. But that would involve writing a specialized app and calling it from my target. Before I start writing one myself... Is there something like this already out there somewhere?
I followed this answer and it sort of works.
The suggested thing was to include the following into my csproj:
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
My main concern is that it also outputs some other DLLs from the framework (such as System.Memory.dll and System.Buffers.dll, among others), which I didn't expect. But maybe that's a good thing. They do seem to be dependencies, just not direct ones. I'll see how it plays out.
If it turns out ok, my only wish would be that this directive was more prominently displayed in project settings (as a simple checkbox, maybe?) so I wouldn't have to hunt the web to find it.
I have my main solution wish is comprised of 4 projects. There is a precedence chain between my 4 projects, that is 1 needs 2, 2 needs 3 etc. Also, some of these projects use Nuget packages.
Like this:
Project Layout
Now I can get my MainPointOfEntry.dll, and include it my a external solution and it works fine, but I don't see how it knows where to find the dependent .dlls (maybe it is somehow looking in the same solution folder)
External Solution with referenced dll
So my single entry point works fine, but how does it know where to find the dlls of which MainPointOfEntry depends?
I feel if I was to make a copy of MainPointOfEntry.dll and create a solution on a different computer it would not work?
Sorry about this badly phrased question
how does it know where to find the dlls of which MainPointOfEntry depends?
It doesn't. MainPointOfEntry does.
I feel if I was to make a copy of MainPointOfEntry.dll and create a solution on a different computer it would not work?
Depends on if you have embedded the library in output or not. If you have added your dependency and configured to be an embedded resource It will work. If not, you need to copy these too.
I have a c# project that generates an EXE file. Now, I'm in a "secure" corporate environment, where I can compile my project, but I cannot execute the EXE file.
As a Java programmer, I'm wondering if there is not a way to compile the c# project into something that would not be an EXE file, but a CIL file and then execute the CIL file by something that corresponds to java.exe in the dotnet world.
EDIT in response to comments:
I can run exe files that have been installed by a package manager
Yes, I know the corporate policy is stupid.
Well, this should be pretty easy.
.NET executables are simply DLLs like any other - the main difference being the executable format itself, and the fact that EXE files have an entry point, while DLLs don't.
It also means that you can load the EXE into memory exactly the same way as you would with a DLL:
Assembly.LoadFrom("SomeExe.exe");
You're already half way there - now we just need to find and execute the entry point. And unsurprisingly, this is also pretty trivial:
var assembly = Assembly.LoadFrom("SomeExe.exe");
assembly.EntryPoint.Invoke(null, null);
For most applications, this should work perfectly fine; for some, you'll have to make sure the thread you're using to invoke the entry point has STAThread or MTAThread respectively (Thread.TrySetThreadApartment if you're starting a new thread).
It might need tweaking for some applications, but it shouldn't be too hard to fix.
So you can just make some bootstrap application ("interpreter") that only really contains these two lines of code. If you can't get even that approved, and you really need something as an "official package", try some .NET application that allows you to execute arbitrary code - for example, LINQPad, or PowerShell.
EDIT:
This does have limitations, of course, and it does introduce some extra setup work:
The bootstrapper has to target the same or higher version of .NET Framework. .NET Portable might be particularly tricky, but I assume you have that well under control. It also has to have the same bitness (if specified explicitly).
You need to run the debugging through the bootstrapper. That actually isn't all too hard - just go to project properties, debug and select "Start external program".
The bootstrapper has to run under full trust conditions - it's necessary for reflection to work. On most systems, this simply means you have to have the exe as a local file (e.g. not from a network share). Tools like LINQPad will run under full trust by default.
The application must not depend on Assembly.GetEntryAssembly. This isn't used all that often, so it shouldn't be a problem. Quite a few similar issues should also be fine since you build the application you're trying to run yourself.
Whats a good approach for white labeling dll and exe with visual studio?
In essence we want to be able to have the name of the dll and exe change based on the client that we are packaging the solution for, e.g.:
Instead of myCompany.exe and myCompany.db.dll, I would like yourComany.exe and yourComany.db.dll or acme.exe and acme.db.dll, etc
Edit:
Currently we are using a straight visual studio build process with a wix project to create an msi.
If the only justification for rebuilding it is to change the name, can you just use something generic in the first place? Imagine having to patch 50 identical DLLs, and build/deploying each one separately because they all must be named different things. Even if it's only for a few clients, I would hate to have to maintain that. Versioning could be a hassle too.
If you must do it, I would probably go with a build task (which can perform fairly advanced operations). You mention that you are "packaged the solution"; the viability of a build task would depend on how it is being packaged.
In response to your comment about naming the EXEs with client-specific names... My obvious suggestion there would be to have those applications contain as little code as possible.
The simplest build integration I can think of would be to create a post-build task which ran upon successful compilation in release mode. The task could then read a config file which defined the unique names, and copy the successfully built EXEs to an output directory.
Some of the operations can be accomplished just from the task config file: http://msdn.microsoft.com/en-us/library/ms171466.
Alternatively, you might want to create a little application to do all the work for you, and just pass config switches to it.
For example, here is a little post-build command that I execute to minify my JavaScript/CSS upon successful build of a web application. The concept is similar:
build
execute an app (like msbuild.exe, or your custom build app)
pass data to the executable (like paths, switches, etc.)
executable writes the files out
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\msbuild.exe
"$(ProjectDir)Properties\build\minify.xml"
/p:SourceLocation="$(ProjectDir)client"
/p:CssOutputFile="$(ProjectDir)client\final\final-full.css"
/p:JavaScriptOutputDirectory="$(ProjectDir)client\final"
You could use ILMerge in whatever post-build process you want on all your outputted assemblies (dll and exe), to create one-off customer-branded builds.
ilmerge /out:CustomerName.exe internalName.dll internalName.exe
I don't know that there is a good way to do this without actually building the project as XYZ company. You could try something like this which will give you the desired result BUT it will change the physical name of the assembly as well which may cause dependency problems.
My company is having trouble figuring out the best way to manage our builds, releases, and branches... Our basic setup is we have 4 applications we maintain 2 WPF applications and 2 ASP.NET applications, all 4 of these applications share common libraries, so currently they are all in one folder /trunk/{app1, app2, app3, app4}.
This makes it very hard to branch/tag a single application because you are branching all 4 at the same time, so we would like to separate it out into something like {app1,app2,app3,app4}/{trunk,tags,branches} but then we run into the issue of where to put the shared libraries?
We can't put the shared libraries as SVN externals because then when you branch/tag the branch is still referencing the trunk shared libs instead of having them branched as well.
Any tips? Ideas?
We are currently using svn and cruisecontrol.net.
EDIT: The shared libraries are changing often as of right now, which is why we can't use them as svn externals to trunk, because we might be changing them in the branch. So we can't use them as binary references.
Its also very hard to test and debug when the libraries are statically built instead of including the source.
I guess it all depends on how stable the shared libraries are. My preference would be for the shared libraries to be treated as their own project, built in CruiseControl like the others. Then the four main applications would have binary references to the shared libraries.
The primary advantage with this approach is the stability of the applications now that the shared libraries are static. A change to the libraries wouldn't affect the applications until they explicitly updated the binaries to the newer version. Branching brings the binary references with it. You won't have the situation where a seemingly innocuous change breaks the other three applications.
Can you clarify why you don't like branching all four applications at the same time?
This makes it very hard to branch/tag a single application because you are branching all 4 at the same time
I usually put all my projects directly under trunk as you are currently doing. Then when I create a release branch or a feature branch, I just ignore the other projects that get carried along. Remember, the copies are cheap, so they're not taking up space on your server.
To be specific, here's how I would lay out the source tree you've described:
trunk
WPF1
WPF2
ASP.NET 1
ASP.NET 2
lib1
lib2
branches
WPF1 v 1.0
WPF1
WPF2
ASP.NET 1
ASP.NET 2
lib1
lib2
WPF1 v 1.1
WPF1
WPF2
ASP.NET 1
ASP.NET 2
lib1
lib2
lib1 payment plan
WPF1
WPF2
ASP.NET 1
ASP.NET 2
lib1
lib2
We are kicking off an open source project to try and deal with this issue. If anyone is interested in commenting on it or contributing to it, it's at:
http://refix.codeplex.com
I agree with #Brian Frantz. There's no reason to not treat the shared libraries as their own project that is built daily and your projects take binary dependency on the daily builds.
But even if you want to keep them as a source dependency and build them with the app, why wouldn't the SVN externals approach work for you? When you branch particular app, there's no need to branch the shared library as well, unless you need a separate copy of it for that branch. But that means, it not a shared library anymore, right?
I've tried solving this problem several ways over the years, and I can honestly say there is no best solution.
My team is currently in a huge development phase and everyone basically needs to be working off of the latest and greatest of the shared libs at any given time. This being the case we have a folder on everyone's C: drive called SharedLibs\Latest that is automatically synced up with the latest development release of each of our shared libraries. Every project that should be drinking from the firehose has absolute file references to this folder. As people push out new versions of the shared libs, the individual projects end up picking them up transparently.
In addition to the latest folder, we have a SharedLibs\Releases folder which has a hierarchy of folders named for each version of each shared lib. As projects mature and get towards release candidate phase, the shared lib references are pointed to these stable folders.
The biggest downside to this is that this structure needs to be in place for any project to build. If someone wants to build an app 10 years from now, they will need this structure. It is important to note that these folders need to exist on the build/CI server as well.
Previous to doing this, each solution had a lib folder that was under source control containing the binaries. Each project owner was tasked with propagating new shared dlls. Since most people owned several projects, things often fell through the cracks for the projects that were still in the non-stable phase. Additionally TFS didn't seem to track changes to binary files that well. If TFS was better at tracking dlls we probably would have used a shared libs solution / project instead of the file system approach we are taking now.
Apache NPanday + Apache Maven Release
... might solve your problems
It gives you dependeny management (transitive resolving), strong versioning support, and automatic tagging/branching on 14+ version control systems, including SVN.
Give me a hint, if I should elaborate more.
I think there is no way you can avoid versioning and distributing your shared libs as separate artifacts, but Maven helps you alot doing that!
And you can allways do tricks to get it all opened in one Solution :-)
A sample workflow:
Dev 1 build A locally using Maven
Checks in sources
Build server builds A and deploys so-called SNAPSHOT-Versions to Repository Manager (e.g. Nexus)
Dev 2 two loads B, NPanday will automatically resolve the A-libs from the Repository Manager (No need to get the source and build)
Dev 1 wants to release A: Maven Release creates a branch or a tag with your source, finalizes the Version (removing SNAPSHOT) and deploys the artifacts to a Repository Manager.
Dev 2 can now upgrade B to use the final release of A (change entry in xml, or use VS-addin to do so)
Now Dev 2 can release B, again with automatic creation of tag or branch and deployment of built artifacts.
If you want to provide zipped packages as output from your build, Maven Assembly Plugin will help you do that.
You can use Apache/ IVY in standalone mode.
http://ant.apache.org/ivy/history/latest-milestone/standalone.html
I need to emphasize "stand alone" mode. If you google for examples....you will find alot of (not standalone) ones.
Basically, IVY works on this premise.
You publish binaries (or any kind of file, but I'll say binaries from this point forward).....as little binary-packages.
Below is PSEUDO code, do not rely on my memory.
java.exe ivy.jar -publish MyBinaryPackageOne.xml --revision 1.2.3.4 (<< where the .xml refers to N number of files that make up the one package.))
"Package" simply means a group of files. You can include .dll and .xml and .pdb files in a package (what I do with a DotNet build of assemblies). Or whatever. IVY is file-type agnostic. If you want to put WordDocs up there you could, but sharepoint is better for documents.
As you make bug fixes to your code, you increment the revision.
java.exe ivy.jar -publish MyBinaryPackageOne.xml --revision 1.2.3.5
then later you can retrieve from IVY what you want.
java.exe ivy.jar -retrieve PackagesINeed.xml
PackagesINeed.xml would contain information about the packages you want.
something like
"I want version '1.2+ of the MyBinaryPackageOne"
(defined in xml)
As you build your framework binaries...you PUBLISH to IVY.
Then, as you develop and build your code...you RETRIEVE from IVY.
In a NUTSHELL, IVY is a repository for FILES (not source code).
Ivy then becomes the definitive source of your binaries.
None of the "Hey, Developer-Joe has the binaries we need" kind of bull-mess.
.......
Advantages:
1. You do NOT keep your binaries in source control. (and thus do not BLOAT your source control).
2. You have ONE definitive source for binaries.
3. Through xml configuration, you say which versions you need for a library.
(In the example above, if version 2 (2.0.0.0) of MyBinaryPackageOne is published to IVY (let's assume with breaking changes from 1.2.x.y)...then you are OK, because you defined in your retrieve (xml configuration file) .. .that you only want "1.2+". Thus your project will ignore anything 2+...unless you change the configuration package.
Advanced:
If you have a build machine (CruiseControl.NET for example)....you can write logic to publish your (newly built) binaries to IVY after each build.
(Which is what I do).
I use the SVN revision as the last number in the build number.
If my SVN revision was "3333", then I would run something like this:
java.exe ivy.jar -publish MyBinaryPackageOne.xml --revision 1.2.3.3333
Thus whenever retrieve the package for revision "1.2.3+" .... I'll get the latest build.
In this case, I would get version 1.2.3.3333 of the package.
It's sad that IVY was started in 2005 (well, that's the good news)...but that NUGET didn't come out til 2010? (2011?)
Microsoft was 5-6 years behind on this one, IMHO.
I would never go back to putting binaries in source control.
IVY is very good. It is time proven. It solves the problem of DEPENDENCY management.
Does it take a little bit of time to get comfortable with it?
Yep.
But it is worth it in the end.
My 2 cents.
.................
But idea #2 is
Learn how to use NUGET with a local (as in..local to your company) repository.
That is the about the same thing as IVY.
But having looked at NUGET, I still like IVY.