What will I need to use Etsy's Statsd in a Windows Environment? My intentions are to create a .net client to use Statsd.
I have statsd+graphite running in my Windows environment using the C# client NStatsD.
Here are my notes for getting the Linux VM setup:
Note: I know enough Linux to be dangerous but am otherwise a noob and could be doing something unwittingly horrible.
Install Ubuntu Server 12.04. I used VirtualBox for dev and then later EC2 for prod.
Download graphite-fabric to your home folder. This is a script that will download, compile and install graphite and statsd. It expects a clean box and uses nginx for the web server.
sudo apt-get install git
git clone git://github.com/gingerlime/graphite-fabric.git
cd graphite-fabric/
Install prereq's for fabric
sudo apt-get install python-setuptools
The next steps are a download, compile and install which can take some time. It is worthwhile setting a keep alive on any putty ssh session before continuing.
Now install as per gingerlime's instructions in the README.md - including the requirements section.
Install statsd as per gingerlime's instructions.
Reboot
Execute netstat -nulp and observe 8125 is in use to confirm statsd is listening.
Check carbon is running tail /opt/graphite/storage/log/carbon-cache/carbon-cache-a/listener.log. If it isn't, try sudo /etc/init.d/carbon start
Now you have your server running, try throwing some counters at it with the NStatsD client.
Timezone fix:
This will fix graphite to graph times in your local zone
cd /opt/graphite/webapp/graphite
sudo cp local_settings.py.example local_settings.py
sudo chown www-data:www-data local_settings.py (check with ls -l that permissions look right)
sudo pico local_settings.py Set TIME_ZONE to something like Australia/Sydney. Discover what timezones you can use in /usr/share/zoneinfo/
Save and restart the box (not sure how to make it pick up the change without restart)
EC2 Notes
root is disabled on EC2. Fabric prompts for a root password which you don't have. Use the -i keyfile argument with fab to give it your ssh keyfile instead.
VirtualBox Notes
VBoxVMService was handy to automatically run the VM as a service in my Windows dev environment.
TL;DR:
statsd.net is a scalable statsd clone with lots of cool extras, and designed for windows-centric environments. First production release is on the 29th of May, 2013
statsd-csharp-client is a lightweight statsd and statsd.net client, available for .net 3.5, 4.0 and 4.5 via nuget.
Full Disclosure: I'm the author of these two projects and I wrote them because I couldn't take etsy's statsd service to production - my organisation isn't ready to deploy and support nodejs-based services.
I had the same problem in my organisation - we're a windows-centric environment that wants to collect stats from all over the world into a single graphite repository. I had initially thought of using etsy's statsd but my company is not yet ready to roll with nodejs services in production. Along the way I found that having my own aggregation service meant I can do lots of interesting things like add memcached support for scalability, add new aggregators and so on.
The service is usable in console mode right now, and will be ready for production use from the 29th of May 2013 onwards.
Your best bet is to have a Linux server with statsd and Graphite installed. You would then just need to write some C# code to make the UDP call to get the metric into the system.
[UPDATE (6/23/2014): I came across a statsd/Graphite clone called statsd.net that looks promising but largely lacking on the Graphite side of the equation. I still think the best bet is to use the actual statsd/Graphite projects since any clone is necessary playing catch-up.]
We've re-built Graphite/StatsD in pure .NET. It is currently being used in production environment, processing around 600M datapoints daily.
Statsify:
There is quite a few open-source StatsD client implementations available in different Language including C#.NET. Etsy provide one on their github examples, also check out AppFirst version of statsd_clients.
Current statsd and nodejs versions allow you to run statsd on Windows without amendments. statsd package.json file already contains scripts to install and uninstall it as Windows service.
The installation procedure would be:
Install nodejs for Windows
Create a directory like C:\StatsD
Open a console window and do cd \StatsD
Then do npm install https://github.com/etsy/statsd.git
In node_modules\statsd directory create your own config.json
In the console window do cd node_modules\statsd and npm run-script install-windows-service
However, Graphite only runs on Linux.
You need a statsd server to connect to.
You also need a client library to connect to it, e.g. this one and the nuget package of it.
Related
I am attempting to debug some C# / .NET 5 code in WSL 2 with Ubuntu on Windows. I have WSL 2 setup with Windows 10 and want to test out creating a Systemd service. Unfortunately, it appears Systemd is not enabled with WSL 2 by default, even though a standard Ubuntu install does have it enabled by default. Is there any way to get Systemd enabled in WSL 2?
Note: See footnote at bottom of this answer for background on this Community Wiki.
There are several possible paths to enabling Systemd on WSL2 (but not WSL1). These are summarized here, with more detail provided below.
Option 1: Upgrade WSL to the latest application release (if supported by your system) and opt-in to the Systemd feature
Option 2: Run a Systemd-helper script designed for WSL2
Option 3: Manually run Systemd in its own namespace
And while not part of this question, for those simply looking to run certain applications that require Systemd, there are alternatives:
On WSL1 and WSL2:
Alternative 1: SysVInit scripts (e.g. sudo service <service_name> start) where available
Alternative 2: Manually configuring and running the service
On WSL2-only:
Alternative 3: Docker
Should you enable Systemd in WSL?
First, consider whether you should or need to enable Systemd in WSL. Enabling Systemd will automatically start a number of background services and tasks that you really may not need under WSL. As a result, it will also increase WSL startup times, although the impact will be dependent on your system. Check the Alternatives section below to see if there may be a better option that fits your needs. For example, the service command may do what you need without any additional effort.
More detail on each answer:
Option 1: Upgrade WSL to the latest application release (if supported by your system) and opt-in to the Systemd feature
Microsoft has now integrated Systemd support in the WSL2 application release (as opposed to the older "Windows feature" implementation).
Starting with WSL Application Release 1.0.0, this feature is available on both Windows 10 and Windows 11. Windows 10 users do need to be on UBR (update build revision) 2311 or later. The UBR is the last 4 digits of your full Windows build number (e.g. 10.0.19045.2311 for Windows 10 22H2). 2311 is installed with KB5020030, an optional Preview update, although if you are reading this later, it will likely be a later (non-Preview) monthly servicing update.
If you are on a supported Windows release, the WSL application with Systemd support can be installed:
Through the Microsoft Store (as "Windows Subsystem for Linux").
Or from the Releases page in the Github repo. To install a release manually:
Reboot (to make sure that WSL is not in use at all). A simple wsl --shutdown may work, but often will not.
Download the 1.0.0 (or later) release from the link above.
Start an Administrator PowerShell and:
Add-AppxPackage <path.to>/Microsoft.WSL_1.0.0.0_x64_ARM64.msixbundle
wsl --version # to confirm
To enable, start your Ubuntu (or other Systemd) distribution under WSL (typically just wsl ~ will work).
sudo -e /etc/wsl.conf
Add the following:
[boot]
systemd=true
Exit Ubuntu and again:
wsl --shutdown
Then restart Ubuntu.
sudo systemctl status
... should show your Systemd services.
Option 2: Run a Systemd-helper script designed for WSL2
There are a number of Systemd-enablement scripts available from various sources. Given the complexities involved in running Systemd under WSL, it is recommended that you:
Use one that is actively maintained
Attempt to understand, as much as possible, how they operate, and how they may impact other features and applications in your distribution(s) under WSL
When asking questions here or on any other site, disclose in the question which script you are using so that others can attempt to understand and/or reproduce your issue in the proper context
Several of the more popular projects that enable Systemd under WSL2 are:
Genie: 1.8k stars, last commit September, 2022
Distrod: 1.4k stars, last commit July 2022
WSL2-Hacks: 1.1k stars, mostly instructional, with a supporting script example. Last commit January, 2022
At the core, all of them operate on the same principles covered in the next option ...
Option 3: Manually run Systemd in its own namespace
One of the main issues with running Systemd in earlier versions of WSL is that both inits need to be PID 1. To get around this, it is possible to create a new namespace or container where Systemd can run as PID 1.
To see how this is done (at a very basic level):
Run:
sudo -b unshare --pid --fork --mount-proc /lib/systemd/systemd --system-unit=basic.target
This starts Systemd in a new namespace with its own PID mapping. Inside that namespace, Systemd will be PID1 (as it must, to function) and own all other processes. However, the "real" PID mapping still exists outside that namespace.
Note that this is a "bare minimum" command-line for starting Systemd. It will not have support for, at least:
Windows Interop (the ability to run Windows .exe)
The Windows PATH (which isn't necessary without Windows Interop anyway)
WSLg
The scripts and projects listed above do extra work to get these things working as well.
Wait a few seconds for Systemd to start up, then:
sudo -E nsenter --all -t $(pgrep -xo systemd) runuser -P -l $USER -c "exec $SHELL"
This enters the namespace, and you can now use ps -efH to see that systemd is running as PID 1 in that namespace.
At this point, you should be able to run systemctl.
And after proving to yourself that it's possible, it is recommended that you exit all WSL instances completely, then doing wsl --shutdown. Otherwise, some things will be "broken" until you do. They can likely be "fixed", but that's beyond the scope this answer. If you are interested, please refer to the projects listed above to see how they handle these situations.
Alternative 1: SysVInit scripts (e.g. sudo service <service_name> start) where available
In Ubuntu, Debian, and some other distributions on WSL, many of the common system services still have the "old" init.d scripts available to be used in place of systemctl with Systemd units. You can see these by using ls /etc/init.d/.
So, for example, you can start ssh with sudo service ssh start, and it will run the /etc/init.d/ssh script with the start argument.
Even some non-default packages such as MySql/MariaDB will install both the Systemd unit files and the old init.d scripts, so you can still use the service command for them as well.
On the hand, some packages, like Elasticsearch, only install Systemd units. And some distributions only provide Systemd units for most (if not all) packages in their repositories.
Alternative 2: Manually configuring and running the service
For those services that don't have a init-script equivalent, it can be possible to run them "manually".
For simplicity, let's assume that the ssh init.d script wasn't available.
In this case, the "answer" is to figure out what the Systemd unit files are doing and attempt to replicate that manually. This can vary widely in complexity. But I'd start with looking at the Systemd unit file that you are trying to run:
less /lib/systemd/system/ssh.service
# Trimmed
[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
Some of the less relevant lines have been trimmed to make it easier to parse, but you can man systemd.exec, man systemd.service, and others to see what most of the options do.
In this case, when you sudo systemctl start ssh, it:
Reads environment variables (the $SSHD_OPTS) from /etc/default/ssh
Tests the config, exits if there is a failure
Makes sure the RuntimeDirectory exists with the specified permissions. This translates to /run/sshd (from man systemd.exec). This also removes the runtime directory when you stop the service.
Runs /usr/sbin/sshd with options
So, if you don't have any environment-based config, you could just set up a script to:
Make sure the runtime directory exists. Note that, since it is in /run, which is a tmpfs mount, it will be deleted after every restart of the WSL instance.
Set the permissions to 0755
Start /usr/sbin/sshd as root
... And you would have done the same thing manually without Systemd.
Again, this is probably the simplest example. You might have much more to work through for more complex tasks.
Alternative 3: Docker
Many packages/services are available as Docker images. Docker typically runs very well under Ubuntu on WSL2 (specifically WSL2; it will not run on WSL1). If there's not a SysVinit "service" script for the service you are trying to start, there may very well be a Docker image available that runs in a containerized environment.
Example: Elasticsearch, as in this question.
Bonus #1: Doesn't interfere with other packages already installed (no dependency issues).
Bonus #2: The Docker images themselves pretty much never use Systemd, so you can often inspect the Dockerfile to see how the service is started without Systemd. For more information see the next option - "The manual way."
Microsoft recommends Docker Desktop for Windows for running Docker containers under WSL2.
Footnote This answer is being posted as a Community Wiki because it can apply to multiple Stack Overflow questions. It is originally based on answers to this Ask Ubuntu question. However, it is hoped that this wiki-answer can be continuously updated by the community as Systemd evolves on WSL.
This question has been chosen since:
It appears to be the most canonical, straightforward, "How do I enable Systemd on WSL?" question.
It is on-topic, as *creating Systemd services is (or at least can-be) unique to programming.
I have been working on .net platform for few years now and I must say I am very impressed by how Microsoft is making .net cross-platform compatible.
I spent hours trying to run a small hello world application built using CoreCLR on a mac. And it worked. While There are still a lot of UNKNOWNS I am still trying to understand, there is this one question I was not able to find answer of on google.
How do you automate the deployment a dnx application. I mean, do you compile your aspnet 5 app into a nuget package and then restore it on your linux server (I have never used linux so not sure how nuget works there), and run dnx command ? Or just zip it and push it to the server directly ?
Sorry this is all very new to me and so my questions might sound stupid. I just want to know what's the best way I can implement continuous delivery for my asp.net 5 applications. My ultimate goal is to host my apps on linux containers.
You can use dnu publish --runtime <name of runtime> --no-source. That creates a folder that has your application, its dependencies, and the runtime. Then, all you have to do is get that folder on your server.
How you move files around really depends on your scenario... It could be FTP, Storage, Kudu (if you're on Azure WebSites), etc.
Another alternative is to do the restore on the server. While this reduces the size of the application when you publish, you will have to restore packages on the server which can be insecure and it can also lead to application breaks because there might be newer, incompatible packages on the feeds.
While there's no right answer to fit all, I found that if you want to most reliable and consistent results, you should publish with everything, test locally and then just copy the bundle on your server.
For docker, I recommend the same thing. Publish with runtime and no sources and create a container that has the resulting folder.
could someone point me to the right documents or describe how Docker would help (or what should be Dockerized) in the following scenario:
4 environments: Developer's laptop, DEV, TEST, PRO
C# web MVC application running under IIS8.
Databse running on Oracle dedicated Exadata server.
Source Control is TFS-TFS, with builds under TFS and a binary repository with rest apis. (Proprietary)
Deployment in TEST and PRO must follow a strict approval process.
with all that said, I know I am not in the most common environment for going to Docker, however I specifically selected this use case in order to get the most of the answers.
I can follow 2 path:
Either try to Dockerize the DEV environment itself, but I doubt Visual Studio can be Dockerized today? So that would leave my with running a smalle database engine in dock as well a maybe a mono instance for running the application, however I doubt developers will see an added value compared to the built-in VSS features.
I could also try to dockerize the 'run' environments, but again for IIS I am not sure Docker will do.
If the answer is "nothing will do on Microsoft stack", please switch to Java, SVN, Jenkins, RAD and WAS where appropriate, that would match some of my use cases too.
For dealing with DEV/TEST/PROD see Dockerfile and dev/test/prod environment
You may have a look at the (343) Oracle Database in container you can find in the registry hub https://registry.hub.docker.com/search?q=oracle&searchfield=
If you choose another webserver replacing IIS, it is easy to use a container with Nginx or Apache or...
I would like to create a custom nuget feed at work, but the computer that I want to host it from doesn't have IIS installed and I can't get it installed either. So I wanted to create my own self-hosted feed, but I'm not sure where to start. I already have a visualSVN server running on this computer, so I know that apache is already installed. Is there a way to host a nuget feed without IIS?
You should be able to use a share. See the section Creating Local Feeds here.
So, if your share is located at \\server\nugetfeed, replace c:\LocalNuGetFeed in the example with that location.
Yes, you can have a local feed running just from a directory on the computer. Once the packages are in a folder, you then need to add that folder as a local NuGet feed in Visual Studio.
Here are a couple of links describing the process:
http://docs.nuget.org/docs/creating-packages/hosting-your-own-nuget-feeds
and this one is a bit older but still relevant I believe:
http://haacked.com/archive/2010/10/21/hosting-your-own-local-and-remote-nupack-feeds.aspx
You could install IIS Express. It could be a simpler solution worth a try if you already know you cannot put the full IIS on the machine.
Unless the same reasons you can't install IIS apply to IIS Express...
Even though it may be too late at this point, my company's product ProGet was designed for hosting private NuGet feeds and ships with an integrated web server - and it looks like the free edition will suit your use case just fine.
I have a small .NET WinForms application, and couple of linux servers, DEV and CL1,CL2..CLN (DEV is development server and CL* are servers which belons to our clients, they are in private networks and it's a kind of production servers)
I want an update mechanism so that
(1) i develop a new version and publish it to a DEV
(2) users of DEV-server install latest version from DEV
(3) users of CL2 (employees of client2) install stable version from CL-2 directly
(4) application checks for updates using server it was installed from (so, if it was installed from CL-2, it should check CL-2 for updates)
(5) i should be able to propogate the update to a selected CL-server (using just file copy & maybe sed; not republishing), if i want that (and if i don't, that CL-server will have an old version until manually i update it)
I tried to use clickonce, but looks like it meets only first two requirements.
What should i do?
ClickOnce should handle 1-4 to be honest. All that would be needed is that for each site you want to deploy/update from, you'll need it to have its own publish, which after looking at your specifications is not incorrect to do.
What you could then do in order to make 5. applicable, is create an automated process to re-publish the file. This could perform a publish and then upload to the correct server.
Remember that ClickOnce needs a new manifest per version, and a new version requires a publish, so I'm not sure that you'll get around 5. with a simple file replacement.
Kyle is right. But for the 5th note, you just need to copy the deployment, and then use mage to modify the installation URL and point it to the new server, and then re-sign the manifests.
I support an app that we deploy to a DEV, QA, PROD servers. The way I handled this is that
I created created a cmd file that has command line calls to MSBUILD. It builds the app once for each server with the appropriate URLs and switches. I give my DEV and QA builds a different AssemblyName that way I can run all 3 environments side by side. This way my build process is automated and I don't have to publish at all.
Here's an article that describes the parameters you can use.
http://msdn2.microsoft.com/en-us/library/ms165431(VS.80).aspx
#Kyle,
For the above solution can the different versions run side by side or do you get errors indicating the app is already installed.