Azure StatefulService not deploying locally due to failure to load - c#

The deployment of an Azure Statefulservice is failing with the following output:
Service Status:
fabric:/Cloud.Application/MyServiceStateless is ready.
fabric:/Cloud.Application/MyServiceStateful is not ready, 1 partitions remaining.
Something is taking too long, the application is still not ready.
Finished executing script 'Get-ServiceFabricApplicationStatus'
This is strange since no other member of my team has seen this problem. To help diagnose it, I set a breakpoint at Program.Main(), but the breakpoint was not hit. Instead, the message "The breakpoint will not currently be hit. A copy of Program.cs was found in StatelessServiceXYZ.exe, but the current source code is different from the version built into StatelessServiceXYZ.exe" when the mouse is hovered over they whited out breakpoint circle. This indicates the assembly isn't loading.
Because no one else can reproduce this problem even though our environments are seemingly identical (we're all using Visual Studio 2017, Microsoft Azure ServiceFabric CoreSDK, .Net Frameword 4.5, and Windows 7), it appears there is some configuration difference or problem with my machine. The other solutions to this problem that I've found online, such as lack of disk space, etc., are not applicable to my situation. I've also tried resetting the local cluster to no avail. Deploying a test StatefulService also worked.

I find a similar issue in the github. if updating the netcore runtime is acceptable, you could have a try to update projects to the 2.0.3 runtime service.

Related

Azure Application Insights HTTP Dependency Tracking Issue

I have a very interesting case with one my applications. To start with, same/similar implementations works fine with my other projects.
Now my main problem is, in this specific project, HTTP remote dependency is not getting tracking. However, everything else is tracked such as normal performance, response times, server requests, custom events, custom exceptions SQL Dependencies. Just the HTTP dependency is not. They used to be tracking and out of nowhere they are not working anymore - this happened without publishing anything to server on the existing production server.
I'm using latest version of the AI packages 2.2.0 and 2.0.7, depending on the package. To point it out again, other projects are fine.
I have tried and checked this behaviour for this project for both Cloud Service and App Service in Azure. Both the same result. I have check tracking logs on debug, I can see SQL and Custom Event tracking but no record for HTTP calls again. Nothing unusual on the Output logs.
Last thing I want to mention is that, I'm deploying this service to multiple cloud services with different Instrument Keys and because of that this is the way I initialise my AI.
TelemetryConfiguration.Active.InstrumentationKey = ConfigurationManager.AppSettings.Get("AzureApplicationInsights");
I have also tried by setting everything into one AI resource via Visual Studio and observed same behaviour.
I'm not sure what else I can add, especially in code level but Azure support wasn't help too much so far, and this is kinda getting important for me as I'm not able see full performance of my system and this putting me in the blind spot for dependency errors.
Edit:
Just to add up, I'm using .Net 4.6.2 with System.Net.Http 4.3.0
Ok, as a last test, I have downgraded System.Net.Http to 4.0.0 from 4.3.0 and locally it tracked dependency now.

Microsoft Service Fabric Host Service (FabricHostSvc) Hangs on Start

I've been working with Microsoft Service Fabric since November 2015 and have encountered many issues but now Service Fabric has become completely non-functional on my development machine. Uninstall/reinstall doesn't help.
I was using 1.5-preview and have since tried 2.0 to no avail.
The problem started when I attempted to run a Service Fabric Application from Visual Studio 2015 Update 1 (as I have done hundreds of times over the past few months).
My machine blue-screened (first time I've seen a Windows 10 blue screen). After rebooting, I was unable to get my Service Fabric Application to deploy via Visual Studio. The PowerShell script failed with the following message:
Starting service FabricHostSvc. This may take a few minutes...
Start-Service : Failed to start service 'Microsoft Service Fabric Host
Service (FabricHostSvc)'.
I went into the SCM and found "Microsoft Service Fabric Host Service" was in a state of Starting. It stayed that way for an hour. I tried stopping and starting the service several times and each time it hangs.
I uninstalled Service Fabric (Service Fabric, SDK and Tools for VS) and re-installed with the latest version 2.0 and it exhibits the same problem.
Reboot, same problem.
Removed c:\SfDevCluster folder, same problem.
Based on some other articles, I looked for any stray performance counters after uninstalling but there weren't any.
I tried looking through the registry but there are other Azure components with "Fabric" in the name. If I delete them, I will probably hose the rest of my Azure dev setup.
Now... when I attempted to start the service again, it did re-create the SfDevCluster folder and give me some logs. It seems to create two trace log files per minute and they have the EXACT same contents.
Every time it fails, the final line of the trace is:
Info ,11176,General.FabricSetup.Main,Operation failed with error
0xffffffff
An earlier trace (SF 1.5) seemed to use a constant rather than the hex value for the error. Seemed to indicate an invalid argument.
Whatever this failure is, it seems to be the cause of my woes. Unfortunately, the error is completely unhelpful.
I'm trying to avoid reinstalling Windows because that will kill an entire day of productivity.
Any help is greatly appreciated.
From an elevated powershell session please run: Unregister-ScheduledTask FabricCounters.
This will fix the issue.
I was seeing very similar behaviour and reinstalls of the fabric SDK and runtime, deleting scheduled tasks, deleting the SfDevCluster contents etc. all didn't work.
I was seeing repeated Docker errors in the event log and when I tried uninstalling Docker for Windows SF instantly woke up. I have no idea what the interaction is between the two but worth checking if you have both installed.
For the benefit of searchers, this is the Powershell script I use to fix my local cluster. It was adapted from this issue fix on github.
#
# WARNING: YOU MUST STOP 'SERVICE FABRIC HOST SERVICE' IN SERVICES FIRST
# IF THE APPLICATION IS STUCK IN 'STARTING', RESTART YOUR MACHINE
#
# This script will completely reset the local cluster
#
Remove-Item 'C:\SfDevCluster' -Recurse -Force -ErrorAction Stop
New-Item -ItemType directory -Path 'C:\SfDevCluster'
Set-Location 'C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup'
./DevClusterSetup.ps1 -PathToClusterDataRoot 'C:\SfDevCluster\Data' -PathToClusterLogRoot 'C:\SfDevCluster\Log'
C:\ drive was full for me. Easy thing to overlook.
Make sure your hard disks aren't full... that was stopping mine from starting. Immediately after clearing up some logs, starts up right away.
It seems like some cluster related settings have gone into an inconsistent state on your machine. This will require looking at Service Fabric traces and figure out the actual cause. I am an engineer on the Service Fabric team. I can help you out if you can email me the Service Fabric traces (from logs folder) at harahma[at]microsoft[dot]com.
If you are familiar with logging support tickets on Azure, I would suggest you do that too so we can track this issue to resolution. In the meantime I will continue to work on this to see how we can unblock you.
I know this is an old and question, but maybe my pain can help someone else out.
You'll get a similar error if the Windows Firewall service is not running when Service Fabric attempts to start.
Check and make sure that the Windows Firewall service is set for Automatic and is running.
I have tried multiple options to resolve this issue like
Uninstall runtime and SDK - Reinstall it.
Removed cluster using powershell commands and setup using the same
and there are couple of more
BUT unfortunately none of them worked for me
Then finally I have Uninstalled Docker Desktop and suddenly issue got resolved.
This is very strange and not sure how docker desktop is obstackle for running FabricHostSvc Service.

Code Cop App just hangs

I written an test application using a Code Cop, a method interception approach.
However, as soon as I ran my first application I hit a snag whereby the application would fire up and hang with no information as to what was happening.
I had followed the code exactly and was able to run the same code on another machine.
There is no error information being output, it just hangs.
Does anyone know how I may be able to solve or debug this issue?
My solution was to contact Ricardo Barbosa at CodeCop who proceeded to help me solve this issue promptly and explaining why this was occurring.
My issue was due to not having the correct CLRJIT.dll on my machine
C:\Windows\Microsoft.NET\Framework
A Windows update solved the issue.
What's Happening
When the CodeCop application runs it creates a folder in %temp%/CodeCop and downloads symbol files from Microsoft to calculate method addresses.
The version I had was 4.6.57.0 in my v4.0.30319 framework folder.
For some reason there was no symbol file from the Microsoft public symbol server for this version of the CLRJIT.dll
Running Fiddler while starting the application showed this to be the case.
After I performed a Windows update I got version 4.6.100.1 of the clrjit.dll the application built and performed as expected.
Thanks to Ricardo for spending the time to solve this issue for me.

SqlServer Types error at build time

I am working on a project in which we use the Geo types form SqlServer.
For this purpose I had to add some DLLs in the solution that gets copied through the build process:
SqlServerTypes.Utilities.LoadNativeAssemblies(Server.MapPath("~/bin"));
This works quite great on my machine but then the guys in my team have build errors because of this.
We are using our local IIS to host the solution and once IIS has started they can't build anymore.
Tho only solution we found now is to restart the app pool (but honestly we're just typing iisreset)
Here is a part of the error message (I stripped out the solution/project name)
Error 15 Could not copy "C:\Workspace\code-...Error 15 Could not copy "C:\Workspace\My-Awesome-Project\Web\MyAwesomeProject.Web\packages\Microsoft.SqlServer.Types.11.0.1\nativeBinaries\x64\msvcr100.dll" to "bin\SqlServerTypes\x64\msvcr100.dll". Exceeded retry count of 10. Failed.
I believe that there is a difference between my computer and the one from my colleagues but we could not point at the one difference that might have an influence on the problem.
Any clue?
Your app pool keeps a lock on those DLLs. Stop the App Pool before building.

InvalidProgramException / Common Language Runtime detected an invalid program

This is the strangest programming issue I have seen in a long time.
I am using Microsoft Visual C# 2010 Express, C# and .NET 2.0 to develop an application. This application references a couple of dll/assemblies (those dlls are all generated on my machine).
Below is part of the code (it is all basic stuff):
public class PowerManagement
{
[TestCase]
public void PrepareTest(){
// Configure according to pre-conditions
Preconditions precondition = new Preconditions();
precondition.SetupPreconditions();
...
}
[TestCase]
public void PerformTest(){
TestcaseData testcaseData = new TestcaseData();
// Set Trigger and perform check
switch (testcaseData.triggerNumber){
case (1):
if ((new Trigger1(testcaseData)).Validate() != 1)
Report.TestStepFail("failed");
break;
...
case (4):
if ((new Trigger4(testcaseData)).Validate() != 1)
Report.TestStepFail("failed");
break;
default:
Report.TestStepFail("Not yet implemented");
break;
}
}
}
This application is then generated into a dll from Visual C# 2010 Express and used elsewhere and all is fine. The problem surfaces when I add another case to the switch-statement above (see below)
...
case (4):
if ((new Trigger4(testcaseData)).Validate() != 1)
Report.TestStepFail("failed");
break;
case (5):
if ((new Trigger5(testcaseData)).Validate() != 1)
Report.TestStepFail("failed");
break;
default:
Report.TestStepFail("Not yet implemented");
break;
I can still build without a single issue and generate the dll but when I use the generated dll I get the following error:
A .NET exception (InvalidProgramException) occured in the module PowerManagement
Error message: Common Language Runtime detected an invalid program.
Throwing method: PowerManagement.PerformTest
(the issue happens even if I copy case(4) and paste it as a new case, so it has nothing to do with Trigger5-class)
What is happening here? I have looked through the other InvalidProgramException and Common Language Runtime in Stackoverflow but none seemed related.
I know this issue is strange so please let me know and I will provide more information. I am using a 64-bit Windows 8 machine, if that matters. I have already checked for any updates on VS and .NET updates. I havet also regenerated all the dlls a couple of time ans also created the solution from scratch a couple of times.
Just wanted to add my experience for this...
In my case, I am hosting my C# Web API on Azure and I encountered this message when trying to log in to my API.
I had to go into my Azure management portal (portal.azure.com), go to App Services, choose my Web API program and click Restart from the Overview screen.
After this, the program worked as normal again.
Did not find any further clues in my logs.
I finally managed to solve this issue.
I unchecked code optimization in C# Express and that solved the issues. Still the weirdest thing, but since we are using old tools and framework we can not really blame anyone.
Try enabling 32-bit applications in your application pool advanced settings.
I had this problem after upgrading to Visual Studio 2017 v15.8.6. The problem went away when I removed the assemblyPostProcessorType attribute in the compilation tag in web.config.
According to MSDN: "Generally this indicates a bug in the compiler that generated the program."
I would start by making sure you have all the updates installed on Windows, .NET and Visual Studio.
You should also check out Q312544 on Microsoft Support.
I've occasionally encountered this error after a deployment to an Azure WebApp using MSDeploy. The error has always disappeared after redeploying for a second time.
Our build and deployment are two different steps, the the redeployment is sending over the exact same files each time - this suggests the problem is not uniquely a compiler issue as suggested elsewhere in this question's responses.
Could be a bug in MSDeploy, or in the version of IIS used for WebApps in Azure perhaps...
Such problem could be caused by bugs in tools manipulating the IL of assembly after compilation, for example if you are using Fody and its plugins. At least there is a bug in Fody MethodDecorator which causes such effect, see
https://github.com/Fody/MethodDecorator/issues/8
If you are having this issue specific to Azure Web Apps - check for installed extension Microsoft.ApplicationInsights.AzureWebSites - or it's friendly name Application Insights extension for Azure App Service and remove it via kudu.
We found this extension was potentially interfering with msdeploy pushes - there was a process snapshotholder_x64.exe running under the IIS w3wp.exe process. Someone likely enabled this extension via the azure portal.
If your issue pertains to a web api dotnetcore deployed to an azure app then this could be caused by application insights. Setting the application insights at the blade level should fix the problem. Also note that there seems to be an unresolved issue with setting it at the blade level as recommended settings vs basic. Basic being the value that works.
Running into this issue deploying a Web Api as an API App on Azure. An initial request to any endpoint would result in the expected response; however, subsequent requests would return the same Common Language Runtime error. I figured out the problem started when I enabled Recommended collection level on the Application Insights blade on my web app. I set recommended and enabled all radio buttons. Reverting this change stopped the error. For reference, the API I am running is running Microsoft.ApplicationInsights 2.8.1
For reference:
https://github.com/dotnet/coreclr/issues/18323
I was doing some powershell automation in a console application using .net core 3.0 when i started receiving this error. I guess .net core is not compatible with System.Management.Automation so i changed it to .net framework 4.7, everything worked well after that
I resolved this issue by doing the following:
Rename C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Performance Tools\vsinstr.exe to C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Performance Tools\vsinstr.exe.broken
Rename C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Performance Tools\vsinstr.legacy.exe to C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Performance Tools\vsinstr.exe
Rebuild the solution
-uncheck "code optimization" (referenced dlls included)
-Upgrade to .net framework 4.6
Out of nowhere we started getting XXX webservice Exception , System.InvalidProgramException: Common Language Runtime detected an invalid program.
Copied compiled website to a different environment - works just fine.
Copied to a different server - works just fine.
I noticed that this exception started when server was reset (either rebooted or app pool reset). When researching this issue, I noticed comment by Brian Reichle on Jun 6 '16 at 12:33
I would expect a bitness mismatch to result in a
BadImageFormatException rather than the InvalidProgramException
described in the question.
I was not familiar with InvalidProgramException exception but I was familiar with BadImageFormatException and the symptom is very similar to the one I get with BadImageFormatException issue. I can't say 100% why either exception happens, current running theory is its a 32-bit application running on a 64-bit machine, but we couldn't prove it nor could we perma-fix it. Enabling 32-bit applications on app pool did not fix the issue.
The only fix we know of, albeit temporary, is to simply recycle App Pool. No need to recompile or anything. Luckily, this happens not too often, maybe once a month or two.
I just ran into this problem myself. Even though VS created the virtual directory for me, it had vb as the default language, but I have a C# application. Changing this setting solved it.
This is an interesting exception that i came across while hosting on IIS. I solved it after finding out that my .NET Framework version on IIS was different from the .NET Framework version that my project was using. Please note if you happen to have other referenced projects/ddl(s) make sure you update their .NET Framework version too.
In my case it was Hasp protection (google: hasp sentinel protection key) SW which ruined the dlls.
To share my experience: had the same issue on x86 computer with my WindowsForms app, found out I've forgotten to copy .exe.config file with all dll redirections, after that everything worked like a charm.
In my case, the exception was caused by datadog tracer after upgrading from 3.1 to NET 6, by upgrading to their latest version, the issue was fixed.
Here is the diff in my dockerfile.
-RUN curl -LO https://github.com/DataDog/dd-trace-dotnet/releases/download/v1.12.0/datadog-dotnet-apm_1.12.0_amd64.deb
-RUN dpkg -i ./datadog-dotnet-apm_1.12.0_amd64.deb
+RUN curl -LO https://github.com/DataDog/dd-trace-dotnet/releases/download/v2.14.0/datadog-dotnet-apm_2.14.0_amd64.deb
+RUN dpkg -i ./datadog-dotnet-apm_2.14.0_amd64.deb
https://github.com/DataDog/dd-trace-dotnet

Categories

Resources