Quartz.Net scheduler service does not start automatically on startup - c#

I have developed a Quartz.Net windows service to run scheduled jobs and I have set up an SQLite jobstore for it to work. Here are the settings of my AdoJobStore:
# SQLite settings
quartz.jobStore.type = Quartz.Impl.AdoJobStore.JobStoreTX, Quartz
quartz.jobStore.misfireThreshold = 60000
quartz.jobStore.lockHandler.type = Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore, Quartz
quartz.jobStore.useProperties = true
quartz.jobStore.dataSource = default
quartz.jobStore.tablePrefix = qrtz_
quartz.jobStore.driverDelegateType = Quartz.Impl.AdoJobStore.SQLiteDelegate, Quartz
quartz.dataSource.default.provider = SQLite-10
quartz.dataSource.default.connectionString = Data Source=.\jobs.db;Version=3
Before giving you a headache, I should say that this is a working solution! That means the job store works just fine! Each job consists of running an .exe file that will get its parameters from an SQL Server database using a connection string that is stored in a text file.
So if the service is started automatically, everything works fine.
After installing the Windows service, we give it a username that has access to the destination database in service's Log On tab. We set the service to start automatically (and have tried the Automatic Delayed mode as well). In the Recovery tab of the service we also command it to Restart in case it crashes.
However, the service does not seems to be starting by itself on some machines and that has kept me wondering for quite a while now. On machines with failure, this is what my service logs when it fails to start:
2015-02-13 15:09:15,674 [1] ERROR Quartz.Server.QuartzServer [(null)] - Server initialization failed:Unable to bind scheduler to remoting.
Quartz.SchedulerException: Unable to bind scheduler to remoting. ---> System.Net.Sockets.SocketException: No such host is known
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
at System.Runtime.Remoting.Channels.CoreChannel.GetMachineIp()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.SetupMachineName()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider, IAuthorizeRemotingConnection authorizeCallback)
at System.Runtime.Remoting.Channels.Tcp.TcpChannel..ctor(IDictionary properties, IClientChannelSinkProvider clientSinkProvider, IServerChannelSinkProvider serverSinkProvider)
at Quartz.Simpl.RemotingSchedulerExporter.RegisterRemotingChannelIfNeeded()
at Quartz.Simpl.RemotingSchedulerExporter.Bind(IRemotableQuartzScheduler scheduler)
at Quartz.Core.QuartzScheduler.Bind()
at Quartz.Core.QuartzScheduler.Initialize()
--- End of inner exception stack trace ---
at Quartz.Core.QuartzScheduler.Initialize()
at Quartz.Impl.StdSchedulerFactory.Instantiate()
at Quartz.Impl.StdSchedulerFactory.GetScheduler()
at Quartz.Server.QuartzServer.Initialize() [See nested exception: System.Net.Sockets.SocketException (0x80004005): No such host is known
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
at System.Runtime.Remoting.Channels.CoreChannel.GetMachineIp()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.SetupMachineName()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider, IAuthorizeRemotingConnection authorizeCallback)
at System.Runtime.Remoting.Channels.Tcp.TcpChannel..ctor(IDictionary properties, IClientChannelSinkProvider clientSinkProvider, IServerChannelSinkProvider serverSinkProvider)
at Quartz.Simpl.RemotingSchedulerExporter.RegisterRemotingChannelIfNeeded()
at Quartz.Simpl.RemotingSchedulerExporter.Bind(IRemotableQuartzScheduler scheduler)
at Quartz.Core.QuartzScheduler.Bind()
at Quartz.Core.QuartzScheduler.Initialize()]
The service is installed on host 127.0.0.1 port 555 and does not seem to need a dependency because it works under some other machines.
I have a feeling that the service is trying to start itself before there is an access to the database or before the TCP host is established. But if that's the case, why does it work on other devices?
I have asked the same question before but have failed to find a working answer. Any help from Quartz.Net dev team or Windows Services experts is greatly appreciated.

Disclaimer. I'm not part of the Quartz.net dev team, nor am I an expert on Windows services
The service is installed on host 127.0.0.1 port 555 and does not seem to need a dependency because it works under some other machines.
Your reasoning is false. It could be that the required services just happen to have always started on the other machines, but without setting a dependency this is not guaranteed.
So, since you are getting a DNS error, I would suggest you start by setting your service do that it depends on Dnscache and Tcpip .

Related

Dns.GetHostAddressesAsync: Resource temporarily unavailable

First, for some context: I am using .NetCore to develop an API with Identity. Everything is on a Cloud server, inside a Docker. When a user is created, an email is sent to the new User using a mailkit and the webmail server through Plesk (Hosted on the same machine). The docker is accessed via a redirection trough Apache using a ProxyPass from the subdomain to the port on localhost
Everything works great while debugging trough JetBrain's Rider, but it is not able to process the email in the docker on the server.
Here is the stack:
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
at System.Net.Dns.InternalGetHostByName(String hostName)
at System.Net.Dns.ResolveCallback(Object context)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
I have yet to try and run the docker on another linux machine to test.
One of my current guess would be a problem with the SSL certificate, but I don't think it would cause a problem with the DNS or any internal socket.
Another guess is thats its a problem for the Docker to get the DNS Hostname, but since it works ok in a local.
Edit: I tried multiple time to run the docker on the mac and the error is still triggered once in a while but not always. It is although always triggered on the server and never send the email
After two weeks of research, I finally stumbled upon a solution for this:
The problem is related to the network, that was obvious, but it's precisely about how containers are isolated from one another. Problem is, the container has no outbound connection. A solution that work inside a standalone container is to use the --network host parameter, which would expose the host network to the container. Note that using this would remove the port mapping from the container since the container's port 5000 is now linked to the host's port 5000
Hope this solution can help others

TIBCO.EMS.IllegalStateException: connect failed: server in standby mode

We have a big system with bunch of services communicating with each other via TIBCO EMS Messaging.
Now, one of the services keeps throwing this exception when it tries to connect to EMS server:
TIBCO.EMS.IllegalStateException: connect failed: server in standby mode
However, other services on the same EMS server are working just fine without any exception.
I tried to reproduce this by creating a console application which would use same credentials to connect to EMS server on the same destination (Queue) but couldn't reproduce this error.
EMS connection is fault-tolerant and it might have been failed over to secondary server, however, now I see primary server is back up.
I also tried to connect to primary server only (which I can see is up & running) but it fails with the same exception.
Below is the stack trace for reference:
TIBCO.EMS.IllegalStateException: connect failed: server in standby mode
at TIBCO.EMS.CFImpl._CreateConnection(String userName, String password, Boolean xa)
at TIBCO.EMS.ConnectionFactory.CreateConnection(String userName, String password)
at Ems.makeConn()
We're using TIBCO.EMS.dll v. 8.1.0.10 on .NET 4.0.
Any idea what can cause this exception ?
Could your connection string only mention one of the hosts inside an FT/HA setup ? This looks like the error message returned when you are trying to connect to only one of the HA/FT host, but not the one active at the moment.
If it is the case, the connection string will work most of the time, but a proper EMS HA connection string include two (or more!) EMS host and port. Only one is active at the same time.
It typically looks like this:
serverUrl=tcp://server0:7222, tcp://server1:7344
See source doc here in TIBCO docs.
It turned out tools we were using to check if server is active (we don't have admin rights on the servers) was misbehaving (showing incorrect status), so none of those servers were really active (not primary nor secondary).
As soon as server was brought up this error message went away.

Azure service bus relay connecting to unknown ip address: 40.112.124.x:9352

We deliver on-premise software that is exposed to the cloud using Azure Service bus relay, the basic code we use to expose is as follows (I have removed everything identifiable):
ServiceHost sh = new ServiceHost(typeof(BasicHttpEntityService));
BasicHttpRelayBinding basicHttpRelayBinding = new BasicHttpRelayBinding();
Uri uriEndPointAddress = ServiceBusEnvironment.CreateServiceUri("https", "ourdomain", "test-url-appendage");
m_shRelayServiceHost.AddServiceEndpoint(
typeof(IMyService),
basicHttpRelayBinding,
uriEndPointAddress
).Behaviors.Add(
new TransportClientEndpointBehavior
{
TokenProvider = TokenProvider.CreateSharedSecretTokenProvider(
"MyUser",
"MyPassword")
});
sh.Open();
This works fine at most of our customers, however, one of our customers has a strict firewall policy.
According to the SB guidelines we have found, we asked them to open ports 9351-9354 to ourdomain.servicebus.windows.net. Now we find out that when there is an incoming request, the service connects to both 'ourdomain' (we see this succeeds in Wireshark, and also in the WCF log) AND an unknown (to us) service on 40.112.124.x:9352 (the last octet changes with every request).
I have been able to reproduce the problem in my development environment by disallowing connections to any 40.x.x.x address on any port. This is what happens in the WCF log:
System.Net.Sockets.SocketException (0x80004005): An attempt was made to access a socket in a way forbidden by its access permissions 40.112.124.25:9352
Server stack trace:
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at Microsoft.ServiceBus.RelayedConnectionSession.ConnectAsyncResult.<GetAsyncSteps>b__4(ConnectAsyncResult thisRef, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
Exception rethrown at [0]:
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.RelayedConnectionSession.EndConnect(IAsyncResult result)
There is no DNS-request going out during this time, so there is no host name that provides any clues to the function of this outgoing connection.
From my investigation, this appears to be a Microsoft controlled subnet, so I'm fine with the relay service connecting to it, but I would like to know:
Is this additional connection optional?
If not, should we allow the entire subnet?
Could this IP-range change in the future? Is it hardcoded somewhere?
In the end, we requested support from Microsoft.
In short their answers were as follows:
Is this additional connection optional?
No it is not optional. For the relay listener, there is a control channel on port 5671, this connection is always there. Then there is a data channel on portal 9352, this connection established when there is a relay client tries to communicate with the listener.
Could this IP-range change in the future?
Currently, for relay this IP can change, so you need to allow the IP range for the entire datacenter in your region (https://www.microsoft.com/en-us/download/confirmation.aspx?id=41653). The SB product team will be working on to significantly reduce this IP range in the future, to make it much more predictable. There is no exact ETA on this future.
So the good news is they are working on it. The bad news is, that right now, we will need to add a LOT of IP addresses to the white-list to ensure smooth operation.

C# SocketException "No such host is known" when creating TcpServerChannel

I'm currently developing a client-server-based game using C# remoting. Everything seemed to work fine until I tried to set up the TcpServerChannel. I'm using the complex constructor (with IDictionary and stuff), but the problem seems to appear even with the normal TcpServerChannel(int port) constructor.
A SocketException with the message "No such host is known" is thrown when I try to create the channel instance. Stack trace:
at System.Net.Dns.GetAddrInfo(String name)
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
at System.Runtime.Remoting.Channels.CoreChannel.GetMachineIp()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.SetupMachineName()
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider, IAuthorizeRemotingConnection authorizeCallback)
at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel..ctor(IDictionary properties, IServerChannelSinkProvider sinkProvider)
at Some.Namespace.Server.Start()
However, after some research I found out that the problem is the CoreChannel.GetMachineIp() method. I found the code of the CoreChannel class here and noticed, that the Dns.GetHostEntry(string hostNameOrAddress) method is called twice. In the first call the return value of the Dns.GetHostName() method (which correctly returns my computer name, lets say we got "hostname" here) is passed as argument while in the second call the returned IPHostEntry.HostName is passed. Here is where the problem occurs: Since my computer is inside a domain (lets call it "domain"), the returned IPHostEntry.HostName contains "hostname.domain" instead of just "hostname". The second Dns.GetHostEntry("hostname.domain") call then throws an exception because "hostname.domain" is not known.
There is a similar question, but it didn't help me in any way.
I don't know why this thing is working that way, but I can't change my computer name or the domain, so I am stuck at this point and really need some help.
I am using Windows 7 64bit (if it matters), VS Professional 2012 Update 4 and .NET Framework 4.5.
Thanks for the help

Why can't I receive a message from my remote public transactional queue?

I'm using C# on Windows Server 2008, and I want to receive a message from a public transactional queue on another machine in the same domain. The error looks like this:
System.Messaging.MessageQueueException: Cannot import the transaction.
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Receive(TimeSpan timeout, MessageQueueTransactionType transactionType)
at JobManagerLib.JobProcessor.Process(Action waitForNewMessageCallback) in C:\Dev\OtherProjects\POC\WindowsService\JobManagerSample\JobManagerLib\JobProcessor.cs:line 132
I've tried DTCPing, which succeeds in one direction but fails in the other. Here is the pertinent part of the log:
++++++++++++hosts ++++++++++++
127.0.0.1 localhost
::1 localhost
08-20, 15:47:22.739-->Error(0x424) at clutil.cpp #256
08-20, 15:47:22.739-->-->OpenCluster
08-20, 15:47:22.739-->-->1060(The specified service does not exist as an installed service.)
++++++++++++++++++++++++++++++++++++++++++++++
DTCping 1.9 Report for DEV-MSMQ2
++++++++++++++++++++++++++++++++++++++++++++++
RPC server is ready
++++++++++++Validating Remote Computer Name++++++++++++
08-20, 15:47:26.207-->Start DTC connection test
Name Resolution:
dev-msmq1-->192.168.22.11-->Dev-msmq1
08-20, 15:47:26.222-->Start RPC test (DEV-MSMQ2-->dev-msmq1)
RPC test failed
Does anybody have any idea why this might be failing? The Windows Firewall has been opened for MSDTC. It's hard to find much info about Windows 2008 and MSMQ.
EDIT: The queue names are FormatName:DIRECT=OS:dev-msmq1\getmap, and FormatName:DIRECT=OS:dev-msmq1\logevent. They are public, transactional queues, and Everyone has peek/receive permission on them. The pertinent part of my code is as follows:
using (TransactionScope tx = new TransactionScope(TransactionScopeOption.RequiresNew))
{
using (var queue = new MessageQueue(QueueName))
{
queue.Formatter = new XmlMessageFormatter(new string[] { _targetParameterType });
var message = queue.Receive(TimeOut, MessageQueueTransactionType.Automatic);
string messageId = message.Label;
...
}
}
Thanks
So I did find a solution: abandon the whole thing altogether and switch to using WCF and the net.Msmq binding. Now the queue communication is working fine.
Make sure, both machine's clocks are in sync. I seen this before where authentication will fail because the server and the client are off time by a minute. This will happen even if the queues are public and have permissions on everyone.
Just for completeness, allowing only MSDTC and MSMQ through the firewall is not enough when running IPv4:
You need to let ICMP traffic through the firewall as well (IPv6 can resolve host names regardless of the firewall, but your DTCPIng log indicates you are running IPv4).
I have struggled with the same error you see in DTCPing, and in my case it turned out to be triggered by the firewall blocking the ICMP traffic.

Categories

Resources