When i have 2000 messages in an import queue, when i read it through get like this
MQQueue mqQueue = null;
MQQueueManager mqQMgr=null;
try
{
//Create connection to queue manager
mqQMgr = new MQQueueManager("Queue Manager name", properties);
//Access the queue
mqQueue = mqQMgr.AccessQueue(QueueName, MQC.MQOO_FAIL_IF_QUIESCING | MQC.MQOO_INPUT_SHARED | MQC.MQOO_BROWSE);
for(int i=1;i<2000;i++)
{
//read the messages
mqMsg=new MQMessage();
mqQueue.Get(mqMsg);
}
}
catch(MQException mqe)
{
//If no messages in the queue , break. (if not, catch any error)
}
finally
{
mqQueue.Close(); //Close the MQ Queue
mqQMgr.Disconnect(); //Disconnect the MQ Manager
}
The manager processes randomly everytime when i read, for example first 800 messages and then it throws back
"MQRC_HANDLE_NOT_AVAILABLE" (comp code:2 , reason code:2017).
Am I not closing the queue/manager in the above piece of code ? Please share your thoughts, so that, all the messages can be processed. I am using 7.5 MQ client via MQ.net classes. The log file contains this below info.
AMQ9051: WebSphere MQ could not find the security policy definition.
Compcode 2 : reason 2017 EXPLANATION: The security policy definition
is not defined. ACTION: Security policy definition must be defined
before this action.
I tried your sample code snippet and works fine for me. I don't see 2017 error.
The entry in error log may have a relation to your problem. Have you set security policy for the queue and expecting signed/signed&encrypted messages to be put/get to/from queue? It may be worth opening a ticket/PMR with IBM.
Related
I'm using a very basic example for testing my MassTransit connection to rabbitMq through C#. Whenever I run this code to connect to my rabbitMq endpoint, it works fine whenever I have a wildcard set as permission. However, when I add the permissions in the rabbitMq admin to only allow this user to access the test event, this code will fail.
For some reason it will first try to connect to an exchange name that I guess is generated (by MassTransit?):
RabbitMQ.Client.Exceptions.OperationInterruptedException: 'The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=403, text="ACCESS_REFUSED - access to exchange '*ComputerName*_dotnet_bus_73451gfsgerqwrefxfddgf' in vhost '/' refused for user 'user'", classId=40, methodId=10, cause='
So the exchange *ComputerName*_dotnet_bus_73451gfsgerqwrefxfddgf, after that it will try to connect to the test exchange. Of course I can add the ComputerName.... exchange to the permissions but then this would need to be done for each computer trying to run this code. Why is MassTransit trying to connect to this exchange? Is the code incorrect or is this just how MassTransit works?
This is the code for the test application (I altered this a bit so it might not run right of the bat, but, in general the code runs fine):
using System;
using MassTransit;
namespace Test
{
public class Testing
{
public string Id { get; set; }
}
}
namespace Consumer
{
class Program
{
static void Main(string[] args)
{
var bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://servername"), h =>
{
h.Username("user");
h.Password("user");
});
sbc.ReceiveEndpoint(host, "test", ep =>
{
ep.Handler<Testing>(context =>
{
return Console.Out.WriteLineAsync($"Received: {context.Message.Id}");
});
});
});
bus.Start();
// For testing purposes, we send a message ourselves.
bus.Publish(new Testing { Id = "X" });
Console.WriteLine("Waiting for messages. Press any key to exit");
Console.ReadKey();
bus.Stop();
}
}
}
Image of the rabbitMq admin user permission:
Is the code incorrect or is this just how MassTransit works?
More than likely this is just how MassTransit works. See this other question: Prevent MassTransit from creating a RabbitMQ exchange for a consumer host
Strangely enough I could not find this information anywhere in the MassTransit docs.
You'll have to grant the configure permission for exchanges containing the string dotnet_bus. More than likely other permissions like read and write will be required.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I'm building a Kafka consumer for my .net proyect. I'm using kafka-net (A native c# client fro Apache Kafka mady by James Roland).
The problem I have is that this code (based on the documentation) fetches all messages from the beginning by default:
private void StartKafkaConsumer(string ipKafka, string portKafka, string topicKafka)
{
string topic = topicKafka;
Uri uri = new Uri($"http://{ipKafka}:{portKafka}");
var options = new KafkaOptions(uri);
using (var router = new BrokerRouter(options))
{
using (var consumer = new Consumer(new ConsumerOptions(topic, router)))
{
foreach (var message in consumer.Consume())
{
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
}
}
}
}
...
StartKafkaConsumer("localhost", "9092", "test"); //this fetches messages sent weeks ago, since the creation of the 'test' topic
Basically, this code does the same than this command:
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
All I want to do is to fetch messages from the time the client gets connected to the Apache server, not from the beginning. I know it's possible since I tried the last command without the "--from-beginning" part and it worked.
Any suggestion would be appreciated.
Look at the ConsumerOptions methods. There should be an option to set a property/config value:
auto.offset.reset
Set the above property/config to latest. At that point, anytime you connect with an unknown/new consumer group id, the consumer will default to start at the latest offset.
If, however, the consumer group id was already known (i.e., had already consumed at least once from that topic/partition), it will attempt to get the last committed offset + 1. If that offset is unavailable, because maybe it has gone past the retention threshold, then it will default to latest.
Here's more detailed documentation:
https://kafka.apache.org/documentation/#newconsumerconfigs
I need to use CCDT file for MQ.NET client to connect to MQ Server, both are running locally, but got error below.
IBM.WMQ.MQException: MQRC_Q_MGR_NAME_ERROR CompCode: 2, Reason: 2058
Please find below the server setup, and MQ.NET client.
Server Setup
I setup and created CCDT file following the links below:
Setting up the server using IBM MQ Explorer
Server-connection Channel: LOCAL.DEF.SVRCONN
MCA User ID: I tried either omitting it or providing it. And either member of mqm or not of it.
Setting up the client using IBM MQ Explorer
Clinet channe: LOCAL.DEF.SVRCONN
Queue Manager name: LocalQM
Connection name: 192.168.1.9 (1415)
192.168.1.9 is localhost address
1415 is queue manager, LocalQM, TCP port.
IBM MQ.NET
The code is below from here
MQQueueManager qm = null;
System.Environment.SetEnvironmentVariable("MQCHLLIB", "C:\\ProgramData\\IBM\\MQ\\qmgrs\\LocalQM\\#ipcc");
System.Environment.SetEnvironmentVariable("MQCHLTAB", "AMQCLCHL.TAB");
try
{
Hashtable props = new Hashtable();
props.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
qm = new MQQueueManager("LocalQM",props);
MQQueue queue1 = qm.AccessQueue("LocalQueue", MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING);
MQMessage msg = new MQMessage();
msg.WriteUTF("Hello this message is from .net client");
queue1.Put(msg);
queue1.Close();
qm.Disconnect();
}
catch (Exception ex)
{
Console.Write(ex);
}
IBM MQ V8 on Windows 10
MQ.NET Client V8 on Windows 10
Creating a client channel definition table
Using a client channel definition table with .NET
UPDATE
If I change to below without using CCDT, it works.
var properties = new Hashtable
{
{MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED },
{ MQC.HOST_NAME_PROPERTY, "192.168.1.9" }, //"localhost" },
{ MQC.PORT_PROPERTY, "1415"},
{ MQC.CHANNEL_PROPERTY, "LOCAL.DEF.SVRCONN" },
{MQC.USER_ID_PROPERTY, "xxx" }, //usrename
{MQC.PASSWORD_PROPERTY, "xxx" } //password
};
_queueManager = new MQQueueManager(_queueManagerName, properties);
UPDATE 2
I changed queue manager to LocalQM
C:\Users\'#.lp\source>SET MQCHLLIB=C:\ProgramData\IBM\MQ\qmgrs\LocalQM\#ipcc
C:\Users\'#.lp\source>SET MQCHLTAB=AMQCLCHL.TAB
C:\Users\'#.lp\source>echo DIS CHL(LOCAL.DEF.SVRCONN) CHLTYPE(CLNTCONN) ALL | runmqsc -n
5724-H72 (C) Copyright IBM Corp. 1994, 2015.
Starting local MQSC for 'AMQCLCHL.TAB '.
1 : DIS CHL(LOCAL.DEF.SVRCONN) CHLTYPE(CLNTCONN) ALL
AMQ8414: Display Channel details.
CHANNEL(LOCAL.DEF.SVRCONN) CHLTYPE(CLNTCONN)
AFFINITY(PREFERRED) ALTDATE(2018-04-16)
ALTTIME(22.16.03) CERTLABL( )
CLNTWGHT(0) COMPHDR(NONE)
COMPMSG(NONE) CONNAME(192.168.1.9 (1415))
DEFRECON(NO) DESCR( )
HBINT(300) KAINT(AUTO)
LOCLADDR( ) MAXMSGL(4194304)
MODENAME( ) PASSWORD( )
QMNAME(LocalQM) RCVDATA( )
RCVEXIT( ) SCYDATA( )
SCYEXIT( ) SENDDATA( )
SENDEXIT( ) SHARECNV(10)
SSLCIPH( ) SSLPEER( )
TPNAME( ) TRPTYPE(TCP)
USERID( )
No commands have a syntax error.
C:\Users\'#.lp\source>
UPDATE 3: MQRC_NOT_AUTHORIZED is Not resolved
I changed
192.168.1.9 (1415) -> 192.168.1.9(1415)
Put AMQCLCHL.TAB also to C:\ProgramData\IBM\MQ (I don't know why which might not be correct, because error on log file:
AMQ9518: File 'C:\ProgramData\IBM\MQ\AMQCLCHL.TAB' not found.
)
3 I tried
MCA User Id ->
1 mqlclient who is memeber of mqm
2 mqlclient who is not memeber of mqm
3 empty
But still got error. Below is exception and error for 3.1
IBM.WMQ.MQException: MQRC_NOT_AUTHORIZED
----- cmqxrsrv.c : 2356 -------------------------------------------------------
17/04/2018 23:50:44 - Process(1848.16) User(SYSTEM) Program(amqzlaa0.exe)
Host(APPLE) Installation(Installation1)
VRMF(8.0.0.5) QMgr(LocalQM)
AMQ5540: Application 'bin\Debug\Producer.exe' did not supply a user ID
and password
EXPLANATION:
The queue manager is configured to require a user ID and password, but none was
supplied.
ACTION:
Ensure that the application provides a valid user ID and password, or change
the queue manager configuration to OPTIONAL to allow applications to connect
which have not supplied a user ID and password.
----- amqzfuca.c : 4311 -------------------------------------------------------
17/04/2018 23:50:44 - Process(1848.16) User(SYSTEM) Program(amqzlaa0.exe)
Host(APPLE) Installation(Installation1)
VRMF(8.0.0.5) QMgr(LocalQM)
AMQ5541: The failed authentication check was caused by the queue manager
CONNAUTH CHCKCLNT(REQDADM) configuration.
EXPLANATION:
The user ID 'mqclient' and its password were checked because the user ID is
privileged and the queue manager connection authority (CONNAUTH) configuration
refers to an authentication information (AUTHINFO) object named
'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with CHCKCLNT(REQDADM).
This message accompanies a previous error to clarify the reason for the user ID
and password check.
ACTION:
Refer to the previous error for more information.
Ensure that a password is specified by the client application and that the
password is correct for the user ID. The authentication configuration of the
queue manager connection determines the user ID repository. For example, the
local operating system user database or an LDAP server.
To avoid the authentication check, you can either use an unprivileged user ID
or amend the authentication configuration of the queue manager. You can amend
the CHCKCLNT attribute in the CHLAUTH record, but you should generally not
allow unauthenticated remote access.
-------------------------------------------------------------------------------
17/04/2018 23:50:45 - Process(14900.9) User(SYSTEM) Program(amqrmppa.exe)
Host(APPLE) Installation(Installation1)
VRMF(8.0.0.5) QMgr(LocalQM)
AMQ9557: Queue Manager User ID initialization failed for 'mqclient'.
EXPLANATION:
The call to initialize the User ID 'mqclient' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2356 -------------------------------------------------------
OP added the following information to the question in UPDATE 2:
SET MQCHLLIB=C:\ProgramData\IBM\MQ\qmgrs\QM1\#ipcc
SET MQCHLTAB=AMQCLCHL.TAB
echo DIS CHL(LOCAL.DEF.SVRCONN) CHLTYPE(CLNTCONN) ALL | runmqsc -n
I see two possible issues with the CCDT:
You have a space in the CONNAME between the IP and the open parenthesis. This may be causing MQ to disregard the port 1415 and use the default port 1414, it also may ignore the space and it may not be a problem, I have never tested with a space.
COMPMSG(NONE) CONNAME(192.168.1.9 (1415))
The QMNAME of the CLNTCONN channel should match what you are specifying in the call to new MQQueueManager, so this should say QM1 not LocalQM, this is likely the direct reason for your 2058 error, but #1 may cause an issue as well once you fix #2.
QMNAME(LocalQM) RCVDATA( )
There are also other uses for QMNAME where you can specify a group name or a blank. See my answer to "Connecting to IBM MQ using CCDT file in JMS
" for more details on this. The answer is related to JMS but the information presented for the setQueueManager would be the same as the first parameter to your call to new MQQueueManager.
Generally speaking, reason code of 2058 (MQRC_Q_MGR_NAME_ERROR) means that you have an invalid queue manager name or that the server does not have that particular queue manager. Check the spelling of the queue manager name that you inputted. Also, queue manager names are case sensitive (e.g. MQA1 is not the same as mqa1).
Also, every queue manager will have its own port #. i.e. 1414, 1415, 1416, etc. So, if there is more than 1 queue manager running on that server then make sure you are using the correct port.
System.Environment.SetEnvironmentVariable("MQCHLLIB", "C:\\ProgramData\\IBM\\MQ\\qmgrs\\QM1\\#ipcc");
System.Environment.SetEnvironmentVariable("MQCHLTAB", "AMQCLCHL.TAB");
Try setting the environment variables before starting your program rather than from inside your program. I have found weird issues in Windows and/or .NET with setting environment variables inside a program - sometimes it works, sometimes it doesn't.
Finally, are you running your application on the same server as where the queue manager is running? If so, then connect in 'bindings mode' rather than 'client mode'. Bindings mode will be way faster as there is no network involved.
I am sending an MQ message and getting a messageID and correlationID back in return as expected, and I can capture the response from specific message that I send using the messageID returned by the MQ server.
I put my application into a load testing tool and I saw that in some cases the messageID returned by the queue manager was the same as a previous message, and in these cases the app failed to read the next message with the same messageID.
I'm not the owner of the queue manager and the response from the admin was "created your own message id".
Can I do that? Does the messageID need to have a specific format?
This is my code :
message = strInputMsg;
queueMessage = new MQMessage();
queueMessage.WriteString(message);
queueMessage.Format = MQC.MQFMT_STRING;
queueMessage.Expiry = timeOutExpiry;
queueMessage.ReplyToQueueName = QueueNameExpiry;
queuePutMessageOptions = new MQPutMessageOptions();
queuePutMessageOptions.Options = MQC.MQRO_COPY_MSG_ID_TO_CORREL_ID;
queue.Put(queueMessage, queuePutMessageOptions);
bytReturn = queueMessage.MessageId;
So can I set the MessageID property to my own message ID value before I send the message, like the below?
queueMessage.MessageId = myOwnMessageId
Yes, it's possible that the code sets the message ID explicitly but your code appears to not reuse the MQMD structure which is how that normally happens. It is more likely based on your description and code provided that the load testing tool is replaying the same messages multiple times and preserving the Message ID while doing so.
If MQ is allowed to set the Message ID it guarantees this to be unique within the queue manager for IDs that it generates. It does not guarantee a GUID across many queue managers but does attempt to ensure no collisions by including 12 characters of the QMgr name in the Message ID. So although we have no information as to which load testing tool is being used and if it employs message replay, that possibility seems much more likely than that MQ has a bug that duplicates message IDs during execution of the .Net MQMessage() class constructor.
Please see:
MQMessage.NET class which says "Creates an MQMessage object with default message descriptor information..." Of course, the default MQMD causes the QMgr to generate the Message ID.
MQMD - MsgId (MQBYTE24) which explains in a note at the bottom how MsgID is made to be unique and in the body how it can be controlled by the application putting the messages.
I am trying to connect to a test instance of a Websphere MQ 7.5 server (on a Windows domain) from a C# application using the full MQ Client install, but I've had no luck. I keep getting the error code 2035 MQRC_NOT_AUTHORIZED from the MQ server. Initially I tried the code below:
string QueueManagerName = "myNewQManager";
MQEnvironment.Hostname = "tst-mqsvr";
MQEnvironment.Channel = "test.channel";
MQEnvironment.Port = 1414;
MQEnvironment.UserId = "domainUser";
MQEnvironment.Password = "********";
//set transport properties.
MQEnvironment.properties[MQC.TRANSPORT_PROPERTY] = MQC.TRANSPORT_MQSERIES_CLIENT;
try
{
queueManager = new MQQueueManager(QueueManagerName);
}
catch(MQException mqexp)
{
//I get the error code 2035
log.Error(mqexp)
}
changing to this doesn't work either:
string QueueManagerName = "myNewQManager";
var queueProperties = new Hashtable();
queueProperties[MQC.TRANSPORT_PROPERTY] = MQC.TRANSPORT_MQSERIES_CLIENT;
queueProperties[MQC.HOST_NAME_PROPERTY] = "tst-mqsvr";
queueProperties[MQC.PORT_PROPERTY] = "1414";
queueProperties[MQC.CHANNEL_PROPERTY] = "test.channel";
queueProperties[MQC.USER_ID_PROPERTY] = "domainUser";
queueProperties[MQC.PASSWORD_PROPERTY] = "*********";
try
{
queueManager = new MQQueueManager(QueueManagerName, queueProperties);
}
catch(MQException mqexp)
{
//I still get the error code 2035
log.Error(mqexp)
}
This error goes away and I can connect/put/get messages from my queue if
I manually set the MCAUSER on my channel, or
I run the application as the domain user who set up the MQ server (using RunAs)
However, I will not be able to use either of these two workarounds for the production set up. I am assuming I'm getting the error code 2035 because the MQ client is sending the wrong user-id. I need to be able to override (or at least determine) the flowed userid during server connection. How do I go about this?
Edit: I got the errors below in my AMQERR01.log file
-------------------------------------------------------------------------------
7/6/2016 13:06:14 - Process(1380.10) User(MUSR_MQADMIN) Program(amqzlaa0.exe)
Host(TST-MQSVR) Installation(DefaultInstall)
VRMF(7.5.0.2) QMgr(MYNEWQMANAGER)
AMQ8075: Authorization failed because the SID for entity 'sinistrian' cannot be
obtained.
EXPLANATION:
The Object Authority Manager was unable to obtain a SID for the specified
entity. This could be because the local machine is not in the domain to locate
the entity, or because the entity does not exist.
ACTION:
Ensure that the entity is valid, and that all necessary domain controllers are
available. This might mean creating the entity on the local machine.
----- amqzfubn.c : 2273 -------------------------------------------------------
7/6/2016 13:06:14 - Process(1380.10) User(MUSR_MQADMIN) Program(amqzlaa0.exe)
Host(TST-MQSVR) Installation(DefaultInstall)
VRMF(7.5.0.2) QMgr(MYNEWQMANAGER)
AMQ8073: Authorization failed because SID: ((None)) could not be resolved.
EXPLANATION:
The Object Authority Manager was unable to resolve the specified SID into
entity and domain information.
ACTION:
Ensure that the application provides a SID that is recognized on this system,
that all necessary domain controllers are available, and that the security
policy is set as you required.
----- amqzfubn.c : 4397 -------------------------------------------------------
7/6/2016 13:06:14 - Process(3008.3) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(TST-MQSVR) Installation(DefaultInstall)
VRMF(7.5.0.2) QMgr(MYNEWQMANAGER)
AMQ9557: Queue Manager User ID initialization failed.
EXPLANATION:
The call to initialize the User ID failed with CompCode 2 and Reason 2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 1975 -------------------------------------------------------
7/6/2016 13:06:14 - Process(3008.3) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(TST-MQSVR) Installation(DefaultInstall)
VRMF(7.5.0.2) QMgr(MYNEWQMANAGER)
AMQ9999: Channel 'TEST.CHANNEL' to host 'WKS-L450 (192.168.10.23)' ended
abnormally.
EXPLANATION:
The channel program running under process ID 3008(2192) for channel
'TEST.CHANNEL' ended abnormally. The host name is 'WKS-L450 (192.168.10.23)';
in some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
Until version 7.5, MQ client always flows the logged-in UserId and not the UserId specified by the application, to queue manager for authorization. You will need a security exit for this scenario.
This changed in MQ v8 where UserId specified by the application is flowed to the queue manager for authorization check. More details here: https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.sec.doc/q113260_.htm
You could use the MQS_REPORT_NOAUTH or MQSAUTHERRORS setting to get more info the authority failure.