Note that this is not my application, it is an application I am pentesting for a client. I usually ask questions like this on https://security.stackexchange.com/, however as this is more programming related I have asked on here.
Granted, RFC 4122 for UUIDs does not specify that type 4 UUIDs have to be generated by a Cryptographically Secure Pseudo Random Number Generator (CSPRNG). It simply says
Set all the other bits to randomly (or pseudo-randomly) chosen
values.
Although, some implementations of the algorithm, such as this one in Java, do use a CSPRNG.
I was trying to dig into whether Microsoft's implementation does or not. Mainly around how .NET or MSSQL Server generates them.
Checking the .NET source we can see this code:
Marshal.ThrowExceptionForHR(Win32Native.CoCreateGuid(out guid), new IntPtr(-1));
return guid;
Checking the CoCreateGuid docco, it states
The CoCreateGuid function calls the RPC function UuidCreate
All I can find out about this function is here. I seem to have reached the end of the rabbit hole.
Now, does anyone have any information on how UuidCreate generates its UUIDs?
I've seen many related posts:
How Random is System.Guid.NewGuid()? (Take two)
Is using a GUID a valid way to generate a random string of characters and numbers?
How securely unguessable are GUIDs?
how are GUIDs generated in SQL Server?
The first of which says:
A GUID doesn't make guarantees about randomness, it makes guarantees
around uniqueness. If you want randomness, use Random to generate a
string.
I agree with this except in my case for random, unpredictable numbers you'd of course use a CSPRNG instead of Random (e.g. RNGCryptoServiceProvider).
And the latter states (actually quoted from Wikipedia):
Cryptanalysis of the WinAPI GUID generator shows that, since the
sequence of V4 GUIDs is pseudo-random; given full knowledge of the
internal state, it is possible to predict previous and subsequent
values
Now, on the other side of the fence this post from Will Dean says
The last time I looked into this (a few years ago, probably XP SP2), I
stepped right down into the OS code to see what was actually
happening, and it was generating a random number with the secure
random number generator.
Of course, even if it was currently using a CSPRNG this would be implementation specific and subject to change at any point (e.g. any update to Windows). Unlikely, but theoretically possible.
My point is that there's no canonical reference for this, the above was to demonstrate that I've done my research and none of the above posts reference anything authoritative.
The reason is that I'm trying to decide whether a system that uses GUIDs for authentication tokens needs to be changed. From a pure design perspective, the answer is a definite yes, however from a practical point of view, if the Windows UuidCreate function does infact use a CSPRNG, then there is no immediate risk to the system. Can anyone shed any light on this?
I'm looking for any answers with a reputable source to back it up.
Although I'm still just some guy on the Internet, I have just repeated the exercise of stepping into UuidCreate, in a 32-bit app running on a 64-bit version of Windows 10.
Here's a bit of stack from part way through the process:
> 0018f670 7419b886 bcryptPrimitives!SymCryptAesExpandKeyInternal+0x7f
> 0018f884 7419b803 bcryptPrimitives!SymCryptRngAesGenerateSmall+0x68
> 0018f89c 7419ac08 bcryptPrimitives!SymCryptRngAesGenerate+0x3b
> 0018f8fc 7419aaae bcryptPrimitives!AesRNGState_generate+0x132
> 0018f92c 748346f1 bcryptPrimitives!ProcessPrng+0x4e
> 0018f93c 748346a1 RPCRT4!GenerateRandomNumber+0x11
> 0018f950 00dd127a RPCRT4!UuidCreate+0x11
It's pretty clear that it's using an AES-based RNG to generate the numbers. GUIDs generated by calling other people's GUID generation functions are still not suitable for use as unguessable auth tokens though, because that's not the purpose of the GUID generation function - you're merely exploiting a side effect.
Your "Unlikely, but theoretically possible." about changes in implementation between OS versions is rather given the lie by this statement in the docs for "UuidCreate":
If you do not need this level of security, your application can use the UuidCreateSequential function, which behaves exactly as the UuidCreate function does on all other versions of the operating system.
i.e. it used to be more predictable, now it's less predictable.
Related
I have two questions regarding implementation of Random class in .NET Framework 4.6 (code available here):
What is the rationale for setting Seed argument to 1 at the end of the constructor? It seems to be copy-pasted from Numerical Recipes in C (2nd Ed.) where it made some sense, but it doesn't have any in C#.
It is directly stated in the book (Numerical Recipes in C (2nd Ed.)) that inextp field is set to value 31 because:
The constant 31 is special; see Knuth.
However, in the .NET implementation this field is set to value 21. Why? The rest of a code seems to closely follow the code from book except for this detail.
Regarding the intexp issue, this is a bug, one which Microsoft has acknowledged and refused to fix due to backwards compatibility concerns.
Indeed, you have discovered a genuine problem with the Random implementation.
We have discussed it within the team and with some of our partners and concluded that we unfortunately cannot fix the problem right now. The reason is that some applications rely on the fact that when initialised with the same seed, the generator produces the same pseudo random sequence. Even if the change is for the better, it will break the applications that made this assumption once they have migrated to the “fixed” version.
For some more context:
A while back I fully analysed this implementation. I found a few differences.
A the first one (perfectly fine) is a different large value (MBIG). Numerical Recipies claims that Knuth makes it clear that any large value should work, so that is not an issue, and Microsoft reasonably chose to use the largest value of a 32 bit integer.
The second one was that constant, you mentioned. That one is a big deal. In the minimum it will substantially decrease period. There have been reports that the effects are actually worse than that.
But then comes one other particularly nasty difference. It is literally guarenteed to bias the output (since it does so directly), and will also likely affect the period of the RNG.
So what is this second issue? When .NET first came out, Microsoft did not realize that the RNG they coded was inclusive at both ends, and they documented it as exclusive at the maximum end. To fix this, the security team added a rather evil line of code: if (retVal == MBIG) retVal--;. This is very unfortunately as the correct fix would literally be only 4 added characters (plus whitespace).
The correct fix would have been to change MBIG to int.MaxValue-1, but switch Sample() to use MBIG+1 (i.e. to keep using int.MaxValue). That would guarantee the that Sample has the range [0.0, 1.0) without introducing any bias, and only changes the value of MBIG which Numerical Recipies said Knuth said is perfectly fine.
I have a very curious thing happening in my application with HashTables.
First off: Yes, I know that HashTables are not supposed to be ordered in the way the were inserted, or any other way (but the hashes of the keys). That doesn't answer my question. I do not need it ordered, I just want to know WHY it differs between two seemingly identical systems.
So, here it is. Left side is IIS ordering, right side is Visual Studio.
Why is it different? Considering that .NET should (?) use the same algorithm to store, and retreive, data from a HashTable, the ordering should be the same on both sides, shouldn't it?
If, as I learned it, the key of a HashTable is hashed, then this hash should be the same on both systems, resulting in the same order of hashes (keys) and therefore the same order of data in the hashtable.
Where am I wrong? What difference is there in the HashTable implementation between IIS and VS?
A few extra notes from comments:
Project is targeted at .NET 4.0
IIS uses .NET 4.0 for the Application Pool
I actually copied the compiled binaries from Visual Studios bin folder to the IIS folder, so they are exactly the same
My assumption is that IIS uses the same .NET implementation as Visual Studio. If not: Why? And what makes the hashing on IIS so different from the one in Visual Studio?
In order for items in the table to have the same ordering several conditions must be met:
Hash algorithm must be the same. This means not only the hash function, but also the way the table grows, shrinks, handles collision, and so on. This is probably the reason in your case (different algorithms).
Environment must be the same. Makes sense if one of the parameters of hash algorithm is something from the environment, like available memory. Some algorithms are pretty sophisticated, trying to avoid page misses or spicing the table for security purposes.
Data must be the same, and stored in hash table in the same order.
As #usr helped me find out, the reason for this behaviour lies in the GetHashCode() function, which is used as the base for the HashTables keys. Depending on the key, the order of Iteration over a HashTable will be different.
A hash function should normally return the same hash vor every input, but... this function returns different hashes, depending on a configuration parameter, namely <UseRandomizedStringHashAlgorithm> which will return a different hash, created by a mysterious function InternalMarvin32HashString().
It was introduced to prevent a Hash Flooding DOS attack vector.
This function however
[...] is imported from an external DLL, clr.dll to be precise.
from Martin Boßlet Dec 14th, 2012
So we cannot really know, what it does exactly without some major (maybe even illegal) refactoring going on.
For those in a hurry, this is NOT related to the controversial Dual_EC_DRBG planted inside NIST's SP800-90A.
About the two RNGs:
The Microsoft BCRYPT layer based one is over their C API. BCryptGenRandom is following NIST SP800-90A's CTR_DRBG (i.e. uses an approved block cipher, AES, to create random bits). However it is unclear if it uses a hardware random source as seed (or part of the seed)...
The Microsoft .NET RNGCryptoServiceProvider is C# based. Looking at the .NET source code (or here), I see it end up calling the C++ method CapiNative.GenerateRandomBytes(). There should have been a P/Invoke stub for the C#=>C++ transition but I couldn't find it anywhere in the framework source. So I have no idea how it's actually implemented.
Does anyone have additional information on these two random number generators? Do either/both use HW random seeds (either via diode noise in older intels or the controversial RDRAND in the latest intels).
PS: Not sure it this should be at Security, StackOverflow or Cryptography ...
The Microsoft .NET RNGCryptoServiceProvider is a C# based
Not exactly, the managed framework class is just a thin wrapper over the Crypto api built into Windows. All the System.Security.Cryptography classes whose name end with ServiceProvider are wrappers for the native API. Those whose name ends in Managed are implemented in pure managed code. Accordingly, the XxxServiceProvider classes use FIPS validated cryptogaphy, and the XxxManaged classes are not.
It is not exactly pinvoke, it uses a common mechanism to make direct calls in CLR code. The jitter consults a table with addresses of C++ functions and compiles the CALL machine code instruction directly. The mechanism is described in this answer. Having a look at the actual code isn't possible, it isn't included in the SSCLI20 distribution and was altered to use the QCall mechanism in .NET 4.
So the assertion is unprovable, but it is pretty likely that RNGCryptoServiceProvider and the algorithm provider you pass to BCryptGenRandom() use the same source for random numbers. Which in Windows is an unnamed exported function in advapi.dll, this answer gives an excellent summary of what it uses.
If this truly concerns you and you want a reliable source of information then don't take advice from a free Q+A web site for your security needs. Call Microsoft Support.
Microsoft RNGCryptoServiceProvider is mentioned in RFC 4086:
7.1.3. Windows CryptGenRandom
Microsoft's recommendation to users of the widely deployed Windows
operating system is generally to use the CryptGenRandom pseudo-random
number generation call with the CryptAPI cryptographic service
provider. This takes a handle to a cryptographic service provider
library, a pointer to a buffer by which the caller can provide entropy
and into which the generated pseudo-randomness is returned, and an
indication of how many octets of randomness are desired.
The Windows CryptAPI cryptographic service provider stores a seed
state variable with every user. When CryptGenRandom is called, this is
combined with any randomness provided in the call and with various
system and user data such as the process ID, thread ID, system clock,
system time, system counter, memory status, free disk clusters, and
hashed user environment block. This data is all fed to SHA-1, and the
output is used to seed an RC4 key stream. That key stream is used to
produce the pseudo-random data requested and to update the user's seed
state variable.
Users of Windows ".NET" will probably find it easier to use the
RNGCryptoServiceProvider.GetBytes method interface.
I'm current stuck in the random generator. The requirement specification shows a sample like this:
Random rand = new Random(3412);
The rand result is not directly given out, but used for other performance.
I'd written the same code as above to generate a random number by a seed 3412.
however, the result of the rest performance is totally different with sample.
The generating result is 518435373, I used the same code tried on the online c# compiler, but getting different result of generation which is 11688046, the rest performance result was also different with the sample.
So I'm just wondering is that supposed to be different in different machines?
BTW, could anyone provide the result from your machine just see if it's same with me.
I would expect any one implementation to give the same sequence for the same seed, but there may well be different implementations involved. For example, an "online C# compiler" may well end up using Mono, which I'd expect to have a different implementation to the one in .NET.
I don't know whether the implementations have changed between versions of .NET, but again, that seems entirely possible.
The documentation for the Random(int) constructor states:
Providing an identical seed value to different Random objects causes each instance to produce identical sequences of random numbers.
... but it doesn't specify the implications of different versions etc. Heck, it doesn't even state whether the x86 and x64 versions will give the same results. I'd expect the same results within any one specific CLR instance (i.e. one process, and not two CLRs running side-by-side, either*.
If you need anything more stable, I'd start off with a specified algorithm - I bet there are implementations of the Mersenne Twister etc available.
It isn't specified as making such a promise, so you should assume that it does not.
A good rule with any specification, is not to make promises that aren't required for reasonable use, so you are freer to improve things later on.
Indeed, Random's documentation says:
The current implementation of the Random class is based on Donald E. Knuth's subtractive random number generator algorithm.
Note the phrase "current implementation", implying it may change in the future. This very strongly suggests that not only is there no promise to be consistent between versions, but there is no intention to either.
If a spec requires consistent pseudo-random numbers, then it must specify the algorithm as well as the seed value. Indeed, even if Random was specified as making such a promise, what if you need a non-.NET implementation of all or part of your specification - or something that interoperates with it - in the future?
This is probably due to different framework versions. Have a look at this
The online provider you tried might use the Mono implementation of the CLR, which is different of the one Microsoft provides. So probably their Random class implementation is a bit different.
How are random numbers generated.? How do languages such as java etc generate random numbers, especially how it is done for GUIDs.? i found that algorithms like Pseudorandomnumber generator uses initial values.
But i need to create a random number program, in which a number once occurred should never repeats even if the system is restarted etc. I thought that i need to store the values anywhere so that i can check if the number repeats or not, but it will be too complex when the list goes beyond limits.?
First: If the number is guaranteed to never repeat, it's not very random.
Second: There are lots of PRNG algorithms.
UPDATE:
Third: There's an IETF RFC for UUIDs (what MS calls GUIDs), but you should recognize that (U|G)UIDs are not cryptographically secure, if that is a concern for you.
UPDATE 2:
If you want to actually use something like this in production code (not just for your own edification) please use a pre-existing library. This is the sort of code that is almost guaranteed to have subtle bugs in it if you've never done it before (or even if you have).
UPDATE 3:
Here's the docs for .NET's GUID
There are a lot of ways you could generate random numbers. It's usually done with a system/library call which uses a pseudo-number generator with a seed as you've already described.
But, there are other ways of getting random numbers which involve specialized hardware to get TRUE random numbers. I know of some poker sites that use this kind of hardware. It's very interesting to read how they do it.
Most random number generators have a way to "randomly" reïnitialize the seed value. (Sometimes called randomize).
If that's not possible, you can also use the system clock to initialize the seed.
You could use this code sample:
http://xkcd.com/221/
Or, you can use this book:
http://www.amazon.com/Million-Random-Digits-Normal-Deviates/dp/0833030477
But seriously, don't implement it yourself, use an existing library. You can't be the first person to do this.
Specifically regarding Java:
java.util.Random uses a linear congruential generator, which is not very good
java.util.UUID#randomUUID() uses java.security.SecureRandom, an interface for a variety of cryptographically secure RNGs - the default is based on SHA-1, I believe.
UUIDs/GUIDs are not necessarily random
It's easy to find implementations of RNGs on the net that are much better than java.util.Random, such as the Mersenne Twister or multiply-with-carry
I understand that you are seeking a way to generate random number using C#. If yes, RNGCryptoServiceProvider is what you are looking for.
[EDIT]
If you generate a fairly long number of bytes using RNGCryptoServiceProvider, it is likely to be unique but there is no gurantee. In theory, true random numbers doesnt mean to be unique. You roll a dice 2 times and you may get head both the times but they are still random. TRUE RANDOM!
I guess to apply the check of being unique, you just have to roll out your own mechanism of keeping history of previously generated numbers.