Is it possible to generate short GUID like in YouTube (N7Et6c9nL9w)?
How can it be done? I want to use it in web app.
You could use Base64:
string base64Guid = Convert.ToBase64String(Guid.NewGuid().ToByteArray());
That generates a string like E1HKfn68Pkms5zsZsvKONw==. Since a GUID is always 128 bits, you can omit the == that you know will always be present at the end and that will give you a 22 character string. This isn't as short as YouTube though.
URL Friendly Solution
As mentioned in the accepted answer, base64 is a good solution but it can cause issues if you want to use the GUID in a URL. This is because + and / are valid base64 characters, but have special meaning in URLs.
Luckily, there are unused characters in base64 that are URL friendly. Here is a more complete answer:
public string ToShortString(Guid guid)
{
var base64Guid = Convert.ToBase64String(guid.ToByteArray());
// Replace URL unfriendly characters
base64Guid = base64Guid.Replace('+', '-').Replace('/', '_');
// Remove the trailing ==
return base64Guid.Substring(0, base64Guid.Length - 2);
}
public Guid FromShortString(string str)
{
str = str.Replace('_', '/').Replace('-', '+');
var byteArray = Convert.FromBase64String(str + "==");
return new Guid(byteArray);
}
Usage:
Guid guid = Guid.NewGuid();
string shortStr = ToShortString(guid);
// shortStr will look something like 2LP8GcHr-EC4D__QTizUWw
Guid guid2 = FromShortString(shortStr);
Assert.AreEqual(guid, guid2);
EDIT:
Can we do better? (Theoretical limit)
The above yields a 22 character, URL friendly GUID.
This is because a GUID uses 128 bits, so representing it in base64 requires
characters, which is 21.33, which rounds up to 22.
There are actually 66 URL friendly characters (we aren't using . and ~). So theoretically, we could use base66 to get
characters, which is 21.17, which also rounds up to 22.
So this is optimal for a full, valid GUID.
However, GUID uses 6 bits to indicate the version and variant, which in our case are constant. So we technically only need 122 bits, which in both bases rounds to 21 ( = 20.33). So with more manipulation, we could remove another character. This requires wrangling the bits out however, so I leave this as an exercise to the reader.
How does youtube do it?
YouTube IDs use 11 characters. How do they do it?
A GUID uses 122 bits, which guarantees collisions are virtually impossible. This means you can generate a random GUID and be certain it is unique without checking. However, we don't need so many bits for just a regular ID.
We could use a smaller ID. If we use 66 bits or less, we have a higher risk of collision, but can represent this ID with 11 characters (even in base64). One could either accept the risk of collision, or test for a collision and regenerate.
With 122 bits (regular GUID), you would have to generate ~ GUIDs to have a 1% chance of collision.
With 66 bits, you would have to generate ~ or 1 billion IDs to have a 1% chance of collision. That is not that many IDs.
My guess is youtube uses 64 bits (which is more memory friendly than 66 bits), and checks for collisions to regenerate the ID if necessary.
If you want to abandon GUIDs in favor of smaller IDs, here is code for that:
class IdFactory
{
private Random random = new Random();
public int CharacterCount { get; }
public IdFactory(int characterCount)
{
CharacterCount = characterCount;
}
public string Generate()
{
// bitCount = characterCount * log (targetBase) / log(2)
var bitCount = 6 * CharacterCount;
var byteCount = (int)Math.Ceiling(bitCount / 8f);
byte[] buffer = new byte[byteCount];
random.NextBytes(buffer);
string guid = Convert.ToBase64String(buffer);
// Replace URL unfriendly characters
guid = guid.Replace('+', '-').Replace('/', '_');
// Trim characters to fit the count
return guid.Substring(0, CharacterCount);
}
}
Usage:
var factory = new IdFactory(characterCount: 11);
string guid = factory.Generate();
// guid will look like Mh3darwiZhp
This uses 64 characters which is not optimal, but requires much less code (since we can reuse Convert.ToBase64String).
You should be a lot more careful of collisions if you use this.
9 chars is not a GUID. Given that, you could use the hexadecimal representation of an int, which gives you a 8 char string.
You can use an id you might already have. Also you can use .GetHashCode against different simple types and there you have a different int. You can also xor different fields. And if you are into it, you might even use a Random number - hey, you have well above 2.000.000.000+ possible values if you stick to the positives ;)
It's not a GUID but rather an auto-incremented unique alphanumeric string
Please see the following code where I am trying to do the same, It uses the TotalMilliseconds from EPOCH and a valid set of characters to generate a unique string that is incremented with each passing milliseconds.
The one other way is to use numeric counters but that is expensive to maintain and will create a series where you can + or - values to guess the previous or the next unique string in the system and we don't what that to happen.
Do remember:
This will not be globally unique but unique to the instance where it's defined
It uses Thread.Sleep() to handle multithreading issue
public string YoutubeLikeId()
{
Thread.Sleep(1);//make everything unique while looping
long ticks = (long)(DateTime.UtcNow
.Subtract(new DateTime(1970, 1, 1,0,0,0,0))).TotalMilliseconds;//EPOCH
char[] baseChars = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
.ToCharArray();
int i = 32;
char[] buffer = new char[i];
int targetBase= baseChars.Length;
do{
buffer[--i] = baseChars[ticks % targetBase];
ticks = ticks / targetBase;
}
while (ticks > 0);
char[] result = new char[32 - i];
Array.Copy(buffer, i, result, 0, 32 - i);
return new string(result);
}
The output will come something like
XOTgBsu
XOTgBtB
XOTgBtR
XOTgBtg
XOTgBtw
XOTgBuE
Update: The same can be achieved from Guid as
var guid = Guid.NewGuid();
guid.ToString("N");
guid.ToString("N").Substring(0,8);
guid.ToString("N").Substring(8,4);
guid.ToString("N").Substring(12,4);
guid.ToString("N").Substring(16,4);
guid.ToString("N").Substring(20,12);
For a Guid ecd65132-ab5a-4587-87b8-b875e2fe0f35 it will break it down in chunks as ecd65132 ,ab5a , 4587,87b8,b875e2fe0f35
but it's not guarantee it to be unique always.
Update 2: There is also a project called ShortGuid to get a url friendly GUID it can be converted from/to a regular Guid
When I went under the hood I found it works by encoding the Guid to Base64 as the code below:
public static string Encode(Guid guid)
{
string encoded = Convert.ToBase64String(guid.ToByteArray());
encoded = encoded
.Replace("/", "_")
.Replace("+", "-");
return encoded.Substring(0, 22);
}
The good thing about it it can be decoded again to get the Guid back with
public static Guid Decode(string value)
{
// avoid parsing larger strings/blobs
if (value.Length != 22)
{
throw new ArgumentException("A ShortGuid must be exactly 22 characters long. Receive a character string.");
}
string base64 = value
.Replace("_", "/")
.Replace("-", "+") + "==";
byte[] blob = Convert.FromBase64String(base64);
var guid = new Guid(blob);
var sanityCheck = Encode(guid);
if (sanityCheck != value)
{
throw new FormatException(
#"Invalid strict ShortGuid encoded string. The string '{value}' is valid URL-safe Base64, " +
#"but failed a round-trip test expecting '{sanityCheck}'."
);
}
return guid;
}
So a Guid 4039124b-6153-4721-84dc-f56f5b057ac2 will be encoded as SxI5QFNhIUeE3PVvWwV6wg and the Output will look something like.
ANf-MxRHHky2TptaXBxcwA
zpjp-stmVE6ZCbOjbeyzew
jk7P-XYFokmqgGguk_530A
81t6YZtkikGfLglibYkDhQ
qiM2GmqCK0e8wQvOSn-zLA
As others have mentioned, YouTube's VideoId is not technically a GUID since it's not inherently unique.
As per Wikipedia:
The total number of unique keys is 2128 or 3.4×1038. This number is so
large that the probability of the same number being generated randomly
twice is negligible.
The uniqueness YouTube's VideoId is maintained by their generator algorithm.
You can either write your own algorithm, or you can use some sort of random string generator and utilize the UNIQUE CONSTRAINT constraint in SQL to enforce its uniqueness.
First, create a UNIQUE CONSTRAINT in your database:
ALTER TABLE MyTable
ADD CONSTRAINT UniqueUrlId
UNIQUE (UrlId);
Then, for example, generate a random string (from philipproplesch's answer):
string shortUrl = System.Web.Security.Membership.GeneratePassword(11, 0);
If the generated UrlId is sufficiently random and sufficiently long you should rarely encounter the exception that is thrown when SQL encounters a duplicate UrlId. In such an event, you can easily handle the exception in your web app.
Technically it's not a Guid. Youtube has a simple randomized string generator that you can probably whip up in a few minutes using an array of allowed characters and a random number generator.
It might be not the best solution, but you can do something like that:
string shortUrl = System.Web.Security.Membership.GeneratePassword(11, 0);
This id is probably not globally unique. GUID's should be globally unique as they include elements which should not occur elsewhere (the MAC address of the machine generating the ID, the time the ID was generated, etc.)
If what you need is an ID that is unique within your application, use a number fountain - perhaps encoding the value as a hexadecimal number. Every time you need an id, grab it from the number fountain.
If you have multiple servers allocating id's, you could grab a range of numbers (a few tens or thousands depending on how quickly you're allocating ids) and that should do the job. an 8 digit hex number will give you 4 billion ids - but your first id's will be a lot shorter.
Maybe using NanoId will save you from a lot of headaches:
https://github.com/codeyu/nanoid-net
You can do something like:
var id = Nanoid.Generate('1234567890abcdef', 10) //=> "4f90d13a42"
And you can check the collision probability here:
https://alex7kom.github.io/nano-nanoid-cc/
Related
I need to generate a 4 bit string randomly and set to a property via Linq.
Right now is hardcoded:
// TODO: hardcode bit string
employees = employees.Select(x => { x.Options = "0101"; return x; }).ToList();
I need Options to be random so I can get all 4 bit possible values: "0001","0010", "0011" and so on.
I was thinking on having a Random 0 to 1 variable and generate the value 4 times and concatenate the string.
Any clue on how to implement this?
If speed is critical, such as this operating being called in a loop, it would be more efficient to use a 16 element string array of all possible values and select the element randomly.
var rand = new Random();
Convert.ToString(rand.Next(16), 2).PadLeft(4, '0')
Explanation:
The first line creates a random object, nothing too hard to understand here... The second line first generates a random number between 0 - 15 (rand.Next(16)). Then it puts the random number into Convert.ToString. The method converts to number to base 2 (because the second parameter is 2). However, this still is not enough because if the random number can be represented by 3 bits or fewer, the returned string will not have the leading 0s. That's why I used PadLeft to add them in.
You can use this
Convert.ToString(int, 2);
This will convert the int to base 2 string.
Where int is a random number up to 16 not included.
I need to generate a unique id which consists of numbers.
Is the following result string uniqueId as unique as the result of guid.ToString()?
Guid guid = Guid.NewGuid();
byte[] guidBytes = guid.ToByteArray();
// Is the result (uniqueId) as unique as guid.ToString()?
string uniqueId = string.Join(string.Empty, guidBytes);
You need seperator between byte values or fill with zero. Otherwise there is intersection.
Example: 3,5,6,7,123 => 003005006007123
Yes, there is a 1:1 mapping of byte arrays to Guids. No information is lost during the transformation so you still retain the same uniqueness as using the normal string representation of a Guid.
A Guid really is just a 16 byte number, it does not matter if you show it as {3F2504E0-4F89-41D3-9A0C-0305E82C3301}, 4AQlP4lP00GaDAMF6CwzAQ== or as 224004037063137079211065154012003005232044051001, it all still represents the same number.
EDIT: Oops, as mkysoft ponted out, you do have to deal with leading zeros. Padding the numbers to 3 digits solves the issue
var guid = new Guid("{3F2504E0-4F89-41D3-9A0C-0305E82C3301}");
Console.WriteLine(string.Join(string.Empty, guid.ToByteArray().Select(x=>x.ToString("000"))));
UPDATE: Actually I just thought of a better solution, a Guid is a 128-bit number, by using 2 64-bit numbers and padding the number's 0's out on the 2nd half you will get a shorter, but still unique number.
var guid = new Guid("{3F2504E0-4F89-41D3-9A0C-0305E82C3301}");
var guidBytes = guid.ToByteArray();
Console.WriteLine("{0}{1}", BitConverter.ToUInt64(guidBytes, 0), BitConverter.ToUInt64(guidBytes,8).ToString().PadLeft(20, '0'));
This will output a unique integer number that is between 21 and 40 digits long, {3F2504E0-4F89-41D3-9A0C-0305E82C3301} becomes 474322228343976880000086462192878292122,
Or you could use BigInteger.ToString() to handle making big numbers into strings (since it's really good at that)
var p = Guid.NewGuid().ToByteArray();
Array.Resize(ref p, p.Length + 1);
Console.WriteLine(new BigInteger(p));
The resize is only if you require positive numbers (otherwise there is a 50% chance you get a negative number). You could also use System.Security.Cryptography.RandomNumberGenerator.GetBytes to have a larger or smaller set of values (depending on how big you want your identifiers to be)
I have a string made up of 2 parts (see code)
I want to know the UnknownDeterministicFunction which returns a Long, which can deterministically produce the same long for a given string.
private void MyProgram()
{
string resultStr = "XXX"+"12345678";
//1st part is a string of characters (the "XXX")
//2nd part is a string of numbers (the "12345678")
long resultLng = UnknownDeterministicFunction(myString);
}
private long UnknownDeterministicFunction(string inputStr)
{
// ???
}
Is this possible in C#?
First of all
there are 8 ** (2 ** 30) different strings (which up to 2 GB long)
there are 2 ** 64 differrent long (which are 64-bit integers)
So you can't guarantee long be unique (good old Pigeonhole principle). If you are ready for possible, though improbable collisions (i.e. different strings can well return the same long) you may want to implement hash functions, e.g.
hash function for string
or
Good Hash Function for Strings
usually, hash function returns Int32; in that case just combine two int into one long
int hash1 = GetHashOneAlgorithm(myString);
int hash2 = GetHashAnotherAlgorithm(myString);
long result = ((long) hash1 << 32) | hash2;
OK the answer is simple.
private long UnknownDeterministicFunction(string inputStr)
{
//not taking care of null...
return (long)inputStr.GetHashCode()
}
I have this name:
string name = "Centos 64 bit";
I want to generate a 168-bit (or whatever is feasible) uid from this name and to be able to get the name from this id vice versa
.
I tried this one GetHashCode() without success.
Result would be something like:
Centos 64 bit (=) 91C47A57-E605-4902-894B-74E791F37C1F
One solution I would recommend is to use a hash function and something like a dictionary. So, get a hash - say SHA256 - of your input string and truncate it to 168 bytes.
Now, to go back from a uid to original string, you would need to have a dictionary which stores pairs like (input_string, string_uid). input_string is original string and string_uid is the uid generated for input_string using method from first paragraph.
Using this dictionary you can easily go back to original input string using string_uid.
This is one way - of course in case, you are allowed to store mappings between string and uid.
The hash normally gives you result as byte array. Converting this byte array to string is a separate step.
For example if you have 10 bytes representing integers in the range [0, 255], converting it to string if you encode the byte array as hex string, will take 20 bytes.
So the next question is do you want the length of the uid as string to be 21 bytes?
Because this will mean the hash output must be somewhere like 10 bytes, this will poorly reflect on collision resistance of the output.
what you want is not achievable. You need to store a lookup table of hash to name. Since you dont give more details of yr system it hard to say if that has to be persistent or in memory. If in memory just use a dictionary of string->string
Here you go sir:
public byte[] GetUID(string name)
{
var bytes = Encoding.ASCII.GetBytes(name);
if (bytes.Length > 21)
throw new ArgumentException("Value is too long to be used as an ID");
var uid = new byte[21];
Buffer.BlockCopy(bytes, 0, uid, 0, bytes.Length);
return bytes;
}
public string GetName(byte[] UID)
{
int length = UID.Length;
for (int i = 0; i < UID.Length; i++)
{
if (UID[i] == 0)
{
length = i;
break;
}
}
return Encoding.ASCII.GetString(UID, 0, length);
}
Caveats: it works for strings up to 21 characters in length that only use ASCII characters (no Unicode support) and it doesn't encrypt the string in any way, but I believe it meets your requirements.
I am attempting to wrap my brain around generating a 6 digit/character non case sensitive expiring one-time password.
My source is https://www.rfc-editor.org/rfc/rfc4226#section-5
First the definition of the parameters
C 8-byte counter value, the moving factor. This counter
MUST be synchronized between the HOTP generator (client)
and the HOTP validator (server).
K shared secret between client and server; each HOTP
generator has a different and unique secret K.
T throttling parameter: the server will refuse connections
from a user after T unsuccessful authentication attempts.
Then we have the algorithm to generate the HOTP
As the output of the HMAC-SHA-1 calculation is 160 bits, we must
truncate this value to something that can be easily entered by a
user.
HOTP(K,C) = Truncate(HMAC-SHA-1(K,C))
Then, we have Truncate defined as
String = String[0]...String[19]
Let OffsetBits be the low-order 4 bits of String[19]
Offset = StToNum(OffsetBits) // 0 <= OffSet <= 15
Let P = String[OffSet]...String[OffSet+3]
Return the Last 31 bits of P
And then an example is offered for a 6 digit HOTP
The following code example describes the extraction of a dynamic
binary code given that hmac_result is a byte array with the HMAC-
SHA-1 result:
int offset = hmac_result[19] & 0xf ;
int bin_code = (hmac_result[offset] & 0x7f) << 24
| (hmac_result[offset+1] & 0xff) << 16
| (hmac_result[offset+2] & 0xff) << 8
| (hmac_result[offset+3] & 0xff) ;
I am rather at a loss in attempting to convert this into useful C# code for generating one time passwords. I already have code for creating an expiring HMAC as follows:
byte[] hashBytes = alg.ComputeHash(Encoding.UTF8.GetBytes(input));
byte[] result = new byte[8 + hashBytes.Length];
hashBytes.CopyTo(result, 8);
BitConverter.GetBytes(expireDate.Ticks).CopyTo(result, 0);
I'm just not sure how to go from that, to 6 digits as proposed in the above algorithms.
You have two issues here:
If you are generating alpha-numeric, you are not conforming to the RFC - at this point, you can simply take any N bytes and turn them to a hex string and get alpha-numeric. Or, convert them to base 36 if you want a-z and 0-9. Section 5.4 of the RFC is giving you the standard HOTP calc for a set Digit parameter (notice that Digit is a parameter along with C, K, and T). If you are choosing to ignore this section, then you don't need to convert the code - just use what you want.
Your "result" byte array has the expiration time simply stuffed in the first 8 bytes after hashing. If your truncation to 6-digit alphanumeric does not collect these along with parts of the hash, it may as well not be calculated at all. It is also very easy to "fake" or replay - hash the secret once, then put whatever ticks you want in front of it - not really a one time password. Note that parameter C in the RFC is meant to fulfill the expiring window and should be added to the input prior to computing the hash code.
For anyone interested, I did figure out a way to build expiration into my one time password. The approach is to use the created time down to the minute (ignoring seconds, milliseconds, etc). Once you have that value, use the ticks of the DateTime as your counter, or variable C.
otpLifespan is my HOTP lifespan in minutes.
DateTime current = new DateTime(DateTime.Now.Year, DateTime.Now.Month,
DateTime.Now.Day, DateTime.Now.Hour, DateTime.Now.Minute, 0);
for (int x = 0; x <= otpLifespan; x++)
{
var result = NumericHOTP.Validate(hotp, key,
current.AddMinutes(-1 * x).Ticks);
//return valid state if validation succeeded
//return invalid state if the passed in value is invalid
// (length, non-numeric, checksum invalid)
}
//return expired state
My expiring HOTP extends from my numeric HOTP which has a static validation method that checks the length, ensures it is numeric, validates the checksum if it is used, and finally compares the hotp passed in with a generated one.
The only downside to this is that each time you validate an expiring hotp, your worse case scenario is to check n + 1 HOTP values where n is the lifespan in minutes.
The java code example in the document outlining RFC 4226 was a very straightforward move into C#. The only piece I really had to put any effort into rewriting was the hashing method.
private static byte[] HashHMACSHA1(byte[] keyBytes, byte[] text)
{
HMAC alg = new HMACSHA1(keyBytes);
return alg.ComputeHash(text);
}
I hope this helps anyone else attempting to generate one time passwords.
This snippet should do what you are asking for:
public class UniqueId
{
public static string GetUniqueKey()
{
int maxSize = 6; // whatever length you want
char[] chars = new char[62];
string a;
a = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
char[] chars = new char[a.Length];
chars = a.ToCharArray();
int size = maxSize;
byte[] data = new byte[1];
RNGCryptoServiceProvider crypto = new RNGCryptoServiceProvider();
crypto.GetNonZeroBytes(data);
size = maxSize;
data = new byte[size];
crypto.GetNonZeroBytes(data);
StringBuilder result = new StringBuilder(size);
foreach (byte b in data)
{ result.Append(chars[b % (chars.Length - 1)]); }
return result.ToString();
}
}