I'm reading a lossy bit stream and I need a way to recover as much usable data as possible. There can be 1's in place of 0's and 0's in palce of 1's, but accuracy is probably over 80%.
A bonus would be if the algorithm could compensate for missing/too many bits as well.
The source I'm reading from is analogue with noise (microphone via FFT), and the read timing could vary depending on computer speed.
I remember reading about algorithms used in CD-ROM's doing this in 3? layers, so I'm guessing using several layers is a good option. I don't remember the details though, so if anyone can share some ideas that would be great! :)
Edit: Added sample data
Best case data:
in: 0000010101000010110100101101100111000000100100101101100111000000100100001100000010000101110101001101100111000101110000001001111011001100110000001001100111011110110101011100111011000100110000001000010111
out: 0010101000010110100101101100111000000100100101101100111000000100100001100000010000101110101001101100111000101110000001001111011001100110000001001100111011110110101011100111011000100110000001000010111011
Bade case (timing is off, samples are missing):
out: 00101010000101101001011011001110000001001001011011001110000001001000011000000100001011101010011011001
in: 00111101001011111110010010111111011110000010010000111000011101001101111110000110111011110111111111101
Edit2: I am able to controll the data being sent. Currently attempting to implement simple XOR checking (though it won't be enough).
If I understand you correctly, you have two needs:
Modulate a signal into sound and then demodulate it.
Apply error correction since the channel is unreliable.
Modulation and demodulation is a wellknown application, with several ways to modulate the information.
Number two, error correction also is wellknown and have several possibilities. Which one is applicable depends on the error rate and whether you have duplex operation so that you can request resends. If you have decent quality and can request resends an approach like the one TCP is using is worth exploring.
Otherwise you will have to get down to error detection and error correction algorithms, like the one used on CDROMs.
Edit after the comment
Having the modulation/demodulation done and no resend possibilities narrows the problem. If you are having timing issues, I would still recommend that you read up on existing (de)modulation methods, as there are ways to automatically resynchronize with the sender and increase signal-to-noise ratio.
Down to the core problem; error correction you will have to add parity bits to your output stream in order to be able to detect the errors. Starting with the forward error correction article #Justin suggests, an scheme that looks quite simple, but still powerful is the Hamming(7,4) scheme.
You need to use forward error correction. An XOR parity check will only detect when an error occurs. A simple error correction algorithm would be to send each chunk of data multiple times (at least 3) and make a majority decision.
The choice of algorithm depends on several factors:
Channel utilization (if you have lots of free time, you don't need an efficient coding)
Error types: are the bad bits randomly spaced or do they usually occur in a row
Processing time: code complexity is limited if data transmission needs to be fast
There are lot of possibilities, see : http://en.wikipedia.org/wiki/Error_detection_and_correction
This can help you with changed bits, but may be unsuitable to check whenever you have all the bits.
In the end, it will probably take much more than few lines of simple code.
Related
I decided to create a feedforward Neural Network without using any libraries. I am fairly new to the subject and completely self-trained.
My Neural Network uses backpropagation to set the weights and the activation function between all layers (input-hidden1-output) is a Sigmoid function.
Let's say that I try to solve a basic problem like the XOr logic gate problem with my NN. Whenever i use the complete training set (all the possible combinations of 1s and 0s) my NN cannot set the weights in such a way that it could produce the desired output. Seemingly it always stops at the middle. (output is ~0.5 in all cases)
On the other hand, when I only iterate one type of input (Let's say 0 and 1) it quickly learns.
Is there a problem in my cost function, number of nodes, hidden layers or what? I would appreciate some guiding words!
XOR problem is not linearly separable and makes single layer perceptron unfit. However, in your network addition of hidden layer makes the network to capture non-linear features, which makes it fine.
The most plausible reason for the poor performance of the network would be due to tortuous initial phase to learn the problem. So increasing the iterations would solve the problem.
And one more possible thing to try is by the smooth nonlinearity of XOR, so the role of bias is crucial as the translation parameter and as important as weights (which you did not mention)
XOR can't be solved with one hidden layer. Because you can't separate your labels (0 and 1) with just one line. You can separate them with two lines and then use AND gate (another hidden layer) to find their common area.
See this post for clarification: https://medium.com/#jayeshbahire/the-xor-problem-in-neural-networks-50006411840b
I am new to neural network and I'm working with Encog3. I have created feedforward neural network which can be train and tested.
Problem is that I'm not sure how to prevent overfitting. I know I have to split data into training, testing and evaluation set, but I'm not sure where and when to use evaluation set.
Currently, I split all data into training and testing set (50%, 50%), train network on one part, test on another. Accuracy is 85%.
I tried with CrossValidationKFold but in that case accuracy is only 12% and I don't understand why.
My question is, how can I use evaluation set to avoid overfitting?
I am confused about evaluation set and any help would be appreciated.
It is general practice to have split 60x20x20 ( another common usage is 80x10x10 )%. 60 percent for training. 20 percent for validating and another 20 percent for validating previous two. Why three parts? Because it will give you better picture how ML works on data which it never seen before. Another part of analysis could include representative learning set. If you have in your training data set values which do not have any representation in validating then most probably you'll get mistakes in your ML. It's the same way how your brain works. If you learn some rules, and then suddenly got some task which is actually exception from rules you'll know, most probably you'll give wrong answer. In case if you have problems with learning, you can do the following: increase dataset, increase number of inputs ( via some non linear transformations with your inputs ). Maybe you'll also need to apply some anomaly detection algorithm. Also you can consider to apply some different normalization techniques.
As a quick aside, you keep referring to the data as an “evaluation” set. Whilst it is being used in that capacity, the general term is “validation” set, which might allow you better success when googling it.
You’re in something of a chicken-and-egg situation with your current setup. Basically, the sole purpose of the validation set is to prevent overfitting – making no use of a validation set will (for all intents and purposes) result in overfitting. By contrast, the testing set has no part to play in preventing overfitting, it’s just another way of seeing, at the end, whether overfitting might have occurred.
Perhaps it would be easier to take this away from any maths or code (which I assume you have seen before) and imagine this as questions the model keeps asking itself. On every training epoch, the model is desperately trying to reduce its residual error against the training set and, being so highly non-linear, there’s a good chance in structured problems that it will reduce this error to almost nothingness if you allow it to keep running. But that’s not what you’re after. You’re after a model that is a good approximator for all three datasets. So, we make it do the following on every epoch:
“Has my new move reduced the error on the training set?” If yes: “Awesome, I’ll keep going in that direction.”
“Has my new move reduced the error on the validation set?” If yes: “Awesome, I’ll keep going in that direction.”
Eventually, you’ll come to:
“Has my new move reduced the error on the training set?” Yes: “Awesome, I’ll keep going in that direction.”
“Has my new move reduced the error on the validation set?” No, it’s increased: “Perhaps I’ve gone too far.”
If the validation error continues to rise, then you’ve identified the point at which your model is moving away from being a good approximator and moving towards being over-fit to the training set. It’s time to stop. Then you want to apply that final model to your test data and see whether the model is still a good approximator to that data too. And if it is, you have your model.
A final word, it’s good to see you’re doing some form of cross validation because I’ve seen that kind of safeguard missed so many times in the past.
I have two computers communicating over a serial modem.
I would like to have a reliability protocol on that line.
I have been looking into PPP, SLIP and RATP. Not all of them are the best fit, and I do not want to write all that code, especially if there is a good code base for that online.
Is there a library or code project in C# that can be used for that purpose?
If not what protocol should you recommend to implement?
The connection speed is 9600, but the amount of data sent is not very big, and speed is not a big issue. Simplicity and ease of development is!
I always just add a CRC to each message, but my higher level protocols are self-synchronizing and loss tolerant by virtue of unsolicited state reports -- if a command is lost and the state doesn't change, that becomes apparent on the next state report. Depending on whether your requirement is to detect or correct errors, and whether you can tolerate extra delays for retransmissions, you might need to look into a forward error correcting code.
Concepts of interest include message sequence numbers, acknowledgement, go-back-N vs selective retransmit, and minimum distance between codes (Hamming distance).
I definitely suggest you look at the design of TCP. The basics are really pretty minimal for guaranteed in-order delivery.
I am using simulated annealing to solve a cryptanalysis problem and I've hit a brick wall. I cannot for the life of me get my probability function to operate correctly, it either takes a worse solution too often (so I bounce around a score of 0.03 and 0.2) or it doesn't take it often enough (so I get stuck at 0.35). I've looked around the internet but I only come across examples where the problems involve finding the MINIMUM value....my problem needs to find the MAXIMUM value, worst score is 0, best is 1.
I need advice on Temperature and what probability function I should use.
The Simulated Annealing article on Wikipedia provides some general guidance on how SA temperatures should be initialized and decreased. Efficient selection of these parameters is normally very problem specific and may need to be identified through tedious trial-and-error.
Normally, optimization algorithms search for the minimum of the objective function. If you want to use such an algorithm as-is on your maximization problem, ask the optimizer to minimize the negation of your objective function. For example, let's say that the objective function for which you want to find the maximum is f(x)=score. You should then request the optimizer to minimize -f(x), i.e. -score (or, as you indicate in the comment above, 1-score).
There are lots of simulated annealing and other global optimization algorithms available online, see for example this list on the Decision Tree for Optimization Software. Unfortunately these codes are normally not written in C#, but if the codes are written in Fortran or C it is normally fairly easy to interface with these codes via P/Invoke.
If you do not require that the optimizer necessarily find the global optimum, there are also some derivative-free optimizers listed here. At least one of these codes is available in a C# version, namely BOBYQA (in fact, this algorithm has been adapted to C# by me :-).
Hey everyone.
This place is like a goldmine of knowledge and it's helping me so much! My next query is:
I have byte data being sent to my c# socket server. I am converting it to an ascii string, then splitting the data based on a common character (like the bar | character) and using the data. Typically the first piece of data is a command as a 4 digit number. I can imagine this not being very efficient! What would be the best way to process the data is an receiving, efficiently?
Related, how I be trapping and processing commands? Multiple if statements or a large case/switch statement. I really need speed and efficiency.
Typically the first piece of data is a command as a 4 digit number. I can imagine this not being very efficient! What would be the best way to process the data is an receiving, efficiently?
No, converting a number to/from a string is not efficient. But the question is: Do it really matter? It sounds to me like you are trying to do premature optimizations. Do not do that. Your goal should be to write code that is easy to read and maintain. Do not do optimizations until someone actually complains about the performance.
Related, how I be trapping and processing commands? Multiple if statements or a large case/switch statement. I really need speed and efficiency.
Again. Determine that the command processing really is the bottle neck in your application.
The whole processing really depends on what you do with the incoming messages. You provide way to little information to give a proper answer. Create a new question (since two questions in one is not really allowed). Add code which shows your current handling and describe what you do not like about it.
If you really need the performance I guess you shouldn't use a string representation for your command but work directly on the bytes. Four numbers in string format are 32 of 64 bits (depending on which charset you are using) in size, whilst a single byte is sufficient to store a four digit number. Using a lot of branches (which if-statements are) also effects your performance.
My suggestion is that you reserve a fixed size prefix in your message for the command. You then use these bytes to lookup in O(1) in a table which command you should execute, this table can be filled with object that have a method execute. So you can do something table[command].execute().
That being said, I don't think the performance-gain would be that large and that you are better off (maintenance-wise) by using one of the serialization libraries out there.