“I can only suggest to keep a pure C++ together with GPU implementation for testing purposes. It is hard to get it right. ”

Do you mean have a CPU based version to check that the GPU version runs in the same way? I would like to do that.

“It seems you are using 8600GT like me. Do you observe 5-10 times speedup?”

I haven’t got a reference point with an otimised C++ program. Using the measure from the last paper, I get 279MCUPS (for a single layer 512×512, batch size of 32), so that would suggest a 27 times speedup. If I’ve got my maths right that is That includes all file reading (though the file is written as a series of floats), conversion to column major format, transfer, weight updates (including momentum) and random number generation.

I’m not sure of the meaning of a few bits of the paper though, the update period is defined as “the time it takes for the implementation to complete a single batch of data”. What’s the batch size? Does it mean a single 512 vector? That’s what my calculation above assumes btw.

Raw performance figures, for training a 784×512x512×2048 network with 10 label units (not softmaxed though) I get a speed of roughly 900 samples per second. That’s for training all layers in sequence. As in, to train all layers in sequence over 50k training samples it takes just under a minute for everything. I get about 3k/s speeds for recognition.

I’m curious about their implementation, they seem to use a matrix multiplication and a matrix addition for the weight updates, as well as a transpose. The transpose worries me a lot, since there’s no need to store a transposed matrix in global memory. You can just change the access pattern (as I assume sgemm does).

Maybe my program is doing something wrong

It’s just a start, but over the next month or so it should become a usable library for others working with RBMs and C++. It’ll be heavily restructured soon to make it easy to create experiments .

In the section that starts with “The algorithm as a whole is:”, you do not have an explicit step that updates the weights & biases. The needed formulas are in a section above, where you wrote that “CD = 0 – n” and “W’ = W + alpha*CD”. Copying those two lines into the list of steps for the algorithm would make the algorithm a bit clearer, I think.

Next, on the diagrams, it might be helpful to explicitly label the blue nodes as visible nodes & the brown nodes as hidden nodes.

Also, when you say, “Si.Sj is just a multiplication of current activation (state) of neuron I and neuron J” it might be slightly clearer to explicitly say that Si is the state of a visible neuron, and Sj is the state of a hidden neuron. This becomes more obvious later on, but at that point in the text one might think Si & Sj could be the states of two different neurons in the same layer, which they’re not.