This post is a simple write-up of the process and problem during my experience. Note: Please replace username to what you need in this post!
Under Linux (Ubuntu here)
1. Create public and private key under Linux
ssh-keygen -t rsa -b 4096 -f <username>
You will see two new file under the directory.
2. You need to send the username.pub file to the server administrator and you keep the private key file username.
3. Test SSH
ssh -i username username@host(or IP)
Note: the first username is your private key file and the second one is the username for login.
4. Test SVN
svn co svn+ssh://username@host/path_to_repository
Under Windows (Win 7)
1. Get the copy of the private key file
2. Convert the key into Putty format. Start PuTTYgen and use Conversions -> Import key Browse to your file keyfile which you got from the server. Finally click on Save private key and save the file as keyfile.PPK.
3. Use this new key file for SSH. Start putty and on the Session tab set the hostname to the name or IP address of your server, the protocol to SSH and save the session as SvnConnection or whatever name you prefer. On the SSH tab set the preferred SSH protocol version to 2 and from Auth set the full path to the .PPK private key file you converted earlier. Go back to the Sessions tab and hit the Save button. You will now see SvnConnection in the list of saved sessions. Click on Open and you should see a telnet style login prompt.
4. Test TortoiesSVN. One way to simplify the URL in TortoiseSVN is to set the user inside the PuTTY session. For this you have to load your already defined session SvnConnection in PuTTY and in the Connection tab set Auto login user to the user name, e.g. svnuser. Save your PuTTY session as before and try the following URL inside TortoiseSVN: svn+ssh://SvnConnection/repos
5. Check out the SVN use the same URL.