Only one resource I found that was explaining how to make it possible to import private key: http://nsayer.blogspot.com/2010/02/import-private-key-into-java-keystore.html. But this article didn't covered for some reason how to make pkcs12 file format. So here I'll put instructions:
Suppose you have
cert.crtin PEM format, that was signed by authority or you got it externally. Firstly export them into pkcs12 format:
openssl pkcs12 -export -in cert.crt -inkey private.key -certfile cert.crt -name "My certificate" -out keystore.p12
Next, use java keytool command to create keystore in JKS format (or any other that keytool supports):
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype JKS
Thats it. You've got it in format that you need it in java.
Compare this for instance with "Old style" suggestions with need to convert into DER format, then use a compiled java class to import into keystore: http://www.agentbob.info/agentbob/79-AB.html