GPG data stream encryption and signing with Bouncycastle

I found it incredibly difficult to use BouncyCastle’s Java library to create data that GPG was actually able to decrypt. Even if compression is not the matter because you only have small tokens, it did not work without a Zip-DataGenerator in between (although there are plaintext variants of these Generators among the BC classes). GPG was complaining about not being able to decode character sequence “2D” or something.

You can save the string to a file if you need this, I had to create string data for passing it to a restful API.

protected final String publickeyFile; // the public key of the receiver, in armored format
protected final String privatekeyFile; // your private key ring file 
protected final String userId; // usually your email-address in your key ring
protected final String password; // password to your private key ring
protected final SecureRandom randomGenerator;

/**
 * Encrypt plaintext message using public key from publickeyFile.
 * 
 * @param message the message
 * @return the string
 */
private String encrypt( String message )
    throws PGPException, IOException, NoSuchProviderException
{
    // read public key file, this is an armored key file, no binary stream
    InputStream in =
        new ArmoredInputStream( Thread.currentThread().getContextClassLoader()
            .getResourceAsStream( publickeyFile ) );
    // find the PGP key in the file
    PGPPublicKey publicKey = findPublicGPGKey( in );
    Preconditions
        .checkNotNull( publicKey, Format.format( "Did not find the GPG key in keyfile ", publickeyFile ) );

    // Encode the string into bytes using utf-8
    byte[] utf8Bytes = message.getBytes( UTF8_ENCODING );

    ByteArrayOutputStream compressedOutput = new ByteArrayOutputStream();
    // compress bytes with zip
    PGPLiteralDataGenerator literalDataGenerator = new PGPLiteralDataGenerator();
    // the reason why we compress here is GPG not being able to decrypt our message input but if we do not compress.
    // I guess pkzip compression also encodes only to GPG-friendly characters.
    PGPCompressedDataGenerator compressedDataGenerator =
        new PGPCompressedDataGenerator( CompressionAlgorithmTags.ZIP );
    try
    {
        OutputStream literalDataOutput =
            literalDataGenerator.open( compressedOutput, PGPLiteralData.BINARY, FAKE_PGP_INPUT_FILENAME,
                utf8Bytes.length, new Date() );
        // update bytes in the stream
        literalDataOutput.write( utf8Bytes );
    }
    catch ( IOException e )
    {
        // catch but close the streams in finally
        throw e;
    }
    finally
    {
        compressedDataGenerator.close();
        Closeables.closeQuietly( compressedOutput );
    }

    // now we have zip-compressed bytes
    byte[] compressedBytes = compressedOutput.toByteArray();

    PGPEncryptedDataGenerator encryptedDataGenerator =
        new PGPEncryptedDataGenerator( PGPEncryptedData.CAST5, true, randomGenerator, BC_PGP_PROVIDER );
    // use public key to encrypt data
    encryptedDataGenerator.addMethod( publicKey );

    // literalDataOutput --> compressedOutput --> ArmoredOutputStream --> ByteArrayOutputStream
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    ArmoredOutputStream armoredOut = new ArmoredOutputStream( byteArrayOutputStream );
    OutputStream encryptedOutput = null;
    try
    {
        encryptedOutput = encryptedDataGenerator.open( armoredOut, compressedBytes.length );
        encryptedOutput.write( compressedBytes );
    }
    catch ( IOException e )
    {
        throw e;
    }
    catch ( PGPException e )
    {
        throw e;
    }
    finally
    {
        Closeables.closeQuietly( encryptedOutput );
        Closeables.closeQuietly( armoredOut );
    }
    String encrypted = new String( byteArrayOutputStream.toByteArray() );
    LOG.debug( Format.format( "Message: {} ", message ) );
    LOG.debug( Format.format( "Encrypted: {} ", encrypted ) );
    return encrypted;
}

/**
 * Find public gpg key in InputStream.
 * 
 * @param inputStream the input stream
 * @return the PGP public key
 */
private PGPPublicKey findPublicGPGKey( InputStream inputStream )
    throws IOException, PGPException
{
    // get all key rings in the input stream
    PGPPublicKeyRingCollection publicKeyRingCollection =
        new PGPPublicKeyRingCollection( PGPUtil.getDecoderStream( inputStream ) );
    LOG.debug( Format.format( "key ring size: {}", publicKeyRingCollection.size() ) );
    Iterator<PGPPublicKeyRing> keyRingIter = publicKeyRingCollection.getKeyRings();
    // iterate over keyrings
    while ( keyRingIter.hasNext() )
    {
        PGPPublicKeyRing keyRing = keyRingIter.next();
        Iterator<PGPPublicKey> keyIter = keyRing.getPublicKeys();
        // iterate over public keys in the key ring
        while ( keyIter.hasNext() )
        {
            PGPPublicKey tmpKey = keyIter.next();
            Preconditions.checkArgument( tmpKey != null );
            LOG.debug( Format.format( "Encryption key = {}, Master key = {}, UserIDs: {}",
                tmpKey.isEncryptionKey(), tmpKey.isMasterKey(), Iterators.toString( tmpKey.getUserIDs() ) ) );
            // we need a master encryption key
            if ( tmpKey.isEncryptionKey() && tmpKey.isMasterKey() )
            {
                return tmpKey;
            }
        }
    }
    throw new PGPException( "No public key found!" );
}

/**
 * Find private gpg key in InputStream.
 * 
 * @param inputStream the input stream
 * @param userId the user id
 * @return the PGP secret key
 */
private PGPSecretKey findPrivateGPGKey( InputStream inputStream, String userId )
    throws IOException, PGPException, NoSuchProviderException
{
    // iterate over every private key in the key ring
    PGPSecretKeyRingCollection secretKeyRings =
        new PGPSecretKeyRingCollection( PGPUtil.getDecoderStream( inputStream ) );
    // look for the key ring that is used to authenticate our reporting facilities
    Iterator<PGPSecretKeyRing> privateKeys = secretKeyRings.getKeyRings( userId );
    // iterate over every private key in the ring
    while ( privateKeys.hasNext() )
    {
        PGPSecretKeyRing secretKeyRing = privateKeys.next();
        PGPSecretKey tmpKey = secretKeyRing.getSecretKey();
        Preconditions.checkArgument( tmpKey != null );
        LOG.debug( Format.format( "Signing key = {}, Master key = {}, UserId = {}", tmpKey.isSigningKey(),
            tmpKey.isMasterKey(), userId ) );
        // we want the signing master key
        if ( tmpKey.isSigningKey() && tmpKey.isMasterKey() )
        {
            return tmpKey;
        }
    }
    throw new PGPException( "No private key found!" );
}

/**
 * Sign a plaintext message using our private PGP key file.
 * 
 * @param message the message
 * @return the string
 */
private String sign( String message )
    throws SignatureException, NoSuchAlgorithmException, PGPException, NoSuchProviderException, IOException
{
    InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream( privatekeyFile );
    PGPSecretKey secretKey = findPrivateGPGKey( in, userId );
    PGPPrivateKey privateKey = secretKey.extractPrivateKey( password.toCharArray(), BC_PGP_PROVIDER );

    PGPSignatureGenerator signature =
        new PGPSignatureGenerator( secretKey.getPublicKey().getAlgorithm(), PGPUtil.SHA1, BC_PGP_PROVIDER );
    signature.initSign( PGPSignature.BINARY_DOCUMENT, privateKey );

    Iterator<String> userIds = secretKey.getPublicKey().getUserIDs();
    // use the first userId
    if ( userIds.hasNext() )
    {
        PGPSignatureSubpacketGenerator subpacketGenerator = new PGPSignatureSubpacketGenerator();
        subpacketGenerator.setSignerUserID( false, userIds.next() );
        signature.setHashedSubpackets( subpacketGenerator.generate() );
    }
    else
    {
        throw new ReportingException( "Did not find userId" );
    }

    byte[] utf8Bytes = message.getBytes( UTF8_ENCODING );

    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    ArmoredOutputStream armoredOut = new ArmoredOutputStream( byteArrayOutputStream );

    PGPCompressedDataGenerator compressedDataGenerator = new PGPCompressedDataGenerator( PGPCompressedData.ZLIB );
    BCPGOutputStream bcOutputStream = new BCPGOutputStream( compressedDataGenerator.open( armoredOut ) );

    signature.generateOnePassVersion( false ).encode( bcOutputStream );

    // literalDataOutput --> bcOutputStream--> compressedOutput --> ArmoredOutputStream --> ByteArrayOutputStream
    // &&
    // signatureOutput --> bcOutputStream--> compressedOutput --> ArmoredOutputStream --> ByteArrayOutputStream
    PGPLiteralDataGenerator literalDataGenerator = new PGPLiteralDataGenerator();
    try
    {
        OutputStream literalDataOutput =
            literalDataGenerator.open( bcOutputStream, PGPLiteralData.BINARY, FAKE_PGP_INPUT_FILENAME,
                utf8Bytes.length, new Date() );

        // update bytes in the streams
        literalDataOutput.write( utf8Bytes );
        signature.update( utf8Bytes );

        // close generators and update signature
        literalDataGenerator.close();
        signature.generate().encode( bcOutputStream );
        compressedDataGenerator.close();
    }
    catch ( IOException e )
    {
        throw e;
    }
    catch ( PGPException e )
    {
        throw e;
    }
    finally
    {
        Closeables.closeQuietly( byteArrayOutputStream );
        Closeables.closeQuietly( armoredOut );
    }
    // }
    String signed = new String( byteArrayOutputStream.toByteArray() );
    LOG.debug( Format.format( "Message: {} ", message ) );
    LOG.debug( Format.format( "Signature: {} ", signed ) );
    return signed;
}

Apache httpcomponents-core NIO SSL example broken

The example snippet of a SSL enabled Apache http-components-core-based Webserver did not work out for me. It crashed after every Request with the following message:

Connection closed: [null]
I/O error: I/O dispatch worker terminated abnormally
Shutdown

Printing the StackTrace revealed the following:

org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor .execute(AbstractMultiworkerIOReactor.java:324)
at com.demandware.tenantssl.NHttpSSLServer.main(SslPaymentServer.java:176)
Caused by: java.lang.UnsupportedOperationException: Entity template does not implement getContent()
at org.apache.http.entity.EntityTemplate.getContent(EntityTemplate.java:57)
at org.apache.http.nio.entity.NHttpEntityWrapper.(NHttpEntityWrapper.java:56)
at org.apache.http.nio.protocol.AsyncNHttpServiceHandler .sendResponse(AsyncNHttpServiceHandler.java:495)
at org.apache.http.nio.protocol.AsyncNHttpServiceHandler .responseReady(AsyncNHttpServiceHandler.java:362)
at org.apache.http.nio.protocol.BufferingHttpServiceHandler .responseReady(BufferingHttpServiceHandler.java:135)
at org.apache.http.impl.nio.DefaultNHttpServerConnection .produceOutput(DefaultNHttpServerConnection.java:221)
at org.apache.http.impl.nio.ssl.SSLServerIOEventDispatch .outputReady(SSLServerIOEventDispatch.java:252)
at org.apache.http.impl.nio.reactor.BaseIOReactor .writable(BaseIOReactor.java:185)
at org.apache.http.impl.nio.reactor.AbstractIOReactor .processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor .processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor .execute(AbstractIOReactor.java:275)
at org.apache.http.impl.nio.reactor.BaseIOReactor .execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker .run(AbstractMultiworkerIOReactor.java:542)
at java.lang.Thread.run(Thread.java:662)

As inline subclassed EntityTemplate does not implement getContent(), you just have to add this tiny snippet in the two inline definitions of EntityTemplate for a minimal working SSL-server:

EntityTemplate body = new EntityTemplate(new ContentProducer()
{

	public void writeTo(final OutputStream outstream) throws IOException
	{
		OutputStreamWriter writer = new OutputStreamWriter(outstream, "UTF-8");
		writer.write("<html><body><h1>");
		writer.write("File ");
		writer.write(file.getPath());
		writer.write(" not found");
		writer.write("</h1></body></html>");
		writer.flush();
	}
}) {
	@Override
	public InputStream getContent() {
		return new InputStream() {

			@Override
			public int read() throws IOException
			{
				return 0;
			}

		};
	}
};

Hadoop MultipleOutputCollector MRunit Testcase

Code-Snippet for a test case with MRUnit and Mockito to test a Hadoop Reducer with MultipleOutputs. MRUnit ReduceDriver unfortunately does not return result data when using multiple Ouptuts.

private HashMap<String, MockOutputCollector> mockOutputCollectors = new HashMap<String, MockOutputCollector>();

@Test
public void test() {
	driver.withInput(new Text(INPUT_KEY, INPUT).run();
	List<Type> resultList = new ArrayList<Type>();

	// This has to be done that complicated way as the above driver.run() only returns an empty list
	// if run with ReduceDriver. For some reason it works with MapDriver, though...
	for (String collectorDirectory : mockOutputCollectors.keySet()) {
		MockOutputCollector<NullWritable, Object> mockOutputCollector = mockOutputCollectors
				.get(collectorDirectory);
		for (Pair<NullWritable, Object> pair : mockOutputCollector.getOutputs()) {
			if (pair.getSecond() instanceof MultipleOutputType1) {
				MultipleOutputType1 multipleOutputType = (MultipleOutputType1) pair.getSecond();
				logger.debug("MultipleOutputType1: "
						+ multipleOutputType.toString());
			}
			if  (pair.getSecond() instanceof MultipleOutputType2) {
				[...]
			}

		}
	}
}

@Before
public void setup() {
	for (Directories directory : Directories.values()) {
		mockOutputCollectors.put(directory.getDirectoryName(),
				new MockOutputCollector());
		when(
				multipleOutputs.getCollector(
						eq(directory.getDirectoryName()),
						argThat(new IsMockReporter()))).thenReturn(
				mockOutputCollectors.get(directory.getDirectoryName()));
	}
	mockOutputCollectors.put(
			OUTPUT_DIRECTORY_NAME,
			new MockOutputCollector());
	
	when(
			multipleOutputs.getCollector(
					eq(OUTPUT_DIRECTORY_NAME),
					argThat(new IsMockReporter()))).thenReturn(
				mockOutputCollectors.get(Constants.OUTPUT_DIRECTORY_NAME));
}

JAAS LoginModule for Linux Authentication

I was looking for a possibility to use the Linux passwd database authentication in Java in order to authenticate a user with a password against the local linux passwd database. Soon I realized that this is not possible out-of-the-box because there is no LoginModule for that. The project http://sourceforge.net/projects/jaas-pam/ seemed exactly the right thing for that but unfortunately it is outdated and lets Sun JRE 1.6 and IBM JRE 1.5 segfault. No matter if using the supplied binary lib or with a recompiled one, no avail.

So I implemented my own one, a JAAS LoginModule that authenticates against a local SSH Server using the sshj project. It opens up a connection to your local SSH server which must be running on the machine you want to authenticate against. Then it tries to log in the supplied user. If it succeeds the supplied username password pair is vaild on the server. You need the sshj library, mavens artifact is net.schmitzz.sshj.

import java.io.IOException;
import java.util.Map;

import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.login.LoginException;
import javax.security.auth.spi.LoginModule;

import net.schmizz.sshj.SSHClient;

public class SshLoginModule implements LoginModule {
	private CallbackHandler handler;
	private Subject subject;
	private String username;

	/**
	 * @see javax.security.auth.spi.LoginModule#abort()
	 */
	@Override
	public boolean abort() throws LoginException {
		return false;
	}

	/**
	 * @see javax.security.auth.spi.LoginModule#commit()
	 */
	@Override
	public boolean commit() throws LoginException {
		try {
			final DummyUserPrincipal user = new DummyUserPrincipal(
					username);
			final DummyRolePrincipal role = new DummyRolePrincipal(
					"admin");

			subject.getPrincipals().add(user);
			subject.getPrincipals().add(role);
			return true;

		} catch (final Exception e) {
			throw new LoginException(e.getMessage());
		}
	}

	/**
	 * @see javax.security.auth.spi.LoginModule#initialize(javax.security.auth.Subject,
	 *      javax.security.auth.callback.CallbackHandler, java.util.Map,
	 *      java.util.Map)
	 */
	@Override
	public void initialize(Subject subject, CallbackHandler callbackHandler, Map sharedState, Map options) {

		handler = callbackHandler;
		this.subject = subject;
	}

	/**
	 * @see javax.security.auth.spi.LoginModule#login()
	 */
	@Override
	public boolean login() throws LoginException {
		final Callback[] callbacks = new Callback[2];
		callbacks[0] = new NameCallback("username");
		callbacks[1] = new PasswordCallback("password", true);

		final SSHClient sshClient = new SSHClient();
		final String fingerprint = "33:**:**:**:..."; //15 hex digit fingerprint of your server.
		final String hostname = "localhost";

		try {
			handler.handle(callbacks);
			final String name = ((NameCallback) callbacks[0])
					.getName();
			final String password = String
					.valueOf(((PasswordCallback) callbacks[1])
							.getPassword());

			sshClient.addHostKeyVerifier(hostname, 22, fingerprint);
			sshClient.connect(hostname, 22);
			sshClient.authPassword(name, password);

			if (!sshClient.isAuthenticated()) {
				throw new LoginException(
						"Authentication failed for unknown reason.");
			}
			username = name;
			return true;

		} catch (final IOException e) {
			throw new LoginException("Authentication failed: "
					+ e.getMessage());
		} catch (final UnsupportedCallbackException e) {
			throw new LoginException("Authentication failed: "
					+ e.getMessage());
		} finally {
			try {
				sshClient.disconnect();
			} catch (final IOException e) {
				throw new LoginException("Authentication failed: "
						+ e.getMessage());
			}
		}
	}

	/**
	 * @see javax.security.auth.spi.LoginModule#logout()
	 */
	@Override
	public boolean logout() throws LoginException {
		try {
			final DummyUserPrincipal user = new DummyUserPrincipal(
					username);
			final DummyRolePrincipal role = new DummyRolePrincipal(
					"admin");
			subject.getPrincipals().remove(user);
			subject.getPrincipals().remove(role);
			return true;
		} catch (final Exception e) {
			throw new LoginException(e.getMessage());
		}
	}
}

ACTA – Ein Angriff auf das freie Internet

Hinter verschlossenen Türen werden gerade wieder Maßnahmen der Durchsetzungen von Software-Patenten verhandelt, die dann OHNE Einspruchsrecht oder Inhaltsdiskussion des EU-Parlaments ein Gesetz werden soll.

Viele Funkamateure setzen heutzutage OpenSource-Software wie Linux, Firefox oder OpenOffice ein, die von Software-Patenten massiv in ihrer Existenz bedroht sind. Patente auf Software spiegeln nicht die Urheberrechte an Quelltext wieder. Diese Rechte werden bereits heute ausreichend geschützt. Software-Patente verbieten den “Nachbau” vorhandener Software, beispielsweise durch OpenSource-Projekte.

Für jeden, der etwas dagegen tun will, empfehle ich folgende Petitionen mitzuzeichnen:

http://www.fsf.org/campaigns/acta/acta-declaration

http://publicacta.org.nz/sign-the-wellington-declaration/

Desweiteren unterzeichne ich, Christian Bayer, Salvador-Allende-Platz 9, 07747 Jena den folgenden offenen Brief  von http://www.laquadrature.net/en/acta-a-global-threat-to-freedoms-open-letter, weil ich der Meinung bin, das derart tiefgreifende Einschränkungen der digitalen Freiheit, wie sie ACTA vorschlägt auf keinen Fall ohne demokratische Legitimation erfolgen dürfen.

ACTA: A Global Threat to Freedoms

open letter

The Anti-Counterfeiting Trade Agreement (ACTA) is a broad intergovernmental agreement under negotiation ranging from the key social issue of access to medicine[1] to criminal Internet regulation. We fear it could seriously hinder European innovation in the digital single market while undermining fundamental rights and democracy at large.

The negotiation process itself raises important questions of transparency and due democratic process, given that the content of the draft agreement has been kept secret for more than 18 months, although some details about the proposals recently leaked to the public. More worrying still, while the European Parliament has been denied access to the documents, US industry has been granted access to them, albeit only after signing non-disclosure agreements.

A recent analysis by the European Commission of the ACTA Internet chapter[2] proves that the topics under discussion go far beyond the current body of EU law. Most importantly, the Commission’s analysis confirms that the current draft of ACTA would profoundly restrict the fundamental rights and freedoms of European citizens, most notably the freedom of expression and communication privacy. These are very much at risk, since the current draft pushes for the implementation of three-strikes schemes and content filtering policies by seeking to impose civil and criminal liability on technical intermediaries such as internet service providers. The text would also radically erode the exercise of interoperability that is essential for both consumer rights and competitiveness.

Consequently, we urge the Parliament to call on European negotiators to establish transparency in the negotiation process and publish the draft agreement, and not to accept any proposal which would undermine citizens’ rights and freedoms. Furthermore, we urge the Parliament to make an unequivocal statement to the Commission and Council that any agreement which does not respect these core principles would force the Parliament to reject the entire text.

[1] See: http://www.oxfam.org/en/pressroom/pressrelease/2009-07-15/criminalize-generic-medicines-hurt-poor-countries

[2] See: http://sharemydoc.org/files/philip/ec_analysis_of_acta_internet_chapter.pdf