The Problem with Annotation Processors

by: Matthias Ngeo

For reasons unknown, broaching the subject of annotation processors seem to elicit some primordial fear in developers. People tend to associate annotation processing with borderline witchcraft and sorcery perform-able only by most adept of basement wizards. It doesn’t have to be that way. Annotation processing doesn’t have to be the big scary monster hiding under the bed.

Image taken from https://sourcesofinsight.com/monsters-under-the-bed/
Image taken from https://sourcesofinsight.com/monsters-under-the-bed/

No doubt, problems with annotation processing do exist, but so do solutions to those problems. One problem that stands out in particular, is the difficulty in unit testing annotation processors. A problem that elementary, a suite of JUnit 5 extensions, solves.

What’s this Annotation Processing Thingamajig?

For the uninitiated, an annotation processor is similar to a compiler plug-in. Like it’s namesake, it can be called by the compiler to process annotations, i.e. @Nullable during compilation. Said process covers an extremely broad and vague expanse. Everything from simple value validation to a full-blown pluggable type system like the checker-framework. A simple @Builder annotation builder to full-blown dependency injection via code generation like Dagger.

Post Java 9, it resides inside the java.compiler module. Inside an annotation processor lies the fabled domain of Elements and TypeMirrors, abstract Syntax Tree (AST) representations of the Java language and counterparts to the reflection framework found in Javaland. Elements represent syntactical constructs such as methods, arrays etc. while TypeMirrors represent, well, types such as reference types (classes) and primitives but we digress.

Why So Difficult?

So what makes testing annotation processing so difficult? In our opinion, everything about the annotation processing environment. We’re not claiming that the environment is some evil grotesque being, it’s actually surprisingly well-designed. The problem lies squarely with the unavailability of the environment outside the compiler. Without its environment, testing an annotation processor is a lost cause.

A good drinking game is taking a shot for each method call in an annotation processor that requires an annotation processing environment.

import com.karuslabs.utilitary.Logger;
import com.karuslabs.utilitary.type.TypeMirrors;
import java.util.Set;
import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.RoundEnvironment;
import javax.lang.model.element.TypeElement;
import javax.lang.model.element.VariableElement;
import javax.lang.model.util.Elements;

class StringFieldLint extends AbstractProcessor {
    Elements elements;
    TypeMirrors types;
    Logger logger;    
    @Override
    public void init(ProcessingEnvironment environment) {
        super.init(environment);
        elements = environment.getElementUtils(); // (1)
        types = new TypeMirrors(elements, environment.getTypeUtils()); // (2)
        logger = new Logger(environment.getMessager()); // (3)
    }
    
    @Override
    public boolean process(Set<? extends TypeElement> set, RoundEnvironment round) {
        var elements = round.getElementsAnnotatedWith(Case.class); // (4)
        for (var element : elements) {
            if (!(element instanceof VariableElement)) {
                logger.error(element, "Element is not a variable"); // (5)
                continue;
            }
            
            var variable = (VariableElement) element;
            if (!types.isSameType(variable.asType(), types.type(String.class))) { // (6) (7) (8)
                logger.error(element, "Element is not a string"); // (9)
                continue;
            }
        }
        return false;
    }
}

Pretty much everything requires an annotation processing environment as illustrated above.

At this junction, we have four solutions to overcome this pickle:

To keep a long story short, we ended up becoming smugglers.

Smuggler’s Discovery

While trawling the web, we discovered Google’s compile-testing project, a hidden gem buried beneath the swathes of GitHub projects. Through some clever hacks, the project managed to provide an annotation processing environment for unit tests albeit a little lackluster and limited. Exploring the project, it became obvious that it wasn’t the panacea that we had hoped. The project suffered from a few limitations that we weren’t able to stomach:

class SomeTest {
  @Rule CompilationRule rule = new CompilationRule();
  Types types = rule.getTypes(); // Throws an exception when the method can be called
  
  @Test
  void test() {
    ...
  }
}

This isn’t to say that the project is bad, just that our objectives are different. In fact, some parts of elementary is based on compile-testing. As its name implies, compile-testing focuses on testing the compilation of code, not annotation processing. That’s not our objective. Our objective is to simplify unit testing annotation processors. Thus, after a healthy dose of “Hold my beer” and Not Invented Here Syndrome, the elementary project was conceived.

Elementary, My Dear Watson

With compile-testing as a foundation, we embarked on a quest to bring Elementary to life. Starting with a clean slate blessed us with the freedom to make decisions that would otherwise incite an angry mob with pitchforks and torches:

Our experience working on Chimera code generation tool told us that tests for annotation processors fell into the classic black-box and white-box testing categories. For small and/or simple annotation processors, it was more efficient to invoke the annotation processor inside a compiler against sample Java source files. As the complexity and size of an annotation processor increases, running the annotation processor against sample files yields diminishing returns. It will be far less tedious to isolate and test the individual logical components. Two different categories with two completely different sets of requirements.

Box of Fun Things

Black-box testing annotation processors can be fun. It doesn’t have to be a myriad of set-up, tear-down and configuration. Not accordingly to JavacExtension at least. For each test, JavacExtension compiles a suite of test cases with the given annotation processor(s). The results of the compilation is then funneled to the test method for subsequent assertions. All configuration is handled via annotations with no additional set-up or tear-down required.

They say seeing is believing so let’s get on with the seeing.

Our imaginary annotation processor is fairly straightforward. All it does is check whether an element that is annotated with @Case is also a string field. If an element isn't a string or variable, an error message is printed. Since it's that straightforward, just black-box testing our annotation processor is enough.

@SupportedAnnotationTypes({"*"})
class ImaginaryProcessor extends AnnotationProcessor {
    @Override
    public boolean process(Set<? extends TypeElement> set, RoundEnvironment round) {
        var elements = round.getElementsAnnotatedWith(Case.class);
        for (var element : elements) {
            if (element instanceof VariableElement)) {
                var variable = (VariableElement) element;
                if (!types.isSameType(variable.asType(), types.type(String.class))) {
                    logger.error(element, "Element is not a string");
                }
            } else {
                logger.error(element, "Element is not a variable");
            }
        }
        return false;
    }
}

Testing our imaginary annotation processor isn’t too difficult either. All we need to do is to sprinkle a few annotations on the test class, create some test cases, check the compilation results, and Voila! We’re done.

import com.karuslabs.elementary.Results;
import com.karuslabs.elementary.junit.JavacExtension;
import com.karuslabs.elementary.junit.annotations.Case;
import com.karuslabs.elementary.junit.annotations.Classpath;
import com.karuslabs.elementary.junit.annotations.Options;
import com.karuslabs.elementary.junit.annotations.Processors;

@ExtendWith(JavacExtension.class)
@Options("-Werror")
@Processors({ImaginaryProcessor.class})
@Classpath("my.package.ValidCase")
class ImaginaryTest {
    @Test
    void process_string_field(Results results) {
        assertEquals(0, results.find().errors().count());
    }
    
    @Test
    @Classpath("my.package.InvalidCase")
    void process_int_field(Results results) {
        assertEquals(1, results.find().errors().contains("Element is not a string").count());
    }
}

Let’s break down the code snippet.

Pandora’s Box

This is where things become really interesting. White-box testing isn’t as simple as invoking an annotation processor since the possibilities of what a test is trying to prove is unlimited. In a black-box test, we need only to prove that the compilation results of a known annotation processor against a fixed number of files matches certain criterion. On the contrary, in a white-box test, we do not know why, what and how a component is being tested. The best we can do is make the annotation processing environment accessible inside the test class.

“It can’t be that difficult to allow class scoped annotation processing environments, compile-testing already does that.”

We too, initially felt the same way and boy, were we wrong. While compile-testing does provide an annotation processing environment, it is limited to the scope of a test method. Not being able to access said environment outside of methods means repetitive and verbose initialization code, which blows. Sadly, we couldn’t just tweak compile-testing’s trick either as it was found to be incompatible with our objective.

The secret sauce behind compile-testing is actually pretty straightforward. Each test method is intercepted by a JUnit rule and wrapped in an annotation processor that invokes the method during processing. The test is subsequently executed inside a compiler that the JUnit rule invokes. Unfortunately, in this technique, an annotation processing environment is available only when a test method. It isn’t possible to tweak the technique to intercept the creation of a test instance and inject the test instance inside an annotation processor either due to the constraints of the JUnit lifecycle.

A great deal of time spent at the drawing board later, we succeeded in creating the ToolsExtension. This extension exploited the fact that a test instance only needed access to an annotation processing environment. Tests didn't need to be executed inside an annotation processor. Once we established that, our trick was run a compiler with a blocking annotation processor on a daemon thread before each test instance was created. With compilation suspended inside the processor, the environment is made accessible to the test instance on the main thread. Only after all tests has been executed does compilation resume.

Here’s a poorly drawn MS Paint diagram illustrating the entire process
Here’s a poorly drawn MS Paint diagram illustrating the entire process

Let’s pretend that as a result of the imaginary processor we described in Box of Fun Things having grown in scope and size, it was refactored into multiple components, one of which checks if an element is a string variable like the original annotation processor.

class Lint {
    
    final TypeMirrors types;
    final TypeMirror expectedType;
    
    Lint(TypeMirrors types) {
        this.types = types;
        this.expectedType = types.type(String.class);
    }
    
    public boolean lint(Element element) {
        if (!(element instanceof VariableElement)) {
            return false;
        }
        
        var variable = (VariableElement) element;
        return types.isSameType(expectedType, variable.asType());
    }
    
}

Using the ToolsExtension to test the annotation processor yields the following code snippet:

import com.karuslabs.elementary.junit.Cases;
import com.karuslabs.elementary.junit.Tools;
import com.karuslabs.elementary.junit.ToolsExtension;
import com.karuslabs.elementary.junit.annotations.Inline;
import com.karuslabs.utilitary.type.TypeMirrors;

@ExtendWith(ToolsExtension.class)
@Inline(name = "Samples", source = {
"import com.karuslabs.elementary.junit.annotations.Case;",
"",
"class Samples {",
"  @Case(\"first\") String first;",
"  @Case String second() { return \"\";}",
"}"})
class ToolsExtensionExampleTest {
    
    Lint lint = new Lint(Tools.typeMirrors());
    
    @Test
    void lint_string_variable(Cases cases) {
        var first = cases.one("first");
        assertTrue(lint.lint(first));
    }
    
    @Test
    void lint_method_that_returns_string(Cases cases) {
        var second = cases.get(1);
        assertFalse(lint.lint(second));
    }
    
}

Let’s break down the code snippet:

The Case for Cases

With the completion of ToolsExtension, we succeeded in our quest to smuggle an annotation processing environment out of the compiler. Yet one final piece in the puzzle still remains. How do we create those elements to test our code against? The jdk.compiler module doesn't provide a way to create elements. While mocking an Element is possible it is far from developer-friendly. Not only is the initialization verbose, unwieldy and convoluted, it is also difficult to guarantee that the mocked element's behaviour matches its actual counterpart. We can't look to compile=testing for guidance either since it doesn't provide anything like that.

After much headache, we managed to find the missing piece. Let’s have the compiler transform our test cases written in idiomatic Java into elements for us. That way, we avoid the mess surrounding the initialization of elements and the resultant code is far easier to understand. To achieve that, we required some way to fetch elements from the compiler. After further refinement of the concept, we eventually developed the Cases class and corresponding @Case annotation.

Returning to our code snippet from Pandora’s Box, let’s analyze it in greater detail.

import com.karuslabs.elementary.junit.Cases;
import com.karuslabs.elementary.junit.Tools;
import com.karuslabs.elementary.junit.ToolsExtension;
import com.karuslabs.elementary.junit.annotations.Inline;
import com.karuslabs.utilitary.type.TypeMirrors;

@ExtendWith(ToolsExtension.class)
@Inline(name = "Samples", source = {
"import com.karuslabs.elementary.junit.annotations.Case;",
"",
"class Samples {",
"  @Case(\"first\") String first;",
"  @Case String second() { return \"\";}",
"}"})
class ToolsExtensionExampleTest {
    
    Lint lint = new Lint(Tools.typeMirrors());
    
    @Test
    void lint_string_variable(Cases cases) {
        var first = cases.one("first");
        assertTrue(lint.lint(first));
    }
    
    @Test
    void lint_method_that_returns_string(Cases cases) {
        var second = cases.get(1);
        assertFalse(lint.lint(second));
    }
    
}

Idea Graveyard

As mentioned at the beginning of this article, we explored a few other avenues which eventually led to dead-ends. We thought them to be interesting enough to discuss in the following sections. Most of them ended up getting shelved due to the impracticality and unacceptable trade-offs for the solution.

Not testing annotation processors goes without saying to be a terrible choice. Just because testing them is difficult doesn’t give us the liberty of skipping that. The problems will only worsen over time if we choose to take the easy route out. Furthermore, most annotation processors usually do code generation and static type analysis. Both of which are extremely difficult to troubleshoot.

“Good things come to those wait. But better things come to those who work for it.”

Had JEP 119: javax.lang.model Implementation Backed by Core Reflection been shipped with JDK 8, I highly doubt elementary would have even been conceived. It solved the issue with accessing an annotation processing environment outside of a compiler by providing a standard implementation. Sadly, it was shelved and future efforts seems to have stalled. A wait and see approach to unit testing annotation processors would thus be unfeasible as there isn’t anything to wait on.

A problem more difficult than testing annotation processing is trying to mock/re-implement the annotation processing environment. Since elements represent an AST for the Java language, we need to be intimate with the language specification to guarantee that the behaviour of mocked/re-implemented elements do not deviate from the original. This honestly makes testing annotation processors seem like a Disney fairy-tale, we don’t want to touch that even with a ten-foot pole. A few existing re-implementations do exist but seem to have been long-abandoned for years. In the end, it boils down to the troubles outweighing the benefits that led us to abandon this avenue.

Final Thoughts

We’ve reached the end of our journey to simplify the testing of annotation processors. Looking back, it has been an absolute blast working on the project. How adopted this project is still remains to be seen. But if anything, I hope that this article encouraged you start playing around with annotation processors.

In summary, Elementary introduces:

That said, this is only beginning of yet another journey. A journey that I am hopeful will bring many new feature and improvements to elementary in the time to come. Until the next time, happy coding!


Shameless advertising* This article is based on Elementary, https://github.com/Pante/elementary


<- Index