For the past four months I've been working on a JSON library for Java.
It's not original. Most of the implementation of the parser I stole from Clojure's data.json and the user facing API is a total ripoff of Elm's JSON library. The only novel engineering I've done has been in translation.
It's also not amazingly fast. Last I benchmarked it, it was around 5x as slow as Jackson, the current king of Java's JSON castle. There are paths to improve that but, whether for lack of time or ability, I haven't explored any of them.
Despite all that, I think you should try it. The rest of this post is going to be an effort to convince you to do so.
First, I am going to go through a basic tutorial to get you up to speed. Then I am going to go through some pitches that I hope convince you.
JSON is a data format. It looks like the following sample.
{
"name": "kermit",
"wife": null,
"girlfriend": "Ms. Piggy",
"age": 22,
"children": [
{
"species": "frog",
"gender": "male"
},
{
"species": "pig",
"gender": "female"
}
],
"commitmentIssues": true
}In JSON you represent data using a combination of objects (maps from strings to JSON), arrays (ordered sequences of JSON), strings, numbers, true, false, and null.
Therefore, one "natural" way to think about the data stored in a JSON document is as the union of those possibilities.
JSON is one of
- a map of string to JSON
- a list of JSON
- a string
- a number
- true
- false
- nullThe way to represent this in Java is using a sealed interface, which provides an explicit list of types which are allowed to implement it.
public sealed interface Json
permits
JsonObject,
JsonArray,
JsonString,
JsonNumber,
JsonBoolean,
JsonNull {
}This means that if you have a field or variable which has the type Json, you know that it is either a JsonObject, JsonArray, JsonString, JsonNumber, JsonBoolean, or JsonNull.
That is the first thing provided by my library. There is a Json type and subtypes representing those different cases.
import dev.mccue.json.*;
public class Main {
static Json greeting() {
return JsonString.of("hello");
}
public static void main(String[] args) {
Json json = greeting();
switch (json) {
case JsonObject object ->
System.out.println("An object");
case JsonArray array ->
System.out.println("An array");
case JsonString str ->
System.out.println("A string");
case JsonNumber number ->
System.out.println("A number");
case JsonBoolean bool ->
System.out.println("A boolean");
case JsonNull __ ->
System.out.println("A json null");
}
}
}You can create instances of these subtypes using factory methods on the types themselves.
import dev.mccue.json.*;
import java.util.List;
import java.util.Map;
public class Main {
public static void main(String[] args) {
JsonObject kermit = JsonObject.of(Map.of(
"name", JsonString.of("kermit"),
"age", JsonNumber.of(22),
"commitmentIssues", JsonBoolean.of(true),
"wife", JsonNull.instance(),
"children", JsonArray.of(List.of(
JsonString.of("Tiny Tim")
))
));
System.out.println(kermit);
}
}Or by using factory methods on Json, which aren't guaranteed to give you any specific subtype but in exchange will handle converting any stray nulls to JsonNull.
import dev.mccue.json.*;
import java.util.List;
import java.util.Map;
public class Main {
public static void main(String[] args) {
Json kermit = Json.of(Map.of(
"name", Json.of("kermit"),
"age", Json.of(22),
"commitmentIssues", Json.of(true),
"wife", Json.ofNull(),
"children", Json.of(List.of(
JsonString.of("Tiny Tim")
))
));
System.out.println(kermit);
}
}For JsonObject and JsonArray, there also use builders available which can make it so that you don't need to write Json.of on every value.
import dev.mccue.json.Json;
public class Main {
public static void main(String[] args) {
Json kermit = Json.objectBuilder()
.put("name", "kermit")
.put("age", 22)
.putTrue("commitmentIssues")
.putNull("wife")
.put("children", Json.arrayBuilder()
.add("Tiny Tim"))
.build();
System.out.println(kermit);
}
}Once you have some Json you can write it out to a String using Json.writeString
import dev.mccue.json.Json;
public class Main {
public static void main(String[] args) {
Json songJson = Json.objectBuilder()
.put("title", "Rainbow Connection")
.put("year", 1979)
.build();
String song = Json.writeString(songJson);
System.out.println(song);
}
}{"title":"Rainbow Connection","year":1979}If output is meant to be consumed by humans then whitespace can be added using a customized instance of JsonWriteOptions.
import dev.mccue.json.Json;
import dev.mccue.json.JsonWriteOptions;
public class Main {
public static void main(String[] args) {
Json songJson = Json.objectBuilder()
.put("title", "Rainbow Connection")
.put("year", 1979)
.build();
String song = Json.writeString(
songJson,
new JsonWriteOptions()
.withIndentation(4)
);
System.out.println(song);
}
}{
"title": "Rainbow Connection",
"year": 1979
}If you want to write JSON to something other than a String, you need to obtain a Writer and use Json.write.
import dev.mccue.json.Json;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
public class Main {
public static void main(String[] args) throws IOException {
Json songJson = Json.objectBuilder()
.put("title", "Rainbow Connection")
.put("year", 1979)
.build();
try (var fileWriter = Files.newBufferedWriter(
Path.of("song.json"))
) {
Json.write(songJson, fileWriter);
}
}
}To turn a class you have defined into JSON, you just need to make a method which creates an instance of Json from the data stored in your class.
import dev.mccue.json.Json;
record Muppet(String name) {
Json toJson() {
return Json.objectBuilder()
.put("name", name)
.build();
}
}
public class Main {
public static void main(String[] args) {
var beaker = new Muppet("beaker");
Json beakerJson = beaker.toJson();
System.out.println(Json.writeString(beakerJson));
}
}This process is "encoding." You "encode" your data into JSON and then "write" that JSON to some output.
For classes that you did not define, the logic for the conversion just needs to live somewhere. Dealer's choice where, but static methods are generally a good call.
import dev.mccue.json.Json;
import java.time.Month;
import java.time.MonthDay;
import java.time.format.DateTimeFormatter;
final class TimeEncoders {
private TimeEncoders() {}
static Json monthDayToJson(MonthDay monthDay) {
return Json.of(
DateTimeFormatter.ofPattern("MM-dd")
.format(monthDay)
);
}
}
record Muppet(String name, MonthDay birthday) {
Json toJson() {
return Json.objectBuilder()
.put("name", name)
.put(
"birthday",
TimeEncoders.monthDayToJson(birthday)
)
.build();
}
}
public class Main {
public static void main(String[] args) {
var elmo = new Muppet(
"Elmo",
MonthDay.of(Month.FEBRUARY, 3)
);
Json elmoJson = elmo.toJson();
System.out.println(Json.writeString(elmoJson));
}
}{"name":"Elmo","birthday":"02-03"}If a class you define has a JSON representation that could be considered "canonical", the interface JsonEncodable can be implemented. This will let you pass an instance of the class directly to Json.writeString or Json.write.
import dev.mccue.json.Json;
import dev.mccue.json.JsonEncodable;
record Muppet(String name, boolean great)
implements JsonEncodable {
@Override
public Json toJson() {
return Json.objectBuilder()
.put("name", name)
.put("great", great)
.build();
}
}
public class Main {
public static void main(String[] args) {
var gonzo = new Muppet("Gonzo", true);
System.out.println(Json.writeString(gonzo));
}
}The inverse of writing JSON is reading it.
If you have some JSON stored in a String you can read it into Json using Json.readString.
import dev.mccue.json.Json;
public class Main {
public static void main(String[] args) {
Json movie = Json.readString("""
{
"title": "Treasure Island",
"cast": [
{
"name": "Kermit",
"role": "The Captain",
"muppet": true
},
{
"name": "Tim Curry",
"role": "Long John Silver",
"muppet": false
}
]
}
""");
System.out.println(movie);
}
}If that JSON is coming from another source, you need to obtain a Reader and use Json.read.
import dev.mccue.json.Json;
import java.io.IOException;
import java.io.Reader;
import java.nio.file.Files;
import java.nio.file.Path;
public class Main {
public static void main(String[] args) throws IOException {
// If you were following along, we created this earlier!
Json song;
try (Reader fileReader = Files.newBufferedReader(
Path.of("song.json"))
) {
song = Json.read(fileReader);
}
System.out.println(song);
}
}If the JSON you provide is malformed in some way, a JsonReadException will be thrown.
import dev.mccue.json.Json;
public class Main {
public static void main(String[] args) {
// Should be in quotes
Json.readString("fozzie");
}
}Exception in thread "main" dev.mccue.json.JsonReadException: JSON error (unexpected character): f
at dev.mccue.json.JsonReadException.unexpectedCharacter(JsonReadException.java:33)
at dev.mccue.json.internal.JsonReaderMethods.readStream(JsonReaderMethods.java:525)
at dev.mccue.json.internal.JsonReaderMethods.read(JsonReaderMethods.java:533)
at dev.mccue.json.internal.JsonReaderMethods.readFullyConsume(JsonReaderMethods.java:543)
at dev.mccue.json.Json.readString(Json.java:369)
at dev.mccue.json.Json.readString(Json.java:364)
at dev.mccue.example.Main.main(Main.java:9)Up to this point, everything has been more or less the same as it is for other "tree-based" JSON libraries like org.json or json-simple.
This is where that will start to change.
To take some Json and turn it into a user defined class, a basic approach would be to use instanceof checks to see if the Json is a particular subtype and navigate from there.
import dev.mccue.json.*;
record Muppet(String name, boolean canSpeak) {
static Muppet fromJson(Json json) {
if (json instanceof JsonObject object &&
object.get("name") instanceof JsonString name &&
object.get("canSpeak") instanceof JsonBoolean canSpeak) {
return new Muppet(name.toString(), canSpeak.value());
}
else {
throw new RuntimeException("Invalid Muppet");
}
}
}
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"name": "animal",
"canSpeak": false
}
""");
var animal = Muppet.fromJson(json);
System.out.println(animal);
}
}This process is "decoding." You "read" your data into JSON and then "decode" it to some type you define.
The problem with the instanceof approach is that you will end up with bad error messages on unexpected data. In this case the error message would just be "Invalid Muppet". The code to get better errors is tedious to write and I haven't seen many folks in the wild do it.
To get good errors, you should use the static methods defined in JsonDecoder.
package dev.mccue.example;
import dev.mccue.json.*;
record Muppet(String name, boolean canSpeak) {
static Muppet fromJson(Json json) {
return new Muppet(
JsonDecoder.field(
json,
"name",
JsonDecoder::string
),
JsonDecoder.field(
json,
"canSpeak",
JsonDecoder::boolean_
)
);
}
}
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"name": "animal",
"canSpeak": false
}
""");
var animal = Muppet.fromJson(json);
System.out.println(animal);
}
}These handle the fiddly process of checking whether the JSON matches the structure you expect and throwing an appropriate error.
You should read this declaration as "at the field name I expect a string."
JsonDecoder.field(json, "name", JsonDecoder::string)If the JSON is not an object, or doesn't have a value for name, or that value is not a string, you will get a JsonDecodeException.
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"canSpeak": false
}
""");
var animal = JsonDecoder.field(
json,
"name",
JsonDecoder::string
);
System.out.println(animal);
}
}Which will have a message indicating exactly what went wrong and where.
Problem with the value at json.name:
{
"canSpeak": false
}
no value for fieldThe last argument to JsonDecoder.field is the JsonDecoder you want to use to interpret the value at that field. In this case a method reference to JsonDecoder.string, which is a method that asserts JSON is a string and throws if it isn't.
For the methods which take more than one argument, there are overloads which can be used to get an instance of JsonDecoder.
// This will actually decode the json into a list of strings
List<String> items = JsonDecoder.array(json, JsonDecoder::string);
// This will just return a decoder
Decoder<List<String>> decoder =
JsonDecoder.array(JsonDecoder::string);This, in conjunction with JsonDecoder.field is how you are intended to explore nested paths.
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"villains": ["constantine", "doc hopper"]
}
""");
List<String> villains = JsonDecoder.field(
json,
"villains",
JsonDecoder.array(JsonDecoder::string)
);
System.out.println(villains);
}
}To decode JSON into your custom classes, you should add either a constructor or a static factory method which takes in Json and use these decoders to make your objects.
import dev.mccue.json.*;
import java.util.List;
record Actor(String name, String role, boolean muppet) {
static Actor fromJson(Json json) {
return new Actor(
JsonDecoder.field(json, "name", JsonDecoder::string),
JsonDecoder.field(json, "role", JsonDecoder::string),
JsonDecoder.optionalField(
json,
"muppet",
JsonDecoder::boolean_,
true
)
);
}
}
record Movie(String title, List<Actor> cast) {
static Movie fromJson(Json json) {
return new Movie(
JsonDecoder.field(json, "title", JsonDecoder::string),
JsonDecoder.field(
json,
"cast",
JsonDecoder.array(Actor::fromJson)
)
);
}
}
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"title": "Treasure Island",
"cast": [
{
"name": "Kermit",
"role": "The Captain"
},
{
"name": "Tim Curry",
"role": "Long John Silver",
"muppet": false
}
]
}
""");
var movie = Movie.fromJson(json);
System.out.println(movie);
}
}With all of that out of the way, here is how you might define a model, write it to json, and read it back in.
import dev.mccue.json.*;
import java.util.List;
record Actor(String name, String role, boolean muppet)
implements JsonEncodable {
static Actor fromJson(Json json) {
return new Actor(
JsonDecoder.field(json, "name", JsonDecoder::string),
JsonDecoder.field(json, "role", JsonDecoder::string),
JsonDecoder.optionalField(
json,
"muppet",
JsonDecoder::boolean_,
true)
);
}
@Override
public Json toJson() {
return Json.objectBuilder()
.put("name", name)
.put("role", role)
.put("muppet", muppet)
.build();
}
}
record Movie(String title, List<Actor> cast)
implements JsonEncodable {
static Movie fromJson(Json json) {
return new Movie(
JsonDecoder.field(json, "title", JsonDecoder::string),
JsonDecoder.field(
json,
"cast",
JsonDecoder.array(Actor::fromJson)
)
);
}
@Override
public Json toJson() {
return Json.objectBuilder()
.put("title", title)
.put("cast", cast)
.build();
}
}
public class Main {
public static void main(String[] args) {
var json = Json.readString("""
{
"title": "Treasure Island",
"cast": [
{
"name": "Kermit",
"role": "The Captain",
"muppet": true
},
{
"name": "Tim Curry",
"role": "Long John Silver",
"muppet": false
}
]
}
""");
var movie = Movie.fromJson(json);
var roundTrippedJson = Json.readString(
Json.writeString(movie.toJson())
);
var roundTrippedMovie = Movie.fromJson(roundTrippedJson);
System.out.println(
json.equals(roundTrippedJson)
);
System.out.println(
movie.equals(roundTrippedMovie)
);
}
}My hope is that at this point you have a sense of how it might look to use this library for your projects.
The rest of the post will just be some pitches to try to push you into the dark side.
Some people are perfectly fine with jackson-databind, gson, and other frameworks which use a class as a schema to read in JSON.
Others seem not to be. Kvetching about annotations and frameworks that make use of annotations is a common past-time in the community.
The current options for decoding without databind kinda suck though. To highlight this - I was talking with someone who takes the "magic bad" position. They said that generally they just use gson and manually construct their objects.
I challenged them to interpret this JSON into classes using their usual method.
{
"title": "Treasure Island",
"cast": [
{
"name": "kermit"
},
{
"name": "gonzo"
},
{
"name": "rizzo"
}
]
}And the following is the code they came up with.
package example.gson;
import com.google.gson.JsonObject;
public record Muppet(String name) {
public static Muppet createFrom(JsonObject muppetObject) {
String name = muppetObject
.get("name")
.getAsString();
return new Muppet(name);
}
}package example.gson;
import java.util.ArrayList;
import java.util.List;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
public record Movie(String title, List<Muppet> cast) {
public static Movie createFrom(JsonObject object) {
String muppetTitle = object
.get("title")
.getAsString();
List<Muppet> cast = new ArrayList<>();
JsonArray castArray = object.getAsJsonArray("cast");
for (int i = 0; i < castArray.size(); i++) {
JsonObject muppetObject = castArray
.get(i)
.getAsJsonObject();
cast.add(Muppet.createFrom(muppetObject));
}
return new Movie(muppetTitle, cast);
}
}package example.gson;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class Example {
public static void main(String[] args) {
String content = "{\r\n"
+ " \"title\": \"Treasure Island\",\r\n"
+ " \"cast\": [\r\n"
+ " {\r\n"
+ " \"name\": \"kermit\"\r\n"
+ " },\r\n"
+ " {\r\n"
+ " \"name\": \"gonzo\"\r\n"
+ " },\r\n"
+ " {\r\n"
+ " \"name\": \"rizzo\"\r\n"
+ " }\r\n"
+ " ]\r\n"
+ "}";
JsonObject json = JsonParser
.parseString(content)
.getAsJsonObject();
Movie movie = Movie.createFrom(json);
System.out.println(movie);
}
}I think this code is pretty representative of the variety one would produce when working against this sort of API.
The follow-up challenge I gave him was to run this code against some malformed input.
{
"title": "Treasure Island",
"cast": [
{
},
{
"name": "gonzo"
},
{
"name": "rizzo"
}
]
}The error message that his code produced was the following.
Cannot invoke "com.google.gson.JsonElement.getAsString()" because the return value of "com.google.gson.JsonObject.get(String)" is nullWhich, while better than it would have been in years past (thanks JEP 358), still isn't amazing.
Compare that to the error message someone will get with what is the most natural way to express this with my library.
record Muppet(String name) {
static Muppet fromJson(Json json) {
return new Muppet(
JsonDecoder.field(json, "name", JsonDecoder::string)
);
}
}
record Movie(String title, List<Muppet> cast) {
static Movie fromJson(Json json) {
return new Movie(
JsonDecoder.field(
json,
"title",
JsonDecoder::string
),
JsonDecoder.field(
json,
"cast",
JsonDecoder.array(Muppet::fromJson)
)
);
}
}Problem with the value at json.cast[0].name
{}
no value for fieldThe code they produced is also pretty heavily "imperative." To make their list of Muppets they have a plain for loop and transform every element individually.
List<Muppet> cast = new ArrayList<>();
JsonArray castArray = object.getAsJsonArray("cast");
for (int i = 0; i < castArray.size(); i++) {
JsonObject muppetObject = castArray
.get(i)
.getAsJsonObject();
cast.add(Muppet.createFrom(muppetObject));
}This is not intrinsically bad by any means, for loops are not evil, but all code lives somewhere on a spectrum from "declarative" to "imperative". Describing what should be done versus how it should be done.
If you compare their code to what one would write when relying on gson's reflection mechanisms the difference is stark.
record Muppet(String name) {}
record Movie(String title, List<Muppet> cast) {}Yes, you need to know the rules for how JSON is automatically mapped to these structures and what different annotations mean if they are present. But this is unquestionably a "declarative schema." If you know the rules it is easier to read.
The code you would write with my library occupies a middle ground.
record Muppet(String name) {
static Muppet fromJson(Json json) {
return new Muppet(
JsonDecoder.field(json, "name", JsonDecoder::string)
);
}
}
record Movie(String title, List<Muppet> cast) {
static Movie fromJson(Json json) {
return new Movie(
JsonDecoder.field(
json,
"title",
JsonDecoder::string
),
JsonDecoder.field(
json,
"cast",
JsonDecoder.array(Muppet::fromJson)
)
);
}
}While there is more of it than when you rely on heuristics, it is still "reasonably declarative."
return new Movie(
JsonDecoder.field(
json,
"title",
JsonDecoder::string
),
JsonDecoder.field(
json,
"cast",
JsonDecoder.array(Muppet::fromJson)
)
);The logic for it is both extensible (there is nothing privileged about JsonDecoder; you can write your own helper methods) and defined in code that you can click-to-definition to.
If a field can be null you would see nullableField. If a field can be missing, you would see optionalField. If it could be both, you would see optionalNullableField.
I don't know about you, but I am absolutely sick of explaining Jackson to students who are still struggling with classes in general.
If a student has JSON like this
[ {"name": "kailee"}, {"name": "fran"} ]Then to read it in as a List<Person> they need to either
Plus maybe some other options I might be unaware of.
To actually understand what they are doing for just that, they need to have a sense for some combination of
And that is really hard to impart at their stage, so often us online helpers just say "ah, make a class that looks like this." and send them on their way.
On the flip-side, when beginners get frustrated with a databind approach and fall back to something like org.json they seem to produce some absolute monstrosities before they come back with another question.
It's not their fault, they learned loops at most a semester ago, but it does present some practical problems.
The tension is between giving an option that there is enough time to teach the mechanics of and teaching an approach that will be ergonomic enough for them to complete their assignments.
I've been testing early drafts of this library against real students and, while there are too many confounding variables to say I've done any real science, I've found it to be far easier.
When a student needs a quick monkey-see-monkey-do, the JsonDecoder.field pattern seems to work just fine. When a student wants, needs, or has time for a full explanation there is a far shorter distance between where they are and where I need to get them.
I just need to make sure they understand interfaces and lambdas then they are ready for some version of the tutorial I gave in the first section.
Students in college aren't the only people who need to be taught how to work with JSON in Java either. If you work for a company that hires juniors or folks who come from different language backgrounds, then there has to be an education step.
It might be worth the boilerplate of writing out fromJson and toJson to have a codebase where onboarding doesn't need to touch the "advanced" side of Java to send JSON over the wire.
There has been a JEP open for years which proposes adding JSON to the standard library
If you use a build tool like maven for all of your programs, it might not seem important. You can pull in Jackson, gson, org.json, this, or any library with one declaration.
There are a few things which make me care about it though
reflection-config.json file it uses for native image, that will read and write JSON. I fear that will be too tempting a target for --add-opens.Regardless of if you agree that support should be in the standard library, I think the previous section illustrates some of the problems that could come if one of the existing APIs were adopted.
List<Muppet> cast = new ArrayList<>();
JsonArray castArray = object.getAsJsonArray("cast");
for (int i = 0; i < castArray.size(); i++) {
JsonObject muppetObject = castArray
.get(i)
.getAsJsonObject();
cast.add(Muppet.createFrom(muppetObject));
}Regardless of applicability, availability could end up making this the default. That is unideal.
An option to avoid this is for the standard library to add direct support for databind. Not only was that ruled out by the existing JEP, it would probably just be a bad idea. Mapping the JSON data model to the wide universe of Java objects has solutions that occupy a very large design space.
Considering the long term commitments the JDK makes whenever it adds a new feature as well as the mental, physical, and emotional damage dealt by its existing Serializable mechanism - I don't see that happening.
If the JDK gave up and just provided a low-level streaming parser akin to jackson-core, then it wouldn't affect the defaults in the ecosystem that much, but it would raise the question of "why not just use jackson-core." In addition, users would still have to add a library to accomplish most tasks. There wouldn't be much of a benefit.
So that's where this library comes in. It's nowhere near seaworthy for that ocean, but the JsonDecoder approach is relatively novel in the JVM ecosystem. The mechanisms needed to make it "work" have only been around since Java 8 and, as far as I know, haven't been tested on any large scale.
The more folks try it, or write libraries that do the concept "but better", or socialize it, etc. the more confidence there can be in whether a decoder based API would be applicable to the needs of the JDK.
You can see my recent conversation on the mailing list about this here.
Maybe I haven't convinced you to give it a try for your work or personal projects. That's fine.
Still, it is a small codebase. It makes (I think) good use of the features added to Java in the last decade. If you aren't caught up it might be a good reference point to do so.
If you want to play with upcoming features like value classes or string templates it could be a nice playground to see how that would affect performance, design, or just how the code feels.
In particular, JSON is mentioned as a use-case in the JEPs and explanations for new features often. Could be nice to have a JSON api to point to that actually works in the way being described.