Everything that it's possible to do in the CloudFlare Dashboard is also possible through our RESTful API. We use the same API to power the dashboard itself.
In order to keep track of all our endpoints, we use a rich notation called JSON Hyper-Schema. These schemas are used to generate the complete HTML documentation that you can see at https://api.cloudflare.com. Today, we want to share a set of tools that we use in this process.
CC BY 2.0 image by Richard Martin
JSON Schema
JSON Schema is a powerful way to describe your JSON data format. It provides complete structural validation and can be used for things like validation of incoming requests. JSON Hyper-Schema further extends this format with links and gives you a way describe your API.
JSON Schema Example
{
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "number" },
"address": {
"type": "object",
"properties": {
"street_address": { "type": "string" },
"city": { "type": "string" },
"state": { "type": "string" },
"country": { "type" : "string" }
}
}
}
}Matching JSON
{
"name": "John Doe",
"age": 45,
"address": {
"street_address": "12433 State St NW",
"city": "Atlanta",
"state": "Georgia",
"country": "United States"
}
}JSON Schema supports all simple data types. It also defines some special meta properties including title, description, default, enum, id, $ref, $schema, allOf, anyOf, oneOf, and more. The most powerful construct is $ref. It provides similar functionality to hypertext links. You can reference external schemas (external reference) or a fragment inside the current schema (internal reference). This way you can easily compose and combine multiple schemas together without repeating yourself.
JSON Hyper-Schema introduces another property called links where you define your API links, methods, request and response formats, etc. The best way to learn more about JSON Schemas is to visit Understanding JSON Schema. You can also visit the official specification website or wiki. If you want to jump straight into examples, try this.
CC BY 2.0 image by Tony Walmsley
Generating Documentation: Tools
We already have an open source library that can generate complete HTML documentation from JSON Schema files and Handlebars.js templates. It's called JSON Schema Docs Generator (JSDC). However, it has some drawbacks that make it hard to use for other teams:
Complicated configuration
It's necessary to rebuild everything with every change (slow)
Templates cannot have their own dependencies
All additional scripting must be in a different place
It is hard to further customize it (splitting into sections, pages)
We wanted something more modular and extensible that addresses the above issues, while still getting ready-to-go output just with a few commands. So, we created a toolchain based on JSDC and modern JavaScript libraries. This article is not just a description for how to use these tools, but also an explanation of our design decisions. It is described in a bottom-up manner. You can skip to the bottom if you are not interested in the technical discussion and just want to get started using the tools.
json-schema-loader
JSON Schema files need to be preprocessed first. The first thing we have to do is to resolve their references ($ref). This can be quite a complex task since every schema can have multiple references, some of which are external (referencing even more schemas). Also, when we make a change, we want to only resolve schemas that need to be resolved. We decided to use Webpack for this task because a webpack loader has some great properties:
It's a simple function that transforms input into output
It can maintain and track additional file dependencies
It can cache the output
It can be chained
Webpack watches all changes in required modules and their dependencies
Our loader uses the 3rd party JSON Schema Ref Parser library. It does not adhere to the JSON Schema specification related to id properties and their ability to change reference scope since it is ambiguous. However, it does implement the JSON Pointer and JSON Reference specifications. What does this mean? You can still combine relative (or absolute) paths with JSON Pointers and use references like:
"$ref": "./product.json#/definitions/identifier"but ids are simply ignored and the scope is always relative to the root. That makes reasoning about our schemas easier. That being said, a unique root id is still expected for other purposes.
json-schema-example-loader
Finally, we have resolved schemas. Unfortunately, their structure doesn't really match our final HTML documentation. It can be deeply nested, and we want to present our users with nice examples of API requests and responses. We need to do further transformations. We must remove some original properties and precompute new ones. The goal is to create a data structure that will better fit our UI components. Please check out the project page for more details.
You might be asking why we use another webpack loader and why this isn't part of our web application instead. The main reason is performance. We do not want to bog down browsers by doing these transformations repeatedly since JSON Schemas can be arbitrarily nested and very complex and the output can be precomputed.
doca-bootstrap-theme
With both of these webpack loaders, you can easily use your favorite JavaScript framework to build your own application. However, we want to make doc generation accessible even to people who don't have time to build their own app. So, we created a set of templates that match the output of json-schema-example-loader. These templates use the popular library React. Why React?
It can be used and rendered server-side
We can now bake additional features into components (e.g., show/hide...)
It is easily composable
We really really like it :)
doca-bootstrap-theme is a generic theme based on Twitter Bootstrap v3. We also have our private doca-cf-theme used by https://api.cloudflare.com. We encourage you to fork it and create your own awesome themes!
CC BY 2.0 image by Maia Coimbra
doca
So, we have loaders and nice UI components. Now, it's time to put it all together. We have something that can do just that! We call it doca. doca is a command-line tool written in Node.js that scaffolds the whole application for you. It is actually pretty simple. It takes fine-tuned webpack/redux/babel based application, copies it into a destination of your choice, and does a few simple replacements.
Since all hard work is done by webpack loaders and all UI components live in a different theme package, the final app can be pretty minimal. It's not intended to be updated by the doca tool. You should only use doca once. Otherwise, it would just rewrite your application, which is not desirable if you made some custom modifications. For example, you might want to add React Router to create multi-page documentation.
doca contains webpack configs for development and production modes. You can build a completely static version with no JavaScript. It transforms the output of json-schema-example-loader into an immutable data structure (using Immutable.js). This brings some nice performance optimizations. This immutable structure is then passed to doca-bootstrap-theme (the default option). That's it.
This is a good compromise between ease of setup and future customization. Do you have a folder with JSON Schema files and want to quickly get index.html? Install doca and use a few commands. Do you need your own look? Fork and update doca-bootstrap-theme. Do you need to create more pages, sections, or use a different framework? Just modify the app that was scaffolded by doca.
One of the coolest features of webpack is hot module replacement. Once you save a file, you can immediately see the result in your browser. No waiting, refreshing, scrolling or lost state. It's mostly used in combination with React; however, we use it for JSON Schemas, too. Here's a demo:
It gets even better. It is easy to make a mistake in your schemas. No worries! You will be immediately prompted with a descriptive error message. Once it's fixed, you can continue with your work. No need to leave your editor. Refreshing is so yesterday!
Generating Documentation: Usage
The only prerequisite is to have Node.js v4+ on your system. Then, you can install doca with:
npm install doca -gThere are just two simple commands. The first one is doca init:
doca init [-i schema_folder] [-o project_folder] [-t theme_name]It goes through the current dir (or schema_folder), looks for **/*.json files, and generates /documentation (or /project_folder). This command should be used only once when you need to bootstrap your project.
The second one is doca theme:
doca theme newTheme projectThis gives a different theme (newTheme) to the project. It has two steps:
It calls
npm install newTheme --saveinside ofprojectIt renames all
doca-xxx-themereferences todoca-newTheme-theme
This can make destructive changes in your project. Always use version control!
CC BY 2.0 image by Robert Couse-Baker
Getting started
The best way how to start is to try our example. It includes two JSON Schemas.
git clone git@github.com:cloudflare/doca.git
cd doca/example
doca init
cd documentation
npm install
npm start
open http://localhost:8000That's it! This results in a development environment where you can make quick changes in your schemas and see the effects immediately because of mighty hot reloading.
You can build a static production ready app with:
npm run build
open build/index.htmlOr you can build it with no JavaScript using:
npm run build:nojs
open build/index.htmlDo you need to add more schemas or change their order? Edit the file /schema.js.Do you want to change the generic page title or make curl examples nicer? Edit the file /config.js.
Conclusion
We're open sourcing a set of libraries that can help you develop and ship a rich RESTful API documentation. We are happy for any feedback and can't wait to see new themes created by the open source community. Please, gives us a star on GitHub. Also, if this work interests you, then you should come join our team!