Introduction to Contract-First REST Development
In the first post of this blog series, I define Contract-First REST Development and discuss benefits of this development style.
In this post, I provide an overview of contract-first development and reasons to follow this style of development. Specifically, I define contract-first REST development, tell the story of a real-life project that migrated to contract-first development, discuss some benefits of contract-first development, and then introduce three styles of generating code to complement contract-first development.
- What is contract-first development?
- A real project’s journey to contract-first development
- Reasons to follow contract-first development patterns
- Code generators
- Next Steps
What is contract-first development?
Contract-first development – also known as API-first development – refers to the practice of defining the interface of an API prior to writing the code that implements the API. Through collaboratively building and refining the API contract, service providers and service consumers can agree upon expected functionality and data models. This agreement facilitates confidence in the requirements for the API and creates efficiency by reducing rework that may result from requirements discovered during or after implementation.
Contract-first development can also apply to other types of services beyond REST. For example, WSDL files provide a contract for SOAP WebServices, and can be written before the implementing code. These other types of contracts are beyond the scope of this series; this series focuses solely on documents in the format of OpenAPI Specification version 3 (OAS3) as contracts for REST services.
A real project’s journey to contract-first development
The journey to and justification for contract-first development can be demonstrated with experience I had inheriting, rebuilding, and maintaining a project over the course of a couple of years.1
No contract at all
When I joined the project team, the application I inherited had a few layers to it. It was a standard 3-tier web application: SQL database, monolithic Java service, JSP-based web frontend. For browser performance reasons, some JSP had a handful of AJAX calls that went to somewhat REST-style endpoints.
With the REST endpoints intended only for team-internal usage, we had zero documentation. New team members needed to ask lots of questions and spend time reading the code in order to build understanding of the purpose and use of each endpoint.
Word-processed documentation
Shortly after I joined, other teams began asking to get direct access to the database so that they could adjust configuration of this application according to the needs of their applications. Of course, we steered these other teams toward making REST requests against our Java service instead. To facilitate use of our REST services, we wrote a word processor document that described URLs, paths, inputs, outputs, etc. Because several of the REST endpoints were intended for use only by the running web application, we documented only the subset of endpoints that were intended to be shared.
There were several issues with this approach: Being that we used a word processor to create a literal document, viewers needed to use the same word processing software to view the document. It also required manual effort to update and couldn’t be used for any kind of automated validation. In other words, it was impossible to use the document in tests to ensure that the services operated as described.
Similarly, it was impossible to generate API implementations from the contract that described the services. Generated API implementations would have reduced the possibility of deviation from the documentation. We often focused only on the code and left the documentation untouched until someone complained that the service did not act as described. In other words, the document became outdated very quickly.
As a final complaint, we stored this document on a Windows Share without version control, which posed its own set of problems as well. For example, the file became locked when another developer had opened the file for viewing, and we were unable to update it.
I often see similar problems occur with API documentation written into wiki systems. It is almost always a manual process to write. It gets outdated and left behind. (I’ve lost count of the number of times I’ve seen requests to update API docs in a wiki.) And it cannot be used for input into testing processes that validate the correct behavior of services.
Auto-generated contract (Code-first development)
The next significant step in this project was when I decided to start breaking the Java service down into smaller logical pieces. I made this decision for a few reasons: I wanted to facilitate scaling up parts of the system without scaling up everything. I also wanted to reduce the amount of regression testing required as code changes were released. (With changes isolated to smaller projects, the amount of code needing to be retested is reduced.)
In order to break the service into smaller pieces, I employed the strangler pattern. First, I worked with the team to select a set of functionality and build a minimally viable microservice. Each microservice took over handling of the selected functionality. Instead of calling code using Java method invocation, the legacy code was updated to initiate REST requests instead. (In some cases, the body of a method was instead replaced with code to make REST requests. This helped limit the range of impact to code as we broke the legacy monolith apart.)
As before, we didn’t initially plan for many of these microservices to be consumed outside the scope of this one application. Again, the main purpose of breaking the application down was to facilitate resource planning and horizontal scaling of the heavier parts of the system. Further, we already had an understanding of the API signature: we knew the exact inputs and outputs; all we needed to decide was the URL path and HTTP method. In the few places that the JSPs were already calling REST, we didn’t even need to make those path and method decisions; we just borrowed them from the existing service and moved the implementation to a smaller project.
During the process of breaking the project down into smaller pieces, I utilized in SpringFox to support documentation of the services. Using SpringFox and a set of annotations, each microservice auto-generated a Swagger contract based upon its own code.2
Swagger is sometimes also referred to as OpenAPI Specification version 2 (OASv2) because it was the direct predecessor to OpenAPI Specification version 3.
At the time, I was pleased with the way the annotations were used in the code and the documentation that it generated. It felt like a similar solution as Javadoc and Doxygen and, as a result, very familiar.
We initially didn’t realize much benefit from these auto-generated contracts as the consumer code was still manually written. The primary benefit that we received from the auto-generated contracts was in ensuring that the documentation was always up-to-date.
Code-first services, contract-first consumers
As we continued building out the microservices to support our application, we looked for a simplified way to enable ourselves and other developers to use the APIs. We initially considered building a separate project that contained consumer code to call our APIs. This would have allowed consuming projects to call the APIs by adding a Maven dependency. One of the reasons we opted to avoid that option is that it didn’t help the developers in our company that were writing .NET, Javascript, and iOS applications – and it seemed to have very limited benefit to Android applications.3
We decided to use swagger-codegen (which was later forked to OpenAPI Generator). By using swagger-codegen, we could generate client code for usage by our own Java backend and Javascript frontend. It was almost as simple to use this generated code as it would have been to use custom-written consumer code. In addition, we showed the .NET and iOS teams how to generate C# and Swift code from the contracts.
This resulted in a hybrid development pattern: We wrote the API provider code manually, and then generated OAS contracts from the code. Following that, we used the generated contracts to generate the consumer code. In other words, the service code was written code-first, and the consumer code was generated in a contract-first way.
Then the bridge to API-topia crumbled
Early on with using SpringFox to generate our API contracts, we ran into issues with some ways data structures were represented. We looked past it at first, figuring that the automated documentation was just that: documentation. We could add notes advising readers of the issues; for example, specifying that Spring’s Pageable object didn’t need to be somehow specified as an object in the HTTP request’s query parameters.4 (Instead, Pageable facilitates use of other query parameters such as page and size that Spring will build back up into a Pageable object.)
In addition to generation of awkward contracts, another issue we faced was with accidental contract changes. It happened occasionally that refactoring internal code caused a service’s generated API contract to change. Oops!
When we started generating code from the contracts, we couldn’t just write notes past these issues. We had to find a solution that would work better. The first thing we tried was to just mark certain objects as ignored. Obviously, this didn’t really help; it just hid the functionality from the contract and, as such, from any applications using generated consumer code. Of course, writing the consumer code manually still allowed use of the functionality, but it undermined the whole purpose of using code generators.
We thought about copying the auto-generated contracts, adjusting as appropriate, and then publishing the hybrid mostly-generated hand-fixed contracts. Nobody on our team wanted to own the tedious responsibility of combing and fixing the API contracts for every release. Understandable, so we dropped that idea.
So we started to adjust the APIs and data transfer models in our code. Because we controlled all the services and their functionality, we could refactor each to use data objects structured in a slightly different way. This led to a game of trial-and-error as we would first build a service and then spend several iterations trying to get the contract to generate just right.
Fully contract-first service and consumer development
At this point in the project, we were following these steps for each service:
- Build the service,
- Run it locally,
- Inspect the generated contract,
- Tear down the service,
- Adjust the code,
- Repeat until happy with the contract.
Needless to say, we were wasting a lot of the time that we thought we were saving with contract generation.
With everybody on the team knowing generally what we wanted the contracts to look like, it made little sense to continue fighting with SpringFox. SpringFox was a great tool to help the team start building familiarity with OAS. It was now time to move past it.
Seeing that swagger-codegen also supported generation of server-side code for Spring Boot, it struck me: We’re already starting to see success in generating the API consumer code, why not also generate the boilerplate for API provider code? With that, I decided to bring contract-first to our service development as well. From that point, we started to design and write the API contracts first, and then generate the project boilerplate from those contracts.
The solution to generate the boilerplate was a definite improvement. We spent less time fighting with the tooling, more time discussing and designing the API contract, and we felt better about the quality of our services. As with most parts of life, there were still opportunities for improvement:
- The code that we generated from the API contract was generated one-time only, which caused problems when we needed to make changes to the contracts, and
- We still missed the opportunity to use the API contract for mocking and validation.
While I have since left the project that I described here, I will demonstrate – through this article series – techniques that can be used on any project to improve the contract-first experience even beyond where I left this project.
Reasons to follow contract-first development patterns
As demonstrated in the journey described above, there are benefits to following contract-first styles for developers of both REST providers and REST consumers.
Benefits for the API provider / server-side development
On the server side, building a service from an existing API contract may feel backwards, like putting the cart before the horse. Contrary to that feeling, it actually helps drive improved code quality with a number of benefits:
- Contract-first development works to facilitate building services that focus on satisfying business requirements. Use of code-generation tools reduces the amount of boilerplate code that developers have to write, and ensure a consistent, usable interface that follows the contract. This allows the developers to focus their hand-written code on solving actual business problems.
- Contract-first development also works to avoid data constructs specific to any particular programming language. Instead of the contract becoming an afterthought to the easiest way to describe an object in a specific programming language, the contract is instead created outside the confines of any implementation. In other words, the usability of the service is protected against strange data structures that appear when generating the contract from code in a code-first way.
- The tools used with contract-first development help ensure that the interface does not change as the code is maintained except when the change is intentional. In other words, developers are more free to refactor the service code without worry that the API interface would be changed by the refactor. This avoids one of the ways requests might be disrupted: For example, a request won’t suddenly start failing because a code refactor accidentally changed a query parameter name. (Changing a query parameter name would require explicit change to the contract.)
- Beyond the service implementation, having a defined contract also facilitates provisioning and configuration of API management platforms such as Red Hat 3scale API Management5.
Benefits for the API consumer / client-side development
For developers that are building applications to initiate requests and consume a service, having the contract available simplifies in a few ways:
- Developers can use tooling to rapidly create service mocks. These service mocks provide an environment for testing consumer code even before the actual service implementation is available. In other words, developers can begin building API consumer applications at the same time that the developers building the API provider begin their work.
- Beyond exploratory testing and one-off validation performed during development, service mocks can also be useful for providing and controlling test environments. These test environments can be controlled to respond in defined ways to prove the behavior of consuming applications for integration tests. This is useful even in situations where the concrete service implementation is already available.
- In addition to the creation of service mocks, consumer code-generation can be run based upon a contract. This type of code-generation allows for the reduction of boilerplate code: By generating client code, developers of consumer applications need not worry about correctness of hand-written HTTP request code. Further, this generated code facilitates more readable business code that is focused on intent rather than implementation: By maintaining a clean separation of generated code and hand-written code, the hand-written code is easier to find and follow.
Code generators
As identified above, one of the significant benefits of defining the contract first is in the ability to then reduce boilerplate code through code generation. This benefit exists on both sides of the request: API provider and API consumer.
Many tools offer this capability in at least one of three ways: one-time code generation, build-time code generation, and runtime code generation. The obvious difference between each method is the time at which the contract is read, parsed, and acted upon. More subtle differences come with the ease of maintenance, repeatability, and performance of each strategy.
Through the rest of this series, code generators will be used to facilitate building code based upon OAS3 contracts. For reasons described below, I heavily favor build-time code generation when possible.
One-time generation
Many code-generation projects start by offering one-time generation that bootstraps a project. The expectation is that the developer will then complete the implementation to match business requirements. Some prominent examples of one-time code generation include Maven Archetypes and Spring Initializr. In the realm of REST and OpenAPI Spec, OpenAPI Generator offers one-time generation options.
One-time generation can be useful to start new projects, especially when unfamiliar with the language or libraries that the project will use. Unfortunately, there are several drawbacks when using one-time generation.
The biggest problem with one-time code generation is that no human working on the project has actually written any of the code. That simple fact reduces the likelihood that any developer on the project fully understands the generated code in the codebase. Because of the lower chance of understanding it, the one-time generated code is also less likely to receive the same maintenance and care as code that was written by hand.
A similarly important problem is that the code generated by one-time generation has no clear path to automated updates. Any modifications applied to the code will need to be redone after any subsequent generation run, which greatly complicates automated update processes. This additional complexity leads to difficulty in setting up and maintaining an update process.
The above two reasons show that the code is unlikely to be maintained manually and even less likely to be updated automatically, which means that the one-time generated code becomes stale and outdated almost as soon as the generator has finished running.
Beyond becoming outdated, the code is usually generated and stored alongside human-written code. Even with annotations marking generated code, it becomes more difficult to quickly glance and identify which pieces of code came from a human and which pieces came from a generator. This undermines the benefit of focusing human-written code on solving the actual business problem. Instead of removing and reducing the boilerplate code held within a codebase, one-time generation will just add it instead.
It is for these reasons that I almost never recommend one-time code generation; instead, I often advise against it. Of course, there are exceptions such as building your own custom Maven archetype for self usage.
Build-time generation
Many of the pain-points in using one-time code generation can be resolved by simply running code generation more often. For example: by running code generation as part of the build process, the code will always use the latest templates available to the generator.
As an example, the OpenAPI Generator project mentioned in “One-time generation” also supports build-time generation. In order to facilitate build-time generation in Java projects, OpenAPI Generator uses a delegate pattern: instead of leaving generated stub methods, it will generate a delegate interface and make calls to that. Developers using this generated code only need to implement that delegate interface in their own class. As such, use of the delegate interface leaves a clean separation between generated code and human-written code.
With the clean separation between human and generated code, all the build-time generated code can be held in a separate source root, which allows placement alongside other compiler output artifacts. In other words, the source repository remains free of generated code and contains only code written by humans.
One drawback of using build-time generation is that it can increase the duration of the build for each and every build. In most cases, this increased build duration is so minimal that it seldom becomes noticeable.
Regardless of any increased build time, I always look for ways that I can generate code during the build. The extra time spent waiting on a build is a small price to pay in exchange for saving time that would have been spent digging through or trying to maintain one-time generated code.
Start-up / runtime generation
A third way to run code generation processes is at runtime. Highly dynamic frameworks and libraries make use of runtime code generation for setting up, configuring, and improving the running application. For example, the Spring family of libraries perform many code-generation tasks at start up, including generation of proxy classes for Aspect-Oriented Programming (AOP), concrete implementations of Spring Data repository interfaces, etc. Even the Java HotSpot JVM uses runtime code generation: As the bytecode interpreter identifies highly used code, it will compile that bytecode to native machine code while the process is running.
As mentioned, runtime generation allows very dynamic behaviors, but it comes at a cost: the startup and operation of an application requires additional CPU cycles for the code generation. Sometimes, this means slowing down the application.
Projects such as Quarkus aim to increase start-up and runtime speed of Java applications by shifting as much application initialization processing into the build phase as possible. In other words, by spending more time during the build process, applications will receive the benefit of a reduced start-up delay before becoming ready. Of course, this is not the only benefit of Quarkus. For more information, see the Quarkus homepage.
In the context of REST services, runtime generation can be useful when consuming APIs. By generating the request code at runtime, an API consumer can ensure that it is always using the latest API contract. This comes at a cost too: updates to the API contract may cause a consuming application to fail during start-up. It also becomes more difficult to control and run in isolation for purposes such as testing. Another way to view this problem is that the runtime code generation may result in production code that is untested if the generation input (i.e., the API contract) had not been tested with the application in lower environments.
Due to the performance implications and the possibility for untested and possibly breaking changes in production, I suggest using runtime code generation only when build-time generation is impossible or impractical.
Next Steps
In this post, I defined contract-first development and discussed benefits of following this style of development. Building from this foundation, the next post will cover the creation of an OAS3 contract for use by subsequent posts that guide through implementation of services and consumers across a few different technologies.
Because I intend to not distract with the actual business problem that was solved, I am intentionally omitting details about the application, business requirements, and its functionality. ↩︎
Code-first contract generators have seemed to gain a lot of traction over the past several years. While I advocate for contract-first development patterns, I also recognize that not everybody is ready to jump fully into contract-first. Some useful libraries for code-first development include SpringFox or springdoc-openapi for Spring-based projects, and Swashbuckle for .NET projects. Many other languages have similar generators as well. Note that these libraries will not be covered in this article series. ↩︎
By this point, I was leading efforts to build native mobile apps for both Android and iOS in addition to my efforts for the service that I’m writing about. Beyond supporting other teams, I was looking to reduce my own burden. I figured that if I could find a way to generate client code, I wouldn’t have to write redundant code across multiple languages. ↩︎
More information about this conflict between SpringFox and Spring’s Pageable can be found in SpringFox’s issue #755 on GitHub. ↩︎
The services I provide through Red Hat include helping customers understand, implement, and thrive with Red Hat 3scale. The 3scale-related topics that I consult on include automation of service configuration, API security, API request/response transformation, high availability and disaster recovery planning, and more. ↩︎
