Use all the Databases - Part 1
PublishedLoren Sands-Ramshaw, author of GraphQL: The New REST, shows how to combine data from multiple sources using GraphQL in this Write Stuff two-part series.
Ever wanted to use a few different databases to build your app? Different types of databases are meant for different purposes, so it often makes sense to combine them. You might be hesitant due to the complexity of maintenance and coding, but it can be easy if you combine Compose and GraphQL: instead of writing a number of complex REST endpoints, each querying multiple databases, you set up a single GraphQL endpoint that provides whatever data the client wants using your simple data fetching functions.
This tutorial is meant for anyone who provides or fetches data, whether it’s a backend dev writing an API (in any language) or a frontend web or mobile dev fetching data from the server. We’ll learn about the GraphQL specification, set up a GraphQL server, and fetch data from five different data sources. The code is in Javascript, but you’ll still get a good idea of GraphQL without knowing the language.
In this first part, we'll look at the databases that will be involved. Then we'll introduce GraphQL before moving on to the query we want to make, the schema we need to create and how to setup the server to make that all happen.
In part two, we'll look at resolving queries on SQL, Elasticsearch, MongoDB, Redis and REST data sources and a look at how to get the best performance before calling things done.
Part 1
Part 2
The databases
We at Chirper Fictional, Inc. were building a Twitter clone, and decided to use these databases:
💾 PostgreSQL: Because like most apps, our data was relational, and our boss said that the database we wanted to use (RethinkDB) was too new to be trusted 😔.
💾 Redis: We wanted to cache frequently-used data, like the public feed, so we could get it quickly and reduce the read load on Postgres.
💾 Elasticsearch: A database built for searching that would function better and scale better than searching Postgres.
💾 REST: We wanted to show our users tweets from their area, and we didn't want to prompt for GPS permissions or pay for a MaxMind IP address database, so we found a REST API for geolocating IP addresses.
💾 MongoDB: We wanted to track some user stats, and we didn't need them to be in the main app database. We put the intern on this, and while he could have just used a second Postgres DB, he used Mongo because he heard it was Web Scale. And we didn't mind because we didn't need ACID or JOINs for our stats.
Now we need a way to combine the data from all of these sources together in whichever ways our clients want it, and the best way to do this is with GraphQL.
GraphQL intro
Gotta be honest here... for the first few months of GraphQL's short existence (it launched in July 2015), I thought GraphQL was a query language for accessing your Facebook friend graph 😳. Turns out that’s FQL, and GraphQL is a replacement for REST! And sorry REST, but GraphQL is kinda better than you for most things 😁. Here's why:
- ✅ Easier to consume: The GraphQL client's job is super simple—just write the data fields you want filled in. When you send the query string like the one on the left side of the image, you get back the JSON response on the right, with the same structure you asked for. Instead of sending multiple REST requests (sometimes multiple round trips in series), you can send a single GraphQL request. And instead of getting more or less data than you need from the REST endpoints, you get exactly the data you ask for.
- ✅ Easier to produce: On the GraphQL server you write resolvers—functions that resolve a field to its value; for instance for the above, there's a
user()
function that responds to theuser(id: 1)
query and returns user #1's SQL record. One nice thing is that they work at any place in the query—looking up the current user's first name (user.firstName
is"Maurine"
in the above example) at the top level runs the same code as looking up the author of a tweet that mentions her name (user.mentions[0].author.firstName
happens to also be"Maurine"
), nested in the query heirarchy (more info on this). Also, sometimes with REST you have endpoints talking to multiple databases. A GraphQL server is more organized, since in most cases each resolver talks to a single data source.
Credit: Jonas Helfer
✅ Types and introspection: Each query has a typed schema (
User
,Tweet
,String
,Int
, etc). At first it may seem like extra work, but it means that you get better error messages, query linting, and automatic server response mocking. It also has introspection—a standard method of querying the server to ask what queries it supports (and their schemas)—which is what powers GraphiQL (with an i and pronounced, “graphical”), the in-browser auto-documented GraphQL IDE described later in this article.✅ Version free: Because the client decides what data it wants, you can easily support many different client versions. Instead of versioning your endpoints (eg GET /api/v2/user), when you add new features, you simply add more fields. When you sunset old features, the associated fields can be deprecated but continue to function.
Fear not—you don't need to rewrite all your REST servers: you can instead add a simple GraphQL server in front of them, as we'll see with the REST data source example below.
Note: you can of course also change data with GraphQL (with functions called mutations), but I won't be covering that in this post.
The query
Let's figure out the query that we'll need for our app's home dashboard. First, here are the things we'd like to display:
- Your name and photo (SQL)
- Recent tweets that mention your name (Elasticsearch)
- Most recent few tweets worldwide (Redis)
- Recent tweets in your city (REST to geolocate and then SQL)
- For each tweet, the number of times it has been viewed (Mongo)
For each tweet, we'll want to display the text of the tweet, the author's name and photo, and when it was created. For the mentions and city feeds, we also want the number of times the tweets were viewed and from what city they were made.
Now to make the query, we write out the pieces of data we need in order to display the above list, choosing names for each field and putting it in a JSON-like format! 😄
const queryString = `
{
user(id: 1) {
firstName
lastName
photo
mentions {
text
author {
firstName
lastName
photo
}
city
views
created
}
}
publicFeed {
text
author {
firstName
lastName
photo
}
created
}
cityFeed {
text
author {
firstName
lastName
photo
}
city
views
created
}
}
`
We'll put mentions
as a field of the user
query instead of at the top level because we'll need to the user's name in order to query Elasticsearch, and we'll have their name from the first step of the user
query (we'll see how this looks when we implement it).
Parentheses are used to pass arguments—for simplicity's sake, we're passing our own user id with (id: 1)
. Usually when fetching the current user’s data, instead of passing your user id as an argument, you'd put your auth token in the Authorization
header, and the server would authenticate you. This is done automatically for you by frameworks like Meteor.
Our query should return the below JSON data. The data mirrors the query format, with values filled in, sometimes with arrays of objects:
{
"data": {
"user": {
"firstName": "Maurine",
"lastName": "Rau",
"photo": "http://placekitten.com/200/139",
"mentions": [
{
"text": "Maurine Rau Eligendi in deserunt.",
"author": {
"firstName": "Maurine",
"lastName": "Rau",
"photo": "http://placekitten.com/200/139"
},
"city": "San Francisco",
"views": 82,
"created": 1481757217713
}
]
},
"publicFeed": [
{
"text": "Corporis qui impedit cupiditate rerum magnam nisi velit aliquam.",
"author": {
"firstName": "Tia",
"lastName": "Berge",
"photo": "http://placekitten.com/200/139"
},
"city": "New York",
"views": 91,
"created": 1481757215183
},
...
],
"cityFeed": [
{
"text": "Edmond Jones Harum ullam pariatur quos est quod.",
"author": {
"firstName": "Edmond",
"lastName": "Jones",
"photo": "http://placekitten.com/200/139"
},
"city": "Mountain View",
"views": 69,
"created": 1481757216723
},
...
]
}
}
Now let's write the simple GraphQL server that will return that data!
The schema
The first thing your server needs is a schema. This is what the server will use to provide type safety and power the introspection and improved error messages. Since we've already written out what we'd like our queries to look like, this will be easy - we just need to list out the fields and their types. First, under type Query
, we list the possible queries (top-level attributes in our query string):
type Query {
user(id: Int!): User
# A feed of the most recent tweets worldwide
publicFeed: [Tweet]
# A feed of the most recent tweets in your city
cityFeed: [Tweet]
}
Each query is followed by the type that is returned. Besides the basic types (String
, Int
, Float
, Boolean
), you can make your own types, which start with a capital letter. So the first line reads, "One possible query is the user
query, which takes one required argument (the exclamation point in Int!
means required) named id
of type Int
and which returns something of type User
." The last line reads, "One possible query is the cityFeed
query, which has no arguments and returns an array of Tweet
s." The #
comments are descriptions, which show up in the GraphiQL IDE described later.
Now to define the User
and Tweet
types, we'll list the fields we chose in our query string:
type User {
firstName: String
lastName: String
photo: String
mentions: [Tweet]
}
type Tweet {
text: String
author: User
city: String
views: Int
created: Float
}
That's our schema! The schema goes into a string:
// data/schema.js
const schema = `
type User { ...
type Tweet { ...
type Query { ...
schema {
query: Query
}
`;
export default schema;
Server setup
The reference implementation of the GraphQL specification is GraphQL-JS
, and it's used by graphql-server-express
, a GraphQL middleware for Express, the most popular Node.js web server. Here's how we set it up:
// server.js:
import express from 'express';
import { graphqlExpress, graphiqlExpress } from 'graphql-server-express';
import { makeExecutableSchema } from 'graphql-tools';
import bodyParser from 'body-parser';
import schema from './data/schema';
import resolvers from './data/resolvers';
const graphQLServer = express();
const executableSchema = makeExecutableSchema({
typeDefs: [schema],
resolvers,
});
graphQLServer.use('/graphql', bodyParser.json(), graphqlExpress({
schema: executableSchema,
}));
graphQLServer.use('/graphiql', graphiqlExpress({
endpointURL: '/graphql',
}));
const GRAPHQL_PORT = 8080;
graphQLServer.listen(GRAPHQL_PORT, () => console.log(
`GraphQL Server is now running on http://localhost:${GRAPHQL_PORT}`
));
import schema from './data/schema'
– the GraphQL schema that we wrote in the last sectionimport resolvers from './data/resolvers'
– an object with our resolve functions, which will do the DB lookups (we'll do this in the next article)/graphiql
– GraphiQL, the IDE for GraphQL. If you visit this URL (for us it’s http://localhost:8080/graphiql in a browser, you'll see the UI shown in the first screenshot.- While you're typing the query string in the left side of the screen, it autocompletes query fields.
- When you hit the run button or
cmd-return
, the response from the server is shown on the right. - There's also a docs sidebar that has automatic documentation of the available queries and data fields.
If you’d like to run this server on your computer, first follow the repo’s setup instructions. Now you can start the server by running server.js
:
nodemon ./server.js --exec babel-node
And make queries in GraphiQL:
http://localhost:8080/graphiql
When you edit the code, the server will restart itself, and you can re-run your query in GraphiQL. Reload the page in order to get the docs and autocompletion to update.
You can try out the GraphiQL of the finished Twitter clone server (powered by Compose!) here:
all-the-databases.graphql.guide/graphql
The only differences between the hosted server and the code running on your own computer are environment variables that contain the database connection info that you get when you set up a new Compose database.
We now have a working server and schema. The server setup was short, and specifying types for the schema was intuitive, but we haven’t done anything database-specific yet. In Part 2 we’ll write the server code that fetches the right data from SQL, Elasticsearch, MongoDB, Redis, and a REST API.s. Add Compose Articles to your feed reader to get the next part!
attributionHyberbole and a half