BLOG

December 30th, 2017

Creating a Node.js and MongoDB REST API Prototype

MongoDB

Node.js

Express

JavaScript

ECMAScript 6

NoSQL

Document Database

Mongoose

Babel

Gulp

REST

API

Basic Auth

Most of my discoveries lately have been about JavaScript. This includes learning the new ES6 syntax and features along with all the quirks that come with the JavaScript language. Recently I have been exploring the MongoDB database and Node.js environment. MongoDB is a NoSQL database that allows you to store JSON formatted documents in a schema-less model. I wrote four discovery posts on MongoDB starting with the basic features of the database . While I haven't written any discoveries with Node.js as the main focus, I have experimented with it when testing out ES6 modules with Babel and Async Functions.

All of this knowledge buildup is for a personal website project that I have planned (and where this blog will call home!). Before I start development on the website directly, I will create a series of prototypes to get a feel for some of the technologies I will use in my websites stack. With these prototypes I can make sure the technology I choose is a good fit for the full project. Also I can use them as templates to complete future discoveries! In general if you have the time to build prototypes with technologies you want to use in a production project it is a great idea!

My planned production web stack will be either MEAN (MongoDB, Express, Angular, Node.js) or MERN (MongoDB, Express, React.js, Node.js). Both of these web stacks share three technologies - MongoDB, Express, and Node.js. I used all three of these technologies to build this prototype!

The prototype is a REST API that lets users look at songs and artists. They can also comment on songs. MongoDB stores this information in a database called music_api. Databases in MongoDB are simply namespaces, which is much different than their Relational Database equivalents. In a RDBMS you need a username and password among other items to connect to a certain database. MongoDB does not have this requirement.

The data is then stored in collections of documents including songs, artists, and end users.

The Node.js run-time environment and Express web application framework then expose a REST API to users so they can query, create, update, and delete items in the database (all CRUD operations).

For the prototype most of the functionality surrounds viewing and manipulating songs, however in a full application these uses would be expanded. This could include creating and updating users, rating songs, user-to-user interactivity, etc. Now let's look at some insteresting aspects of the prototype and certain challenges that I faced while creating it.

For interacting with the MongoDB database from Node.js I chose to use a module called Mongoose. Mongoose allows you to model database JSON objects and perform all MongoDB queries, inserts, and updates. With MongoDB you don't really need to use a ORM (Object Relational Mapping) framework since all the data is already in JSON form which any programming language can deal with. However, Mongoose gives us some additional capabilities which make it desirable.

First off Mongoose allows you to set strict schema rules for your MongoDB objects. As previously mentioned MongoDB is schema-less, so this stricter model can help restrict what exactly a user can place in a document. You can also create nested schemas for complex JSON documents. For example, my song collection schema contains a nested schema for a list of comments.

const Schema = mongoose.Schema; const SongSchema = new Schema({ title: { type: String, trim: true, required: true }, album: { type: String, trim: true }, artist: { type: String, trim: true, required: true }, artist_id: Schema.Types.ObjectId, type: { type: String, enum: ['a', 'j', 'aj'] }, release_date: { type: Date, default: Date.now() }, best_lyric: String, comments: [{ type: CommentSchema }] }, {usePushEach: true});

Each Schema() constructor function takes a JSON object that represents all the properties of the MongoDB document. You can also add additional validations, such as making certain fields required (required: true) or having a default value (default: Date.now())1. When a user uploads a JSON object to a schema and tries updating the database, only the properties seen in the schema will be persisted to MongoDB.

Also there is a comments property which takes an array of type CommentSchema. This CommentSchema can be defined similarly to how I implemented SongSchema.

const CommentSchema = new Schema({ username: String, user_id: Schema.Types.ObjectId, date: { type: Date, default: Date.now() }, content: { type: String, trim: true } }, {usePushEach: true, _id : false});

Another powerful feature of Mongoose is we can define database indexes directly on the Schema object. With advanced features such as this, Mongoose has transformed into something much more powerful than a ORM. With Mongoose I never used the MongoDB CLI since everything I needed could be performed through Mongoose. Here is an index I made on the name property in the artist schema.

ArtistSchema.index({name: 1});

Mongoose allows you to perform any query, update, insert, or delete operation on a Schema. By default Mongoose uses callbacks to perform these operations. We want to avoid callbacks as they become ugly and hard to read once nested database calls are made (commonly known as callback hell). Luckily Mongoose allows us to use ES6 Promises instead2. One of the first things I did when creating my API calls was replace all the callbacks with Promises. Here is some code that performs a find operation on the Song schema and then returns it as a HTTP response using a promise.

Song.find().exec() .then((songs) => { res.format({ 'application/json': () => { res.json(songs); }, 'application/xml': () => { res.render('xml/songs', {songs: songs}); } }); }) .catch((err) => { console.error(err); res.status(500).send(err); });

You may have noticed in the last code sample the res.format() function which returns the song data as an HTTP response. This function also returns either JSON or XML depending on the MIME type specified in the HTTP requests Accept header. Normally I don't think the amount of development work needed to return both notations would be worth it but Express made it so easy to implement3!

There are two ways to implement the conversion to XML. One option is to perform it in JavaScript itself4.

'application/xml': () => { res.write('<songs>\n'); songs.forEach((song) => { let comments = ""; song.comments.forEach((comment) => { comments = ` ${comments} <comment> <username>${comment.username}</username> <date>${comment.date}</date> <content>${comment.content}</content> </comment> `; }); res.write(` <entry> <title>${song.title}</title> <artist>${song.artist}</artist> <album>${song.album}</album> <type>${song.type}</type> <release_date>${song.release_date}</release_date> <best_lyric>${song.best_lyric}</best_lyric> <comments>${comments}</comments> </entry> `); }); res.end('</songs>'); }

This is a bit messy as you are mixing JavaScript and a markup language together. A more elegant solution uses a tempting language to generate the XML. The template language I used is EJS (Embedded JavaScript Templating) which mixes JavaScript with HTML similarly to JSP for Java or PHP (which I used in my saintsxctf.com website)5. I used this template to create the markup on the server before sending it to the client.

<?xml version="1.0" encoding="UTF-8"?> <songs> <% for (var i = 0; i < songs.length; i++) { %> <song> <_id><%= songs[i]._id %></_id> <title><%= songs[i].title %></title> <artist><%= songs[i].artist %></artist> <album><%= songs[i].album %></album> <type><%= songs[i].type %></type> <release_date><%= songs[i].release_date %></release_date> <best_lyric><%= songs[i].best_lyric %></best_lyric> <comments> <% for (var j = 0; j < songs[i].comments.length; j++) { %> <comment> <username><%= songs[i].comments[j].username %></username> <date><%= songs[i].comments[j].date %></date> <content><%= songs[i].comments[j].content %></content> </comment> <% } %> </comments> </song> <% } %> </songs>

One drawback of this approach is that all the JavaScript code I wrote in the EJS template was in ES5. Babel is unable to transpile EJS so if you write ES6 all browsers must be ES6 compatible. Therefore I settled for ES5 just to be safe. Here you can see the XML result in postman:

Text searching is how search engines take user input and return a list of results. The full picture of how it works is beyond the scope of this blog (although it would make for a fun discovery some day!) but just know that MongoDB allows for a barebones version of text searching. You can implement a text search by defining an index of type 'text' on a property (most likely you will define it on a string but you could even place it on an array)6. MongoDB will do the rest of the work for you. Weights can also be placed on properties to specify how important a certain field is for a search7.

Remember how I said practically anything you wanted to do in MongoDB can be done in Node.js using Mongoose? You can implement a text search and its necessary indexes as well8! Here are the indexes which are placed on the SongSchema:

SongSchema.index( { 'title': 'text', 'album': 'text', 'artist': 'text', 'best_lyric': 'text' }, { 'weights': { 'title': 10, 'album': 5, 'artist': 8, 'best_lyric': 2 } } );

Then you can perform the text search using Mongoose:

searchRouter.route('/:query') .get((req, res) => { // Perform a text search and sort based on the text score. // The score is calculated by the indexes placed in the database Song.find({ "$text": {"$search": req.params.query}}) .select({"score": {"$meta": "textScore"}}) .sort({"score": {"$meta": "textScore"}}).exec() .then((songs) => { res.format({ 'application/json': () => { res.json(songs); }, 'application/xml': () => { res.render('xml/songs', {songs: songs}); } }); }) .catch((err) => { res.status(500).send(err); }); });

Now a text search can be executed on the song collection. When a text string is entered, it finds matches on the title, artist, album, and best_lyric fields. When I search the word 'faith', it matches a Mariah Carey song with a matching lyric:

I also found a really nice library to add basic auth to my endpoints. Basic auth works by using the HTTP requests Authorization header to check for a base64 encoded credential string. This credential string before being encoded has the form <username>:<password> followed by the unicode pound sign (U+00A3)9.

All you need to do to implement basic auth in an express app is pass the module a JSON object of usernames and passwords to accept10. This is yet another example of how user made npm modules can make Node.js development easy.

app.use(basicAuth({ users: {'a': 'j'} }));

The final major piece of this prototype was the build automation tool - Gulp. Gulp allows you to build your project in stages using tasks defined as JavaScript functions11. I am still trying to get a hang of gulp but I made a build file that watches for any file changes and automatically restarts the Express server with the changes applied. In the process, Gulp uses Babel to transpile all the ES6+ files into ES5 and moves these transpiled files into a new directory. I also move all my EJS files into this new directory. The code is very hackish (as I don't know gulp well) but gets the job done!

// The main task called. We first execute the 'watch' task and then run a script // to start the server gulp.task('default', ['watch'], () => { nodemon({ script: './dist/app.js', ext: 'js', env: { PORT: 3000 }, ignore: [ './node_modules/**', './dist/**/*.js', './dist/**/*.map', './package.json', './package-lock.json', './dbscripts/**' ] }).on('restart', () => { // Nodemon will restart the server when any of the src files change console.info('Restarting Server with Changes'); }); }); // The 'watch' task will wait for changes in the source files and when they occur invoke // the 'transpile' task. gulp.task('watch', ['transpile', 'move'], () => { livereload.listen(); gulp.watch('./src/**/*.js', ['transpile']); gulp.watch('./src/view/**/*.ejs', ['move']); }); // The transpile task invokes babel to convert ES6+ code into ES5 gulp.task('transpile', () => { gulp.src('./src/**/*.js') .pipe(sourcemaps.init()) .pipe(babel({ presets: ['env'] })) .pipe(sourcemaps.write('.')) .pipe(gulp.dest('./dist')) .pipe(livereload()); }); // The move task takes out ejs files used for xml generation and puts them in the dist folder gulp.task('move', () => { gulp.src('./src/view/**/*.ejs') .pipe(gulp.dest('./dist/view')); });

Now all I have to do to run my server is enter one command in bash:

gulp

Gulp made my life a lot easier in this project. Combining Gulp which restarted my server on all changes and the WebStorm IDE which automatically saves files, I never had to save a file or restart my server throughout development. I put 100% of my focus on developing!

This MongoDB and Node.js prototype was very enjoyable to make and the full JavaScript stack is growing on me. There are still many unexplored areas of Node.js development for me but it is clear why this server side environment was such a game changer.

The code for the prototype is available on my GitHub.

[1] Amos Haviv, MEAN Web Development, 2nd ed (Birmingham, UK: Packt, 2016), 110

[2] "Built-in Promises", http://mongoosejs.com/docs/promises.html

[3] Alex Young, Bradley Meck, Mike Cantelon, Node.js In Action, 2nd ed (Shelter Island, NY: Manning, 2017), 156

[4] Young., 157

[5] Young., 158

[6] Kyle Banker, Peter Bakkum, Shaun Verch, Douglas Garrett &amp; Tom Hawkins, MongoDB In Action, 2nd ed (Shelter Island, NY: Manning, 2016), 252

[7] Banker., 255

[8] "Mongoose - Search for text in three fields based on score or weightage", https://stackoverflow.com/questions/32063998/mongoose-search-for-text-in-three-fields-based-on-score-or-weightage

[9] "The 'Basic' HTTP Authentication Scheme", https://tools.ietf.org/html/rfc7617

[10] "express-basic-auth", https://www.npmjs.com/package/express-basic-auth

[11] Young., 73