dataexamplesjsondatabasemongodbtutoriallocalstoragejavascriptnodejs

#Data management (tutorial)

In this tutorial I will show 3 different ways of storing data.

For this tutorial I'll be using a node.js server and expect the reader to have at least some basic understanding of Javascript.

Different types of data require different methods of storing

Because not all the data is equally important to the owner of the web page, you can choose the approach you find fitting to your needs. In this tutorial I tend to explain when it's best to use one of the three different approaches handled in this tutorial. Here is a heads up of the three different approaches I'll be explaining:

Local storage ★✩✩ (store data in the browser, personal data of the user)
Server ★★✩ (store the data as a JSON file, usually done for small projects or prototypes)
Database ★★★ (store large sets of data and multi user data-objects)

Local storage

Difficulty level - ★✩✩ (beginner)

What is local storage? localStorage is a built-in web storage API that can be used to securely store data as key/value pairs. In Google Chrome you can find it in the developer tools under "Application", see image below.

Before you store the formData there are some things you need to know:

Local storage only allow "strings" to be stored
Data in local storage has no expiration time

|Local storage (chrome) |Similar storage method: sessionStorage | |--- |--- |
| local storage location |Session storage works exactly the same as localStorage but with one major exception:
all data in the session storage gets cleared when the page session ends.*
*whenever the user closes the page. |

When should I use this approach?

Let's say you have a form on your website but the form is rather long and your user doesn't feel like completing it in one turn. This is a perfect opportunity to use local storage. You can store the form data as key = form question and value = user's answer. Before you can store the data you need to parse the form data to a JSON format. This can be easily done using the javascript built-in FormData. Every time the user fills in an answer, you can trigger the following function to parse the form into JSON-object.

index.js

const form = document.querySelector('#myForm') // first get acces to the form

form.addEventListener('change', function () {

    const formData = new FormData(form) // with form available we can parse the answers of the users into our formData object
    // the FormData object itself cannot be read or parsed so we'll have to take it apart

    let formDataObject = {} // define a Object you want the formData to be parsed to

    formData.forEach((value, key) => { // here we take all the values and keys in our formdata and assign it to the formDataObject
        formDataObject[key] = value
    })
    console.log(formDataObject) // now if we change something to the form, the output will be {firstName: "Wouter", lastName: ""}
    // this is perfect because we have a JSON-object containing all our form data

    // Now we can call the function to store our formDataObject with a matching local storage name
    setLocalStorage("formData", formDataObject)
})

Now we have our formData available as an javascript-object, but before we can store this to the local storage we'll have to parse it to a string. We can simply use the javascript built-in JSON.stringyfy(formDataObject) to get the result we want. In the next function I'll show you how to save data to the local storage.

function setLocalStorage(localName, dataToStore) {
    // before we store the data check if there already is data in the localstorage
    let existing = localStorage.getItem(localName)

    // If no existing data, create an object
    // Otherwise, convert the localStorage string to an object
    existing = existing ? JSON.parse(existing) : {}

    // Add new data to localStorage Object
    existing = dataToStore

    // Save back to localStorage
    localStorage.setItem(localName, JSON.stringify(existing))
    // remember to parse your object to a string because localstorage can only store "strings"

}

And here we have our result: we stored data to the local storage! The data is now accessible by calling localStorage.getItem('formData'). Remember: before you can use this as an javascript-object {}, you'll have to parse it to JSON again because the local storage only stores "strings". For a working demo, check the image below or clone this repo and try it yourself! local storage demo

Server

Difficulty level - ★★✩ (intermediate)

In this tutorial we're gonna create our own user list and store the list in a file on our server. For our server-side data management we'll be using the following setup:

For our basic setup I created a simple form which takes a first- and a last name. And our server set up like this:

app.js

const express = require('express')
const routes = express.Router()
const bodyParser = require("body-parser")
const urlencodedParser = bodyParser.urlencoded({ extended: true })
const setJSON = require('./controllers/setJSON')

const config = {
    PORT: 3000
}

const app = express()

app
    .use(express.static('static'))
    .use(bodyParser.json())
    .use(urlencodedParser)
    .use('/', routes)

    .set('view engine', 'ejs')
;

routes 
    // the route required for the POST request this is the route you post to with your form
    .post('/form', setJSON.writeData, (req, res) =>{ // we call the middleware function, writeData, where we extract our form post
        let data = res.locals.users 
        // This is the unsaved data made available from the writeData function that lives in: 'controllers/setJSON.js'
        // we will serve the user a list of all users in the data.json
        res.render('pages/userlist.ejs', {
            users: data.users, // The data is now available on the userlist.ejs
            title: "A list of users"
        })
    })

    // the route to homepage
    .get('/', (req, res) =>{ 
        res.render("pages/form.ejs", {
            title: "Store data to server",
        })
    })      
;

app.listen(config.PORT, () => console.log(`Server running on: http://localhost:${config.PORT}`))

For this approach you need to create a data.json file in the root of your app. We are gonna create the file with the following contents:

data.json

{
  "users": [
    
  ]
}

When we open the file on our server, we can use it like an javascript-object {}. First, we make sure our file is available in our setJSON.js with the following statement at the top of our file: const jsonFile = "./data.json";. We can target the file and parse its contents.

fs.readFile example

fs.readFile(jsonFile, (err, content) => {
    // First, we read the jsonFile and see whats inside
    if (err) return console.log(err); // catch errors as they occur

    const contentJSON = JSON.parse(content); // parse the json file to workable javascript-object

    console.log(contentJSON); // output: { users: [] }
}

Now we have acces to our users = (contentJSON.users), which, at this point, is an empty array: [ ]. Because it's an Array we can add one object to it using contentJSON.users.push(newUser). Now contentJSON looks like this:

{
  "users": [
    { 
      "firstName": "Wouter",
      "lastName": "Heijde" 
    }
  ]
}

Nice, we stored a user in our contentJSON variable next up rewrite the jsonFile with the new content:

fs.writeFile(jsonFile, JSON.stringify(contentJSON, null, 2),(err) => { 
      // I stringyfy the contentJSON before its write-able to the data.json
      // these fancy parameters I put in the stringyfy
      // makes sure the data.json gets nicely formatted
      if (err) console.log(err);
    })

Post form and save user

Great! We have our own little database on the server. But when you'll have to handle more complex and bigger datasets, this might not be an ideal approach. Next step: create your own database!

Database

Difficulty level - ★★★ (advanced) We are going to create a register page and store a user to our database.

We are using mongoose which is an extension build for mongodb. Mongoose comes with a lot of built-in functions that helps us with making models and finding objects in our database.

For our database we are going to use Mongodb. Before you can start this tutorial have the following things ready:

Register an account on Mongodb
Clone this repository: example.3
open your terminal and navigate to example.3
run npm i
.env - file to safely store our local variables

Dotenv is a zero-dependency module that loads environment variables from a .env file into process.env. For more information about dotenv and how to use it check their documentation on: dotenv - npm

After registering an account on mongoDB, log in on mongodb-Atlas with your credentials. Go to the cluster tab and create a database. Fill in a name for your database (name of your app) and a name for your collection (in this case 'users' would make a lot of sense).

You can get the URI to connect with your database really easily. Check the example below! example of mongodb-Atlas

For convenience matters, I paste the complete URI to our .env-file.

MONGODB_URI = mongodb+srv://<username>:<password>@<clustername>-dklfs.azure.mongodb.net/<dbname>?retryWrites=true&w=majority
DB_NAME = YOUR_DATABASE_NAME

First step is creating a connection with our database. Since we have mongoose installed we can call the mongoose.connect() and pass it our mongodb variables. Take a look at the next code example to learn how to do this.

require("dotenv").config() // allows us to acces the .env file
const uri = process.env.MONGODB_URI // Get acces to our mongoDB variables

mongoose.connect(uri, options)
mongoose.connection.on("open", function (err, doc) {
  console.log(`connection established with ${process.env.DB_NAME}`)
  if (err) throw err
})

In your terminal, the following message will appear:

Server running on: http://localhost:3000

connection established with test-Database

Great! We established a connection with our database now lets setup the user-model.

user models are used as a blueprint to store your user objects. more on user models and how to use them check mongoose - schemas

In our models/user.js we start with defining our packages required for this schema.

const mongoose = require("mongoose");
const uniqueValidator = require("mongoose-unique-validator"); // We use this package to validate the user is unique 
const bcrypt = require("bcrypt"); // We use this to create a hash on the password so that the users password remains protected.

Since mongoose has a lot of built-in functionalities, we can set a list of options with each property we want to store.

const userSchema = new mongoose.Schema({
	email: {
		type: String,
		lowercase: true,
		required: [true, "can't be blank"],
		match: [/\S+@\S+\.\S+/, "is invalid"],
		index: true,
		unique: true,  // Since we have unique-validator installed we can use this option
		required: true
	},
	password: {
		type: String,
		required: true
	},
	nickName: {
		type: String,
		lowercase: true,
		required: [true, "can't be blank"],
		index: true,
		required: true
	}
});

Perfect, we now have a model for our user. As you can see we are storing a nickName, a password and an email which has to be unique. Before we can save a user with this model, we will tell the model to hash the password so it doesn't get vulnerable:

userSchema.pre("save", function(next) {
	var user = this;
	if (!user.isModified("password")) return next(); // check if the password is already hashed before otherwise return
	bcrypt.hash(user.password, 10, function(err, hash) { // if not hashed, hash it now and save it to the database
		if (err) {
			return next(err);
		}
		user.password = hash; // tell the model the password is now the hashed-password
		next();
	});
});

The check to see if the user pass is already hashed, is really useful when you allow your users to modify their profile or personal data. If you don't do this, the password might get hashed a second time and therefore the user isn't able to log in anymore.

With our user model all set up, let's create the route we use to register our user. In our app.js we define the register route: .use("/register", user), we'll pass the user as an argument (which refers to our user register function const user = require("./controllers/user")).

First up: in our controllers/user.js we define the packages required for this function to work.

const { userSchema } = require("../models/user"); // get acces to our user-model
const express = require("express"); 
const router = express.Router();
const bodyParser = require("body-parser"); // used for extracting our form-post
const urlencodedParser = bodyParser.urlencoded({ extended: true });

Now lets create the actual route:

router.post("/", urlencodedParser, async (req, res) => {
	// Check if this user already exisits
	let user = await userSchema.findOne({ email: req.body.email }); // mongoose allows us to use .findOne() on the userShema
	if (user) {
		return res.status(400).send("That user already exisits!");
	} else {
		// Insert the new user if they do not exist yet
		user = new userSchema({
			nickName: req.body.nickName,
			email: req.body.email,
			password: req.body.password,
			
		});

		await user.save(); // we save our user to the database

		res.redirect("/succes"); // and redirect to a succes page
	}
});

To check if the user was saved successful, navigate to your mongoDB-atlas cluster and see if the user is in the collection.

Well done! You've now created a database connection and stored a user with the userSchema(user-model).