Project to Calculate COVID Moving Average — Firebase Storage[Part 2]

7 min readJan 19, 2021

This is a series of posts, all related to the same project, but each one is focused on specific issues.

W e definitely need to store our COVID data somewhere, but the options to do it are so diverse that one can be lost amongst so many solutions. For instance, we could use an open-source relational database like MySQL, or maybe bigger ones like Oracle or SQL-Server… Or, considering that our COVID data provided by public Web Services is a JSON Object like this:

{
     “uid”:35,
     “uf”:”SP”,
     “state”:”São Paulo”,
     “cases”:1546132,
     “deaths”:48351,
     “suspects”:5334,
     “refuses”:596,
     “datetime”:”2021–01–11T18:26:31.018Z”
}

How about storing the JSON document in a NoSQL database? That’s why we will use Firebase. If you do not know what JSON is, you can check it here.

1. Firebase Configuration

It is very easy to configure Firebase. Basically, if you have a Google account, you already have a Firebase account. Just go to https://firebase.google.com/ and click on “Go to console” as seen bellow.

Next, click on “Create Project” and choose a name you like to your project. Let’s call it “Teste” for instance. Remember to accept Firebase terms, click “Continue”, turn off Google Analytics (we do not need it for this project), and press “Create Project”. After a while your project’s console will appear. Look at item “Realtime Database” on the left side. This is the place where our Database will be stored. So, click “Realtime Database” as seen bellow:

efefeff — a Realtime Database to be created

Next, click on “Create Database” an choose a location (us-central is fine). Click “Next” and keep the option “Start in locked mode” selected (we will change it soon) and press “Enable”.

Now our NoSQL Realtime Database is ready to be used. As shown below, you will see the root element pointing to null on Data tab, which means that it is just waiting for data to be stored.

One more thing: Remember that the option “Start in locked mode” was chosen? We need to change it because it means “Nobody can read it, Nobody can write it”. So, click on “Rules” tab and change it:

{
   “rules”: {
     “.read”: false,
     “.write”: false
   }
}

to:

{
   “rules”: {
     “.read”: true,
     “.write”: true
   }
}

Remember to press “Publish”. This rules now mean that anyone can read or write on our Database. Of course this is a potential security flaw, but don’t worry we will change it again later.

2. Data Storage

Firebase is capable of storing JSON documents. To verify this you can click the root element on Data tab, press (+) to start the creation of key-value pairs.

But that is not cool :( What is cool is to receive requests from somewhere with data to be stored. Remember that we changed the rules so that, basically, we can receive requests from anyone.

You can see here detailed documentation, but in short, we will use PUT/PATCH to store data and GET to read data. The previous image shows our project ID (teste-c5687-default-rtdb). Now, open the Terminal and submit the PUT command containing a simple JSON with the day and the number of deaths:

curl -X PUT -d '{ "day": "10", "deaths": "15" }' \
  'https://teste-c5687-default-rtdb.firebaseio.com/covid/data.json'

You will receive the following JSON as response: {“day”:”10",”deaths”:”15"}, (and code 200 under the covers, but we will talk about it later…). In addition, you can see that your Data tab was updated similar to this:

Now, let’s say you want to correct the number of deaths from 15 to 17, so use the PATCH command bellow and you will see that the number of deaths is updated do 17.

curl -X PATCH -d ‘{“deaths”: “17” }’ \
 ‘https://teste-c5687-default-rtdb.firebaseio.com/covid/data.json'

But wait! Remember that we collect our COVID data day by day, so an array data structure would be more appropriate. Obviously, you can also insert arrays on Firebase. Let’s say, on day 10 there were 17 deaths, and on day 11 there were 22 deaths. As expected, submit the following code:

curl -X PUT -d ‘[{ “day”: “10”, “deaths”: “17” }, { “day”: “11”, “deaths”: “22” }]’ \
 ‘https://teste-c5687-default-rtdb.firebaseio.com/covid.json'

If everything went Ok, you will receive the inserted array [{“day”:”10",”deaths”:”17"},{“day”:”11",”deaths”:”22"}]. On the other hand, your Data tab will reflect the PUT request recently sent as bellow:

All the commands presented are enough for our COVID project. However, one issue we may have is updating the array, which is not possible using the simple commands presented. The strategy would be to read all the data from database first, add the increments/updates in-memory and re-write the entire tree. You may say: “Ok, it is not that smart”, but remember, JSON is just a STRING, and for our learning needs is more than enough.

3. Security

Remember that our Rules tab is set to accept anything from anyone in the world? It is time to change it! So go to Rules tab and change the content from this:

{
   “rules”: {
     “.read”: true,
     “.write”: true
 }
}

to the one below:

{
   “rules”: {
     “.read”: “true”,
     “.write”: “auth != null”
   }
}

This change means that anyone can still read our data on Firebase, bur ONLY authenticated users can write on it. Do the change and re-submit the last PUT command (changing some array values) to see what happens.

{
 “error” : “Permission denied”
}

This response means that we don’t have sufficient permissions to complete the request. Now let’s see how an authenticated user can login on the database and send successful requests. Now, we will have to login prior to sending requests.

First, on the left side of the console, click on “Authentication” and you’ll see some new tabs. Let’s keep an eye on “Users” and “Sign-in method” tabs. On “Sign-in method” tab enable Email/Password provider [keep Email link (passwordless sign-in disabled though].

The last action allows users to login using previously saved emails/passwords. And where do we add those users? Of course, on “Users” tab. Go there and add an email and password you like (you will need them to login, do not forget them!).

4. Authentication

In order to test our access we need our API_KEY number. Go to the small gear on the left side and click on “Project Settings”. On General tab look for Web API Key. This long string is your API_KEY number. Copy it!

Now you need to require access using the user (email) and a password you had selected previously. This request is presented next:

curl ‘https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key=AIzaSyAOmUTYxItaVhSXCBKn-s8KRkanXSRibpo' \
-H ‘Content-Type: application/json’ \
 — data-binary ‘{“email”:”ferfox@medium.com”,”password”:”12345678",”returnSecureToken”:true}’

Next you can see the response for last command:

{
 “kind”: “identitytoolkit#VerifyPasswordResponse”,
 “localId”: “PCJGyn5CNTMhMSUwiXqAbc2UEhz2”,
 “email”: “ferfox@medium.com”,
 “displayName”: “”,
 “idToken”: “eyJhbGciOiJSUzI1NiIsImtpZCI6IjVmOTcxMmEwODczMTcyMGQ2NmZkNGEyYTU5MmU0ZGZjMmI1ZGU1OTUiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdGVzdGUtYzU2ODciLCJhdWQiOiJ0ZXN0ZS1jNTY4NyIsImF1dGhfdGltZSI6MTYxMDU4MDExNCwidXNlcl9pZCI6IlBDSkd5bjVDTlRNaE1TVXdpWHFBYmMyVUVoejIiLCJzdWIiOiJQQ0pHeW41Q05UTWhNU1V3aVhxQWJjMlVFaHoyIiwiaWF0IjoxNjEwNTgwMTE0LCJleHAiOjE2MTA1ODM3MTQsImVtYWlsIjoiZmVyZm94QG1lZGl1bS5jb20iLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImZpcmViYXNlIjp7ImlkZW50aXRpZXMiOnsiZW1haWwiOlsiZmVyZm94QG1lZGl1bS5jb20iXX0sInNpZ25faW5fcHJvdmlkZXIiOiJwYXNzd29yZCJ9fQ.NCZRMfstI_VnuOMwavhLzvrR50gYbEh_xmTE9_vL7googYA4ELOFWZnc0TTTbd8nLpSfWs2GVv4P5bRCMcjMkfRzxdNicNmatsUC0XRov1LUXC_jdOhzBzGYzilIBvlX_pdpyb6dPzOI1BrKFMdpkw5-BIFftU-LvuhfpummcUNvukrkROgSF9OVXuJhCrrh7IRwHChqrXghpA5Bebke1OOtZSwZP4cgNBbdDlyNJ_eESkLeeJHto26yw8pzGR4ssoIb9Xu20HHirttKQ6WS-8ZkJuaRVXrOyiohpnPU5JrEj8oNPBKYVAEQke9bv7V08dkxl360ABwR8MY9CeBdcA”,
 “registered”: true,
 “refreshToken”: “AOvuKvQx4Bxbc3I8DEu9NpwioXP81qRjI4nnkrrizx2itPXrXbrY61iZa7QiR60HikqeZ8nF-9bwggT97APFNfBp2J8ig5hjlLXsno750R1_8iGBSJirEffqlMK4pNMeud5-X4OoYdWu8O5rr9GYafa1RjgmpBOzd5TsKOoLy2UG2eb42Hq3tgnCXx-xgS81c_b82BocsQm3”,
 “expiresIn”: “3600”
}

This JSON response has 2 important fields:

idToken: It is important to save this series of characters for a while, it will be used to authorize the request;
expiresIn: The number of seconds in which the ID token expires (3600 seconds means 2 hours, so after that a new idToken is needed).

Finally, remember that PUT command that was rejected? Make the following adjustment and submit it one more time.

curl -X PUT -d ‘[{ “day”: “10”, “deaths”: “61” }, { “day”: “11”, “deaths”: “66” }]’ \
 ‘https://teste-c5687-default-rtdb.firebaseio.com/covid.json?auth=eyJhbGciOiJSUzI1NiIsImtpZCI6IjVmOTcxMmEwODczMTcyMGQ2NmZkNGEyYTU5MmU0ZGZjMmI1ZGU1OTUiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vdGVzdGUtYzU2ODciLCJhdWQiOiJ0ZXN0ZS1jNTY4NyIsImF1dGhfdGltZSI6MTYxMDU4MDExNCwidXNlcl9pZCI6IlBDSkd5bjVDTlRNaE1TVXdpWHFBYmMyVUVoejIiLCJzdWIiOiJQQ0pHeW41Q05UTWhNU1V3aVhxQWJjMlVFaHoyIiwiaWF0IjoxNjEwNTgwMTE0LCJleHAiOjE2MTA1ODM3MTQsImVtYWlsIjoiZmVyZm94QG1lZGl1bS5jb20iLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImZpcmViYXNlIjp7ImlkZW50aXRpZXMiOnsiZW1haWwiOlsiZmVyZm94QG1lZGl1bS5jb20iXX0sInNpZ25faW5fcHJvdmlkZXIiOiJwYXNzd29yZCJ9fQ.NCZRMfstI_VnuOMwavhLzvrR50gYbEh_xmTE9_vL7googYA4ELOFWZnc0TTTbd8nLpSfWs2GVv4P5bRCMcjMkfRzxdNicNmatsUC0XRov1LUXC_jdOhzBzGYzilIBvlX_pdpyb6dPzOI1BrKFMdpkw5-BIFftU-LvuhfpummcUNvukrkROgSF9OVXuJhCrrh7IRwHChqrXghpA5Bebke1OOtZSwZP4cgNBbdDlyNJ_eESkLeeJHto26yw8pzGR4ssoIb9Xu20HHirttKQ6WS-8ZkJuaRVXrOyiohpnPU5JrEj8oNPBKYVAEQke9bv7V08dkxl360ABwR8MY9CeBdcA'

You can see that now the Database was updated and the request was accepted. Notice that after the original PUT command a question mark followed by “auth=idToken” were added, and that’s it! Access granted!

5. Firebase & JAVA

The requests required to authenticate and access/update Firebase are simple HTTP requests. Spring provides RestTemplate class that can be widely used to achieve project’s needs. For instance, to authenticate a user in order to get idToken the following Bean is created.

To store COVID data on Firebase we convert an Object to JSON using GsonBuilder and send it as a POST request, as seen bellow.

6. Conclusion

The complete code can be downloaded here. Next part will cover deployment activities on Heroku and the set up of a scheduler to update our COVID data sistematically.

Thank you.