I am creating a worker roster management web app with about 600 workers. Every time I check the roster, it pulls data from the entire database, and the reads add up quickly to the firebase free limit. This feels really inefficient, and I could use some help figuring it out.
My backend is hosted on Railway, the database is on Firebase, and I’m thinking about switching to DigitalOcean for better costs and scalability. Before I make that choice, I would appreciate some insights on:
How can I optimize database reads for a roster app?
Would moving to DigitalOcean or a different platform help with costs or efficiency?
Firebase offers 50k reads per day on the free tier, and that can get used up quickly. Are you running any queries in a loop? Like looping through all users and pulling the roster for each? Sounds like you might have an N+1 issue.
Jagger said: @Shai
That’s what it seems like to me. The person who posted this needs to get familiar with joins and pagination.
Plus, once the app is running, cache the results to avoid reading from the database multiple times. This can boost the app’s response times, though it will use more memory, which is usually a minor trade-off.
Zeek said: @Shai
Yeah, my first thought was the N+1 problem. The person might be able to reduce their database queries by a factor of 10 with a join.
Because Firebase is NoSQL, you really can’t do a join without an extra query.
The person will need to create a new denormalized document in Firebase that includes all necessary data, allowing them to fetch it all in a single request.
They will then have to manage consistency because the same data will be duplicated in different documents. If one part changes, all related documents need to be updated too.
For example, updating a worker involves updating both the specific worker’s document and any larger document that includes a summary of worker data, such as the worker’s role, etc.
In SQL, it’s easy to join a worker table with a job role table, but you can’t do that in NoSQL. Therefore, each worker’s data and their job role data ends up in a separate document.
So, any changes to the job roles mean updates to both the job role document and any summary docs that include worker roles.
This sort of duplication and management challenge is standard in Firebase or any NoSQL database. This is one of many reasons why SQL is preferred by most for complex scenarios, even if they don’t realize it initially.
@Tatum
Oh, I’m so used to Supabase I forgot that Firebase is NoSQL. So yes, he would have to pre-arrange the data needed for each query and be careful to update all the proper documents when any field changes.
It seems the person who posted this is likely running a loop for every employee (with a query) and then going through another loop for each employee (with potentially numerous queries) to get additional data.
This creates a ton of reads.
The better approach would be making a query that selects all employees along with the additional data required and joins them through something like employee.id (primary - foreign key relationship).
@Sky
That’s true for relational databases, but the person who posted this is using a document store. They might be treating it like a relational database, which isn’t ideal for many reasons.
Remy said: @Sky
That’s true for relational databases, but the person who posted this is using a document store. They might be treating it like a relational database, which isn’t ideal for many reasons.
You are correct, I missed that they were using Firebase.
For anyone reading this later, expect significantly higher read rates from a document store compared to a relational database (like pulling images for a single timeline on a social site) but that 50,000 a day limit can be reached quickly (like 500 users looking at 100 posts every day). If you’re not generating any revenue to cover costs, you need to rethink this. And as mentioned before, having a caching service for frequently accessed data would really help reduce excess reads.
