take 🎸 control 🎸
Hey kids.
My third consecutive newsletter! I think I look forward to these now.
This week was a bit slow. Almost feel like nothing consequential took place.
Time left till 1st Jan 2024: 50 Days.
I have 50 days left to figure out how to make my first dollar/subscription.
Okay, so one of the main reasons I like backend stuff is that I can just kind of make these diagrams and charts in my head and keep iterating through them.
Combine that with reading about the things I don’t know and prototyping and there’s a pretty set way to making things.
Here, I’m solving a data or architecture problem of how different components are going to work or how to get them to talk to each other. On the frontend, I’m solving an experience problem which is harder for me imo.
So the scraper for my import feature can be optimized in a few places, mostly on how I proxy the traffic I think. I came across a few services I could use, notably browserless.io, and reached out to them for my use-case. Have a call with them on Wednesday. Will let ya know how it goes.
Making Architecture Diagrams and Feeling Productive.
Here’s what I think the feature would look like if it’s working:
User sends their bookmark tree to import
Server saves it on an Azure blob storage
Server sends a message over the Azure Queue Storage to the Azure Edge Functions.
Azure Edge Functions scrape the URLs and then index them onto Weaviate (the vector database I’m using). Also, use the Azure Redis Cache for optimization
Once indexed, the functions send a success message to the server, which then update the PostgresDB on Supabase
User sees the update on their browser.
Now here’s what sucked in this;
I planned to use the blob storage because the maximum size of a message over the queue was 1MB which seemed low to me. So I planned to use the blob storage.
I realized that the blob storage didn’t work that way. No ID is returned on upload and it’s meant for storing actual large files.
I also read about Supabase’s file storage but as expected, it works in the same way.
Next Iteration:
To figure things out better, I got Azure functions running locally so I could prototype and see how they run. This is what I’ve settled on;
User sends their bookmark tree to be imported.
Gets added to a Postgres table.
This triggers a database webhook to the Azure functions and sends the added row with the bookmark tree
OR
It sends just the UUID of the added row in the table. TBD on how big the webhook data is and queue message size limits.Once scraped and indexed, the functions then update another table (saved_links)
The table update also updates the client through Supabase Realtime.
I’m still iterating this as I prototype and build this out. By the next update, I think I’ll have this feature fully functional.
Azure Annoyances
One major annoyance with Azure: it’s documentation is hard to read. It took me too long to realize that Azure functions “expect” the messages from a queue to be Base64 even though this isn’t the default while sending messages?
Even more annoying is that it’s half-assed buried somewhere in the docs.
I’ll Always Be Amazed At Game Designers
I finished Control. It’s basically a game based on the vibes and premise of the SCP Foundation. Here’s a Wired article if you wanna know more.
There’s a level towards the end of the game that has shifting rooms and floors and a metal track that shifts according to the pace of the player. For eg, a specific riff or lyric comes on based on the location, and the song is basically cut and played in real-time based on the level is being played.
It was badass. I love Remedy Studios.
Miscellaneous thoughts.
GPT-4 Vision API is cheap now. I could very well make a screenshot labeler + searcher in Nous for people who screenshot a lot of things.
Someone should make a tax evader GPT which tells you how to manage your funds to avoid paying taxes.
I’m moving to the “uru” (bengaluru) this week where I’ll work out of my favorite WeWork over at “kora” (koramanagala).
Happy Diwali :)
"tax evader GPT"
👆☝️👆☝️👆