New Builds 2024

Last weekend, I had an amazing opportunity to participate in New Builds, a first-of-its-kind hackathon in the city of Toronto. Now, I've done hackathons before and won a few, but this was different. I could feel it from their registration page. This felt like an event that was curated to be the best, and boy did it deliver. Before writing further, I want to give a special thanks to the organizers (vin, tommy, and the entire New Systems team) for putting together this event.

Draft Night

The most interesting part of the event was not the actual hackathon, but the way the teams were formed. A sports-style draft night was organized where each captain played a bunch of 1v1 mini-games (rock-paper-scissors, beyblades, multiples of 7?) to decide their picks. Each participant was anonymized to ensure picks were made based on skill.

The draft night was a blast. Everyone dressed up in formals, there was a photography booth, conversations were flowing, and the energy was unmatched. After a few rounds of drafting, I found myself on a team with Melody, Kuba, and Kavin. Rather than stick around for the night, we decided to sit down and brainstorm ideas, which led us to a rough concept of what we wanted to build: A 'choose-your-own-adventure' game that the audience would influence and move the story forward, using generative AI.

The game itself

The idea of the game is relatively simple: You start with an overall plot summary, a protagonist that makes decisions, and a few objectives within each chapter. Everything else is generated using LLMs, with the audience providing content for each set of choices. I had a cracked team, and within a few hours, we had a working prototype that we played around with, generating extremely funny choices and some that shouldn't be talked about.

The technical details

I took care of building the entire AI backend, and I chose to use LangGraph to build it out. LangGraph is a framework that allows you to build stateful, multi-actor applications using LLMs in a natural way. It was the perfect fit for this project, and I had a lot of fun building it out. Below is a diagram of the system architecture:

There's a lot going on here, so let's break it down:

Before the game starts, we pre-generate a protagonist using tweets from an actual user. This includes generating a character with a personality, some motivation for the overall plot, and a few quirks, all of which is handled by the ProtagonistGenerator node.
Then, the story begins with the first chapter, which is generated by the ChapterWriter node. This node takes in the plot summary, the protagonist's details, and the current objective to generate an introductory chapter.
Once the chapter is generated, it's presented to the audience. The audience is then prompted to provide input on potential plot developments or character interactions they'd like to see in the story. This is all handled using Websockets and Cloudflare Durable Objects, making it extremely low-latency and real-time.
These messages are then filtered and sent to the ChoiceWriter node, which is configured to output 4 potential choices that utilize the audience's input and make sense within the overall story. These choices are then presented to the protagonist, who makes a decision.
The protagonist's decision is then sent back to the beginning of the graph, and the next chapter is generated. At the same time, the protagonist's decision is scored in the background on a few parameters (we chose to score on effectiveness and friendliness) using the ChoiceScorer node.
The above process repeats until we reach the end of the story, at which point the EndingSceneWriter node is kicked off to generate the final scene of the story, which depends on the average score of the choices made throughout the story. We came up with 4 different endings (it's math, 2^2 = 4), ranging from getting locked in a dungeon, saving a friend, getting betrayed by a friend, and getting a happy ending.
A live image generation pipeline that used fal to run FLUX[dev] and generate images for each chapter to make the story more immersive.

All of the above had to run in real-time and most of the nodes required a human-in-the-loop to either generate choices based on the audience's input or to let the protagonist make a decision. We used Supabase as our database of choice to store the audience messages, the story checkpoints, and the choice scores.

I honestly had a lot of fun building this system out. I learned a lot about how LangGraph works, and how to configure it to do some really cool stuff.

The results

We didn't win any prizes, but for me, the real win was the time I had with my team. We shared a ton of laughs, and we built something that we were proud of. That's all that matters. Also, sleeping in the Cohere office, meeting Ivan (CTO, Cohere), drinking 12 Celsius cans, getting an extremely shitty pizza at 2AM from Mama's Pizza, and watching other teams build some really cool stuff - that's more than enough value for me.

You can see our demo on the X broadcast here (51:30 mark): New Builds '24.

A few other threads:

That's all for now! I want to say thanks again to New Systems, Cohere and my team for an amazing experience.I'm excited to see what the future holds for New Systems, and I'm looking forward to whatever they cook up next.