Lachlan's avatar@lachlanjc/eduCourses
Shared Minds

Week 8: Search

I wrote recently about why I open source all my schoolwork, and one of the reasons I mentioned is because I constantly refer to my past work. However, my site has been lacking a search feature, so I thought it’d be fun to build one. This week was my first time ever playing with embeddings. (Here’s a great explanation of the concept.)

I started out reading Maxime Heckel’s article on building semantic search over OpenAI’s API from scratch. I then found Supabase’s tutorial on integrating them, since I know they make a hosted database. (In hindsight, I should’ve used it!) But I kept looking around, and found this tutorial for ChromaDB, which I was curious about.

First, I set up ChromaDB, and bought $10 credit on OpenAI. I adapted that last tutorial’s basic code for indexing, using a Next.js API Route which:

  1. Queries Contentlayer for all posts
  2. Reformats the content to remove code blocks, links, images, etc, using Maxime's code
  3. Generates embeddings for each post using OpenAI
  4. Indexes the embeddings in ChromaDB

Then, it was time to build the search endpoint. I followed that tutorial’s structure again, adapting it into a second Next.js API Route that queries ChromaDB for the five most relevant posts to a search query, then reformats its output to include the post’s formatted title, date, and URL.

Finally, for the frontend, I figured a client-side component would be fastest to build, which was sort of correct. I added an input, hooked up React state and the form submission to fetch the search API endpoint. I spent some time making the input update the URL of the page with a query parameter, so search results could be shared via URL, and reformatting the page title.

Sadly, I didn't realize until the very end that Chroma doesn't have a hosted version, so for now, this project can only run locally on my computer. Here's the results of 3 search terms, if you're curious what they're like. Unlike traditional search, these terms don't appear directly in several of the results, they're just related to the content of the page.

Food:

  1. How the Other Half Ate
  2. Paper Proposal
  3. Week 3 Readings
  4. Freedom Farmers
  5. Taming Manhattan

Accessibility:

  1. All Technology Is Assistive
  2. Week 3 Responses
  3. Accessible Immersive Experiences
  4. Week 4 Responses
  5. Week 5 Responses

Apple Vision Pro:

  1. Encounter Dinosaurs
  2. Week 2 Spatial
  3. Week 6 Dimensions
  4. Digital Accessibility Options
  5. Accessible Immersive Experiences

Source code for this week: