My Journey through GSoC'23

Oct 4, 2023 · 8 min read · Read on Medium

This blog summarizes my time as a contributor with Postman for Google Summer of Code 2023. It's been a great experience for me throughout this journey. I learnt a bunch of things as a part of this experience and am super grateful to be selected for the project! Some of the things I learnt as part of the project were:

Planning an open source software project from scratch
Using the OpenAPI Specification and working with the ecosystem around it
Making the best and most efficient use of the GitHub API, building scalable search experiences, and working with ElasticSearch

Github OpenAPI Search

The goal of this project was to provide a robust yet easy way to search GitHub for Swagger and OpenAPI definitions. Understanding that there is a lot of noise available, that we only care about OpenAPIs that validate, and that the GitHub API has rate limits that require you to automate the crawling over time. Providing a robust open-source solution that will crawl public GitHub repositories for machine-readable API definitions.

We segmented the problem into 4 subproblems:

Finding valid OpenAPI files using the GitHub API
Validating and bulk loading them to an ElasticSearch database
Searching through the database to look for the files
Updating the database with the latest OpenAPI files

Finding valid OpenAPI files

Finding the OpenAPI files using the GitHub API was one of the crucial challenges we faced. After a lot of iteration and exploration on different methods — comparing them on accuracy of files found and rate limiting — we ended up using the code search API.
The code search API had various limitations though: a low rate limit (10 req/min) and a cap of 1000 results.
We had to work around rate limits and make sure we found the most accurate data we could. We ended up with '"openapi: 3"' or '"swagger: 2"' after a bunch of testing, to run an exact text search on the unique OpenAPI spec syntax using the GitHub search syntax.
This led to us finding 150k files on GitHub (although the GitHub API isn't good at estimating total files). Since GitHub code search caps at 1000 results, we couldn't brute force it.
We then narrowed the scope to the top 2000 organizations on GitHub, selected by star count.

Validating and bulk loading into ElasticSearch

The next problem required validating each of these files against the OpenAPI schema and loading them into the database. For this, we used the oas-normalize library.
We chose ElasticSearch as it was the best fit for building a search experience. Validated files were loaded via the bulk endpoint, with both OpenAPI and Swagger files stored in the same index.
To handle duplicates, we used the ETAG produced by GitHub as the primary key for each file.
Internally, our solution for subproblems 1 and 2 was called Active Search, exposed via the POST /openapi endpoint.

Searching through the database

We experimented with different search strategies in ElasticSearch. Having worked with the GitHub search syntax, we did something similar using the Simple Query String syntax.
Although a bit limited, it was entirely fault tolerant and offered features like searching by custom fields, and/ors, and boosting.
We used a multi-field search as our base, boosting title, repository, and owner fields over raw file data.
Internally, we called this Passive Search, exposed via the GET /search endpoint.

Keeping the database up to date

GitHub files change constantly, so the database had to update too.
We used the ETAG header along with conditional requests from the GitHub API while iterating through database rows — both to detect changes and to optimize rate limits.
We handled deletions and updates, and used throttling to avoid secondary rate limits.
This became the PUT /openapi endpoint.

What we ended up with

A scalable, extendable system which scanned about 80,000 files to identify ~3,000 validated OpenAPI and Swagger files from the top 2000 GitHub organizations.
An API interface for the GitHub code search API to accurately look for OpenAPI/Swagger files with Active Search.
A multi-field search experience over validated OpenAPI files using ElasticSearch, with various search filters, via Passive Search.
Minimal downtime from rate limits and validation, by batch processing every step of the way.
A curated, deduplicated collection of OpenAPI files on GitHub that stays current through the update endpoint.

Future scope

Extend to other API specifications: the server is built for extendability — RAML, GraphQL SDL, Protobuf, and more.
Extend to other types of files, like those that validate certain JSON Schemas.
A better interface: the current admin panel is ad-hoc. It could become a user-facing site to browse collections by organization and convert them to Postman Collections.

Journey through the weeks

This is a short summary of what I did week by week during GSoC. Anshul and I kept a shared working document for communication, meeting notes, and scratch documentation throughout — its version history shows the exact journey of how we implemented everything.

Community bonding period: We discussed basic details about the project, but timing was an issue due to my end-semester exams.

June 1 → June 15: The starting two weeks were hectic for both of us due to exams and a product launch. We decided deliverables and made a roadmap. I researched search strategies and settled on the Passive/Active Search split we still use, and compared ways to find OpenAPI files efficiently through the GitHub API. In hindsight this planning period is why we hit so few errors later.

June 15 → June 22: Started coding — the Node.js boilerplate with basic testing and linting config, plus a v1 active-search endpoint that both found and validated OpenAPI files.

June 23 → July 4: Added pagination and better querying to active search. Finalized the database with Shubham's help, and explored ElasticSearch and Kibana from Node.js. I was on an offsite for my other internship, so bandwidth was lower.

July 5 → July 12: Decided the database structure for storing OpenAPI files, wrote the passive-search endpoint, the v1 README, and a runbook to set up the system.

July 13 → July 19: Researched ElasticSearch search strategies and chose the simple query string method. Improved code quality across both endpoints and employed batch processing to speed up loading.

July 20 → July 26: Iterated on an improved data model, decided how to handle duplicates, and designed the update API and passive-search strategy.

July 27 → August 3: Finished the update API and wrote a seed script for testing. Decided on a simpler Docker-based setup.

August 3 → August 10: Discovered the GitHub API caps at 1000 results, which led to the top-organizations loading approach we use now. Lots of code quality work.

August 10 → August 22: Wrote the seed and gitstar-rankings scripts, a Dockerfile for easy initialisation, and better README documentation. Also had my GRE exam in this period.

August 23 → August 28: Got unit tests working, built the basic admin panel to showcase the APIs, and worked on the final report.

Final thoughts

GSoC has been a very fruitful journey for me. Since the start of my engineering degree, I have always seen YouTube videos about GSoC and was pretty pessimistic about my chances. To now be at a spot having successfully completed this journey — it has been super satisfying. Working with Postman, which without exaggeration was part of my first steps into backend development, made it a full circle moment for me.

Both of us had variable bandwidths throughout — my mentor with a full-time job, and me with a full-time internship and studies. But this project was still super fun to work on and I am super proud of what we cooked up.

A lot of credit goes to Anshul Jain, my mentor during this program. He guided me perfectly throughout and was the primary reason we didn't have a lot of hiccups. I've learnt a lot from him about managing a software product, and working with him has definitely made my code look better. Thanks also to Shubham Ranjan for helping us figure out ElasticSearch.