Scaling Knowledge Management For Technical Teams With Knowledge Repo

Summary

One of the most persistent challenges faced by organizations of all sizes is the recording and distribution of institutional knowledge. In technical teams this is exacerbated by the need to incorporate technical review feedback and manage access to data before publishing. When faced with this problem as an early data scientist at AirBnB, Chetan Sharma helped create the Knowledge Repo project as a solution. In this episode he shares the story behind its creation and growth, how and why it was released as open source, and the features that make it a compelling option for your own team’s knowledge management journey.

Announcements

• Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
• When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
• Your host as usual is Tobias Macey and today I’m interviewing Chetan Sharma about Knowledge Repo, an open source framework for managing documentation for technical users

Interview

• Introductions

• How did you get introduced to Python?

  • EE + CS/AI + Stats degrees
  • Airbnb working on ML models
  • Knowledge Repo itself

• Can you describe what Knowledge Repo is and the story behind it?

  • We started seeing interviewees use ipython notebooks, thought they were great
  • Wanted to push more people to use notebooks, but they weren’t very shareable, vettable
  • Existing notebook hosting services weren’t very good, and weren’t built for people who aren’t data stakeholders. It was especially poor with images, annoying cell blocks
  • Made a simple post processor to remove cell blocks, push the images to s3, and host on flask
  • Once we were pushing notebooks into a Github repo for hosting on a flask app, so many things became possible
    • Review cycles
    • Shareability / collaboration features
    • Indexing / searching
  • Concurrently, great work was happening on developing internal R packages / python libraries to provide consistent, branded aesthetics

• What are some of the approaches that teams typically take for recording and sharing institutional knowledge?

  • Copy and paste to google docs, slides
  • Facebook was using facebook photo albums
  • untrustworthy, not discoverable, divorced from the code

• What are the unique requirements that are introduced when attempting to record and distribute learnings related to data such as A/B experiments, analytical methods, data sets, etc.?

  • Reproducibility is a big one
  • Making sure the learnings are trustworthy (good data? no bugs?)
  • Distributing widely, across the org and across time
  • Experimentation
    • Experimentation is at the end of a research-design-build-measure cycle, strategic analysis is often before
    • Capturing all of the context

• Can you describe how the Knowledge Repo project is architected?

  • Repositories: a store of posts, most commonly a github repo
  • Markdown as original lingua franca, eventually a KR specific “KR post” concept (which is still basically markdown)
  • Post processors
    • Convert whatever upstream file to markdown / KR post (Jupyter notebook, R Markdown, markdown were the original ones)
    • Handle images and other large assets, usually pushing them to cloud storage
    • Evolved to handle PDFs, googledocs, keynotes

• What were the motivating factors for making it available as an open source project?

  • It was such a common problem. Even incredibly sophisticated data teams at Uber, Facebook, etc. were begging us to share the system.

• What is the workflow for creating, sharing, and discovering information in an installation of Knowledge Repo?

  • Create a github repo for hosting strategic analysis
  • Use the KR script to create a stub/template for whatever format you’re wo