Why measure anything?
Data, when used wisely, can help you make better decisions as an open source maintainer.
With more information, you can:
- Understand how users respond to a new feature
- Figure out where new users come from
- Identify, and decide whether to support, an outlier use case or functionality
- Quantify your projectâs popularity
- Understand how your project is used
- Raise money through sponsorships and grants
For example, Homebrew finds that Google Analytics helps them prioritize work:
Homebrew is provided free of charge and run entirely by volunteers in their spare time. As a result, we do not have the resources to do detailed user studies of Homebrew users to decide on how best to design future features and prioritise current work. Anonymous aggregate user analytics allow us to prioritise fixes and features based on how, where and when people use Homebrew.
Popularity isnât everything. Everybody gets into open source for different reasons. If your goal as an open source maintainer is to show off your work, be transparent about your code, or just have fun, metrics may not be important to you.
If you are interested in understanding your project on a deeper level, read on for ways to analyze your projectâs activity.
Discovery
Before anybody can use or contribute back to your project, they need to know it exists. Ask yourself: are people finding this project?
If your project is hosted on GitHub, you can view how many people land on your project and where they come from. From your projectâs page, click âInsightsâ, then âTrafficâ. On this page, you can see:
-
Total page views: Tells you how many times your project was viewed
-
Total unique visitors: Tells you how many people viewed your project
-
Referring sites: Tells you where visitors came from. This metric can help you figure out where to reach your audience and whether your promotion efforts are working.
-
Popular content: Tells you where visitors go on your project, broken down by page views and unique visitors.
GitHub stars can also help provide a baseline measure of popularity. While GitHub stars donât necessarily correlate to downloads and usage, they can tell you how many people are taking notice of your work.
You may also want to track discoverability in specific places: for example, Google PageRank, referral traffic from your projectâs website, or referrals from other open source projects or websites.
Usage
People are finding your project on this wild and crazy thing we call the internet. Ideally, when they see your project, theyâll feel compelled to do something. The second question youâll want to ask is: are people using this project?
If you use a package manager, such as npm or RubyGems.org, to distribute your project, you may be able to track your projectâs downloads.
Each package manager may use a slightly different definition of âdownloadâ, and downloads do not necessarily correlate to installs or use, but it provides some baseline for comparison. Try using Libraries.io to track usage statistics across many popular package managers.
If your project is on GitHub, navigate again to the âTrafficâ page. You can use the clone graph to see how many times your project has been cloned on a given day, broken down by total clones and unique cloners.
If usage is low compared to the number of people discovering your project, there are two issues to consider. Either:
- Your project isnât successfully converting your audience, or
- Youâre attracting the wrong audience
For example, if your project lands on the front page of Hacker News, youâll probably see a spike in discovery (traffic), but a lower conversion rate, because youâre reaching everyone on Hacker News. If your Ruby project is featured at a Ruby conference, however, youâre more likely to see a high conversion rate from a targeted audience.
Try to figure out where your audience is coming from and ask others for feedback on your project page to figure out which of these two issues youâre facing.
Once you know that people are using your project, you might want to try to figure out what they are doing with it. Are they building on it by forking your code and adding features? Are they using it for science or business?
Retention
People are finding your project and theyâre using it. The next question youâll want to ask yourself is: are people contributing back to this project?
Itâs never too early to start thinking about contributors. Without other people pitching in, you risk putting yourself into an unhealthy situation where your project is popular (many people use it) but not supported (not enough maintainer time to meet demand).
Retention also requires an inflow of new contributors, as previously active contributors will eventually move on to other things.
Examples of community metrics that you may want to regularly track include:
- Total contributor count and number of commits per contributor: Tells you how many contributors you have, and whoâs more or less active. On GitHub, you can view this under âInsightsâ -> âContributors.â Right now, this graph only counts contributors who have committed to the default branch of the repository.
-
First time, casual, and repeat contributors: Helps you track whether youâre getting new contributors, and whether they come back. (Casual contributors are contributors with a low number of commits. Whether thatâs one commit, less than five commits, or something else is up to you.) Without new contributors, your projectâs community can become stagnant.
-
Number of open issues and open pull requests: If these numbers get too high, you might need help with issue triaging and code reviews.
-
Number of opened issues and opened pull requests: Opened issues means somebody cares enough about your project to open an issue. If that number increases over time, it suggests people are interested in your project.
-
Types of contributions: For example, commits, fixing typos or bugs, or commenting on an issue.
Maintainer activity
Finally, youâll want to close the loop by making sure your projectâs maintainers are able to handle the volume of contributions received. The last question youâll want to ask yourself is: am I (or are we) responding to our community?
Unresponsive maintainers become a bottleneck for open source projects. If someone submits a contribution but never hears back from a maintainer, they may feel discouraged and leave.
Research from Mozilla suggests that maintainer responsiveness is a critical factor in encouraging repeat contributions.
Consider tracking how long it takes for you (or another maintainer) to respond to contributions, whether an issue or a pull request. Responding doesnât require taking action. It can be as simple as saying: âThanks for your submission! Iâll review this within the next week.â
You could also measure the time it takes to move between stages in the contribution process, such as:
- Average time an issue remains open
- Whether issues get closed by PRs
- Whether stale issues get closed
- Average time to merge a pull request
Use đ to learn about people
Understanding metrics will help you build an active, growing open source project. Even if you donât track every metric on a dashboard, use the framework above to focus your attention on the type of behavior that will help your project thrive.
CHAOSS is a welcoming, open source community focused on analytics, metrics and software for community health.
