Posts Tagged ‘greatness’
When I’ve wanted to play in some new areas lately, it’s been a real frustration because Gentoo hasn’t had a complete set of packages ready in any of them. I feel like these are some opportunities for Gentoo to be awesome and gain access to new sets of users (or at least avoid chasing away existing users who want better tools):
- Data science. Package Hadoop. Package streaming options like Storm. How about related tools like Flume? RabbitMQ is in Gentoo, though. I’ve heard anecdotally that a well-optimized Hadoop-on-Gentoo installation showed double-digit performance increases over the usual Hadoop distributions (i.e., not Linux distributions, but companies specializing in providing Hadoop solutions). Just heard from Tim Harder (radhermit) than he’s got some packages in progress for a lot of this, which is great news.
- DevOps. This is an area where Gentoo historically did pretty well, in part because our own infrastructure team and the group at the Open Source Lab have run tools like CFEngine and Puppet. But we’re lagging behind the times. We don’t have Jenkins or Travis. Seriously? Although we’ve got Vagrant packaged, for example, we don’t have Veewee. We could be integrating the creation of Vagrant boxes into our release-engineering process.
- Relatedly: Monitoring. Look for some of the increasingly popular open-source tools today, things like Graphite, StatsD, Logstash, Lumberjack, ElasticSearch, Kibana, Sensu, Tasseo, Descartes, Riemann. None of those are there.
- Cloud. Public cloud and on-premise IaaS/PaaS. How about IaaS: OpenStack, CloudStack, Eucalyptus, or OpenNebula? Not there, although some work is happening for OpenStack according to Matthew Thode (prometheanfire). How about a PaaS like Cloud Foundry or OpenShift? Nope. None of the Netflix open-source tools are there. On the public side, things are a bit better — we’ve got lots of AWS tools packaged, even stretching to things like Boto. We could be integrating the creation of AWS images into our release engineering to ensure AWS users always have a recent, official Gentoo image.
- NoSQL. We’ve got a pretty decent set here with some holes. We’ve got Redis, Mongo, and CouchDB not to mention Memcached, but how about graph databases like Neo4j, or other key-value stores like Riak, Cassandra, or Voldemort?
- Android development. Gentoo is perfect as a development environment. We should be pushing it hard for mobile development, especially Android given its Linux base. There’s a couple of halfhearted wiki pages but that does not an effort make. If the SDKs and related packages are there, the docs need to be there too.
Where does Gentoo shine? As a platform for developers, as a platform for flexibility, as a platform to eke every last drop of performance out of a system. All of the above use cases are relevant to at least one of those areas.
I’m writing this post because I would love it if anyone else who wants to help Gentoo be more awesome would chip in with packaging in these specific areas. Let me know!
Update: Michael Stahnke suggested I point to some resources on Gentoo packaging, for anyone interested, so take a look at the Gentoo Development Guide. The Developer Handbook contains some further details on policy as well as info on how to get commit access by becoming a Gentoo developer.
“The Hype Cycle describes the way that new technologies and projects are perceived over time, if they do a good job of handling themselves, going from a technology trigger, inflated expectations, disillusionment, enlightenment, before arriving at “the plateau of productivity” – a state where there is no more hype and the new technology is simply a normal part of our lives.”
The perception over the past few years that Gentoo is dying is in reality Gentoo’s arrival at the plateau of productivity. Hype has gone away and remaining is a distribution with a true niche that fits into the broader Linux ecosystem.
Three factors must be present for meaningful organizational change to take place. These factors are:
D = Dissatisfaction with the status quo;
V = Vision of what is possible;
F = First, concrete steps that can be taken towards the vision.
R = If any of these factors are missing or weak, then you’re going to get resistance.
It is important for leaders of organizations going through change to realize the ‘lag time’ between where they are in the process and where others in the organizations are in the process.
The higher a leader sits in an organization the more quickly he or she tends to move through the change process. Because they can see the intended destination before others even know the race has begun, senior managers can forget that others will take longer to make the transition: letting go of old ways, moving through the neutral zone, and, finally, making a new beginning.
I laid out part of a framework for improving Gentoo in my last post. As many of you kindly noted, I explained what I wanted to do but did a poor job of convincing you why it needs to happen. That’s what I’m going to begin doing in this post.
As this post’s title says, when you develop code, you generally want it to be the best code possible. You also want to improve it over time if it’s not yet up to the highest standards. This holds doubly true when you’re doing this for free in an open-source project. First, there’s no time pressure to pound out minimally working code, so you can take the time to make it beautiful. Second, everyone else can look at your work, so you can’t hide ugly code in a dusty corner somewhere as you might be able to in proprietary software.
These same concepts hold true for you, not just your code. When you’re contributing to an open-source project, everyone sees what you do and how you do it. Strive to make yourself beautiful in the context of open-source development. What’s that mean? Always challenge yourself, grow as a person, and learn new skills. Just as beautiful code perfectly suits its purpose, beautiful developers perfectly suit their purpose. Their skills and actions exemplify how open-source developers should act.
If you’re optimizing code, you know that you can’t optimize what you can’t measure. That’s why profilers and other methods of instrumentation exist. This concept has a parallel in your personal development. Your instrumentation is goals, and what you’re measuring is your performance against your own goals. The purpose of this, as with code, is to optimize performance for your relevant metrics. How do you know whether you’ve improved unless you’re profiling yourself? What you’re left with if you don’t set goals and track progress against them is that frustrating comment people always say, “It just feels faster.”
Having the right people in the right places pursuing the right goals is key to Gentoo’s success. Keeping in mind the Pareto principle (20% of the effort produces 80% of the results), I’ve come up with some ideas. The core of these ideas is creating a development community composed only of top-notch contributors and putting the entire Gentoo project into a cycle of continuous improvement, from the bottom up.
Each team and project should have goals. A good goal is SMART (Specific, Measurable, Attainable, Relevant, and Time-bound). That means “Maintain foo packages” is a poor goal, because you cannot answer it in a quantitative way and it has no time limit. One example of a good goal is “6 months from now, 90% of new bugs will be closed within 2 weeks of their report.” Another example is “3 months from now, 100% of our packages will be bumped within 1 week of release, or a bug will exist describing why not.” Progress toward goals should be checked once a month by the team lead and reported to the project lead (or to the council, if no project lead exists).
Each team member should have goals. The performance of that contributor in the context of that team should be measured. People who aren’t up to the team’s high standards (measured by failure to meet goals) should receive coaching from the team lead. There should be a concrete plan for improvement, committed to by both sides, for how these people will reach the high standards they aren’t hitting yet. After a plan exists, failure to improve within 3 months will result in their departure from the team. At the team lead’s judgment, people may be removed without going through coaching if they are considered a lost cause. Progress toward goals should be checked once a month by the team lead.
To minimize the chance that an underperforming developer was simply not in the right place, each developer gets 3 chances to find the right place in Gentoo. Upon removal from the first 2 teams, the team lead will report what happened to the project lead (or to the council, if no project lead exists) and to developer relations. On the 3rd strike, you’re out.
Council members and project leads should take responsibility for mentoring the next generation of leaders. Just as team leads are helping to improve individual developers, so the next tier of leadership should be developing team leads into strong project leads and council members. Nobody sticks around forever, and we need to be ready for when even our most critical developers move on. It’s far better to have too many strong potential leaders than to have a vacuum that’s filled by a poor leader.
Finally, all of our goals should flow downward in a logical manner from the Gentoo mission & vision through projects to teams and individual developers. Gentoo’s perennial problem is a lack of focus. Any work that is inconsistent with our goals should be stopped, and we should rededicate our time to things that tie in with where we want to be next year and 5 years from now. To ensure this, goals at each level should be reviewed by whoever’s at the next level up to make sure things make sense at a global level all the way down, like a waterfall.
Now is a good time to re-read the Gentoo philosophy. I find it extremely powerful and compelling, and I hope you feel the same. We should live this philosophy every day in our work with Gentoo, and every aspect of our work should grow from it:
Every user has work they need to do. The goal of Gentoo is to design tools and systems that allow a user to do that work as pleasantly and efficiently as possible, as they see fit. Our tools should be a joy to use, and should help the user to appreciate the richness of the Linux and free software community, and the flexibility of free software. This is only possible when the tool is designed to reflect and transmit the will of the user, and leave the possibilities open as to the final form of the raw materials (the source code.) If the tool forces the user to do things a particular way, then the tool is working against, rather than for, the user. We have all experienced situations where tools seem to be imposing their respective wills on us. This is backwards, and contrary to the Gentoo philosophy.
Put another way, the Gentoo philosophy is to create better tools. When a tool is doing its job perfectly, you might not even be very aware of its presence, because it does not interfere and make its presence known, nor does it force you to interact with it when you don’t want it to. The tool serves the user rather than the user serving the tool.
The goal of Gentoo is to strive to create near-ideal tools. Tools that can accommodate the needs of many different users all with divergent goals. Don’t you love it when you find a tool that does exactly what you want to do? Doesn’t it feel great? Our mission is to give that sensation to as many people as possible.
I’ve run out of time for tonight. Most of what I’ve written is about forward progress rather than improving individual developers, so I’ll try to write more about that another time. I think we need know how we’re doing if we ever want to improve, so this investment into goals and thinking about performance is worthwhile. What do you think?
Gentoo is good. How do we make it great?
Over the past year or so, I’ve read a few books, and I want to use those ideas to build a better Gentoo. The books:
- Good to Great, by Jim Collins
- The Five Dysfunctions of a Team, by Patrick Lencioni
- The Art of Project Management, by Scott Berkun
- The Power of Focus, by Jack Canfield, Mark Victor Hansen and Les Hewitt
I plan to write a short series of posts discussing the lessons in these books and how to apply them to Gentoo. In this post, I’m going to summarize the concepts of “Good to Great.” I’ll discuss how we can apply them to Gentoo in a later post. The book explains what it takes to transform a good company into a great one. It’s a comparison of companies that made a transition from good to great (thus the title) with companies that remained merely good. Jim Collins and his group reduced the differences to a remarkably small set:
- Level 5 leadership: The leaders of great companies aren’t charismatic, big-name CEOs. They’re humble, and their ambition is for the company, not for themselves.
- First, get the right people: Before you decide what to do and where to go, get the right people in the right spots. Otherwise, you’ve got the wrong people creating the wrong vision, strategy, etc., which the right people are then forced to implement. Concrete implementations:
- “When in doubt, don’t hire—keep looking. (Corollary: A company should limit its growth based on its ability to attract enough of the right people.)”
- “When you know you need to make a people change, act. (Corollary: First be sure you don’t simply have someone in the wrong seat.)”
- “Put your best people on your biggest opportunities, not your biggest problems. (Corollary: If you sell off your problems, don’t sell off your best people.)”
The right people have the right innate abilities and character traits, not necessarily the specific knowledge and skills. For example, look for work ethic, dedication, and problem-solving ability, not ebuild-writing skills and knowledge of bash.
- Confront the brutal facts: Don’t deny reality, or you can’t make the right decisions. To do this, create an environment where everyone can be heard, so the truth can come out. Don’t lose faith that you will win eventually, but accept the reality of now.
- The Hedgehog Concept: Only do things that overlap in the three circles:
- You’re passionate about it
- You can be the best in the world at it
- It drives your resource engine
The resource engine is a combination of time (how well we attract contributors), money and brand (how well we create a community), drawn from an accompanying monograph for the social sectors.
- Build a culture of discipline: Create a culture of self-disciplined people who are “fanatically consistent with the three circles.” Bureaucracy arises to compensate for incompetence and poor discipline due to having the wrong people. If you’ve got the right people, you don’t need the bureaucracy.
- Technology accelerators: If a new technology fits into your Hedgehog Concept, become a pioneer. If not, settle for decent, or drop it altogether if you can.
- The flywheel and the doom loop: Transforming from good to great is not quick. It’s a flywheel, slowly building momentum. From outside, all people see is the breakthrough, but from inside there’s a prior buildup. Merely good companies couldn’t build momentum—they jumped around from focus to focus, never getting anywhere with any of them. You don’t need to spend effort getting people behind your idea. Show them the results, and they’ll follow you.
I recently posted about making Gentoo a better tool. A requirement for being a good tool is being a tool that doesn’t break—thus, we need to improve our quality to a more reliable level. I’m going to mention a few ideas to start this discussion, which I hope the rest of our community will participate in.
First, we essentially have no code review. About the only time any code in Gentoo is reviewed is before and during a developer’s training, with a notable exception being the requirement to post eclasses to the gentoo-dev list. Increasing our code review ought to result in an increase in quality, in ability to justify code in words, and in a stronger community of contributors.
How do we increase code review? One idea is to require reviewer approval prior to committing, but this isn’t the best answer for Gentoo. We’ve always been a pretty open community. Developers aren’t prohibited by ACL from committing anywhere in our ebuild repository, so I don’t think they would accept additional requirements that increased the burden of contributing.
Instead, let’s create a gentoo-commits mailing list or RSS feed(s), with full diffs. We should use this tool in many different ways.
- Each team should use it internally to review all commits to its packages.
- Mentors should continue to follow their mentees’ commits well after they’re granted commit access (6 months minimum, and I recommend forever).
- Mentees should also review their mentors’ commits, first to learn and later to review.
- Every developer should have at least one reviewer and review at least one other developer. This should be formal and documented to ensure it’s happening.
These uses will require that the commit diffs be easily filterable by both committer and files affected. RSS feeds could be made available based on developer or herd, and e-mail lists could contain the information in e-mail subjects or headers.
Second, we should improve our unit testing, where the units are individual packages. This can be both automated and performed by developers and arch testers. Although a number of packages have a useful, working test suite, most lack one. For these packages, we should attempt to provide something automatable in src_test() even when a test suite is absent. Failing that, we should print out a checklist in src_test() of tests to perform before stabilizing a package. There should never be an empty src_test().
Another package-level testing approach is to create solid, automated tinderboxes. This remains unrealistic for our entire database of 10,000+ packages, but we should at least get this going for our “system” set and perhaps for some of the most common sets of packages for servers and desktops. Exactly how to set this up remains a question, since there’s a lot of tinderbox code floating around. Bonsaikitten has some almost-working code based on swegener’s work; Catalyst has some tinderboxing capability; or we could look into using Mozilla’s tinderbox.
Third, we should improve our integration testing, on the entire repository level. Our main source of testing here will be our users, because they have infinitely more combinations of build options and hardware than we can reproduce on Gentoo infrastructure. But how can we take advantage of this testing to improve our quality? By creating an additional, time-lagged set of rsync mirrors with additional QA checks, we could allow users who want to test the latest and greatest software to help those who want stable and solid software.
We already have keywords for ~arch and arch, but they’re still too mixed. A problem in ~arch ebuilds can break the entire tree for all users. They really need a stronger separation. Perhaps the separate repositories should be ~arch versus stable. But another way to do it is to add a delay to the second set of repos, anywhere from 24 hours to a week. This delay allows us time to encounter major problems in the fast-sync repos, fix them, and carry the fixes over to the slow-sync repo. But we’ll need a way to make this really easy to do. It feels like branching with periodic merges, along with cherry picks of major bugfixes, is the right way to do this. Unfortunately, CVS sucks at this. We may need to migrate to a more capable version-control system before this option becomes realistic. In addition to the user testing, we could add a tinderbox into the slow-sync repos to require that they build with the most common configurations.
To sum up, I want to increase code review, unit testing, and integration testing. These three things will strengthen Gentoo’s quality, reputation, and community.
Gentoo is a metadistribution, meaning you’re supposed to built whatever you want with it. And we provide some of the tools to make that a reality, but we stop short. We still make it too hard to do what you want with Gentoo. Here’s some of the ideas I want to explore over the next year:
- Increased binary package support: We already have an experimental tinderbox setup to build packages for a variety of architectures. I want to investigate increasing our collection of binary packages to make a fairly complete installation possible using only binaries.
- Repositories for user-created distributions: We have tools like Catalyst to create basic installations with stage3 or stage4 tarballs, but it’s still hard to get what you want with them. We should make a repository of Catalyst spec files for a variety of purposes, and we should also add an accompanying mirror of the created tarballs so anyone can use them. We could also revive the Seeds project to create not just vanilla stage4 tarballs but fully fledged, preconfigured installations customized for specific purposes like LAMP servers, development machines, various cluster nodes, GNOME/KDE/Xfce desktops, etc. We already have the possibilities; we need to share our tools.
- Creating the Gentoo metacommunity: In addition to letting you create your own distribution from Gentoo, I want Gentoo’s community to be what you want it to be. We’re in the middle of adding a few new mailing lists because our primary development list is drowning in noise. We made gentoo-project for non-technical general talk and gentoo-dev-announce for development-related information so developers don’t need to slog through gentoo-dev. I want to take that a step further by forming stronger Gentoo microcommunities around specific areas and moving discussion about those areas to their IRC channels and mailing lists. Our development community has grown too big to keep everything on a general development list. Everyone tries to chip into every discussion, even if they have no relation to it or are unaffected by it.
- Excellence: Gentoo’s QA is not the greatest. What can we do to improve this? Some automated tools exist for pre-commit checking; can we add anything server-side? Can we add some build servers for critical system packages, so they don’t make it to users before building in a predefined set of common configurations? Can we improve our developer recruitment or add new training for existing developers? How can we renew and strengthen our commitment to excellence? You can’t create a tool with Gentoo if it’s broken.
- New tools for new places: This is more of an exploratory idea. I’d like a team of contributors to research all the places people use Gentoo, and put together a collection of tools we’re missing. Further, we need to do more extensive research on other distributions to see what sorts of tools they have that we lack. I have no symptoms of NIH syndrome, so I’m happy to pull in working tools written for other distributions; I already did this for many of Fedora’s system-config-* tools last year.
- Resolving the meta vs. specificity conflict: Gentoo’s status as a metadistribution sometimes produces conflicts between the goals of various projects and teams within Gentoo. Sometimes a person simply needs to make a decision of how to deal with it: pick one, or find a way to do both? Many people remarked over the past couple of years that we’re lacking strong leadership and overarching goals, which would give us some consistent way to decide which projects will “win.” As Seemant has said, Gentoo is a framework, a metadistribution, not a place that forces people to go a certain way. We should strive to enable contributors to follow whichever path beckons to them. But where is Gentoo going, and where should it go? You decide. The innovators and leaders can only suggest places for it to go, so the rest of the community has to follow them to our destination.