Posts Tagged ‘greatness’
When I’ve wanted to play in some new areas lately, it’s been a real frustration because Gentoo hasn’t had a complete set of packages ready in any of them. I feel like these are some opportunities for Gentoo to be awesome and gain access to new sets of users (or at least avoid chasing away existing users who want better tools):
- Data science. Package Hadoop. Package streaming options like Storm. How about related tools like Flume? RabbitMQ is in Gentoo, though. I’ve heard anecdotally that a well-optimized Hadoop-on-Gentoo installation showed double-digit performance increases over the usual Hadoop distributions (i.e., not Linux distributions, but companies specializing in providing Hadoop solutions). Just heard from Tim Harder (radhermit) than he’s got some packages in progress for a lot of this, which is great news.
- DevOps. This is an area where Gentoo historically did pretty well, in part because our own infrastructure team and the group at the Open Source Lab have run tools like CFEngine and Puppet. But we’re lagging behind the times. We don’t have Jenkins or Travis. Seriously? Although we’ve got Vagrant packaged, for example, we don’t have Veewee. We could be integrating the creation of Vagrant boxes into our release-engineering process.
- Relatedly: Monitoring. Look for some of the increasingly popular open-source tools today, things like Graphite, StatsD, Logstash, Lumberjack, ElasticSearch, Kibana, Sensu, Tasseo, Descartes, Riemann. None of those are there.
- Cloud. Public cloud and on-premise IaaS/PaaS. How about IaaS: OpenStack, CloudStack, Eucalyptus, or OpenNebula? Not there, although some work is happening for OpenStack according to Matthew Thode (prometheanfire). How about a PaaS like Cloud Foundry or OpenShift? Nope. None of the Netflix open-source tools are there. On the public side, things are a bit better — we’ve got lots of AWS tools packaged, even stretching to things like Boto. We could be integrating the creation of AWS images into our release engineering to ensure AWS users always have a recent, official Gentoo image.
- NoSQL. We’ve got a pretty decent set here with some holes. We’ve got Redis, Mongo, and CouchDB not to mention Memcached, but how about graph databases like Neo4j, or other key-value stores like Riak, Cassandra, or Voldemort?
- Android development. Gentoo is perfect as a development environment. We should be pushing it hard for mobile development, especially Android given its Linux base. There’s a couple of halfhearted wiki pages but that does not an effort make. If the SDKs and related packages are there, the docs need to be there too.
Where does Gentoo shine? As a platform for developers, as a platform for flexibility, as a platform to eke every last drop of performance out of a system. All of the above use cases are relevant to at least one of those areas.
I’m writing this post because I would love it if anyone else who wants to help Gentoo be more awesome would chip in with packaging in these specific areas. Let me know!
Update: Michael Stahnke suggested I point to some resources on Gentoo packaging, for anyone interested, so take a look at the Gentoo Development Guide. The Developer Handbook contains some further details on policy as well as info on how to get commit access by becoming a Gentoo developer.
“The Hype Cycle describes the way that new technologies and projects are perceived over time, if they do a good job of handling themselves, going from a technology trigger, inflated expectations, disillusionment, enlightenment, before arriving at “the plateau of productivity” – a state where there is no more hype and the new technology is simply a normal part of our lives.”
The perception over the past few years that Gentoo is dying is in reality Gentoo’s arrival at the plateau of productivity. Hype has gone away and remaining is a distribution with a true niche that fits into the broader Linux ecosystem.
Three factors must be present for meaningful organizational change to take place. These factors are:
D = Dissatisfaction with the status quo;
V = Vision of what is possible;
F = First, concrete steps that can be taken towards the vision.
R = If any of these factors are missing or weak, then you’re going to get resistance.
It is important for leaders of organizations going through change to realize the ‘lag time’ between where they are in the process and where others in the organizations are in the process.
The higher a leader sits in an organization the more quickly he or she tends to move through the change process. Because they can see the intended destination before others even know the race has begun, senior managers can forget that others will take longer to make the transition: letting go of old ways, moving through the neutral zone, and, finally, making a new beginning.
I laid out part of a framework for improving Gentoo in my last post. As many of you kindly noted, I explained what I wanted to do but did a poor job of convincing you why it needs to happen. That’s what I’m going to begin doing in this post.
As this post’s title says, when you develop code, you generally want it to be the best code possible. You also want to improve it over time if it’s not yet up to the highest standards. This holds doubly true when you’re doing this for free in an open-source project. First, there’s no time pressure to pound out minimally working code, so you can take the time to make it beautiful. Second, everyone else can look at your work, so you can’t hide ugly code in a dusty corner somewhere as you might be able to in proprietary software.
These same concepts hold true for you, not just your code. When you’re contributing to an open-source project, everyone sees what you do and how you do it. Strive to make yourself beautiful in the context of open-source development. What’s that mean? Always challenge yourself, grow as a person, and learn new skills. Just as beautiful code perfectly suits its purpose, beautiful developers perfectly suit their purpose. Their skills and actions exemplify how open-source developers should act.
If you’re optimizing code, you know that you can’t optimize what you can’t measure. That’s why profilers and other methods of instrumentation exist. This concept has a parallel in your personal development. Your instrumentation is goals, and what you’re measuring is your performance against your own goals. The purpose of this, as with code, is to optimize performance for your relevant metrics. How do you know whether you’ve improved unless you’re profiling yourself? What you’re left with if you don’t set goals and track progress against them is that frustrating comment people always say, “It just feels faster.”
Having the right people in the right places pursuing the right goals is key to Gentoo’s success. Keeping in mind the Pareto principle (20% of the effort produces 80% of the results), I’ve come up with some ideas. The core of these ideas is creating a development community composed only of top-notch contributors and putting the entire Gentoo project into a cycle of continuous improvement, from the bottom up.
Each team and project should have goals. A good goal is SMART (Specific, Measurable, Attainable, Relevant, and Time-bound). That means “Maintain foo packages” is a poor goal, because you cannot answer it in a quantitative way and it has no time limit. One example of a good goal is “6 months from now, 90% of new bugs will be closed within 2 weeks of their report.” Another example is “3 months from now, 100% of our packages will be bumped within 1 week of release, or a bug will exist describing why not.” Progress toward goals should be checked once a month by the team lead and reported to the project lead (or to the council, if no project lead exists).
Each team member should have goals. The performance of that contributor in the context of that team should be measured. People who aren’t up to the team’s high standards (measured by failure to meet goals) should receive coaching from the team lead. There should be a concrete plan for improvement, committed to by both sides, for how these people will reach the high standards they aren’t hitting yet. After a plan exists, failure to improve within 3 months will result in their departure from the team. At the team lead’s judgment, people may be removed without going through coaching if they are considered a lost cause. Progress toward goals should be checked once a month by the team lead.
To minimize the chance that an underperforming developer was simply not in the right place, each developer gets 3 chances to find the right place in Gentoo. Upon removal from the first 2 teams, the team lead will report what happened to the project lead (or to the council, if no project lead exists) and to developer relations. On the 3rd strike, you’re out.
Council members and project leads should take responsibility for mentoring the next generation of leaders. Just as team leads are helping to improve individual developers, so the next tier of leadership should be developing team leads into strong project leads and council members. Nobody sticks around forever, and we need to be ready for when even our most critical developers move on. It’s far better to have too many strong potential leaders than to have a vacuum that’s filled by a poor leader.
Finally, all of our goals should flow downward in a logical manner from the Gentoo mission & vision through projects to teams and individual developers. Gentoo’s perennial problem is a lack of focus. Any work that is inconsistent with our goals should be stopped, and we should rededicate our time to things that tie in with where we want to be next year and 5 years from now. To ensure this, goals at each level should be reviewed by whoever’s at the next level up to make sure things make sense at a global level all the way down, like a waterfall.
Now is a good time to re-read the Gentoo philosophy. I find it extremely powerful and compelling, and I hope you feel the same. We should live this philosophy every day in our work with Gentoo, and every aspect of our work should grow from it:
Every user has work they need to do. The goal of Gentoo is to design tools and systems that allow a user to do that work as pleasantly and efficiently as possible, as they see fit. Our tools should be a joy to use, and should help the user to appreciate the richness of the Linux and free software community, and the flexibility of free software. This is only possible when the tool is designed to reflect and transmit the will of the user, and leave the possibilities open as to the final form of the raw materials (the source code.) If the tool forces the user to do things a particular way, then the tool is working against, rather than for, the user. We have all experienced situations where tools seem to be imposing their respective wills on us. This is backwards, and contrary to the Gentoo philosophy.
Put another way, the Gentoo philosophy is to create better tools. When a tool is doing its job perfectly, you might not even be very aware of its presence, because it does not interfere and make its presence known, nor does it force you to interact with it when you don’t want it to. The tool serves the user rather than the user serving the tool.
The goal of Gentoo is to strive to create near-ideal tools. Tools that can accommodate the needs of many different users all with divergent goals. Don’t you love it when you find a tool that does exactly what you want to do? Doesn’t it feel great? Our mission is to give that sensation to as many people as possible.
I’ve run out of time for tonight. Most of what I’ve written is about forward progress rather than improving individual developers, so I’ll try to write more about that another time. I think we need know how we’re doing if we ever want to improve, so this investment into goals and thinking about performance is worthwhile. What do you think?
Gentoo is good. How do we make it great?
Over the past year or so, I’ve read a few books, and I want to use those ideas to build a better Gentoo. The books:
- Good to Great, by Jim Collins
- The Five Dysfunctions of a Team, by Patrick Lencioni
- The Art of Project Management, by Scott Berkun
- The Power of Focus, by Jack Canfield, Mark Victor Hansen and Les Hewitt
I plan to write a short series of posts discussing the lessons in these books and how to apply them to Gentoo. In this post, I’m going to summarize the concepts of “Good to Great.” I’ll discuss how we can apply them to Gentoo in a later post. The book explains what it takes to transform a good company into a great one. It’s a comparison of companies that made a transition from good to great (thus the title) with companies that remained merely good. Jim Collins and his group reduced the differences to a remarkably small set:
- Level 5 leadership: The leaders of great companies aren’t charismatic, big-name CEOs. They’re humble, and their ambition is for the company, not for themselves.
- First, get the right people: Before you decide what to do and where to go, get the right people in the right spots. Otherwise, you’ve got the wrong people creating the wrong vision, strategy, etc., which the right people are then forced to implement. Concrete implementations:
- “When in doubt, don’t hire—keep looking. (Corollary: A company should limit its growth based on its ability to attract enough of the right people.)”
- “When you know you need to make a people change, act. (Corollary: First be sure you don’t simply have someone in the wrong seat.)”
- “Put your best people on your biggest opportunities, not your biggest problems. (Corollary: If you sell off your problems, don’t sell off your best people.)”
The right people have the right innate abilities and character traits, not necessarily the specific knowledge and skills. For example, look for work ethic, dedication, and problem-solving ability, not ebuild-writing skills and knowledge of bash.
- Confront the brutal facts: Don’t deny reality, or you can’t make the right decisions. To do this, create an environment where everyone can be heard, so the truth can come out. Don’t lose faith that you will win eventually, but accept the reality of now.
- The Hedgehog Concept: Only do things that overlap in the three circles:
- You’re passionate about it
- You can be the best in the world at it
- It drives your resource engine
The resource engine is a combination of time (how well we attract contributors), money and brand (how well we create a community), drawn from an accompanying monograph for the social sectors.
- Build a culture of discipline: Create a culture of self-disciplined people who are “fanatically consistent with the three circles.” Bureaucracy arises to compensate for incompetence and poor discipline due to having the wrong people. If you’ve got the right people, you don’t need the bureaucracy.
- Technology accelerators: If a new technology fits into your Hedgehog Concept, become a pioneer. If not, settle for decent, or drop it altogether if you can.
- The flywheel and the doom loop: Transforming from good to great is not quick. It’s a flywheel, slowly building momentum. From outside, all people see is the breakthrough, but from inside there’s a prior buildup. Merely good companies couldn’t build momentum—they jumped around from focus to focus, never getting anywhere with any of them. You don’t need to spend effort getting people behind your idea. Show them the results, and they’ll follow you.
I recently posted about making Gentoo a better tool. A requirement for being a good tool is being a tool that doesn’t break—thus, we need to improve our quality to a more reliable level. I’m going to mention a few ideas to start this discussion, which I hope the rest of our community will participate in.
First, we essentially have no code review. About the only time any code in Gentoo is reviewed is before and during a developer’s training, with a notable exception being the requirement to post eclasses to the gentoo-dev list. Increasing our code review ought to result in an increase in quality, in ability to justify code in words, and in a stronger community of contributors.
How do we increase code review? One idea is to require reviewer approval prior to committing, but this isn’t the best answer for Gentoo. We’ve always been a pretty open community. Developers aren’t prohibited by ACL from committing anywhere in our ebuild repository, so I don’t think they would accept additional requirements that increased the burden of contributing.
Instead, let’s create a gentoo-commits mailing list or RSS feed(s), with full diffs. We should use this tool in many different ways.
- Each team should use it internally to review all commits to its packages.
- Mentors should continue to follow their mentees’ commits well after they’re granted commit access (6 months minimum, and I recommend forever).
- Mentees should also review their mentors’ commits, first to learn and later to review.
- Every developer should have at least one reviewer and review at least one other developer. This should be formal and documented to ensure it’s happening.
These uses will require that the commit diffs be easily filterable by both committer and files affected. RSS feeds could be made available based on developer or herd, and e-mail lists could contain the information in e-mail subjects or headers.
Second, we should improve our unit testing, where the units are individual packages. This can be both automated and performed by developers and arch testers. Although a number of packages have a useful, working test suite, most lack one. For these packages, we should attempt to provide something automatable in src_test() even when a test suite is absent. Failing that, we should print out a checklist in src_test() of tests to perform before stabilizing a package. There should never be an empty src_test().
Another package-level testing approach is to create solid, automated tinderboxes. This remains unrealistic for our entire database of 10,000+ packages, but we should at least get this going for our “system” set and perhaps for some of the most common sets of packages for servers and desktops. Exactly how to set this up remains a question, since there’s a lot of tinderbox code floating around. Bonsaikitten has some almost-working code based on swegener’s work; Catalyst has some tinderboxing capability; or we could look into using Mozilla’s tinderbox.
Third, we should improve our integration testing, on the entire repository level. Our main source of testing here will be our users, because they have infinitely more combinations of build options and hardware than we can reproduce on Gentoo infrastructure. But how can we take advantage of this testing to improve our quality? By creating an additional, time-lagged set of rsync mirrors with additional QA checks, we could allow users who want to test the latest and greatest software to help those who want stable and solid software.
We already have keywords for ~arch and arch, but they’re still too mixed. A problem in ~arch ebuilds can break the entire tree for all users. They really need a stronger separation. Perhaps the separate repositories should be ~arch versus stable. But another way to do it is to add a delay to the second set of repos, anywhere from 24 hours to a week. This delay allows us time to encounter major problems in the fast-sync repos, fix them, and carry the fixes over to the slow-sync repo. But we’ll need a way to make this really easy to do. It feels like branching with periodic merges, along with cherry picks of major bugfixes, is the right way to do this. Unfortunately, CVS sucks at this. We may need to migrate to a more capable version-control system before this option becomes realistic. In addition to the user testing, we could add a tinderbox into the slow-sync repos to require that they build with the most common configurations.
To sum up, I want to increase code review, unit testing, and integration testing. These three things will strengthen Gentoo’s quality, reputation, and community.