Posts Tagged ‘development’
When I’ve wanted to play in some new areas lately, it’s been a real frustration because Gentoo hasn’t had a complete set of packages ready in any of them. I feel like these are some opportunities for Gentoo to be awesome and gain access to new sets of users (or at least avoid chasing away existing users who want better tools):
- Data science. Package Hadoop. Package streaming options like Storm. How about related tools like Flume? RabbitMQ is in Gentoo, though. I’ve heard anecdotally that a well-optimized Hadoop-on-Gentoo installation showed double-digit performance increases over the usual Hadoop distributions (i.e., not Linux distributions, but companies specializing in providing Hadoop solutions). Just heard from Tim Harder (radhermit) than he’s got some packages in progress for a lot of this, which is great news.
- DevOps. This is an area where Gentoo historically did pretty well, in part because our own infrastructure team and the group at the Open Source Lab have run tools like CFEngine and Puppet. But we’re lagging behind the times. We don’t have Jenkins or Travis. Seriously? Although we’ve got Vagrant packaged, for example, we don’t have Veewee. We could be integrating the creation of Vagrant boxes into our release-engineering process.
- Relatedly: Monitoring. Look for some of the increasingly popular open-source tools today, things like Graphite, StatsD, Logstash, Lumberjack, ElasticSearch, Kibana, Sensu, Tasseo, Descartes, Riemann. None of those are there.
- Cloud. Public cloud and on-premise IaaS/PaaS. How about IaaS: OpenStack, CloudStack, Eucalyptus, or OpenNebula? Not there, although some work is happening for OpenStack according to Matthew Thode (prometheanfire). How about a PaaS like Cloud Foundry or OpenShift? Nope. None of the Netflix open-source tools are there. On the public side, things are a bit better — we’ve got lots of AWS tools packaged, even stretching to things like Boto. We could be integrating the creation of AWS images into our release engineering to ensure AWS users always have a recent, official Gentoo image.
- NoSQL. We’ve got a pretty decent set here with some holes. We’ve got Redis, Mongo, and CouchDB not to mention Memcached, but how about graph databases like Neo4j, or other key-value stores like Riak, Cassandra, or Voldemort?
- Android development. Gentoo is perfect as a development environment. We should be pushing it hard for mobile development, especially Android given its Linux base. There’s a couple of halfhearted wiki pages but that does not an effort make. If the SDKs and related packages are there, the docs need to be there too.
Where does Gentoo shine? As a platform for developers, as a platform for flexibility, as a platform to eke every last drop of performance out of a system. All of the above use cases are relevant to at least one of those areas.
I’m writing this post because I would love it if anyone else who wants to help Gentoo be more awesome would chip in with packaging in these specific areas. Let me know!
Update: Michael Stahnke suggested I point to some resources on Gentoo packaging, for anyone interested, so take a look at the Gentoo Development Guide. The Developer Handbook contains some further details on policy as well as info on how to get commit access by becoming a Gentoo developer.
As one of my four talks at FOSDEM, I gave one on Gentoo titled “Package management and creation in Gentoo Linux.” The basic idea was, what could packagers and developers of other, non-Gentoo distros learn from Gentoo’s packaging format and how it’s iterated on that format multiple times over the years. It’s got some slides but the interesting part is where we run through actual ebuilds to see how they’ve changed as we’ve advanced through EAPIs (Ebuild APIs), starting at 16:39.
If you click through to YouTube, the larger (but not fullscreen) version seems to be the easiest to read.
It was scaled from 720×576 to a 480p video, so if you find it too hard to read the code, you can view the original WebM here.
App developers and end users both like bundled software, because it’s easy to support and easy for users to get up and running while minimizing breakage. How could we come up with an approach that also allows distributions and package-management frameworks to integrate well and deal with issues like security? I muse upon this over at my RedMonk blog.
Students, this Friday at 1900 UTC is the deadline to apply for this year’s GSoC. It’s an awesome program that pays you to work on open-source projects for a summer (where you == a university/college student).
It’s by no means too late, but start your application today. You can find more information on Gentoo’s projects here (click on the Ideas page to get started; also see our application guidelines) and on the broader GSoC program here.
To my European readers: if you care about the impact of social technologies like Git (and GitHub) & how they’re transforming software development, or the impact of social technology on communities, and you enjoy good beer, you need to be at Monki Gras. I just posted over at my RedMonk blog about how the previous conference in the series, Monktoberfest, was the best conference of my life. And I’ve been to many.
Monki Gras is Feb. 1–2 in London. The timing’s perfect to stop by just before FOSDEM (and that’s exactly what I’m doing). Registration is dirt-cheap, speakers are universally top-notch, and you’ll also get some world-class beers in the package.
I just published an article over at LWN called “The state of Gentoo.” In it, I talk about Gentoo’s progress over the past few years as well as its current problems, how they’re affecting Gentoo’s user and developer communities, and how to begin fixing them.
I’ve gotten a lot of positive feedback from our developers on the article so I’m confident that it’s generally a fair representation of where things stand today. However, Alex Legler (a3li) kindly told me our security team does much more behind the scenes to report vulnerabilities and get updated packages stabilized, even if we don’t see the results in the form of GLSAs.
I’m running for the Gentoo council this year, and I wanted to post my “platform” here: who I am, why I’m running, how I see the council, and how I see Gentoo.
For anyone newer, I’d like to first introduce myself and my history with Gentoo. Over the past 8 years, I’ve spent time doing pretty much everything in Gentoo, from maintaining ebuilds to writing documentation and leading teams and projects. I’ve been a recruiter in devrel, served as desktop manager (back when we had top-level project managers, before GLEP 39), and later chaired the council. I also started the clustering team, led X11 maintenance for about 5 years (when I designed the modular X eclass and wrote all 400+ new modular packages as we transitioned from the old XFree86), spent a term as a foundation trustee, and acted as project manager for our ever-controversial GUI installer.
One of my primary roles at the moment is running our participation in the Google Summer of Code, one of our major sources of both income and new developers. Since I took over 3 years ago, we’ve more than doubled the size of our program (we have 15 students this year!) and greatly increased the proportion of students who eventually become Gentoo devs to roughly 67%.
At this point, I’m convinced I can make the biggest difference to Gentoo in two ways: focusing on specific projects like the git transition and improved eclass testing rather than ebuild maintenance (which we have tons of people doing), and leading Gentoo to greatness as a council member, where I’m in the best position to accomplish that goal.
On my last two terms as a council member, I was extremely active in council-related issues both on mailing lists and at meetings, rather than just showing up at meetings to vote and then disappearing into the sunset. I strongly believe that through preparation and accountability, we can have a more productive council, but this requires a true commitment on the part of every member.
Creating a great community
I see this as an area where the council can set direction for all of Gentoo, both through the positive examples of its members and by taking a stand against anything we won’t stand for in our community. The council should be providing guidance to devrel and userrel on what kinds of behaviors should be encouraged and what kinds shouldn’t be tolerated.
Additionally, we should begin collecting metrics on our community so we know where we’re going. Many other projects (and even distributions) do this already, to give them clues of when things are going wrong, what things are going right, etc.
Making progress now instead of waiting years for perfection
There’s a lot of ongoing projects and GLEPs that have been dragging on for years because people want a “perfect” solution. One personal example is the git transition, where I’ve already made this mistake in spending quite a bit of time trying to get a perfect repository layout working when we could have just done a simple conversion of the whole tree and put it in one repo (the current plan, with some changes).
GLEP 55 is another one. There are some times when any decision is better than no decision.
Removing bureaucracy in favor of more agile development & meritocracy
We all know that GLEP 39 needs some changes. I think that most devs just want to Get Stuff Done without fighting with policies, waiting around forever for votes, not getting majorities without any clear direction, and so forth. I believe that our devs vote in the council because they want those people in charge and trust the council members to do what they think is right, including deciding on changes to GLEP 39 to improve how we run things.
I favor a switch to a smaller group more along the lines of the Linux kernel, with a very small leadership core and more of a hierarchy than a huge committee.
Along the same lines, I favor a switch to a more meritocratic approach such as the lead appointments we’ve been discussing recently. Gentoo isn’t a country government that needs to provide for every sick and needy person in its geography; it’s a nonprofit with specific goals it needs to accomplish and should be run as such. In general, companies, including nonprofits, don’t have checks and balances or long, drawn-out votes. What they do have is a board of trustees at the top, which can replace the leadership when it deems necessary.
Another major problem is the lack of continuity in our leadership, mainly because the whole group is re-elected en masse every year. At a minimum, even if we can’t do the above changes, we should switch to a 2-year term and have half the council up for election each year so new councils can start out quickly.
Promoting innovation from individual developers instead of expecting it to all come from the top
As I mentioned above, one of my focuses now is working on interesting standalone projects. I hope that every developer does the same; in addition, or besides whatever you do every day (maintain ebuilds, write/translate docs, etc.), consider starting a new project that has a clear end in sight and that will help Gentoo get better. It doesn’t have to be anything big; and the great thing about projects with a finite length is that you actually finish them at some point.
Getting rid of policies that were created for a single incident, because they should only exist for patterns of repeated problems
A common problem in Gentoo is feeling like a one-time problem can’t be fixed without making a big policy about it, why it can’t be done, what will happen if it’s done again, etc. If you’ve been around Gentoo for a while and start reading through our developer docs, you’ll be able to connect a specific name and instance with pretty much every line that says something like “Don’t do stupid thing X” and “Don’t do ridiculous thing Y.”
I think part of this is rooted in a fear, for lack of a better word, to take personal responsibility for an action; instead, people want a rulebook to point at and say “That’s wrong” so the rulebook is the bad guy.
But rulebooks should only be for the common cases, the patterns, the problems that keep coming up over and over. One-time problems only deserve one-time solutions.
One of the most powerful aspects of popular high-level languages is the existence of a comprehensive standard library. Unfortunately, the most popular Linux shell, bash, lacks a full-featured library bundled with it. A number of people have written libraries to compensate for this lack, which can make your life infinitely easier if you need to write complex bash scripts. Here are links to all the bash/shell-scripting libraries and collections I’m aware of:
- Marco’s Bash Functions Library (mbfl)
- Bash Shell Function Library (bsfl)
- Bashinator: Bash Shell Script Framework (by our own wschlich)
- shesfw: shell script framework tool
- Wicked Cool Shell Scripts library (from the book)
- UNIX Power Tools library (from the book)
- Portable Shell Programming (from the book)
- Learning the bash shell (from the book)
- Bash Cookbook (from the book)
- Classic Shell Scripting (from the book)
- shunit2: A unit-test framework
- log4sh: A flexible logging framework
- libbash: Enables creation of dynamic-like shared libraries
- bashworks: Framework depending on bash-4 or newer
- bfw: Bash FrameWork (Harvard Neuroinformatics)
- rerun: Turns loose shell scripts into modular automation
It was a real pain to come up with this whole list, so hopefully it’s useful to some of you. In future posts, I’m thinking of reviewing these libraries in more depth and comparing them to equivalent functionality in Python, my other language of choice. My goal is not to have functionality re-implemented in shell script to replace basic tools like head/tail, but instead to discover or create (if necessary) a library using the full suite of standard UNIX tools to provide a fun and convenient experience for shell scripters.
Update (2013/07/08): Added rerun.
Update: Fixed a couple of broken links (bashworks and Portable Shell).
Similarly, you need to quickly build the same level of trust with our community. Do you have the tools it takes to hack into the code? and do you know how to use them? When a contributor uploads a patch to our mailing list or to our tracker (preferred, but to make sure you get noticed please post a link on the mailing list too) he builds instantaneously the same level of trust that the unknown cab driver builds with every new customer: he implicitly shows that he has the tools (hardware, operating system, internet connection, coding environment, repository access) and that he can use them for basic operations.
A post I made on the Gentoo development list last night deserves your attention because it illustrates a key aspect of Gentoo. I want to make sure it’s preserved instead of lost in mailing-list archives. This is in reply to a proposal to move a significant portion into a series of variables instead of being written as scripted functions:
I think the idea of ebuilds as scripts showing directly how to build software is a core part of the Gentoo build-system philosophy. This proposal pushes ebuilds toward a formatted file that is not a script. Instead, it is more like an Ant XML file that more abstractly describes a build. I think this is the wrong direction for ebuilds because they should directly resemble how software is built by hand.
One of the key reasons people use Gentoo is that ebuilds are so easy to “get” for anyone who has ever built software by hand. I will continue to vehemently defend anything that I think retains this key advantage of Gentoo over other distributions.
Gentoo is a distribution for advanced Linux users, or those who want to become advanced. Having packaging scripts that are easy to learn lowers the barrier to entry to editing them, which vastly increases our number of potential developers. It additionally comprises a key part of Gentoo’s ease of customization by allowing people to easily write their own packages or modify existing ones for their purposes. When users do this, it creates a natural pathway from building software by hand to editing ebuilds to becoming a Gentoo developer. The more difficult it is to learn how to write packaging scripts, the more of your potential developer base you repel.