Archive for the ‘Execution’ Category

SourceJuicer: How to contribute a package to OpenSolaris

May 25, 2009

[UPDATE: A few small errors fixed and some clarifications added. See Comments for details.]

I tried recently to add a package to the OpenSolaris contrib repository, but quickly learned I didn’t have enough packaging experience to understand the directions provided at SourceJuicer so I did some homework, asked some questions, and eventually did successfully contribute a package. I’ve documented in this entry everything I’ve learned hoping it will be helpful to others who want to build and submit OpenSolaris packages. Specifically, I’ll describe how I wrote the spec file for Ploticus (my favorite open source plotting/graphing utility) and how I submitted the package to OpenSolaris.

I used SourceJuicer to submit my package because it is the easiest way for a community member to contribute. Before getting into details, a few words about the overall submission process. Packages are first submitted to the pending repository, which is basically a holding area for packages on their way to the contrib repository, the primary repository for community-contributed packages. Once a package has been validated and successfully built, it can then be moved into /contrib. I’ll cover all of this below.

On to the details.

To submit a package to SourceJuicer, you need to supply two files: a text file containing copyright information and a spec file. The spec file contains the information SourceJuicer needs to create a final binary package starting from source code. Ideally the OpenSolaris package will be buildable from the standard, community-released source code without changes, which may require asking the community to adopt changes necessary to build the code for OpenSolaris. In practice, this will often not be necessary since many packages are designed to build on several Unix versions. In cases where changes must be made and those changes have not been accepted by the community, it is possible to specify patches that should be applied to the community source code during the build process. Though not desirable, it is sometimes necessary to do this. I’ll supply pointers to information on how to do this below.

Spec files are not an OpenSolaris invention–they have been used for a long time to build RPM packages. This is good news because there are several excellent web resources that document spec files in detail. I recommend Maximum RPM by Edward Bailey as a detailed reference. One complication: It seems that OpenSolaris spec files are not exactly the same as RPM spec files. However, for the purposes of this exercise, don’t worry about this — the Ploticus example below should give you enough information to create a valid OpenSolaris spec file in most cases. However, if you insist on worrying, you can read the information I found here and here. If anyone knows of a better explanation of the differences, let me know and I will include a pointer here.

Okay, lets get to it. I started with a spec file template and created the following file for Ploticus. My commentary includes all of the tips and other information I discovered during the process of writing the spec file for this particular open source package. While I’ve attempted to give pointers to additional information throughout, this is not meant to be the definitive guide to the full capabilities of spec files. There should, however, be enough information here to allow typical open source apps to be packaged and contributed to OpenSolaris. Consult Maximum RPM for additional details.

spec file commentary
# spec file for package: ploticus
# This file and all modifications and additions to the pristine
# package are under the same license as the package itself.
# include module(s): ploticus
This is all boilerplate commentary. Insert the name of your package twice.
Required for all OpenSolaris packages. For the curious, the source is here.
Name: ploticus

Once you specify the name of your package, you can use the macro %{name} to refer to it later in the spec file. As you will see below, there are other predefined macros available that you will use to write your spec file. You can also define your own macros using the syntax:

%define macro_name macro definition

Summary: ploticus -- creates plots, charts, and graphics from data
Summary is a one-line description of the package that will be displayed by the OpenSolaris Package Manager.
Version: 2.41
The version number can be referenced as %{version} later in the spec file, which can often be used to generalize file and directory names. In the case of Ploticus the version number string (e.g. “2.41”) happens not to be used as part of its filenames (e.g. ploticus241src) so I do not use %{version} in this example, except in one instance of boilerplate.
License: GPLv2
Free text field describing code’s open source license. I’ve seen all of these used: GPL, GPLv2, GPLv3, BSD, LGPLv2.1, New BSD License. If GPL, be explicit if you can: GPLv2 or GPLv3. The “or later” licenses might be appropriate as well, e.g. GPLv2-or-later, GPLv3-or-later, etc. There is a nice discussion here about the pros and cons of “or later” licenses.

The source tag specifies the location of the source-code tarball (possibly gzip’ed) that should be downloaded to build the package. Because Ploticus is hosted on sourceforge I had to specify a manual download URL rather than that of the automated download site (

Note that the source location can also be specified as an ftp:// address.

The open-source community’s web address.
Group:  Applications/Graphics and Imaging
The group tag describes the kind of software in the package and will be used by the OpenSolaris Package Manager to categorize the package hierarchically. I chose a group name based on the package classifications listed here.
Distribution:	OpenSolaris
Vendor: OpenSolaris Community
BuildRequires: SUNWxorg-headers, SUNWzlib, SUNWgcc

These are other OpenSolaris packages that must be available on the build system in order to correctly create the binary package. In this case, I am building Ploticus with X-Windows capabilities, so I need to ensure the X client header files are available. I am also enabling a Ploticus compression option so zlib is needed as well. And, to be safe, I’ve specified which compiler is required. I could have used Sun Studio, but I know for sure that Ploticus compiles with gcc so I’ve used that.

You can find these package names by searching in the Package Manager on your local OpenSolaris system.

Requires: SUNWzlib
This section lists packages that must be installed on the end-user system for the software to work correctly. In this case, Ploticus will be dynamically-linked against zlib so I need to make sure the Package Manager knows about this dependency. When the users asks for Ploticus from the repository, the Package Manager will know it also needs to download and install the SUNWzlib package as well.
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
SUNW_Basedir:   %{_basedir}

This is boilerplate. The intent of BuildRoot is to define a user- and application-specific path that can be used as the root of an area in which your package will be installed on the build server, allowing the build server to support simultaneous builds of multiple packages by multiple users without interference. Note, however, that I do not use BuildRoot in this spec file because this conversation indicates that $RPM_BUILD_ROOT is the officially supported way to refer to the top of a package install area. I don’t know if this is true in the OpenSolaris world as well, but most spec files I’ve seen for OpenSolaris use $RPM_BUILD_ROOT so I have opted to use that as well.

Note that while $RPM_BUILD_ROOT (and BuildRoot) refers to the root of the installation area on the build server, the top of the build area itself — the location where your package will actually be untar’ed and built — is referred to as %{_builddir}.

I do not know how SUNW_Basedir is used.

SUNW_Copyright: %{name}.copyright
This is the name of the copyright file you will upload to SourceJuicer along with this spec file. It must be named as shown (ploticus.copyright in my case.) You will typically find this copyright file on the community’s website and/or included within the community’s source tarball. In the case of Ploticus, the tarball contains a file in src called Copyright, which I have copied, renamed to ploticus.copyright and then edited to remove html markup. This is the file I will then upload to SourceJuicer. The original src/Copyright file is ignored by SourceJuicer. Update: The preceding was actually not sufficient for my package to be validated. I was asked to append the file GPL.txt, which was also in the tarball’s src directory, to ploticus.copyright so that the actual text of the GPL v2 copyright was in the file. The original version of the copyright file (src/Copyright) only refers to the GPL copyleft, it does not include the copyright itself.
Meta(info.upstream): Steve Grubb <>
Meta(info.maintainer):  Josh Simons <>

These fields are specific to OpenSolaris’s packaging system. The upstream field contains the name and address of the individual or group that creates and supports the open-source software. The maintainer field contains the name and email address of the individual responsible for the OpenSolaris packaging of the open-source project. The preferred format is as shown in these examples.

Additional info fields that can be included are documented here.

A free, GPL, non-interactive software package for producing plots,
charts, and graphics from data. It was developed in a Unix/C
environment and runs on various Unix, Linux, and win32 systems.
ploticus is good for automated or just-in-time graph generation,
handles date and time data nicely, and has basic statistical capabilities.
It allows significant user control over colors, styles, options and details.
Ploticus is a mature package, available since 1999, and version 2.40 has
more than 12,000 downloads to date.
A more detailed description of the open source software. This description was taken from the Ploticus web page.
%setup -q -n pl241src

Now we begin specifying what actions are required to build the software. The %setup macro cd’s into the build directory, removes any cruft left over from earlier builds, unzips the source tarball (which will have been downloaded at this point), and then untars the sources into the build directory. It then cd’s into the package’s top-level directory. All of this is done with %{_builddir} as the root directory as described earlier.

Note that %setup assumes the top-level directory specified in the tarball is named %{name}-%{version}. If this is not true for your package, use the -n option to specify the correct name. For Ploticus, all files in the tarball are in the pl241src directory, so I’ve used the -n option to specify this.

See this page for more details about the %setup macro. The %patch macro, which can also be used in the %prep phase, can be used to apply patches prior to building the binaries if the standard community source code needs to be modified in some way to build successfully on OpenSolaris. See the same page for %patch information. Note that you should try to have your OpenSolaris changes accepted by the community to avoid having to apply these patches.

I don’t know what the -q option does.

cd src
make NOX11= XLIBS='-L/usr/openwin/lib -lX11' XOBJ='x11.o interact.o'  \
XINCLUDEDIR=-I/usr/openwin/include WALL= ZLIB=-lz ZFLAG=-DWZ \
PREFABS_DIR=/usr/lib/ploticus/prefabs pl

The %build section contains the commands needed to build the package binaries. At the end of the %prep phase we were left sitting in the top-level directory of the source tarball. Since the Ploticus makefile and sources are one level down from this (pl241src/src), I cd into src before invoking the correct make command for OpenSolaris.

Assuming the make ran correctly, we exit this phase with the binaries and other files all built on the build server in a sub-directory under %{_builddir}.

mkdir -p $RPM_BUILD_ROOT%{_mandir}/man1
cp man/man1/pl.1 $RPM_BUILD_ROOT%{_mandir}/man1/pl.1
mkdir -p $RPM_BUILD_ROOT%{_bindir}
cp src/pl $RPM_BUILD_ROOT%{_bindir}
mkdir -p $RPM_BUILD_ROOT%{_libdir}/%{name}
cp -r prefabs $RPM_BUILD_ROOT%{_libdir}/%{name}

In the install phase, we execute a “make install” or equivalent, moving all files that will be included in the binary package to their final installed locations, but relative to $RPM_BUILD_ROOT rather than to “/” to avoid collisions on the build server. Because the Ploticus “make install” action doesn’t do exactly what I need, I instead manually move each required file to its final location. For many projects, something similar to “make DESTDIR=$RPM_BUILD_ROOT install” would be appropriate in this phase.

If you are moving files manually, do not assume directories exist — make them before you use them. And use the predefined directory macros (e.g. %{_mandir} ) to reference standard installation locations. Others are documented here.

This is boilerplate clean-up code. Insert other commands as necessary.
%attr(0755, root, bin) %dir  %{_bindir}
%attr(0755, root, bin) %dir  %{_mandir}
%attr(0755, root, bin) %dir  %{_mandir}/man1
%attr(0755, root, bin) %dir  %{_libdir}
%attr(0755, root, bin) %dir  %{_libdir}/%{name}
%attr(0755, root, bin) %dir  %{_libdir}/%{name}/prefabs

This can be a complicated section so I suggest reading the Max RPM <a href="%files section.

The %files section specifies the locations and attributes of all files that will be placed onto the end-user’s system when the binary package is installed. The %attr directive is used to specify permissions and ownership for files and directories. The %dir directive identifies directories. Multiple directives can be applied to objects by including them on the same line.

The first line specifies default mode, default user ID and default group ID for all files created during the build process. The dash (“-“) means that a default is not set explicitly for that field. Note that failure to include this line in your spec file will cause an obscure error to be generated when an end-user tries to install your package. That would be very bad.

The next four lines specify the directories in which Ploticus-related files will reside. The last three ensure that the Ploticus binary, all of the Ploticus prefabs config files, and the man page will be included in the binary package. Note again the use of macros to specify standard installation directories.

* Tues Apr 28 2009 - Josh Simons <>
- initial version
Add any changelog information you desire here.

Once you’ve created your spec file, it is time to feed it to SourceJuicer for syntax and other checking and then iterate as necessary until your spec file is correct and has passed validation. The basic flow is shown in the diagram below.

The first step is to submit the spec file to SourceJuicer along with the project’s copyright file. To do so, go to the SourceJuicer Submit page (login required.) Assign a descriptive name to your upload (I used ‘ploticus’) and then specify your spec file. Use ‘add another file’ to add your copyright file. Add whatever other files you may need (see ‘more help’ on the Submit page.) Click Submit and you will see a page like this:

The summary page includes an indication that my spec file successfully passed a syntax check. If an error occurs at this point, make the necessary corrections and use the ReSubmit tab (not shown) at the bottom of this page to upload new versions of your copyright and spec files.

Looking under Reviews, I can see my package has not yet been validated, which means my submission hasn’t yet been checked by someone to ensure my copyright file is appropriate, that someone else has not already packaged this program for OpenSolaris, etc.

The next day I receive two email messages with comments from reviewers. When I log back into SourceJuicer and look at the Review tab, I see the two comments that were submitted. The fact that the package is still marked as not validated means I have issues to address:

Clicking on the “[review]” link takes me to the page with detailed information about the Ploticus review. I can also view this page by visiting the MyJuicer tab and then clicking on the appropriate link under My Submissions. This second method is better since it can be difficult to find your review on the main Review page. In any case, the page looks like this:

As you can see from Amanda and Christian’s comments, I did not use the correct naming convention for the copyright file I uploaded to SourceJuicer. Rather than “Copyright”, the file should have been named “ploticus.copyright” (more generally, %{name}.copyright). Also, Amanda hopes I can remove the html that is for some reason embedded in the standard Ploticus copyright file.

Using this same review page, I submit a clarifying question back to the reviewers to ensure I address their issues. I am not clear on the relationship between the copyright file that is submitted manually to SourceJuicer and the copyright file in the source tarball that is described with the “SUNW_Copyright” tag in the spec file.

Now that I understand the copyright issue and have adjusted my spec file and copyright file appropriately (and also updated the spec file and annotations in this blog entry–meaning you never saw that I had initially called my copyright file “Copyright”), I use the same Review page to Resubmit the spec file and copyright file. Use the tab at the bottom of the Review page to do this:

As of this writing, there is no way to remove a file that has been submitted to SourceJuicer so all three files (Copyright, ploticus.copyright, and ploticus.spec) are associated with the project even though Copyright is now extraneous. Until removal is possible, just ignore the extra files. [UPDATE: As of SJ 1.2.0, files can removed by visiting the MyJuicer review page for the appropriate package.]

I resubmitted the files, the package was subsequently validated, and then it was automatically scheduled to be built on the build server. I did not receive a notification when the build attempt occurred so you need to check status periodically (use the MyJuicer tab). When I checked, I saw my build had completed successfully on the first attempt:

Had the build not succeeded, I would have followed the Log link to view the build log, found the problem, fixed the spec file, and then Resubmitted. The package would then be rescheduled for another build automatically with no need for re-validation.

With the Ploticus build successfully completed, it is now very important to verify that the package installs correctly and that the software actually works. Though I don’t cover it here, my first Ploticus package did not work correctly on my test system. I had to make changes to my spec file, rebuild the package, and reinstall it. Therefore, please do install and test your software!

To do the test installation, I first added the pending repository as a package authority on my 2008.11 system. Note carefully the location of this repository; I had expected it to be, but that is not correct:

% pfexec pkg set-authority -O pending

I then started the Package Manager, selected the Pending repository and did a search for Ploticus. Voila! The package is available:

After selecting the package and clicking on Install/Update, the installation proceeds smoothly. I then start a terminal window and verify that Ploticus does, in fact, work correctly:

Once you are sure your package installs and runs correctly, send an email to requesting that the package be promoted from the pending repository to the contrib repository. Note that you’ll need to subscribe to this mailing list before you can post to it. To subscribe, go here.

Once the package is available in contrib, users will be able to install your package on their systems.


[See my later blog entry for additional information about SourceJuicer and OpenSolaris improvements that make package contributions even easier.]


Apple Success Disaster?

December 3, 2007

I’ve been accused of being an Apple fanboy and that may be true to some degree, but even this fanboy’s patience is being tried by Leopard, Apple’s newest OS release.

Ever since upgrading, my MBP’s keyboard freezes for 10 seconds or so every few minutes until I either sleep and wake the system or reboot it. Sometimes I don’t have the problem for days and then it descends on me again, making the system unusable. Every day more people contribute to this Apple Discussions thread about the problem. Some say Apple is aware of the issue and that a fix is coming soon. Many are getting very, very frustrated.

Time Machine is also having problems, though thankfully it has worked for me without issues so far. After having used Leopard for awhile, my colleague Eric erased his pre-Leopard disk backup and switched to Time Machine. Which then failed to back up his disk, leaving him without a viable backup of his system. When we last talked, he was going to use Carbon Copy Cloner or an equivalent to do an interim backup, but he had run into Leopard compatibility problems with at least one of these tools. This Time Machine problem is apparently also affecting many people, though I don’t have a Discussions link for you.

And then there is X11, which I need to run OpenOffice, the GIMP, and Inkscape. To say Apple’s transition to the Xorg X server has been less than smooth would be an understatement. I’ve had to download and install an unofficial version of X11 to work around some of the problems and get the above applications running again.

Another small issue: /usr/bin/emacs was broken on my system after I did an in-place upgrade to Leopard. After some poking, I found that the Leopard installer had left the Tiger version of the emacs executable on my system. Luckily, emacs can be rebuilt in place with a simple command and that fixed the problem. Go here for how to fix this problem.

And then there is the fact that I can’t run SecondLife on my machine without starting the program using the OpenGL Profiler utility that ships with Xcode. Failing to do this can wedge your machine so badly that one’s only recourse is to power-cycle the system. This is apparently a graphics driver problem and isn’t Leopard specific, but I tripped over this at about the same time all the rest of this nonsense was happening and it has contributed to my annoyance with Apple. For details on the SL problem and its workaround, check here.

This morning Google Earth died mysteriously on startup and found I needed to install a new Leopard version. Thankfully, one is now available.

Apple usually fixes some significant issues in a dot release soon after their initial major release. Unfortunately, 10.5.1 did not fix any of these problems.

Work from Home

October 28, 2007

As I sit here in my office in Burlington on a Sunday, I realize how much I depend on my laptop to let me work from anywhere at any time. And how much additional work I do because I have that capability.

I’ve been without a laptop for about two weeks. Its replacement should arrive in the next two days (I’ve tracked it from Shanghai to East Boston so far), but not quickly enough to avoid having to drive (40 miles each way) to the office to work on a presentation about code review practices that I’m giving tomorrow.

Aside from that one inconvenience, I’ve come to realize how much work I usually do in the mornings before driving to work and how much I do in the evenings afterward. It makes a big difference in my ability to keep up with everything that needs doing at work.

Logical Domains for UltraSPARC: Now available!

May 3, 2007

ldoms architecture diagram

Our group (SPARC Platform Software) has just released V1.0 of Logical Domains (LDoms) for the UltraSPARC T1 processor. With LDoms virtualization, you can run 32 separate Solaris instances on a single UltraSPARC-T1 based system. Customers with Sun Fire T1000 and T2000 systems or with Netra T2000 or CP3060 systems will find the downloadable bits here.

My colleague, Ashley Saulsbury, has more details about LDoms and virtualization on his blog. Or check out the official product page.

Did I mention LDoms software is free? Yes, I guess I did. 🙂

Protecting Your Project From Poisonous People

March 29, 2007

I just watched a Google TechTalk by Ben Collins-Sussman and Brian Fitzpatrick, titled How To Protect Your Open Source Project from Poisonous People. It’s almost an hour long, but well worth watching for people involved in open source communities, or really any communities at all, technical or not. Their “four stages of protection” against poisonous people (comprehension, fortification, identification, and disinfection) are relevant for dealing with misbehavior in any group.

Their abstract:

Every open source project runs into people who are selfish, uncooperative, and disrespectful. These people can silently poison the atmosphere of a happy developer community. Come learn how to identify these people and peacefully de-fuse them before they derail your project. Told through a series of (often amusing) real-life anecdotes and experiences.

focuses on poisonous people in particular, but their talk also shares some more general wisdom about running effective open source projects. Many of their examples come from their experiences with the Subversion project. And, yes, some of the anecdotes are very amusing.

The video is here.

Thanks to Monty for the pointer.

Virtualizing SPARC

March 6, 2007

Interested in running multiple, virtualized Solaris instances on your CoolThreads server? Well, now you can: SPARC virtualization is here! Details on Logical Domains (LDoms) 1.0 Early Access are available on Eric Sharakan’s blog, Virtuality.

Go team!

Put your least creative engineers on that project, please…

February 21, 2007

Sun’s Systems Group holds regular, two-day internal CTO Review meetings at which the status of pretty much every project currently running within the Systems division is reviewed. These meetings are run by Mike Splain (Systems CTO and Sun’s Chief Engineer) with John Fowler (Systems EVP) in attendance as well.

At least week’s meeting, John made the following somewhat surprising comment during a presentation about a new software project starting soon in Systems. He said:

“Do me a favor and put your least creative engineers on that project, please.”

John was right on target. Why? Because the project is about implementing functionality currently available on some non-Sun systems, and customers who have these systems already understand how to use the existing interfaces and utilities. The correct objective should be to do an excellent job implementing those existing interfaces for our systems. Doing a better or different interface is not always the right answer. In fact, it is often the wrong answer, especially when growing aggressively into new markets and taking business from competitors. In such cases, lowering the switching costs for customers is very important. Common interfaces, especially management interfaces, are an important way to do this.

That’s not to say innovation isn’t important–it is. But innovate in ways that add value, not in ways that create gratuitous differences between our products and those of the competition.

Hueston on Tactical Leadership

December 30, 2006

Bob Hueston, a Senior Staff Engineer in Sun’s Systems Group, is writing about leading software projects on his new blog, Tactical Leadership. He’s been wanting for years to write what he calls a field handbook for engineering project leaders and has started to share his thoughts publically.

The content is what I expect from Bob: insightful, thoughtful and well-presented. His introductory entry is here. I especially enjoyed his thoughts on top-down versus bottom-up planning here.

Time to add another Sun blog to my blogroll. This one’s a keeper.

DE Promotion Review Meeting

December 13, 2006

Sun held its biannual Distinguished Engineer promotion review meeting this week. All of the DEs and and Fellows convened in Menlo Park for a 1.5 day meeting to review candidate cases and to recommend promotions to Distinguished Engineer. We reviewed more cases in this session than in any other I’ve attended.

For those not familiar, the process works like this. First, a candidate’s Vice President or Director decides to nominate an employee for consideration. A case is prepared without the knowledge of the candidate, following established guidelines that have been published on the DE website. Case materials are distributed to the DEs and Fellows in advance of the in-person review meeting. At the review meeting, each candidate’s Director or Vice President spends about twenty minutes presenting the candidate and then spends about the same amount of time fielding questions from the attendees. Once the questioning is completed and the speaker has left the room, an internal discussion ensues, followed by a formal vote.

As a participant, I form an initial opinion based on the reading of each case prior to the meeting. That opinion can be affected (positively or negatively) by the case presentation and then again (positively or negatively) by the detailed private discussion prior to the vote.

At this session, I entered the meeting with a tentative determination that I would vote Yes on 30% of the cases being considered. After presentations, that percentage had risen to 60%. After discussions, I ultimately voted Yes on 75% of the cases. The presos and the discussions matter a lot in this process, as you can see.

Promotion announcements should begin rolling out soon.

Systems Leadership Summit, Day 2

August 23, 2006

The second and final day of John Fowler’s System Group Leadership Summit began with a presentation by John on his leadership values, which all of the attendees I spoke with later found very valuable. I guess it’s always good to know what the boss is thinking and what he values. 🙂 In this case, the concept of “stewardship” is something that resonates with John. He defined stewardship as having three axes: Responsibility, Service, and Mature Leadership.

The follow-on leadership training session, run by an external trainer, was not very useful in my view. While it worked reasonably well as a teaming session, I didn’t gain any real insight into improving my leadership abilities. Others seemed to feel the same way.

The balance of the day was filled with presentations by Mike Lehman, Greg Papadopoulos, and Jonathan Schwartz.

I always appreciate an opportunity to hear Mike speak because he is frank and clear about where we are, what we need to be doing, what we’ve done so far, and where we are going.

Greg’s talk was one I’d heard before, but it reinforced the message for me. I agree with him that there is a real shift happening from delivering applications as bits on local systems towards delivering applications as services in the network. I can see that as a Sun employee, but more important, I can see it as a consumer.

Jonathan’s was the last session of the day and it was more of a Q&A session that a set of prepared remarks. He was pleased with the great server market share news that was released today. Since he was talking to the Systems Group, the division responsible for designing and building those servers, he expressed his appreciation to all for the hard work that contributed to these results. He also apologized to John for running late and not being able to pick up a bottle of champagne to open in front of the team. Instead, he simulated the event by throwing a cup of water on John as an homage, which drew a big laugh from the crowd.