Hacking on bug-triage

A lot has happened after the Summer of Code has ended, and I haven't been able to work on bug-triage after that, in part due to "Real Life", in part due to laziness. Anyway, I've finally got some time to work a bit on it today. First, I've taken care of Debian's 452918, where the reporter said that using "pending" on bug status meaning open might cause confusion with the pending tag. Actually, pending is the word used internally by debbugs to mean open bugs, and its also what is returned by the debbugs' SOAP interface (bugs tagged pending get the status pending-fixed instead). Anyway, I've just mapped the internal/SOAP debbugs status to the ones used by pkgreport.cgi, that should avoid misunderstandings. While at it, I also took some time to add a feature I've missed while using bug-triage myself: When browsing upstream bugs in search of a match for a Debian bug, it's necessary to read the upstream bug log; now double-clicking on a upstream bug on bug-triage will open it's log on a browser (the same way that selecting a Debian bug and clicking "Go to" in the toolbar works). It's only small changes by now, and I'm not sure if I'll be able to dedicate myself much more to this in the (near) future, but I'll try my best.

mergeant 0.67-1

The newest version of mergeant, a GNOME GUI for database manipulation, has hit the Debian archives earlier this week (hmm... actually last week, as today is sunday). Thanks Loïc for the sponsorship. This version was a minor one, the biggest change being the use of a library to allow the binary to find its resources in different directories on runtime, which was disabled on the Debian package as the user isn't supposed to move its files around anyway. Another change was the migration of the mime data to the newer shared mime/freedesktop style. It took me some time to figure out that I had to install the mime file through dh_installmime to have it add the mime snippets on the maintainer scripts, as dh_installmime is called before dh_install on CDBS (I wonder if there is some reason for the calling order...) Update: I've missed that .sharedmime isn't a list of files to install like, say, .install; it is the file to install itself.

Mergeant 0.66-1

The latest version of mergeant, a GNOME GUI for database manipulation, has finally reached the Debian archives, closing all bugs which were open against the previous one. Thanks, Loïc, for the sponsorship. As a side note, this version of mergeant has got an implicit pointer conversion bug due to missing includes in two different files. This was my first chance to test the bug forwarding feature of the Bug Triaging and Forwarding Tool (my project in Google's Summer of Code 2007). It isn't as comprehensible as I would like due to limitations on bugzilla, but was pretty handy nevertheless.

September 7th

Last friday was a national holiday here on Brazil (independence day). Even though there were lots of problems (my car got damaged while in the parking lot during the week, our planned paths were closed due to independece parties two times, etc), this was a very cool day. I've joined some friends and have gone to Holambra, 135km away from São Paulo, to see the famous expoflora - an exposition of flowers. It's aways nice to pass some time with good friends, forgetting the boring daily routines. The exposition was very interesting. They had (of course) a lot of flowers, one of the coolest being the "rainbow roses" - roses with various colors. Maybe I'll get some photos up sometime... Besides the flowers, they also had some exhibitions of Netherlands traditions, and of old rural stuff. The only downside goes to the organization; sometimes it was hard to find the way on the place (and to the place also), there should have been a lot of extra signs around.

SoC Final Report

The Google Summer of Code coding period has ended earlier this week, so it seems to be a good time to stop and look what was done (and what wasn't). Let's start with an objective analysis, following the list of planned features: Advanced query of bugs on the Debian BTS In a 1-10 scale, I would give myself something like 6 or 7 here. bug-triage can query bugs by number, source package, package, maintainer and submitter. These parameters can be combined to make an query (reports must match all parameters) and more than one query may be added, subtracted and and'ed. A clearly missing feature is querying by usertag, support for it has been added on btsutils already and is just a question of adding a new input box on the query builder dialog (it should work if you enter the query manually in the bug-triage query line AFAICT). Show all the information related to a bug report bug-triage has a "goto" button which opens the bug in the user's preferred browser. Missing is the support to open the bug in a MUA, which I have promissed to Loïc. Send additional information to a bug report bug-triage has a "followup" button which opens the user's MUA with the right address. Maybe setting the subject would be nice... Manipulate the tags of a bug report I haven't had time to implement this at all. I've started it, the bugs have a context-sensitive menu which have each of the available tags as checkboxes on the right current status, but clicking on them does nothing currently. Search for similar bugs in upstream developer BTS This one would value something like 3 or 4 in a 1-10 scale. It searches upstream bugzillas by product and component, but doesn't do any cool trick like searching words in the bug subjects. The plus is that it does support KDE bugzilla, even though I haven't been brave enough to promisse it before SoC :P Forward a bug report to the upstream developer BTS There isn't really a good interface to report bugs programatically on GNOME bugzilla, which was disappointing. <rant>There is the bug-buddy xmlrpc(?) interface, but the developers say they will actively break anything using it, which I find very free-software-unlike.</rant>. The current bug-triage solution opens a browser with some of the fields filled. It would be nice to fill the full bug description, but its limited by the maximum size of an URL, so it fills only an introductory text (something like This bug was filled on Debian BTS #XXX) currently. Let's hope bugzilla 3 get on the GNOME world soon. Conclusions Unfortunately, this project hasn't evolved as much as I would like in the second half of the GSoC. As I said elsewhere, a lot of stuff has happened, some good, some bad, but on the end everything I hadn't been able to dedicate myself to anything I should. Even so, I think the project may be considered sucessful. Maybe the more interesting product of this project, at least currently, is still on python-btsutils. As I said in the middle-term report, it'll be integrated in the debian_bundle soon, and might be powering important stuff like the devscripts in the future. The bug-triage application has a lot to evolve yet, but is also an interesting tool IMHO. There are a lot of hurdles in the way to get different BTSes integrated, but the effort will for sure pay in the future. I plan to keep developing it (evon though keeping the GSoC pace is impossible) and am anxious to see the results ;) Finally, as I said before, one of the main things on GSoC for me was the potential to learn cool stuff. The best choice I've made during all of this was to use Python. Even though it's still a new language for me, and i still have to search embarassingly simple things in the tutorial/reference, it was a lot more productive than if I was developing in C. Learning to use git was also a great. This goal was the more fulfilled one, I've really learnt a lot of things which will surely be useful in the future. BTOH, I still can't get used to glade :/

Final SoC releases

I've just finished releasing new versions of all modules related to my Summer of Code project, the Bug Triage and Forward Tool. If my mentor doesn't find any serious bug, these will be the last releases under the SoC flag. Thanks again, Google, for this great program and the progress it brings to Free Software. The releases: bzutils 0.2:

• Announcement
• Source .tar.gz

btsutils 0.3:

• Announcement
• Source .tar.gz

bug-triage 0.2.2:

• Announcement
• Source .tar.gz

Debian packages should hit the archive as soon as they get evaluated and sponsored. Of course, a full status report will also be posted during the evaluation time.

Iterating over lists in Python

Note to self: It's a bad idea to change a list in the middle of a loop iterating over it. The following doesn't work:

for item in list:
    if foo:
        list.remove(list)

And must be replaced with:

for item in list[:]:
    if foo:
        list.remove(list)

On the wrong way, the loop doesn't execute on items positioned immediately after the removed ones. This has cost me a big amount of debugging time...

Goodbye vacation

It has been some time since my last blog post... The thing is, my university vacations ended this week, and overall left a bad taste. Unfortunately, complications ranging from buying a very big new toy from some reasonably strong illness have made an already tight schedule simple impossible to follow. On the end, I haven't dedicated enough time for anything I wanted to during these vacations: Didn't sleep as much as I would like, didn't dedicate enough time to my gf, didn't advance my Summer of Code project anything near as much as I wanted, didn't help a friend to get a Linux install in his shining new laptop, didn't fix my work's server RAID array, didn't package the newest mergeant... The list goes on... I guess what I wanted to say is: sorry. I know I'm owing a lot of stuff for a lot of people, and I'm really sorry I haven't been able to deliver what I should. I hope to fix all of these ASAP (maybe excluding the sleeping one, seems like a lost cause :P) Now if just the Sao Paulo's metro stopped this stupid strike...

Glade

Dear lazyweb, Is there any reasonable way to reorder the tabs of a GtkNotebook in glade-3? I needed to do it to add a tab on a configuration dialog, and couldn't find any. I worked around it by editing the XML file by hand, but it just feels counter-productive and ugly...

bug-triage Debian Packages

Thanks to the timely efforts of Loïc Minier, a more usable (even though still featureless) bug-triage version (0.1-2) has found its way to the Debian Archive, alongside with the shining new btsutils 0.2-1. Now it does return something on queries... The development is also progressing, the current version in git can already show the upstream bugs for a given Debian package whose upstream uses bugzilla.

btsutils 0.2 release

For what can be read on Planet SoC, computer problems seems to be the norm between SoC students... I've got my part of this cake today with a power cut and some strange /dev permissions when my PC got back... Fortunately it was somewhat easy to solve (I believe it was the udev upgrade which did the trick), so I've been able to release btsutils 0.2. Please take a look on the announcement for more details. Still on the SoC side, thanks to the great work from the Debian ftpmaster team and Loïc Minier, the packages python-btsutils and python-bzutils have made their way through the NEW queue and are on the Debian official archive. And finally, on Debian side, Loïc Minier also uploaded libgnomedb3 3.0.0-2, a bug fix release for libgnomedb closing two bugs. Thanks, Loïc.

Summer of Code mid-term report

I. Introduction Today, the mid-term evaluations of the Google Summer of Code starts. It means it's a good for a comprehensive summary of what I've done so far. For those who don't remember, My GSoC project is the Bug Triaging and Forwarding Tool, whose aim is to help triaging of Debian bug reports. On my proposal, I had included the following timeline:

• April - May: Study of the resources available for use in the project (Debian BTS SOAP/LDAP interface, Bugzilla interaction resources, etc) and the design of the tool. Request needed unavailable features, if any, with patches if possible
• May - July: Development of initial interface and code to interact with Debian BTS
• August: Development of Bugzilla plugin

During the proposals evaluations and the community bounding period, it was pointed to me that there's already a lot of good tools to handle bug reports, so I've changed this a bit and already started to work on BTS-Bugzilla interaction, which seems to be the more novel part of the tool. The development has been fragmented in three modules: btsutils, bzutils and bug-triage. II. btsutils The btstutils is a python module to interact with the Debian Bug Track System. Currently, it can query the bts by bug number, package, source package, maintainer, submitter and usertag. For each bug, it returns bug number, package, summary, status, severity, submitter and tags. Support for returning usertags is under way. Ths btsutils can use html or soap to access the bug data, and has also a backend-agnostic module wwith tries the best method and, if that fails, tries the other. Steffano Zacchiroli has offered me to include the btsutils on the debian-bundle, a bundle of python modules for dealing with Debian infrastrcture. I'm very excited by this idea, and believe that if all or at least some of the Debian tools which interact with BTS end using a standardized module for access, this might end up being the major contribution of my GSoC project for the Debian community. III. bzutils The bzutils is similar to the bzutils, but is to interact with bugzillas instead. Currently it can do queries based on GNOME's bugzilla boogle and the more generic boolean charts. The later works (at least) on GNOME, KDE and standard bugzilla (i.e., landfill). For each bug, it returns bug id, product, component, status, resolution, reporter, assignee, summary, priority and severity. IV. bug-triage The bug-triage is the Bug triaging and forwarding tool. It's being developed using python, gtk and glade. Currently, it can query the BTS using any of the btsutils' supported fields (even though the usertags aren't really accessible through the interface yet). It can also launch one of the returned bug reports in a web browser. Integration with bugzillas is under development. It already has an interface to establish links between Debian packages and upstream bugzilla's address, product and component. The next steps will be to add an upstream button in the toolbar/menu and make it show the bugs selected through these info, allowing the user to mark the bug as forwarded to one of them. V. The future After that, I'll work on these features (in no particular order): filling a new upstream bug using the Debian bug data as a starting point, more advanced filtering options to filter the upstream bugs (such as filtering by words in the bug summary, and maybe through backtraces if available) and more ways to manipulate the BTS through the interface. To be sincere, it seems this project won't be as advanced as I would like on the end of the GSoC. There are some new, unexpected (and good ^_^) things going on my life, which are taking some of the time I expected to dedicate to GSoC, which means I'll likely keep the current pace until the end of the project (I expected to increase it during July). Even so, I believe the project is going on a good path, and I hope it'll be very useful for the Debian community.

Debbugs, SOAP and Usertags

A small bit of info that can be hard to find: It's possible to call debbugs' SOAP get_usertag function passing only an user as argument, to get all usertags set for this user and all bugs with each of these tags, like this:

>>> import SOAPpy
>>> proxy = SOAPpy.SOAPProxy("http://bugs.debian.org/cgi-bin/soap.cgi", "Debbugs/SOAP/V1")
>>> proxy.get_usertags("bts-link-upstream@lists.alioth.debian.org")

Debbugs SOAP

Following the hints by Bastian Venthur, which answered my last post (thanks, Bastian), and some suspicious mails to debian-debbugs, I've finally gotten to know that the debbugs' SOAP get_bugs is finally fixed. Of course, this means that btsutils has just got its query function implemented in SOAP besides the html parsing stuff. Also great news on the debbugs' SOAP camp is that get_status now works on more than one bug per call. Yay! Debbugs people rocks. I'm not realy in place to ask something more drom these great folks, but it would be great if those cool features they keep adding were announced somewhere. Now, the downside: Querying through SOAP is still quite slower than parsing the html. According to Python's timeit, getting a list with the bugs on all packages maintained by me takes about 17 seconds, while the same operation through HTML parsing takes about 10 seconds. I believe this differences comes from accessing the server one time on the HTML parsing and two times (get_bugs and get_status) on the SOAP.

Exception handling on PyGTK applications

A feature I've been wanting to implement on bug-triage for some time now is to have unhandled exceptions shown in a gtk message dialog instead of getting lost on the terminal. And I've finally got it working now. At first, I had some vague impression of having read about changing the default exception handler on Python. Maybe I've just mixed things up, but I couldn't find anything about that today (maybe it was Java?). Instead, I found out some information about Python decorators. This is the first time I've seen anything like the decorators, and I must say this is a concept a bit hard to get used to, at least for me. Basically, if is a function that gets a callable (ie, a function or anything else that can be called) and returns another callable to be used in its place -- usually the original one with some nice extra. I've created a python decorator to wrap the function call it decorates inside a try statement, whose except clause show some info about any exception it gets. The code can be seen on the bug-triage git repository. On the linked file, errorhandler is the decorator and msg_exception takes an exception as argument and shows a message dialog. Applying the decorator to a function is simple, import it and add an @errorhandler just before its declaration. Some references on decorators:

• PEP 318 - Decorators for functions and methods
• Python Cookbook: Basic exception handling idiom using decorators
• zephyrfalcon.org's "Decorate This"

PS: I just got really happy to discover that this blogger/blogspot draft autosaving stuff works...

Python, os.fork() and GSoC

This one was a bit tricker to find, so I'm going to post it here: if you use os.fork() on Python, you will want to use os._exit() on the child process, or it will clean stuff it shouldn't and you will have segfaults with doubled memory free. That said, python's webbrowser module is quite useful; check it if you ever has to open a browser from a Python script/program. The only downside I see on it is exactly that it doesn't takes care of the forking stuff for you. I'm using it on bug-triage to show a bug report on the user browser. Talking about GSoC, during the last week the Debian BTS was updated, breaking, between other things, btsutils' 0.1 version. The current development version of btsutils, which uses BeautifulSoup to parse the HTML stuff more realiable, wasn't affected by the update. That's all for now, time to code.

bug-triage 0.1

Yet another GSoC release; it's the first release of the real tool this time. From the release announcement sent to bug-triage-devel: ---- I'm pleased to announce the release of bug-triage 0.1. bug-triage is a tool to help triaging Debian bugs. Current features: * Show all bugs which match a given bug number, source package, package, maintainer or submitter * Show details about one of the returned bugs on the user's web browser The source code for bug-triage is available at http://bug-triage.alioth.debian.org/ Debian packages are being worked on and should be available soon.

The world is full of (different) bugzillas....

... but the KDE's wins the strangeness award so far...

>>> import urllib2
>>> opener = urllib2.build_opener()
>>> f = opener.open("http://bugs.kde.org")
>>> print f.read()

[...]

<h1>Page not found</h1>

<p>KDE has switched to bugzilla. Please go to the <a href="/">main page</a>
to search for your bug.</p>

[...]

>>> opener.addheaders = [("User-agent", "bzutils")]
>>> f = opener.open("http://bugs.kde.org")
>>> print f.read()

[...]

<h1>KDE Bug Tracking System</h1>
<p>This is KDE's bug tracking system which files details of wishes, bugs and crashes
reported by users and developers.  Each report is given a number, and is kept on file until it is
marked as having been dealt with. For participating you need a personal account which will gain
you the ability to post reports and comments as well as voting for specific reports and observe
development. You'll need to enable cookies for this site for staying logged in.</p>

Questions: 1) Why does kde bugzilla require user-agent to work? 2) Why it doesn't return something more descriptive?

bzutils 0.1

Following with the Summer of Code, I've just released bzutils 0.1. The release announcement: --- I'm pleased to announce the release of bzutils 0.1. bzutils is a python module to interact with bugzilla servers. Current features: * Query bug reports through boogle (gnome's bugzilla search improvements) or boolean charts. * For each bug report, gets the following metadata: id, product, component, status, resolution, reporter, assignee, summary, priority and severity The source code for bzutils 0.1 is available at http://bug-triage.alioth.debian.org/ Debian packages are being worked on and should be available soon.

BeautifulSoup: Parsing html in Python

Parsing HTML to get the information you need can be a very hard task if you take complex pages like the ones generated by the Debian Bug Track System, which I need to do on my GSoC project while the debbugs people doesn't finish the SOAP interface. I was doing it through regular expressions, heavily based on the reportbug-ng code, when my mentor (thanks, Loïc) mentioned BeautifulSoup, a python module (with a strange name :P) to parse html. If you ever need to parse html code in python, I strongly suggest you take a look on it. As usual with python stuff, it's very well documented, and it has a very good set of features which allows one to easily find anything inside a html document. It also has a xml module, which I haven't tried (yet). BTW, did I already say I think GSoC is a great learning experience? Even I'm surprised by how fast I'm being able to apply GSoC-acquired knowledge in other activities, as I'm already using BeautifulSoup in another project.

Sao Paulo's Metro Strike

I live in São Paulo, The most important (IMO) city of Brazil. It's also the fifth most populous metropolitan region of the world. One of the main public transportation systems in São Paulo is its metro, which was once regarded as a transportation city of major quality. Lately, however, it has been sinking. Fast. Very fast. It just can't keep up with the demand; the trains are getting more and more full, and the timings are getting more irregular as time passes. If this isn't enough; the syndicate of metro workers seems to be formed by a bunch of selfish clowns. So, today, 3,3 million of people are without transport, because these clowns want 13% of income increase. Now, where are the laws which state that this kind of public service can't be paralyzed? The government should just send these clowns back to the circus they fled from. This brings us to another topic: the pathetic laws that regulates public/government workers . They can just work (or not work) however they want, and can't be fired. The ultimate job security here is to get into a government job. Finally, the solution for the (metro) problems: just privatize the damn thing already. Well, rant done, so let's go on with our daily schedules (or what is possible of it without metro, for the paulistans)

btsutils 0.1.1

I've recently released the first version of btsutils, a python module to interact with debbugs servers (such as the Debian Bug Tracking System). The btsutils is part of my Google Summer of Code project, the bug triage and forward tool. Currnetly, the btsutils can query the bts based on bug number, source package, package, maintainer or submitter. A Debian package of btsutils 0.1.1 is already waiting to be processed on the NEW queue. Some useful links:

• Release announce on bug-triage-devel
• Source tarball

Python Soul

I find it very interesting how different programming languages have different styles. My Google Summer of Code project, the Bug Triage and Forward Tool, is my first Python software; and working on it on the last few days, I've got the feeling that the way I've been using to structure the code doesn't fit very well with the way python packages/namespace works. I already wished to separate the project in three independent codebases, so I'll go ahead and do that. These codebases will be:

• python-btsutils: python module to interact with the Debian BTS / Debbugs servers
• python-bugzilla: python module to interact with Bugzilla
• bug-triage: The tool itself.

BTW, I'm just loving python :)

Two weeks later...

It has been two weeks from my last post there... due to this end of semester being a bit thougher than the last ones in my university I couldn't do everything I wanted, particularly on the Summer of Code front. Summer of Code I believe this blog should be on SoC's planet by now. Hello, planet :) I hope the SoC can be a great time for all of the people involved. Today, I finally got some time to make code to talk with debbugs (Debian's bug tracking system software). Thanks to the help of Don Armstrong on debian-debbugs, I have been able to get a python function to get informations from a given bug report through SOAP, but unfortunately it doesn't handle non-ascii characters. I've also made a similar function parsing the debbugs-generated HTML pages; but parsing html doesn't sound reliably at all. BTW, the code to parse HTML was mostly taken from Bastian Venthur's reportbug-ng; thanks Bastian. Free Software rocks because you don't have to rewrite the wheel. Debian Packages On the Debian Packaging front, libgda 3.0.1-1 was uploaded recently (thanks, Loïc Minier). Also, my patch to use libgcrypt instead of openssl was commited on libgnomedb upstream's repository (this time, the thanks goes to Vivien Malerba); I hope a new upstream release is made soon.

Bug Triage and Forward: The Beginning

I've got some free time over the last two days, so I've started to write some code for my Summer of Code project, the Bug Triage and Forward Tool (bugtaf). As I had only a laptop without internet connection in hand, it wasn't possible to implement anything on the tracker integration part, so I've started the user interface. This is the first python application I'm writing, and so far it seems to be a great language. The biggest difficulty I've had so far with it was to find how to call the parent class constructor. I got stuck there yesterday when making QueryBuilder (derived from gtk.Dialog), but today (with internet), I found the answear on Dive Into Python. It's pretty straightforward BTW, I guess I need to sleep more. The pyGTK stuff is pretty simple; there, the background with C GTK programming helps a lot. The only quirk I had there is that calling set_border_width on a already packaged GtkBox (eg, on an Dialog's vbox) doesn't work :( After all that, I've git pushed it to alioth, and git pulled on my desktop. Just then I remembered I had created an ui branch, as I wasn't sure on how my first try with python/pyGTK would be and wished to test the git branching. It took me some fight with git, but everything seems to be properly merged in the master branch now.

Playing with alioth and git

I've been playing a bit with alioth's git support. I didn't find the full procedure to create a new git repository on alioth anywhere (of course, I didn't search a lot though), so I'll post how I did it here. First of all, I asked for a git repository on the Site Admin Support Requests Tracker. They created a directory with the right permissions in git.debian.org/git/bug-triage (bug-triage is the name of my alioth project). After that, I loged on git.debian.org through ssh (the ssh keys registered on alioth work fine) and initialized a new empty git repository there: $ mkdir /git/bug-triage/bug-triage.git; cd /git/bug-triage/bug-triage.git $ git --bare init-db --shared Then, I logged out of git.debian.org and created a git repository on my machine, with an empty README file, and tested pushing it to git.debian.org, which worked fine: $ mkdir ~/bug-triage; cd ~/bug-triage; touch README $ git init $ git add README $ git commit -m "Add an empty README" $ git push --all ssh://grm-guest@git.debian.org/git/bug-triage/bug-triage.git Finally, I've set an origin remote entry on my machine's repository, so that I can use push, fetch and pull without typing the git.debian.org path: $ git remote add origin ssh://grm-guest@git.debian.org/git/bug-triage/bug-triage.git To test that, I've edited the README file and "commited" the change to git.debian.org.