Samuel's Tech Blog

This is the perfect Geany color scheme should (or at least near) :) No... but I'm at least very happy with the color scheme now. I could write a page on what makes this scheme so great, but don't have the time right now. So for now, I'll just say that if you like it, grab it on github!

I just couldn't find a python introduction as condensed and to-the-point as the excellent one for perl, by Kirrily "Skud" Robert, so I simply set out to convert that one to python. Hope you enjoy!

Running python programs

On linux, just run:

python filename.py

Basic syntax overview

A Python script or program consists of one or more statements. These statements can be written in the script in a straightforward fashion. There is no need to have a main() function or anything of that kind (although it can simplify things. See below).

print("Hello world!")

....

After learning to know about Sublime Text 2, it seems to be a perfect companion to VIM and Eclipse, my other favourite editors/IDE:s. Now I also got it set up to debug PHP scripts with Xdebug :) See for yourself:

Sublime as the perfect companion to VIM and Eclipse

VIM, with some modifications, is perfect for quickly editing scripts while navigating around in the command line, and is also available virtually everywhere.

Eclipse is of course the standard workhorse for development, but sometimes it just feels to heavy, and even the simplest debug session will require setting up a new project with a lot of settings, defining a workspace etc etc. And this is where Sublime comes in.

Sublime is more easy to work with than Eclipse/PyDev, Eclipse/PDT (PHP) etc for a number of reasons:

1. Working with projects and folders

The first thing is how it works with folders and project.

In Sublime, most of the time you don't even need a project. It's enough to open a folder (File > Open folder ..., or "sublime <foldername>" on the command line). This will enable the super nice file opening tool (Ctrl + P), that suggests (and previews in realtime) files as you type (with powerful fuzzy-search), which is really handy when you forgot what that file in your project was named.

On the other hand, setting up a project is also very quick and easy. Just use the "Open Folder" feature as described above, and then go "Project > Save Project As ...". This will add one setting in the project definition: the path to the folder. For configuring Xdebug for PHP debugging (more info on installing the required extensions further below), you just need to add a remote URL setting. This is what I configured for working with my RDFIO SMW extension (accessed with "Project > Edit project"):

{
    "folders":
    [
        {
            "path": "/var/www/smw/w/extensions/RDFIO"
        }
    ],

    "settings": {
        "xdebug": { "url": "http://localhost/smw/wiki/Special:RDFImport" }
    }
}

2. The code overview / "minitature"

I really like the "code miniature" that you see to the right when having a file open, so that you get a graphical hint of how the file looks, and where you are. Actually in some ways better than a code outline feature like in other editors, since the "graphical structure" of the code seems easier to recognize for the brain, than the logical, at least for me.

3. Easy access to snippets

I hade trying to figure out how to write that method that makes you python file automatically run the main method, so that you can have it at the top ... In sublime I just go Ctrl+Shift+P and start writing "main()" in a python file, and I will find it right away.

4. Powerful auto-completion

Otherwise, I really like the autocompletion of both keywords and brackets and stuff. It is helpful and seem to just work, a lot more often than many other editors.

5. Easy zooming of text

Something I really miss in Eclipse is easy zooming of the text size. I have quite bad eyes, and sometimes need to really zoom up the text size a lot to be able to work at all. Ctrl + + and Ctrl + - is all you need.

5. Vi(m) mode!

I'm very happy about this. The shortcuts for navigating around in vim are quite hard to beat, even for sublime. Happily enough, sublime has the (vi)ntage mode, so that you can use the most common vim shortcuts within sublime itself.

All in all, Sublime makes me feel more productive than I (think) I have been before!

Setting up Xdebug with Sublime Text 2

Setting up Xdebug to work with Sublime requires a few steps, but it's not too cumbersome.

Note, this assumes you use Linux Mint (or maybe some other recent Ubuntu based distro).

• Install xdebug. Should be enough with something like:

  • • sudo apt-get install php5-xdebug
    • Configure settings in /etc/php5/conf.d/xdebug.ini.

    • My file looks like so:

      zend_extension=/usr/lib/php5/20090626/xdebug.so
      xdebug.remote_enable=On
      xdebug.remote_host="localhost"
      xdebug.remote_port=9000
      xdebug.remote_handler="dbgp"
      xdebug.remote_autostart=1
      ;xdebug.remote_log="/tmp/xdebug.log

    • (Well, the "remote_log" is not really needed, I just use it right now for some debugging)
    • Restart Apache (sudo service apach2 restart)

• Check out the SublimeXdebug Sublime plugin from https://github.com/Kindari/SublimeXdebug and place it in ~/.config/sublime-text-2/

  • cd ~/.config/sublime-text-2/
    git clone git://github.com/Kindari/SublimeXdebug.git .

• Follow Kindari's hint, for resolving the missing py expat stuff in Ubuntu/Linux Mint (see bottom of the github page):
  • Download python2.6 files from Ubuntu Archives
  • Extract the files: dpkg-deb -x python2.6_2.6.5-1ubuntu6_i386.deb python2.6
  • Copy the extracted usr/lib/python2.6 folder to {Sublime Text directory}/lib
  • (My note, leave the existing python26.zip file intact!)
• Create a sublime project:
• Open the folder where your PHP app is ("File > Open folder ...")
• "Project > Save as project"
• "Project > Edit project"
• Add the "remote URL", something like so:
  
  
  

  {
      "folders":
      [
          {
              "path": "/var/www/myapp/"
          }
      ],
  
      "settings": {
          "xdebug": { "url": "http://localhost/myapp/index.php" }
      }
  }
  

• Open the index.php file (or some other file you want to debug)
• Create a breakpoint (Shift + F8 will give you the debug menu. Navigate down to "Add breakpoint)
  • Caveat: Do NOT use the "Right click > Toggle breakpoint" ... it does not seem to come from sublimexdebug, but probably from the GDB plugin that (I at least) did install.
• Start the debugging: Shift+F8 > "Start debugging"
• Use:
  • Ctrl+Shift+F6 - Step over
  • Ctrl+Shift+F7 - Step into
  • Ctrl+Shift+F8 - Step out from
  • Ctrl+Shift+F5 - Continue (until next breakpoint)

That's it, I think!

I just came back home from an interesting PRACE iRODS workshop at the National Supercomputing Centre in Linköping. Find below my random notes and impressions. (The presentations are found here).

Among the top bits were the tutorials by Leesa Brieger, not to mention the talks by DICE director Reagan Moore. He covered a lot of interesting stuff that is going on both with the iRODS software itself, and in how it is being used.

From the tutorials, I was very happy to get a federation up working, between two local datagrids that I managed to install on my laptop, and to actually iput a file from one of them to the other, yay! :) ... and to get the iRODS rule engine debugger idbug to work (See this page for some instructions). The idbug debugger will let you watch - in real-time - exactly which iRODS rules and actions are being executed when you trigger some icommand etc. Seems like a totally essential tool for any iRODS rules development!

Back to Reagan Moore's talks and what is going on with the iRODS software, many nice new things are supposed to come in the upcoming 3.2 release, which is actually supposed to being released next week or so.

Some things that sounded especially interesting:

• Support for registration and sharing of workflows (Reagan showed some nice stuff about this. It looked really interesting, although I have to look closer at it to get it in detail).
• Pluggable Authenication modules (will enable easier use of things like YubiKey, and not the least LDAP integration!)
• Pluggable microservices (you can add them without recompiling the source code)
• Pluggable storage resource drivers
• Support for storage drivers for some interesting protocols like THREDDS, OpenDAP, pyDAP, ERDDAP, NetCDF etc.

Reagan also covered a bit of the things that are expected to show up in the near future. Among the interesting bits there, was DDN:s work on integrating iRODS directly into the storage controllers in their storage systems hardware, and the ongoing NSF funded project to create a nationwide (as in US) and discipline-crossing data federation network. Read more about that at datafed.org.

One thing I found extra interesting, was that Reagan mentioned some group (CIBER-U) doing MediaWiki integration of iRODS! Gotta figure out if they do share their code ... sounds really interesting!

Among my personal notes for further checking up, is also the "Fedora Commons" project, that seems to have a lot of overlap with iRODS, so I'd be interested to figure out more how they do actually compare in terms of key features.

My frequent need to produce plots and graphs of different statistics at UPPMAX finally forced me to learn some R (which is a good thing). With the help of R studio, I finally got started.

My first plot was an overview of the Next Generation Sequencing (NGS) storage at UPPNEX, since it's start in 2009/2010, until today. It's reaching 1 PB even though this graph does not even include all data! (temporary working data is excluded, due to difficulties to track NGS and non NGS data separately). You can imagine some exponential component there, no? :)

For you next gen sequencing bioinformaticians interested in getting some hands on new cloud based technologies for computation, mainly around the hadoop framework, and being around in Uppsala at the end of May 2012, may want to have a look at this:

• Hadoop and sequencing data processing hackathon at UPPMAX
• Here if more info on the COST webpage

As the event page on UPPMAX states:
"The hackathon will focus on next challenges that cloud adoption poses: massively distributed data processing frameworks such as Hadoop, distributed cloud databases and distributed bioinformatics applications."

Based on my experiences from a very useful (as in getting new hands on experience) and interesting (as in making new contacts) hackathon at CSC in Helsinki, Finland, I am sure this will be a highly interesting and useful hackathon as well, for all who are faced with the challenges of big sequencing data.

Apply before April 30 to get (EU/COST action: SeqAhead) funding! See you in Uppsala at the end of May!

(I had this post in draft for too long. Time to publish, as is)

The Semantic MediaWiki conference Fall 2011 in Berlin is over, so time to summarize some thoughts and impressions.

It was my first SMWCon at all. A bit late regarding that I did a Google Summer of Code project for SMW in 2010, but my finances were kind of inexistent then. Happy to get the chance now though.

Before turning to some of the individual talks, just a note of two general things I found interesting (I wish I would have time to review each of them, since there was so much interesting stuff...):

1. There were a remarkable amount of talks on connecting SMW with the rest of the Semantic Web, through RDF, SPARQL etc. Cool, SMW is seemingly becoming a natural choice of platform for SemWeb publishing!
2. The proportion of bio-people were also a bit remarkable. Apart from SNPedia founder Mike Cariaso, there were a whole bunch of others, including Salvadore from the GeneWiki project, Toni .... (and me) .... I guess it reflects how good SMW handles the need to give structure to very heterogenous datasets, so typical for the Life Sciences.

The talks

Note that you can now find slides and videos for most of the talks, online:

• Slides
• Videos

The conference started with one tutorial day, followed by the main days for talks. The tutorials turned out to be so interesting though, so that most people seemed to attend them as well.

Find below my very brief notes/impressions on some talks that I found specially interesting, for my use cases and interests:

Very nice UIs

Daniel Hansh from OntoPrise showed off their SMW+ Community Edition package, which includes SMW, Halo and other extensions. This is quite cool stuff, with really helpful and slick UIs, so let's hope it will remain open source! :) (Slides, Video)

Performance optimizations in SMW

Markus Krötzchs talked on "Saving C02: Top SMW Performance Issues and How to Address Them". The slides are cram full of link to more detailed in formation, so this I'll have to study in more detail. (Slides , Video)

SMW reworkings for better RDF support

As said, there were lots of RDF talks on the conf. One of which is some reworkings of the SMW internals to support the RDF and SPARQL models better. Markus Krötzch gave an overview of the role between SMW and RDF in his talk "Connecting SMW to RDF Databases: Why, What, and How?" (Slides)

Keeping track of changes, even at the fact level!

Jeroen De Dauw presented a new extension: "Semantic Watchlist", to replace a number of (in Jeroens opinion, somehow lacking) extensions. Looks very welldone! (Slides, Video)

Powerful transforming data from RDF on-demand

William Smith, Christian Becker and Andreas Schultz presented some very cool sutff: "Neurowiki: How we integrated large datasets into SMW with R2R and Silk / LDIF". The LDIF framework seems to do a lot of what RDFIO does, but in a much bigger and more capable framework. Really cool stuff! (Slides 1, Slides part 2, Video)

SMW classes as XML

Yaron Koren presented an idea to store "classes" in SMW as XML/JSON in one single location, rather than as now, in three different places (Category, Template and Form). (Slides, Video)

Towards a Semantic Wikipedia!

Denny Vrandečić and Daniel Kinzler presented the WikiData extension, as part of their work to make a "Semantic Wikipedia" a reality ... and based on another nice demo-project they did: Shortipedia. This is gonna be hot stuff! (Slides, Video)

Linked Data - increasingly important topic for SMW

Anja Jentzsch from the LODD presented Linked Data, and the best practices for how to publish RDF data, so that you really "get connected" to the evolving Semantic Web. An increasingly important topic, as shown by the increased interest in connecting SMW with the outside world. (Slides, Video)

RDFIO / Hooking up SMW with external tools via SPARQL

My talk ... (as blogged earler)

More RDF to Wiki Page title mapping strategies

Michael Erdmann also presented how they do Data integration with SMW+ and OntoBroker. They interestingly use a similar strategy as RDFIO in order to nicefy wiki page titles. Interesting! ... maybe there is a way to consolidate all these efforts in a reusable way? (Slides, Video)

More RDF to Wiki Page title mapping strategies

Jeff Pan talked on "Tractable Reasoning". Very interesting! They focus on "making reasoning reasonable", that is, computationally feasible ... and seem to have succeeded as well, their REL reasoner has shown to totally outperform reasoners such as Pellet for more or less any kind of ontology, cool! (I found out it's quite easy to beat pellet though, earlier, with SPARQL/ARC, and especially with PROLOG). (Video)

More RDF to Wiki Page title mapping strategies

Markus Krötzsch and Jeroen De Dauw talked about the next steps for Semantic MediaWiki. Many great things happening: Foundation started, ... (Slides, Video)

Excel-like statistics in SMW

Benedikt Kaempgen demonstrated some supercool stuff, something like a pivot browser for Semantic MediaWiki, in order to get more "Excel-like" statistics in SMW. They use the Spark extension by Jeroen, to query SPARQL endpoints and similar stuff, from javascript. Supercool! (Slides, Video)

SMW as a semantic browser

Benedikt also showed of their Semantic Web browser, that only requires "Equivalend URIs" to be defined for pages, and then let's you browse the data in the wiki in it's original format. Interesting since that matches perfectly with RDFIO, in that RDFIO complements this with also RDF export and querying in original format, using basically the same strategy! (Slides, Video)

SNPedia

Mike Cariaso talked about what's new with SNPedia ... lot's of cool stuff (apart from how cool SNPedia is just in itself!) (Video)

Lightning talks

Not to forget, there were also a whole bunch of very interesting lightning talks ... too many for me to have time to cover here now. One thing you really should not miss though, is the SPARK extension by Jeroen De Dauw, to query SPARQL endpoints via javascript, for visualizations. Wow! (Slides).

Also, for you Bio-people readint this, the SNPedia + GeneWiki mashup, you'll probably find interesting! (Video)

Well, you better watch them all anyway, they are only 5 minutes each, and there's just too much good stuff there, so I can't cover it all:

• Ligthning talk videos

I blogged it earlier, but better to get everything in one post, so taking the summary again:

After doing my GSoC project for Wikimedia foundation / Semantic MediaWiki in 2010, resulting in the RDFIO Extension, I finally could make it to the Semantic MediaWiki Conference, which was in Berlin in September.

Now, the video of my talk, "hooking up Semantic MediaWiki with external tools via SPARQL" (such as Bioclipse and R), is out on YouTube, so please find it below. For your convenience, you can find the slides below the video, as well as the relevant links to the different stuff shown (click "read more" to see it all on same page).

From today evening, I'm taking one week off from work to make a sprint to try to finalize the RDFIO extension, for RDF import and export in Semantic MediaWiki.

This will be a required step for finalizing the vision described in my SMWCon fall 2011 talk, the other month.

I developed RDFIO as part of Google Summer of Code 2010, and it got into a working proof-of-concept state. Some issues, such as with performance, never were resolved though. Also it depended upon two other modules, which, after Semantic MediaWiki changed a lot of it's internals in version 1.6, have still not been updated to support these, leaving RDFIO in a state where it does not support SMW 1.6.

So, a little sprint is definitely needed, to get RDFIO in working condition again. At the same time I hope to look at it with fresh eyes, after having a lot more coding experience now, than when I coded RDFIO, after one year of quite some Java and Python development in my work at UPPMAX.

Things I plan to have a look at (or at least ponder):

• Look at possiblity to use the Wiki Object Model instead of the Page Object Model / SMWWriter combo (which unfortunately is not SMW 1.6 compliant anymore)
• See if/how I can make more use of the new infrastructure in SMW, summarized by Markus in this post in SMW devel mailinglist.
• Take an overall fresh look at the architecture of the code ... try to follow Domain Driven Design principles much better, to get clean maintaineable code.
• Use existing MediaWiki feature uch more, such as the HTMLForm form builder class (for Specialpages and the like).
  (More suggestions like this highly welcome!)
• Import ARC2 library via the ARCLibrary extension rather than with a separate import.
• Use the existing "Equivalent URI" special property, instead of the custom "Original URI" (don't remember why I created a custom one...)
• Run big imports as jobs?
• OWL class import (as categories) ?
• Allow updating Wiki articles from any connected store, by using the new SMW Internals?
• Other things?

I would love to get some feedback and input to the project during this intensive week, so don't hesitate to drop in at #semantic-mediawiki on irc.freenode.net (IRC chat) or in the SMW-devel mailing list! My contact options, summarized:

• Follow code dev at github: github.com/samuell/RDFIO2
• RSS feed for commits of the above repo
• IRC: samuell # chat.freenode.net ( #semantic-mediawiki room )
• Mailinglist: SMW-devel
• Twitter: twitter.com/samuellampa
• G+: gplus.to/saml
• Skype: samuellampa
• Mail: samuel.lampa # gmail.com

Looking forward to your input during this week!

I was upgrading to Ubuntu 11.10 the other day, after sticking to the pre-Unity, Ubuntu 10.10. Now I thought the Unity stuff would have gotten better, and sure it has, but still I couldn't stand it ... so, switched to Xubuntu (sudo apt-get install xubuntu-desktop), which uses XFCE as desktop environment instead of GNOME.

And after some tweaking ... Wow! ... so snappy, so beatiful, so fast, so consistent! just wow ... It's definitely my distro of choice from now on ...

Have to throw a screenshot at you (sorry, haven't got the Lightbox up working yet):

For your referece:

• As theme in "Settings > Appearance", use "MurrinaCrystal"
• In "Settings > Window Manager", use "Aqualemon"
• Install the Geany text editor (sudo apt-get install geany). gedit will look ugly in XFCE, but Geany just is so much more suited for development stuff anyway ... I just love it, too ...
• Do some tweaking of Geany
• Tweak the terminal a slight bit
• Install the Lucida Grande font, and make Lucida Grande, Regular, 9, be the default font in
  "Settings > Appearance > Fonts"
• And probably some minor tweak I forgot ...
• ... and you're there!