Read the rest of this entry »

Avro is a data serialization format with rich features like data structures support, RPC support and lacks requiring generating code to read/write its files. From 1.4.0 upwards you can also use AVRO from within Hadoop’s MapReduce (only Java supports that though).

Here is a sample code snippet that helps you understand how one can serialize (or write, in human terms) a ‘Record’ data type of Avro using its Python module (Installable via `easy_install avro`).

# Import the schema, datafile and io submodules
# from avro (easy_install avro)
from avro import schema, datafile, io

OUTFILE_NAME = 'sample.avro'

SCHEMA_STR = """{
    "type": "record",
    "name": "sampleAvro",
    "namespace": "AVRO",
    "fields": [
        {   "name": "name"   , "type": "string"   },
        {   "name": "age"    , "type": "int"      },
        {   "name": "address", "type": "string"   },
        {   "name": "value"  , "type": "long"     }
    ]
}"""

SCHEMA = schema.parse(SCHEMA_STR)

def write_avro_file():
    # Lets generate our data
    data = {}
    data['name']    = 'Foo'
    data['age']     = 19
    data['address'] = '10, Bar Eggs Spam'
    data['value']   = 800

    # Create a 'record' (datum) writer
    rec_writer = io.DatumWriter(SCHEMA)

    # Create a 'data file' (avro file) writer
    df_writer = datafile.DataFileWriter(
                    # The file to contain
                    # the records
                    open(OUTFILE_NAME, 'wb'),
                    # The 'record' (datum) writer
                    rec_writer,
                    # Schema, if writing a new file
                    # (aka not 'appending')
                    # (Schema is stored into
                    # the file, so not needed
                    # when you want the writer
                    # to append instead)
                    writers_schema = SCHEMA,
                    # An optional codec name
                    # for compression
                    # ('null' for none)
                    codec = 'deflate'
                )

    # Write our data
    # (You can call append multiple times
    # to write more than one record, of course)
    df_writer.append(data)

    # Close to ensure writing is complete
    df_writer.close()

def read_avro_file():
    # Create a 'record' (datum) reader
    # You can pass an 'expected=SCHEMA' kwarg
    # if you want it to expect a particular
    # schema (Strict)
    rec_reader = io.DatumReader()

    # Create a 'data file' (avro file) reader
    df_reader = datafile.DataFileReader(
                    open(OUTFILE_NAME),
                    rec_reader
                )

    # Read all records stored inside
    for record in df_reader:
        print record['name'], record['age']
        print record['address'], record['value']
        # Do whatever read-processing you wanna do
        # for each record here ...

if __name__ == '__main__':
    # Write an AVRO file first
    write_avro_file()

    # Now, read it
    read_avro_file()

I hope the snippet explains enough to understand how one could write/read Avro Data Files. The same technique would work for Java/Ruby also, although they may have certain other abstractions. Comment if there needs to be anything corrected or bettered.

Note: If you’re new to using PyQt but are interested in great cross-platform GUI application development please read the PyQt Introduction article.

Designing a graphical interface for an application could be a tiresome task. There are guidelines to keep track of, layouts to maintain and more of such stuff. In the PyQt code samples you’ve seen so far we’ve written our interfaces in pure code. While this is fun and easy to do for little applications that consist of about 5 to 10 widgets, it’s not worth spending the time upon in creating fully blown up interfaces for complete applications.

The Qt Designer

Fortunately, Qt provides us a tool to design interfaces and to port it to usable code automatically. This tool is called the Qt Designer, and it comes with the Qt library bundle you installed. Additionally we would require a converter for processing the XML .ui files Designer produces into a python module .py file; also installed with the PyQt4 bundle.

Thus, to design in PyQt we need the two following tools:

• Qt Designer from Nokia’s Qt Software
• The pyuic4 script from Riverbank’s PyQt

Once you have Designer and PyQt4 tools installed, let’s fire it up and get started.

The initial interface might look familiar to those who have used Visual Studio, or Glade and such tools. For the rest, it’s intuitive enough to learn smoothly. Read the Qt Designer guide for more elaborate help if you still don’t find it usable.

In this article, we’ll be using Designer to design and create a simple Image Viewer application.

References: Qt Designer, The PyQt + Qt Designer Documentation

Image Viewer

Our Image Viewer application would be a very simple one, with options to Open an image file of major types (JPEG, PNG, GIF, etc.) and another to Quit. Upon opening of an image, we shall also set the status bar to show the dimensions of the image.

Designing the GUI

Open Qt Designer (‘designer’ command if on Linux) and choose File | New | “Main Window” under templates/forms expansion. This creates a new QMainWindow widget for us to work upon. The QMainWindow is basically a composition of a main-area widget, a menu-bar and a status-bar at the bottom, the very usual stuff an application consists of.

From the Widget Box, drag a Label onto the main window’s area. Right-Click at any of the free space available on the same area, choose the “Lay out” sub-menu and then hit the “Lay out Horizontally” option. Designer will automatically stretch your Label across the area available this way. We need this layout (any layout will do here, actually) since we want the image to show in the entire window area.

Now on to menus, Add a File menu and under it add an Open and a Quit option, as shown below. To add these just click where it says “Type here” and get typing; Hit Enter/Return key to get to the next element in the menu after you’ve typed.

Under the Property Editor, select and set the Label’s text property to blank, and its alignment property to AlignHCenter and AlignVCenter. This completes our UI design in Qt Designer. But don’t quit already, do explore the other properties of the widgets used and figure out their possible uses; get used to the interface and available tools like Form Previews, etc.

For instance, check out the properties like geometry (Height and Width), font, tooltip, etc. If you’re in a good reading mood, also checkout the amazing Style Sheet documentation in Qt Assistant, as it explains a lot about each Widget’s construct and how to go about customizing it till your heart’s content.

Finally, renaming the widget elements’ variables would be a good idea, since it will help us code our application in a much more readable manner. This is how I’ve named these widgets but feel free to follow whatever naming conventions you would like to use:

Just double-click on the Object Name items and edit them in-line after that. Note and remember your QMainWindow class’s object name. Once you’re done, save the file as ‘ImageViewerUI.ui‘.

References: QLabel, QAction, QMenuBar, Layouts Guide

Using PyQt’s pyuic4 script

Our next step requires the use of the aforementioned pyuic4 tool. It’s usage syntax is as follows:

pyuic4 input_file.ui -o output_file.py
# Optionally takes -x parameter to make the generated code executable.

Thus we should run, for our UI file:

pyuic4 ImageViewerUI.ui -o ImageViewerUI.py

This will create a file ImageViewerUI.py that we can now use. Looking into the file will show you that its just a long list of widget constructs and of applying property settings. There is no need to edit this file, and is advised not to since pyuic4 will over-write all changes in it if you run it once again after some changes you wanted to do.

Whenever you edit your .ui files, make sure to run the pyuic4 tool to convert it into its Python equivalent (or to update existing Python code such that it reflects the new changes).

Additional info: Try `pyuic4 –help` for more fine-tuning options.

Running the basic GUI

Create a new file ImageViewer.py to finally add the application logic. Before we get to the code part, let me explain how the Qt’s subclass approach works. A Qt Designer file is inherited by the QMainWindow-derived class, and then the interface is setup using the setupUi() call with an instance. This creates all the objects/widgets for the interface as attributes of the derived QMainWindow class so that it’s ready to show. Next, we need to add the application-level logic to this derived class as normal methods. Since we have access to all the widgets used in the interface via our class, we can do as we like with their available features. The following diagram explains the hierarchy we ought to follow each time we need to implement a UI class for use:

Thus, a basic class that can be run would look like:

#!/usr/bin/python

from PyQt4 import QtGui, QtCore
import sys

# Import the interface class
import ImageViewerUI

class ImageViewer(QtGui.QMainWindow, ImageViewerUI.Ui_mainWindow):
    """ The second parent must be 'Ui_<obj. name of main widget class>'.
        If confusing, simply open up ImageViewer.py and get the class
        name used. I'd named mine as mainWindow, hence Ui_mainWindow. """

    def __init__(self, parent=None):
        super(ImageViewer, self).__init__(parent)
        # This is because Python does not automatically
        # call the parent's constructor.
        self.setupUi(self)
        # Pass this "self" for building widgets and
        # keeping a reference.

    def main(self):
        self.show()

if __name__=='__main__':
    app = QtGui.QApplication(sys.argv)
    imageViewer = ImageViewer()
    imageViewer.main()
    app.exec_()
    # This shows the interface we just created. No logic has been added, yet.

Run this class and you can see the window you just created.

You MUST call self.setupUi(self) to make the UI file run the setup statements and build the GUI interface for use by our class.

References: QMainWindow

Adding Application Logic and other Code

Now to add the opening-an-image feature, let’s define a few methods in the class as:

#!/usr/bin/python
# -*- coding: utf-8 -*-

from PyQt4 import QtGui, QtCore
import sys

# Import the interface class
import ImageViewerUI

class ImageViewer(QtGui.QMainWindow, ImageViewerUI.Ui_mainWindow):
    """ The second parent must be Ui_<obj. name of main widget class>. \
      If confusing, simply open up ImageViewer.py and get the class \
      name used. I'd named mine as mainWindow and hence the use. """

    def __init__(self, parent=None):
        super(ImageViewer, self).__init__(parent)
        # This is because Python does not automatically
        # call the parent's constructor.
        self.setupUi(self)
        # Pass this "self" for building widgets and
        # keeping a reference.
        self.connectActions()

    def connectActions(self):
        self.actionQuit.triggered.connect(QtGui.qApp.quit)
        # Connect the Quit action's triggered signal
        # to a proper Quit method
        # given by qApp (which points to your QApplication
        # object).
        self.actionOpen.triggered.connect(self.openImage)
        # Connect the Open action's triggered signal
        # to load an image onto the image label.

    def openImage(self):
        # Lets get a user-provided file to open
        # using PyQt's QFileDialog class.
        fileName = QtGui.QFileDialog.getOpenFileName(
                        self,
                        "Open Image File",
                        QtCore.QDir.homePath(),
                        "Image Files (*.jpg *.jpeg *.gif *.png)"
                    )
        # Don't attempt to open if open dialog
        # was cancelled away.
        if fileName:
            self.imageLabel.setPixmap(QtGui.QPixmap(fileName))
            # Load the image file as a pixmap onto the
            # labelImage QLabel GUI object.

    def main(self):
        self.show()

if __name__=='__main__':
    app = QtGui.QApplication(sys.argv)
    imageViewer = ImageViewer()
    imageViewer.main()
    app.exec_()

Note the two new method additions connectActions() and openImage(). They complete the application logic that’s involved for our image viewing application example. Try to understand them better by seeing the PyQt classes used in them and referring them up in Qt Assistant for much more detailed information.

Now the application can open images via File | Open and can also be quit via File | Quit.

This is the final result of our work:

References: QPixmap, QFileDialog, qApp

Exercise

Try to build a simple text-based evaluator application that contains of two things:

• A text input box that allows for mathematical expressions input.
• An output label or text area (non-editable) for showing results of these expressions.

Classes you could use are QLabel, QLineEdit and QTextEdit. Another hint is to use Python’s ‘eval‘ function. As a bonus try to make the evaluation via Python safe.

Am mixing a lot of words ain’t I? This is for September + Hectic.

There’s the college technical events coming up on 14th, named ‘Pragmatics’. I’m responsible for the code event under it, and can almost smell the deadlines. Then the next day I leave for the college-provided tour across Kerala and a part of TN itself. All nicely ending with the Indian Python Conference 2009, around the month end.

There’s a great feature in Python versions 2.6 and up that I hardly see being used; it’s the ability to install modules and packages in a per-user local directory. I like this feature since it doesn’t have any super-user power requirements and lets me install packages, modules and even scripts in my own Home directory and use it just as normally as the other global files.

To do so, one must first create the local site-packages directory, and then place the required package or module file or folder under it. The following commands are all for UNIX. ~/ expands to the user’s $HOME automatically.

# To create the required directory
mkdir -p ~/.local/lib/python2.6/site-packages

Now place the module or package folder under this directory, or link to it for achieving the same installation effect as you would with a global site-packages directory. For example, for my django copy from svn I’d do:

ln -s ~/.django-svn/django-trunk/django ~/.local/lib/python2.6/site-packages/django

Running the Python interpreter (in the same user account) will show this working:

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> 

So there you have it, when you want a package just for your user account while developing just put it under the local site-packages directory. Python automatically adds this to its path (PYTHONPATH).

References: PEP 370 (Has notes for OS X and Windows users)