Tuesday, July 31, 2007

Django or Rails for Grails user

I had a wonderful experience with Grails after finishing two projects. If you are a new Java programmer who's looking for a good web frameworks, Grails will be your savior just like what it was to me.

Here's what I like about Grails

  • Relatively easy to learn compared to other Java web frameworks.
  • Provides the basic things that one need, it's all there and there's not much
    to configure (ex. No XML).
  • It's agile!
  • Good ORM built on top of Hibernate.
  • Grails was developed mostly in Java so it's enterprise-ready and it scales well.
  • Uses groovy. Easy to learn Java scripting language
  • Although it's not really Java, it does take advantage of the massive amount of Java libraries available, including ones I use like the Java Excel API and JFreeChart.
  • I like database stored procedure programming, and calling stored procedure
    from Grails is easy as well as executing complex sql queries.
  • Integrate my choice of Javascript framework easily. I use jQuery.
  • Grails helps me code properly because of its MVC pattern and services
    concept.
  • Grails plug-in are so cool like Quartz and Searchable

One of the requirements of my next project is that it be hosted on the internet. My client doesn't care what technology I use but wants the hosting fee to be as low as possible. And as we all know, Java web hosting is not cheap compared to others.

I hope to use a non-Java web framework, Django or Ruby on Rails, that I could be comfortable with because of the knowledge and experience I have with Grails. I'd like to see the things that I like in Grails on them. Right now, I'm gearing towards Django because I know a little of Python.

Comments & Suggestions?

Tuesday, July 17, 2007

Beginners guide for Google Spreadsheet API

Programming with web API's is a new thing to me, and this is the first web api I have used. I have encounter a number of troubles before getting things to work and that is why I am writing this post as an additional to their guide especially for beginners like me.

I am using python to work with the api, but this could be applicable to other languages since most of the problems I am dealing with are XML related I guess. For this little & simple application that I am trying to create the operations I perform are:

  • Connecting to Google Spreadsheet Service
  • Opening the spreadsheet and worksheet you wish to work with
  • Writing into the spreadsheet
  • Reading spreadsheet

Connecting to Google Spreadsheet service is easy, here's the code:


gd_client = gdata.spreadsheet.service.SpreadsheetsService()
gd_client.email = 'your_username'
gd_client.password = 'your_password'
gd_client.ProgrammaticLogin()



Opening the speadsheet and worksheet

As of now, the api doesn't allow us to create a spreadsheet, we can only work on the existing spreadsheet. The first thing to do is to get the key which will tell the api what spreadsheet you want to work with. Currently I am getting the spreadsheet key by opening the document itself and you could get from the url.

You can also code your way to get your spreadsheet's key like this.


spreadsheet_feed = gd_client.GetSpreadsheetsFeed()
for spreadsheet in spreadsheet_feed.entry:
print spreadsheet.id.text.rsplit('/', 1)[1]



Now we need to know which worksheet we want to work with, the only way I know to get the worksheet key is to code it. We select one of the spreadsheet key from the above


spreadsheet_key = 'o04042323673667114780.4164488110474914499'
worksheet_feed = gd_client.GetWorksheetsFeed(spreadsheet_key)
for worksheet in worksheet_feed.entry:
print worksheet.id.text.rsplit('/', 1)[1]



Now we know the spreadsheet key and worksheet key, we can now start reading or writing.


spreadsheet_key = 'o04042323673667114780.4164488110474914499'
worksheet_key = 'od6'
gd_client = gdata.spreadsheet.service.SpreadsheetsService(spreadsheet_key, worksheet_key)
gd_client.email = 'your_username'
gd_client.password = 'your_password'
gd_client.ProgrammaticLogin()



Writing into the spreadsheet

Currently, the API provide 2 ways of writing by row and by cell. There are things you need to do and know when writing by rows. The api treats the first row as the column header, so before you write by rows you need to set the column header manually.

We need first define a dictionary first, before we can write by rows.


dict = {'firstname':'Kervin', 'lastname':'Ang', 'age':29, 'birthday': '9/12/1977'}
gd_client.InsertRow(dict, spreadsheet_key, worksheet_key)



The keys of the dictionary will tell to which column we will write the data. As you can see spaces, capital letters, and special character are disregarded. For example, if your column header are $ Money Market and Php Money Market the keys of dictionary are moneymarket and phpmoneymarket respectively.

This where I spend I lot of making things work, underneath the api deals with a lot of XML/ATOM/RSS as I don't have any idea about it.

To check if the insert row is success, you could do something like this


entry = gd_client.InsertRow(dict, spreadsheet_key, worksheet_key)
if isinstance(entry, gdata.spreadsheet.SpreadsheetsList):
print "Insert row is success."



Writing by cell will be very useful in setting up column header.

Reading spreadsheet

Same with writing into spreadsheet, you could read the spreadsheet by rows or cells. An example of reading all the rows of spreadsheet


list_feed = gd.client.GetListFeed(spreadsheet_key, worksheet_key)
for entry in feed.entry:
print "%s: %s\n" % (entry.title.text, entry.content.text)



You could search or filter rows using the gdata.spreadhseet.service.ListQuery like this


query = gdata.spreadsheet.service.ListQuery()
query.sq = 'age==29'
list_feed = gd_client.GetListFeed(ss_key, ws_key, query = query)



Just like writing into spreadsheet by rows, you need to provide the correct key to match the column head.

With this I was able to retrieve some data from the internet and store them in google spreadsheet for my data analysis, and I am looking for more ways of using it.

Wednesday, July 11, 2007

Make your source code look cool on your blog with Pygments

After my previous post that mostly contains a python code, I wasn't satisfy at the way it look where I just simply format it with courier font. Luckily I discover Pygments.

From its website, Pygments is a generic syntax highlighter for general use in all kinds of software such as forum systems, wikis or other applications that need to prettify source code. Highlights are:

  • a wide range of common languages and markup formats is supported
  • special attention is paid to details that increase highlighting quality
  • support for new languages and formats are added easily; most languages use a simple regex-based lexing mechanism
  • a number of output formats is available, among them HTML, RTF, LaTeX and ANSI sequences
  • it is usable as a command-line tool and as a library
  • ... and it highlights even Brainf*ck!
Here are the steps I did to prettify my code in the post. (do this also if you don't want to install pygments)
  1. Go to http://pygments.org/.
  2. Let pygments highlight my code.
  3. Once my code are highlighted, I choose other highlighting style to suits my taste.
  4. View the page source code. Locate and copy the

    <link rel="stylesheet" href="http://www.pygments.org/media/pygments_style.css">

    <div class="hlcode">
    <div class="syntax"><pre>
    ...
    </pre></div>
    </div>

  5. Paste the html code.
  6. Put this style link into my template. Using style link could result to unfix highlighting style, instead you can copy the whole css into your html.

    <link rel="stylesheet" href="http://www.pygments.org/media/pygments_style.css">

Hope support for groovy could come soon. Enjoy highlighting your code.

Tuesday, July 10, 2007

My first Python program excites me: Source code

As someone has requested for the source code, here it is. Improvements are welcome.


#!/usr/bin/env python

from urllib import urlopen

try:
from xml.etree import ElementTree
except ImportError:

from elementtree import ElementTree
import gdata.spreadsheet.service
import gdata.service

import atom.service
import gdata.spreadsheet
import atom
import getopt
import sys

import string
from HTMLParser import HTMLParser
import re
from datetime import datetime


class bdoHTMLParser(HTMLParser):

def reset(self):

HTMLParser.reset(self)
self.getDate = None

self.rawdate = None
self.date = None

self.key = ''
self.data = {}



def handle_data(self, data):
if re.search(r'Unit Investment Trust Funds',data):

self.getDate = data
elif self.getDate:

try:
self.rawdate = data.split(' ', 1)[1].strip()

self.getDate = None
except:
print "Parsing date failed: %s" %(data)

elif re.search(r'BDO\D+Fund',data):
self.key = data

elif self.key:
self.data[self.key] = data.strip()

self.key = None

def main():
# SETUP

columns_header = ['phpmm', 'phpmmchange', 'mm', 'mmchange',

'phpbond', 'phpbondchange', 'bond', 'bondchange',
'balance', 'balancechange', 'equity', 'equitychange',

'fixedincome', 'fixedincomechange']
uitf_to_header_map = {'BDO Peso Money Market Fund':columns_header[0], 'BDO $ Money Market Fund':columns_header[2],

'BDO Peso Bond Fund':columns_header[4], 'BDO $ Bond Fund':columns_header[6],
'BDO Balanced Fund':columns_header[8], 'BDO Equity Fund':columns_header[10],

'BDO Fixed-Income Fund':columns_header[12]}

#ss_key = 'o04042323673667114780.4164488110474914499'
ss_key = 'pHJcUzPY3GTjD6jFA3zvLOQ'

ws_key = 'od6'

# GETTING THE SOURCE
page = urlopen('http://www.bdo.com.ph/NAV.asp')

html = page.read()
page.close()
print "Download Data Success"


# PARSE DATA
parse_data = bdoHTMLParser()
parse_data.feed(html)

parse_data.close()
print "Parsing Success"

# CONNECT TO GOOGLE SPREADSHEET
gd_client = gdata.spreadsheet.service.SpreadsheetsService()

gd_client.email = 'your_username'
gd_client.password = 'your_password'

gd_client.ProgrammaticLogin()
print "Login Success"

# CHECK DATE FORMAT
if re.match(r'\S{3,9}\s+\d{1,2},\s+\d{4}',parse_data.rawdate):

temp_date = datetime.strptime(string.join(parse_data.rawdate.split()), '%B %d, %Y')

parse_data.date = "%s/%s/%s" % (temp_date.month, temp_date.day, temp_date.year)

print 'Date Format Pass: %s' % (parse_data.date)

else:

print """Date Format Failed: %s\n
BDO Log Aborted.\n""" % (parse_data.rawdate)

return

# CHECK LAST DATE LOG
query = gdata.spreadsheet.service.ListQuery()

query.sq = 'date=%s' % (parse_data.date)

feed = gd_client.GetListFeed(ss_key, ws_key, query = query)

if len(feed.entry) != 0:
print """Entry Found.\nBDO Log Aborted.\n

%s %s""" % (feed.entry[0].title.text, feed.entry[0].content.text)

return

# CONSTRUCT DATA
data = {}
data['date'] = parse_data.date

for k, v in parse_data.data.items():

data[uitf_to_header_map[k]] = v
#print "%s = %s" % (k, v)

print 'Construct Data Success'


# WRITE DATA
entry = gd_client.InsertRow(data, ss_key, ws_key)

if isinstance(entry, gdata.spreadsheet.SpreadsheetsList):
print """BDO LOG Success\n

Enter: %s""" % (data)


if __name__ == '__main__':

main()





Thursday, July 5, 2007

My Grails Practices

After learning Grails and finish 2 small project with it in 4 months time, here are a number of the practices I do to develop faster and comfortable for me. As I am new to Grails as well as to java web development, the practices here may not be wrong and your comments are very much welcome.

A controller's action performing post and get.

I have a lot of that in my project, where it really helps me organize my code to be more readable and maintainable and less confusing. This is not against to be CRUD pattern that grails generate, but somehow I feel not comfortable with it, because I have to think two action's name instead of one. For example, I have view employee profile page where you can extends the the employee's contract or terminate the employee's contract. Here's how I code it

def extendEmployeeContract {
if (request.method == 'GET') {
// preparing the page for extending the employee's contract
}
else if (request.method == 'POST') {
// performing the employee's contract extension
}
}

instead of

def createEmployeeContractExtension {
// preparing the page for extending the employee's contract
}

def saveEmployeeContractEntersion {
// performing the employee's contract extension
}


Restricting an action to post only if applicable

Here's how grails make it easy for you, just define it in the allowedMethods attributes.

def allowedMethods = [checkEmployee:'POST']

This makes your application much more secure.

Do you manual assignment after object.properties = params

Sometimes I need to do some manual assignment when an attribute is your domain class, and be able to validate the object that we want to assign to an object's attribute. Doing so we avoid a hibernate error saying that a object's identifier is alter from 1 to 2 which are the id of the object.

Use java.sql.date for date attribute

Using java.sql.date could result into a date field on your database while java.util.date could give you a datetime field in your database which give me a problem when I perform a query with the dynamic find method. This gives me additional work to do because the object.properties = params won't work here but this bug is fix and will be on the next release.

Another reason I use java.sql.date is because of how I implement my user interface. I am not using the gsp datepicker tag, instead I use a regular text field which is much faster to input in for the user and you can use any javascript datepicker solution. For the date format I use yyyy-M-d and it suit very well to java.sql.date, use the java.sql.date.valueOf() static method. Here's how I implement it.

employee.birthday = params.birthday =~ /\d{4}-\d{1,2}-\d{1,2}/ ? java.sql.date.valueOf(params.birthday):null

This validates the user's input and if it's a invalid format, a null will assign to the date attribute which trigger the null constraints I define.

More will be from you, I be glad to hear it.

Monday, July 2, 2007

My First Python Program excites me

As I am decided to learn another programming language in addition to Delphi(pascal), Java, and Groovy, I choose Python. I am not in the position to say which language is better and there's no thing such as the best programming language. But what makes me choose Python over the other is because this is an easy to learn language that has been widely use in software application development.

For me the best and effective way to learn a new language is to develop an application with it. I want it to be something that has to do with the web and I look into what I can do with Django, but I can't come up with an good idea enough to motivate me to go thru the long learning process since I am still very new to python. One day when I am so sick and tired of visiting a certain web page where I need to keep record of a financial data into a excel file on a daily basis, this gives me an idea to automate this boring task. And this excites me as I have to deal with a lot of new things compare to what I do before, database application development with delphi on windows desktop platform.

Here I need to download the page, parse it with some help from regular expression, and write it into an excel file. But since I can't find a module that could read and write into an excel file, a thought came into my mind of exploring solutions from web services, since I've been hearing a lot of it and want to have a chance to work with it. After a short research I discover Google Spreadsheet API is what I need.

Putting it all together, in this short simple python application that I made I have touch a number of new things to me like HTML data processing with HTMLParser, Regular Expression, URLLIB, and Google Spreadsheet API. Now this is something really different from what I've been doing before.