Sunday, 31 March 2013

Fibonacci numbers in Python and Java

Fibonacci numbers,  introduced by Leonardo Pisano Bigollo nearly a thousand years ago, were used to solve a fascinating puzzle about the growth of an idealised rabbit population. Below are the first 11 Fibonacci numbers Fn where n=0, 1, 2,..., 10:

sequenceF0F1F2F3F4F5F6F7F8F9F10
number011235813213455

or we can describe the Fibonacci numbers as
Fn = Fn-1 + Fn-2 (when n>=2)
To write a program to return a Fibonacci number given a sequence, there are many possible implementations. I shall present 2 solutions implemented in both Python and Java, where the first one is using recursion, and the second one is using a while loop, however, the performances of these 2 solutions are drastically different when the sequence is bigger.
Below are the results when running a Python test script. The recursion way is already showing very poor performance comparing to the while loop way.
> python test_fib.py 
.Working out Fibonacci(40)=102334155 recursively took 67.310534 seconds
.Working out Fibonacci(40)=102334155 with while loop took 1.8e-05 seconds
Working out Fibonacci(50)=12586269025 with while loop took 1.6e-05 seconds
Working out Fibonacci(60)=1548008755920 with while loop took 1.8e-05 seconds
..
----------------------------------------------------------------------
Ran 4 tests in 67.312s

OK
If we look at the results when running a similar timed test in Java, the same recursion is not as bad as in Python, though still far poorer than the while loop implementation.
> /somewhere/jdk1.6.0_33/bin/java -classpath ".:" fun.Fib
Working out Fibonacci(40)=102334155 recursively took 408 ms
Working out Fibonacci(40)=102334155 with while loop took 2033 ns
Working out Fibonacci(50)=12586269025 with while loop took 2115 ns
Working out Fibonacci(60)=1548008755920 with while loop took 2368 ns
The reason why recursion is slow in this case is because there are a lot of duplicated operations as the sequence grows. For example, to solve F(5), we need to solve F(4) once, F(3) twice, F(2) three times, and F(1) twice, as shown below

or the number of recursion required given a sequence n is
[1+(n-2)]*(n-2)/2 + (n-3) or n2/2-n/2-2  (when n>=4)
The complexity of using recursion is then actually O(n2), where, on the other hand, the while loop is O(n).
Below are the source code that are used in the test.
fib.py
import datetime


class Fib:
    def recursively(self, n):
        if n == 0:
            return 0
        elif n <= 2:
            return 1
        else:
            return self.recursively(n - 1) + self.recursively(n - 2)

    def whileLoop(self, n):
        if n == 0:
            return 0
        previous, fib, currentIndex = 0, 1, 1
        while currentIndex < n:
            previous, fib = fib, previous + fib
            currentIndex += 1

        return fib

    def timedRecursively(self, n):
        start = datetime.datetime.now()
        result = self.recursively(n)
        end = datetime.datetime.now()
        elapsed = end - start
        print "Working out Fibonacci({})={} recursively took {} seconds".format(str(n), str(result), str(elapsed.total_seconds()))
        return result

    def timedWhileLoop(self, n):
        start = datetime.datetime.now()
        result = self.whileLoop(n)
        end = datetime.datetime.now()
        elapsed = end - start
        print "Working out Fibonacci({})={} with while loop took {} seconds".format(str(n), str(result), str(elapsed.total_seconds()))
        return result

test_fib.py
import unittest
from fib import Fib


class TestFib(unittest.TestCase):
    def setUp(self):
        self.fib = Fib()

    def test_recursively(self):
        self.assertEqual(self.fib.recursively(0), 0)
        self.assertEqual(self.fib.recursively(1), 1)
        self.assertEqual(self.fib.recursively(2), 1)
        self.assertEqual(self.fib.recursively(3), 2)
        self.assertEqual(self.fib.recursively(4), 3)
        self.assertEqual(self.fib.recursively(5), 5)
        self.assertEqual(self.fib.recursively(6), 8)
        self.assertEqual(self.fib.recursively(7), 13)
        self.assertEqual(self.fib.recursively(8), 21)
        self.assertEqual(self.fib.recursively(9), 34)
        self.assertEqual(self.fib.recursively(10), 55)

    def test_whileLoop(self):
        self.assertEqual(self.fib.whileLoop(0), 0)
        self.assertEqual(self.fib.whileLoop(1), 1)
        self.assertEqual(self.fib.whileLoop(2), 1)
        self.assertEqual(self.fib.whileLoop(3), 2)
        self.assertEqual(self.fib.whileLoop(4), 3)
        self.assertEqual(self.fib.whileLoop(5), 5)
        self.assertEqual(self.fib.whileLoop(6), 8)
        self.assertEqual(self.fib.whileLoop(7), 13)
        self.assertEqual(self.fib.whileLoop(8), 21)
        self.assertEqual(self.fib.whileLoop(9), 34)
        self.assertEqual(self.fib.whileLoop(10), 55)

    def test_timedRecursively(self):
        self.assertEqual(self.fib.timedRecursively(40), 102334155)

    def test_timedWhileLoop(self):
        self.assertEqual(self.fib.timedWhileLoop(40), 102334155)
        self.assertEqual(self.fib.timedWhileLoop(50), 12586269025)
        self.assertEqual(self.fib.timedWhileLoop(60), 1548008755920)

unittest.main()

Fib.java
package fun;

public class Fib {


    public static void main(String[] args) {
        recursively(1);
        whileLoop(1);

        long start = System.nanoTime();

        long result = recursively(40);
        long elapsed = (System.nanoTime() - start) / 1000000;
        System.out.println(String.format("Working out Fibonacci(%s)=%s recursively took %s ms", 40, result, elapsed));

        start = System.nanoTime();
        result = whileLoop(40);
        elapsed = (System.nanoTime() - start);
        System.out.println(String.format("Working out Fibonacci(%s)=%s with while loop took %s ns", 40, result, elapsed));

        start = System.nanoTime();
        result = whileLoop(50);
        elapsed = (System.nanoTime() - start);
        System.out.println(String.format("Working out Fibonacci(%s)=%s with while loop took %s ns", 50, result, elapsed));

        start = System.nanoTime();
        result = whileLoop(60);
        elapsed = (System.nanoTime() - start);
        System.out.println(String.format("Working out Fibonacci(%s)=%s with while loop took %s ns", 60, result, elapsed));
    }

    static long recursively(long i) {
        if (i == 0) return 0;
        if (i <= 2) return 1;
        else return recursively(i - 1) + recursively(i - 2);
    }

    static long whileLoop(long i) {
        long previous = 0, fib = 1, currentIndex = 1;
        while (currentIndex < i) {
            long newFib = previous + fib;
            previous = fib;
            fib = newFib;
            currentIndex++;
        }
        return fib;
    }
}

Saturday, 30 March 2013

First impression of Python

Born some two decades ago, Python has certainly accumulated enough interests to become one of the most popular programming languages used in the financial industry.

Designed to be a write-less-do-more language, Python has certain aspects that differ from Java:
  1. It is an interpreted language, which means no compilation is required. 
  2. It is not a strongly typed language
  3. Its statement grouping is done by indentation instead of open and close braces
  4. It does not require declaration of variable or argument
  5. It supports complex number, for example (using Python interactive interpreter)
  6. >>> 1j + 2J
    (-2+0j)
    >>> (1+2j) * complex(2,3)
    (-4+7j)
    
  7. Strings can be subscripted. For example, the first character of a string has index 0.
  8. >>> word="HelloWorld"
    >>> word[4]
    'o'
    >>> word[1:3]   # using the slice notation, returning characters from index 1 to 2
    'el'
    
  9. It supports multiple assignment, for example, 'a,b=b,a+b', where the expression on the right-hand side 'b,a+b' are evaluated first before any of the assignments take place
  10. Not surprisingly, it supports lamda forms
  11. >>> def incresedBy(n):
    ...     return lambda x: x + n
    ...
    >>> f = incresedBy(42)
    >>> f(1)
    43
    
  12. It supports map and reduce functions as part of its functional programming offerings
  13. It supports Tuples, which are immutable and usually contain an heterogeneous sequence of elements
  14. >>> t=123,321,'hello'
    >>> t
    (123, 321, 'hello')
    
  15. It allows multiple inheritance, for example
  16. class DerivedClassName(Base1, Base2, Base3):
        <statement-1>
        .
        .
        .
        <statement-N>
    

Useful Sed command examples in UNIX

  1. Substitute a regular expression into a new value
  2. A simple example is to change the first occurrence of 'old' on each line from file1 to 'new' and save as file2
    sed 's|old|new|' < file1 > file2
    where 's' is the substitute command, '|' is the delimiter, 'old' is the search pattern in regular expression, and 'new' is the replacement string. We can also test string substitution using 'echo' like this
    echo "old, and some other old stuffs" | sed 's|old|new|'
    which will output 'new, and some other old stuffs'. Note that in this case it only replace the first occurrence of 'old' with 'new'. If we want to replace all 'old' with 'new', we could append a global replacement symbol 'g' after the last delimiter '|'
    echo "old, and some other old stuffs" | sed 's|old|new|g'
    which will output 'new, and some other new stuffs'
  3. Pick any delimiter you like
  4. We were using '|' as the delimiter in the above examples. In fact, we can pick any character as the delimiter, as long as it is not in the string we are looking for. For example,
    echo "old, and some other old stuffs" | sed 's/old/new/g'
    echo "old, and some other old stuffs" | sed 's:old:new:g'
    
    have the same effect.
  5. Use & as the matched string
  6. We could search for a pattern and add some characters on top of the original string like this
    echo "old, and some other old stuffs" | sed 's|[a-z]*|(&)|'
    which will output '(old), and some other old stuffs'. The special character '&' represents the matching pattern. We could have any number of '&' in the replacement string
    echo "old, 123 and 456 some other old stuffs" | sed -r 's|[0-9]+|& &|'
    which will output 'old, 123 123 and 456 some other old stuffs'. Note that we use '-r' here to indicate we are using extended regular expression such that sed will support the use of '+'.

Friday, 29 March 2013

Useful Find command examples in UNIX

Find command is one of the most useful commands in UNIX, as there are many options out there to accomplish various search tasks.

  • Find files which have been modified for more than or less than a certain time
  • We can use 'find * -mtime n' to search files under current directory based on modification time. The numeric argument n can be specified as +n, meaning greater than n days; -n, meaning less than n days; or n, meaning exactly n days. For example,
    to find files which were last modified for more than 30 days ago
    find . -mtime +30
    to find files which were last modified for less than 15 days ago
    find . -mtime -15
    to find files which were last modified for exactly 10 days ago
    find . -mtime 10
    There are other options like
    -mmin n, which refers to file's data was last modified n minutes ago
    -amin n, which refers to file's data was last accessed n minutes ago
    -atime n, which refers to file's data was last accessed n days ago
    -type f, which searches for regular file
    -type d, which searches for directory,
    -exec command, which executes command by passing in the find results,
    -perm /mode, which searches files with given permission 
    
  • Remove files which are older than n days
  • Since we understand how to search for files which were last modified for more than n days, we can then use the rm command to remove them
    find /some-dir/* -mtimes +30 -type f -exec rm {} \;
    
    '-exec' is used to pass in commands like 'rm'; '{}' is replaced by the file name that 'find' returns; and the '-exec' command has to end with '\;'.
    Another way will be
    find /some-dir/* -mtimes +30 -type f -print0 | xargs -0 rm
    where 'xargs' reads 'find' results and execute 'rm' on each of the 'find' results
  • Find files which contain particular strings
  • Below is an example to search all txt files starting from current directory for string "some-string"
    find . -name "*.txt" -print | xargs grep "some-string"
    

Using Python SimpleHTTPServer to share files

There are cases when I want to share some files within the intranet to other users.

Python comes with a simple built-in module called SimpleHTTPServer that can turn any directory in the system into the web server directory. I literally just need a single command line to accomplish this task, given Python is shipped with openSUSE!

Assuming I am running linux, and I want to share my home directory ~
cd ~
Start up the http server on port 10001
youyang@monkey-dev-01:~> python -m SimpleHTTPServer 10001
Now the http server is running on port 10001. Open a browser and type the following address, where monkey-dev-01 happens to be my hostname
http://monkey-dev-01:10001
Since there is no index.html under my home directory, the files in the home directory will be listed. Now my purpose is served, but there is one slight problem -- the program is running on the foreground, meaning I can't do anything on that open terminal as it continues to display the http requests when I browse the directories. To change the running process from foreground to background, I need to press CTRL+Z to pause the process, as shown below
^Z
[1]+  Stopped                 python -m SimpleHTTPServer 10001
and type 'bg' to move the process to run in the background
youyang@monkey-dev-01:~> bg
[1]+ python -m SimpleHTTPServer 10001 &
We can also use 'jobs -l' to display the jobs in current session
youyang@monkey-dev-01:~> jobs -l
[1]+  4127 Running                 python -m SimpleHTTPServer 10001 &
We can also start up the http server running in the background directly, by appending a '&' at the end of the command
youyang@monkey-dev-01:~> python -m SimpleHTTPServer 10001&

Unix Cheat Sheet


List a directoryAlternatively, type 'ls --help' or 'man ls' for more information
ls [path]list given path without any options. Without the path, it is listing current directory
ls -l [path]list given path in long listing format
ls -h [path]list given path in human readable format
ls -a [path]list given path, including hidden files (which has prefix dot)
ls -lrt [path]list given path, combining different options. long listing format, sort by modification time in reverse order
ls [path] list given path, redirect listing results into a file
ls [path] | morelist given path, showing listing results on one screen at a time

Change to a directory
cd <directory>change to the given directory
cd ~change to user's home directory
cd /change to root directory
cd ..go to parent directory
cd ../<directory>go to a sibling directory

Print current working directory
pwddisplay the current working directory

Create directory(ies)
mkdir <directory>create the directory under current working directory
mkdir -p <dir1>/<dir2>create directory2 under directory1; create parent directory2 as needed

Remove files or directory(ies)
rm <file>remove the file
rm -rf <directory>remove the contents of given directory recursively without prompt

Move a file or a directory
mv <source> <dest>rename source as dest, and move source to dest

Copy a file or a directory
cp <source> <dest>make a copy from source to dest
cp -r <source> <dest>copy recursively from source directory to dest directory

Download a file from http website
wget <url>download the resource to current directory
curl <url>download the resource to current directory

View a text file
less <file>view file page by page, with a lot of features
more <file>view file with screen length
cat <file>view file, but it will scroll to the end of file
cat <file> | moreview file, and piped the file content to more command

Create a text file
touch <file>create an empty file
vi <file>use vi text editor to create a file (need to save the file though)
echo "foo" >> <file>create a file with content 'foo'
cat > <file>enter multi-line texts and press control-d to save the content to a file

Change a file's modification or access time
stat <file1>display the stat of given file
touch -m -d '1 Mar 2013 02:53' file.txtchange the modification time of file.txt to be '1 Mar 2013 02:53', and create the file if it doesn't exist
touch -a -d '1 Mar 2013 02:53' file.txtchange the access time of file.txt to be '1 Mar 2013 02:53', and create the file if it doesn't exist

Compare two files
diff <file1> <file2>show the differences between file1 and file2
sdiff <file1> <file2>show file1 and file2 side by side

Pipes and redirections
<command> > <file>redirect command output to a file, e.g. ll > file.txt writes current directory listings to file.txt
<command> >> <file>appends command output to the end of a file
<command> < <file>redirect the standard input from the file
<command> < <file1> > <file2>redirect the standard input from the file1, and then redirect the standard output to file2
<command1> | <command2>pipe the output of command1 as the input of command2

File permissionsread=4, write=2, execute=1
chmod 700 <file(s)>file(s) owner can read, write and execute
chmod 600 <file(s)>file(s) owner can read and write, but not execute
chmod 400 <file(s)>file(s) owner can read only
chmod 755 <file(s)>file(s) owner can read, write and execute; both group owner and others can read and write
chmod 644 <file(s)>file(s) owner can read and write; both group owner and others can read only
chmod u+x <file(s)>grant file(s) owner execute permission
chmod u-x <file(s)>remove execute permission from file(s) owner
chmod a+rw <file(s)>grant everyone read and write permission to the file(s)

Other useful comands
df -hreport file system disk space usage in human readable format
du -hestimate file space usage in human readable format
datedisplay or set the current system date and time
topdisplay linux tasks, as well as CPU and memory stats
ps -aefdisplay a snapshot of current processes
ln -s <target> <link_name>create a symbolic link to the target with link_name
aliasdisplay current defined aliases or create an alias
!<command>invoke previous run of the given command

Wednesday, 20 March 2013

Most import aspects of software development

After a few years of software development, it is a good time to recap what are the most important aspects that I should be aware of when working any project.
  1. The code base should be well tested that it has unit-tests, integration/acceptance-tests, external/smoke-tests. Unit test is about testing a single class by mocking its dependencies if necessary. Integration or acceptance tests should be testing the APIs provided by your application, particular front-to-back workflows, story-based features (as someone, I should be able to do something, etc.). External or smoke tests are verifying that the contracts between your application and your external dependencies are intact. There are a few reasons why automated tests are important:
    1. developers have the confidence to refactor or make functional changes as they can run the automated tests to check whether there are any breaking changes
    2. no more painful and error prone manual tests conducted by the QA team -- cost savings, although UAT (User-Acceptance Test) is still required
    3. greatly shortens the release cycle because you spend a lot less time doing the manual tests
  2. Modeling of the business objects is important because it not only affects how easy it is to query your business objects from the persistent layer, but also the performance of querying. One of the project that I worked on was storing a map of data into an oracle database as a CLOB data type (imagine a map is marshalled as XML string and stored as a DB column). It turned out to be rather inefficient to query any field inside that CLOB column, as you have to load the whole CLOB to your application!
  3. Availability and scalability are also crucial to any production system. Even Facebook does not guarantee that their query engine is available 100% of time (as they still have to do the manual failover of their name nodes, part of their Hadoop DFS cluster, from time to time). I won't treat Facebook as a mission critical application that has to be available all the time, but you can sense that to achieve a 100% availability, you will have to really think about your failover or DR (Disaster-Recovery) plan. Scalability is a tricky topic, as you can either scale horizontally by adding more nodes, or scale vertically by adding more cores/memories to the existing nodes (further reading on Amdahl's law). Depends on where the bottleneck of growth is, you might have database scalability, application scalability, caching scalability, etc.