Software

Previous script xman_dos2unix in xman utility (latest) under GPL has problems on files or directories with space.

Simple work around with double quotes:

for file in "$@" ; do
...
done

Note that all references to $file requires double quotes too.

Another tedious work around with IFS and double quotes:

I use IFS=$'\n', and double quote references to files such as "${file}" to solve the space problems.

OLDIFS=$IFS
IFS=$'\n'
for file in $@ ; do
        # FIXME: I should have put these into a subroutine to reduce code duplications in explore().
        # However, I'm not familiar with subroutines in BASH yet.
        # echo "$file"
        IFS=$OLDIFS
        if [ -d "$file" ] ; then
                echo "Entering $file"
                pushd "$file" > /dev/null
                explore
                echo "Leaving $file"
                popd
        elif [ "`${FILE} \"${file}\" | ${GREP} text`" != "" ] ; then
                $DOS2UNIX "$file"
        fi
        IFS=$'\n'
done
IFS=$OLDIFS

I found that current dos2unix will produce an intermediate file and stop when it encounters a directory in the arguments. And it cant process folders recursively. Hence, I wrote these scripts to help me to convert all text files in a list of files and folders into DOS or UNIX format.

Explore directories and files recursively.

explore()
{
        local file=""
        ls -1 | while read -r file
        do
                # Recursively explore if we found a directory.
                if [ -d "$file" ] ; then
                        echo "Entering $file"
                        pushd "$file" > /dev/null
                        explore
                        echo "Leaving $file"
                        popd > /dev/null
                # Process it if we found a text file.
                elif [ "`${FILE} ${file} | ${GREP} text`" != "" ] ; then
                        # echo "Processing: $file"
                        $DOS2UNIX "${file}"
                fi
        done
} # end of explore()

Process each of the files and folders given in the command line.

for file in $@ ; do
        # FIXME: I should have put these into a subroutine to reduce code duplications in explore().
        # However, I'm not familiar with subroutines in BASH yet.
        if [ -d "$file" ] ; then
                echo "Entering $file"
                pushd "$file" > /dev/null
                explore
                echo "Leaving $file"
                popd
        elif [ "`${FILE} ${file} | ${GREP} text`" != "" ] ; then
                $DOS2UNIX "$file"
        fi
done

For the entire file, see xman_dos2unix in xman utility (latest) under GPL.

Get svn
Get git

Pros of svn:

• Good GUI tortisesvn for running svn commands.
• Partial checkout: Choose any sub-dir to checkout as a working copy.
• Centralized server: Save space by maintaining single global repository.
• Portable to Windows and Linux/Unix.
• File-based tracking: More detailed tracking of files.

Cons of svn:

• Centralized server:
  • When the server/network is down or damaged, you cant access your repository, or loss your repository forever.
  • Potential access conflicts to shared files. Frankly, I dont like people manipulating my repository.
  • Flat checkout: Checkout from remote repository transfers file by file, which can be quite slow through the network, especially dealing with many small files.
• Limited move/rename file: You'll need to commit moved/renamed files before manipulating them.
• No good mechanism for pulling or pushing updates to other repositories.
• .svn folders are everywhere in a working copy. Hate this when I want to copy a folder to somewhere.
• File-based tracking: More overhead in tracking.

Pros of git:

• Good GUIs qgit, citool available for visualizing branches and histories.
• Decentralized: If a server is gone, you have nothing to loss in your local copy.
• Space efficient: Single git repository is typically much smaller than a svn repository.
• Compressed checkout: Checkout from remote repository transfers data in compressed format, hence more network friendly.
• Better ways of file moving/renaming/deleting, but this is subjective.
• Simple branch & merge mechanism, for both local and remote repositories.
• Easy branch switching: Short command to switch a working copy from a branch to another branch (with little CPU), requires no folder duplications too. Compared to svn, you need to know which folder to look for for a particular branch. To create a branch in svn, you basically duplicate entire branching folder.
• Requires no user account. Simply publicize your repository, and it's up to anyone to pull your updates. Not really a feature, but git mode of working supports this well.
• Tree-based tracking: For low tracking overhead.

Cons of git:

• Full checkout: Must checkout entire working copy, as well as the entire repository (including all history).
• Multiple repositories: A full repository for each checkout.
• Lonely merging: I have to resolve conflicts by myself when pulling updates from remote repositories. (In svn, the one causing the conflicts will fix the problems.) You can settle conflicts with the authors if you can command them :).
• Use Linux/Unix File System specific features such as hard link in some cases, arguably a pro feature for space efficiency, but trading off portability. Needs Cygwin for Windows, which results in poor performance.
• Missing file-based tracking: Doesn't track file copy operations. We may not know the source copy of a file. It probably takes more time to obtain file specific information, e.g. history of a file.

Upgrading from single core processor to dual-core processor does not give you double performance for free most of the time. Software written to run single threaded has no idea how to fully utilize a dual-core processor. Fortunately, it is not too difficult to parallelize certain software such as Reversi, because:

1. Large amount of time is spent in small evaluation function.
2. Different evaluations can be computed independently.
3. Many evaluations are to be computed.

Let's see how do I quickly parallelize the Reversi in C++ under GPL license using merely a few lines of OpenMP and minor changes to existing codes.

Following is the code snippet that we want to parallelize, to get multiple iterations computed in parallel.

    for(iter = vmoves.begin() ; iter != vmoves.end() ; iter++) {
      m = *iter ;     
      buffer_b = b ; 
      buffer_b.move_at(m, value) ;      
      s = e.eval(buffer_b, value, (value+1)%2) ; 
      cout << "(" << m.x << "," << m.y << "): " << s << endl ;
      if(s > best_s) {
        best_s = s ;
	best_move = m ;
      }          
    }             

Ooops, we have to be able to compute iter statically (iter points to different possible moves), so that OpenMP can schedule and compute different moves in parallel. So we need to change iter a bit. We have:

Pre-requisite:

• freealut
• freealut-devel
• OpenEXR
• OpenEXR-devel
• gettext
• gettext-devel

$ yum install freealut freealut-devel OpenEXR OpenEXR-devel gettext gettext-devel
$ tar zxvf blender-2.42a.tar.gz
$ cd blender-2.42a
Edit compilation flags in config/linux2-config.py
$ scons -j 2  # Uses 2 processors.

When we write programs, we spend lots of time in managing memory. Improper memory management can lead to memory leak, therefore reserving memory that will never be used. Memory leak problems can be serious especially when an application needs to run for a long time. No matter how much memory you have, regular memory leaks can make you run out of memory eventually.

I run "valgrind" to check if my Reversi in C++ has memory leak problems.

$ time valgrind --tool=memcheck ./reversi
==4305== Memcheck, a memory error detector.
==4305== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==4305== Using LibVEX rev 1658, a library for dynamic binary translation.
==4305== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==4305== Using valgrind-3.2.1, a dynamic binary instrumentation framework.
==4305== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==4305== For more details, rerun with: -v
==4305==

(0,0): 9999990
7 1 1 1 1 1 1 1 1
6 1 1 0 0 0 0 1 1
5 1 1 1 0 1 1 0 1
4 1 1 0 1 1 0 0 1
3 1 1 0 0 1 1 0 1
2 1 0 1 0 0 1 0 1
1 1 1 0 0 0 0 1 1
0 1 1 1 1 1 1 1 1
  0 1 2 3 4 5 6 7

White: 44 Black: 20
Recursive evaluate: 53671516
Recursive end eval: 135214583
Evaluation: 44228633

==4305==
==4305== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 1)
==4305== malloc/free: in use at exit: 0 bytes in 0 blocks.
==4305== malloc/free: 289,208,130 allocs, 289,208,130 frees, 6,940,995,120 bytes allocated.
==4305== For counts of detected errors, rerun with: -v
==4305== All heap blocks were freed -- no leaks are possible.

real    142m53.138s
user    139m9.606s
sys     0m37.606s

Hooray!!! But a 3min run becomes 2+hours run. Can it be faster?

Let's catch up with the technology trends on multi-core processors. If you have bought a dual-core, or quad-core, or a large shared memory parallel machine, or a cluster of computers, then you can run this Distributed Parallel Reversi, xversi-java-parallel-v0.8b, to make them busy.

Running Distributed Parallel Reversi: Step-by-step

1. Launch a server in each of your machines that will provide reversi evaluation services.
• xman@sai xversi-java-parallel-v0.8b # java -Djava.security.policy=xversi.policy XversiEval localhost 9394
  Registered XversiInterface
• localhost: Machine name, or its IP.
• 9394: Port number for the client to connect to. Remember to set your firewall to allow this port.
2. Setup a machine info file called nodeinfo.txt.
• xman@sai parallel # cat nodeinfo.txt
  2
  localhost:9394
  localhost:9394
• 2: There are 2 entries about machine information.
  
• localhost: Machine name or its IP. I include two identical lines because I'm running this on a dual-processor or dual-core processor machine. This will make the parallel reversi to utilize the two processors in the system.
3. Launch the Java Applet.
• xman@sai parallel # appletviewer -J-Djava.security.policy=xversi.policy test.html
  Number of Node : 2
  before lookup
  Node 0 localhost:9394
  name 0 //localhost:9394/xversiInterface
  Node 1 localhost:9394
  name 0 //localhost:9394/xversiInterface
  after lookup
• I've forgotten how do you specify the policy to a browser to launch the Java applet. TELL ME IF YOU KNOW! ;)

Notice that in picture below, two processors in the system are running at full speed to evaluate different moves at the same time. :) Hence, if you bought a dual-core, quad-core, ... or even a cluster of computers, enjoy the benefit of parallel computing. :) Ideally it will run N times faster with N processors.

The latest Reversi in Java had been kept safely in my backups for quite some time. I recall that I did backup it into CDs, together with many versions of reversi I developed. After backup into CDs, all the files become capital letters, hence, I have to change all the files into correct case before I can launch the Java applet (Stupid backup). That took me quite some time, together with cleaning up many different versions. It was a mistake I didnt use any version control software such as CVS, SVN, ... during development. From now onwards, all changes shall check-in into my svn repository.
Here is the latest Reversi in Java, xversi-java-v0.7a under GPL, which added some little features on top of xversi-java-v0.6b:

1. Show a green dot of computer's last move.
2. Compute end game when the game is near end.
3. Draws the piece with better aspect ratio as we resize the board.

Lessons learned:

• Always document your changes to your documents, codes, ... Use tools such as CVS, SVN, ... to help tracking the changes.
• I shall prepare a changelog file in the future releases.
• Backup each version of your programs into a package such as .tgz, or .zip, such that when you recover from your backup, program files attributes, from filenames to their permissions, are recovered to the original. Versioning helps you to clean up the mess.
• Portable is good. :) After few years, these reversi programs are still running properly without much issues. :)