postgreSQL – inconsistent performance

August 2nd, 2011 — 8:55pm

We have this query to join 4 tables.

For some reason, the performance of this query was inconsistent on Server A. First run took 6 sec, 2nd run took 300 ms then 3rd run took 6 sec again etc…

We dump the data to other server and run the same query, very consistent result – about 100 ms almost all the time.

We even got “could not read from hash-join temporary file” error and crashed Server A. We end up finding out that work_mem of the Server A is set to very low compare to other servers.

So followed this: http://wiki.postgresql.org/wiki/Performance_Optimization and we changed 2 settings:

work_mem to 16MB – default is 1MB and it was the default value

effective_cache_size – this one was also default of 128MB… so I set it to 3GB (machie has 6 GB and according to the wiki, half of the size of RAM would be good)

Then restart the server and yes, the performance was so consistent and was producing the same result as other server.

Comment » | PostgreSQL

How to get IP address on your mac from Terminal

August 2nd, 2011 — 8:40pm

This will give you ethernet one

ifconfig en0 | grep ‘inet ‘ | cut -d ‘ ‘ -f2

This will give you airport one

ifconfig en1 | grep ‘inet ‘ | cut -d ‘ ‘ -f2

so if you are writing script to get your IP in either case you could:

MYIP=`ifconfig en0 | grep ‘inet ‘ | cut -d ‘ ‘ -f2`
if [$MYIP -eq '']; then
MYIP=`ifconfig en1 | grep ‘inet ‘ | cut -d ‘ ‘ -f2`
fi

MYIP=`ifconfig en0 | grep ‘inet ‘ | cut -d ‘ ‘ -f2`if [$MYIP -eq '']; then MYIP=`ifconfig en1 | grep ‘inet ‘ | cut -d ‘ ‘ -f2`fi

echo $MYIP

Comment » | Bash Shell, mac, OS X

Setting up mod_wsgi for Django app on OS X Snow Leopard (10.6.8) for Dev env

July 30th, 2011 — 4:44pm

We are writing Django app right now. Our dev machine is OS X 10.6.8.

Our dev team decided that we should have similar environment as production so I started to set up mod_wsgi on my macbook pro.

There are a few steps that was pain to figure out hence I decided to post hoping to help others.

[APACHE]

Apache is loaded already. It should be no pain unless your home is set up for FileVault and your application directory is located under your home directory. The error I was getting due to FileVault and having my app under home directory was “Forbidden Error”.

This should not be the case for Production environment since we won’t be using OS X nor FileVault. The fix is not recommended but I think it is acceptable on Dev environment. If you know the better solution, comments would be appreciated.

The fix is to change the  User and Group from www to your user name.

User www
Group www

then problem solved. How to find out your user name and group? Issue $ id from the terminal.

Oh and every time you change your httpd.conf, don’t forget to restart apache

$ sudo /usr/sbin/apachectl -k restart

[Apache - enable virtual host]

Just uncomment the line below in httpd.conf

Include /private/etc/apache2/extra/httpd-vhosts.conf

Then specify your DocumentRoot there in httpd-vhosts.conf

Drop index.html under the DocumentRoot you specified. You should see your page as you point browser to http://localhost

[Download and install mod_wsgi module]

# download

$ curl -O http://modwsgi.googlecode.com/files/mod_wsgi-macosx106-ap22py26-3.3.so

# rename

$ mv mod_wsgi-macosx106-ap22py26-3.3.so mod_wsgi.so

# copy

$ sudo cp mod_wsgi.so /usr/libexec/apache2/

Modify tthpd.conf: add the following line as last line of LoadModule

LoadModule wsgi_module     libexec/apache2/mod_wsgi.so

[Install mod_wsgi]

$ curl -O http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz

$ tar xvfz mod_wsgi-3.3.tar.gz

$ rm mod_wsgi-3.3.tar.gz

$ cd mod_wsgi-3.3

# make sure you will  specify your default python interpreter you are using.

$ ./configure –with-python=/Library/Frameworks/Python.framework/Versions/2.7/bin/python

$ sudo make install

if ALL is good then in /private/var/log/apache2/error_log, you should see:

Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8r DAV/2 mod_wsgi/3.3 Python/2.7.1 configured — resuming normal operations

Do some cleanup

$ make clean

add “django.wsgi” to your path_to_your_application/apache/

the script should look like

import os

import sys

path = ‘/Users/user_name/app_name

if path not in sys.path:

sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = ‘settings’

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

def application(environ, start_response):

status = ’200 OK’

output = “Hello World!”

response_headers = [('Content-type', 'text/plain'),

('Content-Length', str(len(output)))]

start_response(status, response_headers)

return [output]

Then add the following line to your httpd-vhosts.conf if you setup for virtual host otherwise httpd.conf

WSGIScriptAlias / /Users/user_name/app_name/apache/django.wsgi

If you point your browser to http://localhost, you should see ”Hello World!”

Now, you want to run your app. Comment out or remove application function from django.wsgi. So it will look like this:

import os

import sys

path = ‘/Users/user_name/app_name

if path not in sys.path:

sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = ‘settings’

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

Now refresh your browser, you should see your Django app home page! (i hope)

Comment » | Django, Python

HTML5 Offline Application – Debug Manifest

January 31st, 2011 — 5:16am

There are many good tutorial out there to test/experiment offline application such as Dive Into HTML5. In theory, if you follow the instruction, your sample offline application can be up and running in no time. However, that wasn’t the case for me and wanted to share this experience and hoping that this might help someone else out there who is about to pull their hairs.

Debug 1: Did I set web server correctly so that manifest can be served as manifest?

To test this, you can simply run cURL as follows from terminal

curl -I url_to_your_manifest

If you see content type returned as text/cache-manifest, you are good. So this is not the problem…

Debug 2. It doesn’t seems to be all the items on manifest are downloaded successfully…

This really almost killed me. I was using FF and Safari. I know it download something but not quite complete… I used script provided by Jonathan Stark to debug. The script indicates that I might will need to check manifest.

Safari Console

Safari Console

Spent quite long time to make sure everything seems to be correct… still something wasn’t quite right…. I then decided to use chrome and wow… I know exactly what file is the cause. Chrome console shows progress as shown below.

Chrome Console

Chrome Console

I finally got all the files right and now my application runs locally just fine. Yey!

In Summary, here is the list of things that you want to check for manifest/directory:

1. is file name/path correct? (case sensitivity) – to avoid this error, perhaps you should write a code to generate manifest.

2. Does it have right permission? – to avoid 403 error, perhaps you should re-apply permissions once you move the files.

That’s it for now… Now I will need to convert server side code in JavaScript to function locally.

Comment » | HTML, JavaScript, Mobile Web App

Backup File Cleanup task with Shell Script

January 17th, 2011 — 9:53am
How to check whether backup file newer than 3 days ago exist and if so delete older than 3 days via Shell Script?
#!/bin/bash
system_admin_email=sysadmin@email.com;
# check whether backup newer than 3 days ago exist…
filesOnBackup=`find ${network_volume} -name “*backup.gz” -type f -mtime -3`
echo ${filesOnBackup}  >> $logfile
if [ "${filesOnBackup}" != "" ]; then
echo “Deleting old backup files…”  >> $logfile
if [ -d ${backup_dir} ]; then
find ${backup_dir} -name “*backup.gz” -type f -mtime +3 -delete;
echo “done”;  >> $logfile
fi
else
echo “backup doesn’t exist on BU volume”  >> $logfile
SUBJECT=”backup file doesn’t exist on BU volume”
TO=”${system_admin_email}”
echo $SUBJECT | mail -s “$SUBJECT” ${system_admin_email}
fi

Comment » | Bash Shell, System Admin

Yey, Android Emulator on my MacBook Pro (OS X 10.5)

January 15th, 2011 — 10:01am

I got mobile Web Application using jQTouch and this has been tested for iPhone (iPhone simulator and my iPhone). Looking REALLY good and thanks to jQTouch, the implementation was so easy!

Now I wanted to test with Android so I followed instruction provided by Nick Nelson and also referenced Installing the Android Development Environment article from MOTODEV.

1. Download Android SDK for Mac OS X (intel) http://dl.google.com/android/android-sdk_r08-mac_86.zip
2. I copied mine to Macintosh HD: Developer: android-sdk-mac_86
3. Install android-sdk updates
3.1 Open terminal and run (takes a while… a min or two):
$ /Developer/android-sdk-mac_86/tools/android update sdk
then update starts. take a while (i think it was like 3 to 5 min)
4. Download and install Eclipse Classic 3.6.1: http://www.eclipse.org/downloads/
5. Specify Location of the Android SKD for Eclipse
5.1. Open the Preferences dialog
5.2. On the left side of the Preferences dialog, click Android.
5.3. In the SDK Location field, specify the directory that contains the Android SDK. My case is: /Developer/android-sdk-mac_86
6. We are all set: ready to launch.
6.1. Eclipse >> Window >> Android SDK and AVD Manager
6.2. Click New >> Create new AVD window opines
6.3. Name: name your device!
6.4. Target: Android 2.1-update1 – API Level 7
6.5. SD Card: 1024 MiB
6.6. Skin: Select any skin you like (please see image below for a few skins)
6.7. Click “Create AVD” – this will take a moment (it gave me rainbow for my case)
6.8. It will be listed in Android Virtual Devices list on Android SDK and AVD Manager window.
6.9. Select the device and Click “Start” >> “Launch” (it will take a while to launch)
HVGA400

HVGA400

WVGA853 Skin

WVGA853 Skin API Level = 7

Default HVGA Platform 2.3 API Level 9

3 comments » | HTML, JavaScript, Mobile Web App

Hex in HTML Randomly…

December 11th, 2010 — 8:42am

My web application does screen scraping to do some work.

One Monday, the returned page started to contain CR+LF+someHex+CR+LF. When I use cURL all is just fine but when I used Internet Command provided by 4D this sets of characters (CR+LF+someHex+CR+LF) was randomly inserted to returned HTML page.

What I end up finding out was that:

1. the web site I was accessing must have enabled chunked transfer encoding on their web server on “One Monday” or over the weekend.

2. The chunked encoding modifies the body of a message in order to transfer it as a series of chunks

3. chunked body contains chunk size which is represented in HEX (ah…) and the way I was receiving this page did not understand this encoding…

Solution?

Well I could re-write this with cURL that’s the best way to go and I will do that when I have a chance.

Since chunked encoding is only available in HTTP version 1.1. changing the version to 1.0 in request header solved this problem for now…

Comment » | HTML

strace for Mac OS X

August 13th, 2010 — 11:16pm

In Leopard it’s called dtrace, but it’s simpler if you just call dtruss

Here is the manual:

NAME
dtruss – process syscall details. Uses DTrace.
-
SYNOPSIS
dtruss [-acdeflhoLs] [-t syscall] { -p PID | -n name | command }
-
DESCRIPTION
dtruss prints details on process system calls. It is like a DTrace ver-
sion of truss, and has been designed to be less intrusive than truss.
Of particular interest is the elapsed times and on cpu times, which can
identify  both  system calls that are slow to complete, and those which
are consuming CPU cycles.
Since  this  uses  DTrace,  only  the  root  user  or  users  with  the
dtrace_kernel privilege can run this command.
-
OPTIONS
-a     print all details
-b bufsize
dynamic  variable  buffer  size.  Increase  this  if  you notice
dynamic  variable  drop  errors.  The  default  is  ”4m”  for  4
megabytes per CPU.
-c     print system call counts
-d     print relative timestamps, us
-e     print elapsed times, us
-f     follow children as they are forked
-l     force printing of pid/lwpid per line
-L     don’t print pid/lwpid per line
-n name
examine processes with this name
-o     print on-cpu times, us
-s     print stack backtraces
-p PID examine this PID
-t syscall
examine this syscall only
-
EXAMPLES
run and examine the “df -h” command
# dtruss df -h
examine PID 1871
# dtruss -p 1871
examine all processes called “tar”
# dtruss -n tar
run test.sh and follow children
# dtruss -f test.sh
run the “date” command and print elapsed and on cpu times,
# dtruss -eo date
-
FIELDS
PID/LWPID
Process ID / Lightweight Process ID
RELATIVE
relative  timestamps  to  the start of the thread, us (microsec-
onds)
ELAPSD elapsed time for this system call, us
CPU    on-cpu time for this system call, us
SYSCALL(args)
system call name, with arguments (some may be evaluated)
-
DOCUMENTATION
See the DTraceToolkit for further documentation under the  Docs  direc-
tory. The DTraceToolkit docs may include full worked examples with ver-
bose descriptions explaining the output.
-
EXIT
dtruss will run forever until Ctrl-C is hit, or if a command  was  exe-
cuted dtruss will finish when the command ends.
-
AUTHOR
Brendan Gregg [Sydney, Australia]
-
SEE ALSO
procsystime(1M), dtrace(1M), truss(1)

Comment » | Bash Shell

Bathroom Remodeling

August 13th, 2010 — 11:15pm

2008-11
We bought this house in Nov 2008. I am Japanese and taking bath has been just my part of my life. However, we fall in love with this house so much that we accepted fact that the house only had Shower in Master. We were thinking… well, bathtub can be easily installed! We can knock down the closet and place shower there and place bathtub where current shower stall is.

2010-01
So yeah… about one year has passed. We got estimate from a few contractors and I was like wow… placing bathtub is not so cheap especially in our case, major pluming work was involving. First we got estimate from homeDepot. They told me it will be like $30K. Then we met a couple of contractors – both were like $13k or so… because of the limited space we have, I have to pick small bathtub and of course, smaller shower…. I just didn’t see the value paying that much to have small bathtub with smaller shower space… Long story short, I decided that steam sauna is what I want…
1) Greener – it use much less water than bath
2) Don’t have to sacrifice on shower space
3) and a lots of benefit – just google “steam sauna benefit”

old bathroom

Old Bathroom

2010-08
We found one company called Bath-Pro who we feel we can trust and offers very reasonable price. The construction was finally started on Aug 9th.

2010-08-09 [Day 1]
They did demo

Bathroom Remodeling Day 1

Bathroom Remodeling Day 1

2010-08-10/11 [Day 2 & 3]
They placed bench and soffit ceiling so that they can tile the ceiling.

Bathroom Remodel Day 3

Bathroom Remodel Day 3

2010-08-10/11 [Day 4~]
I got too excited about design and forgot to check some critical thing.. Permit. My husband thought that they are professional so he assumed that they already got permit for this construction. But I wasn’t sure so I checked with them… they told me it is up to home owner… We requested Permit from city. Phew… good thing we checked with them. If you google “bathroom remodiling permit”, you will see all kinds of helpful information. We decided that getting Permit is right thing to do especially installation of Steam Sauna involves running electric cable etc…

2010-12-11

So they are still in process of getting permit right… Though good thing we asked them to go ahead get a permit. Because the way they were going to install steam unit actually violates the city code. So now that they have to re-draw some of the plan and re-submitting to the city… Though there are slight improvement since the last time.

1. they installed floor (it was carpet before)

2. they installed steam generator (which needs to be moved to outside in order to meet city code)

3. We did some upgrade by ourselves. Mirror, faucet and light. We really like how it turned out.

Bath Room Pic 2010-12-11

Some floor and New Mirror & Faucet

2010-12-28
Project Manager came back with City Permit and change of plan. According to the City of Goodyear, steam generator needs to have at least one foot of space from each wall and installing underneath of the bench didn’t meet the criteria.

Again, I am so glad to take time to get City Permit before we move forward with this project. It cost us $350 and very long period of time but I think it is worth it. So now the steam generator is installed above the steam shower room.

2010-12-30
Yey! FINALLY, we have wall in the bathroom. We haven’t had wall for let’s see… about 4 months. Now our power bill should be better…

2011-01-05
Installing KERDI paper to cover the space between DensShield (Tile Backer that has built in moisture barrier)

2011-01-11

Some more tiles has been installed.

Space between tile and wall...

Space between tile and wall...

woo, more tiles!

2011-01-11
Oh wow, this is pretty! Now the grout between tiles and steam controller is installed. Pebble floor and wall. I can’t wait to get in this beautiful steam shower!

steam controller and grout are installed

steam controller and grout are installed

2011-01-19

2010-01-19 pic

corner pieces are installed. now it is rounded!

2011-01-24

Glass are installed and looks nice. Quality of the glass are better than expected. Very contemporarily looking. They put extra whole on side-wall. I wonder how they are going to fix…?

Glass Door is in

Glass Door is in

2010-01-24 b

Glass Door Installed.

2010-01-24-c

oops... they placed hole on wrong place... how they are going to fix?

2011-08-24

So we have been enjoying this Steam Room for a while. I just love it. I don’t miss taking bath at all. I use Steam every single day even when outside is 112 degree.
Everything turns out to be okay. I have no complain about final product. The whole on the side-wall was filled and I can’t tell where was the whole :) So it is all good.

Shower Heads

Rain Shower head for my husband and Handheld is for me and kids!

Steam is on!

Steam is on!

Floor

Floor

Full Shower Image

Full Shower Image

Comment » | Home Improvement

Home Solar – Lease or Purchase

August 6th, 2010 — 5:19am

I have read great things about SolarCity and Sungevity – I believe they are the two leading solar leasing company at the time of writing.

I got estimate form both companies. They do lease very differently…

SloarCity ask you for some down payment then low monthly fee for 20 years regardless of power company you are with.

Sungevity on the other hand requires No down payment. This is probably the only way to go if you have no cache. No down payment and you can go green. Sugevity also wants “inflation rate” to be added every year. So my monthly payment will increase 2.5% every year. This isn’t too bad considering recent energy rate increase.

Both SolarCity and Sungevity will give you “Performance Guarantee” with lease option – meaning that if Solar System generates less than what they projected, they will pay you. Also maintenance is included. So going Solar not only help our planet but also save us money guaranteed!!!! At this point I just could not find any reason why we don’t want to go Solar.

Before we decide to go with this lease option I wanted to see what purchase option would be… Lease make Solar system VERY affordable but after 10/20 years, it is not yours… You end up paying about $20K and you will be given option to purchase at the end of the lease.

So I wanted to see what Purchase option might look like. I reviewed some discussion about Solar lease vs purchase. Both side has valid point. Some says lease is scam… but I don’t think so… Yes you end up paying more money than purchase option but there are many people out there who just cannot come up with cache. Though in my research, if you have cache, purchase is the definitely the way to go.

Purchase option with Sangevity and SolarCity weren’t so fascinating. I think this is because they are trying to promote lease option more than purchase option.

I got quote from American Solar Electric. After the Rebate and Federal Credit, I will end up paying something like $8500. I am going to get quote from other company and see but my mind has almost decided. I will go with purchasing option.

I was listening to NPR this morning saying that my power company lost millions of revenue due to energy efficient rebate etc… so they will increase rate again in 2012 or 2014 (can’t remember) so again, it just no brainer… having own power generator is the way to go.

Comment » | Green Living

Back to top