Showing posts with label gearman. Show all posts
Showing posts with label gearman. Show all posts

Monday, June 14, 2010

Symfony config server

The other day at work, we implemented gearman for doing certain expensive data updates in the background. The workers are written in PHP and run as daemons, and our app calls them with PHP gearman client's function doBackground()

We also use Symfony, which in turn uses YAML for its configuration information. Our gearman workers need access to a bunch of config variables from symfony app.yml files (we have multiple applications/environments for our app).

It could be done by either using Syck within the PHP workers, or by sourcing Symfony framework and using sfConfig::get() calls. However, this doesn't work very well if gearman servers and workers live on a separate set of machines, and so in my continuing effort to learn more Python I wrote a simple config server.
Its very simple - all it does is daemonize itself and listen on a socket you specify on command line reading app.yml parameters from a symfony path you specify, so in our case each web server runs the config server, and PHP workers connect via TCP workers to read the configs needed.

To get a config value, all you have to do is open a TCP socket to the config server and issue something like:
app_mystuff_somevariable
All data is returned in JSON format.

You can also switch environments and apps by issuing commands like:
app:my_other_app
env:prod
By default, the app is frontend and env is prod.

Example:
To start:
python symfony_config_server.py -l 127.0.0.1 -p 1025 -s my_symfony _app

[root@hydra ~] # telnet localhost 1025
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
app_gearman_servers
"10.10.10.166"
env:dev
app_facebook
{"secret": "xxx", "apikey": "yyy", "type": "web", "canvasurl": "apps.facebook.com\/myapp\/", "appid": 1234}

Another way of doing this is of course to pass config parameters to the workers themselves when calling them from clients, but that would be too boring : )

I tested it with Symfony 1.0 and I'm pretty sure it would work with any 1.x release.
If you actually use something so obscure and find bugs in it, let me know or just fork and fix!


Tuesday, May 11, 2010

Gearman PECL package on Ubuntu Lucid Lynx oops

So I tried installing gearman today on Lucid.

To install it, i used this command:
aptitude install gearman libgearman-dev libgearman-client-async-perl libgearman-client-perl gearman-tools uuid-dev

Afterwards, tried to install the gearman PECL package:
pecl install channel://pecl.php.net/gearman-0.7.0

And got this error:
/bin/sed: can't read /usr/lib/libuuid.la: No such file or directory
libtool: link: `/usr/lib/libuuid.la' is not a valid libtool archive

After some googling, it turns out that .la files are now gone from many packages, as per this post on debian-devel list. Couldn't really find a good solution for this, so here it is:

Open /usr/lib/libgearman.la as root, and find a line that says:
dependency_libs=' -L/usr/lib /usr/lib/libuuid.la'

Replace it with:
dependency_libs=' -L/usr/lib -luuid'

Now run the pecl install again and you should be good.

Some more background info:

Hope this saves someone an hour..