Alexandre Beelen @ IAS - Posts

STILTS and Python

2017-06-02T00:00:00+02:00

STILTS is a very useful program, in particular to convert VOTABLE between several serializations. But can be difficult to use outside of TOPCAT. It is as simple as

stilts tcopy ifmt=votable in=table.xml ofmt=fits out=table.fits

to convert from an VOTABLE in xml format to its fits serialization. However, STILTS store most of the metadata of the VOTABLE as bytes in the first extension as BINTABLE, with these comments :

COMMENT The data in this primary HDU consists of bytes which                    
COMMENT comprise a VOTABLE document.                                            
COMMENT The VOTable describes the metadata of the table contained               
COMMENT in the following BINTABLE extension.                                    
COMMENT Such a BINTABLE extension can be used on its own as a perfectly         
COMMENT good table, but the information from this HDU may provide some          
COMMENT useful additional metadata.

Reading the binary table returns a lengthy array of indecipherable bytes. Hopefully Python can be of some helps, by transforming it into string representation of a bytearray.

from astropy.io import fits
data = str(bytearray(fits.getdata('table.fits', 0)))

This string is human readable but still not very useful in its form. But it can easily be parsed by ElementTree

from lxml import etree
root = etree.fromstring(data)

Now we can easily access all the keys from the VOTABLE, for, e.g. looking for the value of the CreatorDate PARAM

root.find(".//*[@name='CreatorDate']").attrib['value']

IDL & MPI

2016-06-07T00:00:00+02:00

They are several ways of efficient programming with IDL. The most common way is to use the IDL Thread Pool, which is implemented in several routines and operation on large arrays. But sometimes, matrix or large vector operations are not possible, and/or not sufficient. Sometimes, we would like to break a simple FOR loop into several parallel IDL sessions.

IDL_IDLBridge

This is the purpose of IDL_IDLBridge which allow to create sub instances of IDL and control them within the original IDL instance. A nice tutorial can be find here, and Robert da Silva wrote the routine split_for which allow to split simple loops without entering into the specific syntax of IDL_IDLBridge. Unfortunately, IDL_IDLBridge only creates new processes of IDL on the same computer, as child processes, and sometimes, this is not enough.

When dealing with a cluster of computer, the most convenient way to use several nodes is through MPI. At the expense of loosing the real purpose of MPI, i.e. passing messages, it is possible to launch several instances of IDL in parallel, on one or several nodes in a cluster. It is thus possible to split a long and complex FOR loop into several processes on potentially several nodes.

IDL & MPI

It is possible to launch several instance of IDL using MPI :

nMPI=#ntasks
mpirun -n $nMPI idl mpi_poc.pro

will launch several independent IDL processes, which will all run the same program mpi_poc.pro. This would produce several times the exact same output. However, at least using the OpenMPI implementation, it is possible to retrieve MPI_COMM_WORLD_RANK and MPI_COMM_WORLD_SIZE as system variables and thus split any big FOR loop. The results will then depends on the MPI_COMM_WORLD_RANK value and could be saved using the rank as an unique identifier.

I wrote a few simple scripts to help parallelize a long FOR loop. Note however that combining the results of the FOR loop will require disk IO and thus, the iterations of the FOR loop should be rather CPU intensive in order to be efficient. The following example compute the sum of all indexes between two values.

;; mpi_poc.pro
;; Let's compute the sum of all indexes between start and stop

;; Retrieve the MPI RANK from system variables
mpi_rank, rank

start = 0
stop = 11
sum = 0

;; Divide the task among the different MPI ranks
mpi_helper, start, stop, i_start, i_stop

;; This is the main loop, should be an intensive task....
FOR index=i_start, i_stop-1 DO BEGIN $
   PRINT, rank, index, FORMAT='("rank: ", I2, " doing ", I2)' &$
   WAIT, start &$
   sum += index

;; save the result in an unique output file
SAVE, FILENAME=STRING(rank, FORMAT='("poc_",I2.2,".sav")'), sum

;; Do not forget to exit at the end
exit

Launching 5 instances of this program with

mpirun -n 5 idl mpi_poc.pro

will result in 5 files

poc_00.sav  poc_01.sav  poc_02.sav  poc_03.sav  poc_04.sav

which can be combined using

sum_combined = 0
files = FILE_SEARCH('poc_*.sav')
FOR iFile=0, N_ELEMENTS(files)-1 DO BEGIN $
   RESTORE, files[iFile] &$
   sum_combined += sum

PRINT, sum_combined
PRINT, TOTAL(INDGEN(11))

IDL & MPI & IDL_IDLBridge

When using a cluster of computer, it is thus possible to allocate several nodes and launch one instance of IDL per node which will use efficiently all the core of the node using IDL_IDLBridge. For example, using the slurm scheduler, one could setup a batch script as

#!/bin/bash

#SBATCH -N 2
#SBATCH -n 64
#SBATCH --exclusive
#SBATCH --job-name "MPI IDL"

mpirun -pernode idl mpi_poc.pro

which will launch 2 IDL processes on two different nodes with a total of 64 cores.

accept that the IDL_IDLBridge documentation state that :

Note: On UNIX systems, the IDL_IDLBridge requires that the DISPLAY environment variable be set to a valid X Windows display. If no display is available, execution of the IDL_IDLBridge will halt, with error messages that include:

'IDL_IDLBridge: Unable to establish X Connection.'

which render the sbatch very impracticable, as one need to be sure that X11 is forwarded on all the allocated node of the cluster which are not known in advance. Thanks again IDL... Hopefully Xvfb allows to run virtual x11 environment and thus

srun xvfb-run idl mpi_poc.pro

will be able to run.

D3js test using mpld3

2015-11-18T00:00:00+01:00

After my initial testing of D3js, I came accross mpld3 and I wanted to quickly know if it was possible to integrate those nice plot with pelican_dynamic.

I took the code from the Scatter Plot With Tooltips example, and instead of showing the result with mpld3.show(), I saved it using

mpld3.save_json(fig, 'ScatterPlotWithTooltips.json')

Thus I just need to create a simple js file with

$.getJSON("ScatterPlotWithTooltips.json", function(json){
mpld3.draw_figure("fig_scatter", json);
});

And by defining the proper

in the mardown document-relative

<div class="d3-chart" id="fig_scatter"><div>

Here is the result :

Isn't that a nice looking chart up there?

D3js Test

2015-11-16T00:00:00+01:00

This is a quick migration of old tests using D3js.

Using the same json file

{
    "name": "Carte des services informatiques",
    "children": [
    {
        "name": "Services utilisateurs",
        "children" : [
        {
            "name": "Comptes et accès",
            "children" : [
            { "name": "Régles générales", "size": 500},
            { "name": "Création compte", "size": 500},
            { "name": "Supression compte", "size": 500},
            { "name": "Modification du mot de passe", "size": 500},
            { "name": "Modification des informations personnelles", "size": 500}
                ]
        },
[...]

I wrote some simple D3js javascript code following a tutorial

var diameter = 960;

var tree_r = d3.layout.tree()
    .size([360, diameter / 2 - 120])
    .separation(function(a, b) { return (a.parent == b.parent ? 1 : 2) / a.depth; });

var diagonal_r = d3.svg.diagonal.radial()
    .projection(function(d) { return [d.y, d.x / 180 * Math.PI]; });

var svg_r = d3.select(".d3-1").append("svg")
    .attr("width", diameter)
    .attr("height", diameter - 150)
  .append("g")
    .attr("transform", "translate(" + diameter / 2 + "," + diameter / 2 + ")");

d3.json("ias.json", function(error, root) {
  var nodes = tree_r.nodes(root),
      links = tree_r.links(nodes);

  var link_r = svg_r.selectAll(".link")
      .data(links)
    .enter().append("path")
      .attr("class", "link")
      .attr("d", diagonal_r);

  var node_r = svg_r.selectAll(".node")
      .data(nodes)
    .enter().append("g")
      .attr("class", "node")
      .attr("transform", function(d) { return "rotate(" + (d.x - 90) + ")translate(" + d.y + ")"; })

  node_r.append("circle")
      .attr("r", 4.5);

  node_r.append("text")
      .attr("dy", ".31em")
      .attr("text-anchor", function(d) { return d.x < 180 ? "start" : "end"; })
      .attr("transform", function(d) { return d.x < 180 ? "translate(8)" : "rotate(180)translate(-8)"; })
      .text(function(d) { return d.name; });
});

d3.select(".d3-1").style("height", diameter - 150 + "px");

And putting everything back to pelican_dynamic div, magic happens :

and following the D3js tutorials :

or even with a bit of dynamics :

From XSLT to Pelican

2015-11-03T00:00:00+01:00

Writing and publishing a website used to be a painful process. Some years ago, I developped a simple xslt framework to produce my personnal web pages from simple xml files. This was in part because I did not liked any of the big CMS out at that time, requiring database backend, and also that I wanted to learn XSLT processing. CSS was a must and compliance with W3C standards whitout saying. As long as I did not wanted to change much on the site, this was ok, and the site was pretty robust since 2005.

Before

So I needed a simple framework, what is better than homemade ? So lets remove the style, and start writing content, and only (mostly) content and metadata.

<!-- index.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?DOCTYPE page SYSTEM "my_pages.dtd"?>
<?xml-stylesheet type="text/xsl" href="my_pages.xsl.xml"?>

<page icon="stock_home">
<title>Home</title>
<text>
<section id="section-id" name="Global Section Name">
<p>Blah Blah</p>
<!-- and any basic html tags -->
</section>
</text>
</page>

Simple enough, emacs made, now we need to transform that into a proper html static page, using xsl transformation, this was supposedly the future of xhtml

<!-- my_pages.xslt -->
<?xml version="1.0"  encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns="http://www.w3.org/1999/xhtml"
  >

<xsl:output method="xml" indent="yes" encoding="UTF-8"
  doctype-system="http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"
  doctype-public="-//W3C//DTD XHTML 1.1//EN" />

<xsl:include href="header.xsl"/>
<xsl:include href="footer.xsl"/>

<xsl:template match="page">
<xsl:element name="body">

<xsl:call-template name="make-header"/>

<div id="pageCell">
<!--pageCell-->
<div id="pageName">
  <h2><xsl:value-of select="title"/></h2>
  <xsl:element name="img">
    <xsl:attribute name="src">
      icons/<xsl:value-of select="@icon"/>.png
    </xsl:attribute>
    <xsl:attribute name="alt">page icon</xsl:attribute>
  </xsl:element>
</div>

[....]

This go on and on to construct the web page, with some common header/menu

<!-- header.xsl -->
<xsl:template name="html-header">

<title>
Alexandre Beelen @ IAS - <xsl:value-of select="title"/>
</title>
<meta name="author"      content="Alexandre Beelen" />
<meta name="keywords"    content="astrophysics, astronomy, quasars, black hole, cosmology, IAS, Orsay, MIC, Insitut   d'Astrophysique Spatiale" />
<meta name="description" content="Homepage of Alexandre Beelen" />
<meta name="page-topic"  content="science" />
<meta name="expires"     content="0" />
<meta name="robots"      content="INDEX,FOLLOW" />

<link rel="stylesheet" href="page.css" type="text/css" />

</xsl:template>

<xsl:template name="make-header">
<div id="Header">
  <div id="HeaderTitle">
    <h1><a href="http://www.ias.u-psud.fr/pperso/abeelen">Alexandre Beelen @ IAS</a></h1>
    <div id="HeaderTitleLinks">
    <a href="http://www.ias.fr"                    class="hlink">IAS</a> |
    <a href="http://www.mpifr-bonn.mpg.de"         class="hlink">MPIfR</a> |
    <a href="http://www.astro.uni-bonn.de/~webrai" class="hlink">AIfA</a> |
    <a href="http://www.obspm.fr"                  class="hlink">Obspm</a>
    </div>
  </div>
  <div id="HeaderNav">
    <a href="index.html"                            class="glink">Home</a> :
    <a href="cv.html"                               class="glink">C.V.</a> :
    <a href="talk.html"                             class="glink">Talk &amp; Posters</a> :
    <a href="publication.html"                      class="glink">Publications</a> :
    <a href="database.xml"                          class="glink">High-z CO database</a> :
    <a href="software.html"                         class="glink">Tools</a> :
    <a href="http://www.astro.uni-bonn.de/boawiki/" class="glink">Open Boa</a> :
    <a href="http://www.ias.u-psud.fr/sanepic/"     class="glink">SANEPIC</a> :
    <a href="schedule.html"                         class="glink">Schedule</a>
  </div>
</div>
</xsl:template>

where all the links are hardcoded in the file directly. Same goes for the footer. Of course you need a Makefile produce all the pages, and even publish them through a rsync and ssh commands

SHELL= /bin/sh
srcdir    = .
outdir    = $(srcdir)/html
addstuff  = $(srcdir)/download $(srcdir)/icons $(srcdir)/pictures page.css sitemap.xml

all: update
web: index.html software.html talk.html cv.html publication.html schedule.html gildas.html

update: web
  cp -fr $(addstuff) $(outdir)
  rsync -qavz -e ssh $(outdir)/* abeelen@www:web_perso
  ssh abeelen@www.ias.u-psud.fr chgrp -R www web_perso
  ssh abeelen@www.ias.u-psud.fr chmod -R a+r web_perso

.SUFFIXES:
.SUFFIXES: .html .xml

%.html: %.xml
  xsltproc -o $(outdir)/$@ my_pages.xsl $<

Everything was working fined and was fully W3C HTML 1.1 compliant. And you can still see it at work there. But it was rigid, very rigid... And crazy to maintain... It's 2015 after all !

After

Fast forward to 2015, during a power outage, and with only access to Internet, I started the project to migrate my website. After a rapid survey of simple static web site generator, I decided to use Pelican mainly for its simplicity and the capability to use either Markdown or reStructuredTex. Simple enough to use, eigher with emacs or atom.

The previous home page could be written as a simple file

Home
====

Global Section Name
-------------------

blah blah

You could choose a theme and voilà ! So how did it go ? I just started a Pelican tutorial and finished with a full site in half a day. You start by creating a skeleton project

> pelican-quickstart

and fill the content/pages directory with cleaned version of my previously generated html pages translated to reStructuredTex with pandoc. The basic pages were there.

I just had to find a nice theme, and I quickly choose the bootstrap framework through the pelican-bootstrap3 theme. And I just had to put everything under revision control using git. And I was done with this website. And well, I could also push it with a Makefile !

Even more

The journey did not end by with only migrating the old pages to a new simpler system, but it is now easy to add new features with pelican-plugins, so here is a list of plugins I used :

better_tables, twitter_bootstrap_rst_directives, bootstrapify
- allow a cleaner presentation with pelican-bootstrap3
sitemap
- for a better indexation in web crawler, not having to produce it by hand is a real plus
tag_cloud
- to produce a tag_cloud on the sidebard, utility is still to be proven...
render_math
- allows a clean integration of MathJax. Adding math is now as simple as writing \(\LaTeX\) both with reStructuredTex and Markdown
tipue_search
- allows for simple search over the static website
pelican_dynamic
- to be able to use D3js easily within Pelican

Configuration

Pelican and pelican-bootstrap3 allow for a very fine tuning of the created website. One can very simply define which pages are displayed in the menu, and in which order using the meta data sortorder for example.

The web site can very easily be tracked with piwik or Google Analytics. One can still transfert standalone files, as I needed for the Molecular Database, or even a copy of my old website

SVN to GIT

2015-11-02T00:00:00+01:00

I wanted to move from an bulky svn repository containing several projects to smaller git repositories to allow for easy distribution of the files and tracking of the code. Of course, while doing that, I wanted to keep the file history... I could not use the classic filter-branch as I wanted to extract only a few files withing a directory.

So... First step is to checkgout the svn repository with git :

git svn clone $SVN_REPO_URL
cd $SVN_REPO_URL

or, if you have already done it, you can update the repository with

cd $SVN_REPO_URL
git svn rebase

then you would extract the history of the desired files into a patch file

git log --pretty=email --patch-with-stat --reverse --full-index --binary path/to/file* > /tmp/patch

This will generate a full patch to the file, from its creation to the last commit. The (old) path in the patch file are relative to the old svn structure. Thus one could change it to reflect the new desired path structure

sed -i -e 's/old\/path\///g' /tmp/patch

for example to remove all the path for the new repository

And after creating a new git repository, and within the repository, you can replay the file creation and modification with

mkdir $GIT_REPO
cd $GIT_REPO
git init .
git am < /tmp/patch

This will recreate the files, with their history, in the newly created git repository.

Gildas Tips & Tricks

2010-01-04T00:00:00+01:00

Warning

Long forgotten GILDAS Tips & Tricks. This should probably be extended. Using more examples, in particular for clic and mapping

Gildas/Clic

Change the bandpass calibrator :

let band_source "sourcename" after clicking on select
180 degree Jumpy Phase :
set phase degree jumpy
PLOT /IDENTIFY col
SOLVE phase /PLOT

Change the number of time interval for spline interpolation :

SET STEP 4 (in hours, default 3)

Make polynomial phase fit instead of spline :

SOLVE phase /PLOT /POLYNOMIAL 6

Remove the last drawn plot layer :

CLEAR seg
Plot the closure relations :
PLOT closure

Change the averaging in time when plotting :

SET AVERAGING none
SET AVERAGING TIME 5 (seconds)

Change the reference antenna for RF and phase calibration :

SET REFERENCE x (logical number)

See the correspondance physical/logical antenna :

SET ANTENNA ALL

Gildas/Mapping

Reproject one image to another :

run reproject