Bash expansion

I had posted about the weird behavior of bash expansion, but I think I just ran across a wiki page that discusses this issue.

The key issue seems to be the order of expansions:

1. Syntax analysis (Parsing)
2. Brace expansion
3. Tilde expansion
4. Parameter and variable expansion
5. Command substitution
6. Arithmetic expansion
7. Word splitting
8. Filename expansion
9. Quote removal

Command substitution happens after brace expansion, but before filename expansion (which * expansion is). Similarly, if you attempt to do tilde expansion, it also “won’t work”:

maciek@anemone:~$ echo $(echo \~)
~

Mystery solved. I ran across this when trying to figure out what was it that eval did that’s more interesting than spawning a new subshell. Clearly, the answer is that it lets you get a second pass through these steps, which can definitely be handy.

Strokes and Fills in Flex CSS through ActionScript injection

Flex CSS, while a fine idea, is riddled with limitations. Most notable is probably its lack of proper cascading, but equally unfortunate is that it’s a system defined by programmatic ActionScript APIs. There is a textual way of defining styles, but the Flex compiler, mxmlc, is used to turn this style sheet language into ActionScript, which is in turn compiled into a style module you can load at runtime.

The textual stylesheet language is the black sheep of the family, supporting only a limited set of the features available with runtime Flex styles. For example, take the LineSeries from the data visualization package that comes with FlexBuilder Pro. It defines a number of styles in terms of IFill or IStroke instances. How do you specify IFill or IStroker instances in Flex CSS, so that you can configure look and feel separately from application logic, or alter styles without redeploying the entire application? You don’t. You curse the mxmlc designers for not allowing richer CSS compilation. You can only specify numbers; Strings; embedded images, sounds, and fonts; and references to classes through the ClassReference directive. No instances of any other objects.

Well, what if you’re mad as hell and you won’t take it anymore?

I’m working on a visualization framework with some dynamic styling, and being able to specify “default” styles externally through standard Flex CSS is really, really important. It makes the model much simpler. I was banging my head against the Flex CSS wall and then the scales fell from my eyes:

1. Mxmlc compiles Flex CSS files to ActionScript.
2. That ActionScript must simply be building standard mx.styles.CSSStyleDeclaration classes.
3. Mxmlc has a keep-generated-actionscript attribute.

We take global.css:

AxisRenderer {
    labelAlign: "center";
}

compile with keep-generated-actionscript=true, and we get:

core_properties.as
global.as
global_mx_core_FlexModuleFactory
styles_properties.as

Looking at these, only global.as is interesting. It has a lot of boilerplate code, but then something familiar:

        ...
        var style:CSSStyleDeclaration;
        var keys:Array;
        var selector:String;
        var effects:Array;
        var addedEffects:Array;

        selector = "AxisRenderer";

        style = StyleManager.getStyleDeclaration(selector);

        if (!style)
        {
            style = new CSSStyleDeclaration();
            StyleManager.setStyleDeclaration(selector, style, false);
            selectors.push(selector);
        }

        keys = overrideMap[selector];

        if (keys == null)
        {
            keys = [];
            overrideMap[selector] = keys;
        }

        style.mx_internal::setStyle('labelAlign', "center");
        keys.push("labelAlign");
        ...

This seems fairly routine. It builds a CSSStyleDeclaration programmatically, and then adds values according to our source CSS file. And the style values seem to be just passed along. If only we could somehow “unquote” that style value, we could inject arbitrary ActionScript expressions that would be evaluated before being assigned as style values.

“Unquoting” has a venerable history in programming, so it’s not too crazy an idea. Bash does it (echo "hello $user"), PHP does it (though it’s been a while since my PHP days and I don’t remember the syntax–something along the lines of bash if memory serves), and of course Lisp does it ((let ((user 'maciek)) `(hello ,user)). The tricky part is the implementation.

Fortunately, it’s not really that tricky. With the keep-generated-actionscript flag, mxmlc does most of the work for us. All we need to do is a little post-processing, and re-build the ActionScript into a swf, ignoring the intermediate output swf. Now, everybody stand back. I know regular expressions (not well, mind you, but I get by).

Since Flex has ant task wrappers for its compilers, the easiest way to approach this was to actually build a custom ant filter. Filters can be applied when moving or copying files and they can perform line-by-line transformations on a file. Looking at the generated ActionScript above, we try to find lines matching the calls to setStyle, look for an arbitary special escape sequence that we introduce in String style values (I chose a leading comma, in homage to Lisp), and strip quotes from values matching that escape sequence, to let them execute as arbitrary ActionScript code.

import org.apache.tools.ant.filters.TokenFilter;

public class StyleFilter implements TokenFilter.Filter {
    public String filter(String token) {
        return token.
            replaceAll("(?<=style\\.mx_internal::setStyle\\('[^']{1,100}', )\",([^\"]+)\"(?=\\);)",
                "$1").replaceAll("^import mx\\.core\\.mx_internal;$",
                "import mx.core.mx_internal;\nimport mx.graphics.*;");
    }
}

The regular expression is a little daunting. It can probably be optimized for both performance and correctness (I know I miss edge cases), but it works for now. Briefly,

1. "(?<=style\\.mx_internal::setStyle\\('[^']{1,100}', )\",([^\"]+)\"(?=\\);)": positive lookbehind for the setStyle() call we see above–everything up to and including the comma after the style name. Note that lookbehind won’t let us use an unbounded number of characters due to regex mechanics, so I arbitrarily cap it at 100. I’m sure there are better ways around this, but you’ll seldom see 100-character style names in the wild. Note also that we need to escape the period and parentheses, and escape the backslashes escaping them, since these are Java Strings.
2. "(?<=style\\.mx_internal::setStyle\\('[^']{1,100}', )\",([^\"]+)\"(?=\\);)": the actual quoted value. This expression won’t work on escaped embedded quotes, but since ActionScript supports both single and double quotes, you can typically use single quotes instead. Note that we match the whole quoted value, with the leading comma that signifies the value is to be “de-quoted”, but we only capture the unquoted value, sans leading comma.
3. "(?<=style\\.mx_internal::setStyle\\('[^']{1,100}', )\",([^\"]+)\"(?=\\);)": We make sure the line ends as we expect with positive lookahead. Not really necessary, but a nice sanity check

We replace this whole thing with the first captured group.

Note also the second replace. That’s there to add an import statement for the mx.graphics package, where Stroke and SolidFill, the classes I am most interested in live. Unfortunately, Flex requires import statements even for fully qualified class names (whereas in, e.g., Java, you can say something like System.out.println instead of having to import it explicitly). With this, a style like:

AxisRenderer {
    labelAlign: center;
    axisStroke: ',new Stroke(0x0000ff, 2)';
}

will look like this before the processing step:

        style.mx_internal::setStyle('labelAlign', "center");
        keys.push("labelAlign");
        style.mx_internal::setStyle('axisStroke', ",new Stroke(0x0000ff, 2)");
        keys.push("axisStroke");

and this after processing:

        style.mx_internal::setStyle('labelAlign', "center");
        keys.push("labelAlign");
        style.mx_internal::setStyle('axisStroke', new Stroke(0x0000ff, 2));
        keys.push("axisStroke");

This means you can put arbitrary ActionScript expressions in your style values in CSS. Compiling this yields a standard Flex style module that can be loaded like any other.

Does the special treatment for the mx.graphics package mean that you need to extend the filter with more imports if you need another package? Not necessarily. ActionScript has anonymous functions and imports can be done locally, so you can have CSS values that would give any self-respecting designer a heart attack:

AxisRenderer {
    axisStroke: ',function():Object { import com.example.stroke.CustomStroke; return new CustomStroke(0x0000ff, 2); }()';
}

Fun with xmlns in Flex

So, one of the more interesting things about Flex is that, in addition to Perl-like regular expression literals, it also gives us full-on XML literals. Maybe ‘perverse’ is more accurate than ‘interesting’–the things this must do to the grammar (and the number of issues in Adobe JIRA seems to confirm this). Anyway, it looks like this:

var myXML:XML = <foo><bar baz="quux"><xyzzy/></foo>;

This can be pretty handy to test things when developing something that will eventually consume XML produced by the server. Also, since a number of Flex components (e.g., Menu) consume XML, it’s easy to define it inline.

I am currently experimenting with Degrafa, and using some SVG files as input. Given Flex’s fancy XML support, I decided to just copy and paste the entire text of the SVG file. Everything seems to build fine, but when I launch—no dice. Nothing shows up.

It took me a couple of hours of debugging to track down the problem to this:

var myXML:XML = <foo xmlns="http://www.example.com/foo"><bar baz="quux"><xyzzy/></foo>;

The XML namespace declaration for the XML literal seems to confuse either Flex or Degrafa (I’m not quite sure which), and my Geometry was not being updated. As soon as I took out the xmlns declaration, things are hunky dory.

Update: It seems the problem is in Flex itself. With the following code:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical"
	horizontalAlign="center" verticalAlign="middle">
	<mx:Script>
		<![CDATA[
            private const withNamespace:XML =
                <foo xmlns="http://example.com"><bar>namespaced</bar><baz/></foo>;
            private const withoutNamespace:XML =
                <foo><bar>not namespaced</bar><baz/></foo>;
		]]>
	</mx:Script>
    <mx:Label text="{withNamespace.bar}"/>
    <mx:Label text="{withoutNamespace.bar}"/>
</mx:Application>

only ‘not namespaced’ shows.

Fun with IOWait

I’ve been having intermittent problems with IOWait rendering my laptop completely unusable for minutes at a time after a resume from suspend. I’m not a stickler for performance–I can forgive a little slugishness (in fact, back in the day, I played through GTA III with my brother on a system that was absurdly underpowered for the task, and we’d be lucky to get frame rates above the single digits–but I digress), but it’s just not possible to do anything when the system is like that.

I’ve added the system monitor applet to my taskbar and set the IOWait to show up red (it’s a dark blue that’s almost invisible against the black idle by default). Here is one particularly egregious session:

That’s right: it’s still pegged twenty minutes later. Almost completely unusable the entire time. I haven’t found what could be causing this. There’s an old Ubuntu bug report, but it doesn’t really shed much light on the situation. It seems to blame fglrx, but it links to an older report that doesn’t even mention fglrx.

Bleah. This makes Firefox’s fsync problems look like a Duff’s-device-grade performance optimization.

The voodoo of bash token expansion

Unlike the stunted Windows command shell (I admit I haven’t tried PowerShell), the bash shell combines the ability to put together many different small utilities with some built-in smarts to offer an incredibly powerful interactive experience.

One of the more handy built-in smarts is bash’s “magic” token expansion. If bash sees one of these while evaluating a statement, it expands it before doing anything else. The most common of these is the asterisk, which expands to everything in the current directory:

maciek@potato:~/bash$ ls
maciek@potato:~/bash$ touch foo bar # touch creates empty files named like the arguments
maciek@potato:~/bash$ echo *
bar foo

One interesting aspect is that if there is nothing in the current directory, the asterisk does not expand:

maciek@potato:~/bash$ rm foo bar
maciek@potato:~/bash$ echo *
*

You can also tell bash that you want a literal asterisk by escaping it with a backslash:

maciek@potato:~/bash$ touch foo bar
maciek@potato:~/bash$ echo *
bar foo
maciek@potato:~/bash$ echo \*
*

One of the more powerful aspects of bash is being able to spawn a subshell with $(...) (the output of the subshell is “injected” wherever the subshell block occurs). The interesting thing in terms of token expansion is that the result of the subshell execution seems to be a candidate for token expansion as well:

maciek@potato:~/bash$ echo $(echo \*)  
bar foo  

However, there are other types of expansions that bash does for you. One of my favorite is brace expansion:

maciek@potato:~/bash$ echo foo{1,2,3}
foo1 foo2 foo3

This is extremely handy for, e.g., moving files nested deep in some directory hierarchy. However, a weird thing seems to happen with brace expansion and the results of subshells:

maciek@potato:~/bash$ echo $(echo foo\{1,2,3\})
foo{1,2,3}

The expansion is ignored. I’m far from a shell scripting expert, but I can’t figure this out. It seems that it should be analogous to the ‘*’ expansion, but it behaves differently. Any ideas?

Row numbering hacks in postgres

I recently ran across this post on simulating row numbering in postgres (and announcing that 8.4 will finally have a function for this). I was aware of the temporary sequence solution (I believe that’s the stock answer to that question in postgres) , but the two other approaches are new to me (and rather clever). The last especially is… majestic:

SELECT (SELECT COUNT(*) FROM people 
    WHERE 
    (COALESCE(people.last_name,'') || COALESCE(people.first_name,'')) <= 
    (COALESCE(oldtable.last_name,'') 
    || COALESCE(oldtable.first_name,''))) As row_number, 
    oldtable.*
FROM (SELECT * 
    FROM people 
    ORDER BY 
    last_name, first_name) As oldtable;
&#91;/sourcecode&#93;

Conceptually, for each row (in the order we care about), this constructs a key with a well-defined ordering to it, and counts the number of rows that have keys "smaller" than that key. Slick.

I independently came up with another hacktastic approach to this problem last summer, and I'm rather fond of it:
&#91;sourcecode language='sql'&#93;
SELECT
    row_num,
    (SELECT first_name FROM people
    ORDER BY last_name, first_name
    LIMIT 1 OFFSET row_num - 1) as first_name,
    (SELECT last_name FROM people
    ORDER BY last_name, first_name
    LIMIT 1 OFFSET row_num - 1) as last_name,
FROM
    generate_series(1, (SELECT COUNT(*) FROM people)) as row_numbers(row_num)
&#91;/sourcecode&#93;

This approach is (almost certainly) much less efficient than the first, but I think I may win some style points. From Lucifer.

Essentially, this turns the query on its head: instead of getting a bunch of ordered results and trying to number them, we take a bunch of numbers, and for each one, take the nth row from the (unnumbered) result set with the help of handy LIMIT and OFFSET. Postgres allows queries in the target list (as long as they return a scalar), and since we only care about a single column and LIMIT to a single result, this is perfectly legit.

<hr>

As an aside, I lied above: the result of a subquery in the target list does not have to be scalar: it must have a single column, but it can return more than one row. Things get weird when this happens, though--see, e.g., the following:


SELECT generate_series(1,2) as series1, generate_series(1,3) as series2;

How many rows does this produce? Six. Yes, six.

hello world

Maybe I’m just dim, but it took me ages to realize that

1. WordPress.com completely blocks all JavaScript on user pages
2. syntax highlighting is available through syntaxhighlighter

I only needed (1) to accomplish (2), but both of these data seemed to be rather hard to find (through search–I suppose I should just have hit WordPress support right away).

The availability of syntax highlighting is crucial for talking about code. Surprisingly, some people do rail against it, but usually they’re probably too busy walking through the snow uphill both ways and telling the kids to get off their lawn to tell us how back in their day, they didn’t have syntax highlighting, hell, they didn’t even have any syntax, and monitors only displayed one color: black. Now that’s monochromatic for you.

Anyway, as this is my first post, here’s a hello world in one of my day-to-day languages, ActionScript (not directly supported by syntaxhighlighter, but fortunately, JavaScript is close enough), in all its highlighted glory.

            import mx.controls.Alert;
            Alert.show('Hello world!');