Whalesong: a Racket to JavaScript compiler

Version: 5.2.1

Source code can be found at:https://github.com/dyoo/whalesong. The latest version of this document lives in http://hashcollision.org/whalesong.

Current commit head is 0bcdd61f10af9739e4b757d879b2a1880907ca09.

1 Introduction

Whalesong is a compiler from Racket to JavaScript; it takes Racket programs and translates them so that they can run stand-alone on a user’s web browser. It should allow Racket programs to run with (hopefully!) little modification, and provide access through the foreign-function interface to native JavaScript APIs. The included runtime library supports the numeric tower, an image library, and a framework to program the web in functional event-driven style.

The GitHub source repository to Whalesong can be found athttps://github.com/dyoo/whalesong. If you have any questions or comments, please feel free to use theRacket-users mailing list.

Prerequisites: at least Racket 5.1.1. If you wish to use the JavaScript compression option, you will need Java 1.6 SDK.

1.1 Examples

Here are a collection of programs that use the web-world library described later in this document:

I also gave apresentation of Whalesong at RacketCon 2011, including examples like:

2 Getting started

2.1 Installing Whalesong

Before you begin, if you are using DrRacket,

Please go to the Racket submenu.
Select the Limit Memory item.
Change the setting to Unlimited.

This is to avoid an installation-time issue that prevents Whalesong from fully compiling.

If you want to use Whalesong, run the following to create the "whalesong" launcher:

This may take a few minutes, as Racket is compiling Whalesong, its dependencies, and its documentation. When it finally finishes, you should see a "whalesong" launcher in the current directory.

You should also see a "whalesong-gui" launcher that includes a minimal graphical user interface.

At this point, you should be able to run the "whalesong" executable from the command line.

$ ./whalesong

Usage: whalesong <subcommand> [option ...] <arg ...>

where any unambiguous prefix can be used for a subcommand

The Whalesong command-line tool for compiling Racket to JavaScript

For help on a particular subcommand, use 'whalesong <subcommand> --help'

whalesong build build a standalone html and javascript package

whalesong get-runtime print the runtime library to standard output

whalesong get-javascript Gets just the JavaScript code and prints it to standard output

and if this does appear, then Whalesong should be installed successfully.

To repeat: whenever Whalesong’s source code is updated from Github, please re-run the raco setup step. Otherwise, Racket will try to recompile Whalesong on every single use, which can be very expensive.

2.2 Making .html files with Whalesong

Let’s try making a simple, standalone executable. At the moment, the program must be written in the base language of (planetdyoo/whalesong). This restriction unfortunately prevents arbitraryracket/base programs from compiling at the moment; the developers (namely, dyoo) will be working to remove this restriction as quickly as possible.

Write a "hello.rkt" with the following content

"hello.rkt"
#lang planet dyoo/whalesong
(display "hello world")
(newline)

This program is a regular Racket program, and can be executed normally,

$ racket hello.rkt

hello world

However, it can also be packaged with "whalesong".

$ whalesong build hello.rkt

Writing program #<path:/home/dyoo/work/whalesong/examples/hello.js>

Writing html #<path:/home/dyoo/work/whalesong/examples/hello.html>

$ ls -l hello.html

-rw-r--r-- 1 dyoo dyoo 3817 2011-09-10 15:02 hello.html

$ ls -l hello.js

-rw-r--r-- 1 dyoo dyoo 841948 2011-09-10 15:02 hello.js

Running whalesong build on a Racket program will produce a".html" and ".js" file. If you open the".html" in your favorite web browser, you should see a triumphant message show on screen.

We can do something slightly more interesting. Let’s write a Whalesong program that accesses the JavaScript DOM. Call this file "dom-play.rkt".

"dom-play.rkt"
#lang planet dyoo/whalesong

;; Uses the JavaScript FFI, which provides bindings for:
;; $ and call-method
(require (planet dyoo/whalesong/js))

;; insert-break: -> void
(define (insert-break)
  (call-method ($ "<br/>") "appendTo" body)
  (void))

;; write-message: any -> void
(define (write-message msg)
  (void (call-method (call-method (call-method ($ "<span/>") "text" msg)
                    "css" "white-space" "pre")
              "appendTo"
              body)))

;; Set the background green, and show some content
;; on the browser.
(void (call-method body "css" "background-color" "lightgreen"))
(void (call-method ($ "<h1>Hello World</h1>") "appendTo" body))
(write-message "Hello, this is a test!")
(insert-break)
(let loop ([i 0])
  (cond
   [(= i 10)
    (void)]
   [else
    (write-message "iteration ") (write-message i)
    (insert-break)
    (loop (add1 i))]))

This program uses the JQuery API provided by (planetdyoo/whalesong/js), as well as the native JavaScript FFI to produce output on the browser. If we run Whalesong on this program, and view the resulting "dom-play.html" in our web browser, we should see a pale, green page with some output.

2.3 Using Whalesong functions from JavaScript

Whalesong also allows functions defined from Racket to be used from JavaScript. As an example, we can take the boring factorial function and define it in a module called "fact.rkt":

The files can also be downloaded here:
with generated JavaScript binaries here:

"fact.rkt"

Instead of creating a standalone .html, we can use whalesong to get us the module’s code. From the command-line:

$ whalesong get-javascript fact.rkt > fact.js

$ ls -l fact.js

-rw-r--r-- 1 dyoo dyoo 27421 2011-07-11 22:02 fact.js

This file does require some runtime support not included in"fact.js"; let’s generate the runtime.js and save it as well. At the command-line:

$ whalesong get-runtime > runtime.js

$ ls -l runtime.js

-rw-r--r-- 1 dyoo dyoo 544322 2011-07-11 22:12 runtime.js

Now that we have these, let’s write an "index.html" that uses the fact function that we provideed from"fact.rkt".

"index.html"
<!DOCTYPE html>
<html>
<head>
<script src="runtime.js"></script>
<script src="fact.js"></script>

<script>
// Each module compiled with 'whalesong get-runtime' is treated as a
// main module.  invokeMains() will invoke them.
plt.runtime.invokeMains();

plt.runtime.ready(function() {

   // Grab the definition of 'fact'...
   var myFactClosure = plt.runtime.lookupInMains('fact');

   // Make it available as a JavaScript function...
   var myFact = plt.baselib.functions.asJavaScriptFunction(
        myFactClosure);

   // And call it!
   myFact(function(v) {
              $('#answer').text(v.toString());
          },
          function(err) {
              $('#answer').text(err.message).css("color", "red");
          },
          10000
          // "one-billion-dollars"
          );
});
</script>
</head>

<body>
The factorial of 10000 is <span id="answer">being computed</span>.
</body>
</html>

Replacing the 10000 with "one-billion-dollars" should reliably produce a proper error message.

3 Using whalesong

Whalesong provides a command-line utility called whalesong for translating Racket to JavaScript. It can be run in several modes:

Using whalesong to generate HTML+js documents is relatively straightforward with the build command. To use it, pass the name of the file to it:

$ whalesong build [name-of-racket-file]

A ".html" and ".js" will be written to the current directory, as will any external resources that the program uses.

The whalesong commands support these command line options:

Use Google Closure’s JavaScript compiler to significantly compress the JavaScript. Using this currently requires a Java 1.6 JDK.
Write verbose debugging information to standard error.
Write files to a separate directory, rather than the current directory.
Write each dependent module as a separate file, rather than in one large ".js". This may be necessary if your browser environment prohibits large ".js" files. The files will be numbered starting from 1.

For more advanced users, whalesong can be used to generate JavaScript in non-standalone mode. This gives the web developer more fine-grained control over how to control and deploy the outputted program.

3.1 build

Given the name of a program, this builds".html" and ".js" files into the current working directory.

The ".html" and ".js" should be self-contained, with an exception: if the file uses any external resources by usingdefine-resource, those resources are written into the current working directory, if they do not already exist there.

3.2 get-javascript

Given the name of a program, writes the JavaScript to standard output, as well as its dependent modules. The outputted file is meant to be used as a SCRIPT source.

By default, the given program will be treated as a main module. All main modules will be executed when the JavaScript functionplt.runtime.invokeMains() is called.

3.3 get-runtime

Prints out the core runtime library that the files generated by get-javascript depend on.

4 Including external resources

Programs may need to use external file resources that aren’t themselves Racket programs, but instead some other kind of data. Graphical programs will often use ".png"s, and web-related programs ".html"s, for example. Whalesong provides the(planetdyoo/whalesong:1:=18/resource) library to refer and use these external resources. When Whalesong compiles a program into a package, these resources will be bundled alongside the JavaScript-compiled output.

Defines a resource with the given path name.

For example,

As a convenience, you can also write

which defines a variable named humpback.png whoseresource is "humpback.png".

If the resource given has an extension one of the following:

".png"
".gif"
".jpg"
".jpeg"

then it can be treated as an image for which image? will be true.

If the resource has the extension ".html", then it will be run through an HTML purifying process to make sure the HTML is well-formed.

For example,

5 The web-world API

The web-world library allows you to write functional event-drivenWorld programs for the web; the user defines functional callbacks to handle events, and receive and consume a world argument.

One difference introduced by the web is the web page itself: because the page itself is a source of state, it too will be passed to callbacks. This library presents a functional version of the DOM in the form of a view.

Let’s demonstrate this by creating a basic ticker that counts on the screen every second.

The first thing we can do is mock up a web page with a user interface, like this.

"view.html"
<html>
  <head><title>My simple program</title></head>
  <body>
    <p>The current counter is: <span id="counter">fill-me-in</span></p>
  </body>
</html>

We can even look at this in a standard web browser.

Once we’re happy with the statics of our program, we can inject dynamic behavior. Write a file called "tick-tock.rkt" with the following content.

Several things are happening here.

We require a few libraries to get us some additional behavior; in particular, (planetdyoo/whalesong:1:=18/web-world) to let us write event-driven web-based programs, and (planetdyoo/whalesong:1:=18/resource) to give us access to external resources.
We use define-resource to refer to external files, like "view.html" that we’d like to include in our program.
We use big-bang to start up a computation that responses to events. In this example, that’s clock ticks introduced by on-tick, though because we’re on the web, we can bind to many other kinds of web events (by using view-bind).

5.1 big-bang and its options

(big-bang w h ...) → world
w : world
h : big-bang-handler

Start a big bang computation. The big-bang consumes an initial world, as well as several handlers to configure it, described next:

Provide an initial view for the big-bang. Normally, x will be a resource to a web page.

(stop-when stop?) → big-bang-handler
stop? : ([w world] [dom view] -> boolean)

...
(define-struct world (given expected))
...

;; stop?: world view -> boolean
(define (stop? world dom)
(string=? (world-given world) (world-expected world)))

(big-bang ...
(stop-when stop?))

(on-tick tick-f delay) → big-bang-handler
  tick-f : ([w world] [v view] [e event]? -> world)
  delay : real
(on-tick tick-f) → big-bang-handler
  tick-f : ([w world] [v view] [e event]? -> world)

Tells big-bang to update the world during clock ticks.

By default, this will send a clock tick 28 times a second, but if given delay, it will use that instead.

...
;; tick: world dom -> world
(define (tick world view)
(add1 world))

(big-bang ...
(on-tick tick 5)) ;; tick every five seconds

Tells big-bang to update the world during simulated movement.

During the extent of a big-bang, a form widget will appear in thedocument.body to allow us to manually send location-changing events.

The optional event argument will contain numbers for"latitude" and "longitude".

(on-location-change location-f) → big-bang-handler
location-f : ([w world] [v view] [e event]? -> world)

The optional event argument will contain numbers for"latitude" and "longitude".

(to-draw draw-f) → big-bang-handler
draw-f : ([w world] [v view] -> view)

Tells big-bang how to update the rendering of the world. The draw function will be called every time an event occurs.

5.2 Views

A view is a functional representation of the browser DOM tree. A view is always focused on an element, and the functions in this subsection show how to traverse and manipulate the view.

(->view x) → view
x : any

Coerse a value into a view whose focus is on the topmost element. Common values for x include resources.

(view-focus? v id) → boolean
v : view
id : String

Return true if the view can be focused using the given id.

(view-focus v id) → view
v : view
id : String

Focuses the view on an element, given the id. The view will be searched starting from the toplevelmost node.

See if the view can be moved to the previous sibling.

Move the focus to the previous sibling.

See if the view can be moved to the next sibling.

Move the focus to the next sibling.

(view-up? v) → boolean
v : view

See if the view can be moved to the parent.

(view-up v) → view
v : view

Move the focus to the parent.

See if the view can be moved to the first child.

Move the view to the first child.

See if the view can be moved forward.

Move the view forward, assuming a pre-order traversal.

See if the view can be moved backward.

Move the view backward, assuming a pre-order traversal.

Get the textual content at the focus.

Update the textual content at the focus.

(view-bind v type world-updater) → view
  v : view
  type : string
  world-updater : ([w world] [v view]  [e event]? -> world)

Attach a world-updating event to the focus.

Attach a world-updating event to the focus. When the world-updater is called, the view will be focused on the element that triggered the event.

Common event types include "click", "mouseenter","change". Note that the name of the event should not include an "on" prefix.

A view may have many elements to bind, and it’s a common pattern to focus and view. As a convenience the API provides some syntactic support to bind multiple handlers at once:

Common event types include "click", "mouseenter", or"change". Note that the name of each event should not include an "on" prefix.

As an example:

(define (click-handler w v) ...)

(define (change-handler w v) ...)

(define-resource view.html)

(define my-static-view (->view view.html))

(define connected-view
  (view-bind-many my-static-view
                  ["id1" "click" click-handler]
                  ["id2" "click" click-handler]
                  ["id3" "change" change-handler]))
...

If the collection of ids, types, and handlers can’t be represented as a static list, then view-bind-many* is an alternate helper function that may be helpful to bind a bulk number of handlers to a view.

(view-bind-many* v id+type+updater-list) → view
v : view
id+type+updater-list : (listof (list string string world-updater))

Common event types include "click", "mouseenter", or"change". Note that the name of each event should not include an "on" prefix.

As an example:

(define (click-handler w v) ...)

(define (change-handler w v) ...)

(define-resource view.html)

(define my-static-view (->view view.html))

(define connected-view
  (view-bind-many* my-static-view
                  `(["id1" "click" ,click-handler]
                    ["id2" "click" ,click-handler]
                    ["id3" "change" ,change-handler])))
...

Show the element at the focus.

Hide the element at the focus.

(view-attr v name) → view
v : view
name : String

Get the attribute name at the focus.

(view-has-attr? v name) → boolean
v : view
name : String

Returns true if the element at the focus has an attribute name.

(update-view-attr v name value) → view
  v : view
  name : String
  value : String

Update the attribute name with the value value at the focus.

Remove the attribute name at the focus.

(view-css v name) → view
v : view
name : String

Get the css value name at the focus.

(update-view-css v name value) → view
  v : view
  name : String
  value : String

Update the css value n with the value v at the focus.

(view-id v) → world
v : view

Get the unique identifier of the node at the focus.

Get the form value of the node at the focus.

Update the form value of the node at the focus.

Dom nodes can be created by using xexp->dom, which converts axexp to a node, and attached to the view by usingview-append-child, view-insert-left, andview-insert-right.

Add the dom node d as the last child of the focused node. Focus moves to the inserted node.

Add the dom node d as the previous sibling of the focused node. Focus moves to the inserted node.

Add the dom node d as the next sibling of the focused node. Focus moves to the inserted node.

Remove the dom node at the focus from the view v. Focus tries to move to the right, if there’s a next sibling. If that fails, focus then moves to the left, if there’s a previous sibling. If that fails too, then focus moves to the parent.

5.3 Events

An event is a structure that holds name-value pairs. Whenever an event occurs in web-world, it may include some auxiliary information about the event. As a concrete example, location events from on-location-change and on-mock-location-change can send latitude and longitude values, as long as the world callback can accept the event as an argument.

kvpairs : (listof (list symbol (or/c string number)))

(event-ref evt name) → value
evt : event?
name : (or/c symbol string)

Get an value from the event, given its name.

Get an list of the event’s keys.

5.4 Dynamic DOM generation with xexps

We often need to dynamically inject new dom nodes into an existing view. As an example where the UI is entirely in code:

#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/web-world))

;; tick: world view -> world
(define (tick world view)
  (add1 world))

;; draw: world view -> view
(define (draw world view)
  (view-append-child view
                     (xexp->dom `(p "hello, can you see this? "
                                    ,(number->string world)))))

(big-bang 0 (initial-view
             (xexp->dom '(html (head) (body))))
            (on-tick tick 1)
            (to-draw draw))

Normally, we’ll want to do as much of the statics as possible with ".html" resources, but when nothing else will do, we can generate DOM nodes programmatically.

We can create new DOMs from an xexp, which is a s-expression representation for a DOM node. Here are examples of expressions that evaluate to xexps:

"hello world"

'(p "hello, this" "is an item")

(local [(define name "josh")]
`(p "hello" (i ,name)))

'(div (@ (id "my-div-0"))
(span "This is a span in a div"))

`(div (@ ,(fresh-id))
(span "This is another span in a div whose id is dynamically generated"))

More formally, a xexp is:

‹xexp›	::=	‹string›
	\|	‹symbol›
	\|	( ‹id› ‹xexp›* )
	\|	( ‹id› ( @ ‹key-value›* ) ‹xexp›* )
‹key-value›	::=	( ‹symbol› ‹string› )

To check to see if something is a xexp, use xexp?:

(xexp? x) → boolean
x : any

Return true if x is a xexp.

(xexp->dom an-xexp) → dom
an-xexp : xexp

Return a dom from the xexp.

When creating xexps, we may need to create unique ids for the nodes. The web-world library provides a fresh-id form to create these.

Return a string that can be used as a DOM node id.

We may also want to take a view and turn it back into an xexp.

(view->xexp a-view) → xexp
a-view : view

Coerses a view into a xexp.

5.5 Tips and tricks: Hiding standard output or directing it to an element

For a web-world program, output is normally done by usingto-draw. However, side effecting functions, such asprintf or display, are still available, and will append to document.body.

We may want to disable such printing or redirect it to a particular element on the page. For such purposes, use a combination ofcurrent-output-port and open-output-element to redirect the output of these side effect functions to somewhere else.

For example:

...
;; Redirect standard output to a div called "stdout-div".
(current-output-port (open-output-element "stdout-div"))
...
(big-bang ...
          (on-tick (lambda (world dom)
                     (printf "Tick!\n")
                     (add1 world)))
          ...)

All subsequent I/O side effects after the call tocurrent-output-port will be written out to thestdout-div, which can be easily styled with display: none to hide it from normal browser display.

Opens an output port that will be directed to write to the DOM element whose id is id. Note: writing to this port shouldn’t fail, even if the id does not currently exist on the page.

6 The JavaScript Foreign Function Interface

[[This needs to describe what hooks we’ve got from the JavaScript side of things.

In particular, we need to talk about the plt namespace constructed by the runtime, and the major, external bindings, likeplt.runtime.invokeMains.

The contracts here are not quite right either. I want to use JQuery as the type in several of the bindings here, but don’t quite know how to teach Scribble about them yet.]]

Returns #t if the value is a primitive JavaScript string value.

Coerses str to a JavaScript string value.

Coerses js-str to a string value.

Returns #t if x is a primitive JavaScript number.

Coerses num to a JavaScript number.

Coerses js-num to a number.

(js-null? x) → boolean
x : any

Returns #t if x is null.

JavaScript’s null value.

(get-attr obj key) → js-value
obj : js-object
key : (or/c string symbol)

Looks up the attribute of a JavaScript object.

(set-attr! obj key value) → js-value
  obj : js-object
  key : (or/c string symbol)
  value : js-value

Sets the attribute of a JavaScript object.

Dynamically loads a script from the URL.

Given either a string representation of a JavaScript function, or a javascript function object, returns a procedure that can be called.

For example, the following shows how to lift a simple function:

Caveats: the lifted function does no auto-coersion of values.

Given either a string representation of a JavaScript function, or a javascript function object, returns a procedure that can be called. The first two arguments to the function are special, representing the success and fail continuations that continue the rest of the computation.

For example, the following shows how to lift a simple function:

Note that, unlike js-function->procedure, the JavaScript implementation is expected to call the success or fail continuation explicitly.

As another example that takes advantage of the explicit continuation style:

Caveats: the lifted function does no auto-coersion of values.

(alert msg) → void
msg : string?

Displays an alert. Currently implemented using JavaScript’s alert function.

A JQuery-wrapped value representing the body of the DOM.

(call-method object method-name arg ...) → any/c
  object : any/c
  method-name : string?
  arg : any/c

Calls the method of the given object, assuming object is a JavaScript value that supports that method call. The raw return value is passed back.

For example,

should return the css color of the body.

($ locator) → any/c
locator : any/c

Uses JQuery to construct or collect a set of DOM elements, as described in the JQuery documentation.

For example,

(call-method ($ "<h1>Hello World</h1>")
"appendTo"
body)

will construct a h1 header, and append it to the document body.

Returns true if the running context supports JavaScript-specific functions.

Can only be called in a JavaScript context.

Returns the width of the viewport.

Can only be called in a JavaScript context.

Returns the height of the viewport.

6.1 Adding new event handlers to world programs with the FFI

The callback-driven asynchronous APIs in JavaScript can act as sources of World events. The following function allows web-world programs to bind to these sources.

Creates a new handler type that can be used with big-bang.big-bang calls the first argument at the beginning of the event-loop, and calls the second right before the event loop terminates.

The setup and shutdown functions are usually constructed with js-function->procedure in order to bind to native JavaScript APIs.

The setup function is called with an JavaScript function value that, when called, emits a new event into the world’s event loop. The return value of the setup function will be saved, and when the shutdown procedure calls, that value is passed to it, with the intent that shutting down a service will likely require information that’s produced at setup-time.

For example, we can reimplement some of the behavior ofon-location-change with the following:

#lang planet dyoo/whalesong
(require (planet dyoo/whalesong/js)
         (planet dyoo/whalesong/js/world))

(define setup-geo
  (js-function->procedure
   "function (locationCallback) {
        return navigator.geolocation.watchPosition(
            function(evt) {
                var coords = evt.coords;
                locationCallback(plt.runtime.makeFloat(coords.latitude),
                                 plt.runtime.makeFloat(coords.longitude)); })}"))

(define shutdown-geo
  (js-function->procedure
   "function (watchId) {
        navigator.geolocation.clearWatch(watchId); }"))

;; We create a new event handler type here.
(define on-geo (make-world-event-handler setup-geo shutdown-geo))

;; Once defined, we can use on-geo as just another world-handler type.

(define (move world view lat lng)
  (list lat lng))

(big-bang (list 'undefined 'undefined)
          (on-geo move))

7 Simple world programming

Whalesong provides a library to support writing functional I/O programs (A Functional I/O System). Here’s an example of such a world program:

[FIXME: embed a world program here.]

8 Acknowledgements

Whalesong uses code and utilities from the following external projects:

The following folks have helped tremendously in the implementation of Whalesong by implementing libraries, giving guidence, reporting bugs, and suggesting improvements.