Saturday, December 25, 2010

Qore 0.8.1 Released

Qore 0.8.1 has just been released with a ton of bugfixes and new features. Major new features are: SQL prepared statement API (currently only supported by the soon-to-be released oracle driver v2.0), a much improved type system, support for class constants and static class variables, and a more standard syntax for declaring function and method return types by allowing the type name to be declared at the beginning of the function or method signature, as in C/C++ or Java, for example.

Additionally, there are new parse options that allow for programming without the "$" and "$." signs for variables, class method calls, and object member references.

This last change hopefully will make a lot of people happy - I had a lot of requests to do away with the "$" signs, and now it's possible. Unfortunately, the code highlighting solutions out there will have to be updated again to handle the new %allow-bare-refs and %new-style parse options. %new-style combines both of the new parse options %allow-bare-refs and %assume-local, the latter meaning that all variables are assumed to have local scope unless declared global.

Here is an example with %new-style:

int sub do_something(int p1, string str, *hash h) {
for (int x = 0; x < p1; ++x) {
stderr.printf("error: %s\n", str);
return p1 + 2;
Backwards compatibility is a priority and has been maintained. We'll see if the decision to allow for this new programming style is a good one; sometimes too much choice can just lead to confusion and therefore is counterproductive. However at least some people are very happy with it.

Wednesday, June 23, 2010

Current Qore Status

Qore 0.8.0 has been released along with all updated modules; modules have in most cases been updated to take advantage new APIs (mostly regarding typing, date/time improvements, and new Datasource/DatasourcePool methods).

There was a slight delay before the 0.8.0 release to improve the type system; now the type system is internally capable of supporting very flexible types, where one or more types are accepted and one or more types are returned, making types such as "nothing or string" possible to implement internally.

However Qore's user type declaration support in the parser and the function and class library were not updated to take advantage of the new flexible typing support, as everything was stable and tested and applying the new support for more flexible typing would delay the release by probably a month or two, but at least the internal changes were in place and are a part of the Qore library's API and ABI.

One of the coolest new developments to make use of Qore's new typing support is the new qt4 module, which allows Qore code to implement sophisticated platform-independent QT4-based GUIs (note that the qt4 module does take advantage of Qore's new flexible type system, allowing NOTHING to be passed for some classes to simulate a NULL pointer, for example).

Here are some things to expect in future releases of Qore: implicit typing and other parser improvements (0.8.0 is a great improvement in this area already), improved execution speed, and SMP scalability. And most interestingly JIT compilation support using the LLVM project. This is the most exciting part of future development of Qore that once again should take the Qore language to another level. I am astounded at what an awesome project LLVM is; how well documented it is, how well supported and active it is, what it can do and what a language designer can do with it. This will take some time, but will be some of the most interesting work to date done with Qore, and the results should be nothing short of amazing.

I wish I could give a timeline for any of these new developments, but I cannot; they will be done as time permits.

Sunday, May 16, 2010

The Joy of YAML

I had been working so hard on the next release of qore for so long, I had to take a short break, in which I made the new yaml module. The yaml module is currently a very small module in terms of source code, that allows qore data types (except objects) to be serialized and deserialized in YAML format. It uses libyaml to do the real work.

The great thing about YAML is that it is much better suited to representing data in text format than XML because it's much more concise and readable for humans. Additionally, with the addition of one custom YAML tag (!duration), all native Qore types can be serialized and deserialized as YAML with no information loss.

Compared to XML-RPC, YAML supports time zone information and time resolution to the microsecond (actually YAML's !!timestamp type supports arbitrary fractional seconds), and with our custom !duration type, support relative date/time values in an ISO-8601-like format (with the addition that time values may be negative and an additional character to specify microseconds). Of course YAML is much more readable and concise than XML-RPC.

Compared to JSON, YAML is very similar of course, but is extensible and supports more data types out of the box. JSON is missing !!timestamp and !!binary (base-64 encoded binary type). JSON is as consice and readable as YAML (because YAML, at least YAML 1.2, is a superset of JSON).

Take the following Qore data structure (valid with Qore 0.8.0+):
(1, "two", NOTHING, 2010-05-05T15:35:02.100, False, 1970-01-01Z,
(hash(), (), "three \"things\""), P2M3DT10H14u, now_us(),
binary("hello, how's it going, this is a long string, you know XXXXXXXXXXXXXXXXXXXXXXXX"),
("a" : 2.0,
"b" : "hello",
"key" : True))

Here's how the serialization looks:
[1, "two", null, '2010-05-05 15:35:02.1 +02:00', false, 1970-01-01, [{}, [], "three
\"things\""], '0000-02-03 10:00:00.000014 Z', '2010-05-16 13:27:15.859195 +02:00',
!!binary "aGVsbG8sIGhvdydzIGl0IGdvaW5nLCB0aGlzIGlzIGEgbG9uZyBzdHJpbmcsIHlvdSBrbm93IFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWA==",
{"a": 2, "b": "hello", "key": true}]

XML-RPC (note the duration is not serialized correctly, time zone and us info is lost):
<struct><member><name>data</name><value><array><data><value><i4>1</i4></value><value><string>two</string></value><value/><value><dateTime.iso8601>20100505T15:35:02</dateTime.iso8601></value><value><boolean>0</boolean></value><value><dateTime.iso8601>19700101T00:00:00</dateTime.iso8601></value><value><array><data><value><struct></struct></value><value><array><data/></array></value><value><string>three "things"</string></value></data></array></value><value><dateTime.iso8601>00000203T10:00:00</dateTime.iso8601></value><value><dateTime.iso8601>20100516T13:31:23</dateTime.iso8601></value><value><base64>aGVsbG8sIGhvdydzIGl0IGdvaW5nLCB0aGlzIGlzIGEgbG9uZyBzdHJpbmcsIHlvdSBrbm93IFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWA==</base64></value><value><struct><member><name>a</name><value><double>2.000000</double></value></member><member><name>b</name><value><string>hello</string></value></member><member><name>key</name><value><boolean>1</boolean></value></member></struct></value></data></array></value></member></struct>

JSON (minus the binary data that cannot be serialized, note dates are serialized as strings):
[ 1, "two", null, "2010-05-05 15:35:02.100 Wed +02:00 (CEST)", false, "1970-01-01 00:00:00 Thu Z (UTC)", [ { }, [ ], "three \"things\"" ], "", "2010-05-16 13:32:44.792114 Sun +02:00 (CEST)", { "a" : 2, "b" : "hello", "key" : true } ]

The incredible thing was that I could not find any standard YAML-RPC protocol definition. The closest I could find was a partially-documented protocol called !okay/rpc. So I just implemented a simple YAML-RPC handler and client based on JSON-RPC 1.1 and it works great. I will probably simplify it a bit more to be a little more like !okay/rpc but with a described fault message, then document it and put it online for people to review.

I'm really happy I found YAML; it's conciseness, extensibility, and readability make it a far superior alternative to XML and XML-RPC for data serialization for Qore. In the future I will look at making it possible to serialize and deserialize objects as well - if a class supports writing out its state by using Qore data that can be then passed to the appropriate constructor or other such method on the remote end, it would solve this problem.

Note that Qore's YAML module still is undocumented, but is stable and somewhat tested. IT requires qore 0.8.0+ (still unreleased - only in svn).

Tuesday, April 20, 2010

Time Zone Support

Time zone support has recently been added to qore in svn; version 0.8.0 will support time zones. Qore uses the system's zoneinfo database (if it can find it) to load in information about daylight savings time and time zone names, etc.

The entire date/time implementation was internally reimplemented for this change, although the old APIs remain for backwards-compatibility. Basically now all absolute date/time values not having an explicit time zone identifier will be assumed to be local time.

During the change I also extended the precision of relative and absolute date/time values to the microsecond (1,000,000th of a second), previously it was only to the millisecond.

Wednesday, March 10, 2010

Hard Typing in Qore 0.8.0

Qore 0.8.0 development is at an advanced stage. One of the major features of this release will be the addition of hard typing. This is an interesting subject because it's not always obvious how hard typing should be added to a language that previously was entirely dynamically typed.

Several issues have presented themselves so far. One is how to handle variables declared with a certain type but not assigned any value. Qore's syntax for typed variable declarations looks like this:
my int $x;

One of the reasons for adding hard (aka strong) typing to the language is that is allows many more errors to be caught at parse time that otherwise can only be caught at run time. Consider the following code:
sub func(int $i) {

my int $x;

The dilemma is - should the typed variable declaration cause $x to get a default value for the type, or should the value of $x be undefined (or actually in Qore: NOTHING) and therefore still cause a run time type error to be thrown. I've just checked how perl6 should handle this, and it appears that they take both approaches - as above with a lower-case "int", $x will get the default value for the type (0), with an upper-case "Int" the value will be undefined.

Qore's current implementation in svn (subject to change) is that $x holds NOTHING and a run time type exception will be thrown when func(int) is called.

Qore currently contains some other logic that allows such code to work:
my list $l;
$l += "str";

This will cause $l to be a list with "str" as the sole element; so even though $l was not initialized with a value in the declaration, the += operator still behaves as expected.

The idea is to allow the programmer to check if variables have been assigned a value or not. When we had Qore assigning default values in every declaration without an assignment, we found that it broke a lot of code that we otherwise expected to work.

Furthermore, some types cannot have a default value, such as objects (at least for classes that do not have a constructor that takes no arguments) and for the reference and code types. In order to make type handling consistent, we decided that variables and object members will only get values if they are explicitly assigned.

Now that I'm writing this down, it seems to me that the best solution would be to implement some kind of magic to cover the run time error above and pass the default value for the type to func(int) above - a solution similar to the one implemented for the += operator. That would be great, because then we can avoid run time type errors even though we've declared all our types (something hard typing should give us) and still be able to tell if we've assigned the variable or not. Otherwise we could modify the parser to track if a variable's been assigned or not when it's used, however this would not be a perfect solution because the parser cannot always know this with certainty (halting problem), so in some cases run time type errors would still occur.

It wasn't totally clear to me what would happen in Perl6 if an undefined Int value is passed to a function that expects an Int as a value.

Another issue is making the included function and class library friendly for strongly-typed code. There are a lot of functions and class methods that can return multiple values depending on the argument types. Since qore 0.8.0 in svn now supports overloading, we simply tag each variant with its parameter types and return type (note that an overloaded version of a function or method is called a variant in Qore). However there are cases when the return type depends on the values of the arguments, and not just the types (for example, if you pass a string with an invalid URL to the parseURL() function, it will return NOTHING). In these cases we're writing alternate, new versions of the functions that either throw an exception or return a default value for the return type. In this last example, there is now a parse_url() function that always returns a hash if the URL can be parsed, otherwise an exception is thrown.

Many of Qore's functions simply return NOTHING if the arguments are not of the expected type. In Qore 0.8.0, these functions have a default, untyped variant mapped to f_noop(), a function that simply returns NOTHING. All other possibilities are mapped to the actual functions that perform the useful work that needs to be done. This allows qore to recognize function calls that do not make sense and report that they return NOTHING at parse time (as long as types are declared in all the relevant places). An example of this type of function is parseURL(), a new version, parse_url() has been added that only accepts types that can produce a result.

Most of the other issues that have come up have been solved satisfactorily, method overloading in class trees (implemented successfully), default arguments (implemented for user and builtin code), matching variants at parse time and at run time (Qore tries to match the variant at parse time if types are available, otherwise variant matching is performed at run time).

However another issue is that the type system is very simple and flat; you may declare a variable of type list, but you cannot restrict the element type of the list. The same with hash keys and references (you cannot currently declare a variable as a reference to a particular type). Also it's currently not possible to declare code references to code with a particular signature (the new builtin type code "code" is used to restrict lvalues to being assigned a closure or call reference).

It's all subject to improvement before the 0.8.0 release. The libsmoke-based qt4 module has been converted to use hard typing and has been a very valuable test bed for the hard typing implementation.

Qore in svn is currently stable- if you want to check out qore with hard typing, grab the source from svn and compile it. Qore in svn is also currently compiling on windows with cygwin, although without modules and only as a monolithic binary.

To download qore from svn, use the following command:

svn co qore

Comments are very much appreciated!

Saturday, February 27, 2010

Qore Stuff

I finally got first post, and I just had to start my own blog to do it. Here I will be talking about what's going related to the Qore Programming Language. At the moment a major new release is in development, and it's pretty interesting right now, because hard typing has been added to the language.