Qore Programming Language

Qore Optimizations

2011-11-27T00:16:00.000-08:00

Work on Qore 0.8.4 is progressing slowly but surely.

Here's a rundown of what's in svn right now - targeted for 0.8.4.

Pseudo-Methods

Qore 0.8.4 will support pseudo-methods; these are methods that can run on any type; basically any value will support a set of methods depending on its type. These pseudo-classes are in a (relatively-flat) hierarchy; the base pseudo-class is "any", then there is a class for each data type that inherits from this class.

Pseudo-methods can be used for convenience and will also support operations that are very inefficient to do the traditional Qore way. For example, to get the first key from a hash, previously one had to make an evaluation like this:

    my *string $firstKey = (keys $hash)[0];

For a large hash, this is terribly inefficient, creating a list of all keys in the hash and then returning the first element of the list. Now, with the new pseudo-methods, it can be done like this:

    my *string $firstKey = $hash.firstKey();

Very efficient and clear. Also, to check the data type, the traditional Qore way would be the following:

    if (type($value) == Type::String) {
    }

Whereas the type() function returns a string, and Type::String = "string"; so a string comparison is made. The new way would be to use "any"::typeCode() like this:

    if ($value.typeCode() == NT_STRING) {
    }

Whereas NT_STRING is the integer code for a string (3, the same as the C++ NT_STRING constant coincidentally enough), so this is a much more efficient expression.

Currently there are only a handful of pseudo-methods implemented for each type, but more will be implemented before the 0.8.4 release. In particular I plan on adding more date/time methods to get quick information about date/time values.

Of course pseudo-methods are typed and when resolved at parse-time, particularly for integer operations, run-time optimizations are used. In the last example above (using "string"::typeCode()), no memory allocations or will occur; the operands of the boolean comparison operator will be evaluated in an integer context, meaning that internally native integer values are returned (and not, for example, QoreBigIntNode structures), so there are no memory allocations or atomic references made (which could cause SMP cache invalidations, etc). These kinds of operations are very fast and scalable - runtime optimizations made possible because data types were available at parse-time.

And this leads to the second big feature for Qore 0.8.4:

Performance and Memory Optimizations

Qore 0.8.4 will feature major performance and memory optimizations. I've already implemented a big part of this by implementing optimized handling for integer local variables. Local[ly-scoped] variables are already thread-local (and therefore do not require any locking for reads and writes), and now local variables restricted to integers (declared as int or softint) are also stored as native integers. Assignment operations are also evaluated in an integer context; as above, only native C++ integer values are passed around and stored; there are no memory allocations or atomic reference counts.

Additionally, I've started porting all the operators over to Qore's new operator framework (subclasses of QoreOperatorNode - to replace expressions made with QoreTreeNode which will be removed from Qore when the migration is complete), and the operators ported also support optimized integer operations; ie when types are known to be integer at parse-time, the run-time variants of the operators are used that use the optimized integer operations (again without any memory allocations or atomic operations).

This all leads to major performance improvements with integer operations.

I've also added some of the necessary infrastructure for optimizing floating-point operations (support for local variables, optimized operator variants), but have not finished this work yet.

I also have in mind an LLVM back-end for Qore in the future; I plan on adding more optimizations and propagating additional information about the code during parsing which will be used when generating compiled code for further optimizations. By that I mean not just type restrictions (which can obviously lead to major optimizations like the above) but also, for example, marking lvalues as constant in certain scopes, enforcing the QC_CONST flag for functions and methods and more. But at the moment that's a ways down the road - I won't be able to get to LLVM integration before the 0.8.4 release for sure.

Qore 0.8.3

2011-08-27T22:47:00.000-07:00

The next release of Qore is coming very soon; the major new feature for this release will be Windows support.

Qore is now capable of being built as a native DLL for Windows (XP and up) - finally without Cygwin (which made for a pretty slow binary actually). This was made possible through the MinGW cross compiling environment (http://mingw-cross-env.nongnu.org/).

The main blocking point for this port (which has been requested many times over the years) was my lack of familiarity with Windows development tools (and my dislike of the Windows command line). This software allowed me to use Linux and OSX to make the Windows port. It turned out to be a lot easier than I expected; the MinGW pthreads library worked perfectly (I only needed to make minor changes as pthread_t is not a pointer); the socket code was fairly easy to update (the main thing I had to do there was update the code checking for errors and then getting the Windows error messages instead of using errno); and then I had to write new code for time zone handling to read zone information from the Windows registry instead of using the zoneinfo DB (as on UNIXes). Also the dlfcn library for Windows (http://code.google.com/p/dlfcn-win32/) allowed for seamless loadable module handling with the same code as on UNIX.

The release is basically ready; I just want to port a few more modules to Windows before I make it public (so far I've got the xml, json, yaml, and uuid modules also working on Windows).

Other than a large number of bug fixes since 0.8.2 and a few minor improvements here and there, the only other feature of note is support for simple conditional parsing based on C/C++ preprocessor-type %define and %ifdef, %ifndef, etc directives.

Current (tentative) release notes:

http://qore.svn.sourceforge.net/viewvc/qore/qore/trunk/RELEASE-NOTES?view=markup

Current (tentative) Windows README:

http://qore.svn.sourceforge.net/viewvc/qore/qore/trunk/README-WINDOWS?view=markup

Qore 0.8.1 Released

2010-12-25T09:31:00.000-08:00

Qore 0.8.1 has just been released with a ton of bugfixes and new features. Major new features are: SQL prepared statement API (currently only supported by the soon-to-be released oracle driver v2.0), a much improved type system, support for class constants and static class variables, and a more standard syntax for declaring function and method return types by allowing the type name to be declared at the beginning of the function or method signature, as in C/C++ or Java, for example.

Additionally, there are new parse options that allow for programming without the "$" and "$." signs for variables, class method calls, and object member references.

This last change hopefully will make a lot of people happy - I had a lot of requests to do away with the "$" signs, and now it's possible. Unfortunately, the code highlighting solutions out there will have to be updated again to handle the new %allow-bare-refs and %new-style parse options. %new-style combines both of the new parse options %allow-bare-refs and %assume-local, the latter meaning that all variables are assumed to have local scope unless declared global.

Here is an example with %new-style:

%new-style

int sub do_something(int p1, string str, *hash h) {
   for (int x = 0; x < p1; ++x) {
      stderr.printf("error: %s\n", str);
   }
   return p1 + 2;
}

Backwards compatibility is a priority and has been maintained. We'll see if the decision to allow for this new programming style is a good one; sometimes too much choice can just lead to confusion and therefore is counterproductive. However at least some people are very happy with it.

Current Qore Status

2010-06-23T00:50:00.001-07:00

Qore 0.8.0 has been released along with all updated modules; modules have in most cases been updated to take advantage new APIs (mostly regarding typing, date/time improvements, and new Datasource/DatasourcePool methods).

There was a slight delay before the 0.8.0 release to improve the type system; now the type system is internally capable of supporting very flexible types, where one or more types are accepted and one or more types are returned, making types such as "nothing or string" possible to implement internally.

However Qore's user type declaration support in the parser and the function and class library were not updated to take advantage of the new flexible typing support, as everything was stable and tested and applying the new support for more flexible typing would delay the release by probably a month or two, but at least the internal changes were in place and are a part of the Qore library's API and ABI.

One of the coolest new developments to make use of Qore's new typing support is the new qt4 module, which allows Qore code to implement sophisticated platform-independent QT4-based GUIs (note that the qt4 module does take advantage of Qore's new flexible type system, allowing NOTHING to be passed for some classes to simulate a NULL pointer, for example).

Here are some things to expect in future releases of Qore: implicit typing and other parser improvements (0.8.0 is a great improvement in this area already), improved execution speed, and SMP scalability. And most interestingly JIT compilation support using the LLVM project. This is the most exciting part of future development of Qore that once again should take the Qore language to another level. I am astounded at what an awesome project LLVM is; how well documented it is, how well supported and active it is, what it can do and what a language designer can do with it. This will take some time, but will be some of the most interesting work to date done with Qore, and the results should be nothing short of amazing.

I wish I could give a timeline for any of these new developments, but I cannot; they will be done as time permits.

The Joy of YAML

2010-05-16T04:11:00.001-07:00

I had been working so hard on the next release of qore for so long, I had to take a short break, in which I made the new yaml module. The yaml module is currently a very small module in terms of source code, that allows qore data types (except objects) to be serialized and deserialized in YAML format. It uses libyaml to do the real work.

The great thing about YAML is that it is much better suited to representing data in text format than XML because it's much more concise and readable for humans. Additionally, with the addition of one custom YAML tag (!duration), all native Qore types can be serialized and deserialized as YAML with no information loss.

Compared to XML-RPC, YAML supports time zone information and time resolution to the microsecond (actually YAML's !!timestamp type supports arbitrary fractional seconds), and with our custom !duration type, support relative date/time values in an ISO-8601-like format (with the addition that time values may be negative and an additional character to specify microseconds). Of course YAML is much more readable and concise than XML-RPC.

Compared to JSON, YAML is very similar of course, but is extensible and supports more data types out of the box. JSON is missing !!timestamp and !!binary (base-64 encoded binary type). JSON is as consice and readable as YAML (because YAML, at least YAML 1.2, is a superset of JSON).

Take the following Qore data structure (valid with Qore 0.8.0+):
(1, "two", NOTHING, 2010-05-05T15:35:02.100, False, 1970-01-01Z,
(hash(), (), "three \"things\""), P2M3DT10H14u, now_us(),
binary("hello, how's it going, this is a long string, you know XXXXXXXXXXXXXXXXXXXXXXXX"),
("a" : 2.0,
"b" : "hello",
"key" : True))

Here's how the serialization looks:
YAML:
[1, "two", null, '2010-05-05 15:35:02.1 +02:00', false, 1970-01-01, [{}, [], "three
\"things\""], '0000-02-03 10:00:00.000014 Z', '2010-05-16 13:27:15.859195 +02:00',
!!binary "aGVsbG8sIGhvdydzIGl0IGdvaW5nLCB0aGlzIGlzIGEgbG9uZyBzdHJpbmcsIHlvdSBrbm93IFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWA==",
{"a": 2, "b": "hello", "key": true}]

XML-RPC (note the duration is not serialized correctly, time zone and us info is lost):
<struct><member><name>data</name><value><array><data><value><i4>1</i4></value><value><string>two</string></value><value/><value><dateTime.iso8601>20100505T15:35:02</dateTime.iso8601></value><value><boolean>0</boolean></value><value><dateTime.iso8601>19700101T00:00:00</dateTime.iso8601></value><value><array><data><value><struct></struct></value><value><array><data/></array></value><value><string>three "things"</string></value></data></array></value><value><dateTime.iso8601>00000203T10:00:00</dateTime.iso8601></value><value><dateTime.iso8601>20100516T13:31:23</dateTime.iso8601></value><value><base64>aGVsbG8sIGhvdydzIGl0IGdvaW5nLCB0aGlzIGlzIGEgbG9uZyBzdHJpbmcsIHlvdSBrbm93IFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWA==</base64></value><value><struct><member><name>a</name><value><double>2.000000</double></value></member><member><name>b</name><value><string>hello</string></value></member><member><name>key</name><value><boolean>1</boolean></value></member></struct></value></data></array></value></member></struct>

JSON (minus the binary data that cannot be serialized, note dates are serialized as strings):
[ 1, "two", null, "2010-05-05 15:35:02.100 Wed +02:00 (CEST)", false, "1970-01-01 00:00:00 Thu Z (UTC)", [ { }, [ ], "three \"things\"" ], "", "2010-05-16 13:32:44.792114 Sun +02:00 (CEST)", { "a" : 2, "b" : "hello", "key" : true } ]

The incredible thing was that I could not find any standard YAML-RPC protocol definition. The closest I could find was a partially-documented protocol called !okay/rpc. So I just implemented a simple YAML-RPC handler and client based on JSON-RPC 1.1 and it works great. I will probably simplify it a bit more to be a little more like !okay/rpc but with a described fault message, then document it and put it online for people to review.

I'm really happy I found YAML; it's conciseness, extensibility, and readability make it a far superior alternative to XML and XML-RPC for data serialization for Qore. In the future I will look at making it possible to serialize and deserialize objects as well - if a class supports writing out its state by using Qore data that can be then passed to the appropriate constructor or other such method on the remote end, it would solve this problem.

Note that Qore's YAML module still is undocumented, but is stable and somewhat tested. IT requires qore 0.8.0+ (still unreleased - only in svn).

Time Zone Support

2010-04-20T06:41:00.000-07:00

Time zone support has recently been added to qore in svn; version 0.8.0 will support time zones. Qore uses the system's zoneinfo database (if it can find it) to load in information about daylight savings time and time zone names, etc.

The entire date/time implementation was internally reimplemented for this change, although the old APIs remain for backwards-compatibility. Basically now all absolute date/time values not having an explicit time zone identifier will be assumed to be local time.

During the change I also extended the precision of relative and absolute date/time values to the microsecond (1,000,000th of a second), previously it was only to the millisecond.

Hard Typing in Qore 0.8.0

2010-03-10T12:42:00.000-08:00

Qore 0.8.0 development is at an advanced stage. One of the major features of this release will be the addition of hard typing. This is an interesting subject because it's not always obvious how hard typing should be added to a language that previously was entirely dynamically typed.

Several issues have presented themselves so far. One is how to handle variables declared with a certain type but not assigned any value. Qore's syntax for typed variable declarations looks like this:
my int $x;

One of the reasons for adding hard (aka strong) typing to the language is that is allows many more errors to be caught at parse time that otherwise can only be caught at run time. Consider the following code:
sub func(int $i) {
}

my int $x;
func($x);

The dilemma is - should the typed variable declaration cause $x to get a default value for the type, or should the value of $x be undefined (or actually in Qore: NOTHING) and therefore still cause a run time type error to be thrown. I've just checked how perl6 should handle this, and it appears that they take both approaches - as above with a lower-case "int", $x will get the default value for the type (0), with an upper-case "Int" the value will be undefined.

Qore's current implementation in svn (subject to change) is that $x holds NOTHING and a run time type exception will be thrown when func(int) is called.

Qore currently contains some other logic that allows such code to work:
my list $l;
$l += "str";

This will cause $l to be a list with "str" as the sole element; so even though $l was not initialized with a value in the declaration, the += operator still behaves as expected.

The idea is to allow the programmer to check if variables have been assigned a value or not. When we had Qore assigning default values in every declaration without an assignment, we found that it broke a lot of code that we otherwise expected to work.

Furthermore, some types cannot have a default value, such as objects (at least for classes that do not have a constructor that takes no arguments) and for the reference and code types. In order to make type handling consistent, we decided that variables and object members will only get values if they are explicitly assigned.

Now that I'm writing this down, it seems to me that the best solution would be to implement some kind of magic to cover the run time error above and pass the default value for the type to func(int) above - a solution similar to the one implemented for the += operator. That would be great, because then we can avoid run time type errors even though we've declared all our types (something hard typing should give us) and still be able to tell if we've assigned the variable or not. Otherwise we could modify the parser to track if a variable's been assigned or not when it's used, however this would not be a perfect solution because the parser cannot always know this with certainty (halting problem), so in some cases run time type errors would still occur.

It wasn't totally clear to me what would happen in Perl6 if an undefined Int value is passed to a function that expects an Int as a value.

Another issue is making the included function and class library friendly for strongly-typed code. There are a lot of functions and class methods that can return multiple values depending on the argument types. Since qore 0.8.0 in svn now supports overloading, we simply tag each variant with its parameter types and return type (note that an overloaded version of a function or method is called a variant in Qore). However there are cases when the return type depends on the values of the arguments, and not just the types (for example, if you pass a string with an invalid URL to the parseURL() function, it will return NOTHING). In these cases we're writing alternate, new versions of the functions that either throw an exception or return a default value for the return type. In this last example, there is now a parse_url() function that always returns a hash if the URL can be parsed, otherwise an exception is thrown.

Many of Qore's functions simply return NOTHING if the arguments are not of the expected type. In Qore 0.8.0, these functions have a default, untyped variant mapped to f_noop(), a function that simply returns NOTHING. All other possibilities are mapped to the actual functions that perform the useful work that needs to be done. This allows qore to recognize function calls that do not make sense and report that they return NOTHING at parse time (as long as types are declared in all the relevant places). An example of this type of function is parseURL(), a new version, parse_url() has been added that only accepts types that can produce a result.

Most of the other issues that have come up have been solved satisfactorily, method overloading in class trees (implemented successfully), default arguments (implemented for user and builtin code), matching variants at parse time and at run time (Qore tries to match the variant at parse time if types are available, otherwise variant matching is performed at run time).

However another issue is that the type system is very simple and flat; you may declare a variable of type list, but you cannot restrict the element type of the list. The same with hash keys and references (you cannot currently declare a variable as a reference to a particular type). Also it's currently not possible to declare code references to code with a particular signature (the new builtin type code "code" is used to restrict lvalues to being assigned a closure or call reference).

It's all subject to improvement before the 0.8.0 release. The libsmoke-based qt4 module has been converted to use hard typing and has been a very valuable test bed for the hard typing implementation.

Qore in svn is currently stable- if you want to check out qore with hard typing, grab the source from svn and compile it. Qore in svn is also currently compiling on windows with cygwin, although without modules and only as a monolithic binary.

To download qore from svn, use the following command:

svn co https://qore.svn.sourceforge.net/svnroot/qore/qore/trunk qore

Comments are very much appreciated!

Qore Stuff

2010-02-27T04:05:00.000-08:00

I finally got first post, and I just had to start my own blog to do it. Here I will be talking about what's going related to the Qore Programming Language. At the moment a major new release is in development, and it's pretty interesting right now, because hard typing has been added to the language.