Sunday, June 19, 2016

Less is More in Enterprise Development

Enterprise Development of Mission-Critical Functionality
is Complex, Time-Consuming, and Very Rarely Cheap
"Enterprise" development means development for mission-critical IT support processes for large companies.  Basically if you are developing software supports a mission-critical process for a large business, then you are doing enterprise development.

IT managers and enterprise architects are under pressure to provide IT solutions with a low total cost of ownership that allow for maximum business flexibility with high quality; in other words, solutions have to be cheap, fast, and good.

In these kinds of situations, the stakes are high, and decision makers are allergic to risk.   This is where the old saying, "Nobody ever got fired for buying IBM" comes from - the idea being that it's acceptable to spend more money in the name of reducing risk.

In mission-critical developments for large companies, the timing of the delivery and the functionality take the front seat, and the development is anything but cheap.

The main question I would like to explore is:
How can enterprise development be affordable, fast, and good?
I believe that the only way to do this is to reduce the complexity of the implementation of the solution.   Complex solutions are inherently expensive, either in development, in operations, or most commonly both.

This is the crux of the Less is More argument for enterprise development:
less complexity = more chance for success
I would like to propose several ways to reduce the complexity of enterprise solutions:
  1. Use proven off the shelf applications
  2. Establish and enforce process and technical integration standards
  3. Favor solutions based on configurability over coding
  4. Favor solutions that focus on operations rather than development
The first two points are hopefully self-explanatory, so I'll focus on the last two.

Favor solutions based on configurability over coding
This can also be formulated in a way that is unintuitive, namely: a superficially "weaker" solution with simpler development requirements can provide more value to your business than an extremely powerful and complex open-ended solution through a corresponding reduction in complexity.

A solution that favors configuration over development and simplifies the minimal development necessary can significantly reduce complexity and therefore risk vs an open-ended solution with complex development requirements.

Favor solutions that focus on operations rather than development
The idea here is that a solution focused on operations (the long-term view) will have a lower total cost of ownership than a solution that offers a large and complex development environment but little in the way of operational support.

One critical measure of success for any project is how well it actually performs in operations, therefore a focus on operations should directly lead to higher operational quality as well.

Simplicity is Underrated in Enterprise Development
These last two points hold particularly true for the enterprise integration layer, where problems can have customer visibility and can cost a significant amount of money in manual troubleshooting and in extreme cases in lost market share.

To conclude, enterprise development can be done with dramatically less risk by using solutions that reduce the complexity of the development, and by focusing on operations, such solutions can provide long-term value to customers with the foresight to deploy them.

Saturday, June 18, 2016

Data Serialization

A data serialization standard is required any time you need to to represent data externally from a program.

Two of the most popular data serialization standards are XML and JSON, both human-readable data serialization formats, a feature that has undoubtably contributed to their popularity.

XML is used heavily in enterprise contexts, although currently it seems that JSON is gaining in popularity over XML also in this space.  XML is a document markup language, and was not originally designed for data serialization.  Without a schema such as XSD or external type encoding rules, XML documents do not innately store data type information.

The SOAP protocol is based on XSDs; it is a very complex protocol with a hierarchy of complex specifications.  SOAP originally stood for Simple Object Access Protocol, but this acronym is no longer used as of SOAP 1.2.   SOAP ticks a lot of boxes that make enterprise architects happy, such as:
  • Transport independence
  • Distributed processing model
  • Design by contract (interface specification separate from the implementation)
  • Synchronous and asynchronous messaging
  • Universal standard
In my opinion, there is nothing simple about SOAP.  Also it's status as a "universal standard" is also questionable in practice because of incompatibilities between different vendor implementations of SOAP due to the complexity of the specifications.

I once had a conversation with an engineer from a global tier-1 integration software company (whose name will be left unmentioned) who related an anecdote to me how his company's SOAP solution was so good and so standards-compliant that they had recently won a lawsuit where one of their clients alleged incompatibility in their SOAP implementation due to failure to make a critical interface or set of interfaces work and sued them for negligence.  According to this engineer, it turned out that they were able to prove that their SOAP implementation was correct in court and won the suit.  The fact that the issue had to go to court at all is a testament to the complexity of SOAP; taking pride in having your customer sue you because you couldn't make an interface work (basically your only purpose as an integration ISV) and then winning the suit is another subject better left alone.

One simple extension of XML to support data typing is XML-RPC, which allows for several data types including lists (array) and key-value pairs (struct) to combine the simpler types to make arbitrarily complex data structures.  It's straightforward and simple, easy to ready, easy to serialize and deserialize.  It doesn't have the list of enterprise features that SOAP does, but to me it looks like a simple protocol designed by a single engineer, whereas SOAP looks like a typical protocol "designed by committee", where good engineering ("keep it simple, stupid") takes a back seat to politics and complexity is not avoided but rather embraced (a great way to destroy a technology or project in my experience!).

Nowadays you see more and more JSON.  JSON is an elegant data serialization format that was designed for the purpose it's used for.  JSON is human-readable and is the native format for JavaScript, a language that's grown in popularity along with the web.

For example, here is a JSON string:
 "firstName": "John",
 "lastName": "Smith",
 "isAlive": true,
 "age": 25,
 "address": {
   "streetAddress": "21 2nd Street",
   "city": "New York",
   "state": "NY",
   "postalCode": "10021-3100"

The same string with XML-RPC encoding looks like:
            <string>21 2nd Street</string>
            <string>New York</string>

Because it was designed for data serialization, JSON is better suited to this task than XML, even with simple extensions for data types like with XML-RPC.

Even better than JSON, but not as popular, is YAML (the subject of another blog post some time ago) .  YAML 1.2 is backwards-compatible with JSON, and is also extensible.  With YAML you get an elegant, human-readable data serialization standard that allows itself to be extended to support application or platform types.

The above data structure looks as follows with YAML (using block style for formatting):

firstName: "John"
lastName: "Smith"
isAlive: true
age: 25
  streetAddress: "21 2nd Street"
  city: "New York"
  state: "NY"
  postalCode: "10021-3100"

As with JSON, despite the lack of explicit type information, types are unambiguous and are preserved in the YAML string above.

YAML is extremely suitable for data serialization and therefore also for use in data exchange protocols.

Qore supports XML, JSON, and YAML for data serialization, but YAML is the preferred mechanism when possible to use.

Qore's YAML support is provided by the yaml module.  Qore implements a YAML-RPC protocol, which is basically JSON-RPC 1.1 but using YAML for data serialization.  Additionally, the DataStream protocol is a point-to-point protocol using YAML serialization for sending large volumes of data over HTTP with a small memory footprint.

From an engineering perspective, YAML is simple, elegant, extensible, and does a very important job very well.

While there are compelling arguments to be made for binary data serialization standards in some contexts, when you need an elegant, human-readable data serialization approach, I would highly recommend taking a look at YAML.

Tuesday, June 7, 2016

The Application as a Meta Programming Language


One of Qore's fundamental design goals is to provide a platform that allows programmers to dramatically increase productivity (reduce development time) while producing the high-quality results.

In a given complex functional domain, one approach for achieving this goal is to implement an application framework whose configuration and execution can also be influenced/governed by embedding appropriately-sandboxed user-defined code in the application itself.

Example Telecommunications Billing Application

Let's assume an application that implements some complex functionality, like a billing system for telecommunications services, for example.  With the approach described here, the application allows itself to be extended by supporting embedded and sandboxed user-defined code as attributes of configuration objects which are executed by the application itself in the appropriate context to provide for a very high level of functional customization.

In this hypothetical telecommunications billing system, this user-defined code might by attached to a tariff or discount, for example, and the billing system would then execute the code when calculating fees for consumed services.

The billing system would supply an internal API that could be used by embedded code for monetary calculations and for accessing contextual information, such as the customer's configuration, service usage data, etc.   Furthermore, fees and discounts calculated would be subject to taxation, and the calculation and application of taxation rules could similarly support runtime extensions through code attributes.

To allow for the highest level of user productivity and functional flexibility, our billing system should support simple and standard billing actions through (pure, non-code) configuration while allowing for more complex behavior to be defined through the execution of code attributes on appropriate configuration objects.

Vertical Support in Qore

Qore supports this sort of application development through its sandboxed code / logic containers, which allows applications such as the above to be developed that can be extended and customized with user-defined logic also written in Qore.  These logic containers can be created and destroyed at runtime, which allows for the behavior of the system to be updated without having to restart the server itself.

"Vertical" in this sense refers to the fact that both the server and the embedded code are written in the same programming language.  The nature of a vertical approach means that the programming language itself must feature enough power and flexibility to support the implementation of server applications while also supporting strong sandboxing controls to allow user-defined code to be safely embedded and executed to extend and customize the behavior of the application at runtime.

Specifically, this is implemented by the Program class in Qore.  Sandboxing controls are implemented through Parse Options, which are combination of restriction, capability, parsing and/or execution rules that define the sandboxing rules for the logic container.

Custom APIs can be imported into Program objects through methods such as Program::importClass() and Program::importFunction().  Access to system functionality in sandboxed logic containers can be restricted by functional domain through Parse Options, or the Program object can be created with no Qore system API (i.e. standard functions, classes, and/or constants are not available), and then the desired API can be imported manually.


Using sandboxed Program objects in Qore programs allows for applications to be written in Qore that support runtime extensions also written in Qore.  The fact that these logic containers can be created and destroyed at runtime also allows for such applications to support runtime changes and logic upgrades without the need to restart the system for the changes to take effect.

Programmer productivity in such an application is maximized through the introduction of simple APIs for use in embedded code executed in specific application contexts, which, if properly implemented, allow for the behavior of an application to be customized in ways that could not be imagined when the application was designed and created, and therefore also allow for maximum flexibility.

Finding the right balance between configuration and coding to maximize productivity is important; allowing too much freedom (for example, if the balance of coding vs configuration is shifted too far towards coding) can increase programming complexity and steepen the learning curve, while allowing for too little customization through embedded code might cripple the application (particularly if the application is not very configurable through standard configuration).

In this way, such an application can be considered a special purpose meta programming language; programmers use a combination of configuration and embedded code to develop their solutions in the application.

Saturday, October 18, 2014

Exception-Safe Programming, RAII, and Deterministic GC

I'd like to make a follow-up post with more information on Qore's new Deterministic Garbage Collection support including more details and more information on why it's important for the language.

First of all regarding the language. deterministic garbage collection was always a goal of Qore's because of it's focus on the RAII idiom (wikipedia) for exception-safe programming and resource management (I also found the following very interesting link on this subject:

Some examples of RAII in Qore are (a subset of possible examples):
  • the AutoLock class releases the Mutex in the destructor (this class is designed to be used with scope-bound exception-safe resource management)
  • the Datasource class closes any open connection in the destructor, and, if a transaction is still in progress, the transaction is rolled back automatically and an exception is thrown before the connection is closed
  • the File class closes the file in the destructor if it's open
  • the RWLock class throws an exception if destroyed while any thread is still holding the lock; note that in this case the underlying object is only destroyed when all threads holding locks have released their locks; this is handled with Qore's thread resource handling and strong references to the underlying RWLock object while thread resources are held; thread resource handling is another potential topic for a blog post
  • the Socket class first shuts down any TLS/SSL connection and then closes the connection in the destructor if it's open
  • the ThreadPool class detaches all currently in-progress worker threads, cancels pending tasks not yet executed (by calling their cancellation closure, if any), terminates the worker thread and destroys the thread pool
Basically one of Qore's primary design points is to free Qore programmers from worrying about the details of memory and resource management; the use of the RAII idiom is a large part of this; also above you can see an example of negative feedback provided to programmers when mistakes are made - deleting a RWLock object while a lock is held (note that there is also scope-related resource management support in Qore in the form of the on_exit, on_success, and on_error statements).

Therefore since support for the RAII idiom is a critical feature of Qore's design, the language should always guarantee that objects are destroyed and therefore their resources are managed and associated memory freed when objects go out of scope.  This is tricky when there are circular references involved in the objects, particularly since Qore uses a reference-counted solution with atomic references due to its multi-threaded nature.  Consider the following post on this subject regarding .NET:  This gives a lot of background to the same problem that Qore (and Java) have for providing for deterministic garbage collection.

Basically to summarize, RAII is not supported in Java and .NET because it's hard to do.  Consider also VB6, basically if you have recursive object references in VB6, those objects will never be destroyed or collected by the system; the programmer has to delete objects with recursive references manually in order to destroy them.  This is basically the same situation as Qore before the introduction of deterministic GC.

The current status of the deterministic GC support in Qore svn is stable; it appears to finally be working in large complex object-oriented multi-threaded programs with good performance and (at the moment - knock on wood) free of deadlocks.  In another post I described the high-level approach with emphasis on the deadlock-avoidance approach since this was one of the trickiest parts to solve.  I'd like to provide more detail here on the recursive graph detection and recursive reference algorithm.

To find recursive graphs and calculate the number of recursive references for each member of the graph, Qore does a scan of some starting object.  Note that the recursive graph detection algorithm has to produce the same results for any given recursive graph independently of the starting node in the graph.  So from a high level, Qore scans through all reachable objects from a start object, and, when a recursive reference is found, increments the recursive count for that object (in fact this is done in a transactional way where all changes are committed atomically to the entire recursive graph at the end of the transaction or rolled back in case of certain kinds of lock contention in order to avoid deadlocks, but this is described in my previous blog post).  If a "path" is found from any given object to itself, the recursive count is set to one for all elements of the path, and any recursive reference then has its recursive count incremented.  Due to the recursive nature of the algorithm, during path detection new elements can be added to the recursive set while scanning an object, so processing the path has to also take this into account by checking if parent objects are reachable from existing recursive sets and must process their recursive counts appropriately (for example, if, while processing a path, one of the objects in the path already has a non-zero pending recursive count, then if the preceding object in the path was not already in the set, then its pending recursive count is incremented, otherwise it is not).  Additionally, if any predecessor in the current path of the current recursive node having a different recursive set is reachable from the current recursive set, then the recursive sets are merged into one.

STL sets are used to provide for O(ln(n)) searches of recursive sets found in the scan, however this may not be the ideal algorithm for small sets due to the overhead in managing red-black balanced binary trees normally used to implement sets in STL.  Also these sets are iterated several times, so fast iterator performance is important.  I believe that some analysis of the usage of data structures in the recursive scan could provide some optimizations.

Recursive scans are performed when changes are made to objects that could potentially create a new recursive set or change an existing recursive set.   The approach also features several checks designed to limit the cases where the recursive graph detection algorithm needs to be applied; I believe more cases can be found to further improve performance.

I have not made a systematic performance test of Qore with deterministic garbage collection enabled (it's currently enabled by default in Qore svn trunk), but from subjective testing on existing code, it seems fast.

You can see if deterministic garbage collection is enabled in Qore by typing qore -V; you should see a line like this if it's enabled:
version 0.8.12-6768 (builtin features: sql, threads, DGC)

If you see "DGC" in the output, then it's supported, otherwise not (currently requires svn trunk - 0.8.12 has not yet been released).

In Qore code, there is a parse define, HAVE_DETERMINISTIC_GC, that can be used to optionally parse code depending on the presence of deterministic garbage collection or not; for example, there are now regression tests in Qore that are executed when deterministic garbage collection is present.

Tuesday, September 30, 2014

Deterministic Garbage Collection

Qore has had a big design problem in its memory management regarding object collection; basically it was possible to make circular references to objects, and those objects would not be destroyed automatically, resulting in a memory leak.

The reason for this is that Qore used a simple strong reference count for object scope, so if object A pointed to object B, object B pointed to object A, then a circular reference would exist and neither object would be destroyed (or collected) when the objects would otherwise go out of scope.  Note that Qore treats objects like Java does in that objects are always passed as a copy of a reference to the object instead of by value.  Weak references to objects are also supported, but weak object references do not affect object collection (ie when the destructor is run on the object); normally the object's destructor is run on the object only when the strong reference count reaches zero (or the object is manually deleted with the delete operator).

Also due to Qore's multi-threaded nature, objects can go out of scope at any time since they can be deleted in another thread at any time.  Qore objects are thread-safe and references to an objects are wrapped in a read-write lock - as are all shared lvalues in Qore - basically access to all lvalues in Qore are wrapped in read-write locks except accesses to "pure" local variables, which are local variables where references are not taken of them and also not used in a closure, either of which case makes it possible to use the local variable in another thread and therefore causes the local variable to be created specially so that accesses are wrapped in read-write locks.

I considered using something like Java's garbage collection approach with a dedicated thread that would scan and collect objects, but I always wanted to do a deterministic garbage collector so that Qore's resource management approach with objects could be applied deterministically.  I finally have an initial working implementation of a deterministic garbage collection algorithm for Qore that is thread-safe, does not require a dedicated collector thread, has a solid deadlock-avoidance mechanism, and exhibits acceptable performance (which I expect can be further improved).

With this new approach, objects are collected immediately when they only have strong references that belong to a recursive cycle; so resources managed by the object are released in a deterministic way even when the objects participate in a recursive reference cycle.

I would like to describe the algorithm here including the locking approach and the deadlock-avoidance mechanism.

Basically the idea is to determine the number of strong references to an object that belong to a circular reference.  In fact, it is more complicated than this, because you have to consider the entire recursive directed graph as a whole and not a single object.  The recursive directed graph in this case is the set of all objects participating in the recursive reference chain.  If any object is reachable in the chain and also contains link(s) (ie members) to other objects in the chain, then it is a member of the recursive directed graph.  So the idea is to maintain the recursive strong reference count of every object in the recursive directed graph and then, when any member of the graph is strongly dereferenced, then the entire graph is checked to see if it can be collected, meaning that each member of the graph is checked to see if the strong reference count is equal to its recursive reference count.  If any single member of the graph has non-recursive references, then no object in the graph can be collected; only when the strong reference count equals the recursive reference count for all objects in the graph can the objects in the graph be collected.

This is accomplished by performing a scan of all reachable objects whenever a potentially-relevant change is made to an object.  In this case the recursive reference counts are calculated for all objects in the graph.

To accomplish this, each object is locked specially.  In fact objects now have a special form of read-write lock that includes a special form of the read lock called an rsection lock.  The rsection lock in a Qore object is a read lock that is unique in that first the read lock is acquired and then only one thread can hold the rsection lock at a time.  This allows objects to be scanned for recursive graphs while also allowing them to be read in other threads to maximize concurrency.  Additionally, since multiple rsection locks are acquired when performing recursive scanning (one for each object) and held for the duration of the scan, and since holding multiple locks could lead to deadlocks, and since this can and is performed in multiple threads simultaneously in multi-threaded object-oriented Qore programs, and since otherwise Qore avoids holding multiple locks simultaneously, the deadlock avoidance approach here is to apply a transaction-handling approach to the rsection scan and if any rsection lock cannot be acquired, the transaction is rolled back and we wait for a confirmation from the other thread that the contentious rsection lock has been released.  Also a transaction counter in each object is maintained, and, after waiting for a contentious rsection lock to be released, we see that the transaction count for the root object has been changed, then we know that the rsection scan has already taken place, so we exit the scan immediately.

Basically all changes to objects are stored in temporary data structures and then only committed to the objects in the graph if all rsection locks are acquired.

Also all containers (lists, hashes, and objects) contain a reachable object count, which is a sum of the children (list elements, hash keys, or object members) that have at least one object reachable through them.  This turned out to be efficient to calculate.  This allows us to ignore any container that has no reachable objects when performing rsection scans.

Additionally if an rsection scan fails due to lock contention after an lvalue change, the scanned objects are marked with an invalid rsection so that a deterministic scan is made on the next strong dereference.  When performing rsection scans due to a strong dereference, the scan is repeated after an rsection rollback until it is successful to guarantee deterministic collection when only recursive references remain.

There is still certainly a lot of room for improvement to this algorithm.  For example, the rsection transaction could be compared to any existing rsection graph and left in place if identical results are found, or possibly a delta operation could be performed on an existing rsection graph.

While this algorithm is complicated, the goal of achieving deterministic garbage collection is a valid one in my opinion.

Knowing exactly when your objects will be collected even in the case of participation in recursive directed graphs of strong references provides an advantage to Qore programmers regarding resource management with objects.

Currently deterministic garbage collection is enabled by default in Qore trunk, and I plan on continuing to work on it.

Feedback on this subject would be appreciated.

Thursday, March 7, 2013

REST vs RPC in Web Service Development

Ive been implementing web services using lightweight web service protocols such as XML-RPCJSON-RPC and (my favorite although proprietary) YAML-RPC for some time now.

I had some discussions last year about web service development using REST, a concept I had heard about for some time, but I did not see the advantages of it.  I knew that it had become the predominate web development architecture model, but for some reason I could not see the advantages of REST over pure lightweight RPC approaches.

My first experiences with an RPC protocol over HTTP was with SOAP, which is a technology I've grown to strongly dislike; now I consider SOAP a horrible enterprise technology since it's cumbersome and expensive to implement and often suffers from serious compatibility issues between different vendor implementations (Qore also has SOAP support - both server and client, however I would recommend avoiding it if you can).   My great appreciation for the lightweight RPC over HTTP protocols listed above is to no small degree based on my dislike of SOAP, so when hearing and reading about REST, I could not see that it would be as great a leap as from SOAP to XML-RPC for example.

(As a footnote Wikipedia lists XML-RPC as the precursor to SOAP, which makes SOAP an absolutely classic case of overengineering by committee - since from my point of view XML-RPC with all its faults is vastly superior to SOAP.  Also here is a link which humorously sums up my thoughts on SOAP:

However, finally I've gotten into REST a little deeper, and having developed an initial RestHandler for the Qore HTTP server, the advantages of a REST architecture are completely clear and compelling.

Using REST forces you to have a logical and straightforward organization of your web service development.  Each type of object will have it's own URL, and (in the Qore RestHandler at least) it will have a dedicated class to handle requests to objects of that type.

For example, in the Qore RestHandler, if you register a class with the name "workflows" at the root level of your REST handler, then a request like

   GET /workflows

will be directed to your workflow class's get() method, which will handle the request.

Classes (and therefore URLs) can be stacked, so a request like

   GET /workflows/1/instances/2?action=stop

Will be directed to the appropriate class's "getStop()" method.   If the appropriate method name (derived from the HTTP method name in the request and any action verb given as a URI argument) is not implemented in the class, then the RestHandler returns a user-friendly 400 response.

This easy and logical hierarchical organization of the objects corresponding to the URLs along with the simple dispatching to handler methods of objects of the appropriate class makes implementing the REST service trivial in Qore.

Additionally, the RestHandler transparently takes care of deserializing request bodies based on the HTTP Content-Type header and serializing any response body based on the client's HTTP Accept header.

This also makes the REST approach superior to the lightweight RPC protocols mentioned above.

It's easy to see why REST is the dominant web service architecture; Qore's RestHandler will continue to be under active development for the upcoming release of Qore (0.8.8), and I hope it will be useful for others as well.

Saturday, March 2, 2013

Class Inheritance vs Class Wrapping

Qore supports "standard" class inheritance for polymorphism, but also supports another construct that may be unique to Qore.  In Qore there is the possibility of "wrapping" a class and providing a compatible interface with the wrapped class without using inheritance.

There are several advantages and also some drawbacks to this approach.  Let me first explain what I mean by "wrapping" a class and also explain how it's done in Qore.

To "wrap" a class, you embed an object of the desired class as a member of the new class and implement a methodGate() method that redirects method calls from outside the class to unimplemented methods.  Additionally a memberGate() method can be implemented to redirect accesses to unknown members from outside the class and return an arbitrary value for the member access.

For example, you can wrap a class to provide logging for all method calls or provide external thread synchronization to an unsynchronized class.  The following is an example that does both.

class MySocket {
    private {
        Socket $.sock();
        Mutex $.m();

    static log(string $fmt) {
        vprintf(now_us().format("YYYY-MM-DD Z") + ": " + $fmt, $argv);
    any methodGate(string $method_name) {
        MySocket::log("MySocket::methodGate() method called: %y args: %y\n", $method_name, $argv);
        on_exit $.m.unlock();

        return callObjectMethodArgs($.sock, $method_name, $argv);

This can be very useful for testing and for other purposes, requires very little typing to implement, and its purpose is clear from the implementation (even without comments).  Additionally, this method is completely independent of the definition of the wrapped class.

The main disadvantage is that it doesn't claim compatibility with the Socket class, so you cannot pass an object of class MySocket to code that expects a Socket object.

Currently there is no way in Qore to wrap a class in this way and to remain type-compatible (i.e. to appear to be a subclass of the wrapped class).  To remain type-compatible in this way, every method of the class has to be explicitly wrapped as above, and if the wrapped class has any method interface changes (for example, new method), the subclass has to changed accordingly.

However despite the drawbacks, it's still a very useful construct.   Possibly a future version of Qore will allow such wrapped classes to claim class compatibility with the wrapped class and therefore address the typing issues.