Sunday, June 19, 2016

Less is More in Enterprise Development

Enterprise Development of Mission-Critical Functionality
is Complex, Time-Consuming, and Very Rarely Cheap
"Enterprise" development means development for mission-critical IT support processes for large companies.  Basically if you are developing software supports a mission-critical process for a large business, then you are doing enterprise development.

IT managers and enterprise architects are under pressure to provide IT solutions with a low total cost of ownership that allow for maximum business flexibility with high quality; in other words, solutions have to be cheap, fast, and good.

In these kinds of situations, the stakes are high, and decision makers are allergic to risk.   This is where the old saying, "Nobody ever got fired for buying IBM" comes from - the idea being that it's acceptable to spend more money in the name of reducing risk.

In mission-critical developments for large companies, the timing of the delivery and the functionality take the front seat, and the development is anything but cheap.

The main question I would like to explore is:
How can enterprise development be affordable, fast, and good?
I believe that the only way to do this is to reduce the complexity of the implementation of the solution.   Complex solutions are inherently expensive, either in development, in operations, or most commonly both.

This is the crux of the Less is More argument for enterprise development:
less complexity = more chance for success
I would like to propose several ways to reduce the complexity of enterprise solutions:
  1. Use proven off the shelf applications
  2. Establish and enforce process and technical integration standards
  3. Favor solutions based on configurability over coding
  4. Favor solutions that focus on operations rather than development
The first two points are hopefully self-explanatory, so I'll focus on the last two.

Favor solutions based on configurability over coding
This can also be formulated in a way that is unintuitive, namely: a superficially "weaker" solution with simpler development requirements can provide more value to your business than an extremely powerful and complex open-ended solution through a corresponding reduction in complexity.

A solution that favors configuration over development and simplifies the minimal development necessary can significantly reduce complexity and therefore risk vs an open-ended solution with complex development requirements.

Favor solutions that focus on operations rather than development
The idea here is that a solution focused on operations (the long-term view) will have a lower total cost of ownership than a solution that offers a large and complex development environment but little in the way of operational support.

One critical measure of success for any project is how well it actually performs in operations, therefore a focus on operations should directly lead to higher operational quality as well.

Simplicity is Underrated in Enterprise Development
These last two points hold particularly true for the enterprise integration layer, where problems can have customer visibility and can cost a significant amount of money in manual troubleshooting and in extreme cases in lost market share.

To conclude, enterprise development can be done with dramatically less risk by using solutions that reduce the complexity of the development, and by focusing on operations, such solutions can provide long-term value to customers with the foresight to deploy them.

Saturday, June 18, 2016

Data Serialization

A data serialization standard is required any time you need to to represent data externally from a program.

Two of the most popular data serialization standards are XML and JSON, both human-readable data serialization formats, a feature that has undoubtably contributed to their popularity.

XML is used heavily in enterprise contexts, although currently it seems that JSON is gaining in popularity over XML also in this space.  XML is a document markup language, and was not originally designed for data serialization.  Without a schema such as XSD or external type encoding rules, XML documents do not innately store data type information.

The SOAP protocol is based on XSDs; it is a very complex protocol with a hierarchy of complex specifications.  SOAP originally stood for Simple Object Access Protocol, but this acronym is no longer used as of SOAP 1.2.   SOAP ticks a lot of boxes that make enterprise architects happy, such as:
  • Transport independence
  • Distributed processing model
  • Design by contract (interface specification separate from the implementation)
  • Synchronous and asynchronous messaging
  • Universal standard
In my opinion, there is nothing simple about SOAP.  Also it's status as a "universal standard" is also questionable in practice because of incompatibilities between different vendor implementations of SOAP due to the complexity of the specifications.

I once had a conversation with an engineer from a global tier-1 integration software company (whose name will be left unmentioned) who related an anecdote to me how his company's SOAP solution was so good and so standards-compliant that they had recently won a lawsuit where one of their clients alleged incompatibility in their SOAP implementation due to failure to make a critical interface or set of interfaces work and sued them for negligence.  According to this engineer, it turned out that they were able to prove that their SOAP implementation was correct in court and won the suit.  The fact that the issue had to go to court at all is a testament to the complexity of SOAP; taking pride in having your customer sue you because you couldn't make an interface work (basically your only purpose as an integration ISV) and then winning the suit is another subject better left alone.

One simple extension of XML to support data typing is XML-RPC, which allows for several data types including lists (array) and key-value pairs (struct) to combine the simpler types to make arbitrarily complex data structures.  It's straightforward and simple, easy to ready, easy to serialize and deserialize.  It doesn't have the list of enterprise features that SOAP does, but to me it looks like a simple protocol designed by a single engineer, whereas SOAP looks like a typical protocol "designed by committee", where good engineering ("keep it simple, stupid") takes a back seat to politics and complexity is not avoided but rather embraced (a great way to destroy a technology or project in my experience!).

Nowadays you see more and more JSON.  JSON is an elegant data serialization format that was designed for the purpose it's used for.  JSON is human-readable and is the native format for JavaScript, a language that's grown in popularity along with the web.

For example, here is a JSON string:
 "firstName": "John",
 "lastName": "Smith",
 "isAlive": true,
 "age": 25,
 "address": {
   "streetAddress": "21 2nd Street",
   "city": "New York",
   "state": "NY",
   "postalCode": "10021-3100"

The same string with XML-RPC encoding looks like:
            <string>21 2nd Street</string>
            <string>New York</string>

Because it was designed for data serialization, JSON is better suited to this task than XML, even with simple extensions for data types like with XML-RPC.

Even better than JSON, but not as popular, is YAML (the subject of another blog post some time ago) .  YAML 1.2 is backwards-compatible with JSON, and is also extensible.  With YAML you get an elegant, human-readable data serialization standard that allows itself to be extended to support application or platform types.

The above data structure looks as follows with YAML (using block style for formatting):

firstName: "John"
lastName: "Smith"
isAlive: true
age: 25
  streetAddress: "21 2nd Street"
  city: "New York"
  state: "NY"
  postalCode: "10021-3100"

As with JSON, despite the lack of explicit type information, types are unambiguous and are preserved in the YAML string above.

YAML is extremely suitable for data serialization and therefore also for use in data exchange protocols.

Qore supports XML, JSON, and YAML for data serialization, but YAML is the preferred mechanism when possible to use.

Qore's YAML support is provided by the yaml module.  Qore implements a YAML-RPC protocol, which is basically JSON-RPC 1.1 but using YAML for data serialization.  Additionally, the DataStream protocol is a point-to-point protocol using YAML serialization for sending large volumes of data over HTTP with a small memory footprint.

From an engineering perspective, YAML is simple, elegant, extensible, and does a very important job very well.

While there are compelling arguments to be made for binary data serialization standards in some contexts, when you need an elegant, human-readable data serialization approach, I would highly recommend taking a look at YAML.

Tuesday, June 7, 2016

The Application as a Meta Programming Language


One of Qore's fundamental design goals is to provide a platform that allows programmers to dramatically increase productivity (reduce development time) while producing the high-quality results.

In a given complex functional domain, one approach for achieving this goal is to implement an application framework whose configuration and execution can also be influenced/governed by embedding appropriately-sandboxed user-defined code in the application itself.

Example Telecommunications Billing Application

Let's assume an application that implements some complex functionality, like a billing system for telecommunications services, for example.  With the approach described here, the application allows itself to be extended by supporting embedded and sandboxed user-defined code as attributes of configuration objects which are executed by the application itself in the appropriate context to provide for a very high level of functional customization.

In this hypothetical telecommunications billing system, this user-defined code might by attached to a tariff or discount, for example, and the billing system would then execute the code when calculating fees for consumed services.

The billing system would supply an internal API that could be used by embedded code for monetary calculations and for accessing contextual information, such as the customer's configuration, service usage data, etc.   Furthermore, fees and discounts calculated would be subject to taxation, and the calculation and application of taxation rules could similarly support runtime extensions through code attributes.

To allow for the highest level of user productivity and functional flexibility, our billing system should support simple and standard billing actions through (pure, non-code) configuration while allowing for more complex behavior to be defined through the execution of code attributes on appropriate configuration objects.

Vertical Support in Qore

Qore supports this sort of application development through its sandboxed code / logic containers, which allows applications such as the above to be developed that can be extended and customized with user-defined logic also written in Qore.  These logic containers can be created and destroyed at runtime, which allows for the behavior of the system to be updated without having to restart the server itself.

"Vertical" in this sense refers to the fact that both the server and the embedded code are written in the same programming language.  The nature of a vertical approach means that the programming language itself must feature enough power and flexibility to support the implementation of server applications while also supporting strong sandboxing controls to allow user-defined code to be safely embedded and executed to extend and customize the behavior of the application at runtime.

Specifically, this is implemented by the Program class in Qore.  Sandboxing controls are implemented through Parse Options, which are combination of restriction, capability, parsing and/or execution rules that define the sandboxing rules for the logic container.

Custom APIs can be imported into Program objects through methods such as Program::importClass() and Program::importFunction().  Access to system functionality in sandboxed logic containers can be restricted by functional domain through Parse Options, or the Program object can be created with no Qore system API (i.e. standard functions, classes, and/or constants are not available), and then the desired API can be imported manually.


Using sandboxed Program objects in Qore programs allows for applications to be written in Qore that support runtime extensions also written in Qore.  The fact that these logic containers can be created and destroyed at runtime also allows for such applications to support runtime changes and logic upgrades without the need to restart the system for the changes to take effect.

Programmer productivity in such an application is maximized through the introduction of simple APIs for use in embedded code executed in specific application contexts, which, if properly implemented, allow for the behavior of an application to be customized in ways that could not be imagined when the application was designed and created, and therefore also allow for maximum flexibility.

Finding the right balance between configuration and coding to maximize productivity is important; allowing too much freedom (for example, if the balance of coding vs configuration is shifted too far towards coding) can increase programming complexity and steepen the learning curve, while allowing for too little customization through embedded code might cripple the application (particularly if the application is not very configurable through standard configuration).

In this way, such an application can be considered a special purpose meta programming language; programmers use a combination of configuration and embedded code to develop their solutions in the application.