Formats Library

Generic Read and Write Functions

The Formats library contains read, and write functions for serializing and deserializing data in other formats, e.g. XML or CSV. The XML and CSV plugins are built into the standard DataSonnet distributions while other formats can be supported by implementing DataFormatPlugin interface.

These functions can be used if the data contains embedded data in other formats, for example:

{
    "embeddedXMLValue": "<test>Hello</test>"
}

If the payload or variable itself is in the format other than JSON, it will be automatically detected by the DataSonnet, therefore using the read will cause an error. The input and output of payload and variables is controlled by the headers.

read(string input, string inputMimeType, object params={})

Reads input data in the specified mime type and accepts additional parameters which override default plugin behavior. The list and format of parameters is specific to a plugin implementation.

Example

ds.read(payload.embeddedXMLValue, "application/xml")

Example

local params = {
    "NamespaceDeclarations" : {
        "datasonnet" : "http://www.modusbox.com"
    },
    "NamespaceSeparator": "%",
    "TextValueKey": "__text",
    "AttributeCharacter": "*"
};

ds.read(payload.embeddedXMLValue, "application/xml", params);

write(string output, string outputMimeType, object params={})

Outputs the data into specified data format and accepts additional parameters which override default plugin behavior. The list and format of parameters is specific to a plugin implementation.

Example

ds.write(payload.embeddedXMLValue, "application/xml")

Example

local params = {
    "NamespaceDeclarations" : {
        "datasonnet" : "http://www.modusbox.com"
    },
    "NamespaceSeparator": "%",
    "TextValueKey": "__text",
    "AttributeCharacter": "*",
    "XmlVersion" : "1.1",
    "AutoEmptyElements": false
};

ds.write(payload.someObj, "application/xml", params);

XML Format

MIME types and identifiers

  • application/xml

  • xml

read

Converts input XML string to a JSON object using BadgerFish convention:

  • Element names become object properties

  • Text content of elements goes in the $ property of an object.

  • Nested elements become nested properties

  • Multiple elements at the same level become array elements.

  • Attributes go in properties whose names begin with @.

  • Active namespaces for an element go in the element’s @xmlns property.

  • The default namespace URI goes in @xmlns.$.

  • Other namespaces go in other properties of @xmlns.

  • Elements with namespace prefixes become object properties, too.

  • The @xmlns property goes only in object relative to the element where namespace was declared.

  • CDATA sections go in properties named #1, #2, etc.

  • Text fragments in mixed contents (elements and text) goes in properties named $1, $2, etc.

Example

Embedded XML value
<?xml version="1.0" encoding="UTF-8"?>
<test:root xmlns:test="http://www.modusbox.com">
    <test:datasonnet version="1.0">Hello World</test:datasonnet>
</test:root>
DataSonnet map:
ds.read(payload.embeddedXMLValue, "application/xml")
Result
{
    "test:root": {
        "@xmlns": {
            "test":"http://www.modusbox.com"
        },
        "test:datasonnet": {
            "@version": "1.0",
            "$": "Hello World"
        }
    }
}

Providing an optional params object allows for additional control over JSON generation. The params is a JSON object where following properties can be set:

Parameter Description Default value

NamespaceSeparator

Separator between the prefix and the local name

:

TextValueKey

Key prefix for the text value property

$

CdataValueKey

Key prefix for the CDATA value property

#

AttributeCharacter

Property key prefix which designates an XML element attribute

@

NamespaceDeclarations

Map of internal prefixes to the namespaces which overrides namespaces declarations in the input. Multiple values are allowed, for example:

"NamespaceDeclarations" : {
    "datasonnet" : "http://www.datasonnet.com",
    "test" : "urn:com.foo.bar",
    "": "http://www.modusbox.com"
}

RootElement

if set, the output will be wrapped in a root element with the given name

Example

Embedded XML value
<?xml version="1.0" encoding="UTF-8"?>
<test:root xmlns:test="http://www.modusbox.com">
    <test:datasonnet version="1.0">Hello World</test:datasonnet>
</test:root>
DataSonnet map:
local params = {
    "NamespaceDeclarations" : {
        "datasonnet": "http://www.modusbox.com"
    },
    "NamespaceSeparator": "%",
    "TextValueKey": "__text",
    "AttributeCharacter": "*"
};

ds.read(payload.embeddedXMLValue, "application/xml", params);
Result
{
    "datasonnet%root": {
        "*xmlns": {
            "datasonnet": "http://www.modusbox.com"
        },
        "datasonnet%datasonnet": {
            "*version": "1.0",
            "__text": "Hello World"
        }
    }
}

write

Converts the input JSON object into XML using the Badgerfish convention.

The input JSON must have a single key which will be mapped to the root element of the resulting XML.

Correct:
{
  "person": {
    "firstName": "John",
    "lastName": "Doe",
    "title": "Rookie DataSonnet mapper"
  }
}
Incorrect:
{
  "firstName": "John",
  "lastName": "Doe",
  "title": "Rookie DataSonnet mapper"
}
Incorrect:
{
  "person": {
    "firstName": "John",
    "lastName": "Doe",
    "title": "Rookie DataSonnet mapper"
  },
  "anotherKey": "anotherValue"
}

Example

Payload
{
    "test:root": {
        "@xmlns": {
            "test":"http://www.modusbox.com"
        },
        "test:datasonnet": {
            "@version": "1.0",
            "$": "Hello World"
        }
    }
}
DataSonnet map:
{
    embeddedXMLValue: ds.write(payload, "application/xml")
}
Result
{
    "embeddedXMLValue": "<?xml version=\"1.0\" encoding=\"UTF-8\"?> <test:root xmlns:test=\"http://www.modusbox.com\"> <test:datasonnet version=\"1.0\">Hello World</test:datasonnet> </test:root>"
}

Providing a params object allows for more control over generated XML. In addition to the parameters described in the read section, the following XML output-only parameters are supported:

Parameter Description Default value

XmlVersion

XML version in the XML declaration

1.0

Encoding

XML encoding

UTF-8

AutoEmptyElements

If set to true, empty elements are mapped to self-closing tags. If set to false, start- and end tags are generated.

true

NullAsEmptyElement

If set to true, element with null value is treated as empty element. Otherwise null values are skipped.

true

OmitXmlDeclaration

If set to true, XML declaration is not written in the resulting output.

false
Payload
{
    "test%root": {
        "*xmlns": {
            "test":"http://www.modusbox.com"
        },
        "test%datasonnet": {
            "*version": "1.0",
            "__text": "Hello World"
        },
        "test%empty": {}
    }
}
DataSonnet map:
local params = {
    "NamespaceDeclarations" : {
        "datasonnet" : "http://www.modusbox.com"
    },
    "NamespaceSeparator": "%",
    "TextValueKey": "__text",
    "AttributeCharacter": "*",
    "XmlVersion" : "1.1",
    "AutoEmptyElements": false
};

{
    embeddedXMLValue: ds.write(payload, "application/xml")
}
Result
{
    "embeddedXMLValue": "<?xml version=\"1.1\" encoding=\"UTF-8\"?> <datasonnet:root xmlns:test=\"http://www.modusbox.com\"> <datasonnet:datasonnet version=\"1.0\">Hello World</datasonnet:datasonnet> <datasonnet:empty/> </datasonnet:root>"

CSV Format

MIME types and identifiers

  • application/csv

  • text/csv

  • csv

read

Parses the CSV and converts it to a JSON array of objects. It expects the CSV input to be in a default format, with first row as column headers, comma separator, double quote, backslash escape character and \n newline character. CSV headers are used as keys for the corresponding JSON object values.

Example

Embedded CSV value
"First Name","Last Name",Phone
William,Shakespeare,"(123)456-7890"
Christopher,Marlow,"(987)654-3210"
DataSonnet map:
{
    local csvInput = ds.read(payload.embeddedCSVValue, "application/csv");

    {
        name: csvInput[0]["First Name"] + " " + csvInput[0]["Last Name"]
    }
}
Result
{
    "name": "William Shakespeare"
}

Providing an optional params object allows more control over the format of the input CSV. The params is a JSON object where following properties can be set:

Parameter Description Default value

UseHeader

If set to true, the first row of CSV will be interpreted as a list of column headers and will map to the JSON object property names

true

Quote

specifies the quote character

"

Separator

CSV separator character

,

Escape

CSV escape character

\\

NewLine

New line character combination

\n

Example

Embedded CSV value
'William'|'Shakespeare'|'(123)456-7890'
'Christopher'|'Marlow'|'(987)654-3210'
DataSonnet map:
local params = {
    "UseHeader": false,
    "Quote": "'",
    "Separator": "|",
    "Escape": "\\",
    "NewLine": "\n"
};

local csvInput = ds.read(payload.embeddedCSVValue, "application/csv", params);

{
    name: csvInput[0][0] + " " + csvInput[0][1]
}
Result
{
    "name": "William Shakespeare"
}

write

Creates a CSV out of an array of JSON objects, using default quote, separator, escape and new line characters. The keys of JSON object values are used as a CSV headers.

Example

Payload
[
  {
    "First Name": "William",
    "Last Name": "Shakespeare",
    "Phone": "(123)456-7890"
  },
  {
    "First Name": "Christopher",
    "Last Name": "Marlow",
    "Phone": "(987)654-3210"
  }
]
DataSonnet map:
{
    embeddedCSVValue: ds.write(payload, "application/csv")
}
Result
{
    "embeddedCSVValue": "\"First Name\",\"Last Name\",Phone\nWilliam,Shakespeare,\"(123)456-7890\"\nChristopher,Marlow,\"(987)654-3210\"\n"
}

Providing an optional params object allows for more control over the format of the output CSV. Quote, separator, escape and new line characters can be specified, CSV can be created without headers - in this case the input can be an array of arrays. In addition, a list of columns can be specified to override the JSON object names. In addition to the parameters described in the read section, the following CSV output-only parameters are supported:

  • Headers - an array of strings to use as column names

Example

Payload
[
  [
    "William",
    "Shakespeare",
    "(123)456-7890"
  ],
  [
    "Christopher",
    "Marlow",
    "(987)654-3210"
  ]
]
DataSonnet map:
local params = {
    "UseHeader": false,
    "Quote": "'",
    "Separator": "|",
    "Escape": "\\",
    "NewLine": "\n"
};

{
    embeddedCSVValue: ds.write(payload, "application/csv", params)
}
Result
{
    "embeddedCSVValue": "'William'|'Shakespeare'|'(123)456-7890'\n'Christopher'|'Marlow'|'(987)654-3210'\n"
}

Java Objects

read

Converts POJO to JSON format using Jackson ObjectMapper.

The following read parameters are supported:

Parameter Description Default value

DateFormat

Converts POJO date / time fields to JSON strings using specified date format. See SimpleDateFormat for details.

yyyy-MM-dd’T’HH:mm:ss.SSSZ

FailOnEmptyBeans

If set to true, an exception is thrown if no serializer is found for a Java type. Setting it to false instructs DataSonnet to ignore the field that cannot be serialized. See Jackson Serialization Features for details.

true

write

Converts JSON objects to Java POJOs using Jackson ObjectMapper.

The following write parameters are supported:

Parameter Description Default value

DateFormat

Converts POJO date / time fields to JSON strings using specified date format. See SimpleDateFormat for details.

yyyy-MM-dd’T’HH:mm:ss.SSSZ

OutputClass

Produces an instance of specified class. If parameter is not set, the following conversion rules are used:

  • Objects → java.util.HashMap

  • Arrays → java.util.ArrayList

  • String values → java.lang.String

  • Boolean values → java.lang.Boolean

  • Numerical values → java.lang.Number

JAXBElement serialization and deserialization

When serializing or deserializing Java object with fields of type JAXBElement, the mapping has to include additional fields value, name and declaredType. For example, a JAXB-annotated class may look like this:

@XmlRootElement(name = "WsdlGeneratedObj")
public class WsdlGeneratedObj {
    @XmlElementRef(name = "testField", namespace = "http://com.datasonnet.test", type = JAXBElement.class, required = true)
    protected JAXBElement<TestField> testField;
...
}

Mapping from JSON to the WsdlGeneratedObj is:

/** DataSonnet
version=1.0
output.application/java.OutputClass=com.datasonnet.javatest.WsdlGeneratedObj
*/
{
    "testField": {
        "name": "{http://com.datasonnet.test}testField",
        "declaredType": "com.datasonnet.test.TestField",
        "value": {
            test: "Hello World"
        }
    }
}