JSONtoXML Application Class - An Alternative Method to Parse JSON in Peoplecode

In this article, we will document a solution I developed that makes working with JSON in PeopleSoft easier. We first start of with an introduction to the problem then go into my solution.

Background

First we need to lay the ground work for why I developed this tool. I have been doing a lot of integration in PeopleSoft over the last few years. This includes providing web services to third parties from PeopleTools as well as consuming web services in PeopleSoft. I have found over the last few years that there is a titanic shift in these third parties moving to JSON encoding of data instead of XML and/or SOAP. JSON over REST has become the De facto integration paradigm on the WEB. JSON is being used to store configuration files, data in SQL and NOSQL databases, and structure log entries to name a few. It seems like everything is in JSON now for any application that has been developed in the last 5 years and has a “modern” tool-set (whatever that means).

I was doing some advanced integrations with a particular API that only communicates in JSON. I was tasked with making this integration from PeopleTools to the API work. The JSON processing was really the sticking point. I explored many options to process JSON in PeopleSoft, including but not limited to:

• The PeopleSoft Document Technology
  • This proved to have many issues processing JSON. I could never get it to work. It was riddled with bugs. I started looking at it in 8.53 and it was not great. I revisited the code in 8.54 and it still had a lot of issues. The document designer does not let you produce some JSON structures that are completely valid based on the JSON Spec. One example, I documented in JSON Parsing Limitations in 8.53. Oracle support’s response was to make the third party communicate in a structure that could be parsed by the document technology. This was not an option and a rather amusing response.
  • There was no easy way to process dynamic JSON that was returned from an API. This feature was really important for this integration. For example, if a web service returns different data depending on the input, you CANNOT easily handle this with the document API without some hacks and work-arounds. The document framework assumes a static structure and you pass the string representation which un-marshals that string to the document structure. This is pretty lame when working with web services as you may search and get back a string JSON object or you could get back an array of objects. The document technology cannot handle this. Similarly, if the API returns a mix of objects, it can’t handle this as well. One example of this is maybe an “event stream” that returns a series of objects that changed. The JSON data may have variable objects of what changed. You can’t dynamically parse through the JSON and take action depending on the return using the Document technology. However, you can easily do this in PeopleCode using the XML class. There is no similar functionality to the XML class for JSON to “walk the JSON tree”. I really wish there was.
  • Quite frankly the document technology is just downright painful to work with. It really looks like beta code to me. An interesting thing I noticed is that some of the Campus Solutions delivered web services that produce very large JSON documents do NOT use the documents technology. I hate to read too much into that but the internal development teams are not “Dogfooding” the document technology which does not bode well.

I also searched around the inter-webs as well as the PeopleSoft application code an found different solutions:

• The ever experimenting Jim Marion has written several articles on this topic including but not limited to: Parsing JSON with PeopleCode & JavaScript and PeopleCode Array Parameters
• Campus Solutions has a Peoplecode JSON Parsing library. However, that only exists in CS. Additionally, in my testing I found it had a few bugs in processing some JSON. It is also completely undocumented and likely subject to change so you don’t want to build any custom code on a “sand foundation.”
• Cameron Barre’s Jsoft Code - This only creates JSON and does not parse it. This library is worth checking out if you need to produce JSON in PeopleCode through. Thank you Cameron for sharing this.

None of these solutions I was very happy with for various reasons.

Brainstorming Solutions

I had started to think to myself. You know PeopleCode already has a really solid XML class built in that can parse XML and you can navigate the structure of the data programmatically. I have been using it for well over 10 years without any memorable issues. You can dynamically process any XML and “walk the tree.” I thought:

Why can’t I just somehow convert JSON to XML then just work with the data in XML format?

This led me down a research path of different open source libraries that could be used in a cross platform manner. The cross platform was key as I wanted a solution that would work for Windows and Linux shops. I could not find a solution that was acceptable to me. Most of them came with many dependencies.

However, I been paying really close attention to the Golang programming language over the last few years. It has become first choice programming tool for anything from scripting to building web services and command line applications. The great thing about Golang is that you can actually create the source code on a Mac then cross compile it for windows, linux and many other environments and get a stand alone executable that can be deployed and run on any client. This really makes for a nice and simple distribution model which just requires you to deploy one file with zero dependencies. This is probably one of many reason you are starting to see a lot of different tools like Docker, Terraform and Kubernetes written in Go. The distribution of a simple binary is really easy especially in a cloud infrastructure. Compare this with other distribution models where you may need to have a certain JVM installed prior to installing your code in the case of Java. Or maybe in the Node, Ruby or Python world you need many different toolsets installed and then dependencies downloaded. Golang can remove these issues. Plus it is really fast, easy to learn and the standard library is freaking amazing.

My Solution - JSONtoXML Application Class + Command Line Application

I came up with a working solution that utilizes a PeopleCode wrapper Application class that really ends up calling an operating system level executable to convert from JSON to XML. You then just work with the data in XML using standard PeopleCode classes.

High Level JSON Parsing Flow

1. Load JSON from a file, web service or %Request into a string
2. Instantiate the JSONToXML application class
3. Call the ConvertJSONToXML method passing in the JSON string
4. Check to make sure the conversionSuccess property is true which shows that the JSON could be converted to XML.
5. Get the convertedXML property from the class which is a string representation of the JSON converted to XML
6. Now you can just process the inbound data using the standard PeopleCode XML classes.

What is happening under the hood?

The JSONToXML application class is really just a wrapper that calls a command line utility that you need accessible to the application server or process scheduler running your PeopleCode. The class shells out to the command line application passing the JSON string. The command line application (written in golang) does the conversion if possible and passes XML back. The PeopleCode JSONToXML reads the return code and standard output from the operating system and sets some class properties. The application class is not really doing any conversion work. It is an API to work with the command line application.

The source code for the JSON to XML Command Line Application is on my github account. The CLI is actually a stand alone application that can be run outside the context of PeopleSoft. The real work is currently done by a golang library called github.com/clbanning/anyxml. I just wrote a CLI wrapper around this library.

The XML will resemble the JSON structure. However, since there is not a one to one mapping between XML and JSON encodings some XML nodes are inserted to represent the document “root” as well as array elements. JSON has no concept of a “root” element and arrays are represented differently. Additionally, there is some character escaping differences between JSON and XML so the executable has to escape some characters to generate valid XML.

Using the Conversion tool

Here is a simple example PeopleCode program that uses the tool. We assume you have installed all the tools mentioned in the installation section below. We assume you have created the application class as CHG_UTILITIES:JSONToXML.

import CHG_UTILITIES:JSONToXML


local string &jsonString;

/* &jsonString gets populated with json from somewhere  here */

Local CHG_UTILITIES:JSONToXML &jsonUtil = create CHG_UTILITIES:JSONToXML();

&jsonUtil.ConvertJSONToXML(&jsonString);

If &jsonUtil.conversionSuccess Then
  /* JSON Was converted to XML */
  local XmlDoc &xmlReturn
  &xmlReturn = CreateXmlDoc("");
  
  If &jsonUtil.convertedXML <> "" And
    &xmlReturn.ParseXmlString(&jsonUtil.convertedXML) Then

        /* Process your XML Document &xmlReturn here using standard PeopleCode XML  */
  else
    /* XML was empty or could not be parsed in some way. Handle the error here */
  end-if;
else
  /* The JSON could not be converted. See &jsonUtil.errors string for more information */
end-if;

We assume the reader knows how to use the Peoplecode XML library and will not show examples of processing the converted XML.

Installing the Tool

First you need to download the JSON to XML Command Line Application. Follow the Project Readme file to get a compiled binary for your PeopleSoft server architecture . (I may end up compiling these and distributing them on github let me know if that interests you.)

Take the jsontoxml binary created above and give it to your system admins to put it on your PeopleSoft servers. Your system admin will have to figure out the best place here. The overall requirements are:

• The binary/executable must be accessible to the PeopleCode program where you will use this. This may be application servers and process scheduler servers
• Make sure that the OS level user running either the app or batch servers has execution rights to this file.

Now you need to install the wrapper application class PeopleCode. Create an Application Class as the following path: CHG_UTILITIES:JSONToXML. Then paste in the code below.

You will have to change the code below where the &commandPath variable is being set. This is trying to pickup the location of the command line tool installed above and will vary based on many factors.

class JSONToXML
   method ConvertJSONToXML(&inputJSON As string);
   property boolean conversionSuccess;
   property string convertedXML;
   property string errors;
end-class;

method ConvertJSONToXML
   /+ &inputJSON as String +/
   
   %This.convertedXML = "";
   Local string &commandPath;
   
   /* INSTALLATION TODO: These two lines will likely need to be changed */
   &commandPath = GetEnv("PS_CUST_HOME");
   &commandPath = &commandPath | "/custom/scripts/jsontoxml";
   
   Local JavaObject &runtime = GetJavaClass("java.lang.Runtime").getRuntime();
   Local JavaObject &command = CreateJavaObject("java.lang.String[]", &commandPath, &inputJSON);
   
   
   Local JavaObject &process = &runtime.exec(&command);
   
   Local JavaObject &inputStreamReader = CreateJavaObject("java.io.InputStreamReader", &process.getInputStream());
   Local JavaObject &bufferedReader = CreateJavaObject("java.io.BufferedReader", &inputStreamReader);
   Local integer &returnCode = &process.waitFor();
   
   Local any &inputLine;
   Local any &fullOutput;
   
   
   While True
      &inputLine = &bufferedReader.readLine();
      If (&inputLine <> Null) Then
         &fullOutput = &fullOutput | &inputLine;
      Else
         Break;
      End-If;
   End-While;
   
   
   If &returnCode = 0 Then
      %This.conversionSuccess = True;
      %This.convertedXML = "<?xml version='1.0'?>" | String(&fullOutput);
      %This.errors = "";
   Else
      %This.conversionSuccess = False;
      
      %This.convertedXML = "";
      %This.errors = &fullOutput;
      
      
   End-If;
   
   
end-method;

Contact me if you have suggestions or comments.