The JSONTool allows you to call local or remote json-based RESTful web services and parse the results for inclusion in a velocity template, page or widget. The following examples show how to use the JSONTool and JSONObjects with JSON in dotCMS.
- Methods
- Limitations
- Retrieve and Parse External JSON (
$json.fetch
and$json.post
) - Create JSON Objects ($json.generate)
- Toolbox.xml Configuration
Methods
By default, the JSONTool is mapped in the Velocity Context with the variable name $json
:
- $json == com.dotmarketing.viewtools.JSONTool
The following methods are supported:
Method | Usage | Description |
---|---|---|
fetch | $json.fetch( URL ) ,$json.fetch( URL, Timeout ) ,$json.fetch( URL, Headers ) ,$json.fetch( URL, Timeout, Headers ) | Retrieve and parse JSON from the specified URL. |
post | $json.post( URL ) | Performs a full post to a URL, allowing you to supply headers, JSON data, form params, etc., and retrieves the JSON response. |
put | $json.put( URL ) | Performs a full put to a URL, allowing you to supply headers, JSON data, form params etc., and retrieves the JSON response. |
| generate | $json.generate( Object )
| Generate JSON objects from Java objects, strings, or maps.
Limitations
Keep mind that any use of the $json.fetch("https://remoteUrl.com/")
is a synchronous call and is only going to be as reliable and as responsive as the endpoint https://remoteUrl.com/
that the $json.fetch
is calling. If that remote endpoint has a latency of a few seconds, it means that every dotCMS page that calls that endpoint is going to hang for a few seconds before the page can begin to render. On a loaded site, or a site that does a $json.fetch
call on every page (say in a header or footer) this can block all the receiving threads in tomcat and take a server down.
How to deal with Synchronous Remote Calls:
Use the BlockCache to cache the
$json.fetch
and the results If you want to make the request server side, that is fine, but wrap the call and rendered results in a#dotCache
block. This is especially important when making remote calls. Something like:#dotcache("currentMortgageRates",3600) #set($myJson = $json.fetch("https://remoteUrl.com/")) <div> <b>$myJson.type</b> : $myJson.rate % </div> #end
which will cache it the result for an hour and will not take a site down if the remote endpoint gets slow. For more information on block caching, see: https://dotcms.com/docs/latest/tag-based-caching-block-cach
Make remote
$json
calls client side, in javascript dotCMS blocks a receiver thread when a$json.fetch
call is made and waiting for a response. This means a visitor will have to wait until the remote response is returned before they will see a page rendered. If a site is designed to make the same request client side, using asynchronous javascript, dotcms will render the page fully and no threads in dotCMS will be blocked on dotCMS internally - only the visitors browser will hang trying to render the async request. This prevents a remote API problem from taking a dotCMS server down.Create a cron that imports real time data into dotCMS You can create a
vtl
script file that that calls a remote service and stores that value in dotCMS as a piece of content - and then use this content on your pages. This pattern is robust - the content can be updated asynchronous and your server will survive if there are any issues with the remote API. See this forum thread as an example:
- Final note - it is never recommended to make
$json.fetch
calls to a local dotCMS server This is a inefficient pattern, where 1 request begets 2 or 3 internal requests to fulfill. It is fragile as domain names and ports that dotCMS run on can change. Many times, an API that was calling an internal domainhttp://www.customerdomain.com/api/...
changes tohttps://customerdomain.com/api
which can cause all remote calls to fail. Also a dotCMS server running behind load balancers or a NATed firewall might not be able resolve an external domain name as an internal resource - and callinglocalhost/api
might fail as the ports that dotCMS server listens on (internally) might change. Regardless, it is never recommended to use$json.fetch
to call internal dotCMS APIs as it has proven too brittle and inefficient.
Retrieve and Parsing External JSON ($json.fetch)
The $json.fetch()
, $json.put()
, $json.post()
methods both retrieve and parse JSON from a URL you supply. The methods return a JSON object whose properties can be traversed via the .
(dot) operator.
These two methods behave very similar in most respects. For example, both methods allow you to retrieve JSON data from an external web API, both allow you to send headers with the request, and both return a traversible JSON object. There are 2 main differences between the fetch()
and post()
methods:
- The
fetch()
method uses an HTTP GET to submit the request, while thepost()
method uses an HTTP POST. - Since it uses a POST instead of a GET, the
post()
method allows you to send data with the request.
Usage
The $json.fetch()
method calls HTTP GET and can be called in any of the following ways:
$json.fetch( URL )
$json.fetch( URL, Timeout )
$json.fetch( URL, Headers )
$json.fetch( URL, Timeout, headerMap )
The $json.post()
method calls HTTP POST and can be called in any of the following ways:
$json.post( URL, headerMap, rawData )
$json.post( URL, Timeout, headerMap, rawData )
$json.post( URL, headerMap, paramMap )
$json.post( URL, Timeout, headerMap, paramMap )
The $json.put()
method calls HTTP PUT and can be called in any of the following ways:
$json.put( URL, headerMap, rawData )
$json.put( URL, Timeout, headerMap, rawData )
$json.put( URL, headerMap, paramMap )
$json.put( URL, Timeout, headerMap, paramMap )
Parameter | Methods | Description |
---|---|---|
URL | fetch, post, put | The URL of the JSON resource to retrieve. To retrieve JSON from a file, upload a file into dotCMS and specify the dotCMS URL of the uploaded file (referenced from the root of the dotCMS Site Browser tree). |
Timeout | fetch, post, put | Timeout in milliseconds. |
headerMap | fetch, post, put | A Map of Headers to be passed in the HTML request. |
rawData | post, put | The raw data or json passed with the request. The format and contents of the data is determined by the URL/API being called, not by dotCMS. |
paramMap | post, put | If supplied, this is a Map of form parameters that will sent with the post or put |
Timeout Parameter
The timeout specifies the total time to wait to both make the connection and perform the read. The Import Tool will automatically kill the connection after the specified time, regardless of what state the import is in (even if it is in the middle of a read). Therefore the timeout should be greater than the time it takes for both the connection and read of the JSON resource.
Note:
- The timeout is specified in milliseconds (not seconds).
- If no timeout value is specified, the default timeout is used.
- The default value may be changed by modifying the
URL_CONNECTION_TIMEOUT
property in the dotmarketing-config.properties file. - Important: It is strongly recommended that all changes to the dotmarketing-config.properties file be made through a properties extension file.
- The default value may be changed by modifying the
Headers Parameter
You may pass headers to be sent with the JSON request. This may be necessary, for example, to perform a fetch from a server that restricts access to certain types of clients (such as regular browsers) by checking the User Agent header.
The headers must be in a map, with each header to be passed consisting of both a header label and a header value.
The following code demonstrates how to generate and use the headers map:
#set($headers = {})
#set($dummy = $headers.put("User-Agent", "Mozilla/5.0"))
#set($results = $json.fetch("http://www.omdbapi.com/?t=Battlestar%20Galactica&y=&plot=short&r=json", $headers))
Form Parameters
You may pass form params to be sent with the request. This may be necessary, for example, to POST data to a server.
The form params must be in a map, with each param to be passed consisting of both a string key and a string value.
The following code demonstrates how to generate and use the param map:
#set($headers = {})
#set($params = {})
#set($dummy = $params.put("firstName", "Will"))
#set($dummy = $params.put("lastName", "Ezell"))
#set($results = $json.put("https://demo.dotcms.com/testing",$headers, $params))
Raw Data
You can send raw json data as well - just pass the json in as a fully formed json string
#set($headers = {})
#set($data = "{'firstName':'Will','lastName','Ezell'}")
#set($results = $json.put("https://demo.dotcms.com/testing", $headers, $data))
Example
The following example fetches a JSON object from an external site and displays different properties of the object.
#set($myjson = $json.fetch("http://www.omdbapi.com/?t=Battlestar%20Galactica&y=&plot=short&r=json"))
<p>The title is: $myjson.Title</p>
<p>The year is: $myjson.Year</p>
<p>The rating is: $myjson.Rated</p>
The results of the code above are:
The title is: Battlestar Galactica
The year is: 2004–2009
The rating is: TV-14
Parsing Arrays and Loops
If the JSON property is an array, you can loop over the array using Velocity's #foreach
directive:
#set($myjson = $json.fetch("http://imdbapi.poromenos.org/json/?name=battlestar%"))
#foreach($show in $myjson.shows)
<p>$show.name ($show.year)</p>
#end
The code above lists every version of the Battlestar Galactica television show, including the year it was broadcast.
Limiting JSON Results
To limit the number of results returned in the JSON query, use Velocity $foreach.count
and #break
directives to exit the #foreach
loop as soon as the count is met. The following example limits the previous code to only display 3 results:
#set($myjson = $json.fetch("http://imdbapi.poromenos.org/json/?name=battlestar%"))
#foreach($show in $myjson.shows)
#if( $foreach.count > 3 )
#break
#end
<p>$show.name ($show.year)</p>
#end
Handling Alternate JSON Array Formats
The JSONTool handles JSON arrays in certain specific formats automatically; however JSON arrays can be formatted in different ways, and for some JSON array formats, additional processing may be required to be able to iterate through the array results. If you can not iterate over a JSON array using the standard loop syntax (above), you must first read the JSON array and create a JSON object from it, and then iterate over the JSON object, as in the following example:
#set($jsonData = $import.read("http://imdbapi.poromenos.org/json/?name=battlestar%"))
#set($jsonData = "{data:${jsonData}}")
#set($jsonObject = $json.generate($jsonData))
#foreach($show in $jsonObject.data)
<p>$show.name ($show.year)</p>
#end
Passing Data to a Post
The format and contents of the Data object passed with the post()
method must match the requirements of the API/URL the post is being sent to. The Data object is created in the same way as the Headers object used in both the fetch()
and post()
methods.
#set($data = $contents.getEmptyMap())
#set($dummy = $data.put("Query", "+contentType:news"))
#set($results = $json.post($url, $data))
Create JSON Objects ($json.generate)
The $json.generate()
method generates JSONObjects from Java Objects,Strings or Maps. The JSON object values can be populated with static or dynamic content from your dotCMS instance.
Usage
$json.generate( Object )
Example
The following code generates a JSONObject from a java.collections.Map, demonstrating how to serve a JSON object from your site.
#set($mymap = $contents.getEmptyMap())
#set($dummy = $mymap.put("one","value 1"))
#set($dummy = $mymap.put("two","value 2"))
#set($dummy = $mymap.put("three","value 3"))
#set($myjson = $json.generate($mymap))
<p>$myjson</p>
<p>The value of key "one" is: $myjson.one</p>
The code generates the following results:
{"two":"value 2","one":"value 1","three":"value 3"}
The value of key "one" is: value 1
Parse Internal JSON Files
The following example code will allow you to parse JSON files that are stored on the dotCMS file system. Since the #include
directive is only aware of files on the system and not files in dotCMS's virtual file system, use $webapi.getAssetPath
to convert the virtual path to the system path, and then $json.generate
to convert the file contents into a variable.
#define($file)#include($webapi.getAssetPath("/your/dotCMS/file/path/data.json", $host.identifier))#end
#set($data = $json.generate($file.toString()))
-Thanks to Leonardo Bell for providing this example :-)
Toolbox.xml Configuration
The following example shows how the JSONTool is mapped in the toolbox-xml file:
<tool>
<key>json</key>
<scope>application</scope>
<class>com.dotcms.rendering.velocity.viewtools.JSONTool</class>
</tool>