Attention: This article is a follow-on blog of the blog article Persist additional task data with a small DB footprint
Like in the previous article shown it is possible to store a lot of additional data in a generic database table with a small footprint of a table structure. This solution is working fine but in a productive environment with a heavy load and the requirement of fast access to this structured data the shown solution is not 100% comfortable with focus on performance.
Inside of this article you will see to optimize the performance of this small DB footprint solution with the usage of fasterxml and JSON.
Technical reason for inperformance
In the previous implementation was used the marshaller and unmarshaller of Java package javax.xml.bind and these classes are not the fastest implementation when Java object should be transformed into XML and vice-versa.
Requirements
This optimization needs the following external Jars from fasterxml:
These Jars can be downloaded e.g. from the maven repository (https://mvnrepository.com/artifact/com.fasterxml.jackson).
These files should be stored in a central External Library DC and accessed via DC dependencies.
The existing implementation must be extended and changed at different locations:
The performance will be increased drastic with this surgically optimazations inside some lines of code. Especially when you have bulk operation with high volume data. In some standalone tests inside NWDS with 10000 and 100000 items I have identified a delay of runtime between the fasterxml and original implementation of more than 60 seconds.
Like in the previous article shown it is possible to store a lot of additional data in a generic database table with a small footprint of a table structure. This solution is working fine but in a productive environment with a heavy load and the requirement of fast access to this structured data the shown solution is not 100% comfortable with focus on performance.
Inside of this article you will see to optimize the performance of this small DB footprint solution with the usage of fasterxml and JSON.
Technical reason for inperformance
In the previous implementation was used the marshaller and unmarshaller of Java package javax.xml.bind and these classes are not the fastest implementation when Java object should be transformed into XML and vice-versa.
Requirements
This optimization needs the following external Jars from fasterxml:
These Jars can be downloaded e.g. from the maven repository (https://mvnrepository.com/artifact/com.fasterxml.jackson).
These files should be stored in a central External Library DC and accessed via DC dependencies.
Realization
The existing implementation must be extended and changed at different locations:
- The required annotations @JacksonXmlRootElement and @JacksonXmlProperty shoud be added to the model classes like DataPair and DataPairs. With these annotation transform fasterxml the String XML content into Java objects.
package <Vendor>.common.data.model;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlRootElement;
@XmlRootElement(name = "Pair")
@JacksonXmlRootElement(localName = "Pair")
public class DataPair implements Serializable {
private static final long serialVersionUID = 1L;
@JacksonXmlProperty(localName = "Key")
@JsonProperty("Key")
private String key;
@JacksonXmlElementWrapper(useWrapping = false)
@JacksonXmlProperty(localName = "Value")
@JsonProperty("Value")
private List<String> value = new ArrayList<String>();
public DataPair() {
super();
}
public DataPair(String key, List<String> value) {
super();
this.key = key;
this.value.addAll(value);
}
@XmlElement(name = "Key")
public String getKey() {
return key;
}
public void setKey(String key) {
this.key = key;
}
@XmlElement(name = "Value")
public List<String> getValue() {
return value;
}
public void setValue(List<String> value) {
this.value = value;
}
}
package <Vendor>.common.data.model;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlRootElement;
@XmlRootElement(name = "Pairs")
@JacksonXmlRootElement(localName = "Pairs")
public class DataPairs implements Serializable {
private static final long serialVersionUID = 1L;
@JacksonXmlElementWrapper(useWrapping = false)
@JacksonXmlProperty(localName = "Pair")
@JsonProperty("Pair")
private List<DataPair> dataPair;
public DataPairs() {
dataPair = new ArrayList<DataPair>();
}
@XmlElement(name = "Pair")
public List<DataPair> getDataPair() {
return dataPair;
}
public void setDataPair(List<DataPair> dataPair) {
this.dataPair = dataPair;
}
public void add(DataPair dataPair) {
this.dataPair.add(dataPair);
}
public void addAll(List<DataPair> dataPair) {
this.dataPair.addAll(dataPair);
}
public void remove(DataPair dataPair) {
this.dataPair.remove(dataPair);
}
public void removeAll(List<DataPair> dataPair) {
this.dataPair.removeAll(dataPair);
}
}
- Inside of the DTO classes must be created new operations that use fasterxml classes to marshall und unmarshall XML and JSON content.Hint: The XML implementations beside the original code is necessary and will be used for content that already exists in your runtime system than this optimization should have also an effect to the existing data.Hint: Operation toEntity() has now an parameter where the requester can control if the content should be marshalled into a specific format. Be aware that consumer operation should set or forward this format parameter from there requesters, etc.
The following snippet shows only the content of TaskDataDto (, please addapt this template to other Dto implementations):
package <Vendor>.common.data.model;
import java.io.IOException;
import java.io.Serializable;
import java.io.StringReader;
import java.io.StringWriter;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import <Vendor>.common.data.service.TaskDataServiceLocal;
import <Vendor>.common.data.utility.JsonCheckHelper;
import com.sap.tc.logging.Location;
import com.sap.tc.logging.Severity;
import com.sap.tc.logging.SimpleLogger;
public class TaskDataDto implements Serializable {
private static final long serialVersionUID = 1L;
private static Location logger = Location.getLocation(TaskDataDto.class);
private String taskInstanceID;
private DataPairs taskDataPairs;
public TaskDataDto() {
super();
}
public TaskDataDto(TaskDataEntry taskDataEntry) {
super();
if (null != taskDataEntry) {
this.setTaskInstanceID(taskDataEntry.getTaskInstanceID());
this.setTaskDataPairs(taskDataEntry.getContent());
}
}
public String getTaskInstanceID() {
return taskInstanceID;
}
public void setTaskInstanceID(String taskInstanceID) {
this.taskInstanceID = taskInstanceID;
}
public DataPairs getTaskDataPairs() {
return taskDataPairs;
}
public void setTaskDataPairs(DataPairs taskDataPairs) {
this.taskDataPairs = taskDataPairs;
}
private void setTaskDataPairs(String content) {
try {
if (JsonCheckHelper.isValidJSON(content)) {
taskDataPairs = unmarshallJsonContent(content);
} else {
taskDataPairs = unmarshallXMLContent(content);
}
} catch (IOException e) {
taskDataPairs = null;
SimpleLogger.trace(Severity.INFO, logger, "An error occured:", e);
}
}
public TaskDataEntry toEntity(String persistFormat) throws JsonProcessingException {
TaskDataEntry taskDataEntry = new TaskDataEntry();
taskDataEntry.setTaskInstanceID(taskInstanceID);
if (null != persistFormat && persistFormat.matches(TaskDataServiceLocal.PERSIST_FORMAT_JSON)) {
taskDataEntry.setContent(marshallTaskDataPairsIntoJson(taskDataPairs));
} else {
taskDataEntry.setContent(marshallTaskDataPairsIntoXML(taskDataPairs));
}
return taskDataEntry;
}
@Deprecated
public static String marshallTaskDataPairs(DataPairs taskDataPairs) throws JAXBException {
try {
JAXBContext jaxbContext = JAXBContext.newInstance(DataPairs.class);
Marshaller marshaller = jaxbContext.createMarshaller();
StringWriter strWriter = new StringWriter();
marshaller.marshal(taskDataPairs, strWriter);
return strWriter.toString();
} catch (JAXBException e) {
SimpleLogger.traceThrowable(Severity.ERROR, logger, "JAXBException inside marshallTaskDataPairs()", e);
throw e;
}
}
@Deprecated
public static DataPairs unmarshallContent(String content) throws JAXBException {
try {
JAXBContext jaxbContext = JAXBContext.newInstance(DataPairs.class);
Unmarshaller unmarshaller = jaxbContext.createUnmarshaller();
StringReader strReader = new StringReader(content);
return (DataPairs) unmarshaller.unmarshal(strReader);
} catch (JAXBException e) {
SimpleLogger.traceThrowable(Severity.ERROR, logger, "JAXBException inside unmarshallContent()", e);
throw e;
}
}
public static String marshallTaskDataPairsIntoXML(DataPairs taskDataPairs) throws JsonProcessingException {
try {
return new XmlMapper().writeValueAsString(taskDataPairs);
} catch (JsonProcessingException e) {
SimpleLogger.traceThrowable(Severity.ERROR, logger, "JsonProcessingException inside marshallTaskDataPairsIntoXML()", e);
throw e;
}
}
public static DataPairs unmarshallXMLContent(String xmlContent) throws IOException {
try {
return new XmlMapper().readValue(xmlContent, DataPairs.class);
} catch (IOException e) {
SimpleLogger.traceThrowable(Severity.ERROR, logger, "IOException inside unmarshallXMLContent()", e);
throw e;
}
}
public static String marshallTaskDataPairsIntoJson(DataPairs taskDataPairs) throws JsonProcessingException {
try {
return new ObjectMapper().writeValueAsString(taskDataPairs);
} catch (JsonProcessingException e) {
SimpleLogger.traceThrowable(Severity.ERROR, logger, "JsonProcessingException inside marshallTaskDataPairsIntoJson()", e);
throw e;
}
}
public static DataPairs unmarshallJsonContent(String jsonContent) throws IOException {
try {
return new ObjectMapper().readValue(jsonContent, DataPairs.class);
} catch (IOException e) {
SimpleLogger.traceThrowable(Severity.ERROR, logger, "IOException inside unmarshallJsonContent()", e);
throw e;
}
}
}
- Inside of service / stateless EJB implementations some operations must be maintained, like inside TaskDataService:
...
private TaskDataDto createTaskData(TaskDataDto taskDataDto, String persistFormat) throws TaskDataServiceException {
if (taskDataDto == null) {
throw new TaskDataServiceException(DTO_NULL_ERROR);
}
EntityManager em = emf.createEntityManager();
try {
utx.begin();
// merge/persist task data
TaskDataDto newTaskDataDto = new TaskDataDto(em.merge(taskDataDto.toEntity(persistFormat)));
em.flush();
utx.commit();
// update cache
taskDataCache.setCacheEntryByTaskDataDto(newTaskDataDto);
taskDataCache.broadCastCacheUpdate(TaskDataCache.ACTION_UPDATE, newTaskDataDto.getTaskInstanceID());
return newTaskDataDto;
} catch (IllegalStateException e) {
throw new TaskDataServiceException(e);
} catch (IllegalArgumentException e) {
throw new TaskDataServiceException(e);
} catch (TransactionRequiredException e) {
throw new TaskDataServiceException(e);
} catch (JsonProcessingException e) {
throw new TaskDataServiceException(e);
} catch (NotSupportedException e) {
throw new TaskDataServiceException(e);
} catch (SystemException e) {
throw new TaskDataServiceException(e);
} catch (SecurityException e) {
throw new TaskDataServiceException(e);
} catch (RollbackException e) {
throw new TaskDataServiceException(e);
} catch (HeuristicMixedException e) {
throw new TaskDataServiceException(e);
} catch (HeuristicRollbackException e) {
throw new TaskDataServiceException(e);
} finally {
em.close();
}
}
...
@Override
public TaskDataDto createAndUpdateTaskData(TaskDataDto taskDataDto, String persistFormat) throws TaskDataServiceException {
if (taskDataDto == null) {
throw new TaskDataServiceException(DTO_NULL_ERROR);
}
EntityManager em = emf.createEntityManager();
try {
// get task data from DB
TaskDataEntry taskDataEntry = readTaskDataEntryByTaskInstanceID(taskDataDto.getTaskInstanceID());
if (null != taskDataEntry) {
if (null != persistFormat && persistFormat.matches(TaskDataServiceLocal.PERSIST_FORMAT_JSON)) {
taskDataEntry.setContent(TaskDataDto.marshallTaskDataPairsIntoJson(taskDataDto.getTaskDataPairs()));
} else {
taskDataEntry.setContent(TaskDataDto.marshallTaskDataPairsIntoXML(taskDataDto.getTaskDataPairs()));
}
utx.begin();
// merge/persist task data
TaskDataDto newTaskDataDto = new TaskDataDto(em.merge(taskDataEntry));
em.flush();
utx.commit();
// update cache
taskDataCache.setCacheEntryByTaskDataDto(newTaskDataDto);
taskDataCache.broadCastCacheUpdate(TaskDataCache.ACTION_UPDATE, newTaskDataDto.getTaskInstanceID());
return newTaskDataDto;
} else {
// in case the entry doesn't exists create a new one
return createTaskData(taskDataDto, persistFormat);
}
} catch (IllegalStateException e) {
throw new TaskDataServiceException(e);
} catch (IllegalArgumentException e) {
throw new TaskDataServiceException(e);
} catch (TransactionRequiredException e) {
throw new TaskDataServiceException(e);
} catch (JsonProcessingException e) {
throw new TaskDataServiceException(e);
} catch (NotSupportedException e) {
throw new TaskDataServiceException(e);
} catch (SystemException e) {
throw new TaskDataServiceException(e);
} catch (SecurityException e) {
throw new TaskDataServiceException(e);
} catch (RollbackException e) {
throw new TaskDataServiceException(e);
} catch (HeuristicMixedException e) {
throw new TaskDataServiceException(e);
} catch (HeuristicRollbackException e) {
throw new TaskDataServiceException(e);
} finally {
em.close();
}
}
...
- At every operation where the XML/JSON content will be used with a helper class (named JsonCheckHelper) should be check if the content is in XML or JSON format, like here inside readTaskDataKeyListByTaskInstanceID of TaskDataService:
@Override
public List<String> readTaskDataKeyListByTaskInstanceID(String taskInstanceID) throws TaskDataServiceException {
if (taskInstanceID == null) {
throw new TaskDataServiceException(TID_NULL_ERROR);
}
List<String> taskDataKeys = new ArrayList<String>();
// get TaskDataDto from cache
TaskDataDto cachedTaskDto = taskDataCache.getCacheEntryByTaskInstanceId(taskInstanceID);
if (null != cachedTaskDto) {
DataPairs cachedTaskDataPairs = cachedTaskDto.getTaskDataPairs();
Map<String, List<String>> cachedTaskDataMap = taskDataPairToMap(cachedTaskDataPairs.getDataPair());
taskDataKeys.addAll(cachedTaskDataMap.keySet());
return taskDataKeys;
}
try {
// get task data from DB
TaskDataEntry taskDataEntry = readTaskDataEntryByTaskInstanceID(taskInstanceID);
if (null != taskDataEntry) {
DataPairs entryTaskDataPairs = (JsonCheckHelper.isValidJSON(taskDataEntry.getContent())) ? TaskDataDto.unmarshallJsonContent(taskDataEntry.getContent())
: TaskDataDto.unmarshallXMLContent(taskDataEntry.getContent());
Map<String, List<String>> entryMap = taskDataPairToMap(entryTaskDataPairs.getDataPair());
taskDataKeys.addAll(entryMap.keySet());
TaskDataDto newTaskDataDto = new TaskDataDto(taskDataEntry);
// update cache
taskDataCache.setCacheEntryByTaskDataDto(newTaskDataDto);
taskDataCache.broadCastCacheUpdate(TaskDataCache.ACTION_UPDATE, newTaskDataDto.getTaskInstanceID());
} else {
// in case the entry doesn't exists do nothing
}
} catch (IllegalStateException e) {
SimpleLogger.trace(Severity.INFO, logger, STANDARD_ERROR, e);
} catch (IllegalArgumentException e) {
SimpleLogger.trace(Severity.INFO, logger, STANDARD_ERROR, e);
} catch (TransactionRequiredException e) {
SimpleLogger.trace(Severity.INFO, logger, STANDARD_ERROR, e);
} catch (IOException e) {
SimpleLogger.trace(Severity.INFO, logger, STANDARD_ERROR, e);
}
return taskDataKeys;
}
Content of JsonCheckHelper:
package com.lidl.p2p.common.data.utility;
import java.io.IOException;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
public class JsonCheckHelper {
public static boolean isValidJSON(String toBeChecked) throws IOException {
try {
ObjectMapper objectMapper = new ObjectMapper();
// objectMapper =
// objectMapper.enable(DeserializationFeature.FAIL_ON_READING_DUP_TREE_KEY);
objectMapper.readTree(toBeChecked);
return true;
} catch (JsonProcessingException e) {
return false;
}
}
}
- Build and deploy the optimation to the runmtime system
Conclusion
The performance will be increased drastic with this surgically optimazations inside some lines of code. Especially when you have bulk operation with high volume data. In some standalone tests inside NWDS with 10000 and 100000 items I have identified a delay of runtime between the fasterxml and original implementation of more than 60 seconds.
- SAP Managed Tags:
