News

Our latest news releases

Using DataGate to pre-process data for AVEVA NET Portal

No Comments 14684 Views0

DataGate is designed for complete flexibility and scalability. Data can be sourced from anywhere, checked for conformance to business requirements, and exported to anywhere.

You may wish to source data from a database, file, file system, document management system, SAP, SharePoint, or any 3rd party system. Likewise, once data has been processed you may want to export to a plethora of data stores. DataGate is designed to adapt to specific requirements.

In this Engineering demonstration video we import a list of 100,000+ Equipment tags, validate asset numbers against a specified format, and check other data fields conform to a variety of rules.

Mapping data from disparate sources is simple with DataGate. Using concepts from the ISO 15926 standard, the format and language of source data becomes a trivial problem. It can easily be mapped to a data model we care about using DataGate‘s inuitive Auto Map and drag and drop interface.

Validation is performed against a business model (Class Library). The model defines data structure, associated business rules, and associations. The model can be managed in small parts and is fully scalable, meaning it can be adapted to reflect differing data requirements through a project lifecycle.

In this video, all valid Tags are exported in a format that can be loaded into the 3rd party AVEVA NET system. Whether you wish to export valid data, invalid data, or all data, the choice is up to you. You can export Valid data to a database, and Invalid data to a file. The choice is yours.

Additionally, a validation report is produced in html format. This can be emailed or deployed to a company’s intranet for easy access.

 

DataHarvester Bench Testing II

No Comments 2862 Views0

Following on from Bench Test I, a further test has been conducted on a huge 1 terabyte document store.

The test proved incredibly successful, scanning a total of 650,720 files and returning 63,663 matches. No issues were encountered and the software proved to be reliable, robust, and efficient. The test ran very smoothly and consistently with no memory issues.

The testing of DataHarvester has proved very successful and sets the standard for eSensible data management software in the near future.

DataHarvester - Bench Testing II

DataHarvester - Long running memory test

DataHarvester Bench Testing

No Comments 2658 Views0

The purpose of DataHarvester is to scan huge quantities of documents to extract textual patterns. This can be an intensive process, and therefore necessitates an efficient algorithm this is conducted in the fastest time possible.

The test set used for bench testing has been a set of 5,317 OCR’d Adobe PDF documents amounting to 10.9 gigabytes in disk space, with file sizes ranging from 10kb to almost 300mb.

The machine used was a 2.2 GHz Sony i7 quad core laptop.

DataHarvester Bench Testing

DataHarvester Bench Testing

Initial bench tests proved very efficient, extracting 1362 matches from all documents in a time of 44 minutes and 42 seconds. This utilised a single-thread process.

Concurrent processor threading has been incorporated to allow configuration of CPU resources. This allows DataHarvester to be configured based on the specification of the host machine, therefore utilising as much processing power as possible.

A further test has been conducted on the same set of documents in a time of 19 minutes 56 seconds, a significant increase on the previous test.

More information of DataHarvester.