Latent Semantic Indexing

Conceptual analytics in Relativity uses latent semantic indexing. Rather than referring to a master dictionary, mathematics is used to identify concepts in documents. The approach is based on the co-occurence of terms used in the documents that an analytics index is based on. The content of the workspace determines how documents are related to one another, and which concepts are present in those documents.

Latent Semantic Indexing is the mathematical basis of a conceptual analytics index which is based on a set of documents. Conceptual analytics can also use a classification index which is based on coded examples. This type of index uses a Support Vector Machine learning.

LSI as used in Relativity has several key characteristics:

1. It is language agnostic. Latent Semantic Indexing will discover correlations between documents and concepts inside them no matter which language the documents are in.

2. The training data source used for a LSI conceptual analytics index can be the same as the full set of documents to be analyzed, or be a subset of those documents.

3. It generates a multi-dimensional concept space. This is a mathematical model. The documents which are indexed are mapped on to the concept space - which is a spatial index. Documents which are closer together in the concept space, will be more conceptually similar.

4. The similarity between two documents or two words is measured by rank value, also referred to as a coherence score. The higher score, the higher the degree of similarity.

5. The coherence score is not a percentage of shared content, but a measurement of distance.

6. Analytics indexes are always in memory.

Latent Semantic Indexing processing technique is based on the following:

1. The principle that documents which are conceptually similar will use similar sections of text.

2. Use of a matrix, or chart, in which each word is listed on a separate row, and each document in a separate column.

3. Singular value decompositon is used to decrease the number of rows while maintaining the similarity relationships between the columns.

4. The degree of similarity between two documents is calculated by finding the cosine of the angle between the two vectors formed by two columns. A value close to 1 will indicate they are very similar. A value closer to 0 will show that they are more dissimilar.

Getting started with Relativity Analytics

If you are using Relativity analytics to get an overview of the contents of a data set, keep in mind that these are the basic steps which should be performed at the start:

• Repeated content identification should be run to find disclaimers and confidentiality footers which occur frequently in the full set.

• The repeated content filters should be linked to an analytics index.

• Assemble a set of conceptually rich documents, which exclude computer generated files, to use for the index in a saved search.

• Set the index to optimize the training set in order to automatically remove conceptually irrelevant documents.

• Set the index to automatically remove email signatures and footers.

• Populate and build the index.

• Perform clustering and categorization to find the kind of documents present in the full document set.

Changing the Location of an Analytics Index

An admin can change the location of an analytics index or a structured analytics set from their original location on a server. The indexes and structured analytics sets must be in the same location, and only one location can be used for both. While the move occurs, analytics operations cannot be run.

On the server, the admin has to log into Analytics REST (Representational State Transfer). REST is an API which can be used to create applications, and to access resources using template URLs. Template URLs provide a means to re-write HTML content.

Before proceeding with the relocation, the Content Analyst (CAAT) Relativity Analytics Engine Window service must be stopped.

The data may or may not be stored in a single date directory mode. The admin will be able to determine which mode it is in using REST. If it is in single data directory mode the device will be designated single-data-directory.

When data is in single data directory mode, the indexes and sets are copied from the original location to the new location - including any empty folders. The complete structure must be maintained. The new location should have a folder named .ufx; a file named '.initialized', and folders for prior indexes and sets:

In the C:\CAAT\etc directory, a file named 'single-data-dir.properties' will have to be edited so that the new index location is referenced.

If it is not in single data directory mode, the full contents, with subfolders, of these locations on the server:

• C:\CAAT\data-default

• C:\CAAT\indexes;

• C:\CAAT\staging)

. . . must be copied to the new location. A number of steps must be taken but in general the link at C:\CAAT\device-links is deleted, and a new index location is designated.