DeepDoc Client Action

The DeepDoc Client Action interfaces with the JIVAS DeepDoc service for document processing, chunking, metadata extraction, document handling, serving, and integrated management. Users can upload documents or URLs in batches across multiple formats, including PDF, DOCX, Excel, PPT, and TXT. Processing occurs asynchronously, culminating in ingestion and indexing within your configured vector store.

Document Handling & Management

The DeepDoc Client Action now provides comprehensive document handling and management capabilities through built-in file serving integrated with your configured vector store:

• File Serving and Integration: Uploaded documents can be served directly and are linked seamlessly to stored content within the vector store. When deleting a processed document through this interface, all associated references and indexed vector content will also be automatically removed, maintaining data integrity.

• Document Management Interface: Users have access to an intuitive management interface to easily view a comprehensive list of all ingested documents. Documents may be quickly added or removed, thereby making repository management straightforward and efficient.

• Duplicate Prevention Safeguards: Duplicate processing of documents is automatically prevented. The system verifies and ensures that each uploaded document is unique within the repository and vector store, eliminating redundancy and potential confusion.

This package is defined as a singleton action, requiring the Jivas library (version 2.0.0 or higher) and a properly configured vector_store_action. It also requires the Jivas-modified DeepDoc service.

Package Information

• Name: jivas/deepdoc_client_action
• Author: V75 Inc.
• Architype: DeepDocStoreAction

Meta Information

• Title: DeepDoc Client Action
• Group: core
• Type: action

Configuration

• Singleton: true

Dependencies

• Jivas: ^2.0.0

---

API Usage

The DeepDoc Client Action can be accessed via an API endpoint to perform various operations using different walkers. The standard endpoint for interfacing with operations is /action/walker. Below are the details and examples for each walker that can be called:

Endpoint

POST /action/walker

Required Parameters

• agent_id: The ID of the agent
• module_root: For this action, the module root will be 'actions.jivas.deepdoc_client_action'
• walker: The name of the walker (e.g., list_documents, add_documents, delete_documents)
• args: The arguments passed to the respective walker
• files: A list of files for upload, supplied as a list of dictionaries, each containing name, content, and type.

Walkers and Payload Examples

1. Add Documents

This walker processes documents and ingests them into the vector store.

Payload Example:

{
  "agent_id": "12345",
  "module_root": "actions.jivas.deepdoc_client_action",
  "walker": "add_documents",
  "args": {
    "urls": ["http://example.com/document.pdf"],
    "metadatas": [{"author": "John Doe"}],
    "from_page": 0,
    "to_page": 100000,
    "lang": "english"
  },
  "files": [
    {
      "name": "document1.pdf",
      "content": "byte data",
      "type": "application/pdf"
    }
  ]
}

2. List Documents

This walker lists all documents processed by the deepdoc service.

Payload Example:

{
  "agent_id": "12345",
  "module_root": "actions.jivas.deepdoc_client_action",
  "walker": "list_documents",
  "args": {}
}

3. Delete Documents

This walker removes documents from the vector store and deletes local file system entries.

Payload Example:

{
  "agent_id": "12345",
  "module_root": "actions.jivas.deepdoc_client_action",
  "walker": "delete_documents",
  "args": {
    "documents": [
      {
        "job_id": "67890",
        "filename": "document1.pdf"
      }
    ]
  }
}

---

How to Configure

This action supports configuration either via environment variables or directly from within the action application settings.

Configuration Options

Variable Name	Description	Default Value	Required
DEEPDOC_API_URL	API endpoint URL of your DeepDoc service	http://localhost:8001	Yes
DEEPDOC_API_KEY	Your DeepDoc API authentication token	api-key (replace with secure value)	Yes
JIVAS_BASE_URL	Base URL for your Jivas instance; required for the deepdoc callback to function	http://localhost:8000	Yes
VECTOR_STORE_ACTION	Action used for storing vector data	TypesenseVectorStoreAction	Yes

Option 1: Using Environment Variables

You can configure these values as environment variables so they are accessible to your runtime environment.

Linux/MacOS (.bashrc or shell configuration):

export DEEPDOC_API_URL="https://your-custom-deepdoc-url.com"
export DEEPDOC_API_KEY="your-secure-api-key"
export JIVAS_BASE_URL="https://your-jivas-base-url.com"

Docker (docker-compose.yml):

environment:
  DEEPDOC_API_URL: "https://your-custom-deepdoc-url.com"
  DEEPDOC_API_KEY: "your-secure-api-key"
  JIVAS_BASE_URL: "https://your-jivas-base-url.com"

Option 2: Using the Action App’s Config

Alternatively, you can configure these variables within the action app's settings interface in the JIVAS manager. This is useful if you prefer not to use environment variables or want easy adjustments through the GUI.

From within the action's configuration page:

1. Navigate to your JIVAS Manager Dashboard.
2. Find and select DeepDoc Client Action under Actions.
3. Enter the required configuration values (such as api_url, api_key, and vector_store_action) into their respective configuration fields.
4. Save the configuration once complete.

Note: Configuration values set directly within the action app override those provided via environment variables.

After adjusting settings, restart your service or action to apply your changes.

---

Required Configuration for Local File Serving

If operating locally, you must ensure proper configuration of either local or Amazon S3 file serving to enable this functionality. Configure the following variables in your .env file:

# Set Environment
JIVAS_ENVIRONMENT=development

# JIVAS File Serving Config (Local)
JIVAS_FILE_INTERFACE="local"
JIVAS_FILES_ROOT_PATH=".files"
JIVAS_FILES_URL="http://127.0.0.1:9000/files"

# Alternative: Amazon S3 Configuration
JIVAS_FILE_INTERFACE="s3"
JIVAS_S3_ENDPOINT="your-s3-endpoint"            # optional, typically for custom endpoints
JIVAS_S3_ACCESS_KEY_ID="your-access-key-id"
JIVAS_S3_SECRET_ACCESS_KEY="your-secret-access-key"
JIVAS_S3_REGION="us-east-1"
JIVAS_S3_BUCKET_NAME="your-bucket-name"

In addition, running locally requires configuration of MongoDB and a suitable vector store (we strongly recommend Typesense). Include the following configuration in your .env file:

# MongoDB Config
DATABASE_HOST="mongodb://localhost:27017/?replicaSet=my-rs"

# Typesense Vector Store Config
TYPESENSE_HOST="localhost"
TYPESENSE_PORT=8108
TYPESENSE_PROTOCOL="http"
TYPESENSE_API_KEY="abcd"
TYPESENSE_CONNECTION_TIMEOUT_SECONDS=2

---

📖 Best Practices

• Validate your API keys and model parameters thoroughly before deployment.
• Test pipelines extensively in staging environments before promoting to production.

---

🔰 Contributing

• 🐛 Report Issues: Submit reports for bugs identified or feature requests.
• 💡 Submit Pull Requests: Review open PRs and submit your implementations.

Contributing Guidelines

1. Fork the repository using GitHub’s fork button.
2. Clone your fork locally:

   git clone https://github.com/YOUR_USERNAME/deepdoc_client_action
   

3. Create a new descriptive branch:

   git checkout -b feature-xyz
   

4. Implement your changes and ensure testing.
5. Commit your changes clearly describing updates:

   git commit -m 'feat: add XYZ feature'
   

6. Push changes back to your branch:

   git push origin feature-xyz
   

7. Submit a pull request detailing your changes clearly for review.

Contributor Graph

---

🎗 License

This project is licensed under the Apache License 2.0. See the LICENSE file for additional licensing information.

---

0.0.5 → This Version

• Catered for spaces in document filenames which affect the rendering of URLs

All Changes

0.0.1

• Initialized package using jvcli

0.0.2

• Added file serving of uploaded documents and document management which links to vector store content. Deleting a processed document will also remove all of its references from the vector store.

0.0.3

• Updated action to update descriptor export on doc_manifest update
• Updated action to allow custom metadata to be added to each document ingested

0.0.4

• Patched file_path bug in action
• Updated docs to include API docs

0.0.5

• Catered for spaces in document filenames which affect the rendering of URLs