File Operations

Sietch's file operations are built around a sophisticated chunking and encryption system. Every file you add is automatically chunked, encrypted, and stored with deduplication support. Understanding these operations is crucial for effective vault management.

How Sietch Handles Files

The Chunking Process

When you add a file to Sietch:

1. Chunking: The file is divided into chunks (default 4MB) using your vault's chunking strategy
2. Hashing: Each chunk gets a content hash for deduplication and integrity verification
3. Encryption: Chunks are encrypted using your vault's encryption settings
4. Storage: Encrypted chunks are stored in .sietch/chunks/ with cryptographic names
5. Manifest: A YAML manifest tracks file metadata, chunk references, and tags

Vault Structure

your-vault/
├── .sietch/
│   ├── chunks/           # Encrypted chunk storage
│   ├── manifests/        # File metadata and chunk references
│   ├── keys/            # Encryption keys
│   └── sync/            # Sync-related keys
└── vault.yaml           # Vault configuration

Adding Files: sietch add

Add individual files to your vault with optional organization and tagging.

Usage

sietch add <source_path> <destination_path> [flags]

Options

Key Behaviors

• Individual files only: Sietch add works with single files, not directories
• Automatic encryption: Files are immediately chunked and encrypted
• Deduplication: Identical content chunks are automatically deduplicated
• Path organization: Destination paths create logical organization within the vault

Examples

# Add a file to vault root
sietch add document.txt documents/

# Add with organizational path
sietch add research-data.csv research/datasets/

# Add file with tags for organization
sietch add report.pdf reports/ --tags "urgent,quarterly,finance"

# Add sensitive document with force flag
sietch add secrets.txt secure/ --force

# Add multiple individual files (no directory support)
sietch add file1.txt archive/
sietch add file2.txt archive/
sietch add file3.txt archive/

What Happens During Add

When you run sietch add document.txt documents/:

1. File Analysis: Sietch reads the file and calculates its size and modification time
2. Chunking: The file is divided according to your chunking strategy
3. Content Hashing: Each chunk receives a SHA256 (or Blake3) hash
4. Deduplication Check: Sietch checks if identical chunks already exist
5. Encryption: New chunks are encrypted with your vault's encryption key
6. Storage: Encrypted chunks are stored with cryptographic filenames
7. Manifest Creation: A YAML manifest records file metadata and chunk references

Example output:

File Metadata:
File: document.txt
Source: document.txt
Size: 1.2 KB (1,234 bytes)
Modified: 2025-09-23T10:30:15+00:00
Destination: documents/

Beginning chunking process...
Chunk 1: 1.2 KB bytes, hash: a1b2c3d4 (encrypted)
Total chunks processed: 1
Total bytes processed: 1.2 KB

✓ File added to vault: document.txt
✓ 1 chunks stored in vault
✓ Manifest written to .sietch/manifests/document.txt.yaml

Retrieving Files: sietch get

Extract and decrypt files from your vault back to the filesystem.

Usage

sietch get <file_path> <destination_path> [flags]

Options

Important Notes

sietch get documents/file.txt ./retrieved/
base64 -d ./retrieved/file.txt > actual-file.txt

Examples

# Get a file to current directory
sietch get documents/report.pdf ./

# Get file with specific name
sietch get vault/data.csv ~/Downloads/financial-data.csv

# Force overwrite existing file
sietch get reports/quarterly.pdf ~/Documents/ --force

# Retrieve for recovery (raw chunks)
sietch get corrupted-file.dat ./recovery/ --skip-decryption

# Standard workflow to get actual file content
sietch get documents/report.txt ./temp/
base64 -d ./temp/report.txt > ./actual-report.txt
rm ./temp/report.txt

What Happens During Get

When you retrieve a file:

1. Manifest Reading: Sietch reads the file's manifest to get chunk information
2. Chunk Assembly: All chunks are located and read from storage
3. Decryption: Each chunk is decrypted using your vault keys
4. Reassembly: Chunks are reassembled in correct order
5. Base64 Encoding: Content is encoded for safe output
6. File Creation: The encoded file is written to destination

Example output:

Retrieving documents/report.pdf from vault
Reassembling file from 3 chunks
Processing chunk 1/3
Processing chunk 2/3
Processing chunk 3/3

File retrieved successfully: ./report.pdf
Size: 2.1 MB
Tags: [important, quarterly]

File successfully decrypted

Listing Files: sietch ls

View and organize files in your vault with powerful sorting and filtering options.

Usage

sietch ls [path] [flags]

Options

Display Formats

# Basic file listing
sietch ls
# Output:
# documents/report.pdf
# research/data.csv
# vault/secrets.txt

# Long format with metadata
sietch ls --long
# Output:
# SIZE   MODIFIED             CHUNKS  PATH
# 2.1 MB 2025-09-23 10:30:15  3       documents/report.pdf
# 856 KB 2025-09-23 09:15:22  1       research/data.csv
# 1.2 KB 2025-09-23 11:45:33  1       vault/secrets.txt

# Show file tags for organization
sietch ls --tags
# Output:
# documents/report.pdf [important, quarterly]
# research/data.csv [research, economics]
# vault/secrets.txt

Sorting and Filtering

# Sort by file size (largest first)
sietch ls --sort=size

# Sort by modification time (newest first)
sietch ls --sort=time

# Sort alphabetically by name
sietch ls --sort=name

# List specific directory
sietch ls documents/

# List research files
sietch ls research/

# Combined with formatting
sietch ls research/ --long --tags

Understanding the Output

The long format provides crucial information:

• SIZE: File size (original, before chunking/encryption)
• MODIFIED: Last modification time from original file
• CHUNKS: Number of chunks the file is split into
• PATH: Virtual path within the vault
• TAGS: User-defined tags for organization

Deleting Files: sietch delete

Remove files from your vault with automatic chunk cleanup.

Usage

sietch delete <file_path> [flags]

Options

Deletion Process

• Confirmation Required: By default, Sietch asks for confirmation
• Manifest Removal: The file's manifest is deleted immediately
• Chunk Cleanup: Orphaned chunks are automatically removed
• Deduplication Aware: Shared chunks used by other files are preserved

Examples

# Delete with confirmation
sietch delete documents/report.pdf
# Prompt: Are you sure you want to delete 'documents/report.pdf'? (y/N):

# Force delete without confirmation
sietch delete old-files/backup.zip --force

# Delete manifest but keep chunks (for recovery scenarios)
sietch delete corrupted-file.dat --keep-chunks

# Standard deletion with cleanup
sietch delete temporary/cache.tmp --force

What Happens During Delete

1. Confirmation: User confirms deletion (unless --force used)
2. Manifest Removal: File manifest is deleted from .sietch/manifests/
3. Chunk Analysis: Sietch identifies which chunks were used by the file
4. Reference Check: Checks if chunks are used by other files
5. Cleanup: Removes orphaned chunks not referenced elsewhere
6. Deduplication Update: Updates deduplication indices

Example output:

Warning: Failed to delete chunk a1b2c3d4...: chunk still referenced by other files
✓ Successfully deleted 'documents/report.pdf' from vault
✓ Cleaned up 2 orphaned chunks
✓ Freed 1.8 MB of storage space

Advanced File Management

Understanding Manifests

Each file in your vault has a YAML manifest that stores metadata:

file: research.txt
size: 52
mtime: "2025-09-23T13:05:35+05:30"
chunks:
  - hash: 9fa35c8f680d121b...
    encrypted_hash: 06a4261c9d152baed...
    size: 52
    encrypted_size: 200
    index: 0
destination: research/
tags:
  - important
  - survival
added_at: 2025-09-23T07:35:50.635546248Z

Deduplication in Action

Sietch automatically deduplicates identical content:

# Add two files with identical content
echo "Same content" > file1.txt
echo "Same content" > file2.txt

sietch add file1.txt test/
sietch add file2.txt test/

# Check deduplication statistics
sietch dedup stats

# Both files share the same chunk
sietch ls --long
# SIZE  MODIFIED             CHUNKS  PATH
# 13 B  2025-09-23 10:30:15  1       test/file1.txt
# 13 B  2025-09-23 10:30:16  1       test/file2.txt

# But only one chunk is stored physically
find .sietch/chunks/ -type f | wc -l
# Output: 1

File Tags and Organization

Tags provide powerful organization capabilities:

# Organizational tags
sietch add report.pdf reports/ --tags "quarterly,finance,2024"

# Priority tags
sietch add urgent-memo.txt alerts/ --tags "urgent,action-required"

# Project tags
sietch add research-data.csv projects/ --tags "project-alpha,research,confidential"

# View all tags
sietch ls --tags

# Filter by examining tags in long format
sietch ls --long --tags | grep "urgent"

Best Practices

File Organization

• Use descriptive paths: documents/contracts/2024/ instead of docs/
• Tag consistently: Develop a tagging strategy for your use case
• Group related files: Keep project files in dedicated paths
• Regular cleanup: Use sietch delete to remove outdated files

Security Considerations

Performance Optimization

• Chunk size matters: Larger chunks for big files, smaller for deduplication
• Monitor deduplication: Use sietch dedup stats to track efficiency
• Regular garbage collection: Run sietch dedup gc to clean orphaned chunks
• Tag organization: Use consistent tagging for easier file management

Troubleshooting

Common Issues

File won't add: Ensure the source file exists and you have read permissions.

Can't retrieve file: Check that the file path exists in the vault with sietch ls.

Base64 output: Remember to decode retrieved files with base64 -d.

Deletion warnings: Warnings about chunk cleanup are normal and indicate successful deduplication.

Recovery Scenarios

# Retrieve raw chunks for manual recovery
sietch get corrupted-file.dat ./recovery/ --skip-decryption

# Check chunk integrity
sietch dedup stats

# If manifest is corrupted but chunks exist
# Delete the corrupt manifest entry
sietch delete broken-file.txt --keep-chunks

# Re-add the original file
sietch add original-file.txt vault/