Ingestion

Introduction to Ingestion

In the context of the FastAPI API, "ingestion" refers to the process of uploading, managing, and retrieving file data. This domain is crucial for applications that need to handle large volumes of data efficiently, enabling users to upload files, access their contents, and perform operations like search and preview. By leveraging these endpoints, developers can build robust data pipelines and content management systems.

Key Concepts and Entities

Files

Files are the primary entities in the ingestion domain. They can be uploaded, deleted, previewed, and searched. Each file is uniquely identified by a source_id, which is used to perform operations on the file.

API Key

The X-API-Key is an optional header parameter used for authentication. While some endpoints may function without it, using an API key is recommended for secure access and to ensure proper authorization.

Common Workflows

Ingesting Files

To upload a file, use the POST /v1/files/ endpoint. This operation allows you to send files to the server for storage and further processing.

Example:

curl -X POST "https://api.example.com/v1/files/" \
-H "X-API-Key: your_api_key" \
-F "file=@/path/to/your/file.txt"

Deleting Files

To remove a file from the server, use the DELETE /v1/files/ endpoint. This operation requires the file's source_id to identify which file to delete.

Example:

curl -X DELETE "https://api.example.com/v1/files/" \
-H "X-API-Key: your_api_key" \
-d '{"source_id": "12345"}'

Retrieving File Content

To access the content of a specific file, use the GET /v1/files/{source_id}/content endpoint. This operation retrieves the file's data based on its source_id.

Example:

curl -X GET "https://api.example.com/v1/files/12345/content" \
-H "X-API-Key: your_api_key"

Previewing Files

To generate a preview of a file, use the POST /v1/files/preview endpoint. This is useful for displaying a summary or a small portion of the file content.

Example:

curl -X POST "https://api.example.com/v1/files/preview" \
-H "X-API-Key: your_api_key" \
-d '{"source_id": "12345"}'

Searching Documents

To search through documents, use the POST /v1/files/search endpoint. This operation allows you to query files based on specific criteria.

Example:

curl -X POST "https://api.example.com/v1/files/search" \
-H "X-API-Key: your_api_key" \
-d '{"search_query": "example search term"}'

Important Considerations

Authentication

While the X-API-Key is optional, it is highly recommended to use it to authenticate requests. This ensures that only authorized users can access or modify files.

Error Handling

Ensure to handle errors gracefully. Common errors include invalid source_id or missing required parameters. Check the response status and message to diagnose issues.

Pagination

For endpoints that may return large datasets, consider implementing pagination to manage the volume of data returned in a single response.

By understanding and utilizing these endpoints effectively, developers can create efficient workflows for managing file data within their applications.