Skip to main content

Jobs Preview

Dremio Arctic enables you to schedule optimization jobs to help you manage the accumulation of the data files that occurs through DML operations. This API allows you to retrieve job status and run a one-off job.

Jobs Object
{
  "type": "OPTIMIZE",
  "catalogId": "5d138bd7-6513-46d7-b9cb-d236e195b34a",
  "id": "8c03f8c8-2c21-49c2-aa5a-dfaed20f5f42",
  "state": "COMPLETED",
  "username": "dremio_user@company.com",
  "startedAt": "2023-02-23T20:25:44Z",
  "endedAt": "2023-02-23T20:28:50Z",
  "engineSize": "XX_SMALL_V1",
  "config": {
    "tableId": "zip_lookup",
    "reference": "main",
    "targetFileSize": "256 MB",
    "minFileSize": "192 MB",
    "maxFileSize": "460 MB",
    "minFiles": 5
  },
  "metrics": {
    "rewrittenDataFiles": 6,
    "newDataFiles": 2,
    "rewrittenDeleteFiles": 5
  }
}

Job Attributes​

{{< api-section file="data/api/arctic-jobs.json" data="jobs-attributes" >}}

config Object Attributes​

{{< api-section file="data/api/arctic-jobs.json" data="config-object" >}}

metrics Object Attributes​

{{< api-section file="data/api/arctic-jobs.json" data="metrics-object" >}}

Creating a Job​

Create a job for the specified Arctic catalog.

Method and URL
POST /v0/arctic/catalogs/{catalogId}/jobs

Parameters​

{{< api-section file="data/api/arctic-jobs.json" data="addingJob" >}}

config Object Parameters​

{{< api-section file="data/api/arctic-jobs.json" data="addingJobConfigObjectParameters" >}}


Example Request
curl -X POST 'https://api.dremio.cloud/v0/arctic/catalogs/{catalogId}/jobs' \
--header 'Authorization: Bearer <personal access token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "type": "OPTIMIZE",
  "config": {
    "tableId": "zip_lookup",
    "reference": "main",
    "targetFileSize": "256 MB",
    "minFileSize": "192 MB",
    "maxFileSize": "460 MB",
    "minFiles": 5
  }
}'
Example Response
{
  "type": "OPTIMIZE",
  "catalogId": "5d138bd7-6513-46d7-b9cb-d236e195b34a",
  "id": "8c03f8c8-2c21-49c2-aa5a-dfaed20f5f42",
  "state": "SETUP",
  "username": "dremio_user@company.com",
  "engineSize": "XX_SMALL_V1",
  "config": {
    "tableId": "zip_lookup",
    "reference": "main",
    "targetFileSize": "256 MB",
    "minFileSize": "192 MB",
    "maxFileSize": "460 MB",
    "minFiles": 5
  }
}

Response Status Codes​

{{< api-responses file="data/api/arctic-jobs.json" data="addingJob-Responses" >}}

Listing All Jobs​

Returns a listing of all jobs for the specified Arctic catalog.

Method and URL
GET /v0/arctic/catalogs/{catalogId}/jobs

Parameters​

{{< api-section file="data/api/arctic-jobs.json" data="listingAllJobs" >}}

filter Object Attributes​

{{< api-section file="data/api/arctic-jobs.json" data="jobs-filter-object" >}}


Example Request
curl -X GET 'https://api.dremio.cloud/v0/arctic/catalogs/{catalogId}/jobs' \
--header 'Authorization: Bearer <personal access token>' \
--header 'Content-Type: application/json'

In the response for a request to retrieve all Arctic optimization jobs, the job objects are wrapped with a data array. Each object in the data array represents one job.

Each object in the response also includes two attributes that are not part of the individual job object:

  • errorMessage (String): For unsuccessful jobs, a description of the problem.
  • scheduleId (String): Unique identifier for the schedule that created the job, if applicable. Empty for jobs created via API or the "Optimize Once" option in the Dremio Cloud application.
Example Response
{
  "data": [
    {
      "type": "OPTIMIZE",
      "catalogId": "5d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "8c03f8c8-2c21-49c2-aa5a-dfaed20f5f42",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-23T20:25:44Z",
      "endedAt": "2023-02-23T20:28:50Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "1907c1f6-5fa7-4cf2-ae9c-4089620f7e28",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-23T19:06:52Z",
      "endedAt": "2023-02-23T19:09:57Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "c0a2648f-3400-4613-8a5e-e4ee71386ca2",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-16T17:18:12Z",
      "endedAt": "2023-02-16T17:20:18Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "7d0ebe6b-908e-42fe-b753-ba4b277a38de",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-16T17:12:29Z",
      "endedAt": "2023-02-16T17:14:34Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "e72dff53-bb7c-4737-8cd1-a788adc209b3",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-16T17:10:47Z",
      "endedAt": "2023-02-16T17:14:52Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "3777cde6-88df-4477-8662-77f7e45e9b69",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-16T17:06:29Z",
      "endedAt": "2023-02-16T17:10:34Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "90522152-efa5-4ed8-8692-7f1fd59b62a0",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-16T16:03:23Z",
      "endedAt": "2023-02-16T16:06:29Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": "55dbb0fa-d7e2-4520-9748-a48d4b96f837"
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "830c7a49-1408-4047-a6df-4ecf684fb918",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-16T13:53:23Z",
      "endedAt": "2023-02-16T14:07:30Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    },
    {
      "type": "OPTIMIZE",
      "catalogId": "6d138bd7-6513-46d7-b9cb-d236e195b34a",
      "id": "12946fe4-267a-4398-808c-4e03a97fb961",
      "state": "COMPLETED",
      "username": "dremio_user@company.com",
      "startedAt": "2023-02-15T23:14:43Z",
      "endedAt": "2023-02-15T23:17:48Z",
      "engineSize": "XX_SMALL_V1",
      "errorMessage": "",
      "scheduleId": ""
    }
  ],
  "nextPageToken": "a"
}

Response Status Codes​

{{< api-responses file="data/api/arctic-jobs.json" data="listingAllJobs-Responses" >}}

Retrieving a Job​

Retrieve the specified job.

Method and URL
GET /v0/arctic/catalogs/{catalogId}/jobs/{id}

Parameters​

{{< api-section file="data/api/arctic-jobs.json" data="retrievingJob" >}}


Example Request
curl -X GET 'https://api.dremio.cloud/v0/arctic/catalogs/{catalogId}/jobs/5d138bd7-6513-46d7-b9cb-d236e195b34a' \
--header 'Authorization: Bearer <personal access token>' \
--header 'Content-Type: application/json'
Example Response
{
  "type": "OPTIMIZE",
  "catalogId": "5d138bd7-6513-46d7-b9cb-d236e195b34a",
  "id": "8c03f8c8-2c21-49c2-aa5a-dfaed20f5f42",
  "state": "COMPLETED",
  "username": "dremio_user@company.com",
  "startedAt": "2023-02-23T20:25:44Z",
  "endedAt": "2023-02-23T20:28:50Z",
  "engineSize": "XX_SMALL_V1",
  "config": {
    "tableId": "zip_lookup",
    "reference": "main",
    "targetFileSize": "256 MB",
    "minFileSize": "192 MB",
    "maxFileSize": "460 MB",
    "minFiles": 5
  },
  "metrics": {
    "rewrittenDataFiles": 6,
    "newDataFiles": 2,
    "rewrittenDeleteFiles": 5
  }
}

Response Status Codes​

{{< api-responses file="data/api/arctic-jobs.json" data="retrievingJob-Responses" >}}