> ## Documentation Index
> Fetch the complete documentation index at: https://docs.datanumia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Consumption data

A site's energy consumption is described through three types of data:

* **Index**: a cumulative meter reading, used to calculate a site's energy consumption over a given period and bill it according to the pricing terms of the customer's contract. Indexes are collected for both electricity and gas.
* **Load curve**: a time series of power measurements taken at regular intervals (e.g. every 30 minutes), giving a detailed view of how a site's consumption varies over the course of a day. Load curves are collected for electricity only.
* **Max power**: the highest power level reached by a site over a given period (typically a day). It can be compared with the customer's subscribed power to check whether that capacity is suited to actual usage. Max power is collected for electricity only.

These three data types are collected once a site has been subscribed and its associated contract(s) registered.

## Collection Process

Collection process to add, modify or invalidate (indexes only) energy consumption data is based on a **mass** collection API that manages both **electricity** and **gas** data.

This method enables bulk collection of consumption data of multiple sites by sending JSON files transmitted in a compressed archive in tar.gz format. The archive must contain between 1 and 5 JSON files and its size cannot exceed 2,5 MB. Each JSON file can contain up to 10 000 indexes or 1500 load curves.
It is recommended to send a single type of content per archive.
Once an archive has been sent, it is possible to track its processing through a dedicated service.

The bulk method includes three endpoints:

* `POST /indexes-collect-jobs`: Allows you to submit an archive containing consumption data for processing
* `GET /indexes-collect-jobs/{job-id}`: Allows you to monitor the processing of an archive through its statuses and the detailed report containing the OK/KO operations
* `GET /indexes-collect-jobs`: Returns a list of jobs between two dates, with their statuses and an overview of the OK/KO operations

## Expected information to be sent

### Index collection

The following types of indexes can be collected:

* Cyclic gas: gas indexes collected monthly
* *Smart gas (to be implemented)*: gas index collected daily
* *Cyclic electricity (to be implemented)*: electricity index collected monthly
* Smart electricity: electricity index collected daily

The expected object format varies depending on the type of index, even though most of the fields are common.

| Information | Description | Required |
| - | - | - |
| `Site identifier` | IDs associated to the site that has been generated at the end of the subscription process | Yes |
| `Point of delivery identifier (POD)` | Identification number of the electricity or gas point of delivery | Yes |
| `Nature` | Indicates how the index was read and transmitted | Yes |
| `Meter reading date` | The day of the meter reading | Yes |
| `Generation Timestamp` | Date used to historize index content for a given ts | Yes |
| `Meter change` | Indication of a meter change during the day | No (Specific to cyclic indexes) |
| `Consumption data` | Can vary depending on fluid (gas or electricity) and data type: value(s), volume, units & conversion factors | see details in API documentation |

#### Index natures

Five generic index natures have been defined for both electricity and gas indexes.
In case of concurrent meter readings (indexes with the same ts), the nature will be used to prioritize the index to use for consumption calculation.

Each nature has an associated invalidating nature used to invalidate an incorrect index previously sent, so it is no longer used for consumption calculation (see dedicated section):

| Nature name | Description | Priority | Invalidating nature |
| - | - | - | - |
| `MKT_SMART_DISTRIBUTOR` | Smart index sent by the marketer | 1 | MKT\_SMART\_DISTRIBUTOR\_INVALID |
| `MKT_CYCLIC_DISTRIBUTOR` | Cyclic index sent by the marketer | 2 | MKT\_CYCLIC\_DISTRIBUTOR\_INVALID |
| `MKT_CUSTOMER` | Index entered by the customer on the marketer's portal | 3 | MKT\_CUSTOMER\_INVALID |
| `DTN_CUSTOMER_ACK` | Index entered by the customer on Datanumia's portal and which has been approved by the marketer | 4 | DTN\_CUSTOMER\_ACK\_INVALID |
| `MKT_ESTIMATION` | Index estimated by the distributor or the marketer | 5 | MKT\_ESTIMATION\_INVALID |

#### Example of JSON files

##### Cyclic gas index (with Datanumia internal identifier)

```json theme={null}
{
  "cyclicGasIndexes": [
    {
      "siteId": "site-3000",
      "podId": "pdl-38",
      "nature": "MKT_CYCLIC_DISTRIBUTOR",
      "ts": "2021-07-21T22:00:00Z",
      "generationTs": "2022-05-17T14:28:37Z",
      "indexBeforeChange": false,
      "value": 1234,
      "conversionFactors": {
        "kWh": "10.3"
      },
      "unit": "m3"
    }
  ]
}
```

##### Smart electricity index (with Marketer external identifiers)

```json theme={null}
{
  "smartElecIndexes": [
    {
      "siteCustomerId": "AccordCo123",
      "personCustomerId": "5bp72c86b4",
      "podId": "pdl72c86b42-3",
      "ts": "2023-08-15T22:00:00Z",
      "generationTs": "2023-08-18T13:43:55Z",
      "nature": "MKT_SMART_DISTRIBUTOR",
      "value": {
        "1": 3143, 
        "2": 6534
      }
    }
  ]
}
```

### Load curve collection

Load curves are collected for a given day and are composed of data points whose number depends on the load curve's time step (for instance: if the time step is 30 minutes, a daily load curve is composed of 48 data points).
Each data point can use either the start or the end timestamp of its time step.

| Information | Description | Required |
| - | - | - |
| `Site identifier` | IDs associated to the site that has been generated at the end of the subscription process | Yes |
| `Point of delivery identifier (POD)` | Identification number of the electricity point of delivery | Yes |
| `Meter reading date` | The day of the meter reading | Yes |
| `Generation Timestamp` | Date used to historize content for a given ts | Yes |
| `Step` | Time step between two consecutive points of the load curve, expressed as an ISO 8601 duration (e.g. PT30M = 30 minutes) | Yes |
| `Values` | An array of load curve data points consisting of a timestamp, a value and a reliability score. See details in API documentation | Yes |

#### Example of a JSON file (with Marketer external identifiers)

```json theme={null}
{
  "loadCurves": [
    {
      "siteCustomerId": "AccordCo123",
      "personCustomerId": "5bp72c86b4",
      "podId": "pdl-38",
      "meterReadingDate": "2022-05-16",
      "generationTs": "2022-05-17T14:28:37+02:00",
      "step": "PT30M",
      "values":[
        {
          "beginTs":"2025-05-14T00:30:00+02:00",
          "value":4838,
          "reliability":0
        },
        // OR
        {
          "endTs":"2025-05-15T01:30:00+02:00",
          "value":151,
          "reliability":0
        }
      ]     
    }
  ]
}
```

### maxPower collection

Max powers are collected for a given day.

| Information | Description | Required |
| - | - | - |
| `Site identifier` | IDs associated to the site that has been generated at the end of the subscription process | Yes |
| `Point of delivery identifier (POD)` | Identification number of the electricity point of delivery | Yes |
| `Timestamp (ts)` | Time at which maximum power was reached during the day | Yes |
| `Generation Timestamp` | Date used to historize content for a given ts | Yes |
| `Value` | max power reached for the day | Yes |

#### Example of a JSON file (with Marketer external identifiers)

```json theme={null}
{
    "maxPowers": [
      {
        "siteCustomerId": "AccordCo123",
        "personCustomerId": "5bp72c86b4",
        "podId": "pdl72c86b42-3",
        "ts": "2023-08-15T22:00:00Z",
        "generationTs": "2023-08-18T13:43:55Z",
        "value": 4700
      }
    ]
  }
```

## Energy consumption data modification & invalidation

#### Modification

A marketer can modify an index, a load curve or a max power previously collected by sending it again with a new value using the same ts but a more recent generationTs.

In the specific case of indexes, the marketer can also invalidate previously collected data (see dedicated section).

The data taken into account in the consumption calculation is the one with the most recent generation ts that has not been invalidated.

#### Invalidation (indexes only)

Invalidating natures are used to invalidate an incorrect index, so it is not taken into account for a site's consumption calculation. To invalidate an index, the marketer must resend the same index (ts, value), replacing its nature with the associated invalidating nature.

##### Invalidate an index sent through legacy collection process

Index collected through legacy collection process (batch) used historical natures that have been replaced by the five generic index natures defined as part of this API.

The marketer can invalidate a historical nature using this API by sending the corresponding generic invalidating nature.

Mapping between historical natures and generic natures is the following:

###### Elec indexes

| Historical natures | Generic invalidating nature |
| - | - |
| `LINKY_OK`, `LINKY_INTERPOLE`, `LINKY_SUJET_A_CAUTION` | MKT\_SMART\_DISTRIBUTOR\_INVALID |
| `DISTRIBUTEUR` | MKT\_CYCLIC\_DISTRIBUTOR\_INVALID |
| `RELEVE_CONFIANCE` | MKT\_CUSTOMER\_INVALID |
| `SCORE_RC_ACQUITTE`, `SCORE_ACQUITTE`, `SCORE_RC`, `SCORE`, `SUIVI_CONSO` | DTN\_CUSTOMER\_ACK\_INVALID |
| `ESTI_EDELIA`, `ESTI_ACC` | MKT\_ESTIMATION\_INVALID |

###### Gas indexes

| Historical natures | Generic invalidating nature |
| - | - |
| `MES`, `EST`, `COR`, `INTERPOLATED` | MKT\_SMART\_DISTRIBUTOR\_INVALID |
| `DISTRIBUTEUR` | MKT\_CYCLIC\_DISTRIBUTOR\_INVALID |
| `RELEVE_CONFIANCE` | MKT\_CUSTOMER\_INVALID |
| `SCORE_RC_ACQUITTE`, `SCORE_ACQUITTE`, `SCORE_RC`, `SCORE`, `SUIVI_CONSO` | DTN\_CUSTOMER\_ACK\_INVALID |
| `ESTI_EDELIA`, `ESTI_ACC` | MKT\_ESTIMATION\_INVALID |

## Job processing report status

Job reports provide information on archive processing status at three levels:

* Archive (current and historical)
* Files
* Operations

### Archive state & errors

Job processing state is one of the following:

| State | Description |
| - | - |
| `SUBMITTED` | The archive is received but processing has not started |
| `IN_PROGRESS` | The archive is being processed. |
| `DONE` | Archive processing is completed without errors and a detailed report on each operation is accessible. |
| `ERROR` | Archive processing is completed but has encountered at least one functional or internal error. The details are available in the detailed report. Actions on the customer's side are necessary to correct them and resend the corresponding contracts. |
| `INTERNAL_ERROR` | Archive processing was interrupted due to a series of internal errors that occurred while processing a JSON file. Datanumia requires intervention to correct these errors. |
| `DELETED` | Archive has been deleted (occurs 60 days after processing). Detailed status report is no more available |

In case of error, a code is returned:

| Error code | Error case |
| - | - |
| `INVALID_ARCHIVE_FORMAT` | Unable to read or extract the contents of the archive. |
| `TOO_MANY_FILES` | The archive must contain between 1 and 5 JSON files |
| `INTERNAL_ERROR` | Archive processing was interrupted due to a series of internal errors that occurred while processing a JSON file. Datanumia requires intervention to correct these errors. |

**Example 1: Archive processing is completed without errors**

```json theme={null}
{
  "archive": "indexes.tar.gz",
  "state": {
    "current": "DONE",
    "history": [
        {
            "status": "SUBMITTED",
            "ts": "2024-09-26T14:25:22.552544+02:00"
        },
        {
            "status": "IN_PROGRESS",
            "ts": "2024-09-26T14:25:36.94735+02:00"
        },
        {
            "status": "DONE",
            "ts": "2024-09-26T14:25:37.206544+02:00"
        }
    ]
  }
  "_links": {
  // ...
  }
}
```

**Example 2: Archive processing is completed but has encountered at least one functional or internal error**

```json theme={null}
{
  "archive": "indexes.tar.gz",
  "state": {
    "current": "ERROR",
    "history": [
      //...
    ]
  },
  "report": {
    "error": {
      "code": "INVALID_ARCHIVE_FORMAT",
      "message": "string"
    }
  }
  //...
}

```

If the job state is DONE, ERROR or INTERNAL\_ERROR, the report contains a file object for each JSON file sent in the archive.

### File processing errors

Controls are made on JSON files' content and format before processing. The following errors may occur:

| Error code | Error case |
| - | - |
| `INVALID_FILE_FORMAT` | File must be in JSON format and contain valid content |
| `TOO_MANY_OPERATIONS` | Each JSON file can contain up to 10 000 indexes or 1500 load curves |
| `INTERNAL_ERROR` | File processing is interrupted to prevent disruption to production systems because 15 INTERNAL\_ERROR errors occur within a sliding window of 20 operations. Datanumia intervention is required to resolve this issue. |

**Example**

```json theme={null}
{
  //...
   "files": [
      {
        "file": "test.json",
        "error": {
          "code": "INVALID_FILE_FORMAT",
          "message": "string"
        }
      }
    ]
  //...
}

```

#### Operation status & errors

For each operation in a file, there is a corresponding status list that specifies whether the operation was successful.

The status is either OK, meaning the operation was processed successfully, or KO, with an error code and a message describing the issue:

| Error code | Error case |
| - | - |
| `FIELD_IS_MANDATORY` | A required field is missing |
| `FIELD_IS_INVALID` | A field content do not match the regex specified in the documentation |
| `INCONSISTENT_REQUEST` | A functional requirement related to multiple attributes is not respected |
| `INTERNAL_ERROR` | This error requires investigation on Datanumia's side to restore service and/or correct the anomaly |

**Example 1 - Operation is successfull (OK)**

```json theme={null}
{
  //...
  "files": [ 
    {
      "file": "DQ.json",
      "content": {
        "smartElecIndexesCollectStatus": [ 
          {
            "status": [ 
              {
                "code": "OK"
              } 
            ],
            "personCustomerId": "personId",
            "siteCustomerId": "siteId",
            "podId": "podId",
            "ts": "2024-11-02T00:00:00Z",
            "generationTs": "2025-01-01T13:43:55Z",
            "nature": "MKT_SMART_DISTRIBUTOR_INVALID"
          } 
        ]
      }
    } 
  ]
}
```

**Example 2 - Operation failed (KO)**

```json theme={null}
{
  //...
  "files": [ 
    {
      "file": "indexes.json",
      "content": {
        "cyclicGasIndexesCollectStatus": [ 
          {
            "status": 
            [ 
              {
                "code": "FIELD_IS_INVALID",
                "message": "generationTs format is invalid !"
              }
            ],
            "personCustomerId": "personId",
            "siteCustomerId": "siteId",
            "podId": "podId",
            "ts": "2024-03-01T00:00:00Z"
          }
        ]
      }
    }
  ]
}
```