Process Config

Description

A process config json file contains paths to files used by the program. It acts like a linker which links the required input, output and additional files with the program. It consists of the following fields:

#	Field	Description	Example	Mandatory
1	id	Identifier to get fileid from DHMetadata table which is used to retrieve process config file from FileDef table.	M_PC001	true
2	name	Name of the process config file.	BALM_DOMESTIC_GL_AGGR	true
3	file_type	Type of process_config file	CONFIG	true
4	base_input_id	Identifier to get fileid from DHMetadata table which is used to retrieve input file from FileDef table.	M_IN001	true
5	base_input_metadata	Path to input metadata file downloaded form filedef table.	{M_IN001} (this place holder will get replaced by the actual file path like test-bed/M_IN001.json) Note: We can directly provide the actual file path here. [Click here to check Input Metadata]	true
6	additional_input	Contains details to fetch additional input files required by the program.	Click here to check Additional Input	false
7	base_output_id	Identifier to get fileid from DHMetadata table which is used to retrieve output file from FileDef table.	M_OP001	true
8	base_output_metadata	Path to output metadata file downloaded form filedef table.	{M_OP001} (this place holder will get replaced by the actual file path like test-bed/M_OP001.json) Note: We can directly provide the actual file path here. [Click here to check Output Metadata]	true
9	additional_output	Contains details of additional output files generated by the program.	Click here to check Additional Output	false

Example

{
    "id": "M_PC001",
    "name": "BALM_DOMESTIC_GL_AGGR",
    "file_type": "CONFIG",
    "base_input_id": "M_IN001",
    "base_input_metadata": "{M_IN001}",
    "additional_input": [
        {
            "id": "M_IN002",
            "input_metadata": "{M_IN002}",
            "input_type": "Exchange_Rate"
        },{
            "id": "AGGR_Rule_1000",
            "input_metadata": "{getrules(1000,AGGR,{G_BALM_CONN_STRING},{E_2_SH_RESOURCES}/{B_2_SH_RESOURCES}/PC_ID.json)}",
            "input_type": "AGGR_Rule"
        },
    ],
    "base_output_id": "M_OP001",
    "base_output_metadata": "{M_OP001}",
    "additional_output": []
}

In the above example "{M_IN001}", "{M_IN002}", "{M_OP001}" gets replaced by actual path to input metadata, additional input files and output metadata file fetched from DHMetadata table using "id" parameter. getrules is a function call which downloads the rule file to the provided path, in above example it is {E_2_SH_RESOURCES}/{B_2_SH_RESOURCES}/PC_ID.json.

The getrules function takes four parameters: fileid, rule_type, connection_string to database, path to write rule file.

Input Metadata:

A input metadata json file contains path to input file which is the output of another program like PEO and also contains metadata fileds derived from input file.It describes about the input file fields. It consists of the following fields:

#	Field	Description	Example	Mandatory
1	id	Unique identifier for input metadata	M_IN001	true
2	name	Name of the input metadata file	BALM_DOMESTIC_GL_AGGR_IP	true
3	file_type	Type of file	InputMetadata	true
4	input	Path to the input file.	{E_2_SH_CFDATA}/{B_2_SH_CFDATA}/{ddmmyyyy}/PEO_GL.cf	true
5	separator	Input fields separator	"\|"	false
6	fields	Contains metadata derived from the input file.It contains three fileds: id, name, data_type [Click here to check all data-types], input_format [Click here to check all formats]	"id": 1,"name": "as_on_date","data_type": "ShortDate"	false
7	derivations	Depends on the process type. Each process type has it's own derivations type.	""	false
8	validations	For data validation	""	false

Example

{
    "id": "M_IN001",
    "name": "BALM_DOMESTIC_GL_AGGR_IP",
    "file_type": "InputMetadata",
    "input": "{E_2_SH_CFDATA}/{B_2_SH_CFDATA}/{ddmmyyyy}/PEO_GL.cf",
    "fields": [
        {
            "id": 1,
            "name": "as_on_date",
            "data_type": "ShortDate",
            "input_format": "dd-mm-yyyy"
        },
        {
            "id": 2,
            "name": "mat_date",
            "data_type": "ShortDate",
            "input_format": "dd-mm-yyyy"
        },
        {
            "id": 3,
            "name": "curr_code",
            "data_type": "String"
        }
    ],
    "derivations": {
        "fields": []
    }
}

In the above example, data inside the fields parameter represent the metadata derived from the input cf file.

Additional Input:

A additional input json file contains paths to additional files required by the aggregator like exchange rate, overdue_llg file etc. It consists of the following fields:

#	Field	Description	Example	Mandatory
1	id	Identifier for file.	M_IN002	true
2	input_metadata	Path to additional file. This get replaced by the download path mentioned in DHMetada table using it's id.	{M_IN002} (this place holder will get replaced by the actual file path like test-bed/M_IN002.json) Note: We can directly provide the actual file path here.	true
3	join_type	Type of sql join between files.	left	false
4	input_type	Name of additional file.	Support	true

Example

{
    "id": "M_IN002",
    "input_metadata": "{M_IN002}",
    "join_type": "left",
    "input_type": "Support"
}

Output Metadata:

A output metadata json file contains path to output file and required fileds for the aggregator.It consists of the following fields:

#	Field	Description	Example	Mandatory
1	id	Identifier for output metadata file.	M_OP001	true
2	name	Name of the output metadata file.	BALM_DOMESTIC_GL_AGGR_OP	true
3	file_type	Type of file.	OutputMetadata	true
4	input	Path to the output file.	{E_2_SH_SUMMARY}/{B_2_SH_SUMMARY}/{ddmmyyyy}/AGGR_GL	true
5	separator	Input fields separator.	"\|"	false
6	fields	Contains metadata derived from the input file.Contains metadata derived from the input file.It contains three fileds: id, name, data_type	""	false
7	derivations	Depends on the process type. Each process type has it's own derivations type.	""	false
8	validations	For data validation	""	false

Example

{
    "id": "M_OP001",
    "name": "BALM_DOMESTIC_GL_AGGR_OP",
    "file_type": "OutputMetadata",
    "input": "{E_2_SH_SUMMARY}/{B_2_SH_SUMMARY}/{ddmmyyyy}/AGGR_GL",
    "fields": [],
    "derivations": {
      "fields": [
        {
          "name": "aggr",
          "data_type": "None",
          "derivation": {
            "derivation_type": "BALM_AGGR",
            "sub_type": "Balm Maturity Aggr",
            "req_fields": {
              "account_number": "ac_type",
              "concat": "deal_no",
              "cashflows": "cashflows",
              "institution": "curr_code",
              "interest_rate": "int_rate",
              "next_rep_date": "deal_date",
              "exchange_rate": "",
              "npa": ""
            }
          }
        }
      ]
    }
  }

Additional Output:

A additional output json file contains paths to additional files apart from the output file that the program will generate like health-check report etc. It consists of the following fields:

#	Field	Description	Example	Mandatory
1	id	Identifier for file	M_OP290	true
2	output_metadata_path	Path to additional output files	{M_OP290} (this place holder will get replaced by the actual file path like test-bed/M_OP290.json) Note: We can directly provide the actual file path here.	true

Example

{
    "id": "M_OP290",
    "output_metadata_path": "{M_OP290}"
}

Data Types:

Data Types being used to organize and process io metadata:

#	Data_type	Description
1	Amount	Number(24,3), This can contain total of 24 numbers including both before and after decimal point. Maximum number of digits after decimal point is 3.
2	ExchangeRate	Number(19,8), This can contain total of 19 numbers including both before and after decimal point. Maximum number of digits after decimal point is 8.
3	InterestRate	Number(19,8), This can contain total of 19 numbers including both before and after decimal point. Maximum number of digits after decimal point is 8.
4	MaxRatePrecision	Number(19,8), This can contain total of 19 numbers including both before and after decimal point. Maximum number of digits after decimal point is 8.
5	Percentage	Number(19,8), This can contain total of 19 numbers including both before and after decimal point. Maximum number of digits after decimal point is 8.
6	Number	Number(24,3), This can contain total of 24 numbers including both before and after decimal point. Maximum number of digits after decimal point is 3.
7	ShortDate	Supported formats are: dd-mm-yy, dd-mm-yyyy, dd.mm.yyyy, dd/mm/yy, dd/mm/yyyy, ddmmyyyy, mm-dd-yyyy, mm/dd/yyyy, mmddyyyy, yyyy-mm-dd, yyyy/mm/dd, yyyymmdd
8	ShortMonth	Provide the short name of the month. For ex- Sep
9	ShortDateTime	Supported formats are: dd-mm-yy hh:mm:ss, dd-mm-yyyy hh:mm:ss, dd.mm.yyyy hh:mm:ss, dd/mm/yy hh:mm:ss, dd/mm/yyyy hh:mm:ss, ddmmyyyy hh:mm:ss, mm-dd-yyyy hh:mm:ss, mm/dd/yyyy hh:mm:ss, mmddyyyy hh:mm:ss, yyyy-mm-dd hh:mm:ss, yyyy/mm/dd hh:mm:ss, yyyymmdd hh:mm:ss
10	LongDateTime	Supported formats are: dd-mmm-yy hh:mm:ss, dd-mmm-yyyy hh:mm:ss
11	LongDate	Supported formats are: dd-mmm-yy, dd-mmm-yyyy
12	LongMonth	Give long name of the month. For eg - September
13	Char	Single character. For eg - 'Y'
14	String(1024)	String of 1024 character
15	String(128)	String of 128 character
16	String(16)	String of 16 character
17	String(2048)	String of 2048 character
18	String(3)	String of 3 character
19	String(32)	String of 32 character
20	String(512)	String of 512 character
21	String(64)	String of 64 character
22	String(8)	String of 8 character
23	String(max)	String of 2048 character

Input Format:

This field is used to derive the date_format:

dd-mm-yyyy
mm-dd-yyyy
yyyy-mm-dd
dd/mm/yyyy
mm/dd/yyyy
yyyy/mm/dd
dd.mm.yyyy
mm-dd-yyyy
yyyy-mm-dd
ddmmyyyy
mmddyyyy
yyyymmdd
dd-mmm-yy
dd-mm-yy
dd/mm/yy
dd-mmm-yyyy
yyyy-mm-dd

Description​

Example​

Input Metadata:​

Example​

Additional Input:​

Example​

Output Metadata:​

Example​

Additional Output:​

Example​

Data Types:​

Input Format:​

Description

Example

Input Metadata:

Example

Additional Input:

Example

Output Metadata:

Example

Additional Output:

Example

Data Types:

Input Format: