balm_aggr_super

1. Description

Balm_aggr_super program takes list of balm aggregator output directory path as input path and list of sources. It then merges all the nth file from different balm sources and prepare 13 merged files. It then performs aggregation on the merged balm aggregator output to generate 13 super-aggregated output files.

In case of without-dim type (WODIM and CONV_WODIM): it aggregates on the basis of SubType_ID, As_on, Currency_ID, SchemeID and SLRorIRS for INFLOW/OUTFLOW files for maturity products. In case of summary file, it also includes CashflowType as the aggregator key for both maturity products and non maturity products.

In case of with-dim type (DIM and CONV_DIM): it aggregates on the basis of SubType_ID, Item_ID, Dim_ID, As_on, Currency_ID and SLRorIRS for maturity products. In case of summary file, it also includes CashflowType as the aggregator key for both maturity products and non maturity products.

2. Program location

<installation_path>/balm_aggr_super

3. Input

Click ⬇️ to download the sample data.

3.1 Mandatory Parameters

#	Parameters	Description	Example
1	as_on_date	The date for which the program has to run.	04-10-2023
2	output_path	Path to the output file.	test-bed/output/super_aggr
3	input_path	Path to the input directory. Note: input path should end with /.	test-bed/input/
4	log_file	Path to write logs.	test-bed/log.txt
5	diagnostics_log_file	Path to write diagnostics logs.	test-bed/diag_log.txt
6	data_type	Type of data to be processed and load.	DIM/WODIM/CONV_WODIM/CONV_DIM

3.2 Non Mandatory Parameters

#	Parameters	Description	Default value	Example
1	log_level	Level of diagnostics written to the log file.	info	error/warn/info/debug/trace/none
2	is_perf_diagnostics_enabled	The flag that decides whether performance diagnostics will be written to the diagnostics log file.	false	true or false
3	source_list	Comma-separated source name. Note: if no source are provided, it takes all the sources by default.	""	loans, borrowings

4. Output

This program will take n different products aggregated output, so 13*n aggregated output files as input and generate 13 merged files as temporary intermediatery files. For example product1 aggr1.txt will be merged with Product2 aggr1.txt. These 13 intermediatery files will generate 13 super-aggreagted output files following the same bucketing structure as the balm aggregated output.

Note: For without-dim bucketing structure, please check balm_aggr_m documentation🔗.

Note: For with-dim bucketing structure, please check balm_aggr_dim_m documentation🔗.

4.1 Output files-1 to 12 : 6 master inflow/outflow files each

When data-type is WODIM: If its an Inflow file, it will be uploaded in MasterInflowN table where N represents the file number. Eg: aggregated-0.txt will be uploaded to MasterInflow1 table and so on. If its an Outflow file, It will be uploaded in MasterOutflowN table where N represents the file number. Eg: aggregated-6.txt will be uploaded to MasterOutflow1 table and so on.
When data-type is CONV_WODIM: If its an Inflow file, it will be uploaded in MasterInflowConvertedN table where N represents the file number. Eg: aggregated-0-consol.txt will be uploaded to MasterInflowConverted1 table and so on. If its an Outflow file, It will be uploaded in MasterOutflowConvertedN table where N represents the file number. Eg: aggregated-6-consol.txt will be uploaded to MasterOutflowConverted1 table and so on.
Note: MasterInflowConvertedN/MasterOutflowConvertedN structure is same as MasterInflowN/MasterOutflowN. For MasterInflowN/MasterOutflowN field details, please check balm_aggr_m documentation🔗.
When data-type is DIM: If its an Inflow file, it will be uploaded in DimMasterInflowN table where N represents the file number. Eg: aggregated-0.txt will be uploaded to DimMasterInflow1 table and so on. If its an Outflow file, It will be uploaded in DimMasterOutflowN table where N represents the file number. Eg: aggregated-6.txt will be uploaded to DimMasterOutflow1 table and so on.
When data-type is CONV_DIM: If its an Inflow file, it will be uploaded in DimMasterInflowConvertedN table where N represents the file number. Eg: aggregated-0-consol.txt will be uploaded to DimMasterInflowConverted1 table and so on. If its an Outflow file, It will be uploaded in DimMasterOutflowConvertedN table where N represents the file number. Eg: aggregated-6-consol.txt will be uploaded to DimMasterOutflowConverted1 table and so on.
Note: DimMasterInflowConvertedN/DimMasterOutflowConvertedN structure is same as DimMasterInflowN/DimMasterOutflowN. For DimMasterInflowN/DimMasterOutflowN field details, please check balm_aggr_dim_m documentation🔗.

4.2 Output file-13 : aggregated summary file

When data-type is WODIM: It will be uploaded in tblProductTotals table.
When data-type is CONV_WODIM: It will be uploaded in tblProductTotalsConverted table. Note: tblProductTotalsConverted structure is same as tblProductTotalsConverted. For tblProductTotals field details, please check balm_aggr_m documentation🔗.
When data-type is DIM: It will be uploaded in DimMasterProductTotals table.
When data-type is CONV_DIM: It will be uploaded in DimMasterProductTotalsConverted table. Note: DimMasterProductTotalsConverted structure is same as DimMasterProductTotals. For DimMasterProductTotals field details, please check balm_aggr_dim_m documentation🔗.

4.3 Output file-14 to 25 : LLG level health check report

12 Health check report for each of the 6 inflow and 6 outflow files containing file level summary about the processed accounts such as success counts, failed counts, etc.🔗

4.4 Output file-26 : Aggregated health check report

The health check report gives summary about the processed accounts such as success counts, failed counts, etc.🔗

Output derivation logic:

For without-dim data :

- INFLOW/OUTFLOW files:

  It aggregates on the basis of SubType_ID, As_on, Currency_ID, SchemeID and SLRorIRS for maturity products.

  The amount field is aggregated by taking simple summation.

          Amount = Σ amount

  The InterestRate and AvgDate field is aggregated by taking weighted average.

          InterestRate = Σ (amount * interestrate) / Σ (amount)

          AvgDate = Σ (amount * avgdate) / Σ (amount)

- Summary file:

  It aggregates on the basis of SubType_ID, As_on, Currency_ID, SchemeID, SLRorIRS and CashflowType for both maturity products and non maturity products.

  The amount field is aggregated by taking simple summation.

          Amount = Σ amount

  The InterestRate field is aggregated by taking weighted average.

          InterestRate = Σ (amount * interestrate) / Σ (amount)

For with-dim data :

- INFLOW/OUTFLOW files:

  It aggregates on the basis of SubType_ID, Item_ID, Dim_ID, As_on, Currency_ID and SLRorIRS for maturity products.

  The amount field is aggregated by taking simple summation.

          Amount = Σ amount

  The InterestRate and AvgDate field is aggregated by taking weighted average.

          InterestRate = Σ (amount * interestrate) / Σ (amount)

          AvgDate = Σ (amount * avgdate) / Σ (amount)

- Summary file:

  It aggregates on the basis of SubType_ID, Item_ID, Dim_ID, As_on, Currency_ID, SLRorIRS  and CashflowType for both maturity products and non maturity products.

  The amount field is aggregated by taking simple summation.

          Amount = Σ amount

  The InterestRate field is aggregated by taking weighted average.

          InterestRate = Σ (amount * interestrate) / Σ (amount)

5. Process configuration in streamdef table:

data_type = WODIM

  .........
    "processName": "Super Aggr",
    "processId": "1",
    "processBinary": "{SH_PROGRAMS}/balm_aggr_super",
    "processArguments": [
      "--as-on-date",
      "{AsOnDate}",
      "--output-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/balm_super",
      "--input-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/balm",
      "--log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/balm_super_log.txt",
      "--diagnostics-log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/balm_super_diag_log.txt",
      "--data-type",
      "WODIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/balm_super-summary-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/balm_super_log.txt",
    "errorLog": ""
  .........

data_type = DIM

  .........
    "processName": "Dimensional Super Aggr",
    "processId": "2",
    "processBinary": "{SH_PROGRAMS}/balm_aggr_super",
    "processArguments": [
      "--as-on-date",
      "{AsOnDate}",
      "--output-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/dim_balm_super",
      "--input-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/dim_balm",
      "--log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/dim_balm_super_log.txt",
      "--diagnostics-log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/dim_balm_super_diag_log.txt",
      "--data-type",
      "DIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/dim_balm_super-summary-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/dim_balm_super_log.txt",
    "errorLog": ""
  .........

data_type = CONV_WODIM

  .........
    "processName": "Consol Super aggr",
    "processId": "3",
    "processBinary": "{SH_PROGRAMS}/balm_aggr_super",
    "processArguments": [
      "--as-on-date",
      "{AsOnDate}",
      "--input-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_balm",
      "--output-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_balm_super",
      "--log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_balm_super_log.txt",
      "--diagnostics-log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_balm_super_diag_log.txt",
      "--data-type",
      "CONV_WODIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_balm_super-summary-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_balm_super_log.txt",
    "errorLog": ""
  .........

data_type = CONV_DIM

  .........
    "processName": "Consol Dimensional Super aggr",
    "processId": "4",
    "processBinary": "{SH_PROGRAMS}/balm_aggr_super",
    "processArguments": [
      "--as-on-date",
      "{AsOnDate}",
      "--input-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_dim_balm",
      "--output-path",
      "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_dim_balm_super",
      "--log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_dim_balm_super_log.txt",
      "--diagnostics-log-file",
      "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_dim_balm_super_diag_log.txt",
      "--data-type",
      "CONV_DIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_dim_balm_super-summary-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_dim_balm_super_log.txt",
    "errorLog": ""
  .........

1. Description​

2. Program location​

3. Input​

3.1 Mandatory Parameters​

3.2 Non Mandatory Parameters​

4. Output​

4.1 Output files-1 to 12 : 6 master inflow/outflow files each​

4.2 Output file-13 : aggregated summary file​

4.3 Output file-14 to 25 : LLG level health check report​

4.4 Output file-26 : Aggregated health check report​

Output derivation logic:​

For without-dim data :​

For with-dim data :​

5. Process configuration in streamdef table:​